They studied brain scans of 1,839 people aged between 34 and 88 from the Framingham Offspring study. Participants were separated into groups based on how much they drank, ranging from non-drinkers to heavy drinkers.
People were branded as heavy drinkers if they consumed more than 14 alcoholic drinks per week.
"Research has shown that there is a beneficial effect of alcohol in reducing incidence of cardiovascular disease in people who consume low to moderate amounts of alcohol," said study author Carol Ann Paul.
"However, this study found that greater alcohol consumption was negatively correlated with brain volume."
The study found that people who had more than 14 drinks per week had an average 1.6 percent reduction in the ratio of brain volume to skull size compared to people who did not drink.
It also revealed that women saw a greater effect on their brain volume from drinking than men did. The negative impact on brain volume appeared to be greatest in women in their 70s.
The research was presented at the American Academy of Neurology's 59th annual meeting in Boston.
Click here to run an energy price comparison, and see if you could be paying less for your gas and electricity.