Cancer Research UK's survey also showed that people are unaware that there are steps they can take to reduce their chances of developing cancer.
Two-thirds underestimated the proportion of cancers that could be prevented and just one in five correctly said that half of all cancers could be prevented by lifestyle changes.
Commenting on the findings, Cancer Research UK's director of cancer information, Dr Lesley Walker, said: "Even though survival for most types of cancer is better now than it ever has been, this survey highlights just how frightened we are of getting cancer
"Crucially the survey highlights a lack of awareness about how many cancers could be avoided. If people know what they can do to help prevent the disease, we hope that it will allay some fears."
The charity is launching 'awareness roadshows' today which will tour the UK to improve people's knowledge about the link between lifestyle and cancer.
Singer Ronan Keating, whose mother Marie died of cancer, has lent his support to the campaign.
"Because I lost my mother to breast cancer, I understand how frightening the disease can be," he said.
"I believe that knowing more about how cancer can be prevented or picked up early will take away some of the fear and ultimately help to save lives."
In 2003, 276,678 persons were diagnosed with cancer; breast, lung, bowel (colorectal) and prostate cancer account for over half of all new cases of cancer.
Click here to run an energy price comparison, and see if you could be paying less for your gas and electricity.