The groups are calling for a Commons debate on the issue and have written to opposition politicians to gather support for the campaign.
It comes after the government announced that a further £675 million from the National Lottery fund will be diverted towards London 2012 finances.
According to the groups, that will result in the Arts Council of England losing £62 million over four years, the Heritage Lottery Fund (HLF) having its funding cut by £90 million and Sport England being deprived of £55 million.
But the government has moved to allay such fears, with a spokesman for the Department for Culture, Media and Sport (DCMS) insisting the concerns were misplaced.
"The Olympics is a leading good cause and will benefit the country in all measure of ways," the spokesman said.
"Everyone can benefit from the Games including the arts sector."
But the campaign leaders remain unconvinced that the transfer of funds, announced by Olympics minister Tessa Jowell last month, will not hit charities and community groups hard.
Stuart Etherington, chief executive of NCVO, said: "NCVO welcomed the commitment that the Big Lottery Fund's resources to the voluntary and community sector will be protected, but we are very concerned that similar protection has not been provided for charities, voluntary organisations and community groups that apply for funds from Arts Council England, Sport England and the Heritage Lottery Fund.
"I hope that the wider voluntary and community sector will support sports, arts and heritage organisations in this campaign."
John Sell, acting chair of Heritage Link, added: "We want Britain to look its best for the Olympics.
"The contribution of the voluntary heritage sector is central to explaining Britishness to visitors as well as promoting our rich and diverse heritage as venues for national, regional and local events and community activities. But these groups do not necessarily benefit from a post-Games tourism legacy.
"Diverting HLF funding from these groups, many of them run entirely by volunteers, will hold back their involvement and reduce the sense of ownership felt in the Games."
According to the government, the "difficult decision" to divert the Lottery funding was made because the Olympics do "immeasurable good" throughout the country.
It added that core funding of the arts has gone up 73 per cent since 1997 in real terms and that the Lottery was always intended to be used for important one-off events such as the Olympics.
Click here to run an energy price comparison, and see if you could be paying less for your gas and electricity.