Mr Ito was seeking re-election for his fourth term in office this weekend and doctors said in a press conference that two bullets had reached his heart.
Police arrested Tetsuya Shiroo, a 59-year-old man said to be a senior member of a gang linked to one of Japan's leading criminal groups called the Yamaguchi-gum.
Mr Abe said the timing of the killing, during an election campaign, made it a threat to the democratic process.
He added that the murder should never be forgotten and that and that the people should show strength to prevent violence of this nature.
Gun laws and controls are strict in Japan and attacks on politicians are very rare, although Mr Ito's predecessor was injured in a shooting in 1990 by a right wing extremist.
The motive behind Mr Ito's murder is not confirmed, but reports suggest it may be linked to the city's handling of a traffic accident four years ago.
Click here to run an energy price comparison, and see if you could be paying less for your gas and electricity.