Two of the 17 injured babies are in a critical condition while the others remain stable.
A brave nurse who tried to save the children is also being treated in hospital for burns to her face and hands.
The blaze started at 06:00 local time (04:00 BST) local time at a bedroom of the Ljubica Ivezic orphanage and gutted three other rooms.
Firefighters managed to rescue 23 babies from the three rooms and extinguished the fire after ten minutes.
Biljana Jandric, a spokeswoman for Sarajevo Kosevo hospital, said: "All the children we hospitalised are babies.
"One baby was dead on arrival and for the other four [victims] everything was done but we could not save them. Seventeen other babies and a nurse are also hospitalised."
An eyewitness said: "These children have such bad luck; first they are orphaned and now as babies they die in fire."
Police have launched an investigation into the fire at the orphanage in downtown Sarajevo which houses over 100 children.
Click here to run an energy price comparison, and see if you could be paying less for your gas and electricity.