But he did not return as expected at 18:00 BST that night and police have issued warnings to members of the public about approaching him.
Varga is said to have a condition that makes him "potentially violent without medication" and has previously impersonated a journalist.
Police say Hungary-born Varga may be suffering from excessive salivation and shaking as a result of missing his medication and suggest that he may attempt to conceal this fact by hiding his hands in his pockets.
Varga was convicted of murdering an employment manager he had a disagreement with in 1968.
He was released on life licence in 2005 but has now had that revoked and will be taken back to prison once captured.
Varga was said to be wearing a red shirt, light-coloured trousers and a beige suit jacket when he left the facility and was carrying a dark-coloured sports holdall, a black laptop-sized bag and a brown laptop-sized bag.
His details have been distributed to ports and airports amid fears he may attempt to return to Hungary or fulfil a desire to visit the US.
Detective Inspector Mark Daughters, who is leading inquiry for Surrey police, said: "We are taking the direct approach of warning people as we believe Mr Varga to be potentially dangerous.
"Surrey police officers and colleagues in other forces are working very hard to locate him. I would ask people who see a man matching his description not to approach, but to call us immediately on 999."
Click here to run an energy price comparison, and see if you could be paying less for your gas and electricity.