The 58-year-old acknowledged at the beginning of May that he had a four-year relationship with Jeff Chevalier following a failed attempt to prevent the Mail on Sunday newspaper revealing details of the affair.
Lord Browne, who had already announced he would leave BP in July, stands to lose a retirement package worth at least £3.5 million, although it could have risen to £12 million.
And today Goldman Sachs announced his resignation from its board, eight years after he became a director when the firm became a public company.
"Lord Browne has made an enormous contribution to Goldman Sachs and our culture of teamwork, excellence and client service," said Goldman Sachs chairman and chief executive officer Lloyd C Blankfein.
"He has approached every issue with extraordinary business acumen and an absolute commitment to the firm's success.
"His ability to identify and deal with the important issues affecting our business and our people has helped guide this institution since we became a public company. On behalf of the people of Goldman Sachs, I would like to thank Lord Browne for his service to the firm and our shareholders."
Click here to run an energy price comparison, and see if you could be paying less for your gas and electricity.