Arthur Goldberg took the post from JFK and later regretted it; he resigned to challenge Nelson Rockefeller for governor.
He left the court and became UN ambassador for LBJ and then lost to Rockefeller in 1970.