Congress and the president can hold them accountable. Congress has the power to require testimony. It can also change any laws regarding the Fed. And the president has the power of appointment. It would also help if the press didn’t grant infallibility to central bankers.
Yes, they could.... if our elected officials had a collected economic IQ higher than a typical woman’s shoe size.
Trouble is, they don’t. I’ve watched most all the Congressional Humphry-Hawkings hearings over the past six years and have to say, the vast, vast majority of those firing questions at the Fed Chairman are... well, not interested in asking pointed questions. They’re asking questions to make themselves look good in front of the cameras, not solicit real monetary policy information.
And as for the press — their adoration is, IMO, the result of the Cult of Greenspan. Before Greenspan, the Fed Chair caught plenty of heat from the press, but by and large, most people didn’t know much about the Fed and the Fed chair didn’t jet around the world, shooting his mouth off at ever economic PR opportunity.