I would be more worried about populism controlling our financial sector. As it is now a non-elected private body is helping to ruin the economy and the financial sector. All the while personally benefiting financially from the havoc they create.
Congress still retains oversight over the Federal Reserve as is confirmed by the hearings before Congress. Congress has oversight of many things and they are all going to crapper because Congress does not do what they are supposed to do.
I suspect that Congress ceded power to the Federal Reserve so members of Congress could avoid blame for financial problems - so much for Congress being beholden the people. That is true but just because someone else allowed Congress to skip its duties does not mean we should also allow them to continue in this.
If the recent polls showing Congress' unfavorable ratings being so high and if large numbers are returned to office in November, what does that tell you about Congress being beholden to the people?