That's true of the Supreme Court, but it would be a horrible idea not to appoint one.
The laws the Supreme Court deal with can be incredibly complex. They need to know canons of construction, etc.
It takes a legal mind to deal with the law. And it takes a legal mind familiar with the Constitution to deal with constitutional law.
Any good reader with reasonable intelligence could figure it out. The issues in most appellate cases are very narrow. And the litigants do all the heavy lifting - they present the Court with exhaustively researched and thorough briefs, along with all the amicus briefs from other interested parties. The Court simply has to apply common sense and make a decision, then have the law clerks put it all into written form that adequately supports and explains the decision. There's not any magical alchemy going on. Sure, they dress it up in terms of precedent and constitutional scholarship, but it's all common sense and politics at the core. Don't believe that Wizard of Oz stuff about "legal minds".