They are right. Silence implies consent.
BUT -
The respondent can send the panel a letter responding to their decision and tell them to do something anatomically impossible with it.
THAT will hold up in a court of US law, and the respondent can go on about his business.
:-)
On what legal grounds would it "hold up?" I'm afraid that, at best, you are uninformed on this subject.