Yes, of course he could refuse. In fact, since the challenge is imminent, he should refuse. If he offers a prejudgment he will have to recuse himself.
Rush talking about this now.