Yes, well, FoxNews thought so too, poking fun at his use of twitter and saying that he would appoint his twitter followers to his cabinet. IOW, FoxNews said his actions are, as you say, "looking frivolous." So he decided not to put up with it. People will make of it what they will. Some will say he's cutting his own throat, others will see it as decisive and looks like leadership. Some other network will give him airtime, and FoxNews loses nothing, they still get to have a debate. Win-win!
IMO its clear his internals are telling him that for whatever reason it would be better for him politically if he not attend.