“The point is – Trump needs to think long and hard before snubbing the voters by refusing to debate. It has the air of arrogance, and in some sense can look cowardly.”
That would be true if there were any credible candidates, besides Vivek, running. Also, Trump has his track record in the White House to fall back on, and while that record is far from perfect, his supporters primarily blame the GOP for the problems.
Keep in mind, Trump supporters only make up about half of the GOP and about 20% of the voting public at large and while they may make excuses for all of Trump's shortcomings, the rest of the voting public does not.