In such a context, the “refusal” would seem to mean very little if it’s commonplace.
I wouldn't say commonplace but I can think of at least three Presidents who were not sworn in on Bibles and I'm sure there are a number of Congressmen and Senators who weren't.