That's because Paine got weirder and weirder as time went on. The Fathers were embarrassed by his outspoken support of the French Revolution and the way he spoke out against Washington.
John Adams never liked Paine. He always considered him too radical and called "Common Sense" a "crapulous mess".
Paine appears to have made his religious leanings well known before the revolution, and it is submitted that "a majority of the founders turned on him" because of those beliefs. Thomas Jefferson was president at the time, and appears to have publicly supported it, so Paine wasn't alone in that regard.