You are correct that Mill deserves admiration, or at least a helthy level of respect. But I believe Earley is right that there are many people who've mistaken the Founders for proto-Mills.
The founders believed in liberty, but had standards (they were still Kantish on universal values).