Well, Jefferson would disagree. Freedom of political association is an absolute Right that Free men and women possess.
My own contention is that Lincoln was a monster. The more I read of him, the more my opinion is confirmed.
Would he?
"If there be any among us who would wish to dissolve this Union or to change its republican form, let them stand undisturbed as monuments of the safety with which error of opinion may be tolerated where reason is left free to combat it." -- Jefferson's first inaugural address.