Until the late ninetheenth century, everyone knew your vote because voting was public. This made fraud less likely and worked to discourage voting by people who knew too little about the issues to justify themselves to their neighbors. The reason the secret Australian ballot came to America -- and it was a tremendously controversial decision with conservatives against -- was so that vote buyers could not know how supposedly bought voters had voted. But now with easy absentee voting in many states we have the worst of both worlds -- vote buying is again practical, while voters are often not manly enough (sorry ladies) to publicly state their beliefs.
As far as the bill of rights in early America, check out the Alien and Sedition Acts. There is a great deal to Madison's description of bills of rights: "Parchment promises, easily made, easily broken."
P.S. Yes, I know Madison eventually said he changed his mind, just as Lincoln changed his famously low opinion of Jefferson. Politics.