I think you are referring to the Roman Empire; I can't think of another state in antiquity that took such a latitudinarian view of other people's belief systems. But I think you confuse mere broadness or size of the polity with the consequences of conscious policy. The Romans were systematic in their acceptance of extant cults where they found them (even importing them; a large temple of Isis stood near the Pantheon), but they were not indiscriminate in their acceptance, as the Christians found out to their sorrow. Paul of Tarsus was tried for spreading the beliefs of Christianity, and the complaint of the Jews against him was in part that he was spreading mere superstitions and cultic practices that were not accepted, or acceptable (enter here several canards about Christian practice), within the broad guidlines of Roman policy.
Notwithstanding the prejudice of modern historians and students of politics against "particularisms", I don't think you can correlate the size or multiethnicity (imperial spread) of a state with toleration. The Soviet state extended through what, eleven time zones? And yet it was an intolerant state with regards to belief and opinion.
Madison wasn't arguing for size per se, but for a federal system that diffuses intense local conflicts over a larger and more heterogenous area. The separation of powers would make it harder for local disputes to bulk so large on the national scene or for national conflicts to disrupt local politics. The late 20th century system didn't reflect this perception, but neither did the Articles of Confederation.
As between unionists and secessionists, it's not clear to me that the secessionists were closer to the Madisonian balance than the unionists. I'd say the reverse was true. Of course if you had no federal government, you wouldn't have had such an imbalance of federal power in the next century, but we make our choices on present conditions. We may look generations into the future, but we can't anticipate all contingencies and shouldn't throw away something that works because of what it might become if our decendents don't have the virtues we do.
This is a living debate, since there are those even now, who see the US breaking up along regional lines. Part of the dispute here is about whether a divided or fragmented America would be more like those peaceful and prosperous Northern European city states or more like the contentious and tumultuous Italian city-states. What would happen now is anyone's guess, but the talk of absolute state sovereignty, distrust of industry, reliance on single crop agriculture, racial questions and slavery makes me expect that the Southern states of 1860 would have been in the unfortunate category had they truly tried to go it alone. You may dislike the chicanery and economic empire building of the Gilded Age, but they did direct the energies of the ambitious away from politics towards technology and business. Where this path isn't open, the ambitious young turn towards coups and political mischief. It's also possible that the Confederate government would have tried to keep the states in line and the same conflicts would have developed as we have known since the Civil War, as the new national elites sought to use the government for their own projects.
Not sure about ancient history, but the empires of the Hellenistic period allowed Greeks, Jews and others to move throughout the world, so there must have been some openness to other cultures. The opportunity for different peoples in pagan empires to just add each others gods to their own must have helped as well. As you note, such empires could be hard on those who weren't polytheists.