Jefferson may have been rhetorically consistent about the tariff, but his party took a different course when it was in power. What looked like tyrannical usurpations when Federalists ran the country were apt to be taken as wise efforts to secure greater independence from Britain when the Jeffersonians were in office. Like many another ex-President, Jefferson was more ideologically consistent after leaving office than he was -- or anyone could be -- in office.
Madison was a "moderate protectionist" as President. In 1816, Congress passed the first protective tariff with President Madison's approval. The 25% basic tariff was considered "moderate." In later years Madison supported the federal government during the tariff/nullification crisis of 1830. So he can't be numbered as a consistent opponent of protective tariffs.
As Chief Justice, William Howard Taft dealt with the constitutionality of the protective tariff in J.W. Hampton & Co. v. United States, 276 U.S. 394 (1928). Part of his argument was that even the first Congress had listed "the encouragement and protection of manufactures" as a reason for legislation in its first tariff law. Those who sat in that first Congress included many who had worked out the Constitution and they recognized that promotion, protection and encouragement of industry was a valid power for the federal government implied by the Constitution.
Was the tariff a good idea for early 19th century America? Are tariffs ever a good idea? I don't know. But for many late 18th and early 19th century Americans a protective tariff was essential for the security and independence of the country.
One could survey American history and say that a national bank or tariffs or internal improvements were the first step towards tyranny. But it certainly took a long time to reach that goal. Was the growth of a massive federal bureaucracy a necessary result of the protective tariff? Could that growth of bureaucracy and regulation have been avoided if there had been no tariff? Did the fate of liberty really depend on the tariff question? And why did it take so long for the result?
One could just as well argue the other side and argue that the Jeffersonian policy would have meant a poorer, less united, more fractious and weaker America. It would also be possible to find support for that argument. The fate of the country under the Articles of Confederation would be a pretty good start. In the beginning of the republic any step taken would have had a wide variety of good and bad consequences which wouldn't always be easy to sort out.
If you go back and read what Tocqueville wrote in the 1830's, you will see that he predicted that (a) the U.S. Constitution would lead to a concentration of power, and (b) slavery was the thing which would be most likely to cause an American civil war. Blaming tariffs for big government is like blaming a cheeseburger for a heart attack.