Both of those are definitely factors. The tariff was raised and there was inflation (although nothing like what the south saw--Union inflation was pretty much in line with wartime inflation during the World Wars). Nonetheless, the loss of the south during the war did not, in fact, cripple the economy of the north, so all claims that it would do so were false.
I looked up the tariff and inflation rates and calculated the net imports in 1860 dollars.
Tariff rates are from Table 1 of Taussig's The Tariff History of the United States, 1910 edition: Taussig
Yearly Customs income is from Table 3 of Taussig. Same link as above. Actually, income from collected import duties was a bit lower than Customs income, but I'll use the higher figures since you had posted those rather than duties collected.
Inflation rate was from Inflation Rates. I've seen higher rates than these posted for the period, but I'll be conservative and use these.
After applying the tariff rates to the revenue to determine the value of the imports, I then adjusted the number by the inflation figure. I find that the value of imports to the North relative to the 1860 import value was:
1860: 1.00
1861: 0.82
1862: 0.50
1863: 0.52
1864: 0.54
1865: 0.38
Seems that the war did have a substantial impact on northern imports after all when expressed in 1860 dollars. Please correct me if I've made an error in my calculations somewhere.