Well sure, if you call 200 million dollars per year (in 1860) dollars, a "drag" on the economy. Slavery was paying for the vast bulk of Federal government because all of those tariffs came from importing European goods in exchange for American exports, 75-85% of which were produced by slaves at that time.
Slavery is incredibly inefficient component of an economic system it is factored into, and when placed alongside a capitalist system, it cannot complete. Back then, and with an agricultural based economy, that was probably the only thing keeping it going.