Yep he got rid of the bank (same as the Federal Reserve today) and was the only president to get rid of the US debt completely. Amazing what can be done when you do not have the Federal Reserve, which by the way is private not federal.
Abolishing the Second Bank of the United States and paying off the national debt contracted the American money supply and had the (now predictable) effect of creating a depression, which ran for six years.