Many patriots at the time opposed Jackson's Trail of Tears. Rep. David Crockett, whose grandparents had been murdered in an Indian raid before he was born, was Jackson's main public opponent in this. The Indians Jackson forced to move had long been abiding by a treaty, had grown civilized and prosperous, and were living under a tribal government modeled on our Federal one. It was a mystifying campaign by Jackson that was purely political in motivation.
Jackson's response to the controversy was to pour money into a campaign to unseat Crockett in Congress. Crockett lost, went to Texas, and died a hero at the Alamo.
It's thanks to Jackson that Florida is part of the United States. Jackson won one of the most decisive victories against the Brits during the War of 1812. And, once again, he was an ideologically significant figure because of "Jacksonian Democracy" and opposition to Central Banking (a mixed bag, but important). There may be other historical figures more important and worthy than Jackson to be on the $20, but Harriet Tubman isn't one of them by a long shot.
One final point: I oppose this move to remove Jackson for the same reason that I oppose removing monuments to Confederate commanders. Not because I’m particularly dedicated ideologically to either Jackson or the Confederacy, but because if you let the PC left tear down the memories of Jackson or of Confederate commanders, that will embolden their broader agenda. The PC Left has no more use for George Washington (another “racist slave owner”) than they do for Robert E. Lee or Andrew Jackson. The latter are just easier targets for them to tear down in their march forward. If we let them replace Jackson with Tubman today, 20 years from now they’ll be replacing Washington with Cesar Chavez.