Since the Whig Party was founded by those opposed to Jackson, it shouldn't be surprising that its members opposed his policies.
Andrew Jackson did some admirable things before his Presidency, such as volunteering to fight in the American Revolution as a young teenager and beating the British at the battle of New Orleans, but there is lots to criticize in his Presidency, beyond his Indian policy.
The original Tennessee constitution of 1796 had not barred free black men from voting. The new one adopted while Jackson was President did--but I don't know if he had any influence on that decision at the state level.
The Cherokee Trail of Tears happened under the next President, Martin van Buren (also a Democrat), but it was a consequence of the Jackson policies.