It was under the Bill of Rights. Slavery in the states wasn't ended until explicitly outlawed by the 13th Amendment.
And the 14th Amendment applied, among other things, the Bill Of Rights - and hence the 2nd Amendment - to the states as well. If SCOTUS decrees the 14th doesn't apply after all (per the 2nd), then the 13th doesn't apply either, and pretty much the whole "Constitution" contract gets flushed.