I appreciate your modifications on the question of ownership (illegals, and the seriously mentally ill).
I would actually add another point that I'm sure you'd agree with: Even ex-felons have some natural right to self-defense. It must be balanced against their potential danger to society, but still, there's a natural right to self-defense that is not erased by past behavior.
Hard@$$ me, I only consider a pardoned citizen to be an "ex-felon". Once a felon, always a felon. Now, a felon MIGHT be an ex-inmate, and maybe even an ex-criminal.
By definition, a felon is one who has been convicted of a felony; no "ex" applies without a pardon. Same applies to "convict".
However, that is hairsplitting. As I said about "violent felons": They don't belong on the streets in the first place.
For the first century and half, and a bit more, of the existence of the country, no federal law prohibited felons who had served their time from owning the means of self defense, most state laws did not either, probably none that date to the time of the passage of the second amendment and it's state constitutional analogs.