I think you'd be surprised at the stupid and trivial things that are now considered "felonies".
If your freedom of movement is not being restricted (i.e., you are in prison), the government has no legitimate power to remove rights you were born with.
You make a very important point. ‘Felony’ is usually only tied to the length of prison time associated with the crime, not the type or level of violence of a crime. The length of prison sentence associated with a felony is usually one year, though it may vary depending on the state. But as you point out, many non-violent as well as blue collar crimes have sentences over a year, thus getting the ‘felony’ label. For these non-violent crimes, Once a person has paid their debt to society, all rights should be restored.