Yes, if you are a nonviolent felon, your punishment should end when you have completed your sentence. Further denial of rights is an ex post facto punishment and violated the double jeopardy prohibition in the Constitution.
I would argue that even violent criminals should have rights restored once a custodial sentence or legal restitution is completed. If they are OK to be let out of jail they are full citizens.
Notice, there are a lot of crimes where the criminal should never get out