Get rid of the secret ballot. Make all votes public and accessible as public records. Problem solved. Anyone then can cross check who voted and if they were legally allowed.
I absolutely agree with you. The secret ballot is a horror show. Have people vote publicly where you can look it up.