No. No you don't. Not even Lincoln.
President Lincoln wrote to the new governor of Louisiana in 1864, Michael Hahn, asking him to consider a provision in the new LA state constitution to allow blacks to vote. President Lincoln said this would help keep the jewel of liberty in the family of freedom.
A year later, President Lincoln came out publicly in support of the franchise for black Union soldiers. That is why Booth shot him.
It is also a simple fact of history that blacks -could- vote in five northern states in 1860.
Walt