Except that the life expectancy for women is higher than men, and the life expectancy of minorities is usually less than whites. And single people have a lower life expectancy and on average lower incomes than married couples. And lower income people have lower life expectancy than the middle class and wealthy.
So while overall your proposal is theoretically revenue neutral, in practice it exacerbates what is already a wealth transfer from lower income males and minorities to middle class and wealthy white women.
Being married to a white woman and having studiously pursued a career that does not contribute to social security until I achieved financial independence, I’m OK with this. I don’t have any illusions that Social Security will ever be moral or fair, because in the end it’s just theft coupled with a biased welfare program.