Do you have a professional do your taxes?
If you are self-employed, unless you meet some criteria I've never heard of, you are supposed to be paying the full 15% (employee and employer's share) payroll tax, which is for Social Security under the Self-Employment Contributions Act on IRS Schedule SE.
Maybe you have zero income?
Now I guess you could be taking the tax as a business deduction AFTER paying it. Maybe that is what you are referring to. But you do pay the tax first.
If you truly are not paying the Social Security (payroll) tax, your Social Security "contribution" form you get every year will show you getting zero credits.
Yes, I have zero credits on those social security forms, all my income is rental income that is handled differently than other types of self employment income. I pay no social securiy of medicare tax on that income.
I will be able to get social security and medicare though (just like a stay at home mom), because I will be able to claim it as a spouse on my wifes plan which she does contribute too through her salary.