There's some truth in that statement. In this country (and the UK), you are being punished for being a member of the Middle Class. So you have a lower birth rate.
By the end of the Roman Empire, the effective fertility rate (children surviving to adulthood) was probably around 1.0–1.5, well below replacement. This helps explain why Rome became increasingly reliant on outsiders (Goths, Vandals, etc.) to fill its armies and settlements.
“demographics is destiny”