I think the bigger question here is why does the IRS get access to this info in the first place? Why are my mortgage payments a matter of public record?
What’s reported to the IRS by lenders is mortgage interest, and since many people do claim the mortgage interest deduction, it is reasonable for the government to get such info from banks to check that interest deductions are reported correctly. What I am very wary of is the government using the mortgage interest data for other purposes.