Yes, that makes sense to me, and that’s been my understanding. I would hesitate to make money off any image that contains an identifiable private building, even if the image is in the public domain.
If you really want people to go crazy on you, go onto some Navaho reservation, take photos and try to sell them without a permit. Yikes...! Not to sound racist..when you see them go after a photographer for breaking the rules, you understand where the term “on the warpath” comes from!