The grandfather tree thing does not work when everyone moves every few years, often to different parts of the country. The fruit trees we planted in northern Illinois are mostly grown now, being enjoyed by strangers, unless they were cut down.
Of course you are right. It’s even worse on suburban 1/4 acre lots.
But it is magical to a small child and maybe as that child grows into his or her 20s or 30s. Lots of people stay put long enough for that magic to work.
Even later in life those adults can go back to their long-sold childhood home and, if they are lucky, say to their kids “See that tree there? That’s my tree. Grandpa planted it for me.”
Your observation points out the literally rootless impermanence our cultural habits engender. Frequent relocation (either of oneself or the others around you if you stay) severs ties to family, community, place, and heritage, leaving a world of strangers. Choices do have to be made, but they always have a price.