You have nailed my one complaint about letterboxing, and why it was not very big until the 90's. It looks terrible on a small screen.
Remember, until the Mitsubishi 30" tube that came out in the early eighties (I think) at a cost of over $3,500 when $7 an hour was pretty good money, 25" was "big." Not much letterbox back then.
I wouldn't even say it was a resolution thing. Europe seemed to accept letterboxing for broadcast films before laserdisc was popular in America (and LD was even more of a niche market in Europe, so much so that an NTSC/PAL capable player went on the market and sold as well as the initial region free DVD players, like hotcakes).