If the HR gets too big it becomes even more unwieldy then it already is !
That is certainly a downside. But even if we double the size of the House there would still be 350,000+ constituents per rep.