Both O'Connor and Thomas directly address your observation in their dissents. They do a better job than would a brief rejoinder from myself.
O'Connor's standard seems to be if it benefits the poor, or poorer, it is OK, and if it benefits the rich, or the richer, as she sees it through her mind's eye, it isn't.
Do you really find such an analysis persuasive as opposed to well, embarrassing?