$165K with a wife and a couple of kids is not a lot of money for DC. You can likely make it work if you have preexisting wealth but if not then the wife is going to likely be working as well.
Maybe if you buy a a house in Old Town or McClean, or a condo right downtown. But I can't see anyone having trouble with that amount on Capitol Hill, Southwest or Columbia Heights, or anywhere in Fairfax County.
I think Justice Roberts is out of touch with how most Americans live.