I have noticed this, too. A lot of smaller families have large houses, while our family of seven has a tiny house. I thought maybe if we had less children, we could have had a bigger house. Then I looked at my children and couldn't think of one of them I would not want to have.
One thing I noticed about large families. Unless there is a dysfunctional problem, the children with large families will have close friends for life. A few weeks ago my sister-in-law was talking about how close her family is. As an only child it's something I will never know.
Probably you don't have a larger house because houses with five or more bedrooms are hard to find, at any price.