Not really true. Most often the head of the household would one slaves. So if a family was composed of five or six people the numbers would look low if you considered the ratio of slaves to all free people. If you look at families or households the numbers are pretty surprising.
49% of Mississippi households and 46% of South Carolina households had slaves, and about a third of households in Alabama, Georgia, and Florida. The numbers were lower in other states. Stats here.
People could save or borrow money to buy slaves or they could inherit them. A summer home is a luxury and doesn't bring in any money. Slaves, by contrast were part of the productive economy and brought in money for their owners.
Virginia? I don’t agree with your stats at all. People in the South who do their genealogies generally find out that there were no slaveowners in their ancestry—and they are expecting to find it. The internet gives the illusion that you can find out everything, and I think the whole ancestry.com stuff is actually rather dangerous, but documentation of slave ownership is completed unless there’s a will. Many southerners attempt to trace their ancestry to prove that they have enough Cherokee to cash in on casino/tribal money, and find they can’t really prove what their families have told them.