I'm sure there was a goodly number of Southern rural white people who suffered the same fate. Most whites in the South were NOT plantation owners. Many were simply folks trying to eke out a living off land that hadn't been kept up for several years. And after the war, there were fewer men to do the heavy work.
Somehow, I suspect Scarlett O’Hara types found survival in the post-war South a good deal more challenging than their former slaves. Who do you think had grown the food before the war? Not Scarlett.