Slavery was hardly an economic drain, rather it was what made rich Southern planters rich southern planters. The 4 million or so slaves in the U.S. in 1861 had a total value of upwards of $4 billion.
The South had as much of what we would call a middle class as the North did. And most slave owners where probably part of it. Most slave owners were small time, owning 5 or fewer. And many, perhaps most slaves never saw the inside of a cotton field, either. They were cooks, butlers, maids, grooms, gardeners, what have you. Thomas Jackson, a university professor and as solidly middle-class as they come, owned as many as 10 slaves at one time, buying them and selling them as the need arose.
Sure a few planters got rich but the army the south fielded were often unshod and ill-equipped. Slavery cost the south its economic potential and retarded the economic growth of the region for more than a century after the war.