The Southern mindset of the time was perhaps best expressed by Louis Wigfall, a Texas senator, in a conversation with William Russell, a correspondent of The Times of London shortly after the rebellion began: "We are an agrarian people; we are a primitive people. We have no cities - we don't want them. We have no literature - we don't need any yet. We have no press - we are glad of it…We have no commercial marine - no navy - we don't want them. We are better without them. Your ships carry our produce and you can protect your own vessels. As long as we have our rice, our sugar, our tobacco, and our cotton, we can command wealth to purchase all we want from those nations with which we are in amity, and to lay up money besides."
So why weren't there entrepreneurs or industry or manufacturing? Because the southern aristocracy was making more than enough money doing what they knew, investing that money in more slaves and more land to produce more agricultural produce for export.
Sure a few planters got rich but the army the south fielded were often unshod and ill-equipped. Slavery cost the south its economic potential and retarded the economic growth of the region for more than a century after the war.
The South built it's economy on slave labor. When that labor was gone then there was no substitute, so they did suffer as a result of losing their rebellion. But prior to the war the South was one of the richest areas of the nation, with a per-capita income far in excess than that of the North.
Due to the small population, the wealth of the few obscenely rich planters may have resulted in a higher per capita income average but the wealth of the south was insignificant compared to the north. As you said, without the slaves the south had no middle class to replace these lost workers. An asset that evanescent is not an asset it's a drain.
With slaves they could never establish the capitalist elements of true wealth, capital, factories, banks and other businesses. Farmland you can't work because of a lack of labor is not an asset, it's a fallow field.