That's not an impression, that's a fact.
They WERE one of the wealthiest colonies in the British Empire back when the Brits were in charge. Ditto for Haiti and the French Empire when the French were in charge.
What you suggest has a real life example in Liberia. It was a (relatively, by African standards) a wealthy functional independent county from the time it was settled by freed American slaves in the 1850s until it decided to revert to traditional tribalism some 120 years later.