This applies both to the highly skilled professionals like medical doctors and unskilled workers. You need some degree of freedom and fairness as well for the society to prosper.
If you consider the wealth (which is is different from income) you can see the countries where the few owes so much that the working people of both categories (skilled and unskilled) get discouraged and demoralized. Some great countries of the past have fallen for this reason.
Partly the reason for such "fallen countries" is that the countries are perhaps the size of one US state, and that entering another country is in fact -- entering another country with all that pertains. Quite different from moving from one US State to another. The European Union "proposals" from years back attempted to address this issue.