Two major factors that will slow the adoption of robots are vandalism and theft. And an increase in robots should, in theory, create an increase in skilled service technician jobs to create and maintain them. Of course if too many jobs disappear, there will be an even more natural factor limiting the expansion of robots -- there will be no need for a robotic McDonalds because there won't be enough people with paying jobs going to McDonalds to justify them.
That said, there is a line in the Original Star Trek episode "The Ultimate Computer" where Dr. Daystrom claims that "Computers will free man so that man may achieve." That always makes me wonder, "Achieve what? And what about the people who don't want to achieve? What are all those idle hands going to do and who is going to feed them?"
For me so much it isn't about the top half of the gene pool, but the lower half. What exactly purpose will they serve in an economy. If you have an IQ of 80, but are a hard worker, you can mop floors today. Those who are smarter, more aggressive can do something better. When robots come into full force, the smart people can work on source code, design, sales, etc, that is what always happens when technology comes.
The key difference I think here though is, what happens when the technology is created specifically to get rid of the lower functioning people?