I knew a guy with probably 10's of millions of dollars of rental properties and his advice was do what you have to but in the end have at least one house that you live in that is paid for in full.
Intelligent use of leverage can be a good thing. The "intelligent" part of that equation seems to be lost on many people.