We are talking about inside the home defense.
Rifles are better for long range. For instance, like hunting Deer.
I partially agree with your statement about inside the home.
The only downside to a rifle I can see is use in urban environment where you have to worry about where the round goes after it has blown open the perp in your house. Not necessarily good for your neighbors in the next apartment or house right next door. In those cases a shotgun or pistol might be better suited.