1. TIME. Not enough of it. Calculate the subliminal speeds those rocks travel, and you'll find they're still far too slow to make it around the galaxy. Additionally, they've had to GET VERY LUCKY and "find" a planet to capture it/crash into. Infintessimally infintessimal odds of that, and, of course, not nearly enough time.
2. Logical fallacy of BEGGING THE QUESTION. Where did the first life come from? It wasn't seeded. Because of (1) and (2), it's far more logical to conclude that life arose here on Earth, by some means, Divine or...?
As far as needing to go all the way around the Galaxy, how about within 30 light years of Earth? Plenty of stuff there eh!