Gold mining probably wasn't their only motivation for being here. They also may not have been the most powerful race out there. Perhaps they were forced to pick over the meager takings in the outer bands of the Milky Way, due to territorial claims by others.
Good thinking, W.