A lot of the gold and silver was already mined when the conquistadors got there. They may have STOLEN it, or at least traded it for trinkets, but they didn't enslave the natives to mine it. Unlike in the Caribbean islands, where they DID makes slaves of the natives to produce sugar cane and tobacco.
Native Americans made lousy slaves.