How so? Oil is fungible, if it goes elsewhere we'll go elsewhere too. No net loss. Big deal.
The other place it could hurt us is in with our imports of already refined gasoline which Venezuela also exports to us.
Both crude and gasoline are fungible products, so it wouldn't affect world prices, but it would increase our transportation costs to bring both here, and we'd have to retool our refineries.
It's not an entirely empty threat.