I suspect they weren’t deliberately set loose on the barrier islands from Assateague and Chincoteague on down to Ocracoke Island. They’re still there, known as Banker Ponies in NC. Their origin has long been attributed to shipwrecks, there are shipwrecks including Spanish Galleons going back to the earliest records kept.
I would not have cared to travel on that particular ship BUT I am sure they dined well.
There were thousands of private ventures that were never reported and show no evidence here of ever having happened, but there are some extensive reports about some of them.
I suspect the death rates were spectacular for those 16th century voyages ~ but you would have had people coming from every Hapsburg land ~ so they were already here to some extent when your more familiar Brits and French arrived.
As a side venture in looking for these earlier settlements I've been looking for practical connections between the Swedes and the Spanish and there do seem to be "such events" and business deals. I guess they were just too far apart to cause each other trouble, but the Swedes were a house afire in pursuing improvements in common technology ~ better boats, better firearms, better steel, better armor, better machines ~ new designs.
As the Spaniards found out America was a huge place and it took everything they could to do anything at all.