You're right! What started out as a somewhat reasonable "insurance" scheme has morphed into a casino-type betting system.
And when the house (the big banks) can't pay after a bad run, the taxpayers will step in to make everything whole again.
Without speculators there is not a liquid supply of counterparties for hedgers.