I suppose an anarchist could answer, the same people who stop the government from running a protection scheme like the mafia, there'd be a lot to that, but the government can be restrained by the public, at least in a republic, while a protection agency, if it starts acting like the mafia, will put itself outside the market (which is the check they think will act against misbehavior), i.e., will coerce people to pay, without necessarily providing a service. So what you'd have is a small state, and a very despotical one. Unless the agencies get together and invade small states like that, there'd be a lot of reinstatement, and not in a nice way.
You want to gamble ? A system is needed. The mafia provides the system. If they decide to kill you, who is to stop them ?
If you believe that, I have a ..., never mind. I can't take advantage of the gullible.