Then they have to pay it, attempting to leave without doing so constitutes theft, and many states allow businesses limited detainment rights against individuals believed to have engaged in theft.
My suspcion is the restaurant is well within its rights for what it did.
I agree mandatory tip on a group of 5 is a low number for it to kick in, however its policy and posted, the family could have gone elsewhere knowing this, but chose to eat there, so they have to pay it.
I would usually fight for parties of 8 or larger just because I knew if they liked my service I would usually get a monster tip because they would tip on top of the 15-20% already expected of them.