I fly almost every week. When the door closes, the crew and pilot are in charge - of everything, including the safety of ALL 137 (or 143) passengers.
What I suspect happened is a couple kids acted up, some chaperones got involved (re: became overly aggressive with defending the kids and not making them behave) and rather than sorting through the complexity of kicking the offenders and selecting chaperones to stay with the kids the decision was made to remove everyone.
Like it or not, the captain is in charge.
I know of people getting kicked off of flights for much less.