Mill jumped on to Benthamite "utilitarianism", did he not?
Utilitarianism is an incorrect and dnagerous moral stance that has been a basis for 'collectivism' ethics in the 20th century. It could be for that reason that it was deemed harmful.
Now that you mention it (and now that I've dug further), it seems he did. I hadn't been aware of that before. Yes, that would be a problem.