It doesn't necessarily mean they actively bullied him.
Why would kids hang out with someone if they thought he was creepy.