He said it both ways. Take your pick.
Matthew 12:30 - Whoever is not with me is against me, and whoever does not gather with me scatters."
Luke 9:49-50 - Master, said John, we saw someone driving out demons in your name and we tried to stop him, because he is not one of us. Do not stop him, Jesus said, for whoever is not against you is for you.
Thanks Gourmet. :)