20 armed committed wolves can easily take out 80 docile sheep.
Each one of the terrorists in Paris killed 15 innocents (on average.) That is considering that they used a very simple tactic of running into the crowd with guns blazing. More sophisticated attackers (like Breivik) can kill far more.
Besides, if those photos of "refugees" are in any way representative, it would be hard to find 20% of non-terrorists among them. They look like ISIS soldiers taken from the front lines for another duty. There are only a few women and children in each photo. Most likely nobody can leave Syria without going through a filter that is controlled by ISIS - and we can imagine who they will let through and who will be sent back.