That’s what concerns me.
Now, one can rightfully say that Jesus associated with sinners and tax collectors, but Jesus did not invite them into His inner circle until they demonstrated a repentant heart (Matthew).
So basically, the sinners can to him because their ears and hearts were at least open to repentance. Of course, the Pharisees missed that point. I think we need to point this out when people use the argument you referred to. It's not like Jesus didn't care about them - he came for sinners and tax collectors to preach, not to hang out with.