The point is some weird guilt by association.
Have you ever heard the old saying, a man is known by his enemies?