Yes, I think the point he was making was the potential for a double standard - if a hate group / violent group associated with the left wing of politics were planning the rally they may not have blocked it even without an invitation.
Spencer spoke on the A&M campus last year.
If an Aggie student group were to invite him to campus, the A&M administration would have difficulty stopping it.
But for now, no Aggie wants him on their campus.