That's why they've tacked on the "due process" claim. Meaning, that once he had a pass, there had to be some documented process to remove it, where he had a chance to protest and defend his right to not be deprived of "his property", namely that press pass.
That doesn’t make sense.
The pass was issued by The White House, just like the microphone was.