What power did A&E have to remove Phil from the show?
There are plenty of contracts that forbid certain types of speech or behaviors of celebrities so that they do not besmirch the name of the parent company. Phil already had YouTube videos out there prior to the tv show that told A&E what he might say. Which is why they had a minder for him in the interview.
Phil had the right to say anything he wanted in the interview - freedom of speech is fine. Apparently A&E has the right to remove him from the show when he becomes an embarrassment (their point of view).
and people have the right to boycott and try to punish A&E too
Look you don’t seem to understand the English language. Let me clarify. A contract that has language violating the law is an invalid contract. A contract that infringes religious speech is invalid per Title VIi of the the US Civil Rights Act. Nobody can sign their rights away to an employer.