However, the questionable judgment aside, this has nothing to do with the first amendment which has never covered defamation.
Your assertion allows for all kinds of exceptions to the 1st amendment, depending on a judge's whim. If a judge rules that the word "irresponsible" can be "defamation per se," as in "Pelosi is irresponsible," that doesn't have anything to do with the 1st amendment?