If he committed no crime there should be no punishment of any sort. Monetary damages alone should be forthcoming in a civil matter.
There are criminal infractions and civil infractions. What you are saying is that slander should not be punished. Sexual harassment should not be punished. Breaking a contract should not be punished. People can basically do or say whatever they want and as long as it isn't criminal there are no protections from it.