This entirely correct! The problem now is that many people make the assumption that if something is NOT CRIMINAL then it is NOT IMMORAL and therefore should become LEGAL and by natural extension becomes MORAL.
This is a classic case of flawed logic that is used too often.
At the end of the day - most of ours laws are based on a MORAL vrs. IMMORAL code that is rooted in religious beliefs. If laws and rights are not based in a large part on religion - which results in an unquestionable legitimacy, all laws and rights would flow from only the opinion of only the most powerful.