The right to swing your fist ends at the other guy’s nose.
That was the exact phrase my father used when discussing rights.