I notice you didn't answer the question.
Yours sounds exactly like the argument, "I'm against abortion, but I don't think I should tell women they should get one or not."
Abortion in and of itself violates a person's rights - pot smoking, tobacco smoking, and reading Marxist literature don't.
Neither does assisted suicide. Are you now on board with that?