A minimalist government (i.e., a legitimate, just one) can really hold a man responsible for "doing wrong" only to the extent that his wrong-doing can be shown to be injurious of the life, liberty, and property of another person.
Alas, this being the case, it is also true that evil behavior (even snorting cocaine, or a suicide attempt, for two examples) is definitely harmful to other persons, whether they are those who love us, those who pay taxes to clean up our messes, or those whose lives we affect in future generations by the society to which our influence is detrimental.
We live interrelated and interdependent lives. That is of the truths self evident.
Wow, that's the same kind of expansive rationale that big government types use to argue that a man can be prohibited from growing carrots in his own backyard garden under the Interstate Commerce Clause.