Why does an individual have a "responsibility" to not ingest certain substances or engage in certain consensual adult sexual behavior? Those acts violate no other individual's rights. They may affect other individuals - but that's far too broad a standard for governmental action, as almost everything anyone does "affects" someone else in some way.
I think you answered your own question. Because an action "affects" someone else, there may be a need for some sort of third-party arbiter, or even proscription under the law of certain kinds of behavior, depending on the extent of the "affect". What that extent is has to be determined by the will of the people (though that is not always perfect judgement). That is what we have legislatures for. No I don't think the government should be as extensive and intrusive as it is today, but neither do I want it to simply go away. Governments must be kept in check and they must tread lightly, but they can serve a good purpose.
It is also a good idea to seriously examine the principle of "as long as it doesn't hurt someone else" or as you put it in a slightly different variation, "as long as it violates no other individual's rights." Personally, I think I have a right to drive on streets where I do not have to risk my or my family's safety because someone has decided to drive under the influence of whatever chemical influence they are under at the time. You may argue rightly that is of a different order than whether someone can ingest their drug of choice. That is true, but it is all about where we draw the line--the extent of the affect. There are societal costs to drug use and even certain sexual behaviors that are not immediately measured in a direct affect on another individual. Society has to weigh those costs and make a decision. For now the mechanism that we have to do that is through representative government.