Almost everything anyone does "affects" someone else; the proper test for government force is the narrower "violation of rights" (cf. the Declaration of Independence), which pot sale and use are not.
Your response if stupid. Of course actions have consequences, some actions are known to have worse or higher probability of bad outcomes. Painting my window frames is not likely to harm anyone. smoking drugs that alter ones mind state is however and doing other dangerous things ( driving, painting your window frames) increases risk to others. If you can’t find waldo in this equation, you may not ought be on your own.
But hey, we all set our own standards on FR...