There are crimes against all of the things people on drugs do that "bother" other people.
Sex is legal, but few people have any problem with laws that require that it not be performed in public venues, nor in places where it will be seen/heard by those who do not with to see/hear it. Why could drug use not be regarded in the same category?