Your equating "affirmative action" with restrictions on illicit drugs?
It's like this:
Some people would say the only reason one would want to end affirmative action is because they are racists, and want the chance to discriminate. They dismiss the point that some those opposed to affirmative action hold that position because of principles that have nothing to do with racism.
It is the same with the WoD. Some people would say that the only reason one would want an end of the WoD is because they are users. They dismiss the point that many opposed to the WoD hold that position because of principles that have nothing to do with wanting to use themselves.