You do that, and you get 90% of the Libertarian vote coming your way (the ones who have legal narcotics as the main issue won’t budge).
Then there are the libertarians who are just constitutional conservatives believing in minimal government to support the common good.