"I've often wondered at what point a line might be crossed..."
You can see where a whole line of points can be crossed, if you use a 2 dimensional model of political ideologies. (There is no perfect model -- but, a 2D model is far better than the usual 1D "spectrum" model.) Here's one such model.
Your libinlaw and you are somewhere over the line toward libertarianism, at about the point where local police acquire tanks. Not very long ago, that point would have been somewhere on the "authoritarian" half of the model.