I'm afraid there is. Libertarians do not call upon anything higher than the individual in their moral reasoning. That isn't good enough. Our laws much reference something higher than ourselves to be consistent. Otherwise what I believe is no better than what you believe. In addition, libertarians hold ot a Rights of Man argument that, in their reasoning, has nothing to back it up.
All you're saying is that if I don't agree with your religious views, I can't make moral judgements, because morals MUST come from YOUR God.
Nice try. I don't buy it. And I'm not convinced your pipeline to God is any better than mine. Care to show me why yours is better ?
Morals can also be established by other means - harm to one's self, or another, for instance. This doesn't require ANY religious belief - just a little thinking.