Do you have a preference for which side of the initiation you want to be on? If you have no preference whether force is initiated against you or if you are the instigator, you have a (small) point. On the other hand, if you prefer not to have force instigated against you, welcome aboard.
Whether you like me killing you is not relevant. What matters is the morality of the action. Whether you like it or not you have no authority to tell me what is moral. So if that morality does not come from the author of Creation then it is only a thought and holds no authority even on he who thought it.
If you truly believe that Morality is a concoction of man and you further believe men can force others to comply with full legitimacy, you must admit that some men have the right to form and force morality. That sounds allot like socialism and for a reason. Without a Moral Authority outside man, moral authority is force by man. There is no other way, you always come to that basis.