I see your logic, but think it a stretch to say that all rifles are of military origin because all firearms derive, at some point, from firearms that were designed for military use.
Clearly, some firearms are designed specifically for hunting and not for military use, even if they can be used militarily.
Sure, the early matchlocks and hand cannons were first designed for military usage, and all modern designs derive to some extent from them.
But it is rather like saying that all knives are military derivatives.
I think we can make a useful distinction about rifles that were designed for military usage, and those that were not. Many were designed with both purposes in mind. For a lot of designers, the military market was the brass ring to aim for.
On the other hand, I cannot think of a single double rifle manufacturer that tried to get military contracts for their rifles.