I see your logic, but think it a stretch to say that all rifles are of military origin because all firearms derive, at some point, from firearms that were designed for military use.
Clearly, some firearms are designed specifically for hunting and not for military use, even if they can be used militarily.
Sure, the early matchlocks and hand cannons were first designed for military usage, and all modern designs derive to some extent from them.
But it is rather like saying that all knives are military derivatives.
I think we can make a useful distinction about rifles that were designed for military usage, and those that were not. Many were designed with both purposes in mind. For a lot of designers, the military market was the brass ring to aim for.
On the other hand, I cannot think of a single double rifle manufacturer that tried to get military contracts for their rifles.
There are a lot of avid hunters that are union member Democrats that hate the 2nd amendment and would be thrilled if we could ban all "military weapons", including pistols, as long as they can shoot some ducks/doves/deer whatever. We need to fight against the sporting purposes rationale whenever we can while reminding these Fuds that their toys are going to be banned next.