Appeal to authority? I'm talking about the authority of the evidence. It's like chain of custody. Heresay and inference aren't valid evidentiary methods. Theory is not truth. You may call it fact; but, truth and fact are not identical in nature. That is perhaps why you might prefer to use the word fact. If you have nothing but modern evidences and infinite possibilities for causality, an inference isn't true, nor is it necessarily fact. It is a working hypothesis. In absence of capacity to falsify and exclude all other possibilities, you are left with a hypothesis at best - not a fact.
This seems like a reasonable statement, but I think you have closed your eyes to 200 years of accumulated evidence in physics, chemistry, biology and geology. We send people to the gas chamber on far less evidence. Using your standard, no criminal would ever be convicted.