You keep getting hung up on the word "fact". Comfort yourself in knowing that that the word fact rarely appears in the literature. Just use a filter so that when people use the word fact in a layman's discussion, you know that they mean "know it to be true as much as we can know anything to be true in science". That should help you with your problems.
That is an excellent rule of thumb. Science can get a lot done without conjecture in certain directions and degrees, and yet it must rely upon conjecture to press ahead.