The fossil record does not offer an observed instance of man evolving from another species. It offers samples from which, by observation, we may reasonably infer as much. If one cannot tell the difference between direct observation (experience) and inference, how should one expect him to know the difference between science and philosophy?
What is your objection to inferring in this context?
It seems to me that you would object to historians believing anything. For instance, there is not a human alive with "direct observation" of Julius Caesar. Do you therefore argue that the prior existence of Caesar is a lie?
Police forensic science is useless by that logic. The police don't observe that a guy was murdered, they just take samples from which, by observation, we may reasonably infer as much.