All of science is inference. You think anyone has ever seen a graviton? How about cosmic strings? Quantum Physics is 100% inference -- the items being studied cannot be directly observed.
So explain why the Theory of Gravity (which is much more vague than TToE) is OK. Or maybe you have a problem with that, as well?
I don't know why evo's use this example because it's so bad...anyways we know the effects of gravity we see it and feel them constantly every second. We can see it's cause and effect.