I’m not impressed. The correct answer is: we don’t know what caused past warming because we don’t know what the temperature was before it was first measured.
But we do have long histories of last first frost, last frost for most places there were farmers, or cherry blossom opening dates in the imperial court of Japan. It isn't laid in our lap in degrees celcius, but it's enough to go on. There a lot of temperature proxies in the natural world too.
Inference isn't perfect, but it is a valid scientific tool.