But we do have long histories of last first frost, last frost for most places there were farmers, or cherry blossom opening dates in the imperial court of Japan. It isn't laid in our lap in degrees celcius, but it's enough to go on. There a lot of temperature proxies in the natural world too.
Inference isn't perfect, but it is a valid scientific tool.
It's great for building "scientific consensus". Objective truth? Not so much.