That's actually the almost the entire problem. Likewise, one mustn't confuse intelligence with learning. I would guess (based on pretty good evidence) that the ancient Egyptians were every bit as intelligent as we are today, but they just didn't know as much (not counting what we have forgotten.)
OTOH, we do know that there are heritable and selectable "intelligence traits," as any dog breeder can tell you, as evidenced by the suitability of certain breeds for certain tasks. So it's not at all implausible to assume that the differences between, say, the Egyptians and the folks in equatorial Africa were due to something more than learning.