Asimov had another option in The End of Eternity; time travel is possible, but needs some sort of machinery - sort of a station - to do it. Therefore you cannot travel back further than the first station. That is why we don’t see time travelers - the first “station” hasn’t been invented yet.
This thread reminded me of Asimov, then I saw your post.
https://en.wikipedia.org/wiki/The_End_of_Eternity
In the future, humanity uses time travel to construct Eternity, an organization “outside time” which aimed to improve human happiness by observing human history and, after careful analysis, directly making small actions that cause “reality changes”, as well as to help establish trade between the various centuries to help those in most need. Its members, known as “Eternals” and by the roles they hold, prioritize the reduction of human suffering, at the cost of a loss to technology, art, and other endeavors which are prevented from existing when judged to have a detrimental effect. Those enlisted travel “upwhen” and “downwhen” and re-enter time in devices called “kettles”. Their rules prevent them from earlier travel to the Primitive times before the 27th century, when the temporal field powering Eternity was established, to prevent accidental damage to pre-temporal history. Also, humanity’s fate is unknown – the earth is empty by the 150,000th century, but this is preceded by a period called the Hidden Centuries from the 70,000th–150,000th centuries in which for unknown reasons they cannot access the world outside Eternity to learn more.