Ah, it was the first temple I was speaking of. I still can’t find the biblical reference, but I feel certain it’s there... somewhere... the second temple was destroyed by the Romans
You are on the right track. The Zadok Priests of the Essene line of Judaism continued to keep the solar calendar, even as the Pharisaic line of Judaism fabricated the lunar calendar, which of course was not given to them by God so it was never going to align properly with what God gave to Israel to keep the High Holy Days properly.
The destruction of the First Temple was how all those Jews got to Babylonia.
Prophets like Ezekiel and Daniel were included, as were Temple priests and Levites. Ezra and Nehemiah note the genealogical records they had. Only a minority of exiles returned to the Land of Israel. Not a chance any group lost track of the Sabbath.