It doesn’t work like that.
Lewis Powell and William Rehnquist were sworn in January 7, 1972
Clarence Thomas was sworn in October 23, 1991.
Neil Gorsuch was sworn in April 8, 2017.
Justices are sworn in mid term quite often. The court can reset and rehear for reargument cases that were tied prior to a new justice taking office.
Yes, they can be sworn in mid-term but they can’t vote on the cases that have come up during the initial hearings. Essentially you’ll have a 4-4 situation on the court.