It’s because of the presidential election. That one needs time to go from the popular/ technically meaningless vote to the electoral college/ actually chooses a president vote, with time to deal with potential tie-breaking procedures/ other strange political junk before the president actually gets inaugurated. All the other offices take their schedule from that.
Consider the Bush/Gore debacle
Bush could have moved in on November 5 (or whatever day it was) and let the counts and re-counts happen
If he had lost, we kick him out ... but until then ... the winner goes in.
Why must the timing of the Presidential election affect the start of the new Congress? Prior to the ratification of the 20th Amendment, both the new Presidential and the new Congressional term started on the same day, March 4. With the ratification of the 20th Amendment, the new terms now start on different days, Jan 3 for the new Congress, Jan 20 for the new President. I agree that a new amendment would be needed, but why could the date for the new Congressional term not be changed without changing the date for the new Presidential term?
I would suggest that the new Congress start 2 days after the election. It should be possible to get from anywhere in the US to DC in 48 hours using commercial airlines. If there’s a contested election, perhaps include some remedy such as allowing the incumbent to continue to serve until the contest is decided or leave the seat open. The requirement that the new Congress be seated nearly 2 months after being elected is an anachronism from the time when travel to DC was time consuming and difficult.