When Roger Maris hit 61 home runs, beating Babe Ruth's record of 60, a lot of people wanted there to be an asterisk next to his name because he had not hit the 61st home run in the first 154 games of the season.
In the original 8-team leagues, you played each of 7 opponents 22 times, for 154.
When they expanded to 10, you played 9 opponents 18 times each, or 162. It’s stayed at 162 since, despite ever-increasing layers of playoffs that have occasionally caused the season to extend into November.
There is nothing any more sacrosanct about 162 than about 154.