I had to allow about an hour and a half to drive to Rutland, and about 5-6 hours to Penn Station, depending on how late it was. Down on Monday and back on Thursday.
The train wasn’t always that empty. Often crowded as far as Albany, and sometimes quite a few all the way to Vermont in Ski season. And you had to reserve a seat for the week of Thanksgiving several months ahead.
Also, the tracks past Saratoga were basically owned and maintained by the freight trains, although most likely Amtrak contributed to fixing the rails and ties.
I have never been to Vermont, one of the few states I have not visited (Vermont, NH, Maine, Wisconsin and Minnesota). I hear it is beautiful.