You are right about all the states that need to buy in, but it's worse because small towns will go crazy all up and down the line. In fact I've been told that the Accela (the high speed bullet trains they put in service several years ago) are prevented from going faster than conventional trains because of the rules in the various municipalities. A glance at a schedule will show that it is barely faster than the old fashioned regional trains taking the same route.
But as a frequent commuter on the line you showed I can say that your scenario of five minutes per stop is not right. More like 30 seconds to a minute, tops.
Having shunted back and forth between DC and NYC in the past for more than three decades I find it hard to believe that Amtrak stops now are of 30 to 60 seconds in duration. Those times are faster than NYC subway stops as I recall.
What’s your route?