Correct. The Bank can not force you to renegotiate your contract.
In other words, the "fixed" part of a fixed-rate mortgage only applies to the bank, since the borrower can always refinance the mortgage without paying a penalty.
In which case the Bank gets all of it's money plus the interest that's been paid during the course of the loan. The Bank isn't harmed in the least. In fact they've made a tidy little profit as the interest is front loaded into mortgage loans.
When interest rates fall, the banking industry deals with billions of dollars in refinancing at lower rates -- but they're still obligated to pay higher long-term rates on term deposits. A bank that must cast mortgages at 5% while paying out 6% on long-term certificates of deposit isn't going to stay in business very long.
This is precisely why banks have run into so much trouble in recent decades (the S&L crisis of the 1980s being a good example of this) during periods of falling interest rates. When rates decline and large numbers of people refinance their mortgages, the banks that held the original mortgages do (as you said) get lump-sum cash payments representing the outstanding balances on those mortgages. But they are getting this cash at a time when interest rates -- and therefore investment returns on this money -- are LOWER than they had been before.
This is why the banking industry has been resorting to all kinds of other sources of revenue -- including credit card interest rates, late fees, overdraft fees, etc. -- when interest rates are low.
I would imagine this is why it’s so hard to pay off stuff because everything is frontloaded with loan shark interest rates.