They can't. The holders face two risks - refinance if rates drop, and default, no matter what the rates do. Indeed, it is fairly rare for a lender to actually hold the whole mortgage. Loans are generally packaged and securitied.
Thazt is the deal, the mortgage brokers are screaming now because they cannot move the paper anymore.