We always shopped for and got zero-cost refis. No points and no costs. We refi’d several times and snagged a 2.25% rate a few years ago. I found the ridiculously low rate on bankrate.com and it was with a small boutique bank in Ohio. They were easy to work with and we pushed paper electronically. We never met anybody in person. People on FR thought it was a scam and we’d never really get that rate, but we succeeded. A lot depends on finding a lender who is eager to deploy money.
Refinancing takes a bit of work, but is worth it. Don’t think of it as a one-time event. Shop for the best rate. Don’t be concerned about working with an out-of-state lender or doing everything via email and internet. Look for low- or no-cost deals. Preserve your cash and don’t pay points or costs. Years ago, we used a mortgage broker (again no cost) to find us a low rate and zero costs.
All of my mortgages were with local credit unions. I always found them to be the lowest rates. In addition, those some of those same credit unions did NOT sell the note to Fannie or Freddie. IF they did, they kept the servicing in house. I always sent the payment to the same place.