If there’s a formal setting involved or if the PhD person is acting in their professional capacity, then addressing such a person as ‘Dr.’ is appropriate, IMHO.
Outside of those conditions, it should be optional, and certainly the addressee shouldn’t be insisting upon the reference... that becomes self-serving and arrogant-sounding rather quickly.
I once worked for a woma who had a PhD in Math from a top flight mathematics department. A great accomplishment one where someone should be proud. However she took the pride to an extreme. She legally changed her name from “First name” to “Dr First Name” so the “Dr” had to be on all official documents. All in all she wasn’t a bad person to work for, all you had to do was acknowledge her “self-proclaimed genuis”. It got to be routine, sort of like steppeing over or around an annoying obstacle to get work done. You just did it and got on with your day.