I've learned about the Pagan - Buddhist interrelationship from being married to a Buddhist for 35 years. In that time I've had countless discussions with various Buddhist Monks, Lamas and Abbots who are friends of my family. A simple Google search on Pagan and Buddhism will give you thousand of references, including several of the texts that I have read along the way.
That quote is from Pagans and Christians by Robin Lane Fox, University Lecturer in Ancient History at Oxford.
I don't dispute that there was a center of Buddhism in Burma called "Pagan." What I'm fairly sure of, is that 4th century Christians knew nothing of Buddhism, nor indeed of Burma. They were Latin- or Italian-speakers, and chose a bit of slang from their own language, referring, not to Buddhists, but to the polytheists among whom they lived.