“You must hang out with that crowd from Hampshire. “
—
Nope,but I’ve lived in Eastern MA all of my life. Never heard it with a silent “H”.
.
I have family members (distant) who are 3 generation Amherst College (not UMass).
I don’t know about the town, and I don’t know about UMass Am Herst, but graduates of the college pronounce it Am Erst.