That did seem a bit strange at first, but Ashville is overrun by aging hippies and other liberals, and Spartanburg is the nearest population center. While there would be some smaller churches that were closer, he might have been concerned with the impact in might have had on a smaller body.
That part of the world has a lot of Presbyterians and Methodists; Baptists might be 3rd or 4th in that area.
Which means a Baptist congregation on every 3rd or 4th block :-). But I see your point, and given Rev. Graham's age and state of health, it's unlikely he'll be a regular attendee, so the distance may not matter.