I have to admit that after having worked in academe most of my career, I'm fond of college towns. There is some downside but overall, the presence of a university enhances the desirability of a community, IMO.
I'm beginning to think about retirement. I still enjoy work, so that is probably 8-10 years down the road. My wife and I are thinking a little bit about where we want to settle. We might just stay in Manhattan, Kansas or we might move, probably to the Northwest. Although our retirement location will probably be determined by where our kids wind up, Missoula, Montana is the current front-runner if we move. Coeur d'Alene, Idaho is the only candidate for a move which doesn't have a university located there.
I love college towns too and they always seem to rate high on “quality of life” surveys. Especially in the smaller cities and towns where the institution tends to dominate the population and culture.