I've been on the property before. It's hardly a military base. It covers approximately two blocks in downtown San Diego. Also, it's mainly administrative offices. Furthermore, it's only two blocks from other commercial high rises and one block from luxury condos. So I doubt there's anything contaminated there.
You're right, it does sound like choice real estate...