Not remembering seeing someone in college isn't that big a deal. I recently found out that a coworker and I were classmates in college, graduating at the same time with the same major from a fairly small university. We do not remember each other from college. People who weren't big socializers, and who spent their time studying alone when not in class, won't be remembered by anyone.
Was your friend president of the Harvard Law Review? Probably not.