It’s implied that Rick didn’t stick with one woman for very long after he left France. Ilsa was his one true love, and when she sent him the letter as he was on the train out of Paris, he probably hardened his heart against any woman again.
It can happen.