Actually, my wife and I refer to our new daughter as “the baby” all the time. The callers use of that phrase means nothing.
Phew! I was beginning to think I was the only one who called the youngest “the baby”. Even though he’s almost five.:)