Because stars are hard to see in the daytime even for Magi?
I can only assume that Mary and Joseph spent only one night in Bethlehem, took care of the newborn baby and did the census paperwork with the Romans and left to go back to Nazereth the next day.
The New Testament says the ‘wise men’ found the child in a house, not a manger, so Jesus could have been a month old at the time of their coming.