[pgyanke]

Without looking at text, I think Matthew says that everyone had to return to their place of origin...

No. They went to the place of their family name. Joseph was from the line of David so he took his family to the city of David, Bethlehem of Judea.

[Natufian]

The census was for tax purposes. Making people move away from where they make a living makes no sense in that context.