Well there are two family histories in the infancy chapters. The Gospel of Mathew’s family history is of that of Joseph’s, Mary’s is covered by the Gospel of Luke’s.
Yes, you are right, and that has been mentioned in this thread. I have personally pored over both, and am not neglecting the Matthew account (Jesus the King of royal Hebrew lineage) but Luke's (Jesus the Son of the Human) where later on in His life the scribes and Pharisees tried to intimate that His conception was premartial, therefore a product of fornication, with perhaps the true father not known for sure.
Of course, the whole Hebrew-institutionalized process of espousal was invoked to make one's descent secure and irrefutable. It is not until our recent past that, under the gentile system, paternity could be fully and inescapably confirmed by DNA testing.
Under the Law, the methodology was given in Deuteronomy 22:13-26. The espousal was a preview of establishment of a contracted marriage, with a truly committed envowed husband and wife, but without consummation until the espousal time elapsed guaranteeing paternity, whence the validating virginity test would be conducted immediately without further contention.
Very obviously Husband Joseph would have justly been allowed to put away/divorce espoused Wife Miriam, but he yielded in faith to the Holy Ghost speaking through Gabriel, thus giving The Anointed One his legal royal title, and fatherly protection, as well as apostle brothers and loyal sisters in the future consummation of their marriage according to the will of The God: "Be fruitful and multiply!" an ordinance to not defer by rhythm or mechanical contraception, nor by unnatural continence/celibacy in marriage or abortion.