Unless Joseph was older and had previously been married and these were Jesus’ older half brothers.
That’s a possibility, but you would think as much as that time period was orally and written by multiple people and historians that fact would at least be hinted at or referenced in some form or fashion...
I think I will stick with ocams razor...