Good point. Also, if Joseph was so concerned of himself that he contemplated putting Mary aside for being "with child" and unwed, it hardly makes sense he should himself father children out of wedlock, so these "others" could not have been younger siblings from other spouses.
One more thought..It is a sin for a woman to refuse to be a wife in all respects to her husband...so if Mary refused that right to Joseph that was a sin..
Now in the long runs none of this matters much ..Jesus is still the product of a Virgin Birth , Son of God..King of Kings and Lord of Lords..