If Mary had children by another man, we would give her every benefit of doubt and say that she married again after Joseph died. Mary is THE blessed Mother of Jesus.
They were referring to the Father of Jesus AND Joseph (or any other man) also fathering children by her when the father of her first child was still living.
They were trying to avoid the charge that would come from that.