They were unbelievers/not spiritual brothers and possibly pretty young...
“My mother and my brothers are those who hear the word of God and do it.”
Do you see the last three words of the verse you have been basing everything on? DOERS. Jesus himself says his mother and his brothers are those hear and DO the word of God. Being a spiritual brother isn’t enough. You have to be a DOER of the word of God. Our good works are the fruits of our accepting that we are forgiven. Faith without works is dead.
As to the idea that he disowned his mother via this verse, it is preposterous. Mary is to be revered. She has an honored place. Jesus was speaking as to principle and not as to fact. We know he did not reject his mother. Indeed, at least by the time of the wedding feast of Cana, she knew that he was a miracle worker and, so, some kind of man of God. We also know that his brothers took some time to come around to accepting him as the Messiah, but they did, and they were crucified for the faith. What good works have you performed relative to the martyrs, that you say they were rejected by Jesus as not being his spiritual brothers.