Matthew 13:53-5753And when Jesus had finished these parables, he went away from there, 54 and coming to his hometown he taught them in their synagogue, so that they were astonished, and said, "Where did this man get this wisdom and these mighty works? 55 Is not this the carpenter’s son? Is not his mother called Mary? And are not his brothers James and Joseph and Simon and Judas? 56And are not all his sisters with us? Where then did this man get all these things?" 57And they took offense at him. But Jesus said to them, "A prophet is not without honor except in his hometown and in his own household."
These were not outsiders. This event happened in His own hometown.
All they knew was a large family with kids, and called them collectively "brothers". Middle-Easterners do so to this day.