Context provides the definition of the words, not vice versa. So, the burden would seem to be upon the people who read the Matthew passage as speaking of his non-nuclear-family relatives. To be as succint as possible, the question is: Why would the townspeople try to point out Jesus' humble origins by saying "Hey, that's just Jesus. We know who his mom and his cousins are?"
The meaning that most people would take from a plain reading is that they list Mary, his mother, and his siblings. As for SoothingDave's argument that we know these people's real parents from other Scripture he would have to establish that the people being spoken of here are indeed the ones we know by the same names elsewhere.
Why not? "Who does this guy think he is? Isn't he Mary's boy? Don't we know all his family?"
The point is that this kid ain't something special. His cousin married my barber's daughter. They ain't special, they're just garden variety first century Hebrews.
Imagine if you saw someone on TV claiming to be a prophet. Wouldn't you turn to your wife and say "Hey didn't your brother use to date his cousin?" And then scoff.
(There could also be an argument made from the use of the phrase the son of Mary as opposed to a sonof Mary, but I don't know the Greek or Hebrew behind the phrasing.)
SD