"I would however like to know what reason is there to ignore the natural reading of brother = literal brother."
Because until very recently, all Christians believed that Mary was perpetually virgin. Therefore the Scripture was interpreted in the light of this common, ancient belief and thus "brother" was never understood in its literal physical sense.
Luther, Calvin and Zwingli all believed and defended her perpetual virginity, and this doctrine was never part of the original Catholic vs. Protestant controversy. It was something that all were agreed on.
That it has become an issue in more recent times is probably due to the way the English language has developed over the last few hundred years. Words change their force, meaning, and latitude of interpretation over time. e.g. what do the words "gay" and "wicked" mean in colloquial speech now compared to 100 years ago?