The Bible is not ambiguous about it. The Bible even names some of his brothers.
Why couldn't it have been his half-brothers?
Most Lutheran churches hold that it was Joseph's children from a previous marriage. The reason given for John being tasked by Jesus to take care of Mary was that His other brothers didn't follow Him at that time.