Why couldn't it have been his half-brothers?
I suppose by definition they WERE his half-brothers, seeing that Joseph was their father. What is your point ? That Joseph had sons by someone other than Mary ??