I think the best response would have been, “He has a right to do it, end of story.”
That would have been just as good, or probably better, than my suggestion.