If a father’s visitation rights are being violated it is a legal matter to be handled by the court. The father tried to physically force his legal right on his own, which is not OK. The child wasn’t even there, so even from that perspective the father had no legal right to be there.
Unless the father had some reason to believe his children were being abused, at that momnet, call the cops, film eveything, bbut comply with the homeowner's command to leave the premises.