He caught them at it.
Translation: He was angry that they caught him.
Yes, it’s all their fault. The fact he did the deed isn’t the point as far as he’s concerned.