Yes, that’s probably where the anti-government accusation comes from, but the real question is why the mother doesn’t have custody?
How did they get placed at Grandma’s instead of left with Mom? The only thing I can think of that wouldn’t involve a public record is military deployment.
If Dad got arrested while Mom was deployed, Grandma would get a call. Then Mom comes home and Grandma doesn’t want to give up the kids (or the stipend she’s been getting for them) and disagrees with Mom about the viability of her marriage or suitability of the Dad and you’ve got a ugly custody dispute in progress.