I faced a similar situation with my daughters and my ex many years ago. In our case though, I was awarded sole custody with supervised visitation after he failed to appear for joint counseling sessions. In time, the effort to see the girls was too much, and he just faded into oblivion. He hasn't see my daughters for about 12 years.
The Lord is able, and has your grandson's best interests at heart...watch for a "creative" solution from Him to your problems. :)