You can, but you have to prove that you had no knowledge or understanding of the details of the plea agreement and you're withdrawing your agreement. Chansley may not be the sharpest tool in the shed but he'd have a hard time proving that.
I assume that you can probably appeal if there was misconduct by the prosecution.