We have a horse at the barn who had very severe ear infections a long time ago - but he STILL doesn't like his ears handled. We have a special clip release on the left side of his headstall so that we can unclip the bit and let it drop out of his mouth. Then you unbuckle everything else and wait until he drops his head to signal that he's ready - then swiftly run your right hand under the reins and the top of the headstall and he backs out of the bridle. He has to do it on his terms.
I had to braid this boy for a hunt (for my daughter, she is learning how but her braids still look lumpy) - it literally took me two hours. Instead of braiding head to shoulders I braided him backwards, gradually working up towards his head. By the time I finally braided the last braid just behind his ears and his forelock, he was totally bored and had realized that I wasn't going to hurt him. Now he lets me scratch his head just behind his ears - I haven't tried scratching his ears yet because it would probably undo all my good work - but next time I braid him up I'll give it a go.
Have you tried gradually stroking around his ears and giving him his favorite treats at the same time? I wonder if he might not have had an episode like this horse - a bad infection that is gone but he still remembers the pain?
He's gotten better about it.... we can now cup your hand over the whole ear and rub, the way labs like, if you do it careful. He doesn't enjoy it, he gets this worried look on his face and looks right at me, but he'll tolerate it. We can pet over the outside of the ear in stroking his whole head. But if we grab the ear flap at all (when I grabbed it the first time, it was to rub-flop his ears the way my old lab used to like) he cries out and recoils.