Hi RB! I am so flattered that you think we're sane!
No, I don't think it's rude at all for you to want father's day to yourself. My folks split up when I was still home, and neither parent would have tried to horn in on Mother's or Father's day for the sake of a grandfather, or step-dad, and I had both. Plenty of other get-together opportunities on other days.
Whose feelings are going to be the most hurt? - Yours if you let them go, or theirs if you don't? Is it worth the battle in the short run?
I appreciate everyone's insight. I get stuck in the forest sometimes and fail to see the bigger picture.
Everyone involved is local so they can make plans for another dinner.