Remember what they said about his great-great uncle, Edward VIII, who also gave up a throne for love, and married American divorcée Wallis Simpson, "He used to be the ruler of Royal Navy, now he's third mate on an American tramp."
To be fair, they probably did everyone a favour. They stayed happily married till he died and spared the country from having a king who was too cosy with the Nazis.