I liked the first Philip too. And one of my very favorites in the first 2 seasons was Tommy Lascelles, the Private Secretary. He was exceptional.
I struggled through Season 3 with the new cast... it was like watching a totally different show, and I kept wishing things hadn’t been changed, just did not enjoy the show nearly as much. So I haven’t even bothered to watch Season 4 yet. Maybe I will at some point.
Season 4 was good!
I recommend you watch 4. I struggled vtoo but the story is worth it. The only false note was the treatment of thatcher.