David Duchovny isn't what you would call a real actor. He's from that William Shatner school of dramatics. And, yeah, Olivier from all accounts wasn't that bright. Neither was Jack Lemmon. They could "act" bright, but that's different. And not to say they weren't nice guys. It has something to do with the "craft," it doesn't allow for depth of intellect. Hitchcock knew this and used it to his advantage.
The conventional wisdom suggests that to be a great actor you have to understand human nature, psychology and so forth. Which suggests intelligence. I suppose all around intelligence isn't rarely granted to people who have an exteme gift in one particular field. Beethoven's letters show his intellect to have been pretty ordinary to say the least.
David Duchovny's parents are both professors I believe. I know Duchovny's dad was a liberal. Before there were "Quaylisms" or "Bushisms", there was the "Collected wit of Spiro T. Agnew", that was a compellation put together by Duchovny's dad.