I think it’s more than that.
Bond is the product of a classic British education.
For a black actor to try and pull that off, he would look “wimpy”.
A white actor can do it and look suave, and tough all at the same time.
The only black actor who could have ever pulled off Bond was (maybe) Sidney Portier.
Idris Elba is a fine actor, but not meant to play Bond. No way.
All that you mentioned are the character traits the public has come to expect in both the novels and films.
However....
The defining quality is that Bond is a SECRET agent with a license to kill, with the need to blend into the crowd.
PC crowd not withstanding, Britains main adversaries remain white Europeans full stop.