The problem with taking an average of all polls is that averaging the bad polls in with the (few) good polls results in an average that is most likely to be wrong.
You say 2 + 2 = 4
I say 2 + 2 = 6.
Hmm. Can’t figure out who is right...so let’s average.
2 + 2 = 5!