Because way too many folks are reading SF (or Fantasy) almost exclusively because they think it is more FUN to read. To me, a lot of SF is like eating highly calorated air. Look, I've read and enjoyed a lot of SF but at some point I decided to check out literary novels.
Nothing wrong with reading Asimov or Vonnegut but how about reading them AND James Jones AND Robert Penn Warren AND Ralph Ellison AND Tolstoy? As to the SF ghetto, way too many people volutarily put themselves in that ghetto by reading SF and Fantasy almost exclusively.
I have learned more of life, of science and wonder and the way people are from SF than from "literature".
Pick out, in any given year, a non-SF novel that recieves awards and accolades, and an SF novel that recieves the same sort of praise. I would bet you that in 85% of the cases, the SF novel will be read in 50 years, and no one will remember the 'real' novel.