Very true. He liked to call people who liked that kind of music "Rollos". Not to turn this into a Charles Ives thread, but one of my favorite Ives quotes comes from a performance of his and Carl Ruggles' music. Several people booed Ives' composition, but when they did the same during the performance of his friend Ruggles, Ives turned around to the audience and shouted "Stop being such a God-damned sissy! Why can't you stand up before fine strong music like this and use your ears like a man?"
He was very insecure about his reputation as a ‘real man’. Serious music in America at the time was considered a women’s advocation. That’s why he didn’t want to be a full time composer.