First off, I'm trying to understand why music should be selectively harmful! But there are plenty of things we do a lot, like purely recreational sex for instance, that probably are marginally detrimental (risky) in an immediate sense, so I can't throw your question out immediately...
Maybe it's the long-term risk of having such a powerful and generalized thinking machine inside our heads.
I'm no expert on the different theories on the development on the human brain, but here's what I think happened: Our ancestors' big brains proved very helpful in surviving in their environment, and our appreciation for things like music & art & humor & junk food are all side-effects of that big-brain capability. Similarly, civilization and the moral codes that make it possible have been very beneficial to humanity, yet there are those who fret that our compassion for the sick will eventually hurt the gene pool as a whole. If we assume this is true, its harmful effects (in a population genetics sense) haven't outweighed the positive benefits of the civilization that generated it ... yet.
Here's another example: Until the rise of the Industrial Revolution, the most prosperous societies didn't necessarily have smaller family sizes than the poorer countries, because they were all agricultural societies and more children meant more loyal farm hands. But today the prosperous societies (which tend to be more industrialized & capitalist) have far smaller families on average than the poorer peasant societies. In the long run the prosperous societies could theoretically go extinct. According to your question's logic this should never have been able to happen in the first place. Obviously the answer is that a long term negative consequence takes time to hurt the conditions that made it possible in the first place.