I used to think this. However, I installed Ubuntu linux on a spare machine this summer, just to see where Linux was at, and was quite surprised. It was very user friendly and had a very windows-esque look and feel to the desktop. Installing programs (even those downloaded from the net) has become automated and is no longer the "geeks only need apply" hassel it used to be. If all I was using a computer for was office apps, internet and music I would switch in a second.
I wasn't sure myself, but was able to mess with Fedora this past weekend and was very impressed. I think the main thing would be to have someone install the OS for you (well, for me, anyway). After it's stable and running, it really isn't any worse than Windows, and probably better in a lot of ways. I'm going to install Ubuntu on my old desktop & see how many apps there are to replace the ones I normally use. If I can find ones that mirror what I already use, I'll probably get rid of Windows. The virus worries and constant updates are too much to worry about.