Here is problem number one. Nemesis could have gotten an extra $40 million if they had opened at Thankgiving, like they normally do, against Harry Potter. But they don't know their audience!! Harry Potter was never a major challenge to their fan-base, but Lord of the Rings encompasses their entire fan base.
Problem number two. They never kill any characters! Want to know why Star Trek II: Wrath of Khan was the best Trek film so far? They killed Spock! Not that fans wanted Spock to die, but the whole production was in your face, "Wow! Redshirts aren't the only ones who die!" kind of movie with danger and thrills the whole way through it.
Problem number three. New situations that are easy to write your way out of stink. It's a matter of "ridiculon" particles to save the Enterprise yet again! Or Picard's sheer will power to out think the villian. Give me a break!
Anyone remember Babylon 5? Remember the episode where Sheridan dove out of the lift on the station when a bomb was about to go off? Me and my friends were on the edge of our seats during the commercial because there was no way for them to save him before he hit the ground. Then Kosh "revealed" himself, every race saw him as an angel from their point of view, and flew up and grabbed him out of the air. Incredible and unforseen! Then three episodes later Kosh is killed by the "shadows" that were just starting to take on a major role. A major character killed off in the middle of a season! Is it any wonder why B5 had such a loyal following? No one knew what was going to happen, or more importantly, how the situation would be resolved.
If Star Trek can get back to being inventive and shocking (the Borg was the best thing they ever introduced, and Spock's death was incredible), the franchise will survive. If they can't get back to great storytelling, Star Trek will increasingly become a parody of itself.