Most Americans are moderates. (And, no, that doesn't mean they don't know what they think). The Dems have gotten so Liberal that the moderates are voting GOP. If the GOP leaders decided to take a hard Conservative line, the Dems just might go back to clintoon-style triangulation, which gets them the moderate votes.
But, yes, if both parties decided to take the extreme view, a third party would have a very good chance. It would just need a leader, and one could emerge.
The problem is that GOP politicians have a very strong tendency to articulate mushy positions that are absolutely indefensible. If the Democrats propose some new $10B/year program that is a just plain Bad Idea, Republicans need to oppose it outright, rather than "compromise" for a $5B/year version. Even if the refusal to "compromise" means the Democrats force through the $10B program, the fact that the Republicans opposed the program from the get-go will help them kill it after it proves to be ineffective. By contrast, if the program is funded for $5B with Republican support and then fails, the Democrats will successfully argue that it was underfunded and thus needs to be funded for $10B. The Republicans will have no effective counter and thus instead of a program costing $10B and being cancelled, it will cost $5B the first year, $10B the second, and even more after that.