If your goal is to fly around and shoot down other fighters than I'd agree with this. However I'd say most missions would be to put weapons precisely on targets which are otherwise inaccessible. To do so, you need to fly past the enemies' air defense network of radars and surface to air missiles. You don't want the ground radars seeing you, and if they do, you don't want the missiles to be able to hone in on you. Hence stealth.
If our only potential enemies were third world nations with archaic defense networks, well then we don't need good new planes, just planes that are better than theirs (and their missiles). But if you want to be prepared to go against China or Russia or rogues armed by them, then you need something better.
To clarify your point, fighter are designed to shoot down other fighters and stealth allows a level of ‘sneakiness’ to do so.
Strike aircraft are the ones that drop bombs, if the platform can also be used as a ‘fighter.’ The F-15E is an excellent fighter/bomber (’strike’) aircraft.
The F-22 can be used to blind the enemy because they can do so without being seen, rush in, drop a JDAM or SDB and never be seen, thereby opening the door for the non-stealthy F-15E to run in and drop bombs in great numbers and with great effect.
Strategic strike like the B-1’s and Next Gen Bomber are something else.
Other nations are buying 4th-gen fighters from Russia and China and that is what we build against. . .the platform.
To be sure, a third-world camel jockey pilot is still a third-world camel jockey pilot even if he is flying a first world advanced jet. See Iraq in Gulf War 1 for example.
While we produce the best pilots in the world, along with the RAF and RAAF, we must always be prepared to fight those that might not be camel jocks. That is why we build to the fighter. There is no guarantee we will NOT end up fighting a capable enemy behind the stick. We should not bet our national security that all we will ever face is some third-world camel jock.