F22 Raptor: Source code crashes lead to problems in system software.

Strategypage ^ | April 2, 2004

[spetznaz]

April 2, 2004: For a long time is was said, only half in jest, that you built a new warplane by finding the most powerful engine you could and then building an airplane around it. That has changed in the last two decades. The engine and the airframe are now easy, the software to make everything work together is the hard part. The F-22 software comprises some two million lines of source code (the text and numbers that are converted, or "compiled', into computer readable instructions). This code is complex, and it must work reliably for long periods of time, because the F-22 cannot fly without it's software (or at least most of it) working properly. By current standards, the F-22 software is not particularly huge. The Windows operating system comes in at over 40 million lines of source code, and has been in development since 1988, a few years less than the F-22 software. A competing operating system, Linux, contains a bit more than 30 million lines of code (and began development in 1991). Compared to the F-22 software, the two operating systems are rock solid, able to operate for thousands of hours without a crash. Why is the F-22 software so delicate. As of 2003, after over a decade of effort, the code was crashing, on average, every three hours. Some parts of the software system were failing every 90 minutes. There has been some improvement as of 2004, but the F-22 is still not ready for prime time because of the unreliable software. No one is quite sure why the F-22 software is in such a sorry state, and fingers are pointing everywhere. There are some suspects though.

Part of the reason is that this software has been in the works for over two decades without any effective direction from above to fix the problems. In that time, the hardware the software was based on became obsolete and the software had to be rewritten to run on more modern hardware. When the software creation began in the early '80s, the Department of Defense was mandating the use of a unique software language, ADA, that was to replace all the other software languages then being used for defense work. ADA didn't work out as expected, but some 80 percent of the F-22 code is still written in ADA. Getting good ADA programmers was always a problem, as ADA never caught on outside the Department of Defense. As a result of the limited pool of programmers would could work on the F-22 software, quality and imagination (in the design of the software) suffered. To further complicate matters, the hardware (electronic and mechanical components) the software was to control kept changing, and the software had to be modified to adjust. As a result, F-22 is further delayed from being ready for service.

[spetznaz]

My only question (and maybe some Freepers with necessary knowhow can answer this) is why the F22 software cannot be upgraded. How difficult would it be to replace the flawed software with something more up to date .....even something that is off the shelf? I guess that is a rhetorical question since if it was simple it would have already been done, but i am still curious as to the degree of difficulty imbedded in such a decision. And even if it is difficult i am certain it would still be more cost-effective than having the most expensive fighter jet being wracked by flawed software. Or so it would appear, but the fact this is an issue inherently implies that it is not so straight forward.

Any freepers with an explanation for this?

[Support Free Republic]

---

Calling All Freepers
	Donate Here By Secure Server Or mail checks to FreeRepublic , LLC PO BOX 9771 FRESNO, CA 93794 or you can use PayPal at Jimrob@psnw.com
STOP BY AND BUMP THE FUNDRAISER THREAD- It is in the breaking news sidebar!

[spetznaz]

F-22 Raptor Ping (software problems).

[Spruce]

The classic software dilemma. When does the Kludge end? When can we re-write this crap?

[HamiltonJay]

There is nothing off the shelf about the F22's software. Custom one of a kind hardware, and custom routines to control every aspect of the plane.

I work for a company that is doing many research projects for nasa and the military, some really cutting edge stuff... and stuff that will save lives.

The F22 software is what I would call evolutionary obsolete. Meaning that it has evolved over time, while requirements continually changed for it... leaving a hodge podge of kludgy code, none of it designed or architected from the top... and what is not running is nothing like anything that has been designed in the grand scheme.

THe original coders are likely long gone, new code is being hooked into old code that no one understands, new requirements and features require even more code to be put on top of that... etc etc etc...

What needs to be done is a complete rearchitecture and rewrite of the entire system. A costly undertaking that RARELY happens to software once it is in a production environment. At least based on the story as told in this article.

[cooldog]

There is nothing "off the shelf" available, that's why.

This code was custom-built for one job only; controlling a Raptor F22, and it's the *only* code in existence for the job.

If it is as bad as this article says, then a rewrite is in order.

[coloradan]

[irv]

What needs to be done is a complete rearchitecture and rewrite of the entire system. A costly undertaking that RARELY happens to software once it is in a production environment.

Indeed, if you're going to rewrite that much code from the ground up, I, personally, would want to carefully consider the hardware components, too, in order to look for interactions and common interface requirements. (That's off the top of my head)

Translation: Scrap the whole project and start again.

[Spruce]

Re-write is the first solution for any coder tasked with working on another's code.

[Filibuster_60]

The F-22 software is what you would call a "hard real-time system" - in which event-handling and deadlines are extremely stringent. Even apart from the fact that ADA's the main language, this is a far more complex system we're talking about than a Linux box. ADA incidentally is the language of choice on many troublesome DoD platforms - Osprey being another prime example.

[Spruce]

Could you imagine having to extend proprietary code written over 2 decades? Before terms like Inheritance, Extends and Polymorph were even parts of the toolkit? What a friggin nightmare. It would be a blast, though.

[Filibuster_60]

I'm positive that the F-22 code overall is not only poorly written and documented, but the whole software development process used for it is highly flawed and would help explain why new problems keep showing up even as older ones are tackled. However that shouldn't be surprising - software development back in the early 80s is vastly different from software development today. If you ask me, a brand-new program using existing best practices has a good chance of fielding a reliable product before the clunky system out there now.

[Spruce]

The challenge is to deliver the machine on time. Again it comes to the simple ratio of Kludge/code. When Kludge grows too large it is time to scrap and restart.

[mikegi]

How difficult would it be to replace the flawed software with something more up to date .....even something that is off the shelf?

It can't be replaced by something off the shelf. Rewriting code always seems like the Holy Grail but the problem is that it's ALWAYS more difficult than it seems. All the old hard lessons have to be learned all over again. Things in the old code that look stupid may actually have an important purpose. Having said all that, I have rewritten large chunks of old code and made it much cleaner and easier to work with.

I definitely would not want to be responsible for custom software that sits on top of constantly changing custom hardware. Especially in a real-time system. Tight timing requirements in an RT system make using an off the shelf OS kernel less likely. The benefit of using an OS kernel is that it would force the software to be organized in an app->kernel->driver model. Better have a damn good scheduler in the OS!

One last note, I have no idea how the F22 software is organized but I suspect that the failure "every 90 minutes" is not a system crash but a less serious problem like a timeout in a non-critical system.

[Oztrich Boy]

you built a new warplane by finding the most powerful engine you could and then building an airplane around it.

It has to be a bit more complicated than that

[Myrddin]

A project started at my company in 1993. It was a classified DoD project that mandated use of ADA. By 1998, the project had consumed $75 million and produced a hodge podge of trash. The project required 2167 documentation standards. That meant almost 10 pages of documentation for every page of source code.

DoD became impatient in 1998 and directed the project manager to switch to C++. He dutifully obeyed and turned his lame cadre of Ada "engineers" into an army of feckless C++ hackers. The project is still in the toilet today. The original "senior" "Ada engineer" disappeared back into the hole from whence he had appeared.

Thankfully, my total involvement in that fiasco was just a couple project review meetings. Yuck. I'm not sure that I would be anxious to be an F-22 pilot.

[spetznaz]

LOL .....that is basically an engine with wings.

Another winner in that department is the F-4 Phantom. Don't get me wrong, i love the ole' Phantom but there is something about it that reminds me of a brick with huge powerplants and wings.

[Izzy Dunne]

Source code crashes lead to problems

Maybe they should COMPILE the source code before they run it...

[Spruce]

Sounds like you folk ran into that nasty ISO-900x certification. What a bunch of crap that is still turning-out to be.

[GeronL]

Windows F22 Patches are on the way!