F22 Raptor: Source code crashes lead to problems in system software.

Strategypage ^ | April 2, 2004

[spetznaz]

April 2, 2004: For a long time is was said, only half in jest, that you built a new warplane by finding the most powerful engine you could and then building an airplane around it. That has changed in the last two decades. The engine and the airframe are now easy, the software to make everything work together is the hard part. The F-22 software comprises some two million lines of source code (the text and numbers that are converted, or "compiled', into computer readable instructions). This code is complex, and it must work reliably for long periods of time, because the F-22 cannot fly without it's software (or at least most of it) working properly. By current standards, the F-22 software is not particularly huge. The Windows operating system comes in at over 40 million lines of source code, and has been in development since 1988, a few years less than the F-22 software. A competing operating system, Linux, contains a bit more than 30 million lines of code (and began development in 1991). Compared to the F-22 software, the two operating systems are rock solid, able to operate for thousands of hours without a crash. Why is the F-22 software so delicate. As of 2003, after over a decade of effort, the code was crashing, on average, every three hours. Some parts of the software system were failing every 90 minutes. There has been some improvement as of 2004, but the F-22 is still not ready for prime time because of the unreliable software. No one is quite sure why the F-22 software is in such a sorry state, and fingers are pointing everywhere. There are some suspects though.

Part of the reason is that this software has been in the works for over two decades without any effective direction from above to fix the problems. In that time, the hardware the software was based on became obsolete and the software had to be rewritten to run on more modern hardware. When the software creation began in the early '80s, the Department of Defense was mandating the use of a unique software language, ADA, that was to replace all the other software languages then being used for defense work. ADA didn't work out as expected, but some 80 percent of the F-22 code is still written in ADA. Getting good ADA programmers was always a problem, as ADA never caught on outside the Department of Defense. As a result of the limited pool of programmers would could work on the F-22 software, quality and imagination (in the design of the software) suffered. To further complicate matters, the hardware (electronic and mechanical components) the software was to control kept changing, and the software had to be modified to adjust. As a result, F-22 is further delayed from being ready for service.

[Mannaggia l'America]

Maybe they should COMPILE the source code before they run it...

He he - I noticed that line too.

Maybe they should rewrite the software in Java.

But then the F22 would have to be slowed down to go as fast as this:

[IronJack]

you're rather behind the times. S/360 went out of date many years ago. We're past S/370 and on to S/390 and even Z/OS now.

I'm well aware of that. I didn't say I programmed ON a 360. I said I programmed 360 Assembler. (Also known as BAL or ALC.) The instruction set has remained the same for IBM mainframes since the introduction of the 360, even though the OS's have changed numerous times.

[7.62 x 51mm]

" Some parts of the software system were failing every 90 minutes."

Imagine The Blue Screen of Death at 20,000ft... Yikes!

[EQAndyBuzz]

I think that everytime there is a hardware change, the modification of the code becomes even more complex. It is not like change a module and run with it. All 2million lines have to be reworked and retested.

The evolution becomes even more complex. Pristine code no longer is pristine. One new line effects thousands of lines of code.

An example would be, if a module was updated for wind lets say, it would impact fuel flow and god knows how many other variables. Let's say that you miss a speed to weight ratio while updating. Missing one variable now can have a major impact three revisions down the road.

Think this is probably what happened.

In my company this happened with a distrubition package we were writing. Someone forgot about the size of the truck and the capacity it could hold. No one took this into account. When new trucks were purchased, they purchased smaller trucks. Guess what? They needed more deliveries to distribute goods. And they then wondered why the costs escalated. They wound up laying off people. Was sad actually.

Funny thing about it though. A year later the owner thought something was up and called the feds. Turns out the union asked for the smaller vehicles. The union leadership was in bed with a truck dealership. The programmers brother in law was the union head.

Both are now in jail for fraud. I love happy endings.

[IronJack]

I was fascinated, IronJack, to learn 360 BAL is still in use. It must have some legs

You'd be surprised some of the places it's used. If you use a credit card, odds are it was authorized via an Assembler-based online. If you get cable TV, high-speed internet, or telephony services, there's a good chance your bill comes from an Assembler-based system. Many insurance companies still use it and a number of banks.

To those who would rather drive a sports car than a Cadillac, Assembler is still a very useful language. Our company estimates we need 25-40 percent less CPU to process the same volume compared to a COBOL/CICS online. More transaction-intensive platforms would benefit even more.

[IronJack]

a pipelined execution queue.

I'm not sure what you mean by that. But whatever the code is, it ultimately becomes machine language. I've always wondered why the executable wasn't written in the language the machine speaks to begin with.

[mikegi]

Software designers, most of whom in the US are not degreed engineers, quick and smart as many are, have not learned the discipline of engineering, finite state machines, time dependency analysis.

Hey, I represent that remark! Well, I did flunk out of EE school in the early 1980s so I did have a bit of an engineering background. Thank goodness that, back then, the PC world was desperate for programmers. All the CS grads went into minis and mainframes.

As Microsoft has shown with their Common Language Interface (CLI) standard, procedural languages are equivalent. Fortran = C++ = J# = Python = Basic = Cobol ... They all compile to an intermediate language. Last I heard there are some 60 languages which .NET will translate to the the Intermediate Language IL (like Java's byte code), where they are dynamically compiled to machine code.

I haven't used the CLI stuff so I don't know how its performance compares to, say, MSVC++ 6.0's output. I do know that the guy at msft who did the runtime compilation stuff for msjava was really good (IIRC, he came from Borland) and I assume he went on to do the CLI stuff, too.

What's funny is that of all the languages I've used I found Modula2 to be the easiest to create bug-free code. For the systems I created with it, though they were small scale projects, if the source compiled, then it ran without error. I even embedded programs written in TopSpeed Modula-2 on that old NEC 8088 clone chip with integrated RS-232 ports. TopSpeed's compiler was great because it had built-in multithreading.

[Mr Rogers]

Trust me - the Lockheed software engineers who write code for the F-16 are even worse than those writing for the F-22!

F-16 software is updated on a 2 year cycle. Lockheed (to maximize profits) requires everything on the airplane to run thru a central computer. Therefor, even minor improvements are purchased on a 2 yr cycle - except Lockheed's contract freezes the software about 5 years out. No way you can keep requirements frozen for 5 years, so you get requirements creep driven by Lockheed's incredibly slow turn around on writing code.

The requirements creep results in things that don't work right, so to get the OFP out the door on time band-aids are applied. The band-aids usually break almost as much as they fix, so another bandaid approach is used.

The F-16 software is now a huge hairball of accumulated band-aids, and no one understands how all those band-aids interact. This makes ANY change a potential show-stopper.

But Lockheed gets paid to put band-aids on, and they get paid in spite of their 5 year turnaround driving the need for changes, and Lockheed makes a ton of money to fix what they broke in the first place.

Lockheed has done the same thing with the C-130J, and will undoubtedly do the same thing with the JSF. And Rummy wants to streamline the system by 'trusting' the contractor - YGBSM!!!

[Mr Rogers]

A software engineer I know who saw how Lockheed works F-16 software says their entire process is badly flawed - no configuration control & run-amuck patches applied without knowledge of how they will affect other areas.

I'm just a dumb EWO, but even I can tell Lockheed screws up every piece of software I've ever seen them touch. FWIW - I'm told Boeing is at least as bad...

[Mr Rogers]

I'm very concerned about the software problems they are having. Yes, they will eventually get something that works right - way behind time & way over cost. Every month delay impacts combat capability - either by delivering it late or by reducing the number we can afford.

In addition, the failure to fix it years after the problems arose means they don't fundamentally understand what they are doing. So when they finally DO get it to work, no one will know what the next upgrade (that will run thru the central computer) will do. It isn't software DESIGN, it is software LUCK.

[chilepepper]

want to know why?

ADA was designed in FRANCE

(i kid you not!)

[chilepepper]

the problem w/ writing in assembly language is that the scope of the problem, e.g.- the functions that have to be performed, are just too complicated and it would take too many folks to do it in assembly. how many lines of ASSM would it take to run the F22, a *BILLION* ??

what ADA was *supposed* to do was to foster shared libraries and functions, e.g.- the same idea as the classes in C++ and other OO languages.

i suspect something like LABVIEW or at least the VRTX platforms (such as are being used on Spirit & Opportunity) would be really good choices now a days...

i am, however, very much surprised that the F22 isn't already using higher level programming abstractions which take into account 100 years of aviation... the things that have to happen on the F22 are very similar to what has to happen on the F18, and not THAT different than what has to happen on a F15 or F16, and not THAT wildly different than what has to happen on an F4.

[thoughtomator]

I dunno, but I'd be happy to help the DoD fix it if they wanted an extra programmer.

[spetznaz]

our schools are handicapping our brightest students (Students in math and engineering from abroad require about two years less coursework preparation for a Ph.D at U.C. Berkeley)

Wow! Are you serious? 2 years!

[Criminal Number 18F]

[ADA] is not a very hard language to learn

True. It's basically an extended Pascal, and Pascal was once the teaching language in universities. The problem is, finding people motivated to learn it. Defence contractors don't want to spend money on people, so they want someone who already has a knowledge of ADA and a final security clearance. Means they are fishing in a small pond.

Many of the people opposing the F22 are those that oppose all defence innovations (John Kerry springs unbidden to mind). But the USAF has done an extremely poor job of selling this aircraft.

People who advocate changing to a desktop operating system don't understand just how unreliable those systems are. One avionics firm made primary flight displays that ran (underneath it all) on the NT kernel. They found that to pass FAA certification, much less stringent than military, they had to redo their software in a real-time OS. The Windows based system could not make the reliably standard by a staggeringly large margin.

d.o.l.

Criminal Number 18F

[CasearianDaoist]

ADA programmers with emebbed real time experience have a very high pay scale at the moment, and the way they are popping people through clearances these days for FCS it must not that rough to get through, particularly for eager young people. The USAF perhaps has not sold it well but they have done a good job building an industrial constituency for it on both sides of the aisle, including the unions. I wonder if they did not sell it so well or rather the case was that their opponents sold their view point better. They let them get away with the "Cold War Weapon" and "fighter mafia" nonsense far too long. the F22 is not the Comanche or the Crusader - it is a whole other problem. It will be daces before UCAVS can safely take on this role, if indeed they ever do. In the end it might me a mixture of manned and unmanned weapons. The F22 is no "niche platform" we need it to keep overmatch.

Almost all the people in the know that oppose the f22 want a weaker nation. I cannot understand the criticism I hear from some around here. Do we want the Europeans to have a better fighter. In my book it is a must have. It gives us Air dominance for another 35 years. I wish that they would build the original 700+ flight. I would even like to seem some sort of "navalized" smaller version of it. It is unclear if super hornet is good enough against leading edge land based air dominance fights that we might face. I think that the transition of the Navy from a purely blue water force to an "Blue and Green" water navy (See strike at the littoral) exposes a gapping whole here in the weapons strategy. We cannot always count on the Air Force being there. We shall see. The USAF will cancel other major programs before they give it up.

Now all we need to do finish the ERAAM. the F-22 and the ERAAM, now that's a platform.

[CasearianDaoist]

whole = hole

[VOA]

"LOL!!! That was the exact aircraft I had in mind when I read that line."

-- nearly a verbatim version of my thought as well...


And for the lurkers/posters who are asking themselves "what the h-ll is that"
concerning the airplane in post #15:

http://www.geebee.com/

[Criminal Number 18F]

CD,

You make some good points. However, the USAF needs to sell this thing to John Q public. Why is it better than an F-15? You know and I know (Jeez, supercruise alone is revolutionary, and that is one DETAIL of an integrated whole) but I talk even to fighter pilots who don't know.

If this program dies it will be USAF mismanagement that killed it. A few examples of that include the sudden repurposing it as the F/A 22 after what, 15 years of development as an air superiority fighter? It makes the AF look like whores.

Finally, failure to think intelligently about costs has *always* been a characteristic of USAF planning, and still is. As the size of the buy shrinks, the unit costs grow, which shrinks the size of the buy further. It's as if no one in the Pentagon has ever heard the engineering saw, "the best is the enemy of the good."

If, in a conflict with China 20 years from now, we have two dozen F-22s and otherwise are relying on 1970s engineering in the form of the F15 and -16, are we better off? What if the chinese can match our F15s ten to one with F15-equivalent Su27s or successors?

d.o.l.

Criminal Number 18F

[Paul C. Jesup]

THe original coders are likely long gone, new code is being hooked into old code that no one understands, new requirements and features require even more code to be put on top of that... etc etc etc...

Sort of like what goes on at the IRS.

Sorry I could not help myself. ;P