F22 Raptor: Source code crashes lead to problems in system software.

April 2, 2004: For a long time is was said, only half in jest, that you built a new warplane by finding the most powerful engine you could and then building an airplane around it. That has changed in the last two decades. The engine and the airframe are now easy, the software to make everything work together is the hard part. The F-22 software comprises some two million lines of source code (the text and numbers that are converted, or "compiled', into computer readable instructions). This code is complex, and it must work reliably for long periods of time, because the F-22 cannot fly without it's software (or at least most of it) working properly. By current standards, the F-22 software is not particularly huge. The Windows operating system comes in at over 40 million lines of source code, and has been in development since 1988, a few years less than the F-22 software. A competing operating system, Linux, contains a bit more than 30 million lines of code (and began development in 1991). Compared to the F-22 software, the two operating systems are rock solid, able to operate for thousands of hours without a crash. Why is the F-22 software so delicate. As of 2003, after over a decade of effort, the code was crashing, on average, every three hours. Some parts of the software system were failing every 90 minutes. There has been some improvement as of 2004, but the F-22 is still not ready for prime time because of the unreliable software. No one is quite sure why the F-22 software is in such a sorry state, and fingers are pointing everywhere. There are some suspects though.

Part of the reason is that this software has been in the works for over two decades without any effective direction from above to fix the problems. In that time, the hardware the software was based on became obsolete and the software had to be rewritten to run on more modern hardware. When the software creation began in the early '80s, the Department of Defense was mandating the use of a unique software language, ADA, that was to replace all the other software languages then being used for defense work. ADA didn't work out as expected, but some 80 percent of the F-22 code is still written in ADA. Getting good ADA programmers was always a problem, as ADA never caught on outside the Department of Defense. As a result of the limited pool of programmers would could work on the F-22 software, quality and imagination (in the design of the software) suffered. To further complicate matters, the hardware (electronic and mechanical components) the software was to control kept changing, and the software had to be modified to adjust. As a result, F-22 is further delayed from being ready for service.

Did Richard Clark Work on the F22?

Let's assume the source of the original article is trustworthy, and not an engineer responsible for the problems he's complaining about like a former counter-terrorism czar. Among the rules in place for coding on several platforms under development, Joint Strike fighter included, is the constraint that source code may not be modified. Yes, I said SOURCE code. DOD and the Air Force are at the leading edge of platform independent Model Driven Architecture. The code is generated automatically.

As Microsoft has shown with their Common Language Interface (CLI) standard, procedural languages are equivalent. Fortran = C++ = J# = Python = Basic = Cobol ... They all compile to an intermediate language. Last I heard there are some 60 languages which .NET will translate to the the Intermediate Language IL (like Java's byte code), where they are dynamically compiled to machine code. Further, I can say from personal experience that compilers will almost always achieve better performance from a system because processors are so much faster than memory. Many, if not most, processors are designed as pipelines, a set of parallel processors to comprise the computation engine with critical timing to achieve claimed performance. The timing requirements are embodied in the compiler. In fact compiler designers are key to the design of modern processors.

There have been modeling languages for decades, created to help address the complexity of system design. The current contender is the UML, or Unified Modeling Language. The core idea is not so hard; identify closely related activities, package them, name them and use them as building blocks - again and again. The blocks can be represented graphically. What matters is how functional blocks relate to one another. Those relationships get represented graphically as lines with a family of arrows and labels. I won't wax on, but suggest a look at www.omg.org, the Object Management Group, to learn more.

It is possible to design formally validated systems (provably correct) by using a subset of the UML. One could also use Petri nets or other equivalent techniques. Semiconductor designers have had to learn about modeling because the cost of failure was too great. Software designers, most of whom in the US are not degreed engineers, quick and smart as many are, have not learned the discipline of engineering, finite state machines, time dependency analysis.

Modeling is the future of software. Much of the design process is graphical. Ada, other than that there is a dearth experienced programmers, isn't the issue. Procedural languages deal with one function at a time. In the real world there are lots of things happening in parallel. Some of those things (the word object is usually used, and vague, but I'll avoid it) need to know the state of other things - does the door lock control need to know when the car is moving? The UML and its relatives are implicitly parallel discriptive languages. They all provide rich mechanisms for defining the state - door open/closed - engine running/stopped. For activites to follow, one after another, would requre that the designer express functional blocks as related by a squential order. More important is that the design must characterize the required relationships between the blocks which are needed to describe the system.

Who uses modeling besides the DOD? IBM, Microsoft, Nokia, Lockheed, Borland, Motorola, GE, Grumann, Tellabs, Cisco, Nortel, Mentor, Cadence,.. I see semiconductor vendor job reqs for MS degree Engineers with UML experience using Rose or Rhapsody in Hyderabaad (India). I'm glad the Indians are getting educated because our companies can't compete without well educated engineers, and our schools are handicapping our brightest students (Students in math and engineering from abroad require about two years less coursework preparation for a Ph.D at U.C. Berkeley)

When the F22 project was begun, there were tools, but they were used primarily by the telecom industry. Telecom developed a language, an ITU standard is called the System Definition Language, SDL, which is quite similar to the UML. Ptolemy, Esterel, Simulink, Colored Petri-Nets, and lots of other languages exist. SDL is still in use, but being displaced by the UML. I thought the UML was being used for the F22 project, but don't know first hand.

Idealy fully executional modeling separates the design from the deployment. You can run the model, and decide later what hardware, how many signal processors or general purpose cpus, memory, message transports, displays, etc. you need. The intellectual property, how to control the elevators, how to evaluate targets, how to monitor engine condition, is all preserved in models. If no one touches source code, fixes are made by fixing the model. The integrity of the design is preserved, ready for translation to new hardware without touching the model, though work will have to be done to the translation module.

The model is also the specification - the document - of the design. Models, because their allowed states are well defined, provide sufficient information for generating tests automatically. Products are commercially available from hundreds of vendors.

The DOD appears to be addressing software complexity. NASA isn't as disciplined, as the Rover programming problems atest. (Rover wasn't modeled) How about Boeing, which built an airplane, the 767, with software which cost more than the airframe. I hear that Boeing "end-of-lifed" the 767 before the first plane flew. I'll bet they model the software for the next airplane!

No, they are not similar. APL is basically an algebraic approach to programming, and mostly deals with analytic and abstract information processing problems. Since it has good set manipulators it also fond its way into early database work. It is an extremely high level language, one of the most level in fact. Outside of I/O mechanisms there are hardly any features in it that have anything to do with systems engineer, practical manipulation of computing resources and real time system programming.

The core of APL was invented by Kenneth E. Iverson when he was a grad student at Harvard but most of the work was paid for by IBM. Iverson joined IBM after Harvard and I believe that he became an "IBM Fellow" out at the T. J. Watson Research Center, which at the time was their "blue sky" research center, not an engineering center. Incidentally I knew and worked briefly with his son on an APL project once.

It is worth noting that IMB mainframe terminals had APL symbols on their keyboards.

APL has become a bit of a relic as far as computer languages go. There are still some investment banks, program traders and hedge funds that still use it in their programming trading departments, outside of that it seems to be have been left to the "antiquarians" and hobbyists in academia. It appears to have fallen by the wayside as a primary language much like LISP or Prolog has. As a mathematical language it has been superseded by "Mathematica," a language that incorporates many of APLs better features.

ADA on the other hand was designed expressly for software engineering. Most particularly its focus was embedded systems and generally in the domain of aerospace. It was created under the auspices of the DoD (but really the USAF.)

It emphasizes modularity, encapsulation and "generic programming." If you know OO concepts, then strip out thing like inheritance, polymorphism and message passing and replace them with a notion of encasulated "flat objects" (called Packages in ADA) that can be "parameterized" and "generic functions." Ada forces the notion of "Code everything once and in one place" approach to engineering. Its basic and most powerful features are Packages and modularity, Exception handling (it is very good at this,) "generic programming "(essentially writing algorythms that operate on unknown and undefined data types,) and parallel and concurrent programming particularly in an embedded real time environment.

It is a actually great language but never managed to get the "mindshare that other languages got. Part of the problem may be that it came just a little too early in the OO "revolution" that occurred in the late 70s and the early 80s. In got superseded by newer languages like C++, smalltalk and later java.

As a UML buff and someone that has worked in DoD avionics facilities let me answer that.

The F-22 predates MoD approaches, or let us say that it uses "incipient" MoD approaches - it obviously predates the UML or the MOD approach as it is a 20+ year old project. And of course as you say they are a leader and this gets transitioned in one way or another back to this effort though it is not its major thrust. Designs were to be "backed modeled" to the UML, and new design models in the UML but these tend to be higher level usages of the UML and aimed at "RUP like" requirements engineering. Another area that the UML comes in is obviously describing state machines. The problem with these approaches is that it is deeply hard to capture in a reasonable way all of the millions of interaction between the various objects solely in the UML. This is particularly true in descibing very complex real time state machines that are not really of the linear "nested" sort that the UML is so good at describing (I know that you can describe other systems with the UML, but just reflect on it for a second and you can imagine how unwieldy it might get.) Whatever the Air Force might say, the UML approach has in fact been used as a design aid rather that the sort of "compilable design" that you are pointing to, at least in embedded real time systems. They tend to use it more in "traditional" information systems (eg. non embedded systems.) They are less than honest about this and, of course, there are competing camps and differnet schools of thought and practice in that community.

But in a way you are quite right about the source code issue but for an entirely different reason. Typically in avionics enviroments there are at least three development environments: 1) a developer's simulated "sandbox" which is ussauly just software and has nothing to do with the target environment (it could just be a standard UNIX environment but could be a real time UNIX - or other - environment,) 2) An "intermediate testbed" that could just be a software environment or could have some real target hardware (or combinations there of,) and 3) the actual target hardware ans software test bed. Each one othese envionments could have entirely different Hardware, machine laguages and operating systems.

The point is that in an ADA shop you would work in ADA and cross compile to whatever machine language the testbed ran in. So architects and programmers really are just thinking in ADA, not machine code or in many cases not even middle level constructions like "setjumps" that you might see in "C" or "C++." <

As an aside, while I am a big believer in the UML, there have been quite a bit of problems using it to capture every aspect about a problem domain and practically work out the sort of "compilable design" you favor, and this is particularly true on large projects and most particularly true in embedded real time. Three seems to be some middle ground that encompasses round trip engineering approaches and other "unwisely" engineering approaches. It is a laudable goal but so far more honored in its breach than in its realization, no matter what the MoD geeks at the OMG say about it.

Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.