The Shocking Reality: Developers Behind Boeing 737 Max’s Software Issues

Medium ^ | 21 Aug, 2023 | Vishnu Ravi

[MtnClimber]

Unveiling the Truth: Boeing’s 737 Max Software Development Unraveled

In the realm of aviation technology, the Boeing 737 Max series has remained a topic of intense scrutiny and debate. Recent revelations have shed light on a pivotal aspect of its development, exposing a web of intricate details that could reshape our perception of the entire saga.

Boeing’s Strategic Outsourcing Approach

Startling reports have surfaced, divulging an unconventional strategy employed by Boeing during the development and testing phase of the 737 Max’s software. This strategy involved outsourcing crucial responsibilities to a pool of temporary workers and fresh college graduates. These individuals found themselves under the employment or contract of esteemed Indian technology firms, namely HCL Technologies and Cyient Ltd.

The Quest for Efficiency and Its Consequences

Intriguingly, the workforce engaged by Boeing for this pivotal task was comprised of low-paid professionals. This unconventional move, while seemingly aimed at enhancing efficiency and cutting costs, ultimately raised serious concerns within the industry. Veteran engineers who have had the privilege of observing this process firsthand have pointed out alarming irregularities in the code’s execution.

The Shadows of Imperfection

These seasoned engineers, with their wealth of experience, did not mince words when it came to assessing the work undertaken by the outsourced engineers. They assertively state that the code, which forms the backbone of the 737 Max’s software, was marred by inadequacies and inaccuracies. Their unanimous sentiment was that the code had not been meticulously crafted and executed in the manner it deserved.

Anatomy of the Flawed Software

While it’s essential to cast a discerning eye on the role played by the outsourced workforce, it’s equally imperative to understand the broader context of the software’s malfunction. The glitch-ridden software, which has become synonymous with the ill-fated 737 Max series, can be traced back to a fundamental design flaw embedded in the plane’s Maneuvering Characteristics Augmentation System (MCAS).

The Crucial Link: MCAS in Focus

MCAS, heralded as a pioneering advancement in aviation safety, was meticulously designed to avert potential stalls during flights. However, a single point of vulnerability within its architecture led to catastrophic consequences. MCAS was intrinsically engineered to depend on data sourced from a solitary sensor. This inherent design choice, while meant to streamline operations, inadvertently exposed the system to erroneous readings.

A Chain Reaction of Devastation

The vulnerability stemming from the sole reliance on one sensor spawned a chain reaction of events that would go on to haunt the 737 Max series. The discrepancies in sensor readings had the potential to trigger the MCAS into action, causing it to erroneously force the aircraft’s nose downward. This unsettling behavior played a pivotal role in the tragic sequence of events that culminated in two fatal crashes, shaking the very foundation of Boeing’s credibility.

Lessons Learned and the Path Forward

In hindsight, the calamitous outcomes attributed to the 737 Max’s software issues underscore the dire consequences of compromise in aerospace engineering. This sobering saga serves as a testament to the indispensable nature of precision and rigor in such critical endeavors. It compels the industry, regulators, and stakeholders alike to reevaluate their approach to safety, innovation, and quality assurance.

Conclusion: Navigating Turbulent Skies

The saga of Boeing’s 737 Max software development illuminates the intricate interplay between innovation, oversight, and accountability. The outsourcing strategy, though well-intentioned, cast a spotlight on the importance of striking a delicate balance between efficiency and excellence. As the aviation industry charts its course into the future, the lessons gleaned from this tumultuous chapter will undoubtedly serve as a compass, guiding the way towards safer skies for all.

In conclusion, the revelation of Boeing’s choice to outsource critical software development tasks to a contingent of low-paid workers and recent graduates opens a window into the complex world of aerospace engineering. The resultant issues that plagued the 737 Max’s software underscore the significance of maintaining an unwavering commitment to quality and precision in such a high-stakes domain. This introspection must steer the aviation industry toward a future that prioritizes not only innovation but also the unfaltering assurance of safety and reliability.

[cymbeline]

“... (and a failed sensor input would be one such obvious case, especially if it’s not redundant”

I am not an aviation expert but a pilot friend blames the pilots of the crashed planes because they should have known to disengage the MCAS. Perhaps inadequate training.

If I were designing a plane, I’d want its autopilot to have as many redundant inputs as possible, and I’d want it to tell the pilots if it’s seeing anything strange in its inputs, and even suggest what the pilots should do in those cases.

Prototypes were flown a lot by test pilots. I doubt that those pilots are allowed to speak publicly on what they thought. And since the MAX was deployed, they’ve been flown by hundreds of pilots. Wonder what they would have to say about the plane and specifically the MCAS.

My final poorly informed conclusion is that the plane could have been more mediocre-pilot-friendly.

[T.B. Yoits]

The Boeing 737 Max should never have been built. It's unstable and flies "goofy foot" which might be acceptable for military aircraft such as the F-16, F-117, etc. but is NOT acceptable for civilian aircraft. That's on the executives, the engineers, and the FAA.

The MCAS systems should have never been implemented but once installed, should have had redundant systems. That's on the engineers and the FAA.

The code writers should have called full stop, asking what the code writing would be for the redundant sensors. If they had done so, and found out there were no redundant sensors, they should have refused to write the code.

If someone walks into your shop and asks you to program a throttle for their car with no shut-off, you don't write the program.

[citizen]

Winning comment!

[Skywise]

You could design in something like a logarithmic analysis of the data over the previous X readings and if something seems out of whack (like a sudden spike change in the reading) you could ignore the reading and not engage. But that’s kind of dependent on how the sensor fails. Without that second sensor the software has to “guess” if the sensor data is valid and engage safety protocols or not.

I agree there was a complete business process failure here on all sides. Software design should’ve pushed back on the design in general and hardware shouldn’t have gone with a faulty design to begin with. That I will place the blame squarely at the feet of poor management (maybe H1B there) and/or an emphasis on hiring for identity rather than capability.

But unless there’s an actual bug there - I’d say the software guys did the best with what they could from a pushed down design.

[flamberge]

The shoddy work of Musk’s H1-Bers on display.

These were outsourced developers working in India for Boeing. Boeing cut out all the middlemen and got them for rock-bottom prices. Somewhere between $5 and $9 per hour. Let's see American or European developers "compete" with that!

The results do not speak well for the competence of Indian software development training or Boeing management. "But they are so cheap", say the executives.

[nagant]

I follow the “South Main Auto” Youtube channel. The mechanic there just encountered his 2nd Kia SUV where a failure in the backup camera rendered the cars into bricks.

This cr@p has to stop. Stop making drive systems rely on computers. Period.

[citizen]

Really? I did not know BOTH crashes were piloted by Moslems. hmmm

I speculate that a woke company like Boeing would admit to anything other than casting suspicion on Islamic piolots.

[Dave Wright]

Some minor corrections to this story. First, the MCAS software and related systems performed exactly as designed and were not responsible for the Ethiopian and Indonesian aircraft accidents. The MCAS sensor selection depended on the aircrew’s manual selection of flight controls. The left and right seat controls indicated a large difference in angle-of-attack, which should have caused them to select the reading corresponding to their attitude relative to the horizon. This would have restored normal flight operations.

Instead, they misdiagnosed the false stall warning and nose-down trim, believing the automated system rather than the reality they could see outside the cockpit. This corresponded to a memory checklist item, Runaway Stabilizer Trim, that all pilots must train on, so they don’t have to use the operations manual. Due to a lack of training, the inexperienced first officer began thumbing through the manual rather than immediately switching off the electric stabilizer trim. This would have again restored normal flight operations.

The incorrect angle of attack reading on the pilot’s side was due to maintenance crews damaging the left side AoA vane and not repairing and recalibrating it properly when a prior flight crew reported it.

None of this relates to Boeing software engineers or where they came from. The MCAS was modified to alert crews when there is a discrepancy between the AoA sensors, but this just added a bell-and-whistle to the standard flight training that requires them to crosscheck their instruments for these anomalies.

[Paal Gulli]

"Regardless of the software issues, the problem goes back to management who decided it was more expedient and cheaper to install much larger engines then the original airframe had been designed to accommodate."

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ THIS ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

If a pilot has a type rating on one aircraft "type" (B-747, MD-80, SAAB-340, etc) he can do more of his FAR-required training and check-ride "flights" in the simulator rather than the actual aircraft, which obviously amounts to a YUGE reduction in overhead for the operator.

So it's a BIG selling point to an air carrier if you offer them a new a/c that their pilots' existing type ratings will cover. This is why Boeing elected to put engines so oversized on a 737 airframe that it required longer landing gear. Because it would be the same "type" as the older 737s.

Have you ever seen a Boeing 737-400? Did you notice how oddly shaped the engine nacelles were?

The original 'Jurassic' 737s had engines with a much lower bypass ratio, so the engine was skinnier and the engine intake (and to an extent, the rest of the nacelle) could be made circular.

But as higher-bypass (and hence more efficient) engines became available, Boeing's engineers were faced with the problem of how to cram them into the space between the bottom of the 737's wings and the surface of the runway.

And that's why the -400's engines have flat bottoms.

But the engines for the Max were going to be so large that that trick was no longer effective, so they resorted to taller landing gear.

But having such large engines also moved the thrust they created further below the centerline of the airframe. Which reeked havoc with the aerodynamic balance of the airplane. Which necessitated a different stall prevention system.

So if Boeing hadn't been hell-bent on getting the FAA to certify the Max as a 737, this might never have happened.

[flamberge]

That was a great "true story"!

I don't believe a word of what that ex-CEO said, but it was still a great story. Very entertaining.

Thank you.

[citizen]

I understand. I see cheap labor, as the prime motivation of mgmt, the same - whether on foreign soil or in-country as H1-Bers.

[GMThrust]

I was the primary flight tester for the Air Force’s development of the MH-60 Simulator. I spent months working directly with the software engineers. They would make a change and down the line, some other item was affected. Bottom line, it was a painstaking experience going over line by line of code to get it close to reality.
Pilot training is the larger issue here. We used to have to be a “little drop of oil or gas or electron” and draw out the schematics of the systems in detail and go through all the possible emergencies. Now, if it’s green it’s good.

[citizen]

Your comment and some others illustrate the danger of having MBAs controlling instead of a cadre of seasoned engineers.

[RckyRaCoCo]

You're welcome.

While many (among the rank and file) called Bob Crandall a tool and a "stool"...he had his moments.

[fella]

MBA think; Cheapest (Efficiency) is best.

[Tahoe3002]

I worked 30± years in engineering & construction companies. A big selling point for many clients was the use of “low cost” engineering centers,usually in India or South America. Frequently, when engineering drawings, calculations or software were received in our Houston office, it required massive amounts to of rework by American trained engineers.

The client effectively paid twice for the same work. But, hey all billable hours to the client.

[Sequoyah101]

Don’t tell me, let me guess. The developers and coders were a bunch of H1B dot Indians who won the low bid for the work to Boing.

[PLMerite]

[Have you ever seen a Boeing 737-400? Did you notice how oddly shaped the engine nacelles were?]

Makes for an easier wheels-up landing when the onboard systems fail.

[Sequoyah101]

Interesting isn't it? Interesting how total reliance on software and systems generally leads to total failure when it happens not to work.

Of course, everyone knows that software and sensors are totally reliable and never ever fail. Right? /s

[Montana_Sam]

MCAS was intrinsically engineered to depend on data sourced from a solitary sensor.

The pseudo code for the initial meeting looked somewhat like:
... if(sensor.Reading == isBad)
{
crash();
exit(1);
}
Code passed review and the dev team was teeing off the first hole by 10 a.m.