HealthCare.Gov Needs Five Million Code Lines Rewritten

national review ^

[Sub-Driver]

HealthCare.Gov Needs Five Million Code Lines Rewritten By Andrew Johnson October 21, 2013 9:13 AM Comments 42

Obamacare’s online exchanges have been riddled with problems since they came online three weeks ago, and those issues may continue for at least the next few weeks. Contractors said fixing the problems by the November 1 deadline set by the administration would be “unrealistic,” according to the New York Times.

From the sluggish websites to garbled enrollment information, the flaws require the extensive rewriting of code: “One specialist said that as many as five million lines of software code may need to be rewritten before the Web site runs properly,” the Times reports — that’s out of a total of approximately 500 million lines of code, according to another expert.

Others experts warned that some of the website’s problem are yet to come. One technical specialist involved in the repair effort said, “The account creation and registration problems are masking the problems that will happen later.”

[cpdiii]

Wally’s dream job. Any attempt to fix the 5 million lines will certainly break the other xxx million lines. So why even start?

DING DING DING, we have a winner. It does not work and the fix will cause problems with the code that is working.

[Redleg Duke]

Okay, how about Helen Thomas, Hitlery Clinton, Andrea Mitchell, Mika, Madonna, Joy Behar, Woopie Goldberg, Barbara Streisand, and Seibelius...and no, you cannot have any paper bags!

[Mr. K]

Someone said they spotted copywrited code in there, so I think the programmers cut and pasted snippets of whatever they found that worked (illegally or not)

[dfwgator]

Just rewrite it from scratch, it will be quicker.

[Lazamataz]

I plead the 5th.

[i_robot73]

Nor would I, nor my group, given the (6) days of testing have allowed it to be pushed into PRODUCTION.

Of course, being MILLIONS over-budget, using other peoples money, what’s the matter to gov’t? /s

[Windcatcher]

FR might run on PHP, but it doesn’t require secure transaction management with rollback and faiover.

Let me second the OMFG. They are so unbelievably SCREWED!

[Redleg Duke]

Don't plead it...guzzle it!

By the way, Madonna has begged off, but Nazi Pelosi is available.

[PapaBear3625]

THEY USED PHP????!??!??!??!??!??! Unfixable. This thing may never work. That said, FR runs on PHP, but it is the brainchild of one person (more manageable), much smaller scope (more manageable), and evolved over 12 years (more manageable) — and it still has some bugs.

There is a reason why major software projects use something scalable like Java, or C#.

[sergeantdave]

Try vacuum tubes and punch cards.

[PapaBear3625]

No, you only saw Oct 1 get here, and media people could not even complete a form. Again, the whole rollout was flawed from the beginning, and if the Tea-Party fight forced the White House to push the site out before it's time - which many of us suspect, then Ted Cruz et al did their job - perhaps without even knowing it. Someone high up HAD to have seen these errors, and said "Launch it anyway"

It's funny. I'm wondering if Cruz had some inside info on the problems of the website.

He and the House Repubs demanded that ObamaCare be postponed for a year. So naturally Obama COULD NOT allow Obamacare to be postponed for even a day, since that would mean that Republicans had "won". So it rolled out on schedule, to be a complete disaster, because the boy-child could not give even the appearance of the Repubs winning at anything.

This suggests a strategy: whatever you want done, issue a demand to Obama to do the opposite.

[Robert Teesdale]

You would think that there would be screaming done at some level about it not being ready.

I cannot imagine security is so tight on the teams involved, that no one leaked "this is going to bomb" ahead of time to any serious degree.

[Safrguns]

>>> Finding faulty logic in a program that doesn’t work, is another thing entirely.

As a programmer on IBM mainframes for 12+ years, one of the terms frequently used to describe aged or cumbersome software was: “Spagetti Code”.

Spagetti Code happens as a result of last minute or rushed requirements forced into a system to make it work by the deadline. It also happens as a result of modifying an existing program to do new things instead of streamlining the logic and re-writing a new program. If you look at the Obamacare flowchart, you might get a simplified visualization of what spagetti code can end up looking like.

Besides the obvious problems of decreased maintainability of the code, and increased risk of logic errors, the greatest risk associated with “spagetti code” is that only a select few programmers who are most familiar with the code are able to maintain it... let alone understand it.

It is also the most tried and true way a system programmer can actually manufacture his/her own job security into the software with the valid claim that time constraints prohibited a “cleaner” end product.

Everything I’ve been hearing about what has been done, what has NOT been done, and what they are doing now tells me that at some point in the very near future they will be forced to revert to a manual process while maintaining the requirement that everyone enroll in the system (set up an account) I just read an article yesterday reporting that the Obamacare website included FREE software that was available and was modified to remove copyright credentials... and that the software company that was infringed upon is filing suit.

I’t looks to me like the people who were responsible for designing Healthcare.gov on paper, and passed those requirements on to developers are the same people responsible for manufacturing Obama’s on-line birth certificate...

It reminds me of a popular cartoon where a program manager says to a developer.. “Go ahead and start writing the program... we will provide the parameters and requirements next month.

Now it is next month, and the developers are being asked to fix an un-fixable product in a timeframe that makes the task literally impossible to do.

OH... and the real kicker is this: Starting from scratch will not work either because of the GIGO priciple (Garbage In = Garbage Out) Programs at their core are nothing more than the automation of a manual process that is already proven. Even the designers of the system do not know what they are doing, and will therefore NEVER be able to properly communicate their requirements to developers.

[Cyropaedia]

>To write 5 million lines of code of that type is about 1,000 man months just to write it. Toss in the typical government overhead of ‘management’, the endless meetings, document rewrites, etc, and that can easily be 50,000 man months.

Lucky for them, the new project manager is on the case :

"Um, yeah... looks like I'm gonna need you to come in on Saturday...".

[Cyropaedia]

With this administration, they are so used to having the MSM/JournOlistas conducting De Facto damage control for them ( Fast and Furious, Benghazi, IRS etc. ) that they were simply at a loss with this debacle.

Here's a system that everyone in the country is going to have to be able use directly; - and discover the actual truth for themselves.

[Lou L]

Everything I’ve been hearing about what has been done, what has NOT been done, and what they are doing now tells me that at some point in the very near future they will be forced to revert to a manual process while maintaining the requirement that everyone enroll in the system

HealthCare.gov 2.0 will be an online form that if one is lucky, they'll be able to print or download, and send in the mail.

[Hieronymus]

The problem being that manual entry into the system and billing will be prone to error and time consuming as well—plus it will be relying on the U.S. postal service, which loses enough stuff to complicate matters. It also isn’t sexy enough, so I doubt that DC is capable of humbling itself that far.

[Billthedrill]

As a thankfully-ex PM I think I can see a way out of this, but it's politically impossible. What they need to do is (1) delay the program for a year, (2) completely review the underlying document with an eye to simplification of the decision trees, (3) continue the signups to convey the impression to the public that everything is just fine.

Item (2) is the stickler. The legislation is broken. Code Toad posted the - well, I suppose you could call that very high-level flowchart, and it's a mess. If you try to recreate that with one monolithic piece of code without breaking it up you'll never finish.

There is another issue. The ACA contains a large number of "The Secretary shall determine" statements that cannot be coded because they refer to decisions that haven't been made yet. Well, they can be coded, but I don't think very many people will be satisfied with a window that pops up and says, "We'll let you know later. Trust us."

The real issue is that the ACA was written on a legislative model that was already pretty clumsy - every Democrat Representative and Senator inside the beltway slathered everything he or she ever wanted or imagined that they might want later, the idea being to get the thing approved and massage it to do pretty much anything they wanted once it was in place and unassailable. It isn't a working piece of legislation, it's a giant piñata, it's broken, the candy is all over the ground, and some poor PM is being tasked with putting it back together in a form that works when it never worked in the first place.

[OneWingedShark]

Right. Perl. I'm sorry, I was wrong. But isn't Perl about as bad? Weak typing, lack of programmatic boundries (no isolation), etc. Still, John pulled it off.

I think you're right, for the most part.
This site has the advantage of being, pretty much, all text-processing… healthcare, not-so.

That the ACA site was written in PHP is utterly mind-boggling.

I agree.

[OneWingedShark]

From a webdev standpoint, The HealthCare.gov website's functions should be as follows:

1. Accept input data via forms
2. Write input data to a database
3. Read input data to display on screen, reporting or document creation
4. calculate insurance costs based in data input

Conceptually, yes; implementation wise, no… precisely because data needs to be validated between #1 and #2.
Here's why: Healthcare data.

Consider something as simple as a social security number; off the top of my head:

1. JavaScript dependence/disabling: nothing in this site should require javascript, as its use is [essentially] mandated by the government and therefore must comply with ADA and accessibility standards, including text-only browsers.
2. Because of #1 you cannot count on client-side validation, therefore *all* validation needs to be done in the client-side code.
3. Because of #2, a SSN may come in in several forms: dashes, no-dashes, spaces, etc. (You can do this w/ regex, though regex solutions are typically too brittle for complex [i.e. non-regular] data.)
4. Because of #3, combined with the general inability to declare types, it is now incumbent on the programmer to do [at least some] flow analysis when making any changes. (i.e. ensure all inputs to the function are validated.)
   (Mitigated if using OOP; however, OOP is rather bolted-on in PHP, there are many tasks for which OOP is an unnatural solution, and there can be memory concerns on some systems [admittedly may or may not be an issue in newer PHP versions].)
5. Now we need to add it to the DB&hallip; wait, did you know that a SSN isn't unique? (Did you set up the DB assuming they were? Is your code assuming they are?)

Validation though is the 'core', if you will, of verifiable S/W — which you want for critical medical systems. (I'd argue for even non-critical medical systems.) The above is actually very small, and as such there are things you don't even need to really account for, like poor Bobby Tables, because the data is just a numeric-string — once you get into validation PHP quickly becomes terrible, because of the loose take on types.

The above validation-insurance for SSN can be expressed in Ada as this:

    Type User_ID is new Positive; -- ID for a record; cannot be confused w/ a integer-count.
    
    Subtype Digit is Character range '0'..'9';

    -- The following is a string, of length 9, that has ONLY digits.
    Subtype Social_Security_Number is String(1..9)
    with Dynamic_Predicate =>
      (for all C of Social_Security_Number => C in Digit);
    
    -- The following function cannot have an invalid invalid SSN parameter.
    Function Save( SSN : Social_Security_Number ) return User_ID;

I can make the above guarantees because they are ensured by the language; without ever looking at the body of Save I can tell you it cannot be called w/ an invalid SSN and that it will return a positive integer. Moreover, the guarantees can be used to prove the program's correctness w/o relying on any flow analysis. (Flow analysis might be needed on the internals of functions, but not on the interfaces; thereby allowing for far better modularization and subsystems.)

---

Sorry folks, but all that is Web Development 101, and even other government websites that do the same thing seem to work without a problem. I estimate the HealthCare.gov site could have been written in no more than 5000-6000 lines of code using existing open-source frameworks (and that's being generous).

Perhaps, but how confident are you that you can say the data in such a DB is valid? [IIRC, there's stuff in ACA about mandating electronic medical records.] Moreover, how much would you want to be forced, under penalty of law, to use a site that isn't actually validated? Howabout submitting sensitive medical data, upon which your life may one day depend (drug allergies, perhaps)?

I don't like that idea.