Code Review of Ferguson’s Model

Lockdown Skeptics ^ | Unknown | Sue Denim

[Renkluaf]

Imperial finally released a derivative of Ferguson’s code. I figured I’d do a review of it and send you some of the things I noticed. I don’t know your background so apologies if some of this is pitched at the wrong level.

My background. I wrote software for 30 years. I worked at Google between 2006 and 2014, where I was a senior software engineer working on Maps, Gmail and account security. I spent the last five years at a US/UK firm where I designed the company’s database product, amongst other jobs and projects. I was also an independent consultant for a couple of years. Obviously I’m giving only my own professional opinion and not speaking for my current employer.

. . .

Conclusions. All papers based on this code should be retracted immediately. Imperial’s modelling efforts should be reset with a new team that isn’t under Professor Ferguson, and which has a commitment to replicable results with published code from day one.

On a personal level, I’d go further and suggest that all academic epidemiology be defunded. This sort of work is best done by the insurance sector. Insurers employ modellers[sic] and data scientists, but also employ managers whose job is to decide whether a model is accurate enough for real world usage and professional software engineers to ensure model software is properly tested, understandable and so on. Academic efforts don’t have these people, and the results speak for themselves.

[Renkluaf]

Thought the conclusion said about everything that was needed.

[sauropod]

bkmk

[FreedomPoster]

Can’t wait to get comments on this from my software pro buddy.

[mewzilla]

To my mind the most important question is did Ferguson screw up, or was this deliberate?

If it’s the latter...

[Bitman]

You mean that someone wrote code that is worse than H-1B product? That is saying something!

[ml/nj]

To my mind the most important question is did Ferguson screw up, or was this deliberate?

I'm a 40+ year software guy. I did a quick scan of the article at the link; and also followed one of the links there to a code example..

My own belief is that this is totally ineptitude.

ML/NJ

[moovova]

From Robert, in one of the comments below the article...

“Science is about discovering the truth, no matter how inconvenient or unwelcome it may be to particular interested parties. Politics is about accomplishing the goal of interested parties and hiding any truth that would tend to impede that goal. The problem is not that “government has being doing it wrong;” the problem is that government has been doing it.”

[kvanbrunt2]

All software models used to determine public policy must be open source. problem solved. global warming bs also fails with this requirement.

[MAGAthon]

6 May: Breitbart: Who Would Buy a Used Computer Model from ‘Bonking Boffin’ Neil Ferguson?
by James Delingpole
Professor Neil Ferguson is currently the most reviled, scorned and mocked man in Britain.
Such a pity, though, that it’s for all the wrong reasons…
Being an arrogant, sexually incontinent, hypocrite is a fairly venial slip, after all, when compared to destroying the world’s fifth-largest economy – and with it the livelihoods, job prospects and prosperity of a nation of 65 million people...

Few men in recent British history, I would argue, have done more long-term damage to their country. It’s about time this was recognised by two important groups: by the politicians who, up till now, have been taking him seriously as their guru; and by the general populace, a significant portion of which has been terrified by his doom-laden prognostications into a hysteria out of all proportion to the nature of the actual threat posed by coronavirus.
Ferguson, let us not forget, has a long track record of failure...
https://www.breitbart.com/europe/2020/05/06/would-buy-a-used-computer-model-from-bonking-boffin-neil-ferguson/

don’t forget his married lover, Antonia Staats, is an extreme gloBULL warming activist with Avaaz. their dream is to shut down western economies:

2012: Flickr: 350.org
#EndFossilFuelSubsidies London petition delivery. Jenny Rosenberg (Friends of the Earth EWNI), Antonia Staats (Avaaz), Marco Cadena (Push Europe), Emma Biermann (350.org), Parker Liautaud (young Arctic explorer).
https://www.flickr.com/photos/350org/7393804452

[ArcadeQuarters]

Two things stand out in the article.

> the problem goes away if you run the model in single-threaded mode

In other words, multiple threads are running and each is accessing the same memory without synchronization, corrupting the numbers. When run as designed, NOTHING valid can come from this program.

> Reports of random results are dismissed with responses like “that’s not a problem, just run it a lot of times and take the average”,

Averaging results is NOT for dealing with bugs. It makes sense to use random sampling and average the results rather than run through every possibility - such as in gambling for example. They are corrupting the data in unpredictable ways and expecting it to somehow average out.

This is a multi-trillion dollar software problem.

[canuck_conservative]

All software models used to determine public policy must be open source

Can't be emphasized enough!! ... deciding public policy is no place for incompetent power-hungry prima donnas

Notice also that Bill Gates IHME team in Washington State also refuses to share their data or computer model code ... you can draw your own conclusions.

[ThunderSleeps]

My background - 40 years of coding, been getting paid to do it for 35 now. But you know what - forget all that and the rest of my resume. Let's sidestep the "appeal to authority" fallacy altogether. Anyone with a year or two experience under their belt doing professional software development knows what is reflected here is pure excrement.

Anyone who has ever seen "academic code" knows that the majority of it is nearly worthless. About the only thing you can do with it is pull some equations out of it and throw the rest away. It is typically the byproduct of a professor with an interest in some other field (other than software development) - he or she is merely using it as a tool, a means to an end, rather than a pursuit in and of itself. I'm not knocking that - hey we build tools for people to use. But I am saying that kind of focus results in poor practices and poor code quality. Academic code is also typically written and maintained by barely interested grad students just hoping to get by and keep the prof happy.

This is not a recipe for good code. The inability to reproduce results given the same inputs speaks volumes. A single 15,000 line long file is also a joke. That's indicative of the lack of professionalism and care or concern for the project. It is the path of least resistance. Well, perceived as such by people who don't know what they are doing. While good organization and file structure requires a little more work and thought to set up, you reap the rewards many times over down the road. Unless of course you're a grad student just banging out a change for the prof and you're not going to be here next semester and it'll be someone else's problem...

Why is this program even still under development? At a mere 15k lines it should have been tossed out and re-written long ago. 15k lines sounds like a lot - and it is in one file - but in the world of programming that's a couple months work for a small to modest sized team - to produce tested verified code. Heck, when I was a consultant I personally managed a codebase of about 120k lines as sole programmer. Our team right now is managing in the neighborhood of 3 million SLOC. If this project were given to me tomorrow the first thing I'd do is set it aside and never compile it again. Clean-sheet design for something that size should take a few days, maybe a week. I could have a team code it in a couple of months, or knock it out myself in several weeks of undisturbed coding bliss. ;-)

The author is right - this code is an embarrassment. Ever action taken based on the results of running it should be retracted and an apology issued.

[ArcadeQuarters]

Also... the problems exposed by the article are about the implementation. The “expert” assumptions and formulas that went into the model could be a whole other layer of problems.

[Vince Ferrer]

I am in IT, and it doesn't surprise me. In fact I wrote an email to Chanel Rion of OAN if she would ask Fauchi to release the code. This is exactly what I would expect. Academics, even in computer fields, are crap coders. What is described is probably the norm of any simulation written in academia. Never ever be impressed by an academic who has writted code.

This is also why climate scientists like Michael Mann will never realease their source code. Mann was involved in a lawsuit in Canada where the source code became part of the discovery. He refused to give the court the code. The court ruled against him and he lost the lawsuit.

[FewsOrange]

"The code. It isn’t the code Ferguson ran to produce his famous Report 9.What’s been released on GitHub is a heavily modified derivative of it, after having been upgraded for over a month by a team from Microsoft and others. "

[thoughtomator]

I already knew that academics were crap coders; this experience has taught me that they know nothing about preserving integrity or meaning of data. Some of the metrics I see are beyond stupid in the way they are constructed.

[sphinx]

For later

[lakecumberlandvet]

The author compares the Ferguson Model to Simcity. I think that is a good comparison. Simcity is a game and may be fun to play but it, like the Ferguson Model, is not reality. A model is just one tool in the toolbox that may be used to solve a problem, not an end-all.

[marktwain]

Anyone who has ever seen "academic code" knows that the majority of it is nearly worthless. About the only thing you can do with it is pull some equations out of it and throw the rest away. It is typically the byproduct of a professor with an interest in some other field (other than software development) - he or she is merely using it as a tool, a means to an end, rather than a pursuit in and of itself. I'm not knocking that - hey we build tools for people to use. But I am saying that kind of focus results in poor practices and poor code quality. Academic code is also typically written and maintained by barely interested grad students just hoping to get by and keep the prof happy.

I have had personal experience with this. Academic code becomes a tool used by politically minded professors to convince policy makers to approve of the policy they want.

It makes them look authoritative, but the fact is a computer simulation easily gives the person(s) writing and running it the results they want.

If it doesn't, then "something is wrong with the code" and it has to be tweaked to get the "right" result.

I saw a hundred million dollar copper zinc mine in Wisconsin be legislated into oblivion (never developed or opened) because the legislators taxed it to death based on bogus academic code, written by grad students for a Marxist professor.

When I read the code, years later, it was clear one character change was the major culprit. A cost was being treated as income. It made all the difference.

Computer models should never be relied on for political decisions.

[SteveH]

i am skeptical that microsoft would be the optimal entity to fix any problems. imho microsoft has been unable to fix the windows os, and windows was created by microsoft decades ago.

ideally, a software simulation would require some kind of independent verification, especially if it is monte carlo. one possibility would be an independently developed software simulation.

software management is difficult in the best of situations.
it sounds as if the university research management suffers from tunnel vision. regardless, regression suites are standard practice. monolithic source code program files do sound very much like ancient practice and not something originally written in either C or C++.

i presume fauci and birx referenced ferguson in all of the covid19 task force deliberations and conclusions, and if so, this would seem to explain a great deal.