'Blue Screen of Death' on Oil Rig's Computer ( Deepwater Horizon Oilspill )

HardOCP ^ | Saturday July 24, 2010 | Al

[Ernest_at_the_Beach]

Testifying at a federal hearing on Friday, the chief electronics technician told of numerous instances of a "Blue Screen of Death" on the computer system responsible for monitoring and controlling drilling. The largest oil spill in American history may be due to a simple computer glitch.

The machine had been locking up for months, Williams said, producing what he and others on the crew called a "blue screen of death." "It would just turn blue. You'd have no data coming through," Williams said today, according to the New York Times' story. With the computer frozen, the driller would not have access to crucial data about what was going on in the well.

[bigbob]

Slight clarification - there are versions of Windows with real-time extensions, such as iNTime and Venturecom RTX, that are used by some industrial applications. No idea what BP was using, but I’m sure it will come out. And despite what Microsoft says, I wouldn’t trust CE or Mobile or embedded XP for anything boom-able or critical. Been there, done that.

[loungitude]

If it’s a high-vibration environment, that can wreck havoc on hard drives and connectors.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

A marine environment is rough on electronics, I’d bet.

[auboy]

Some people just ain't very bright. And never rule out the lazy factor.

[loungitude]

I recall reading the specs on that rig. Seven 10,000 HP diesel generators.

[Ernest_at_the_Beach]

Found this from link at TOD:

Software Products for Industrial Automation

[UCANSEE2]

Disabled gas sensor, broken BOP, failing computer systems

Audible alarms disabled.

Unusually high pressure readings.

Unexpected spikes and drops in pressure.

Seismic indications of unstable seafloor. Leaks of oil and gas from seafloor (NOTICED WAY BEFORE THE RIG CAUGHT ON FIRE).

Waivers to skip the waiting period for the cement plugs to cure.

Waivers to jump to replacing heavy mud with seawater ahead of schedule.

The 'crew' was ready to go home. (truth was they wanted off the rig before it 'blew'.)

The "MISSION ACCOMPLISHED" signs were already paid for.

There was a 'party' going on celebrating their success.

The EXPERIENCED CREW wanted to follow safety procedures, the INEXPERIENCED STUFFED SHIRTS were only concerned with the 'official turnover' of the RIG going ahead as planned, damn the torpedoes.

Missing Hangers.

Cheap CHINESE steel pipe casing (with Melamine for more flavor). (another 'waiver' from MMS for BP).

Dead batteries in 'REMOTE BOP ACTIVATORS'.

And that's just off the top of my head.

[worst-case scenario]

Your list is BRUTAL. Thank you for compiling it I only wish you didn’t have to.

Definitely trial for murder time. I don’t care HOW high this goes.

Take Salazar and Hayward and Suttles and put them on the stand. Americans need to know what’s happening to their people and their land.

[Ernest_at_the_Beach]

More from #65:

Productivity3000® PAC Software

***************************************EXCERPT***********************************

... more cost effective than any control solution in the market!

ProductivitySuite Programming Software

The Productivity3000 programmable automation controller (PAC) is a compact controller that combines the features and capabilities of a PC-based control system with that of a typical programmable logic controller (PLC). The Productivity3000 offers you a more powerful control solution that is easier to use and more cost effective than any control solution in the market.

ProductivitySuite is user-friendly programming software designed to allow quick and easy programming of ladder logic programs for the Productivity3000 PAC. The online help file provides information that will help you get acquainted with the software quickly.

• User Defined Tag-based Programming
  Descriptive naming of I/O points and other internal tags that make sense for the application
  • Strong data typing to reduce obscurity
  • Arrays for process management (easier data management, recipe handling)
    
    
• Simple to Use Instructions
  Practical instructions that help solve problems
  • No stack to manipulate
  • Easy math eliminates the need for using complex rungs to do simple calculations
    
    
• Application Tools
  Allow quick access to many of the advanced features of the ProductivitySuite software
  
  • Setup – Hardware Config, Security Accounts, Set PAC Time/Date
  • Write Program – Tag Database, I/O Overview, Rung Comments
  • Monitor & Debug – Data View, Data Logger, PID Tuning, Bit Histogram, Word Histogram, PAC Error History, PAC Event History
  • Control PAC – Offline, Online, Choose PAC, Transfer to PAC, Run, Stop, Debug
    
    
• Task Management
  Task Manager allows quick and easy prioritization of Ladder Logic code. By breaking code into logical tasks, the task manager can then easily change priorities or disable parts of the code.
  • Run First Scan Only, Run Every Scan, Run Every Second, Run When Called, Disable Task

Visit the in-depth Productivity3000 product Web site

Application Instructions

• Contacts (5) – NO, NC, NOE (NO Edge), NCE (NC Edge), CMP (Compare)
• Coils (10) – Out, Set, Reset, OR, Flasher, Debounce, Timed, Toggle, Program End, No Operation
• Application functions (12) – Alarm, Average, Change, Min/Max, Learn Alarm, Limit, Ramp, Ramp Generator, Scale (linear), Scale (nonlinear), Summation, Switch
• Array functions (4) – Array Statistics, Copy, Fill, Shift/Rotate
• Counters/Timers (4) – Simple Counter, Counter, Simple Timer, Timer
• Communications (13) – ASCII in, ASCII Out, Clear Serial Port Buffer, Custom Protocol In, Custom Protocol Out, GS Drive Read, GS Drives Write, Modbus Read, Modbus Write, Send Email, DataWorx Request
• Data Handling (13) – Absolute Encoder, Compare Values, Copy Data, FIFO/LIFO, First Bit On/Off, Inc/Dec, Logical Bits, Logical Words, Lookup Table, Pack Bits, Shift/Rotate Bits, Sign Magnitude, UnPack Bits
• Drum sequencers (2) – Drum, Sequencer
• Math functions (2) – Math Editor (MATH), Data Statistics
• PID (2) – PID Loop, Ramp/Soak
• Program control (6) – Call Task, For Loop, For Loop Break, Next Block, Stop Program, User Defined Fault
• String functions (7) – Compare Strings, Copy Character, Extract String, Find String, Pack String, UnPack String, String Length
• System functions (2) – LCD Page, Set PAC Time

ProductivitySuite PAC Programming Software

PART NO.	DESCRIPTION	PRICE	INFO
P3-PGMSW	Productivity3000 PAC programming and documentation software Free Download	$495.00
P3-USER-M	Productivity3000 User Manual Free Download	$29.00

DataWorx P3K PAC Data Logging Software

Part No.	Description	Price
PC-DATP3K-1	Support for one Productivity3000 PAC system	$595.00
PC-DATP3K-UN	Support for unlimited Productivity3000 PAC systems	$1,595.00
PC-DATP3K-UPG	Upgrade from a single license to unlimited	$1,000.00

Software Features

• Built-in database connectivity
• Tag name based programming
• Powerful task manager
• Easy integration with C-more HMI
• Auto-discover I/O modules
• Auto-discover variable frequency drives (VFDs)
• Documentation stored on CPU
• Comprehensive HELP files
• Plug and play USB programming
• Ethernet programming
• Nimble math instructions
• Multi-level security options
• Run-time edits
• Application specific instructions

PC Requirements

• Windows® 2000 Service Pack 4, XP Home or Professional, or Vista (32 bit only).

• Personal Computer with a 333 MHz or higher processor
• SVGA 800x600 pixels resolution (1024x768 pixels resolution recommended)
• 150MB free hard-disk space
• 128MB free RAM (512MB recommended)
• CD-ROM or DVD drive for installing software from the CD
• USB or Ethernet port for project transfer to PAC

[norraad]

..all the way to the tippy top(..or the dirty ugly bottom

[Delta 21]

They will have a 20 BILLION dollar account set up out of Microsofts coffers just like thay did with BP.

NO rule of law for the deep pockets.

[theymakemesick]

My suspicion from day one, McAfee pushed a DAT file that day that caused thousands if not millions of PC’s to boot-loop the morning the rig caught fire: http://www.bing.com/search?q=mcafee+pc+reboot&form=QBLH&qs=n&sk= McAfee admitted to not testing the DAT on XP SP3. The DAT file detected a Windows system file as a virus and deleted it which caused PC’s to endlessly reboot. It was not a Windows issue.

[jonascord]

"It took a bunch of people not doing the job for this to happen."

It took the Harvard MBAs that the Seven Sisters hire, rather then some poor tool pusher who grew up in the Oil Patch, and went to school at night. This is what you get when you treat a business and a mechanical process as a practical application of economic theory.

[jonascord]

"It took a bunch of people not doing the job for this to happen."

It took the Harvard MBAs that the Seven Sisters hire, rather then some poor tool pusher who grew up in the Oil Patch, and went to school at night. This is what you get when you treat a business and a mechanical process as a practical application of economic theory.

[printhead]

It seems they’ve left out HP’s RTE and RTE-A real-time OS’s. I can think of several sites still running HP-1000’s. Bullet-proof, maybe but I can personally attest to it being oil-proof.

[Smokin' Joe]

You will hear old hands carping about the d@mn computers being screwed up again, and it does happen, but usually it is a bad sensor in the system not tracking depth or the sample lagging program getting dumped and a couple of cuttings samples not getting caught as a result. These, while aggravating can be worked around by correcting depth and having someone paying attention to what is going on.

Looking at the wrong data is pretty bad, and the BSOD should never be tolerated in a rig data package.

Still, 'older' hands would likely rely on gauges and pit levels by eyeball rather than believe a computer which tells them something different than their observations, just because they don't trust the computers anyway.

This might have led to the disagreements which were alleged to have occurred between crews and BP management, and I can see where someone might be inclined to blow off readings which indicated adverse conditions in the wellbore as a computer error or miscalibration.

The critical error would be in not verifying that either the computer readouts were in error or that the readings were indeed correct before proceeding, especially when considering the potential for disaster. While the driller's console is an important one, there should be multiple redundencies onsite (or on the rig), not just that one display.

I wonder whose program they were running?

[Yo-Yo]

Just Press CTRL-ALT-DELETE

[UCANSEE2]

You don’t want Windows to run your car’s electronics.

Oh.... but some of them do. Chryslers do. Their car electronics runs Windows. I used to pop in a CD to install factory updates.

[cynwoody]

While I am no MS fan, is Windows the only OS that throws a blue screen when things go wrong?

In the Unix world, it's called a kernel panic, after the name of the privileged function that is called to take the system down:

PANIC(9)		 BSD Kernel Developer's Manual		      PANIC(9)

NAME
     panic -- bring down system on fatal error


SYNOPSIS
     #include <sys/types.h>
     #include <sys/systm.h>

     void
     panic(const char *fmt, ...);

DESCRIPTION
     The panic() function terminates the running system.  The message fmt is a
     printf(3) style format string.  The message is printed to the console and
     the location panicstr is set to the address of the message text for
     retrieval from the OS core dump.

     If the kernel debugger is installed control is passed to it, otherwise an
     attempt to save a core dump of the OS to a configured dump device is
     made.

     If panic() is called twice (from the disk sync routines, for example) the
     system is rebooted without syncing the disks.


RETURN VALUES
     The panic() function does not return.

BSD				August 11, 1995 			   BSD

As far as color, black and white tends to be favored over blue and white.

Linux

Mac OS X

[catnipman]

BSOD can mean either a Windows software failure, a hardware failure, or a stealth hardware failure that corrupted the software, causing it to fail.

Sounds like Transocean was using a Windows product, and most likely were using a garden variety commercial Dell PC or the like, a most deadly combination in terms of reliability, robustness, and uptime. I would really hate to see something like that used on, say, a nuclear reactor controller.

Basically, Transocean was using Radio-Shack-Quality components for this particular monitoring function, rather than an industrial computer with a reliable OS like one of the UNIX-derivative OSes.

[UCANSEE2]

Windows: n 32 bit extensions and a graphical shell for a 16 bit patch to an 8 bit operating system originally coded for a 4 bit microprocessor, written by a 2 bit company, that can't stand 1 bit of competition.

Based on an OS stolen from APPLE, who had stolen it from XEROX.