US power outage: Sequence of voltage cascade revealed

The Inquirer ^ | Saturday 13 September 2003 | Eva Glass

[mjp]

MY PAL Marty says he's seen the official reports so far of the power out that affected swathes of North America on the 14th of August last and it's not a happy story. The Joint US/Canada Task Force investigating the power down has so far isolated the time line of the cascade in the electrical grid on that day. Marty assures me that Adamson Rust's attempt to pin the blame on Chicken Little and Foxy Woxy is up to his usual lame standards. [Can't you get over this guy, Eva? Ed.]

The time line is pretty much OK, though the Task Force is unhappy about the time stamps that should note the disconnects. Some time stamps are inaccurate because the computers that record the disconnects either got backlogged or the clocks weren't calibrated properly to the national time standard. Heads will no doubt roll because of this – it makes it harder to pin down the times and the events.

It's pretty clear the whole North American power grid needs quite a bit of care and attention.

According to Marty, nearly all of the events in the blackout happened from noon Eastern Daylight until 4.15PM Eastern Daylight. But the power sleuths are tracking from 8AM on the day, EDT, just to see if anything happened earlier.

Conesville Unit, central Ohio (375MW) A generator trip happened at 12.05 Greenwood Unit, north of Detroit Generator trip at 1:14 Eastlake Unit (597MW), Northern Ohio Generator trip 1:31 Stuart Atlanta (345kV) Transmission disconnect at 2PM, because a brush fire shorted the conductors Harding Chamberlain Transmission line disconnect at 3PM Hanna Juniper Transmission line disconnect at 3.30PM

The Hanna Juniper line hit a tree, and that caused a short circuit to ground and the line disconnected. An acorn fell of the tree, hit Chicken Little on the head, and she went off to tell the rest of her mob that the "sky was falling down" as Adamson Rust puts it in his rather non technical description.

Star South Canton Transmission line disconnect at 3.40PM Canton Central Tidd Disconnect at 3.45 Samnis Star Disconnect at 4PM Galion Ohio Central Muskingum Disconnect at 4.08PM East Lima Fostoria Central Disconnect at 4.10 Kinder Morgan Disconnect at 4.10

At this point, says Marty, power flows from Indiana and over the west-east Michigan transmission loads started to feel the strain, while at around ten past four, the Eastern Interconnection frequency showed a demand loss of between 700 to 950MW. The following sequences happened almost at the same times.

Harding Fox Disconnect 4.10 Lake Erie Twenty generators stop at 4.10PM West-East Michigan Disconnect Midland Cogeneration Venture Disconnect Detroit Transmission system northwest of Detroit goes down Perry Ashtabula Erie West 4.10PM, transmission line cuts northern Ohio off from Pennsylvania

The investigators at this point reckon that when the lines along the south of Lake Eyrie fell, the power reversed direction and started going in a counterclockwise loop heading from Pennsylvania to NY, to Ontario and into Michigan.

Homer City to Watercure Road 4.10PM a disconnect Homer City to Stalle Road 4.10PM a disconnect South Ripley to Dunkirk 4.10PM a disconnect East Towanda to Hillside 4.10PM a disconnect

Exceedingly heavy power flows heading towards the New York and New Jersey lines.

Fostoria Central Galion Down at 4.10PM Perry I Nuclear unit This 1252MW unit disconnected off at 4.10PM Avon Lake 9 Off line Beaver-David Besse Cleveland load auto disconnected

Campbell Unit Three Disconnect Keith Waterman Disconnect 4.10PM Wawa Marathon Disconnect 4.10PM Branchburg Ramapo Down 4.10PM

The Eastern Interconnect was divided into two sections.

Picture copyright 2003 Juan Leger

New York split east west 4.10 Transmission lines disconnect. New England becomes an electricity island for a while New York split continued 4.10PM. Northern New Jersey and south west Connecticut stay attached to New York system east. Ontario, east Michigan stays connected to New York West Ontario, New York separate at Niagara Falls 4.10PM Long Mountain, Plum Tree disconnect 4.11PM Ontario, east Michigan Separation 4.12PM

Eastern New York blacks out with small pockets of power left. Western New York manages to support 50% of load.

The End Game The cascade at 4.13PM is nearly finished. Some small pockets stayed up for a few minutes more

According to the Task Force, there's more data to be collected and research to do before it can say exactly what and why it happened, and what the relationship between all of these events are.

[mjp]

Another timeline from CNN:

(AP) -- A timeline from the joint U.S.-Canadian task force investigating the August 14 blackout. It uses information from the North American Electric Reliability Council (NERC), Independent System Operators, utility companies and regulatory agencies in the United States and Canada.

Generators shut down:

12:05 p.m. EDT -- American Electric Power's Conesville power plant in central Ohio.

1:14 p.m. -- DTE Energy's Greenwood power plant north of Detroit.

1:31 p.m. -- FirstEnergy's Eastlake power plant in northern Ohio.

2:02 p.m. -- Transmission line disconnects in southwestern Ohio due to brush fire under a portion of the line.

Transmission lines disconnect between eastern and northern Ohio:

3:05 p.m. -- FirstEnergy's Harding-Chamberlain line goes out of service, reason unknown.

3:32 p.m. -- FirstEnergy's Hanna-Juniper line disconnects after contact with a tree.

3:41 p.m. -- Star-South Canton line, shared by AEP and FirstEnergy, disconnects.

Remaining transmission lines disconnect from eastern into northern Ohio:

3:45 p.m. -- AEP's Canton Central-Tidd line disconnects; reconnects 58 seconds later.

4:06 p.m. -- FirstEnergy's Sammis-Star line disconnects.

Transmission lines into northwestern Ohio disconnect; generation interrupted in central Michigan:

4:08-4:09 p.m. -- AEP's Galion-Ohio Central-Muskingum and East Lima-Fostoria Central transmission lines disconnect.

4:09 p.m. -- Kinder Morgan's generating unit in Central Michigan.

Transmission lines disconnect across Michigan and northern Ohio; generation interrupted in northern Michigan and northern Ohio; northern Ohio separates from Pennsylvania:

4:10 p.m. -- Twenty generators along Lake Erie.

4:10 p.m. -- West-east Michigan lines.

4:10 p.m. -- Midland Cogeneration Venture.

4:10 p.m. -- Transmission system separates northwest of Detroit.

4:10 p.m. -- Perry-Ashtabula-Erie line.

Four transmission lines disconnect between Pennsylvania and New York:

4:10 p.m. -- Homer City-Watercure Road, Homer City-Stolle Road, South Ripley Dunkirk and East Towanda-Hillside lines disconnect within four seconds of each other.

4:10 p.m. -- Fostoria Central-Galion line disconnects.

4:10 p.m. -- FirstEnergy's Perry nuclear unit 1 on southern shore of Lake Erie and Avon Lake 9 unit near Cleveland.

4:10 p.m. -- Beaver-Davis Besse line disconnects.

Transmission paths disconnect in northern Ontario and New Jersey, isolating the northeast portion of the Eastern Interconnection:

4:10 p.m. -- Campbell unit 3.

4:10 p.m. -- Keith-Waterman line disconnects.

4:10 p.m. -- Ontario system separates when Wawa-Marathon line disconnects along northern shore of Lake Superior.

4:10 p.m. -- Branchburg-Ramapo line disconnects.

4:10 p.m. -- Over nine seconds, New York-New England transmission lines disconnect. New England (except southwestern Connecticut) and Canada's Maritime provinces -- Nova Scotia, New Brunswick and Prince Edward Island -- separate from New York and remain intact.

Ontario separates from New York west of Niagara Falls and west of St. Lawrence. Southwestern Connecticut separates from New York and blacks out:

4:10 p.m. -- Ontario system west of Niagara Falls and St. Lawrence separates from New York.

4:11 p.m. -- Most of Ontario blacks out.

4:11 p.m. -- Long Mountain-Plum Tree disconnects.

4:11 p.m. -- Remaining transmission lines between Ontario and eastern Michigan separate.

4:13 p.m. -- Cascading sequence essentially complete, with blackout stretching from eastern Michigan and southeast Canada to New York state, New Jersey and parts of New England.

[Cicero]

There are local failures all the time. The real question is, why did they cascade and bring down the whole system. Other articles suggest that the system where the failure started neglected to inform neighboring systems, so they could have started to shed loads or prepare themselves for what was coming. I'm an amateur at this, but as I understand it the mathematics of large grid systems like this are exceedingly complicated.

[Iris7]

I know a little about this stuff, but a long way from the "special expertise" level.

The real question as I see it is why the cascade failure? Obvously when that western Ohio line went down causing the reversal of flow with Michigan the line to Michigan should have tripped. Ohio was trying to keep power to the Cleveland area too long. Lots and lots of protection relays never tripped. Maybe hundreds. Very odd. Makes one suspect the grid operators have been setting them higher so they won't trip, and then crossing the fingers. Finally didn't work.

[KarlInOhio]

In general, the grid is set up to
1. protect the generating and transmission equipment
2. provide powers to customers

All of the various shutdowns did the job of protecting the equipment.

I don't know why everyday localized shutdowns caused a nationwide failure, but I fear that the changes the Feds want to put in will end up preventing companies from getting off the grid in emergencies. That would cause damage to the equipment (generators trying to run at the wrong frequency, transmission lines handling too much power) and what required hitting the reset button this time will cause a much longer term failure next time as major equipment has to be repaired or replaced.

One of the things which need to be done is building more localized power generation, so the regional grid is used for temporary generation/use imbalances instead of constant movement of power from one area to another. Another is a quicker restart cycle if the grid shuts down again, so a regional power outage lasts an hour instead of a day.

[Cicero]

It's possible. When we lived in Connecticut about 15 years ago, we used to have a power outage every single time there was a thunderstorm in the area. Someone in the know told me a few years after it happened that CLP had installed faulty lightning arrestors on the lines, that they knew they were faulty, but that they didn't want to pay to install new ones. After about 15 blackouts, they quietly changed the arrestors without ever telling customers what the problem was.

I'm sure competition and badly managed partial deregulation has persuaded a lot of companies to cut corners, and it's common knowledge that the carriers are not keen on developing or upgrading their networks when the authorities are forcing them to carry power for other people over their lines and when any change requires regulatory and environmental approvals.

[Cicero]

One of the things which need to be done is building more localized power generation

Just so. But that violates the "not in my backyard" principle which governs in most neighborhoods today.

[Porterville]

EMP test gone wrong??? Just a thought.

[JennysCool]

Our apologies, Earthlings.

[Publius6961]

Some time stamps are inaccurate because the computers that record the disconnects either got backlogged or the clocks weren't calibrated properly to the national time standard. Heads will no doubt roll because of this – it makes it harder to pin down the times and the events.

I suspected this from the beginning, but since all plants have backup power for their logging computers (otherwise heads should roll all the way to the top) it is an easy matter to note the time difference after the event and apply a correction to the logged times.

Unless complete idiots began screwing around with the clocks after the event, and before the official inquiry began. Those should be jailed for sabotage and criminal incompetence.