FirstEnergy Blamed for Blackout in Report

Associated Press Writer ^ | Wed, Nov 19, 2003 | JOSEF HEBERT

[presidio9]

WASHINGTON - The nation's worst blackout began with three power line failures in Ohio and should have been contained by operators at FirstEnergy Corp., a three-month government investigation concluded Wednesday.

The report by a U.S.-Canadian task force said the FirstEnergy operators did not respond properly, allowing the Aug. 14 outage to cascade, eventually cutting off electricity to 50 million people in eight states and Canada.

The task force also cited outdated procedures and shortcomings at a regional grid monitoring center in Indiana that kept officials there from grasping the emerging danger and helping FirstEnergy deal with it.

"This blackout was largely preventable," Energy Secretary Spencer Abraham (news - web sites) said.

The task force said it found "no computer viruses or any sort of illicit cyber activities" to blame. It also concluced that there was no deliberate damage or tampering with equipment associated with the outage.

Among the faults found at FirstEnergy, however, was a simple failure to keep trees around power lines trimmed.

FirstEnergy, the nation's fourth largest investor-owned utility company, had no immediate comment on the report. The company, based in Akron, Ohio, has maintained that its problems were but some of many in the Midwest power grid on the day of the blackout and that it should not be singled out.

The task force report cites the failure of a FirstEnergy line near Cleveland, follow by problems with two of its other lines, as the "initial events" of the blackout.

The loss of the three lines caused too much electicity to flow into nearby lines, causing an overload. Because those lines not prepared for the sudden increase in power, the system became unstable as the balance between available power and demand deteriorated, said the report.

It said the company's failure to adequately trim trees along the lines "was the common cause" for the lines tripping and said overall FirstEnergy "failed to ensure the security of its transmission system."

Abraham and Canadian Natural Resources Minister Herb Dhaliwal released the findings in the 134-page report on the causes for the blackout that spread across eight states, from eastern Michigan to New York City and into Canada.

It was the worst blackout in the nation's history, costing at least $6 billion in economic and other losses. It prompted new calls for upgrading the nation's high-voltage electric transmission systems and giving the government power to enforce reliability standards.

Congress is expected this week to complete a massive energy bill that includes, for the first time, federal reliability rules for companies to follow to safeguard the grid system. Currently the industry regulates itself with no direct penalties for violations.

The report raises questions about the monitoring of the power grid by the Midwest Independent System Operator, or MISO, a group responsible for overseeing power flow across the upper Midwest.

The MISO operators, from a control center in Carmel, Ind., were using outdated information and didn't have the means to identify significant transmission problems developing in the system, said the report. That prevented MISO operators from assisting FirstEnergy control operators, who themselves were hampered by a faulty computer and other mechanical glitches.

The task force cited both human error and equipment failures, noting that FirstEnergy's ability to analyze its problems was hampered for nearly an hour and a half by a computer failure.

From the time the computer failed "to when they began to recognize their situations, (FirstEnergy) operators did not understand how much of their system was being lost" or that their interpretation of events did not reflect the system's true condition, the report said.

Abraham said of the hours before the blackout: "A number of relatively small problems combined to become a very big one."

The report found four violations of industry reliability standards by FirstEnergy and another violation by the Midwestern Independent System Operator. The industry is largely self regulating and such violations in themsevles would carry no fines.

The FirstEnergy violations included not reacting to a power line failure within 30 minutes as required by the North American Electricity Reliability Council, not notifying nearby systems of the problems, failing to analyze what was going on and inadequate operator training.

The report said the MISO, the multi-state grid operator, did not notify other regions about the potential problems.

While the blackout was preventable, Abraham said "once the problem grew to a certain magnitude, nothing could have been done to prevent it from cascading out of control."

The report said the cascade began at 4:06 p.m. EDT when a FirstEnergy high-voltage line called Sammis-Star tripped.

"Within seven minutes the blackout rippled from the Akron area across much of the northeast United States and Canada," the report said. In those seven minutes more than 263 power plants went down.

[_Jim]

ALL fingers point to FE ...

[flamefront]

And to the likely involvement of the
Blaster worm linked to US blackout.

[snopercod]

What a lame report. Is the system so fragile that one line tripping can bring down the whole NEastern US? Obviously it is.

Did Arlen Specter write the "single line" theory?

[meyer]

What a lame report. Is the system so fragile that one line tripping can bring down the whole NEastern US? Obviously it is.

The article is rather poorly written. The Sammis-Star line tripping was the "straw that broke the camel's back" so to speak. 2 major lines into the Cleveland load center had already tripped during the hour prior to this event so the system was already highly taxed. (I know that the second of those first two lines was tree-related, but I've since heard that the first line that tripped was due to vegitation growth as well)

Additionally, the Cleveland area had several hundred MW less generation capacity than in the years just prior to the merger, some of which was eliminated due to emissions requirements and some of which was removed for economic reasons after the merger. There were a couple of generator outages as well, most notably the 600 mw Eastlake unit #5 which tripped earlier that afternoon. Cleveland is a pretty big importer of both power and voltage support due to the relatively large line loading from the south.

BTW, thanks for the ping!

[meyer]

ALL fingers point to FE ...

... As well they should! I'm gloating again!

[tubebender]

Among the faults found at FirstEnergy, however, was a simple failure to keep trees around power lines trimmed.

PG&E had this problem a few years ago when untrimed trees caused a massive forest fire in the Sierra. Only problem was the fact the state PUC had told them to cut cost to reduce rates...

[_Jim]

I just downloaded the joint US-Canada report and skimmed though it - VERY detailed ... here's an excerpt, the summary, from section 4:


4. How and Why the Blackout Began

Summary

This chapter explains the major events?electrical,
computer, and human?that occurred as the
blackout evolved on August 14, 2003, and identifies
the causes of the initiation of the blackout. It
also lists initial findings concerning violations of
NERC reliability standards. It presents facts collected
by the investigation team and does not offer
speculative or unconfirmed information or
hypotheses. Some of the information presented
here, such as the timing of specific electrical
events, updates the Sequence of Events1 released
earlier by the Task Force.

The period covered in this chapter begins at 12:15
Eastern Daylight Time (EDT) on August 14, 2003
when inaccurate input data rendered MISO?s state
estimator (a system monitoring tool) ineffective.
At 13:31 EDT, FE?s Eastlake 5 generation unit tripped
and shut down automatically. Shortly after
14:14 EDT, the alarm and logging system in FE?s
control room failed and was not restored until
after the blackout. After 15:05 EDT, some of FE?s
345-kV transmission lines began tripping out
because the lines were contacting overgrown trees
within the lines? right-of-way areas.

By around 15:46 EDT when FE, MISO and neighboring
utilities had begun to realize that the FE
system was in jeopardy, the only way that the
blackout might have been averted would have
been to drop at least 1,500 to 2,500 MW of load
around Cleveland and Akron, and at this time the
amount of load reduction required was increasing
rapidly. No such effort wasmade, however, and by
15:46 EDT it may already have been too late
regardless of any such effort. After 15:46 EDT, the
loss of some of FE?s key 345-kV lines in northern
Ohio caused its underlying network of 138-kV
lines to begin to fail, leading in turn to the loss of
FE?s Sammis-Star 345-kV line at 16:06 EDT. The
chapter concludes with the loss of FE?s Sammis-
Star line, the event that triggered the uncontrollable
cascade portion of the blackout sequence.

The loss of the Sammis-Star line triggered the cascade
because it shut down the 345-kV path into
northern Ohio from eastern Ohio. Although the
area around Akron, Ohio was already blacked out
due to earlier events, most of northern Ohio
remained interconnected and electricity demand
was high. This meant that the loss of the heavily
overloaded Sammis-Star line instantly created
major and unsustainable burdens on lines in adjacent
areas, and the cascade spread rapidly as lines
and generating units automatically took themselves
out of service to avoid physical damage.

Chapter Organization

This chapter is divided into several phases that
correlate to major changes within the FirstEnergy
system and the surrounding area in the hours
leading up to the cascade:

- Phase 1: A normal afternoon degrades
- Phase 2: FE?s computer failures
- Phase 3: Three FE 345-kV transmission line failures
and many phone calls
- Phase 4: The collapse of the FE 138-kV system
and the loss of the Sammis-Star line

Key events within each phase are summarized in
Figure 4.1, a timeline of major events in the origin
of the blackout in Ohio. The discussion that follows
highlights and explains these significant
events within each phase and explains how the
events were related to one another and to the
cascade.

[ALOHA RONNIE]

HILLARY for President HQ =

Shaker Heights/Cleveland, Ohio Area


HILLARY Booksigning night before Power Blackout =

Shaker Heights/Cleveland, Ohio Area


HILLARY blasts BUSH's Power Energy Policies on TV night before Power Blackout =

Shaker Heights/Cleveland, Ohio Area


Power Blackout fires up =

Shaker Heights/Cleveland, Ohio Area


HILLARY then blames BUSH on TV for Power Blackout that fired up =

Shaker Heights/Cleveland, Ohio Area

[meyer]

Thanks, Jim. I'm about to review the report, since I have a vested interest in it. I spent enough time in that company's control room to easily visualize everything that the report will tell me. As well as gaining some vindication for us ex-Cleveland people, it also serves as a learning tool for those of us that operate in other areas of the country.

[meyer]

Hillary, hillary, hillary....

Makes me think of a certain picture of Hitlery where she was blamed for the power outage (the one where she's holding here electric "toy"). Oddly enough, the substation behind her looks hauntingly like Star substation, located at one end of the Star-Sammis transmission line. I've been there!

[_Jim]

As well as gaining some vindication for us ex-Cleveland people,

I understand ...

Meanwhile, here's a nice timeline extracted from the report that shows the functional areas that were seen to have failed:

It's kind of big; reducing it in size loses some detail so I left it large ...

[meyer]

Jim, I'm reading the report now! It has all kinds of good information in it. The idea that FE didn't perform any kind of contingency analysis after Eastlake-5 generator tripped, nor after the Harding Chamberlain 345 kv line tripped tells me that they really didn't realize the extent of their problems. Of course, the failed computer didn't help.

[ALOHA RONNIE]

Q: How much human supervision was there, or lack thereof, of this Shaker Heights/Cleveland, Ohio Area originating Power Blackout at First Energy..?

Q: How close is this supervision to the HILLARY for President Supporters on the Shaker Heights City Council..?

[meyer]

Just so you know, Aloha Ronnie, the city of Shaker Heights has no major transmission lines nor any transmission facilities within its borders. Its an old-money community, though degraded a bit since the good old days. Its full of limo-liberals, but I really can't make a connection between that community and the power outage. If anything, the people there were the first and loudest to complain about lack of power and water (the Cleveland water system had no backup power of diesel pumps).

The blackout started in an area much larger than Shaker Heights, and included the vast greater Cleveland area as well as a good portion of the Akron area - Shaker heights is but a little piece of that area.

[meyer]

I'm just digging in to the computer failures. If I had a dollar for every time a "node" failed on the EMS network during the 3+ years I spent at FE, I would be a rich man. It was so bad that we joked that failure was the norm and that a working computer was some kind of abberation. :)

[meyer]

OMG, Jim, the entire sequence of events and their description in the first few pages following the summary are very detailed, and so absolutely believable. I picture myself sitting right there, witnessing this. Unfortunately, its easy to see how it happened. At least the beginning portion of it.

[ALOHA RONNIE]

Thank you for your up close info.

Don't the Power Lines that went down, and the First Energy Power Station in question, sit about 20 miles or so from 'HILLARY for President' Stronghold Shaker Heights..?

Isn't the President of the Shaker Heights City Council a leading supporter of HILLARY for President..?

Didn't her husband treat former Gore2000 Campaign Manager TONY COELHO for his lifetime Epilepsy..?

Before and after the Power Blackout didn't HILLARY suddenly publically attack President BUSH for his Power Energy Policies and then blame him for the Blackout itself all over TV..?

Why did HILLARY's Friend, the Liberal Canadian Prime Minister,
immediately announce on TV for us to erroneously digest after the blackout began that it was fired up by a lightning strike on a Niagra Falls Power Station ...when he knew darn well there was no such lighting strike that day..?

Is there any link between the Supervision of First Energy and the Supervision of the City of Shaker Heights..?

All kinds of WHY's that need to be answered as the CLINTONS' Lifetime M.O. of...


"It's the TV, Stupid, no matter WHO pays for it'


...continues.

[meyer]

Well, I'm running out of time - its getting late and I've got to get up early, so I'll be a little concise.

None of the power lines actually "went down" in the sense that no conductors fell to the ground. There are a lot of "figures of speech" that are mis-interpreted or mis-stated, but the reality is that several power lines tripped off due to excessive tree growth under the lines coupled with high line loads. The first line to trip was about 20-30 miles from Shaker heights, but it was also within 30 miles of about 4 million other people.

I am not sure about the city politics of Shaker Heights, but I don't think that there's much of a political connection between FE and the Shaker Heights city council. Shaker is a suburb of Cleveland. When the old Cleveland Electric Illuminating Company and Ohio Edison merged to form FirstEnergy, the Ohio Edison gang from Akron became dominent. If there was any Shaker influence before the merger, it disappeared afterwards.

I can't explain the statements of the Canadian Prime minister except to state that he is a politician and thus knows next to nothing about electric power.

In short, I doubt that Hitlery had anything to do with the blackout, given that there's plenty of ordinary explanations for it. She is a politician as well - she took advantage of the blackout to try to blame Bush (as if he could cause it).

I'm with you - I don't trust the Clintons, but I don't think that they have even a remote connection with the blackout (other than the Clinton-supported attempt at de-regulation which wasn't actually de-regulation in the first place).

[ALOHA RONNIE]

Thank you for your detailed CLARITY.

Please keep an eye out for us on the 'HILLARY for President' doings over at the Shaker Heights for anything else peculiar that stands out to ya in the future, like the WEEK THAT WAS the Massive Power Blackout.

Not having any direct knowledge of what's going on in the planing room's over at the 'HILLARY for President' Headquarters in Shaker Heights doesn't mean it isn't happening.

Please be Aware & Vigilant.