Blackout Pinned on Three Ohio Failures (claim they found "real" cause)

Atlanta Journal Cosntitution ^ | Aug. 16, 2003 | H. JOSEF HEBERT

[Cold Heat]

However a signal such as for example a change in the frequency will trip the breaker, no?

Yes, at the power source as a result of self protection.

Frequency relays accomplish this task and that is why the grid went down.

If a generator loads too quickly it slows and the frequency slows. The trip has a built in delay and the setting is optional but usually within a very short window,(seconds)

A disturbance in the grid would be like a signal. It would be felt instantaneously and more strongly as you neared the source.

Grid protection is more concerned with current however. Those devices trip on load and not so much frequency. It is the power plant that is damaged by frequency disruptions, however these disruptions create additional load that should trip the grid protection at the substations.

Small disruptions of frequency occur all the time. The grid will tolerate them as long as they only last a few seconds.

I have not got a clue as to what actually happened. i think the fault was indeed in the North. (everything points that way) But, what the cause was that stated the mess is not at all clear to me. Not clear at all.

I am very suspicious of a initial report that a Canadian plant had a malfunction. It will be a while before it becomes clear as to the initial cause and I would be very suspicious of a plant that suffered damage and does not go back on line within a week or so.

Very suspicious.

[Cold Heat]

It is my understanding that the nukes want to get off the grid early to prevent problems with cooling water. They want to switch to backup before line power is lost. They may have contributed big time to the disruption, but how the problem spread so far can only be a result or poor management or something not yet revealed in the news.

[FairOpinion]

According to this timeline, things started to happen at 14:06 EST, and it took some two hours before the large area went dark. So why were they telling us a that it all started at 16:09 ( I think) and it cascaded in 9 SECONDS?!

[meyer]

According to this timeline, things started to happen at 14:06 EST, and it took some two hours before the large area went dark. So why were they telling us a that it all started at 16:09 ( I think) and it cascaded in 9 SECONDS?!

The list is in eastern STANDARD time - we're in Daylight savings time. That is why there's an hour difference.

[meyer]

Also, the 16:09 edt may or may not have come from a written log. The EMS systems that utilities use are generally syncronized as far as time is concerned, and will give precise times of events. I think that's why 16:11 edt is probably the correct time for the actual multiple trippings, at least in Michigan.

This report isn't complete yet - no data from Ontario, PJM, NY, or other affected areas is in yet, but it appears to be adequate to telling a good part of the story.

[FairOpinion]

I was wondering about that. But EDT is one hour ahead of EST, and if it started at 14 EST, that would make 15 EDT, 3 pm, but the blackout occurred after 4 pm EDT, after the markets were closed, and this timeline still talks about 2 hours, which is vastly different from the 9 second timeline we were told initially.

IOW, it seems to me that there was plenty of warning that things were tripping off, so weren't able to do something about it?

[meyer]

Finally, a reasonable analysis from someone who knows something. From the preliminary sequence of events, it does indeed look like the Sammis Star trip/reclosure was involved in the subsequent power swings. And a mitigating factor that may come out in the investigation is how the big nuke plant protections (min/max bus voltage limits due to safety systems, power limits, etc.) can drop these units offline quickly during power/voltage swings. I know the power plant side (nukes), but how the ISO's monitor and initiate isolations is not my area of expertise. Still, If I was to guess at this point, the midwest ISO (MISO) and or First Energy dropped the ball big time - this could have been isolated had the isolations been performed early enough. Once the big nukes started poping off line, that was it though, no way could the remaining generators pick up the load. From the preliminary NERC report (http://www.nerc.com):....

It should be noted that when the Sammis-Star line tripped and reclosed, it was essentially the only south feed left into the Cleveland system, given that the Star - S. Canton line from AEP had already locked open, the Star-Carlisle line goes to the northwest, and the Star-Juniper line represents the 3rd 345 kv line to Cleveland from the south. Frankly, I'm surprised that Perry Nuclear Power Plant didn't trip off line at the same time that the Sammis-Star line operated, given the sensitivity of that plant.

Cleveland was almost an island at this point, but an island with 4000+ megawatts of load and only around 3000-3500 megawatts of generation. It was going down regardless.

There was some strong speculation that lake Erie "loop flows" were a factor in this event, and I beleive that will prove correct. Power, for some reason, tends to flow around the lake. I haven't studied the phenomenon myself, but I do know of its existence and I know that Michigan was considering putting phase-shifting transformers on their tie lines to Ontario to control the flow across the ties. I don't know if those have been put in service or not.

[FairOpinion]

There seems to be an updated version of this story, still with the same title at ABC with more detailthan the original article:

http://abcnews.go.com/wire/US/ap20030816_979.html

Blackout Pinned on Failed Lines in Ohio
Investigator Says Problems With Transmission Lines in Ohio Likely Caused Worst Blackout in U.S.

The Associated Press



WASHINGTON Aug. 16 —
A failure to contain problems with three transmission lines in northern Ohio just south of Cleveland was the likely trigger of the nation's biggest power blackout, a leading investigator said Saturday.

Experts are working to understand why the local line disruptions, some of which occurred an hour before the blackout reached its peak, were not isolated, allowing a cascade of power system shutdowns stretching from Michigan to New York City and into Canada.






"We are fairly certain at this time that the disturbance started in Ohio," Michehl Gent, head of the North American Electric Reliability Council, said in a statement. "We are now trying to determine why the situation was not brought under control after three transmission lines went out of service."

Gent said the transmission system was designed to isolate such problems and suggested that human error might have been involved in not containing the situation.

"The system has been designed and rules have been created to prevent this escalation and cascading. It should have stopped," said Gent in a telephone conference call.

Later, in a statement suggesting human failings for the events last Thursday, Gent said in the future "system operators ... will be extremely vigilant" when local transmission problems arise.

Energy Secretary Spencer Abraham, who is co-chair of a U.S.-Canadian task force that will look into the cause of the blackout, said it's still too early to pinpoint a cause.

"We're not going to prejudge where the problem is," Abraham told reporters in Albany, N.Y., on Saturday where he met with the governors of New York and New Jersey to discuss the blackout. "We're also not going to prematurely leap to conclusions."

Abraham said the task force is putting together investigative teams that will include experts from the government's research laboratories as well as private resources, to find out what caused the power grid breakdown and recommend actions to prevent a repeat.

Gent did not identify specifically the three power line failures that have become the focus of the NERC investigation. But other council officials said they were among five reported transmission failures in the Cleveland area during a period of just over leading up to the blackout peak Thursday afternoon.

According to NERC, the first report came in at 3:06 p.m. EDT on Thursday and involved a 345-volt line that had "tripped" or gone off line. That was followed by reports on other lines failing at 3:32 p.m., 3:41 p.m., 3:46 p.m. and 4:06 p.m.

Two minutes later, according to the NERC summary, "power swings (were) noted in Canada and the U.S." and three minutes after that power disruptions hit across eight states.

The transmission system in northern Ohio is operated by FirstEnergy Corp., based in Akron, Ohio. The company has declined to comment on the investigation. "Those reviews have not even come close to being completed and we're not going to speculate," FirstEnergy spokeswoman Kristen Baird said Saturday before Gent's announcement.

"It appears the train left the tracks in Ohio but we don't know who's responsible," said Alan Schriber, chairman of the Public Utilities Commission of Ohio.

Gent said he is confident the specific reason for the failures and who is responsible will be learned, but that it could take many weeks.

Among the things yet to be determined is the relationship between lines tripping in Ohio and the unusual power swings that were observed in lines leaving Michigan and going into Canada and then back again, according to investigators.

There are more than 10,000 pages of data, including automatically generated logs on power flows over transmission lines, that need to be closely examined, said Gent. Complicating the matter, he said, is that at the time of the power breakdown "events were coming in so fast and furious that (some reports) ... weren't even being logged in a timely way."

Nonetheless, Gent said he is convinced that no data was lost and whatever was not recorded will be recovered.

"We will get to the bottom of this," he said.


Contributing to this story was AP Writer Andrew Welsh-Huggins in Columbus, Ohio.


On the Net:

North American Electric Reliability Council: www.nerc.com

[Enlightiator]

The MISO list above does not capture all the nukes that went offline due to the grid problems. From the Event Notification Report for August 15, 2003 at http://www.nrc.gov (dynamic page changes daily except weekends, so I am not linking), here are the reported times for August 14, 2003 that the nukes experienced a variedy of grid related events (most were trips due to loss of offsite power). Take these times with a grain of salt, they are not official times, but what the operator put in the logbook, could be in error a few minutes. Times are EDT (EST +1). I have edited the event data below for brievity:

August 14, 2003 event times:

16:07 EDT: Palisades (MI)
POWER GRID DISTURBANCE OCCURRED RESULTING IN A MOMENTARY DROP IN VOLTAGE. At 1607 EDT a significant electrical power grid disturbance occurred resulting in a momentary drop in voltage at the site, including both 2400VAC safety busses 1C and 1D. The reduced voltage on the 2400VAC safety busses caused both emergency diesel generators to auto start, but not load. The plant is stable at pre-event conditions.

16:10 EDT: Perry (OH)
UNUSUAL EVENT DECLARED DUE TO LOSS OF OFFSITE POWER GREATER THAN 15 MINUTES. Automatic reactor scram due to a loss of offsite power. All rods fully inserted. Supplying power to vital buses via emergency diesel generators. All system operating properly.

16:10 EDT: Davis Besse (OH)
UNUSUAL EVENT DECLARED DUE TO A LOSS OF OFFSITE POWER. All systems operated as expected. Decay Heat pumps are available if needed.
[Note: Davis Besse, did not trip, it was already offline when the Loss of offsite power occurred. ~Enlightiator]

16:10 EDT: Fermi (MI)
PLANT ENTERED AN NOUE DUE TO THE LOSS OF OFF SITE POWER. The reactor scrammed from 100% power due to fluctuations occurring on the main generator and a loss of off site power.

16:11 EDT: Oyster Creek (NJ)
REACTOR SCRAM FROM 100% POWER DUE TO OFF SITE ELECTRICAL GRID INSTABILITY. Reactor scram from 100% due to off site electrical grid instability, but did not lose off site power. Reactor conditions at 1730 are normal reactor water level, reactor pressure 850-1000psig, MSIV's closed and all recirc pumps are off and reactor is in hot shutdown with all rods inserted.

16:11 EDT: Fitzpatrick (NY)
UNUSUAL EVENT DECLARED DUE TO A LOSS OF OFFSITE POWER. Automatic reactor scram due to a loss of offsite power. All rods fully inserted into the core. All emergency core cooling systems and the emergency diesel generators are operating properly.

--- [Note: Ginna has two entries, the 16:11 time appears to be an update. ~Enlightiator]
16:12 EDT: Ginna (NY)
UNUSUAL EVENT DECLARED DUE TO LOSS OF OFFSITE POWER GREATER THAN 15 MINUTES. Automatic reactor trip due to a loss of offsite power. All rods fully inserted into the core. All emergency systems operated as expected.
16:11 EDT: Ginna (NY)
REACTOR TRIP/AUXILIARY FEEDWATER ACTUATION Received grid disturbance that caused reactor trip at 1611. The trip occurred due to over temperature delta set point being reached from load swings experienced on the generator.

16:23 EDT: Indian Point 3 (NY)
UNUSUAL EVENT DECLARED DUE TO LOSS OF OFFSITE POWER GREATER THAN 15 MINUTES. Automatic reactor scram due to a loss of offsite power. All rods fully inserted. Supplying power to vital buses via emergency diesel generators. All systems operating properly.

16:25 EDT: Indian Point 2(NY)
UNUSUAL EVENT DECLARED DUE TO LOSS OF OFFSITE POWER GREATER THAN 15 MINUTES. Automatic reactor scram due to a loss of offsite power. All rods fully inserted. Supplying power to vital buses via emergency diesel generators. All systems operating properly.

16:35 EDT: Nine Mile Point 1 (NY)
UNUSUAL EVENT DECLARED DUE TO LOSS OF OFFSITE POWER GREATER THAN 15 MINUTES. Automatic reactor scram due to a loss of offsite power. All rods fully inserted into the core. All emergency core cooling systems are operating properly and the emergency diesel generators are operating properly.

17:00 EDT: Nine Mile Point 2 (NY)
NOUE WAS DECLARED AT 1700 DUE TO LOSS OF OFFSITE POWER. The plant lost off site power due to grid disturbance which resulted in a reactor trip from 100% power. The NOUE was declared because of loss of off site power was greater than 15 minutes.

[meyer]

IOW, it seems to me that there was plenty of warning that things were tripping off, so weren't able to do something about it?

I think that will be a major point of contention. Realistically, the first line tripping isn't a real big deal - these things happen more often than one would think, though not that frequently on the extra-high voltage lines, such as this 345,000 volt circuit. But the second line tripping after sagging (I'd bet there was a tree under there that needed trimming) tells me that there was a lot of load on that line. It also would be the point where FirstEnergy or MISO should have taken action IMHO. Surely, after the third line tripped, immediate emergency load reduction was probably in order. NERC made note of there being little or inadequate action after the third tripping in their teleconference today, but again stressed that not all data was in. They will be looking over about 10,000 pages of data in the next few days, according to their report.

[meyer]

FirstEnergy spokeswoman Kristen Baird said Saturday before Gent's announcement.

She's been a spokesperson for the company for a while. I bet she'd never expected national coverage. :)

[meyer]

16:10 EDT: Perry (OH) UNUSUAL EVENT DECLARED DUE TO LOSS OF OFFSITE POWER GREATER THAN 15 MINUTES. Automatic reactor scram due to a loss of offsite power. All rods fully inserted. Supplying power to vital buses via emergency diesel generators. All system operating properly.

This seems odd - I think that off-site power would have been lost at 16:10 or 16:11 when the grid went down, not 15 minutes prior. On the other hand, this report may be incomplete. The plant would certainly trip on overfrequency if system load was lost, or underfrequency if Cleveland became an island. Without more info, either scenario is possible.

BTW, thanks for the info - another resource to help sort through the mess.

[Salman]

I am very suspicious of a initial report that a Canadian plant had a malfunction. It will be a while before it becomes clear as to the initial cause and I would be very suspicious of a plant that suffered damage and does not go back on line within a week or so.

Very suspicious.

Hillary was Johnny-on-the-spot blaming W. I think that blaming W for whatever goes wrong has become a conditioned reflex with liberals, rather that a thought out strategery. Many FReepers were not far behind blaming Bill Clinton.

So now what? Blame Canada, of course. They always screw us up, especially when all that cold air starts pouring in every winter. Global warming will fix that.

[supercat]

It is my understanding that the nukes want to get off the grid early to prevent problems with cooling water.

If a nuclear plant is running at e.g. 90% capacity, how much load can vanish instantly without the plant having to execute an emergency shutdown? My impression is that it takes awhile to slow down a nuclear reaction, and until the reaction slows down it will produce a lot of heat that has to go someplace. If too much load vanishes, that energy has nowhere to go except the emergency cooling systems. Is that an accurate assessment?

If so, would it be practical to make a man-made pond near a nuclear plant with lots of resistive elements underwater to serve as an almost-emergency load dump? Even for a 50MW plant the amount of water required would not seem unreasonable.

[meyer]

Experts are working to understand why the local line disruptions, some of which occurred an hour before the blackout reached its peak, were not isolated, allowing a cascade of power system shutdowns stretching from Michigan to New York City and into Canada.

I think there was a little misunderstanding on the part of the media here. The lines that tripped isolated theirselves as they should through protective relaying schemes. If they were referring to automatic load-shedding or some such scheme, it isn't clear in this sentence. I know that Cleveland, like the entire eastern interconnection, has automatic underfrequency load-shedding schemes, but I don't think there was an underfrequency event until the cascade happened and then the frequency was zero. Even then, the charts I've seen here in Tennessee show an overfrequency around 16:10 EDT indicating that more load was lost than generation initially. The frequency spike was big!

[meyer]

Enlightiator - since you're a plant type, I have a graph of the frequency as taken at a coal plant in Western Kentucky on 8/14/03. Its an old mechanical freq. chart, so there may be some mechanical overshoot. But it shows an overfrequency spike of 60.33 HZ at roughly 16:10 EDT, immediately settling back to 60.27, then back to 60.05 over roughly a 5-10 minute period. I've not seen a swing that big before!

[supercat]

Small disruptions of frequency occur all the time. The grid will tolerate them as long as they only last a few seconds.

BTW, I was just reading that induction motors can work as generators whose output is automatically synchronous to a supplied voltage provided they are run within a certain RPM range (slightly over the synchronous rotation rate). I wonder to what extent such asynchronous generators could have advantages over synchronous generators?

[2nd Amendment]

What's the difference between NYC and Disney World? In NYC you have 20.000 people trying to catch one ferry. In Disney World you have 20,000 fairies trying to catch one person.

[Cold Heat]

I was just reading that induction motors can work as generators whose output is automatically synchronous to a supplied voltage provided they are run

Sounds like a description of what is called a magamp.

These are still used today to keep generators in sync and numerous other applications. The technology is some 50 years old or more.

[Cold Heat]

Nukes are not in my area of expertise. Your premise sounds logical.