Posted on 08/17/2003 7:33:34 AM PDT by Brian S
Ohio Lines Failed Before Blackout By RICHARD PÉREZ-PEÑA
The events that led to Thursday's blackout began when several high-voltage transmission lines near Cleveland failed, investigators said yesterday. The utility that owns the lines said an alarm that should have alerted controllers to the shutdowns also failed.
The line failures — including one triggered by contact with a tree — began at 3:06 p.m., 65 minutes before a wide swath of the United States and Canada lost power, said Michehl R. Gent, president of the North American Electric Reliability Council, the power industry group that is investigating the blackout.
It is not clear whether the problem with the alarm delayed action by the utility, FirstEnergy Corporation, or the consortium that controls the regional grid, the Midwest Independent System Operator.
The council's first detailed timeline for the initial system failures, which it released yesterday, also does not answer what many people in the power industry say is a more pressing question: how a failure in one place, no matter how big, could have spread catastrophically to other regions, overwhelming mechanisms designed to halt such a spread.
But Mr. Gent said he was prepared to say for the first time, "We're confident this started in Ohio."
The first failure hit a 345-kilovolt transmission line near Cleveland, for reasons that remain unknown. Ordinarily, that would have no effect on service because the load would be transferred to other lines nearby.
But adding power to a line makes it heat up, expand and sag. Twenty-six minutes after the first line failed, Mr. Gent said, a second 345-kilovolt line in the same area, probably one helping to carry the load from the first failed line, sagged into a tree, causing it to shut down. Trees near transmission lines are supposed to be cut back to prevent such accidents.
With the remaining grid in northern Ohio becoming more strained, three more 345-kilovolt lines in the area failed, at 3:41, 3:46 and 4:06 p.m. Over the next five minutes, systems throughout the Eastern United States and Canada began to see huge swings in voltage and in the direction of power flow, and more lines went down. Power plants shut themselves off to protect their equipment from harmful fluctuations in the flow of electricity, first in Ohio, then Michigan, then New York, then Ontario. At 4:11, the blackout began.
Of the first five lines to fail, four belong to FirstEnergy. Based in Akron, it is one of the nation's biggest utilities, with millions of customers in Ohio, Pennsylvania and New Jersey. The fourth of the five lines to go down belongs to American Electric Power, another major power company.
FirstEnergy released a statement last night saying that "its computerized system for monitoring and controlling its transmission and generation system was operating, but the alarm screen function was not." A company official confirmed that this meant that an alarm system that was supposed to alert controllers did not do so. It was not clear whether the system, besides flashing messages on control-room computer screens, also included an audible alarm. The official would not provide additional details, and no one from the Midwest energy consortium could be reached for comment.
Even without the alarm, "it's inconceivable to me that the utility and the I.S.O. wouldn't know what was going on, and they probably ought to have been able to do something about it, to stop it from spreading," said Karl E. Stahlkopf, senior vice president of the Hawaiian Electric Company and a former vice president of the Electric Power Research Institute in Palo Alto, Calif.
He said there is a precedent for grid managers failing to stop a cascading series of transmission lines, which then spread from one region to another, notably the 1996 blackout that hit most of the West Coast. But since then, the industry has tightened many of its operating rules, including those requiring systems to keep their neighbors apprised of their problems.
Business leaders, electric power experts and government officials have warned for years that the nation's transmission grid is seriously strained, especially in the Midwest. Some of those warnings have focused on Ohio.
During the same time period, companies that are large consumers of electricity have complained of a rapid rise in the number of times that transmission bottlenecks have led regulators to restrict the flow of power around the region. A report by a group called Industrial Energy Users-Ohio cited a nearly fivefold rise in such incidents between 1999 and 2000, and called for changes in the system.
The transmission lines in the Cleveland area are part of what the power industry calls the Lake Erie Loop, a ring of lines in the United States and Canada that have become a sort of electricity interstate, moving vast amounts of power from one state to another. In assessments of the reliability of the national grid, Mr. Gent told reporters Friday, that loop "has always been a big, big problem" in part because it is so heavily used.
While Mr. Gent put the start of the chain of events at 3:06 p.m. Thursday, there were indications that trouble might have started earlier.
FirstEnergy said in its statement that on Thursday, before the lines failed, a coal-burning power plant it owns in Ohio, Eastlake Unit 5, shut itself off, for reasons that are not clear. And readings taken by a Wisconsin company that monitors the flow of electricity for its commercial customers suggest that the problems in Ohio might have started the day before the blackout. Sensors used by the company, SoftSwitching Technologies, show that its customers in Ohio experienced frequent swings in voltage starting after 3 p.m. on Wednesday and continuing into Thursday, according to a database accessible on the company's Web site.
The readings show power lines dropping below their usual operating voltage for periods ranging from a few seconds to several minutes, while the flow to consumers in other parts of the country remained far more stable. Small voltage swings are not uncommon, but they are usually very mild and brief. These were sharper, more prolonged and much more numerous than usual, and the database shows that some customers lost power completely on Wednesday night.
"Instability in the voltage is something you really worry about," Mr. Stahlkopf said. "In the '96 outages in California, we had very large voltage swings, which caused relays to trip, which caused plants to go offline, which caused blackouts."
The company's database also shows Ohio consumers repeatedly experiencing a much rarer condition: voltage running higher than normal for a minute or more, from mid-June through late July. Several experts said that such "overvoltage" can indicate systemic trouble in the local grid, and that if severe enough, it can damage power systems. But they also cautioned that it is not clear whether any of these patterns were more widespread, or had anything to do with the blackout.
Blackouts are often caused by summer heat, as peak demand strains the system and equipment overheats. Mr. Gent and others have ruled that out in this case, noting that neither temperatures nor demand were terribly high on Thursday.
But a report by the Electric Power Research Institute proposes that warm weather could have played an important role, though an indirect one. When a problem causes voltage to drop, a lamp or a television might dim, and when the voltage rises again, the appliance returns to normal without a notable increase in power consumption.
Air-conditioners, though, work differently, and they account for a huge proportion of summer electricity use. Air-conditioners and some other devices, including many elevators, have electric motors that slow down when voltage drops. The motor's response is to draw much more power, trying to resume its usual speed.
The research institute's report proposes that that can lead to damaging swings in voltage, and turn a moderate voltage drop into a much steeper one that causes equipment to fail.
On September 1, 2000, ATSI became the first electric transmission subsidiary of an investor-owned utility in the United States. FirstEnergy Corp. transferred ownership and control of its electric utility operating companies' transmission system assets to ATSI, a wholly owned subsidiary of FirstEnergy.
See also: Alliance RTO
The Alliance Regional Transmission Organization (AllianceRTO) is a for-profit organization proposed by FirstEnergy, Amercan Electric Power, Consumers Energy, Detroit Edison and Virginia Power, that will own and/or operate the transmission systems of several companies.
And BTW, if the NYT is so concerned about reliability of their power system, why don't they interview the NY Public Service Commission, which has prevented the State of NY from joining a Regional Transmission Organization designed to prevent these sorts of things?
I BLAME ALGORE!
2 p.m. FirstEnergy Corp.'s Eastlake Unit 5, a 680-megawatt coal generation plant in Eastlake, Ohio, trips off. This is just east of Cleveland on lake Eire.
Beginning at 3:06 four 345 KV transmission lines in the Cleveland area fail, the last at 3:46. First Energy appears to be disconnected from American Electric Power to the South in Ohio.
At 4:06 p.m. FirstEnergy's Sammis-Star 345-kilovolt line, also in northeast Ohio, trips, then reconnects.
4:08 p.m. Utilities in Canada and the eastern United States see wild power swings. One wonders whether the trip and reconnect wasn't the real problem. If this occurs with the right timing, it may deliver much more of a shock to the rest of the network than simply disconnecting.
4:10 p.m. The Campbell No. 3 coal-fired power plant near Grand Haven, Mich., trips off. This plant is far to the west on the shores of Lake Michigan west of Grand Rapids. The loss of 800 MW may have been what made the incident unrecoverable, and it will be interesting to know why this plant tripped. Was there a first westward surge of power that had nowhere to go when it reached Lake Michigan?
4:11 p.m. Orion Avon Lake Unit 9, a coal-fired power plant in Avon Lake, Ohio, trips. This plant is in west suburban Cleveland.
4:11 p.m. The Perry Unit 1 nuclear reactor in Perry, Ohio, shuts down automatically after losing power. This is just east of Cleveland.
4:11 p.m. The FitzPatrick nuclear reactor in Oswego, N.Y., shuts down automatically after losing power. Oswego is at the Southeastern corner of Lake Ontario. This is a very long way from Grand Haven MI, but there seems to have been a surge of power eastward across Michigan, Ontario, and New York.
It appears that the nuclear plants are very "nervous", and are now programmed to trip off immediately upon sensing "disturbances" on the grid. Although Eastlake, Grand Haven, and Avon Lake, the first three to trip, were all coal-fired, what followed is a cascade of nuclear plants going off-line whether they really needed to or not. This sensistivity appears to be what led to the widespread blackout.
4:12 p.m. The Bruce Nuclear station in Ontario, Canada; Rochester Gas & Electric's Ginna nuclear plant near Rochester, N.Y.; and Nine Mile Point nuclear reactor near Oswego, N.Y., all shut down automatically after losing power.
4:15 p.m. FirstEnergy's Sammis-Star 345-kilovolt line, in northeast Ohio, trips and reconnects a second time.
4:16 p.m. Oyster Creek nuclear plant in Forked River, N.J., shuts down automatically because of power fluctuations on the grid. This is 10 minutes after the first trip-reset of the Sammis-Star and the blackout is now at its full geographic extent.
It looks to me like:
- the lack of alarm displays at First Energy resulted in the operators letting the problems caused by Eastlake tripping get out of hand.
- the Midwest system operator, which had the backup alarm display, may not have been notified that they were now responsible.
- the trip-reset twice of the Sammis-Star line is a bad thing and may indicate a malfunctioning relay.
- the grid is not simply a bunch of DC batteries and light bulbs hooked together; it is a giant oscillator (albeit a low-frequency one), and it obeys Maxwell's equations whether politicians like it or not.
- the emergency actions by the nuclear plants are guaranteed to turn any medium sized problem into a total collapse of the system.
FirstEnergy is part of the Midwest ISO, a Regional Transmission organization. It didn't help.
Once 3 major sources from the south into the Cleveland area were tripped out of service, there was more than a medium-sized problem. Whether it was the MISO's responsibility or not, steps should have been taken when the second line tripped. I'm having a little trouble with the assertion that the alarming didn't work, but that the EMS was still in operation. Something stinks there. I guess it could happen, but that's quite odd. The alarms are brought through the same paths that bring all other substation and plant data, and are displayed by the same energy management computer.
- the trip-reset twice of the Sammis-Star line is a bad thing and may indicate a malfunctioning relay.
Since most line faults are transient in nature, the normal design is to reclose the breaker. If the fault still exists, the line will retrip immediately and lock open. I'm suspecting line sag here as well, with a momentary flashover tripping the line, then it recloses automatically. Normally, this kind of operation is not a problem.
In this particular case, since the system was already weakened significantly, this operation may have started the oscillations around the "lake Erie loop". The Sammis - Star line, in light of the Star-south Canton line already being out, represents the only direct path from Cleveland to the generation-rich Ohio valley. I'm certain that it was loaded very high, given the shortage of generation along lake Erie, even before the event began.
Good post, by the way. I would wait for more information before making the final assertion that the nukes came off line before the grid split - I have the frequency chart from that day and it indicates a greater loss of load than the amount of generation lost. I suspect that when more data comes in from other sources, we'll see that a ton of circuits opened up just after the power oscillations began.
FERC is trying to create a "super ISO" - the NERTO, but New York State refuses to join.
They're supposed to do that. Have you never experienced the power going off...then a 5 second time delay and it flickers on again...then another 5 second time delay and one more flicker before it goes off completely?
This is how the automatic reclosers work. They try to restore service three times before giving up.
I have issues with the way the grid is run nowdays - the responsibility, risks, and rewards are all misaligned. Marketers are financially rewarded for overselling the transmission grid, ISO's are supposed to be responsible, but the buck stops at the system dispatcher/operator who assumes all the risk when things go awry. Notice how the the ISO isn't even mentioned in most articles.
I used to work for FirstEnergy as a system dispatcher/operator. I left 2 1/2 years ago. I saw a problem coming down the pipe, though I didn't think it would affect half the stinkin' grid. Spent a little too much time micromanaging electricians and substation operators, and too little time watching the grid.
I used to work for PG&E out west, and they had a vertically-integrated system which worked really well. They were in charge of generation, transmission, and distribution, and it worked really well until the politicians got involved.
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.