Posted on 09/12/2003 9:01:48 AM PDT by Robert357
NEW YORK -- Since last month's Northeast blackout, utilities have accelerated plans to automate the electric grid, replacing aging monitoring systems with digital switches and other high-tech gear.
But those very improvements are making the electricity supply vulnerable to a different kind of peril: computer viruses and hackers who could black out substations, cities or entire states.
Researchers working for the U.S., Canadian and British governments have already sniffed out "back doors" in the digital relays and control room technology that increasingly direct electricity flow in North America.
With a few focused keystrokes, they say, they could shut the computer gear down -- or change settings in ways that might trigger cascading blackouts.....(snip)....
The "Blaster" worm that flummoxed an estimated half-million computers around the world last month might have exacerbated utilities' problems during the August blackout, bringing down -- or perhaps blocking communications -- on computers used to monitor the grid, said Joe Weiss, a utility control system expert.
"It didn't cause what happened, but it could've exacerbated what happened," said Weiss, with Kema Consulting in Cupertino, Calif., The blackout followed the Aug. 11 Blaster outbreak by just three days.
The Ohio utility that is the chief focus of the blackout investigation, FirstEnergy Corp., is investigating whether the Blaster worm might have caused computer trouble that was described on telephone transcripts as hampering its response to multiple power line failures.
"We haven't detected a worm or a virus, but we're not ruling anything out," said FirstEnergy spokesman Ralph DiNicola.
The binational task force investigating the country's biggest blackout, which also affected parts of Canada, is also looking into the issue, U.S. Energy Department spokesman Joe Davis said.
In January, the "Slammer" Internet worm took down monitoring computers at FirstEnergy's idled Davis-Besse nuclear plant. A subsequent report by the North American Electric Reliability Council said the infection blocked commands that operated other power utilities, although it caused no outages.....snip....
(Excerpt) Read more at seattlepi.nwsource.com ...
Now back to work and meeting impossible deadlines.
The level of the cascading of the power system in the blackout just didn't seem right. I still think it will take a bit more time for things to come out, but the picture is kind of interesting. One firm that we sometimes use as a specialty subconsultant on transmission line design, Commonwealth, has a computer model of the east coast blackout rolling through the transmission grid that they are selling for $200. I'm trying to get a free-be of it so I can better visualize what happened.
Well got to log off now and get back to my clients. Ernest, you are in my prayers. Have fun with this one gang.
I full believe that most of the SCADA systems are not windows based machines and that the SCADA systems were not infected with the Blaster virus. Let's treat that as a given and not debate it. Let's also not debate windows, versus other OS.
However, (and this is why I posted the story) I think that lots of other computers used by the utilities for lots of things that help dispatchers, help operations folks, and are within substations operating things across the blackout out area were windows based and could have been infected with the Blaster virus.
Because I have done load flow analysis and multiple power system component failure analysis in the past, I feel that a weak area in most utility contingency planning has to do with "common mode" failures. It wouldn't take too many computers using a slowed down internet, or if infected with the Blaster virus and rebooting their heart out, to cause an truly unanticiapted common mode failure event that could lead to power grid conditions that were not anticipated.
Note that in the above, I didn't say that the infected computer was the utility SCADA computer. It could have been something connected to the utility's PBX telephone system that just slowed down the ability to talk to other dispatcers. The infected computer could have been somewhere else that just slowed down the connections between electric utility un-infected computers or it could have been something out in the field, like a bunch of smart meters or smart relays. It could even have been a number of PC's that engineers used to run something like a spreadsheet that calculated a value needed by a dispatcher during an abnormal switching event. BPA transmission remediation nomigraphs are usually posted as part of Excel files on the BPA OASIS site. I wouldn't be surprised to learn that they are created on PC's that if infected by a virus could make it difficult to run a fresh set of transmission remediation nomigraphs that some dispatcher might feel he really needs before doing something drastic.
Again, the point is that it will take time for the true story of the east coast blackout to come out, but it is odd that the problems cascaded to so many systems as there are suppose to be protections in place to stop cascading blackouts.
Back to work for me
Hopefully, we will get the whole story of what happened before too long.
That would be me. This has been discussed before, at length. If I weren't familiar with this computer from having used it in the past, I might suspect hackers or viruses as a cause for the computer problems that contributed to (but aren't the root cause of) the blackout. However, I know enough about this computer and the way it is used to know that it is extremely likely that it just plain crashed or locked up. It has a less than stellar track record, and unless it was drastically upgraded in the last 2 years or so, it is still less than acceptable in the reliability department.
My experience with the transmission grid in the Cleveland area tells me that FE, for whatever reason, failed to realize the importance of the 3 major 345,000 volt circuits that feed the Cleveland area from southeast Ohio (where, coincidentally, the bulk of FirstEnergy's generating plants are located). The Cleveland transmission system used to be run separately from Ohio Edison, the two of which make up part of FirstEnergy. Cleveland was run with the understanding by its dispatchers of the contingencies that were involved with the loss of any of those 3 paths to the south, particularly in conjunction with the loss of generation along Lake Erie. Once Cleveland was absorbed into the FirstEnergy envelope, these important contingencies became more minor and more local, at least in the view of the company. They were viewed as but a small piece of the bigger picture, despite their actually being an important link between a large load center and a large generation center.
Keep in mind also that the power generation world has changed considerably over the last few years. There are several factors that make that portion of the grid more vulnerable than it has been in the past. Economic considerations have shifted generation patterns to the cheaper units, most of which reside in southeastern Ohio, at the expense of more essential but expensive generation around the Cleveland load center. That means relying more on the unchanged transmission grid and less on local generation.
Also remember that non-utility generator plants and Independent Power Producers seldom have had to construct adequate transmission facilities to get their power from often remote plants to the load centers several miles away. They have usually tended to rely on existing transmission lines that were not constructed to carry their additional loads, but are still expected to provide the margin of safety necessary to maintain reliable service after contingencies. Finally, note that many end-user entities, such as Cleveland Public Power do not generate power of their own, but instead rely on the existing grid built and owned by other utilities to transport their power from remote locations. Neither of these entities has had to bear the cost of building and maintaining the grid and yet they present an increasing burden on the same infrastructure due to deregulation.
I'll attack deregulation again at a later date - suffice to say that while its principle isn't bad, its application is very poor and serves only to separate risk and reward.
Pardon my long-windedness.
Absolutely, and if I find out that it was, I'll personally go up to Ohio and B-slap the idiot that approved such a connection. It is NOT common practice to connect a control system to the internet, though there can be specific dial-up links for maintenance/analysis (which are hopefully protected by unusually long multi-character passwords and such).
You obviously have better insights into that SCADA computer, which I appreciate your sharing.
I know of a lot of smart metering in substations that is polled by using either dial-up, the internet, or some LAN.
Alot of equipment vendors brag about how they can call up their smart control devices and do a diagnostic with their office computers. It wouldn't take much to contaminate such equipment with a virus if one was sloppy. Remember that the Blaster virus cause computers to reboot until they frooze up.
In the good old days, SCADA RTUs would have been connected to the SCADA main station either via dedicated mircowave, dedicated radio, leased copper phone line, or dedicated utility owned fiber optic. That was the "good old days" now low bid is often considered. (Yes, radio isn't as secure as it use to be considered with the ME Mag Pringles directional antenna.)
Earlier today I read a report based on a Commonwealth Associates, Inc. power flow model of the grid. They started with the summer peak load standard load flow, then did some modifications to take off line certain power plants, then changed some Canadian utilities from exporters to importers of power. Then they walked throught the NERC sequence of events, transmission line outage by outage to see if that would cause more transmission lines to exceed emergency ratings.
They couldn't model the exact sequence of events. After a number of lines had tripped out, they started to be able to come close to actual conditions for a while. Although there is an interesting discussion in the report at one place that the lines exceeded their normal maximum ratings, but that dispatchers at this point should have switched over to using the transmission line's emergency rating but some lines that triped off according to their models were over normal max rating but not emergency ratings. The also found that a lot of 138 kV lines seemed very heavily loaded at the beginning of the grid collapse. Toward the end when things started to really happen fast, they couldn't get their power flow model to converge to a solution so they think it will take dynamic modeling to really track things.
I don't really have anything intelligent to add, other than National Instruments [Nasdaq NATI] has a line of Web-Enabled SCADA systems which could be used for contolling remote substations and such.
It is fairly obvious that rates need to be raised for transmission lines, regardless of who owns them. The people in New York and other nimby areas must be forced to pay for their habit of demanding cheap power without building power plants.
You can see already that the New Yorkers are trying to make Ohioans pay for upgrading the transmission lines that the New Yorkers depend upon.
You obviously have better insights into that SCADA computer, which I appreciate your sharing.
I have the perspective of having used about 5 different SCADA systems, including the old Sigma-5 built in the 1970's. But, my experience is primarily that of an end user, and while I have some knowledge of the other side of the business, I am not a computer expert.
I know of a lot of smart metering in substations that is polled by using either dial-up, the internet, or some LAN.
This is becoming more prevalent in the industry. I don't have too much problem with data collection using these means, but control needs to remain very secure. I think that utility attitudes were becoming lax during the 1990's, but I also know that we've gotten very serious about security since 9/11 - I suspect that FE and the rest of the industry has as well.
Alot of equipment vendors brag about how they can call up their smart control devices and do a diagnostic with their office computers. It wouldn't take much to contaminate such equipment with a virus if one was sloppy. Remember that the Blaster virus cause computers to reboot until they frooze up.
Had I not known the characteristics of this particular computer and of this particular part of the grid, I would be much more inclined to believe that a virus may have contributed to the disturbance, but I think that there is way too much logical evidence to the contrary.
In the good old days, SCADA RTUs would have been connected to the SCADA main station either via dedicated mircowave, dedicated radio, leased copper phone line, or dedicated utility owned fiber optic. That was the "good old days" now low bid is often considered. (Yes, radio isn't as secure as it use to be considered with the ME Mag Pringles directional antenna.)
I long for the days of physically separate links, but there are a lot of leased lines used these days. I do have to say, however, that most major substations (those with extra-high voltage facilities like 345 kv lines and such) still use secure data links such as utility-owned fiber and microwave. Smaller distribution stations are the predominant users of leased phone lines and even automatic-dialing cell-phone schemes. There are a handful of exceptions, I'm sure.
Earlier today I read a report based on a Commonwealth Associates, Inc. power flow model of the grid. They started with the summer peak load standard load flow, then did some modifications to take off line certain power plants, then changed some Canadian utilities from exporters to importers of power. Then they walked throught the NERC sequence of events, transmission line outage by outage to see if that would cause more transmission lines to exceed emergency ratings.
They couldn't model the exact sequence of events. After a number of lines had tripped out, they started to be able to come close to actual conditions for a while. Although there is an interesting discussion in the report at one place that the lines exceeded their normal maximum ratings, but that dispatchers at this point should have switched over to using the transmission line's emergency rating but some lines that triped off according to their models were over normal max rating but not emergency ratings. The also found that a lot of 138 kV lines seemed very heavily loaded at the beginning of the grid collapse. Toward the end when things started to really happen fast, they couldn't get their power flow model to converge to a solution so they think it will take dynamic modeling to really track things.
I would think that given all the variables involved, it would be very difficult to model this event with a great deal of accuracy. There are simply too many pieces of data that can vary from the model. For example, the standard summer peak probably estimates that those generators on line are capable of their proven "net generation capacity". I know that a few smaller generators in the Cleveland area have been burning inexpensive, low quality (lower BTU output) coal and have had trouble meeting their maximum output due to the poorer burn quality. That's probably not in the standard mode. I'm sure that there were a couple of 138 kv lines out of service for regular maintenance as well, and this wouldn't be unusual. Finally, keep in mind that most of the 138 kv system is simply a parallel path for the much more robust 345 kv system, so that whenever a 345 kv line tripped out, some of the slack fell onto the weaker 138 kv lines.
Please note as well that transmission lines are almost never protected with simple overload tripping, but use more complex impedance relays that allow fairly heavy overloads to pass while tripping very quickly for actual short-circuit faults. What that means is that dispatchers really can't "switch" over to emergency ratings for the purpose of preventing line tripping. It would only serve to allow, on paper, the increased loading of a line with some loss of the safety margin that is often built into the rating curve. Only severe overloads and/or very low voltage (as happened with all 6 138-kv lines feeding from the R. E. Burger plant) would trip the impedance relay, and it is more likely that the wire would sag far enough from heating up to fault to a tree or other ground-mounted object (which will probably turn out to be the case in several of the early trippings).
Many of the EHV lines tripped well before their thermal rating was met. For example, FE's Hanna - Juniper 345 kv line tripped at around 85% load when it sagged enough to flash over to a tree. This was witnessed, ironically, but a tree-clearing crew about 2 towers away. Trees were probably a factor in quite a few other lines, particularly those that tripped and re-closed back into service several times in the period leading up to the final cascade. I recall a few tree/line trips in my time at FE, though mostly on the lower 138 kv system.
I appreciate your commentary on the computer side of things - these are concerns and should be addressed where applicable.
Thanks for the excellent insights guys.
You're certainly welcome, and thanks for the ping.
I don't really have anything intelligent to add, other than National Instruments [Nasdaq NATI] has a line of Web-Enabled SCADA systems which could be used for contolling remote substations and such.
Given the recent blackout events and the concerns for internet security of late, I wouldn't invest in NATI unless they have a vast line of non-internet products to sell.
It is fairly obvious that rates need to be raised for transmission lines, regardless of who owns them. The people in New York and other nimby areas must be forced to pay for their habit of demanding cheap power without building power plants.
There definately isn't any money in upgrading the grid, and in fact the system is set up to reward a weaker grid, given the laws of supply and demand. That needs to change. We as an industry simply cannot risk reliability so that a few marketers can gain huge rewards. There are rules for grid operation as well, but they have no teeth - there is really no penalty for breaking NERC policy. Ironically, I had a little "discussion" on NERC policy today at work.
You can see already that the New Yorkers are trying to make Ohioans pay for upgrading the transmission lines that the New Yorkers depend upon.
Unfortunately, there will likely be a political solution to a practical problem and it won't be pretty, but it will probably make a hand full of "connected" people gain advantage at the expense of someone else. That is often the case when pols get involved in things they don't understand (or things that will gain them votes).
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.