Posted on 04/26/2009 8:35:06 AM PDT by 2banana
A Cyber-Attack on an American City
Bruce Perens
Just after midnight on Thursday, April 9, unidentified attackers climbed down four manholes serving the Northern California city of Morgan Hill and cut eight fiber cables in what appears to have been an organized attack on the electronic infrastructure of an American city. Its implications, though startling, have gone almost un-reported.
That attack demonstrated a severe fault in American infrastructure: its centralization. The city of Morgan Hill and parts of three counties lost 911 service, cellular mobile telephone communications, land-line telephone, DSL internet and private networks, central station fire and burglar alarms, ATMs, credit card terminals, and monitoring of critical utilities. In addition, resources that should not have failed, like the local hospital's internal computer network, proved to be dependent on external resources, leaving the hospital with a "paper system" for the day.
Commerce was disrupted in a 100-mile swath around the community, from San Jose to Gilroy and Monterey. Cash was king for the day as ATMs and credit card systems were down, and many found they didn't have sufficient cash on hand. Services employees dependent on communication were sent home. The many businesses providing just-in-time operations to agriculture could not communicate.
In technical terms, the area was partitioned from the surrounding internet. What was the attackers goal? Nothing has been revealed. Robbery? With wires cut, silent alarms were useless. Manipulation of the stock market? Companies, brokerages, and investors in the very wealthy community were cut off. Mayhem, murder, terrorism? But nothing like that seems to have happened. Some theorize unhappy communications workers, given the apparent knowledge of the community's infrastructure necessary for this attack. Or did the attackers simply want to teach us a lesson?
Although they are silent on the topic, I hope those responsible for emergency services, be they in business or government, are learning the lessons of Morgan Hill. The first lesson is what stayed up: stand-alone radio systems and not much else. Cell phones failed. Cellular towers can not, in general, connect phone calls on their own, even if both phones are near the same tower. They communicate with a central switching computer to operate, and when that system doesn't respond, they're useless. But police and fire authorities still had internal communications via two-way radio.
Realizing that they'd need more two-way radio, authorities dispatched police to wake up the emergency coordinator of the regional ham radio club, and escort him to the community hospital with his equipment. Area hams dispatched ambulances and doctors, arranged for essential supplies, and relayed emergency communications out of the area to those with working telephones.
That the hospital's local network failed is evidence of over-dependence on centralized services. The development of the internet's communications protocols was sponsored by the U.S. Department of Defense, and they were designed to survive large failures. But it still takes local engineering skill to implement robust networking services. Most companies stop when something works, not considering whether or how it will work in an emergency.
Institutional networks, even those of emergency services providers, are rarely tested for operation while disconnected from the outside world. Many such networks depend on outside services to match host names to network addresses, and thus stop operating the moment they are disconnected from the internet. Even when the internal network stays up, email is often hosted on some outside service, and thus becomes unavailable. Programs that depend on an internet connection for license verification will fail, and this feature is often found in server software. Commercial VoIP telephone systems will stay up for internal use if properly engineered to be independent of outside resources, but consumer VoIP equipment will fail.
This should lead managers of critical services to reconsider their dependence on software-as-a-service rather than local servers. Having your email live at Google means you don't have to manage it, but you can count on it being unavailable if your facility loses its internet connection. The same is true for any web service. And that's not acceptable if you work at a hospital or other emergency services provider, and really shouldn't be accepted at any company that expects to provide services during an infrastructure failure. Email from others in your office should continue to operate.
What to do? Local infrastructure is the key. The services that you depend on, all critical web applications and email, should be based at your site. They need to be able to operate without access to databases elsewhere, and to resynchronize with the rest of your operation when the network comes back up. This takes professional IT engineering to implement, and will cost more to manage, but won't leave you sitting on your hands in an emergency.
Communications will be a problem during any emergency. Two-way radios have, to a great extent, been replaced by cellular "walkie-talkie" services that can not be relied upon to work during an infrastructure failure. Real two-way radios, stand-alone pager systems, and radio repeaters that enable regional communications are still available to the governments and businesses that endure the expense of planning, acquiring, maintaining, and testing them. Corporate disaster planners should look into such facilities. Municipalities, regardless of their size, should not consider abandoning such resources in favor of the less-robust cellular services.
Satellite telephones can be expected to keep operating, although they too depend on a land infrastructure. They are expensive, and they frequently fail in emergency situations simply because their users, administrative officials rather than technical staff, fail to keep them charged and have no back-up power resource once they are discharged.
A big plus for Morgan Hill was that emergency services had an well-practiced partnership with the local hams. Since you can never budget for all of the communications technicians you'll need in an emergency, using these volunteers is a must for any civil authority. They come with their own equipment, they run their own emergency drills and thus are ready to serve, and they are tinkerers able to improvise the communications system needed to meet a particular emergency.
Which brings us to the issue of testing. No disaster system can be expected to work without regular testing, not only of the physical infrastructure provided for an emergency but of the people who are expected to use it, in its disaster mode. But such testing takes much time and work, and tends to trigger any lurking infrastructure problems, creating outages of its own. It's much better to work such things out as a result of testing than to meet them during a real disaster.
We should also consider whether it might be necessary to harden some of the local infrastructure of our communities. The old Bell System used to arrange cables in a ring around a city, so that a cut in any one location could be routed around. It's not clear how much modern telephone companies have continued that practice. It might not have helped in Morgan Hill, as the attackers apparently even disabled an unused cable that could have been used to recover from the broken connections.
Surprisingly, manholes don't usually have locks. They rely on the weight of the cover and general revulsion to keep people out. They are more likely to provide alarms for flooding than intrusion. Utility poles are similarly accessible. Much of our infrastructure isn't protected by anything so tough as a manhole cover. Underground cables are easily accessible in surface posts and "tombstones", boxes often located in residential neighborhoods. These can be wrecked with a screwdriver.
Most buried cable cuts are caused by operating a back-hoe without first using one of the "call before digging" services to mark out the location of all of the buried utilities. What's done accidentally can also be done deliberately, and the same services that help diggers avoid utilities might point them out to an attacker.
The most surprising news from Morgan Hill is that they survived reasonably unscathed. That they did so is a result of emergency planning in place for California's four seasons: fire, floods, earthquakes, and riots. Most communities don't practice disaster plans as intensively.
Will there be another Morgan Hill? Definitely. And the next time it might happen to a denser community that won't be so astonishingly able to sustain the trouble using its two-way radios and hams. The next time, it might be connected with some other event, be it crime or terrorism. Company and government officers take notice: the only way you'll fare well is if you start planning now.
Ask Barney Frank about “man holes”!
L
The answer is three. I'm not kidding!
Sounds like it could be a DNS server problem. Instead of http://www.google.com, try this: http://74.125.95.104; instead of http://www.drudgereport.com, try http://66.28.209.219. If that works, then their DNS server is having problems looking up the proper IP addresses.
a) be as independent as possible from "authorities", be it police, medical services or fire department, or grocery stores. In case of a serious trouble of any kind your pleas for help will be ignored anyway.
b) get yourself a ham radio. There is no single answer to this (and like) threats; having an independent radio that can contact other people is incredibly valuable. I have Yaesu VX-6 among others, it's waterproof, receives every emergency frequency, transmits on three ham bands with good power, has a decent built-in battery and can take external +12V from your car or another source.
That is how I was convinced that the progressive tax system does have some fairness to it...those with the most have the most to lose, and should bear more costs of insurance against loss (i.e., military and LE). However, I also think that taxation needs to be more wealth-based than income-based, as a progressive income tax just keeps down hard-working, productive people, dampening incentive for productivity.
This destroys wealth, both in the targets and the ammo.
What? You don't believe in the Keynsian idea of prosperity through government waste? ;-) (Yes, it destroys wealth, despite creating a temporary sense of prosperity via the creation of a production need; the costs aren't as easily seen, so they are often ignored.)
It also diverts lots of highly competent people from the production of wealth.
One of my major problems with the Iraq war deployments was that many, many, many good productive Americans were diverted from the production of wealth. (My other concern is that I believe that properly raising children is a critical task, and many family separations have led to irreparable damage to young'ns.) Regardless of necessity of war, there are many hidden costs. Regardless of the need for a military, there are many hidden costs. Regardless of the need of LE, there are many hidden costs.
The responsible thing to do is to face and manage those costs, not ignore them. I think you advocate the former, but many Americans want to do the latter.
I'm not in favor of eliminating those necessities to preserve our freedom; I'm in favor of viewing them honestly and directly.
Just checked back on Time Warner’s Channel 10 here in Rome. It looks like the problem has been fixed. Hope yours is up and running too.
Was probably an attack by the union. Were any negotiations going on? It is pretty common afterall.
Ditto.
Military and LEO are essential. But they are expenses, not assets, much less income-generating assets. They’re overhead. Essential overhead, to be sure.
A larger city will have more redundancy in the system and you’ll need to cut more than 8 cables to cut it off. And of course even if you do there needs to be something along with it. We’ve had blackouts that more thoroughly shutdown larger chunks of real estate and things by and large turned out OK.
Not the Union as far as I can see. The report said they had an equipment failure and had to replace something.
You got it. No foreign involvement, terrorist plots or grand conspiracies needed. Just a couple of CWA brothers with T lifts, axes and a knowledge of the network.
http://www.cwa-union.org/news/busloads-of-cwa-members-head-to-dallas-for-at-t-annual-meeting.html
Sorry, disregard previous post. I thought you were referring to the California outage. Time Warner is not now in contract negotiations.
Thanks neighbor, (I'm actually in Marcy) we're up and running
COBOL2Java
Again, thanks for the help. My daughter has been burning oil on this project relating to 1962. Fortunately, a sixties survivor myself, I was able to fill in for Google on about 80% of the material!
Both have a good Sunday afternoon, I'm back to my woodshop
Regards,
Wilum
Morgan Hill.
Last June, the Family Practice Group with several offices, one office is where my wife works as the head RN, went to a new electronic Medical Record System.
There have been the expected problems and a few more.
I kept asking the bosses/Docs and IT people what would happen if there was a power failure.
We were told that there were several backup servers/systems.
I told them that wasn’t the problem, lack of electrical power in the system or users would render their computers useless. They totally ignored that possible reality.
Eventually, there was a power failure that took out 5 of the offices and the servers for several hours.
Then, they found out what I was trying to tell them. All access to and from the patients electronic records were down, the appointment system was down, no appointments could be made or checked to see if and when a patient had an appointment that day. There was no way to check the patients insurance cards to see if they were still valid, up to date and what the copays were. They couldn’t send electronic RXes nor receive RX requests from the pharmacies. Orders to and from the hospital were not working. Lab results that day and in the past could not be accessed
Medically/legally the doctors were in the dark about their patients medical histories and treatment plans.
Finally after the first hour, they closed the offices just like grocery stores and other stores on electronic systems do in a power failure.
Later, a group of the Doctors gathered in the office where my wife worked. On her way out of the dark offices at high noon to come home, she said, “This was what my husband was trying to warn you about, not your backup server resources.”
Besides the inconveniance to the patients, the $ loss to the group for that day was significant and basically gone forever.
Don't ask me...I have no idea what that's about!
I don’t have a ham license. Or do I just buy one and listen to it??? I know nothing about what to buy, what is good, what is not.
What about CB radio, limited but better than nothing.
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.