Posted on 03/19/2025 6:55:05 PM PDT by george76
Data Center Alley, a 30-square-mile stretch outside Washington D.C. and home to more than 200 data centers, consumes roughly the same electricity as Boston. So power company officials were alarmed when a big chunk of those centers – 60 of them – suddenly dropped off the grid one day last summer and switched to on-site generators.
The mass reaction was triggered by a standard safety mechanism across the data center industry, intended to protect computer chips and electronic equipment from damage caused by voltage fluctuations. But it caused a huge surge in excess electricity, according to federal regulators and utility executives.
The magnitude of the imbalance forced grid operator PJM and local utility Dominion Energy (D.N) to scale back output from power plants to protect grid infrastructure and avoid a worst-case scenario of cascading power outages across the region.
The near-miss – reported here in detail for the first time – forced federal regulators to recognize a new vulnerability of America’s electrical grid: unannounced disconnections by data centers.
“As these data centers get bigger and consume more energy, the grid is not designed to withstand the loss of 1,500-megawatt data centers,” John Moura, Director of Reliability Assessment and System Analysis for NERC, told Reuters in an interview. “At some level it becomes too large to withstand unless more grid resources are added.”
Historically, grid operators have planned for large power plants tripping offline. But the rapid expansion of data centers processing the vast amounts of information used for AI and crypto mining is forcing grid operators to plan for new contingencies and complicating the already difficult task of balancing the country’s supply and demand of electricity.
“What it tells us is that the behavior of data centers has the potential to cause cascading power outages for an entire region,” said Alison Silverstein, a former senior adviser to the chairman of the U.S. Federal Energy Regulatory Commission.
The event last July 10 occurred near the D.C. suburb of Fairfax, Virginia, an area known as Data Center Alley for its concentration of facilities serving Microsoft, Google and Amazon. About 70% of the world’s internet traffic flows through the area.
A month after the incident, the North American Electric Reliability Corporation (NERC), the federal regulator for grid reliability, founded a taskforce to study en masse disconnections by data centers and crypto miners.
For this story, Reuters examined thousands of pages of regulatory documents and interviewed about a dozen industry executives to determine the origins of the fault – a failed surge protector on Dominion’s Ox-Possum 230-kilovolt line near Fairfax, Virginia – and its spread across the area.
NERC reviewed the incident in a report in January but did not disclose the exact location of the fault, the number of data centers involved, or how PJM and Dominion worked to rebalance the grid’s supply and demand of electricity.
...
NEAR-MISS EVENTS INCREASING
The number of near-miss events like the one in Data Center Alley has grown rapidly over the last five years as more data centers come online.
The amount of power used by data centers has tripled over the past decade and could triple again by 2028, according to a report produced by the Lawrence Berkeley National Laboratory for the Department of Energy in December.
A Reuters review of disclosure filings by the Electric Reliability Council of Texas (ERCOT), the state’s main grid operator, identified more than 30 near-miss incidents since 2020, triggered by big energy users like data centers and crypto miners switching offline.
In December 2022, a failed transformer at a substation in west Texas caused nearly 400 crypto miners, data centers and oil and gas production facilities to unplug without warning.
The mass exodus produced an oversupply of nearly 1,700 megawatts of electricity – equivalent to about about 5% of the grid’s total demand – and forced 112 megawatts of power generation to shut down, according to ERCOT.
The risk of power outages will only grow as new data centers come online, the NERC forecast in a December report. Nearly all of the United States will face higher risks of energy shortfalls over the next 5 to 10 years, the report said.
The regulator urged utilities to consider updating federal reliability standards for data centers and crypto miners.
A CONTROVERSIAL FIX
Many data centers are engineered by their operators to switch to local generators at the smallest hint of a problem on the grid to minimize the risk of an interruption to services like Google search or crypto mining, according to NERC.
Some grid operators have proposed requiring data centers to “ride through” routine voltage dips without disconnecting. But data center operators are opposed because of the risk of damaging electronic equipment and cooling systems.
ERCOT last year withdrew a proposal that would have imposed ride-through restrictions on data centers and crypto miners after facing pushback from an industry group, the Data Center Coalition.
The group, whose members include Amazon, Google, and Meta, cited costs and the risk of damaging computer chips and cooling systems exposed to fluctuating voltage levels.
“Data center hardware and power supplies, similar to other electronics, are very sensitive to power supply stability,” the coalition said in January 2024 comments filed with ERCOT.
“Deviating from this range will deteriorate the optimal performance, reduce longevity, or damage the components beyond repair.”
The coalition declined to comment for this story. Amazon, Google and Meta did not return messages seeking comment. ERCOT did not return messages seeking comment.
There is “high potential” for the magnitude of these disconnection events to grow as larger operations plug into the Texas grid, ERCOT operations engineer Patrick Gravois said in a December presentation to NERC’s Large Load Task Force.
Gravois said the grid operator is still working to determine exactly what prompts big users of electricity to unplug from the grid, so that it can avoid surprises.
Ari Peskoe, director of the Electricity Law Initiative at Harvard Law School, said regulators could require data centers to ride through voltage dips – but that could risk Big Tech decamping for states with more relaxed rules.
Jim Simonelli, chief technology officer for Schneider Electric’s secure power division, said utilities and the data center industry have a lot of lessons to be learned from what happened outside Washington DC this past July.
“One thing that doesn’t exist yet for the data center industry is how to be grid-friendly,” Simonelli said.
Mr. Fusions for all.
Come’on Elon, you can do it.....
X AI has a new center and they have gas turbines in place. They make their own power and also ran a line from the nearby TVA combined cycle plant.
Backup/redundancy is a good plan.
An easy fix is the data centers use solar power and wind power and batteries from electric vehicles owned by the employees.... No need for them to tapinto the grid.
Such facilities should be mandated to produce their own power on site and sell surplus to the grid (i.e., illegal to use grid power).
The alternative is reckless & negligent on multiple levels.
*TECH PING*
At 3.5hp each. That was a mighty big instantaneous power drain on PG&E's system, at least at the time, and resulted in surges back and forth. PG&E's complaints got no response from the Navy so they complained to then-Senator Dianne Feinstein and she got those experiments closed down.
So the Navy changed the experiments to the Lawrence Livermore Labs area in the bare dusty hills to the southwest of the intersections of Highways 580 (east-west) and Highway 5 (north-south). There was an old experimental area already there where, I believe, testing was done of the high-explosive lenses for the fissionable cores of plutonium-fueled nuclear weapons.
Fast-forward thirty years to 2010 or so. The population in the barren area of around the Livermore Labs testing facility increased some, and so did road traffic, particularly rush hour vehicle density on both the southbound lanes of Highway 5 and the various two-lane country roads to the west of Highway 5 that commuters to the Bay Area started to use.
And their cars started stalling out for no good reason. Eventually somebody figured out that the Lawrence Livermore electronic warfare tests might be responsible.
And high school kids in that area began blaming the Navy for their being late to school. I heard that from my wife, who taught at a continuation high school in Patterson.
The problem went away at some point.
These data centers should pay double the normal rate for electricity. Just wait till people experience the hell upon mankind that they are about to unleash.
Utilities and worthwhile regulatory agencies can eliminate this problem. These data centers need to pay for transmission and generation upgrades needed for their power consumption. It’s not hard. Just don’t get greedy, make sure it makes sense and do it.
Agreed
Lol. EV’s, data centers.....meet green climate change folks. Invest in popcorn.
BETTER:
Put a large battery bank — essentially a giant capacitor — between The Grid and the Datacenter.
If The Grid goes down, the Datacenter remains powered from the battery bank.
If The Grid stays down too long, and the battery bank is becoming depleted, the Datacenter can spin up onsite generation and switch to that.
Setting up Datacenters with this power architecture would make what happened EXTREMELY unlikely. Modern, intelligent switchgear is easily able to pull this off.
And if some errant signal did trip the onsite generation while The Grid was up, the excess would be safely routed into the battery bank.
Well... I skimmed over that, and yeah, that is a sudden and huge loss, up there in the unacceptable scale.
Yeah they left the grid and it probably cost them 5 times as much when they did. Its not that they wanted to leave the grid. They did it for self preservation to protect against power surges and such. The were very happy to get back on it.
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.
