10 years ago, 55 million people were left without power; Utility experts say repeat is unlikely

On Aug. 14, 2003, Harvey Schmidt, of Schmidt & Sons Pharmacy in Dundee, left, Ace Hardware store employee Ryan Williams, and Troy Briggs of United Bank and Trust, scrounge for phone cords. Three seconds after Toledo went dark at 34 seconds after 4:10 p.m., Detroit and southeast Michigan went dark.

One decade later, the events of August 14, 2003, are still as clear in Mike Bryson’s mind as a flash of lightning.

The then-manager of transmission for PJM Interconnection, the operator that controls the electric power grid for much of the East and the Midwest, including most of Ohio, was on an airplane that had just landed that afternoon at the Philadelphia International Airport.

As he stepped off the plane, he quickly got wind of what was taking place on that hot and humid afternoon: The second largest power blackout in history was unfolding across parts of the Midwest and northeast United States and parts of Canada.

Nearly 55 million people in eight states, including Ohio and Michigan, and the province of Ontario, were suddenly without electricity after dozens of high-voltage lines failed, and more than 100 power plants had automatically shut down in a cascading blackout that investigators later learned was triggered by events near Cleveland in territory controlled by FirstEnergy Corp. of Akron.

RELATED: Remembering the 2003 blackout

“When I got off the plane and turned on my phone I got a message from PJM which said, ‘You should come in and not go home,’ ” Mr. Bryson recalled last week.

At PJM’s control room in Valley Forge, Pa., Mr. Bryson walked into a beehive of activity.

“The full shift was on and they were already, at that point, working on restoration efforts because the actual event had already happened. There was no attempt at detailed analysis because the main focus was on just getting the power back,” he said.

Start-up operations continued around the clock and well into the next day, he said. “I was lucky. Because I was traveling, I had my suitcase with me,” Mr. Bryson said with a laugh.

Nowadays, as executive director of system operations for PJM, Mr. Bryson remains ever-vigilant for the slightest system anomaly that could once again take down large swaths of America’s electrical grid system.

But as the 10-year anniversary of the 2003 blackout approaches, industry experts say the impact from that catastrophic event — which estimates later said cost up to $10 billion, contributed to 11 deaths, and left 540,000 homes in Ohio without power for up to to 24 hours — was the catalyst for numerous changes.

As a result, experts agree, the likelihood of such a widespread blackout ever happening again, while not impossible, has become very slim.

“They still can happen. But we just have the system in place now to make it less likely,” said David Hilt, a former vice president of compliance at the North American Electric Reliability Council, and the man who led the NERC team that investigated the blackout.

Mr. Hilt and other experts agree that one primary change has made the grid system more reliable and large-scale blackouts less likely: The federal Energy Policy Act, passed in 2005, codified standards and practices by which electric utilities operate. The standards, which took effect in 2007, are mandatory and enforceable by punitive actions, usually in the form of heavy fines.

“All the reliability standards that were in place at the time of the blackout, they were all voluntary and implemented in a peer-pressure environment,” Mr. Hilt said.

After the new standards were put in place, utilities became accountable. For example, he said, in 2008 Florida Power & Light received a $25 million fine for causing a blackout in southern Florida that cut power for several hours to millions of consumers.

“One of the things I like to tell people is that complicated systems fail in complicated ways,” Mr. Hilt said. “The hope now is we have smaller system failures — that is, smaller scale than 2003. I can’t say it’s not possible to have [a big] one again because the system is operated as one huge interconnected system.

“During the blackout, things started in Ohio with trees touching wires, and it cascaded into a large-scale event. But I think the likelihood of that happening now is much much less,” he added.

Carl Bridenbaugh, vice president of transmission at FirstEnergy, which is the parent firm of Toledo Edison, agrees that enacting mandatory standards removed the “vagueness” from operating the grid.

“Before there were standards, but they maybe were more like guides. It was like, ‘Here’s how we think everybody should do it,’ ” Mr. Bridenbaugh said. “... Not that everybody has to do everything the exact same way, but there’s got to be some commonality on how you do things.”

A chief cause of the blackout, according to investigators, was FirstEnergy’s lack of awareness of critical events as they unfolded. That issue kept the company from seizing opportunities to curtail the crisis well before it spun out of control.

During the last 10 years FirstEnergy has devoted tens of millions of dollars to new devices and technologies to ensure that it is fully aware at all times of what is happening, not only on its own electrical system, but on neighboring systems as well.

It joined PJM Interconnection in 2009, placing its transmission assets under the control of the Pennsylvania-based grid operator.

And in its latest move to see that it doesn’t become the focus of a massive blackout, it is spending $45 million on a state-of-the-art operations control center that will direct power over nearly all of its 10 subsidiaries.

New technology

Located about seven miles from the utility’s downtown Akron headquarters, the new Akron Control Center (or ACC for short) will be finished by year’s end and operational sometime in the first quarter of 2014.

The 76,000-square-foot building, which has multiple layers of security, will feature an oval-shaped control room surrounded by dozens of 14-foot high video screens transmitting a map of every power line, substation, power plant, and other facility in FirstEnergy’s territory and beyond.

Forty control operators working 12-hour shifts will man a ring of computer consoles, and in the center is a raised island where two or three supervisors will monitor all operations.

It will replace the company’s Wadsworth center, which will become a backup.

“We will have the latest technology in spades. Quite frankly, from an employee perspective, this facility really tells us that the company cares,” said Robert Austin, FirstEnergy’s executive director of transmission operations services and the man supervising construction of the Akron Control Center.

Mr. Austin would not speculate on whether having such a control center 10 years ago would have made a difference in preventing the blackout.

“But what we’re doing today is light years away from what we did 10 years ago,” Mr. Austin said.

The control center “stands up with anything currently in the industry,” Mr. Austin added.

A broad view

Mr. Bridenbaugh said the technology for the new center is just an enhancement of what FirstEnergy has added to its Wadsworth center since the blackout.

“We did a big renovation at Wadsworth. We put in enhanced situational awareness [software] and we have visualization ability that makes it a lot easier for operators to get a broad picture of what’s going on on the system,” he said.

“So say an operator … is doing switching at a particular substation on the system, and they’re very focused on the screen in front of them and what’s going on at that substation. In order to not really lose sight of what’s going on in the broader system you have this big screen up in front of the operators that they can glance up at from their screen and see what’s going on on the broader system,” Mr. Bridenbaugh said.

The large viewing screens not only give data on electrical loads and voltages across the system, they also indicate where a facility is out or where a power line has just tripped out of service, the executive said.

In its final report, a joint U.S.-Canadian task force investigating the blackout stated that FirstEnergy “did not recognize or understand the deteriorating condition of its system.”

Part of that was because of a software bug that kept alarms from reaching control operators’ computers, meaning they could not see the larger picture of how things were unfolding.

Now, he added, even if an alarm isn’t sent, operators can still glance up at a large “dynamic map board” in the control room and understand at a glance what is happening throughout the FirstEnergy system and beyond.

But even if that failed, there is another backup.

Warning ignored

In 2003, FirstEnergy received a call from neighboring American Electric Power as the blackout was unfolding. The call warned about the tripping and reclosure of a 345-kilovolt shared line in northeast Ohio.

FirstEnergy dismissed the call because it did not receive any alarms about the line tripping out.

Mr. Bridenbaugh said that now FirstEnergy’s system closely coordinates with PJM’s system.

What one sees, the other sees, he said.

“Having all the enhanced visualization makes it real easy for our operators when they’re talking with PJM to say, ‘Hey, we’re seeing something developing over here, do you see it?’ ” Mr. Bridenbaugh said.

“Or if there’s any disagreement, then everybody has lot of ways to confirm results and say, ‘Well, you know, what are we missing? Why aren’t we seeing the same thing?’ and then you can drill down to a good decision,” he said.

Mr. Bryson of PJM said one of the great lessons of the blackout and later power failures out west was the need for redundancy. FirstEnergy had no backup system — a second set of eyes — to check what was happening in 2003.

So PJM has built a second control room to serve as a backup, a training system, and a fact-checker against data from the main system.

As computing power has evolved during the last decade, so has the ability of utilities to gather more data points to keep them aware of how the grid is functioning at any given second.

Utilities and grid managers like PJM or the Midwest Independent System Organization, to which FirstEnergy previously belonged, use “state estimators,” which are sophisticated computer programs that take frequent snapshots of the state of the system and provide operators with an estimate of what is occurring at that moment.

Every few minutes the estimators provide an analysis of potential problems in the making, allowing a control operator to plan for or avoid future problems, rather than merely reacting to ones already in progress.

In 2003, FirstEnergy’s state estimator should have been refreshing operators’ control screens every 3 seconds, but the software bug slowed that to every 59 seconds.

New tools

Mark McGranaghan, vice president of power delivery and utilization for the Electric Power Research Institute, said technology, such as measuring devices called phasors, is allowing utilities to collect even more real-time data about grid activity.

“The energy-management systems that were in place that kept track of what was going on in the system were refreshing with 5-second scans of the system,” said Mr. McGranaghan, a University of Toledo graduate.

Today, phasor managers can scan a system 30 times per second, permitting much greater monitoring of a grid system.

“It’s a huge improvement. But a problem we’re having now is we haven’t figured out all the ways to take advantage of that data. We don’t have the applications yet that can use all of that data,” he said.

One way to use it would be to add more automation to grid systems so that operators, who react slower, would have less control, Mr. McGranaghan said.

“Because the data would be coming so fast you don’t want to have to rely completely on the operators. You could have load-shedding schemes that kick on automatically under certain conditions,” he said.

Software and data gathering aren’t the only areas that have improved since 2003 to make blackouts less likely.

FirstEnergy spent $15 million between 2005 and 2010 to add four substations that have large-scale capacitors that can increase or decrease the voltage on the grid as needed.

High voltage current is used to move electricity over very long distances while reducing the amount of lost energy along the way. During the blackout, voltage on FirstEnergy’s system decreased as power stations tripped out of service and customer demand drew heavily on the available power on the grid.

The capacitor substations, located at strategic points, can ramp up power quickly or decrease it as needed to keep the grid’s generating plants and customer demand in balance.

Tree trimming

Lastly, the mandatory standards that took effect in 2007 now call for strict maintenance of trees and other vegetation along electrical power right-of-ways, with fines levied if such activity is neglected.

Two key events in 2003 were high-voltage lines that sagged into trees near Cleveland, causing the lines to be shut down and other lines overloading as a result.

During the last decade, FirstEnergy has spent more than $320 million clearing trees, using cherry-pickers, cranes, and even helicopters outfitted with saws. In Toledo Edison territory this year the utility is spending $5.5 million on tree-trimming.

Mark Durbin, a FirstEnergy spokesman, said that the utility used to be somewhat amenable to customers' pleas not to cut down or severely trim certain trees, but since 2007 it has become absolutely ruthless in carrying out its federal mandate to keep vegetation at bay.

“We have adopted a zero-tolerance policy on reliability. The bottom line is we have a right-of-way, and we intend to maintain it,” Mr. Durbin said.

While some customers don’t like it, the vigilance on tree-trimming is just another step in avoiding a repeat of 2003, Mr. McGrahanghan said.

“It reduces the number of things that can produce faults on the line. It doesn’t mean a lightning storm won’t cause a fault on the line, but it’s all about reducing what is in your control.”

Contact Jon Chavez at: jchavez@theblade.com or 419-724-6128.