Toledo weathers computer meltdown

8/29/2005

It was Tom Rascke's day to troubleshoot. But trouble came to the city of Toledo that fateful day bigger than he could possibly bring down.

Last January, the calls trickled in from departments across the city starting at 9 a.m., each one louder than the last. The city of Toledo's Department of Information and Communications Technology's two phone operators soon were overwhelmed, and the incredulous chorus of calls flooded into the department's work areas.

Mr. Rascke, a data communications specialist for the department, kept thinking: "Help!"

His boss, department manager Valerie Robertson, remembers that day with blunt humor.

"I've been through the evolution," said Ms. Robertson, her expression half smiling, half jaded. "I've been through it."

That evolution of the city's core computer system - the transition from the old system to the new, for which they'd spent millions preparing - peaked in force that day last January.

In force, and too early.

"It opened a lot of people's eyes," Ms. Robertson said. "That day, people here realized how essential information technology can be."

Mr. Rascke had gone home the previous weekend with the city's computer system on his mind, as usual. The 1,000-gigabyte storage unit held everything: all the public folders that any city department might need; all the common files and forms and rigmarole they used on a daily basis to keep the city on track and on time.

But the system had been "flaking" for the last several months. Mr. Rascke had been applying minor repairs, patches, anything to keep it all together. Just two more weeks.

Two more weeks and the new system, for which the city had spent $2.5 million, would be up and functional. "Every now and then, the Band-aid would slip," Ms. Robertson said. "We've had so many near misses, people don't even know."

And that Monday the system started sputtering, getting more and more troublesome.

Until, all at once - nothing. "It finally failed totally," Mr. Rascke said. "Totally."

By 9 a.m., everyone working for the city had lost their e-mails. Fire and police - or any city department, for that matter - couldn't get access to their most basic day-to-day forms.

And it was hard to get the word out. "We couldn't broadcast anything - well - because the e-mail was down," Mr. Rascke said.

But word got in: a wave of consternation from callers citywide.

Mr. Rascke said there was no question something had to be done - quickly. But "it wasn't the type of thing you could just go pick up at Best Buy," he said.

They called in technicians from their vendor, Dell Inc., who responded even though the equipment's warranty had expired. They'd just signed a new city contract worth millions; they could cut a few corners. They replaced a few drives.

And by the end of the day all the system's eggshells were back together again. Workers went home shaken but relieved. Ms. Robertson slept well after the 12-hour workday.

Then the next day came - the day of the City Council meeting.

And it happened all over again. Council's two-week cycle had come to a head, and much of the city's legislation never got forwarded, up against the wall of the system-wide crash. City employees were forced back into the archaic mechanics of walking legislation drafts from office to office, building to building. Many ordinances were trapped on the system. Only a few made it to council.

Perhaps 10 items - compared to a typical 50 to 60 - were discussed by the city that week, and the next.

When speaking of it today, Ms. Robertson frowns, then settles on a simple description of her time fielding the flurry of calls from angry administrators across the city: "It was a bad day."

While the crippled system sputtered, going back up, crashing, up again, down again, Mr. Rascke realized the problem was with a part known as a "back-plane" - the apparatus plugging all the network's workhorse "hard drives" together. The news came after they'd already replaced several of the drives.

And there was more good news. After Dell conducted a worldwide search, they came back with a tiny little problem: The part wasn't made anymore.

"The next couple of days were interesting," Ms. Robertson said.

Interesting because the $2.5 million infrastructure the city just bought for exactly this reason was untested, not scheduled to go online for a few weeks. Still in boxes, the key servers were to be the last thing installed.

Now they would be the first.

Mr. Rascke got two of the new servers up - and it held.

The new server network, nearly four times as fast and able to hold six times the memory, hummed away. By Thursday, all the kinks were out.

Mr. Rascke leaned back, calling it "a blessing," then launching into talk of "everyone pulling together."

The system became fully integrated into the city's desktop computers just last month.

In hindsight, Ms. Scott sees that day as indicative of past attitudes toward the department she has commanded since 2003. "The city hadn't invested anything for some time. It wasn't seen as a priority," she said.

"We have accomplished more in the last year than I've seen us accomplish in the last 22 years," Ms. Robertson said. "Change is really difficult for people. And virtually everything has changed."

Nobody in the city had internal e-mail until around the new millennium, she said, waving her arms in a motion that encourages the question: "Can you believe it?!" Each department in the city went out and bought their own computers, their own programs, with no worry of whether they were compatible.

She has litany of additional transgressions: a lack of "redundancies," or systems that automatically switch on when another fails; patchy anti-virus protection; backups stored in basements. "We didn't have the basics," Ms. Scott said.

Now, walking among the new, 6-foot computer towers on the third floor of One Government Center, Ms. Scott appears comfortable.

She gives credit to Mayor Jack Ford, the man who hired her, for creating her department, taking three sections split between different departments and consolidating them, giving them heavy authority. City Council member Betty Shultz strove equally hard on council, she said.

"You could see the eyes glaze over when we talked about this [on council]," said Ms. Shultz, chairman of council's information technology committee. "I didn't hold [many] meetings about it because of lack of attendance."

Mr. Ford said earlier this month the city's previous technological foot-dragging "borders on disgrace," while Ms. Shultz gives credit to the city's Information System Advisory Council, created in 1996, from which Ms. Scott was recruited, for spearheading changes. Not surprisingly, Ms. Scott touts the $14 million allocated for various project over the last 18 months.

"It's like refuse, or water and sewer," Ms. Scott said. "Until something happens, everyone takes it totally for granted."

Still, with new payroll and water billing systems, with a total cost of $6.5 million, not due to replace the city's decades-old existing systems until next January, Ms. Scott puts in a caveat common to the engineering field.

"This whole thing could happen again," she said.

Contact Tad Vezner at: tvezner@theblade.com or 419-724-6065.