Internet, telecom networks put to test in wake of terrorist strikes on U.S.
By Network World Staff
Network World, 09/17/01
Last week's kamikaze-style terrorist attacks on the World Trade Center and Pentagon exposed the weaknesses and strengths of America's communications systems, which played such a critical role in the aftermath of the destruction.
As people reached for phones in horror and shock on Tuesday morning to contact family, friends and business associates, many found landline circuits jammed and cellular service unavailable.
Verizon's networks struggled to handle twice the usual call volume even as the carrier lost key cell towers and voice switches housed near the site of destruction. Reviving a major Verizon facility adjacent to where the World Trade Center towers stood is considered key to getting the U.S. stock markets reopened.
The aftermath: Networks
How you used the Internet to stay in touch with loved ones and colleagues.
Other thoughts on what happened.
Many individuals say they found Internet-based e-mail and the Web to be the more reliable form of communication as events unfolded.
"The cellular and phone systems are horrendous," says Phyllis Lampell, executive director of the Wall Street Technology Association, which is setting up a chat room on its Web site for its 1,200 members to offer each other data center space or other resources in the wake of the attacks. "I've had people trying me since [Tuesday] who can't get through, and I'm in New Jersey."
She adds: "The Internet seems to be fine, although I'm not getting tons of e-mail."
Still, news and airline Web sites, including those for CNN, ABC News and The New York Times, were overwhelmed by traffic volume during the first hours after the attacks. Content providers, though, had some success in increasing bandwidth with the help of hosting providers and content delivery network companies to better handle up to tenfold spikes in visitor traffic.
Though two-fifths of the Pent-agon was damaged by the hijacked airliner slamming into it, the military's command and control communications re-mained intact, as did its classified and nonclassified intranets. The Pentagon's private phone system survived, though circuits to the outside world were often unavailable in the Washington area. The main problem, Pentagon officials say, was cellular.
"The [Defense Department] systems continue to function across the board, but the impact was on people trying to use cellular phones from Verizon, which are distributed to military personnel," says spokeswoman Sue Hansen, who survived the terrorist attack, though up to 200 others may have died at the Pentagon, according to last week's estimates.
Winstar Communications has been able to deploy fixed wireless radio equipment that is equivalent to four T-1 lines at the Pentagon, says Bill Rouhana Jr., CEO of Winstar. "We've been able to kludge a network together for them."
These fixed wireless lines are connecting rescue and emergency personal to Cingular Wireless' network. They are using wireless voice services to stay connected throughout the burned out area where workers are searching for victims.
Rouhana says that Sprint, the Federal Emergency Management Agency and New York City have all requested assistance from Winstar for everything from replacing T-3 lines to establishing emergency services.
At the World Trade Center towers, Morgan Stanley Dean Witter, the New York Port Authority, Lehman Brothers and others among the 1,200 businesses located there had extensive data centers and significant network installations on site. Many if not all of them will be forced to reevaluate their disaster-recovery plans in the weeks and months to come, though a number of them also indicated they were prepared to be open for business the day after the attack.
"The firm is in great shape, all of our systems are working," Morgan Stanley CEO Philip Purcell told employees worldwide last Wednesday from the firm's headquarters on Broadway, encouraging them to relay that message to clients. However, Morgan Stanley, which had 3,500 people in the World Trade Center buildings, was reporting that up to 40 of its employees were missing as of late last week.
Investment firms last week were reluctant to provide details about their alternate systems, though disaster-recovery vendor SunGard Recovery Services disclosed that 15 customers have been affected so severely as to move into SunGard's sites, where they are provided with full telecommunications and computer equipment to duplicate their office needs. Another 70 customers alerted SunGard that they may need to use its services. Separately, disaster-recovery service provider Comdisco says 32 customers declared disaster emergencies.
Keeping the network up
Some businesses near New York's ground zero shared their experiences. SoundView Technology Group, an online investment banking firm in lower Manhattan, which was declared a disaster zone, kept its private data network operational for Internet access and e-mail services for 400 employees in New York, Stamford, Conn., and San Francisco. However, part of the company's voice network was cut off when the World Trade Center towers were hit.
SoundView had dedicated, private voice lines - called ring-down circuits - at all the major brokerage firms in New York, and most of those lines were down.
"With the ring-down circuits, you just pick up a line and you're talking to the broker," explains Bart Anthony Lavore, manager of corporate telecommunications for SoundView. "These are voice lines that can be copper or T-1 circuits that go straight to the brokerage house."
Lavore last week was trying to set up speed-dial lines between SoundView and the temporary locations of brokerage houses affected by the disaster. These speed-dial lines won't be as fast as the ring-down circuits and will cost more because they use the public telephone system, but at least they will provide connectivity, he says.
"We're still trying to figure out when we'll be able to set up the speed-dial lines," Lavore says. "[The carriers] haven't told us when there will be power restored to the three central office switches closest to the financial district."
Bill Moore, telecommunications manager at the Museum of Modern Art in midtown Manhattan, says the museum's network was largely unaffected, though Internet connectivity slowed. Still, he's concerned about how long it will take to rebuild the overall network infrastructure serving the city.
"My gut feeling is that it's going to take quite a long time and a very major effort before we get our communications infrastructure anywhere back to normal," says Moore, who is also an officer with the Communications Managers Association. "It's going to take months, knowing what's down in that area."
Here is how Moore described the communications infrastructure in lower Manhattan: "[It] is a maze of tunnels. Most of the [fiber-optic] cables run below ground from building to building in those tunnels. That area down there is where Manhattan narrows to a point, and everything converges down there. As the World Trade Center buildings collapsed, they collapsed down into those tunnels. And as they start doing the excavation down there, there are likely to be some tractor cuts, too.''
Wall Street's main interconnection point is the heavily damaged Verizon West Street facility, which supported 200,000 voice lines and 3 million private lines prior to the Sept. 11 attack. Last week, top officials at the New York Stock Exchange (NYSE) and Nasdaq said the financial industry, with its extensive data-storage and back-up systems, would be ready to reopen the stock exchanges today - if Verizon's facility can be restored.
"Verizon is working mightily in this building, and there is great danger to workers," said Richard Grasso, chairman of the New York Stock Exchange, which like other exchanges is dependent on interconnected voice and data networks.
At a press conference last week, Grasso acknowledged that the financial industry - even though the U.S. banking and Federal Reserve networks were unaffected by the attacks - needs to come up with a better telecommunications infrastructure plan to respond to emergencies. "As an industry, after we've resolved the challenges of the moment, first and foremost taking care of this human side of the tragedy, we will partner with the Securities and Exchange Commission, the Federal Trade Commission and others and ask: What are the points of vulnerability? How do we improve security from both the physical and systemic standpoint?" he said.
Nasdaq Chairman Hardwick Simmons, who spoke alongside Grasso, said the stock exchanges were in continuous discussion with Verizon executives and long-distance firms. For the financial services industry, "billions of messages per second are the business of this business," said Simmons, adding that there's concern Verizon can't yet handle the traffic volume it did before the attack.
Verizon had no easy answers to questions about why there was not more alternate routing. Despite using highly reliable technology such as SONET, the reality is that even SONET's dual-ring architecture can't help when both the fiber rings and the equipment attached to them are ruined. A Verizon spokesman says the carrier has already begun routing some calls to its Broad Street facility, which Verizon claims handles 80% of the NYSE and Nasdaq telecom needs (the West Street facility handles the rest). Meanwhile, repair work was proceeding under treacherous circumstances on both the outside of the facility, showered by heavy debris, and deep underground where water had seeped into cables running five stories deep.
"This could be a very long process," said Larry Babbio, vice chairman at Verizon, which had almost 500 employees at the World Trade Center, a small number of whom were unaccounted for at press time. "The building is filled with smoke and dust. It's difficult to breathe and to see."
"Thousands of circuits have been moved to get customers up and running," said Ivan Seidenberg, co-CEO and president an Verizon. As of Friday, the company had rerouted 24 OC-48 circuits, which is the equivalent to more than 20 million T-1 lines, according to Seidenberg.
Among service providers, Verizon's network appears to have been hit the hardest, and Verizon Wireless lost 10 cellular transmitter sites. But others also took major hits. Cingular lost six transmitter facilities in New York, while Sprint lost four cell sites in lower Manhattan.
AT&T; lost landline connectivity to its wireless cell sites but reported that its local switching gear located several stories down in the basement of the World Trade Center remained unscathed. Still, AT&T; says it rerouted calls. Qwest Communications and WorldCom say their networks were not affected by the attacks.
Internet trouble spots
While Internet-based communications fared well overall, ISPs weren't immune from network interruptions.
"It doesn't matter which ISP you have. There are only so many cable conduits under the city, and everyone buys lines from everyone else," says Perry Metzger, CEO of start-up Wasabi Systems, which is located on 14th Street, at the edge of the disaster zone. "All of the ISPs in New York have been having trouble.''
Wasabi is a 20-person company that backs up its systems to a site in northern Virginia every four hours. But even with a disaster-recovery plan in place, it took more than 24 hours for the company to get its e-mail system and Web site running on spare systems in Virginia. In the meantime, the company's engineers located around the country couldn't communicate with each other or access corporate network resources.
Popular news sites were also overwhelmed, according to Keynote, a Web monitoring service, which reported that The New York Times, CNN and NBC News all had 0% availability between 9 and 10 a.m. last Tuesday, undoubtedly because of traffic overload.
"All hell broke loose," says Vice President of Technology at Washingtonpost.com Eric Schvimmer.
The number of page views was estimated to be running at 10 times normal, and the visual nature of the terrorism story meant Web site visitors, when they could get access to washingtonpost.com, were downloading high-bandwidth digital photos and audio. Schvimmer says the Web site, which collocates with Exodus Communications but also uses Akamai Technologies' content delivery services, had Akamai provide extra bandwidth under its FreeFlow service.
Exodus, which has four data centers in the New York metropolitan area and five in Washington, D.C., says that while its Web hosting service wasn't interrupted, company personnel was evacuated from the New York centers and a skeleton crew remained in the Washington facilities, which all were being monitored remotely.
Ellen Messmer, Carolyn Duffy Marsan, Denise Pappalardo, Jennifer Mears and Kathleen Ohlson contributed to this story.
More news on the attacks
Cancelled, postponed IT trade shows and conferences
Microsoft letter tests negative for anthrax, 10/19/01
Snail mail squashed by anthrax anxiety, 10/18/01
Former federal agency heads stress data sharing, 10/17/01
Study: Corporate IT purchasing continues to weaken, 10/15/01
Anthrax-tainted mail sent to Microsoft, 10/15/01
Corporate America's wake-up call, 10/15/01
IT execs rethinking disaster recovery, 10/15/01
Congress going wireless, 10/12/01
Will liberty be the next casualty? - Ann Harrison argues we shouldn't let federal officials take away our electronic rights in pursuit of terrorists.
aftermath: Networks - How you used the Internet to find out about loved ones and colleagues.
Attacked - Other
thoughts on the attacks.
spam - Have some people no shame? Discuss the worst post-attack spam you've gotten.
Apply for your free subscription to Network World. Click here.