AT&T Wireless Self-Destructs [Offshore IT Outsourcing Disaster Kills Company]

CIO ^ | 4/15/2004 | Christopher Koch

[Southack]

AT&T Wireless Self-Destructs The story of a botched CRM upgrade that cost the telco thousands of new customers and an estimated $100 million in lost revenue. Hard lessons learned. BY CHRISTOPHER KOCH
Executive Summary Last fall, AT&T Wireless frantically tried to complete a CRM upgrade to Siebel 7. It had to be done in time to handle the customer service challenges accompanying a Federal Communications Commission deadline for allowing customers to change carriers without changing their phone numbers. The effort was a failure. Systems crashed and stayed down. Customer reps could not keep up, and angry customers abandoned the carrier. The weakened company was ultimately bought out by rival Cingular. Through exclusive interviews with company insiders, we piece together what went wrong and highlight the key project management lessons, including the pitfalls of poor planning and the deleterious effects of an ill-timed outsourcing push.    Learn More... CRM at Work: Banks Fight Customer Flight Following mergers, banks used to sacrifice customer service in favor of speedy integration and cost-cutting. But no more. Here's how CIOs are helping banks implement a new postmerger mandate to focus on customer service. Getting To Know Them Tackling CRM is a huge challenge for companies across all industries. Here's how four of this year's winners are able to gather and mine customer data for real value. Measuring the ROI of CRM Services (From Computerworld) From the Store Customer Relationship Management: Maximizing Rewards, Minimizing Risk This CIO FOCUS guide brings clarity to the capabilities of CRM, and showcases through several case studies companies that have found affordable ways to make it work, along with others who have made costly mistakes and learned valuable lessons. More on Project Management Too bad AT&T Wireless couldn't get it right. Even the $3 trillion worldwide construction industry is adopting IT to manage its immensely intricate projects. And a small publisher in Salt Lake City could have taught the erstwhile wireless pro about project prioritization. We covered both recently: Cures for Complexity and First Things First, respectively. Resources by Topic Customer Relationship Management Research Center IT Value Research Center In This Issue of CIO: Security Hand Over Security Project Management AT&T Wireless Self-Destructs Management Strategies In Sync with His CEO CIO-VC Partnerships Something Ventured, Something Gained Book Excerpt | The Future Of Competition How to Put Your Customers to Work The Exchange The Top 10 Drivers for Innovation Total Leadership How to Build a Great Team Making I.T. Work Mad as Hell Reality Bytes All Work and No Play Essential Technology Use It and Lose It		JACK LEE MARKED Nov. 24, 2003, on his calendar because that was the day he would finally be able to change his cell phone carrier without losing his phone number-thanks to a Federal Communications Commission ruling. But Lee, president of Tangara Technologies, a company that develops software for forms, decided to wait a day before switching to AT&T Wireless to let the chaos of "number porting" die down a little. Little did he know that the chaos was just beginning. Advertisers Lee ordered his new phone on Nov. 25. When he went to the AT&T Wireless website to check the status of his order a day later, he was greeted with a message: "We could not find a porting request for this number in the system. Please contact Customer Care." It was the beginning of a two-month odyssey in which Lee estimates he made 15 to 20 calls to AT&T Wireless, sent nearly as many e-mails and spent 60 hours on the phone dealing with customer service representatives or waiting on hold-with the line often going dead when AT&T Wireless's customer service lines became overloaded. After being routed all over the company, Lee finally discovered what was going on. A major CRM system had crashed during an upgrade, and customer service representatives could not set up or access new accounts. The system breakdowns, which continued through February 2004, swamped other AT&T systems, gridlocked customer service phone banks and sent furious customers scurrying to other providers. The breakdown couldn't have come at a worse time for AT&T Wireless. It deprived the telco of thousands of potential new customers and cost the company an estimated $100 million in lost revenue. But that wasn't all. The failure so damaged AT&T Wireless's reputation that many analysts believe it hastened its sale to Cingular in February for $41 billion, or $15 per share, which was just under half the value of AT&T Wireless's shares when it went public in April 2000. While an AT&T Wireless spokesman says the company would have been sold regardless of the fiasco because "it was the right thing to do," the crash and resulting confusion could not have helped AT&T cut a good deal. "The system problems made AT&T look like a wounded provider, and the sharks smelled blood and circled," says Roger Entner, a wireless and mobile services analyst at The Yankee Group, a research company. AT&T Wireless's mistakes offer valuable lessons for CIOs. For one, it's unwise to freight major system upgrades with external complications. AT&T Wireless's CRM upgrade was hamstrung from almost the very beginning by rumors of outsourcing deals and future layoffs. These rumors generated pervasive morale problems that hurt the productivity of project staff. Second, it should be understood that complex projects require flexible deadlines. AT&T Wireless undertook a difficult upgrade that affected roughly 15 systems just before it was faced with an immovable deadline-the federally mandated Nov. 24 number portability date. Finally, it always pays to have a plan B. Without one, AT&T Wireless was forced to move forward even as it became apparent that its upgrade would not be completed in time. Playing Catch-Up with the Competition When AT&T Wireless began its Siebel CRM system upgrade in 2003, it was a company that had slipped from unquestioned market leader to middle of the pack. Its overall market share had slid from an industry-leading 25 percent at the end of 2001 to 17 percent in 2003, third behind Verizon and Cingular. AT&T Wireless vs. Verizon Wireless: Losing Ground Between 2001 and 2003, AT&T Wireless lost significant ground to Verizon Wireless (now the industry leader). That's what its CRM upgrade was intended to fix. It didn't. Read More

Worse, AT&T Wireless was playing catch-up on its most important technology asset, its phone network. One of the older wireless companies, AT&T Wireless made an early bet on a technology called TDMA that could not handle data transfer over cell phones-the next big thing for business customers. Even before AT&T Wireless was spun out from its parent AT&T in 2001, it had begun a furious buildout of an expensive new network-global system for mobile communications, or GSM-that could not only handle data but had the added advantage of global compatibility with overseas providers. Only one other major carrier, Cingular, was saddled with the challenge of building out a new GSM network while still servicing the old one. The other carriers had all chosen network technologies that could handle data transfers.

GSM was a great opportunity for AT&T Wireless, but it was also a huge CRM challenge. The company had to convince its old customers to move off TDMA, which worked as well as most other carriers' networks for voice calls, and onto GSM, which had poorer voice quality, according to Morgan Stanley. It also had to convince new customers that GSM was the wave of the future, that they would soon be shipping data over their phones instead of their laptops. But customers didn't buy the pitch. By 2003, AT&T Wireless's percentage of customers on GSM was hovering at 15 percent, according to analysts, while Cingular had 35 percent of its customers moved over (thanks, in part, to its acquisition of a cellular provider with an existing GSM network). And things weren't improving. In the third quarter of 2003, less than half of AT&T Wireless's new customers were choosing GSM, while Cingular was signing up 75 percent, according to Morgan Stanley.

Everyone at AT&T Wireless agreed that the company would keep its existing customers and add more new ones if its customer representatives could handle more calls and get customers up and running faster. Customer service representatives needed about 20 minutes, on average, to work through five or six screens that fetched information from about 15 legacy systems, say former employees. Slow access to customer information records was just one of the reasons why AT&T Wireless possessed the second-highest cost per subscriber (behind Sprint PCS) of the top six national carriers in 2003, according to financial services company UBS.


Why AT&T Needed to Upgrade
AT&T Wireless installed Siebel's CRM system in 2001 to be the front end of its customer service process. The back end, however was a complex mishmash of systems, say former employees. Telco billing systems, for example, were stuffed full of different rate plans and arcane metering processes. Systems that tracked calls and set up new phone numbers (provisioning) communicated with hundreds of thousands of different telephone switches around the country and the world. To work for AT&T Wireless, Siebel's version 6 had to be highly customized. Though the software came with integration tools, consultants usually resorted to writing point-to-point scripts to hook the systems together. Policing the overall integration in a scenario like this is difficult at best. Indeed, a former AT&T Wireless employee who worked on the project recalls the test system crashing and remaining down for six weeks during the summer of 2002 when AT&T Wireless began preparing Siebel version 6 to deal with number portability. And when Siebel 6 was finally up and running, it still couldn't handle all the information that customer service representatives needed.

The next release of Siebel, version 7, was much more powerful. Siebel developed an industry-specific version of the new software that had many more features and could capture more telco-specific information. And new Web capabilities meant that AT&T Wireless could potentially reduce the number of customer service screens to one by building a Web portal that put all the different systems and information sources together in one place. "That was the point of the upgrade," says a former employee. "They wanted to get everything on one screen so they could sign up customers faster."

Marc Siegel, a spokesman for AT&T Wireless, agrees:"As we needed to handle more transactions, more customers, we needed a more robust system. So we upgraded. [The upgrade] was going to make it easier for our representatives to get access to information and give them a fuller array of information on the customer."

In the spring of 2003, the company decided to upgrade to version 7.5 for the roughly 3 million customers on the GSM and general packet radio service (the data portion of the new network). TDMA customers would continue to be serviced through AT&T Wireless's legacy Axys CRM system until they could be moved over to GSM and the new Siebel system. The project was called Odyssey.


The Upgrade Begins, Badly
Though the upgrade promised gains in speed and simplicity, getting there was neither speedy nor simple. AT&T Wireless's IT staff had to rip apart old links from the different legacy systems to the client/server-based Siebel 6 system and rewrite them to work over the Web. The back-end systems-integration work was so complex that it wasn't unusual to see teams of 20 or more people assigned to write connections for a single system, says a former employee. Coordination between the teams-the responsibility of the lead integrator on the project, Deloitte and Touche-quickly got out of hand. (Deloitte and Touche did not respond to repeated requests for an interview.)

"Everything was siloed among the different groups, and we all worked independently of each other," says a project team member. Teams would work on a revision to their piece of Odyssey, for example, only to find that when they finished testing, code had changed elsewhere in the system, rendering the testing meaningless. "In other projects I've worked on, the project managers would freeze code while the teams did revisions to their pieces so you could test everything against the same code base," he says. "I didn't hear of that happening on this project."

Meanwhile, rumors of layoffs and offshore outsourcing began swirling around Odyssey. "[The rumors] slowed things down," says a former employee. "When stuff like that happens, people start looking for other work. I know I was looking for other work when I should have been testing."


Meet the New Boss
On April 10, 2003, Mike Benson, a 15-year AT&T veteran, sent a memo to employees announcing that he was leaving after three years as CIO. AT&T Wireless's Siegel says the 48-year-old Benson "decided to retire." Christopher Corrado, the head of the security solutions practice for Wipro, an Indian offshore outsourcing company, was named executive vice president and CIO. Before joining AT&T Wireless, Corrado had been CTO at two divisions of Merrill Lynch, where he presided over the offshore outsourcing of portions of Merrill Lynch's IT. Employees speculated that it was just a matter of time before offshoring began at AT&T Wireless.

CIO Christopher Corrado never directly addressed rumors that AT&T Wireless planned to outsource much of the support and maintenance for its CRM system.

"Corrado was the hatchet man-everyone knew it," says a former employee. "We'd see our people going into these long meetings with people from Indian companies. We'd see whiteboards that had questions like, 'What opportunity do we have to offshore/outsource?'"

Former employees say morale wasn't helped by Corrado's first presentation to the IT group, in which they say he proclaimed, "Come in every day and expect to be fired." Intended to inspire the troops to greater effort, the talk backfired, says another former employee. "We all came away saying, 'Who is this arrogant jerk?'" Corrado could not be reached for comment.


The Deadline Looms
As November approached, AT&T Wireless was one of the last industry holdouts opposing the Federal Communications Commission's new rule on number portability, along with Cingular and Alltel. The industry had succeeded in delaying the change since 1998 on legal technicalities, and AT&T Wireless was counting on another postponement, say industry sources. But on Halloween, the U.S. Court of Appeals for the District of Columbia Circuit rejected the appeal to move back the FCC's Nov. 24 deadline. An AT&T Wireless spokesman told Bloomberg News that the company was ready for porting.

It wasn't.

Things were going so badly with the Siebel CRM upgrade that there was talk on the project of trying to roll back to the old Siebel 6 system before the porting date. "Two weeks before launch there was rollback talk," a former employee says. But project managers hadn't preserved enough of the old system to make it feasible. There was no plan B. Project members estimated that the team needed two or three more months of bug fixing and testing to get the system running reliably.

And the Siebel system wasn't the only thing AT&T Wireless needed to get working before the deadline. Wireless number porting required new systems that had to be integrated with the different carriers' CRM systems. Five of the six top providers outsourced the administration of the number changes to TSI Communications and used software built by Telcordia. But in June 2002 (before the other carriers had announced they were going with TSI), AT&T Wireless chose its longtime software and administration provider, NeuStar. Ultimately, this would create serious interoperability problems for AT&T Wireless.

AT&T Wireless defends its decision by saying that NeuStar was the most experienced provider in the market, having worked on landline portability in the early '80s. It also says NeuStar's software offering was more complete than TSI's at the time AT&T made its decision.


No Treat, Just Trick
On Halloween evening, the project team tried to bring up the new Siebel system for the first time. It crashed. "It went down and stayed down," says a former employee. "They couldn't get it working again." Project staffers were brought in to work on the system for three straight days to try to rescue it, with no luck, say former employees. The system was unstable.

Though observers could not trace specific causes for the crash, a few factors played a role. As the go-live date neared, former employees say that Deloitte and Touche project managers relaxed testing requirements for various pieces of the system. Rather than freeze the code for the system once problems began, teams continued to add new pieces to the project in an attempt to get it all working. But the new pieces simply added more errors to an already bug-ridden system, which complicated the process of finding root problems and fixing them.

"[Integration was] far more technical and far more difficult to test in advance and also fix problems once they appeared," acknowledged Michael Keith, president of AT&T Wireless Mobility Operations, in a conference call on the company's fourth quarter results.

While employees struggled to fix glitches in the CRM upgrade, CEO John Zeglis confirmed the layoff rumors, saying AT&T Wireless would soon lay off 1,900 workers.

Compounding these difficulties was the hit employee morale had taken on Nov. 16, when AT&T Wireless President and CEO John Zeglis confirmed the layoff rumors. Zeglis told analysts in a third quarter conference call that the company would lay off 1,900 workers. He did not say where the cuts would come or when. But IT managers started telling employees that layoffs would begin in February 2004, according to former employees.

Some AT&T Wireless IT employees had gotten hold of a report prepared by Tata Consulting Services of India outlining plans for offshoring pieces of AT&T Wireless's IT infrastructure. The report, obtained by CIO, also lays out plans for offshoring a portion of the support and maintenance for the new Siebel system. "You had to wonder why you were working so hard if the whole thing was going to go away anyway," says a former project employee.

On Nov. 19, The Wall Street Journal ran a story on planned layoffs and outsourcing at AT&T Wireless, and CIO Corrado responded in a memo to the staff: "We are in the awkward position of having reports in the press about a contract with HP before we have communicated with the employees who will be affected. We have just today signed a letter of intent with HP to deliver the services provided by the following teams: Desktop Services; Retail Field Services; Business Office WAN/Telecom; MSI Packaging; Service Desk-password reset and desktop calls." But as for bigger questions, namely that support and maintenance for the Siebel system would be outsourced to offshore provider Wipro and Tata, Corrado did not address the rumors directly. Instead, he concluded: "We currently outsource work throughout the company, including work within both Customer Services and IT. We will continue evaluating the best mix of internal and external resources as we work to achieve best-in-class margins."

In fact, AT&T Wireless was planning to move overseas more than 3,000 positions in its computer operations and customer service.


The Second Crash
The Odyssey team was still struggling to get the Siebel system up and keep it up when, on Nov. 24, customers began trying to port their numbers between carriers. That's when the software for handling number portability crashed.

TSI and NeuStar had arrived at "differing interpretations" of the industry's data standard for exchanging and verifying porting requests, known as Wireless Intercarrier Communications Interface Specifications (WICIS). The Yankee Group's Entner says that TSI modified its version of the WICIS standard without telling NeuStar. "They made changes in fields and reporting that didn't impact all their five customers but no longer made it possible to communicate with [NeuStar]," says Entner. TSI's Michael O'Brien, vice president of marketing, denies that changes were made without telling NeuStar but acknowledges that the two vendors had different interpretations of WICIS. (NeuStar declined to be interviewed for this story.) AT&T Wireless says that TSI crashes prior to the deadline prevented NeuStar from being able to test the process adequately, and on Nov. 24, the electronic flow of porting requests between NeuStar and TSI broke down right away.

The industry had estimated that automated ports would require 2.5 hours to complete, so the porting messages were programmed with time and date limits. If the other carrier's system didn't respond within the allotted time, the port that failed was supposed to drop into an electronic bin to be handled manually.

Five Lessons from AT&T Wireless's Project Failure

It did a lot wrong. Here's what you can do right.

Read More

NeuStar's software could not respond to TSI's systems before time ran out. And the error messages weren't all getting through either. When AT&T Wireless's Porting Administration Group tried to track and resolve errors using a workflow management tool, it crashed or froze. That meant AT&T Wireless customer service representatives were in the dark when their phones lit up. Combined with their continued problems accessing Siebel, there was little they could do to help their customers. In that first week of porting, more than half of the complaints filed with the FCC singled out AT&T Wireless, according to The Associated Press. The FCC sent a letter to AT&T Wireless on Dec. 4, asking for an explanation, triggering a torrent of bad publicity.

"There were 50,000 [AT&T] customers per week that didn't have service because of this," says Phil Cusick, a telecom industry analyst with Bear Stearns. And many of them, like independent consultant Ian Drake, didn't stick around for a fix. After three weeks of trying to track his newly purchased AT&T Wireless phone, Drake gave up and signed on with Verizon.


A Company Fire sale
As word of AT&T Wireless's difficulties spread, customers began deserting at a rapid clip. The problems continued through December, according to AT&T Wireless President and CEO Zeglis. "It took most of December to expand the systems capacity so that the care reps could have as much access as they needed in order to handle customer calls at the same time the salespeople were processing new orders," he said in a conference call with analysts on Jan. 22. "Frankly, our customer service went pretty far south."

For independent cell phone retailers, which account for about 60 percent of new sales in the industry, there was no choice. "The indirect sellers pushed other brands when they heard that AT&T was having problems," says Greg Teets, telecom analyst for A.G. Edwards. AT&T Wireless was forced to bump up commissions to independents and lower prices to try to stem the tide. It didn't work. After adding 229,000 customers in the third quarter of 2003, fifth out of the top six providers, AT&T Wireless added just 128,000 in the fourth quarter, last among the top providers and less than one-tenth the number of industry leader Verizon, which added 1.5 million new subscribers.

By January, AT&T Wireless was in the advanced stages of plans to move overseas more than 3,000 positions in its computer operations and customer service, according to The Wall Street Journal. (It backed off from the plan only after agreeing to be purchased.) Some employees became part of AT&T Wireless's "buddy program," in which consultants from Indian outsourcing companies Tata Consultancy Services and Wipro were assigned to AT&T Wireless employees to learn their jobs.

In February, 220 AT&T Wireless IT employees were told that they would be released from the company in March. According to one of the terminated employees, another round of 250 layoffs is planned for June and the same number for September. "It's tough," an employee said in February. The Indian workers "basically follow people around all day and pepper them with questions." Other former employees say some staffers resisted helping the consultants. "People would make project decisions when the Indians weren't around," a former employee says.

On Feb. 17, Cingular announced that it was buying AT&T Wireless.

That same day, Deborah Sawyer's voice-mail box began filling up with messages. Sawyer, a partner for executive search firm Morgan Howard Worldwide, says she received inquiries from 12 different AT&T Wireless IT executives looking to move on.

But for their company, the odyssey was over.

Executive Editor Christopher Koch can be reached at ckoch@cio.com.

[Southack]

This is *the* prime example of IT outsourcing (some offshore, some by domestic body shops) gone horribly, horribly bad.

[Southack]

Look carefully. Most of the "usual suspects" are caught mismanaging a large IT project so badly that AT&TW sold to Cingular for *HALF* of its IPO share price.

[Southack]

Hey, didn't Corrado land on his feet after his disasterous AT&TWireless stint??

[Cicero]

If true, it was a $40 billion mistake. So much for saving money by layoffs and outsourcing.

[Benrand]

Corrado Soprano? He's ok.

[Benrand]

"Come in every day and expect to be fired."

Man, what an IDIOT. he COULDN"T have said something as stupid as this.

That is amazing.

[WayneM]

Killed by offshoring...

This is music to my ears. How many time are we, as consumers, told that price is only part of the equation.

As an engineer IT professional, I must admit that the sight of an established American company brought down by offshoring is extremely sweet. The inevitable lawsuits should be another push in the right direction.

This is all made even sweeter by the fact that I don't own any AT&T stock.

[Citizen of the Savage Nation]

I'm curious, what happened?

[oceanview]

ut the point is - the people who make the decisions don't suffer any consequences. the executives all walk off with big bonuses. so where is the dis-incentive to conduct these same programs in the future? there is none.

[oceanview]

but the point is - the people who make the decisions don't suffer any consequences. the executives all walk off with big bonuses. so where is the dis-incentive to conduct these same programs in the future? there is none.

[lelio]

Killed by offshoring...

It looks more like "killed by outsourcing and no controls" -- the offshoring threat (not sure if it ever came about) was at the end of this debacle.

I remember when Corrado was hired on and everyone at ATTWS was fearing for their jobs as he was a former exec of Wipro. Looking over his bio I can't really see anything that shines as an example of leadership, it looks like ATTWS thought that they could get a former exec of an offshoring firm and magically everything would work.

To top it all off, ATTWS execs gave themselves bonus shares just before the sale to Cingular, making themselves millions while the rabble only had their underwater shares. Company was run by a bunch of crooks. Its no wonder they tried the outsourcing route and had it fail -- they wouldn't know how to manage their way out of a paper bag.

[gitmo]

I don't think you can say this was killed by offshoring. It looks like they'd pretty much committed suicide without the offshoring. I've watched projects going to pot like this.

I had an IT exec at one of the 5 largest companies in the world tell me we should implement a system of similar magnitude to this one without a backout plan. He told that stupid story about Cortez burning his ships when he landed in Mexico. I told there was no way my company would do something like that. We implemented our system (ignoring almost all of his recommendations) with barely a burp.

[Blue Jays]

Good Evening All-

AT&T Wireless could not even process my change-of-address request. For six consecutive months, I would send in my (late) payments with the new address box marked. They were unwilling and/or unable to make this change....even after endless Customer Service Center calls.

Fast forward to a few months ago when my family received a letter that essentially said, "Our performance and customer service have both been dreadful. We're working to improve, so please stay with us."

Yeah, right. The only way they may have prevented me from migrating to Verizon was if they sent me a $500 check and two new phones. Even then I would have paid out-of-pocket for reliable and robust wireless communications...which AT&T Wireless isn't.

There have been several recent articles in the Wall Street Journal that have blasted this provider. How they can go from a front-runner to having the greatest number of FCC complaints is beyond comprehension. Not only that, but their proprietary technology as described in the article above makes their handsets worthless if you wish to switch carriers for whatever reason.

Stay away from this nightmare company. There, I feel better now. [/rant]

~ Blue Jays ~

[lelio]

He told that stupid story about Cortez burning his ships when he landed in Mexico.

The ATTWS version of that involves a fictional Buzz Aldrin.

Have there been any large SAP / Siebel rollouts that a) don't have cost overruns in the millions and b) are several quarters late?

[gitmo]

What I don't understand is how do you get into this shape? It ain't that hard to do it right.

[old-ager]

> the sight of an established American company brought down by offshoring is extremely sweet

Ditto, dude! Hahahahahahahaha!!!!

[Theo]

bump for later reading ...

[FITZ]

I must admit that the sight of an established American company brought down by offshoring is extremely sweet.

If you can call it an "American" company anymore. No loss to us here in this country. Who cares if all the multinationals suffer the same fate? The CEO's can't make money by keeping operations in the USA --- they blame that on the American worker --- but who can they blame when they outshore and still lose money? Time to replace the CEO's it seems.

[cardinal4]

My wife and I are stuck with att for two years. Trust me when I say they are terrible.

[Formoore04]

"My wife and I are stuck with att for two years. Trust me when I say they are terrible."

I see you didn't read the "indentured servitude clause" before you signed the contract.

Tsk tsk. That's how they get you.