EMC Posts Loss, to Cut About 1,350 Jobs

The money could have been better spent establishing a fund for those they laid off that are in desperate situations.

I'm not criticizing a fund for the firefighters, but many charities have already donated to the firemen. With the economy the way it is, EMC should be helping those they let go instead of using this donation as publicity for the company. There are many ex-EMC employees having a hard time supporting their families.

Companies are also using more and more "off-shore workers". This is different than visa workers. The jobs are farmed out to foreign countries, most notably India. The contractors will come in for a few months, learn the job, then work on the job back in India.

Sorry to hear of your bad experience with EMC. However my comment was about treatment of employees, not customers.

I work in a technical support area and can make no informed statements or comments about the treatment of customers by sales or management. I do not doubt the truth of your statements. I was only commenting on the culture within the company as it relates to employees.

...IT workers that could have filled the poisons that were given to foreigners.

Freudian slip?

You may be interested in another EMC customer - GunBroker.com.
http://www.gunbroker.com/user/EmcCrash/EmcCrashFollowup.asp

This story was on the Yahoo Message Board a few months ago.
It is still on Gunbroker.com web site in the link above.

EMC Storage Array Crash - The Real Story

March 20, 2002

Two months ago we reported the EMC storage array crash that caused us nearly 40 hours of downtime. Many people wrote us and asked for additional information about the crash, but at the time we had no additional information to provide. After a month of demanding a detailed explanation we finally got some additional information which is provided herein.

It has also been brought to our attention that EMC Corporation disseminated some false and misleading information to customers, potential customers, and the press. We received a copy of this information and received independent confirmation from multiple sources that the information came directly from EMC Corporation. Because EMC Corporation has attempted to portray us as liars and has attempted to blame their failure on us we feel obligated to reveal the facts of the matter to defend our good name.

The Players

GB Holdings / WebVentures / GunBroker.com - www.gunbroker.com - that's us EMC Corporation - www.emc.com - the manufacturer of the Clariion storage array on which our database was being hosted.

ManagedStorage Inc. (MSI) - www.managedstorage.com - ManagedStorage is an EMC partner and is the actual owner of the storage array our data was hosted on. According to the EMC/ManagedStorage partnership press release "As a pure-play storage service provider achieving platinum status in EMC's xSPerience Provider Program, ManagedStorage will work with EMC to bring the benefits of ManagedStorage's unique expertise in storage and networking, and EMC's information infrastructure to Internet-based businesses".

Inflow - www.inflow.com - the data center that our systems are hosted in. Inflow also acts as a reseller for third-party services, and resells ManagedStorage services under the Inflow brand name StorageFlow

The Equipment

The storage array is an EMC Clariion. We were never told the model. Our systems were connected to the Clariion via dual redundant Fibre Channel cards hooked up to redundant Fibre Channel Switches.

What Caused the Crash and Downtime?

According to the information we have been given, the problem was caused by a bug in the EMC firmware that caused the EMC storage processor to hang when an MSI technician terminated a telnet session with the EMC array. The EMC storage processor hang was so severe that the system was unable to fail over to the backup storage processor.

Inflow's exact words: "On 18-Jan-2002 at 12:37 PM, an engineer from our SSP was performing remote status checks on host names and IP addresses on our disk platforms. Remote status verification was accomplished by accessing the Clariion disk platform through an IP connection utilizing a telnet session. For this particular status check, the host names and IP addresses were verified at the initial login and password challenge screen. Therefore, the SSP engineer did not proceed to actually login, but executed a "control"-"right bracket" command to terminate the telnet session. The termination of the telnet session in this manner caused the active storage processor to hang, preventing failover to the redundant processor."

EMC Corporation, which provides training, support, and on-site technical assistance to ManagedStorage, dispatched technicians to the Inflow data center to attempt to fix the problem. It was not until nearly eleven hours later that EMC technicians had the array back online.

Because we feared data corruption, we requested that EMC/MSI create a bit by bit backup of our data before we restarted our servers. Several more hours passed before this task was completed.

It took a few hours to restore the old database from tape. The majority of the remaining downtime was caused by the need to meticulously recover as much data as possible from the corrupted database.

What Caused the Data Corruption?

Inflow, EMC, and ManagedStorage apparently do not want to address this issue because we have been unable to get a solid written response. Our opinion is that EMC's technicians caused the data corruption. During the time that the array was offline we kept asking why it was taking so long to bring the system back up. The response we were given was that there was a 'dirty cache' on the storage processor and that the EMC technicians were having difficulty flushing the cache to physical storage. We believe that the EMC technicians were never able to flush the cache of the hung storage processor and instead chose to reset the equipment, losing the data in the cache and corrupting our database in the process.

Of course, if EMC's firmware did not contain a serious bug that compromised the system then this whole situation would never have occurred. It is simply ridiculous that a simple telnet session could crash the storage processor.

Inflow told us that EMC has since created an update to their 'flare' code (EMC's term for its firmware) that fixes this bug.

Supporting Documentation

Inflow Trouble Ticket revision 1 Inflow Trouble Ticket revision 2 EMC's Response to Gunbroker.com

EMC's False and Misleading Statements

"Gunbroker.com [sic] outsources most of its IT operations to a service provider, an EMC customer" - No, we don't outsource "most" of our IT operations. At the time we outsourced storage management to ManagedStorage, a company that is a platinum member of EMC's xSPerience Provider Program and that EMC lists as a "Tier 5 Participant" in their EMC Proven E-Infostructure program. MSI is not simply some random EMC customer; MSI is a company that EMC has partnered with specifically to generate demand for EMC products by providing services to the small business market segment occupied by companies like GB Holdings.

"While EMC Customer Service has been involved in diagnosing the problems experienced by the service provider, EMC storage systems have not been found to be at fault for the outage" - EMC is 100% at fault for the outage and the data corruption. The bug that caused the storage processor to hang was a bug in EMC firmware. Furthermore, EMC technicians were unable to flush the cache and instead reset the equipment, causing loss and corruption of our data.

"Gunbroker.com [sic] was responsible for conducting its own backups and the service provider involved has suggested this was the source of Gunbroker.com’s [sic] corrupted database and lengthy downtime" - No, EMC, your buggy firmware and the actions of your technicians were the source of our corrupted database and lengthy downtime. See above.

What the Vendors did After the Crash

We would love to be able to report that EMC, ManagedStorage, and Inflow took responsibility for their problems and worked with us to fix the issues and to cover our financial loss that occurred as a direct result of the problem. Unfortunately this did not happen.

ManagedStorage - because of the possibility that the logical structure of the volume on the EMC was corrupted we backed our data up off the EMC volume and migrated to a new database server running Windows 2000 (the old server had been on NT 4.0). ManagedStorage was never able to get Windows 2000 to see the EMC volume. After a great many hours of following MSI's advise trying to resolve this issue we gave up and canceled the MSI / StorageFlow contract.

EMC Corporation - EMC contacted us directly and asked us to take down the original information we had written on Jan 20 that explained the outage to our customers. Their PR guy suggested that we explain that we had "hardware problems" instead of directly naming EMC Corporation. According to Inflow, EMC also pressured Inflow to try to get us to take the information down.

Inflow - we had high hopes that Inflow would step up to the plate and reimburse our loss. After initially being told that Inflow would take "full responsibility" for the matter, Inflow decided to back out and now refuses to reimburse us for the loss. Inflow's marketing states "Our Internet data centers and managed services provide 100% availability, performance optimization and comprehensive system management. We help customers ensure that their critical applications are always up, always open for business". They also advertise a "100% customer satisfaction guarantee". Inflow makes a lot of promises but when the chips are down Inflow does not deliver. As a result of their failure to deliver according to the terms of their SLA and their subsequent decision to hang us out to dry, we have elected to move out of their data center at the end of the current contract.

If you are looking to build a reliable infrastructure that depends on quality vendors who stand behind the products they sell we strongly recommend that you avoid EMC, Inflow, and ManagedStorage.

Preventing Future Problems

We understand that in the technology business problems are going to occur. What separates the wheat from the chaff is how you deal with the problems. The vendors named in this document serve as illustration of how some companies handle problems poorly. We are reviewing our procedures to make sure that we handle problems quickly and minimize the impact of any issues that arise.

Prior to this matter we relied too much on vendors who made grand promises but failed to deliver. Subsequent to the disaster we have rearchitected our infrastructure to take greater control in-house, reevaluated our choice of vendors for key services, and substantially overhauled our disaster recovery plan. Current and future changes will reduce our dependence on third-party vendors, minimize the possibility for loss of data, and allow us to more quickly recover in the event of a catastrophic failure. This has been a painful and costly lesson, but one that we learned well.

As always we are working hard on behalf of our customers to make sure that GunBroker.com provides the highest level of reliability.

EMC and Customers -- published in The May Report
http://www.themayreport.com/

05/09/2002
Scoop

A Culture of Arrogance, disregard for customer wishes....

A Culture of Arrogance, disregard for customer wishes, and for good accounting practices continues as forged signatures are found at EMC's Professional Services Mid-Atlantic division, by Ron May

So, you thought EMC eliminated its problems when they fired Manish Shah?

Well, think again. It turns out that the problems with the way EMC conducts business and, in particular, the way it bills customers and handles revenue recognition issues, are pervasive and ongoing --- pervasive in the sense that they don't just relate to Chicago. In fact, there are serious problems in the way the Mid-Atlantic region operates and they continue to this very day. Beyond the billing issue is a broadly based cultural problem within the company. Quite simply, the old guard, which includes many higher and mid-level managers, does not get it.

The irony is that in the grand scheme of things the people who are pulling the tricks are doing so when it comes to relatively small amounts of money, perhaps thinking that this way they won't be caught.

Here are the specifics:
In the Mid-Atlantic region (which includes New York, Virginia, and Pennsylvania, that area), now run by people from the Southern
region because the two areas have merged, there have recently been several cases of fraudulent billing in the Professional Services division, tied to issues of revenue recognition. There is a little thing called the PCF (Project Completion Form). Billing on projects is supposed to be done only at the point at which the work is completed. A project may be slated for X dollars but it will be broken up into its component parts based on when the work is done. The customer is required to sign off on the PCF before the bill can be sent to the corporate office at EMC and the accounts receivable department. The customer's signature acknowledges that a certain milestone in the project has been completed.

Recently, in the Mid-Atlantic region several PCFs were forged. In one instance, the person at EMC who signed for the customer was so inept that he misspelled the customer's name. According to my sources, the forging of the customer signatures on the PCFs occurred in at least two instances. The amount of money involved was in the range of $40,000 in each case.

This gets a bit technical but I think it's worth going over. Some projects are bundled, meaning that the customer is making lump sum payments or paying for hardware and services together. From an accounting practices point of view, some of that revenue, even if it is already sitting in EMC's bank, is considered to be deferred revenue. A PCF must be signed by the customer before it can go from being deferred revenue to real revenue.

Just to jump ahead a minute, the phony billing was discovered and the bill was pulled back by the Mid-Atlantic region from the corporate office before a bill was sent out to the customer. The East Coast PS management claimed that a mistake had been made.

So all's well that ends well, you say. Not true. The specific numbers may seem tiny and in the end the problem was corrected but the culture that created it marches on as Sherman marched through Georgia. The use of that analogy is not purely coincidental since my sources are telling me that much of the "sleaze factor" at EMC can be tied to the Southern region. While I don't have the names of the culprits at this time, I do know that one of the "stand up" people is Maurice Dupas who is, I believe, a division head for the PS East Coast area.

Several important things should be noted about the way this whole thing has been handled. First, EMC was pressuring the customer at least in one instance to sign the PCF and the customer said "no" - - twice. The pressure to sign off early on the PCF was coming from management within Professional Services. This fact alone is astonishing. Second, is there retaliation or the threat of retaliation against whistleblowers? You better believe there is. Anyone who so much as asks questions is singled out and targeted on any number of grounds. I have heard stories of backlash and character assassination. One person discovered the problem and another leaked it within the PS division. Third, the whistleblowers are using the threat of going to the media as a defense against being
subjected to retaliatory moves. Whistleblowers are telling EMC management that they better not screw with them or they will go to the Wall Street Journal. Fourth, the people who forged the signatures were not punished. The person who discovered the problem and reported it is being targeted and others who are asking questions are also being subjected to harassment. Management has written this off simply as a mistake and has not acknowledged the severity of the problem. Another source told me late today that one of the top East Coast PS managers has been canned very recently. I don't know if heads have rolled or not. Remember, EMC is getting rid of a lot of people right now.

The general cultural problem at EMC is this: this is a company that sells boxes. Many of the old-timers at EMC are not college educated and have a "good ol' boy" view of the world. In fact, many of them are proud of the fact they never went to college. Most of the people running the company come out of a sales background. There is little respect even at the highest levels of the company for knowing and following good accounting practices such as FASB and GAAP. It is true that there are some MBAs and/or people who come out of a professional services background, but that is not the crowd running the company.

As one person put it to me late this afternoon, the old cowboy philosophy that has governed the firm is that we will "shove it down the
customer's throat." Customers are told, "Here, sign this." The customer just bought a million dollar piece of hardware and he is being treated like an order taker from a salesman who will be fired if he is not making $350K a year. It is not about customer service or satisfaction, it is about winning at any cost. Well, the days of "take no prisoners," the guiding EMC philosophy, are over. There is competition from Compaq, Hitachi and others. That $200 billion in asset value is now $16 billion; the stock of $100 is now $8.

One source told me that there are non-stop meetings going for 48 hours straight at the VP level to determine how the firm is to be reshaped and who will run it. It is "land grab" time. There are a lot of good people who are now gone and the survivors are looking at what remains. It could have real value if the firm gets its act together.

One source told me that this is an "internal controls" issue and that the Professional Services area is too small to pop up on the radar screen for the firm's auditor PriceWaterhouseCoopers. The advice that my sources tell me every EMC customer should heed or ignore at their own peril is READ YOUR BILL - - CAREFULLY! Make sure that you have actually signed off on everything you are being billed for.

It did not escape my attention that The May Report legitimately scooped the story of Manish Shah being fired. And that neither the Wall Street Journal nor the Boston Globe, which did stories on the topic gave credit for the story being broken. I will be watching carefully on this one to see if the major media outlets have reformed their ways. One source told me today that the Shah story was dismissed as being pure quackery written by that nutcase Ron May, but that they had a much harder time using that excuse when the story appeared in the Wall Street Journal. Look, this is not just Ron May tooting his horn. It is important that an avenue such as
this report have the appearance of legitimacy. A story seemingly as minor as this might not be picked up by the Journal or the Globe until some pressure builds. That is why TMR should be seen as a legit pub. Recall that it was The Drudge Report that broke the Lewinsky story.

Think for a moment about what it took in dissatisfaction for people who work far away from Chicago to call me about the internal workings of EMC when the problem was caught and corrected. The point is that only a symptom of the disease has been treated. The core disease, which is the very culture of EMC, continues unabated.

...Therefore, the SSP engineer did not proceed to actually login, but executed a "control"-"right bracket" command to terminate the telnet session. The termination of the telnet session in this manner caused the active storage processor to hang, preventing failover to the redundant processor.

I would be surprised if this is the cause, the ctrl-] is captured locally and all the remote system sees is the dropping of the TCP connection.

Sorry to hear of your bad experience with EMC. However my comment was about treatment of employees, not customers.

Thank you for displaying the typical "who cares about the customer" arrogance that has brought EMC to this point.

Couldn't happen to a nicer company, IMO.

I knew about this story, and about Manish Shah among others.

Oh the stories I could tell you about some of the larger EMC customers in my area, and the b.s. EMC tried to pull!!!

I talk to them all, and have consulted with a few on strategies to shift from EMC to other products and how to best handle the technical apsect of their move.

Come to think of it, I think I've just found a new career: Storage Consulting and helping companies get off of EMC's products! LOL!!! I remember the day they told me I'd fail in moving my information off of their storage arrays onto HDS. They said I'd lose data and cause downtime. Funny, neither happened. We moved all our data in less than 30 days (a full month ahead of or schedule, and more than a month ahead of our lease end date) without so much as 30 seconds of downtime.

After EMC's threats of having to pay a $250k penalty if I was so much as one day late in returning a box to them from lease, I ended up sticking them with the bill for leaving their horse-crap laden storage array's in our building for 2 months after we were off of them and they refused to pick them up! I felt sooooooooooo good sticking them with a storage bill. Even better when they paid it. LMFAO.

I'm EMC's worst nightmare and they know it. Best thing about it is, I know they know it.

Did you go to Boston College by any chance? I'm really just curious here....

"who cares about the customer"

When did I say that? I was clarifying my comment which you took out of context in the first place.

As a matter of fact I do care very much about the customer and your earlier post troubles me deeply.

Wouldn't you be troubled if someone told you the same thing about the company you work so hard for?

Did you go to Boston College

No, I did not.