Posted on 04/25/2005 7:25:04 PM PDT by Ramius
Computer local area networks at East Japan Railway Co. and some media organizations were inaccessible Saturday morning, apparently due to a bug in antivirus software made by Trend Micro Inc. "A bug was detected in a file (designed to detect viruses), and it is highly likely that computer networks that were updated (with the file) encountered a system failure," the software company said.
Trend Micro said its virus analysis and support center in Manila released the file worldwide at around 7:30 a.m. Saturday as part of an update for its Virus Buster software.
It said the file was replaced by a bug-free one before noon.
The buggy file slowed down computer performance substantially by making CPUs run at almost full capacity, the software company said.
JR East encountered the trouble with its LAN from around 8 a.m. to noon, but train services and ticketing operations were not affected, company officials said.
View Plaza, a travel agency operated by JR East, said it suffered access failure. It asked customers to buy their tickets at JR station ticket offices.
Computer networks at Osaka's municipal subway system were also affected.
Kyodo News experienced LAN access failure from around 8:20 a.m. to shortly before noon. The Asahi Shimbun and Yomiuri Shimbun also had trouble with their LANs at their Tokyo and Osaka bureaus, but the problems did not affect editing or printing of their evening editions.
The Nihon Keizai Shimbun's sales bureau and the Shinano Mainichi Shimbun in Nagano Prefecture encountered minor trouble.
The problem also affected absentee voting for mayoral and municipal assembly elections in the city of Toyama, where votes had to be counted manually, while the Tottori Prefectural Government and Beppu City Hall in Oita Prefecture were also hit.
No LAN-related trouble was reported at central government entities such as the prime minister's office.
Trend Micro said its call center would be open over the weekend. It posted directions on its Web site to remove the bug-infected file from affected computers.
Trend Micro's call center can be reached at (03) 5334-1441 for individual customers and (03) 5334-3620 for corporate clients.
The Japan Times: April 24, 2005
There was another story here:
http://www.marketwatch.com/news/story.asp?guid=%7BD5848B22-B9F0-44A0-A00C-EEE1C452B2D7%7D&siteid=mktw
Seems Trend stock took a little bath on the news.
Pingish, to my "I'm still cheesed off about this" thread.
Though... to be fair, I really do like Trend's AV products... but this really messed us up.
Security vendor Trend Micro distributed a faulty virus definition file on Friday that slowed thousands of PCs worldwide to a crawl, the company admitted Monday.
The virus definition file was released Friday at about 3:30 p.m. PDT to both the Trend Micro Web site (where users could retrieve it manually) and to the firm's automatic update servers. The file was to update Trend Micro's OfficeScan, PC-cillin, ServerProtect for NT, Client/Server Suite for SMB, and Client/Server/Messaging Suite for SMB.
Rather than simply update the anti-virus files, however, the new definition brought machines to their knees by chewing up virtually every processor cycle.
"We confirmed that a virus pattern file which we distributed on April 23, 2005, from 7:33 a.m. to 9:02 a.m., Tokyo Local Time, significantly slowed the performance of our customers' computers and in some cases made their computers inaccessible," said Trend Micro in a statement from its Tokyo office on Monday. "This trouble was caused by insufficient work in compatibility testing of the product with the operating system before it was released."
Earlier, Trend Micro had pinned the blame on improvements in its software's ability to detect viruses in compressed files.
Approximately 90 minutes after releasing the defective definition file, Trend Micro posted a fixed file that it urged all users to download and install from its site or its update servers. The company also posted suggested tactics for affected users, including instructions on how to roll back the update.
Although it was too late for users plagued with inoperable PCs, Trend Micro's Tokyo's office said in a statement, "We promise that we will further strive for and improve our quality control to avoid future occurrences of similar problems."
The company also said it was trying to put a dollar amount of the gaffe to account for any possible changes to its business forecasts and guidance.
I was away doing real life (personal) stuff and my PCs were turned off at the time.
I never knew we had a problem.
That's precisely why we developed an early deployment group for immediate pattern file delivery. We built in a delay to wait a couple hours before deployment to everyone else. Saved our butts this time. Trend Micro's QA has been horrible lately.
I'm cheezed that they didn't send out an apology of any kind, saying, "our bad, here's a gift cert. for an ice cream cone, we're sorry" ;~D This thing locked my computer up ~but~ good.
RBA, who do we ping for computer Windows/Trend Micro articles?
Yah... but normally, it's that first couple of hours during an outbreak that you really *need* to make sure the user base is covered. I don't recall this ever happening before. With software updates, I can see... but with mere pattern files, earlier is better than later. Except this time. :-)
All things considered... this probably nailed others more than it nailed us.
We only had about 40 or 50 computers freeze, out of about 800. Only one server out of about 50. Mainly 'cuz it hit late in the day on the west coast, and a friday. Machines connected but not running any IP apps at the moment, mostly never noticed.
Its been my experience that if we're hit by an outbreak, its going to be a few hours before Trend gets pattern files ready anyway. At that point, its better to put some kind of remediation in place -- email rules, port blocking, etc. We've had some problems in the past that forced us to put the delay in -- namely, Trend mistaking critical OS files as containing a virus. Not fun.
Huh. Okey doke. I haven't had Trend misidentify OS files that I can remember... but my memory ain't whut it used to be... :-)
The problem though, is that worm that gets inside the firewall to just that one host, and starts running the table in the ol' corporate host billiards game.
All things considered... this little gotcha cost me about two hours of major annoyance for a small minority of users, followed by a couple more hours for just a few stragglers. Compared to the damage that a clever worm could do in mere minutes to hosts on a scale orders of magnitude worse... we got off pretty easy.
That doesn't mean I'm a super happy camper, but I'm a camper with some perspective.
You take this kind of ping?
ping
Yup, I've got a dead hard drive to prove it. One of our four XP Home machines runs TM, and it went wacko on Friday, using all the CPU, wouldn't shut down, open programs, reboot, etc. Tests reported that the NTFS MFT bitmap was corrupted. Couldn't fix it with chkdsk, recovery console or anything else I tried.
Of course, that's the only machine I don't have a backup for...but I won't be re-installing Trend Micro ever again.
I had a problem with TM a year ago -- it started to go crazy one day, downloading every five minutes. I uninstalled it and installed Avast and Sygate on that machine, but since I had almost a full paid subscription left, I moved TM to the least important machine.
Bad move -- I should have just cut my losses and tossed the TM altogether.
Not directly, but I'm playing now. I've been trying to lever Symantec's package off my servers with a crowbar and Trend was one of my recommendations. Oopsie.
I read that changing antivirus solutions or backup solutions was like paddling your little boat from one spot to another on the Lake Of Fire. Yer still in Hell...
Your hard drive shouldn't be dead. There's a fairly easy fix to this one:
- Boot the computer into Safe Mode. Locate the file in the C:\~program files~Trend Micro~ folder with a ".594" or .593" extension. Delete those files. Reboot the computer.
Fixed. Update again, and by now you'll get at least .598.
No worries. :-)
Thanks!
This little foible aside... I do have to say I've had really good experience with Trend products over the several years I've used them.
This was certainly an annoyance, but its not yet enough to have me shopping for another solution. Trend's corporate edition is a really nice way to centralize and push out updates, and they do it for a way better price than McAfee.
But for last weekend... Grrrrrr....
Trend Micro Ping!
Trend Micro's "Housecall" web-based service is excellent. Never had any trouble with it. It appears that this only affected the paid installable software, correct? I've not paid for anti-virus protection since I heard about housecall and AVG during graduate school.
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.