Posted on 06/23/2026 10:40:26 AM PDT by ShadowAce
Oops: Twenty-eight years ago, Pixar nearly lost 90% of Toy Story 2's digital files – not because a system crashed, but because someone ran a routine Unix command that engineers had been using for years without a second thought. The command /bin/rm -r -f instructs the system to recursively delete everything under a directory without asking for confirmation. At Pixar in 1998, it was apparently executed in the wrong location.
The studio's animation pipeline at the time ran across a network of Unix and Linux machines holding hundreds of thousands of production files. Artists and technical staff had broad access to both personal workspaces and shared production directories. The setup made collaboration easier, but meant a routine cleanup command could reach critical files if issued from the wrong directory.
According to people familiar with the incident, the command propagated beyond its intended scope and began erasing core production data. Oren Jacob, an associate technical director on the film, watched it happen in real time. Files began disappearing from his screen, first individual assets, then entire characters and sequences.
Within moments, roughly 90% of the movie was gone.
"You don't often watch a company vaporize in front of your eyes," Jacob told The Wall Street Journal.
The immediate response was to contain the damage. An emergency call went out to shut down the system, cutting off the deletion mid-process. That worked. The fallback plan did not.
Pixar's backup system was designed precisely for this kind of scenario, but it had been failing silently. Nobody realized it until they tried to use it. "The mechanism we had in place specifically to help us recover from data failures had itself failed," Pixar co-founder Ed Catmull later wrote in Creativity, Inc.
What remained of the film was fragmented and inconsistent. Reconstructing it from pieces would have been slow and uncertain, and the timeline was already tight. Pixar had recently committed to releasing Toy Story 2 theatrically rather than as a direct-to-video sequel, raising both the creative bar and the financial stakes. A long delay could have had serious consequences for the company.
To Pixar's leadership, identifying who triggered the error was beside the point. "Looking for someone to blame doesn't help us learn from mistakes," Catmull said. "We understood that the deletion of the movie was an accident because somebody typed in a command when they were in the wrong directory. We don't know who typed the command, or if they even knew that they were the one who did it, but it didn't matter." What mattered was whether the film could be recovered at all.
The answer turned out to be sitting off-site. Galyn Susman, the film's supervising technical director, had been maintaining a working copy of the project on a machine at her home. She had built a remote workflow during her pregnancy, periodically syncing updated versions of the film so she could continue working after hours. It wasn't part of Pixar's formal backup infrastructure, but it was current enough to matter.
That machine held what was likely the only intact version of the film. Susman and Jacob went to retrieve it. The computer was carefully packed into a car and driven back to Pixar's campus. According to those familiar with the trip, they took no chances with the hardware, treating it as if it were fragile infrastructure rather than a personal workstation.
Once powered up inside the studio, the system delivered what the internal backups could not: a usable copy of the movie.
Recovery still wasn't instantaneous. Teams spent days combing through tens of thousands of files, checking for corruption, missing assets, and inconsistencies. Given how tightly coupled the animation pipeline had become, the process was as much verification as restoration.
The incident exposed several now-obvious gaps: broad access to critical directories, backups that were never tested under real conditions, and a single central store of production data with no distributed redundancy. Modern pipelines use version control, automated backup testing, and distributed storage to make it far harder for one errant command to take out an entire production. In 1998, those safeguards were either incomplete or absent.
Despite the close call, Pixar didn't simply finish and ship the film as it was. In the final year before release, Toy Story 2 underwent extensive creative overhaul. Catmull later described it as "the cinematic equivalent of a heart transplant."
Susman stayed closely tied to the project through completion. "Personally," she said, "I still think Toy Story 2 is the best of the franchise."
Her son, Eli, appears in the film's credits as part of Pixar's "production babies" tradition, which recognizes children born during a movie's development. In this case, the connection runs deeper. The same home setup built around that period ended up preserving the film itself.
I worked in IT, saw a lot of unfortunate errors. All right, who deleted the root account from the UAT machine? It was some rookie SA in India; fortunately, it’s not a production server.
Then there was the Oracle DBA who was asked to truncate one of the test instances, and truncated production instead. She was quite experienced, but she typed the command into the wrong window.
***I don’t buy into this story. It has threads of plausibility, but I’m highly skeptical that all of this is true.***
Agreed! It would be almost impossible to believe that *any* major film studio would develop a multi-million dollar project, having only one copy without continuous, multiple off-site backups.
***Not in 1998***
YES, in 1998 (and also in 1991 when I began work in a UNIX shop!)
They know darn well who did the deed. It was the boss, thus the cover-up.
Also, this is what happens when you hire based on things other than experience and aptitude. So many violations of common sense.
Saving this thread /pun>
Right? How could they not have a backup of a valuable property? Criminal.
ps
I was a Unix admin for 30 years. I cannot think of one time I used the rm -fr . command without cd /top directory and still using the full path of the top directory.
Windows? MSDOS vs Unix? pulllllleazzzzeeee
Right? I had to present my qualifications to the client’s IT VP to receive permission to use root authority.
It’s not hard to figure out who did it. I used to run Big Brother on our systems. One of the functions I wrote did a who every 5 minutes so I knew who was logged on and when.
We had an incident where some files went missing. A month after they asked me find out who did it. I found out who was logged on at the time the files went missing and went to his .sh file and found the rm command. Busted. I felt bad that I had to narc on a nice guy who meant well by freeing space.
Such a silly comment. Why would you even write that?
That includes larger servers, but not mainframes.
RM *.* alert
How did we ever survive not having Git?
And this is the magic of a sneaker network.
Firewalls like this always work; software backup systems not so much.
I’d like to warn of trusting valued data to the now-ubiquitous USB “thumb” sticks. After the second “cannot read corrupted drive” message I bought a good external drive for a master repository and just use the thumbs for moving data and some redundancy.
When shopping for the external drive I was surprised to learn that many of them are scams - just USB sticks put in larger cases, so stick to name brands if you look to buy one.
Research saves assets and asses.
I’ve been a UNIX engineer for a very long time. You would be amazed at how common it is for backups to be taken but never tested. In my view, if you haven’t tried to restore your backups you don’t have any.
The same is true for disaster recovery plans. Countless plans are created but are not tested and without them being successfully tested, they simply don’t exist.
I did that once ("rm -rvf *"), deliberately, on an SGI workstation that was
(1) disconnected from our LAN and
(2) being decommissioned
It stayed "up" for several minutes before some critical files got deleted (things in "/dev", IIRC) and the display went blank. Continued to chatter the disk for several more minutes before it totally quit doing anything.
I felt like Dave Bowman ...
In Unix, before they die good parents will kill all their children before they become zombies.
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.