Posted on 06/23/2026 10:40:26 AM PDT by ShadowAce
Oops: Twenty-eight years ago, Pixar nearly lost 90% of Toy Story 2's digital files – not because a system crashed, but because someone ran a routine Unix command that engineers had been using for years without a second thought. The command /bin/rm -r -f instructs the system to recursively delete everything under a directory without asking for confirmation. At Pixar in 1998, it was apparently executed in the wrong location.
The studio's animation pipeline at the time ran across a network of Unix and Linux machines holding hundreds of thousands of production files. Artists and technical staff had broad access to both personal workspaces and shared production directories. The setup made collaboration easier, but meant a routine cleanup command could reach critical files if issued from the wrong directory.
According to people familiar with the incident, the command propagated beyond its intended scope and began erasing core production data. Oren Jacob, an associate technical director on the film, watched it happen in real time. Files began disappearing from his screen, first individual assets, then entire characters and sequences.
Within moments, roughly 90% of the movie was gone.
"You don't often watch a company vaporize in front of your eyes," Jacob told The Wall Street Journal.
The immediate response was to contain the damage. An emergency call went out to shut down the system, cutting off the deletion mid-process. That worked. The fallback plan did not.
Pixar's backup system was designed precisely for this kind of scenario, but it had been failing silently. Nobody realized it until they tried to use it. "The mechanism we had in place specifically to help us recover from data failures had itself failed," Pixar co-founder Ed Catmull later wrote in Creativity, Inc.
What remained of the film was fragmented and inconsistent. Reconstructing it from pieces would have been slow and uncertain, and the timeline was already tight. Pixar had recently committed to releasing Toy Story 2 theatrically rather than as a direct-to-video sequel, raising both the creative bar and the financial stakes. A long delay could have had serious consequences for the company.
To Pixar's leadership, identifying who triggered the error was beside the point. "Looking for someone to blame doesn't help us learn from mistakes," Catmull said. "We understood that the deletion of the movie was an accident because somebody typed in a command when they were in the wrong directory. We don't know who typed the command, or if they even knew that they were the one who did it, but it didn't matter." What mattered was whether the film could be recovered at all.
The answer turned out to be sitting off-site. Galyn Susman, the film's supervising technical director, had been maintaining a working copy of the project on a machine at her home. She had built a remote workflow during her pregnancy, periodically syncing updated versions of the film so she could continue working after hours. It wasn't part of Pixar's formal backup infrastructure, but it was current enough to matter.
That machine held what was likely the only intact version of the film. Susman and Jacob went to retrieve it. The computer was carefully packed into a car and driven back to Pixar's campus. According to those familiar with the trip, they took no chances with the hardware, treating it as if it were fragile infrastructure rather than a personal workstation.
Once powered up inside the studio, the system delivered what the internal backups could not: a usable copy of the movie.
Recovery still wasn't instantaneous. Teams spent days combing through tens of thousands of files, checking for corruption, missing assets, and inconsistencies. Given how tightly coupled the animation pipeline had become, the process was as much verification as restoration.
The incident exposed several now-obvious gaps: broad access to critical directories, backups that were never tested under real conditions, and a single central store of production data with no distributed redundancy. Modern pipelines use version control, automated backup testing, and distributed storage to make it far harder for one errant command to take out an entire production. In 1998, those safeguards were either incomplete or absent.
Despite the close call, Pixar didn't simply finish and ship the film as it was. In the final year before release, Toy Story 2 underwent extensive creative overhaul. Catmull later described it as "the cinematic equivalent of a heart transplant."
Susman stayed closely tied to the project through completion. "Personally," she said, "I still think Toy Story 2 is the best of the franchise."
Her son, Eli, appears in the film's credits as part of Pixar's "production babies" tradition, which recognizes children born during a movie's development. In this case, the connection runs deeper. The same home setup built around that period ended up preserving the film itself.
|
Click here: to donate by Credit Card Or here: to donate by PayPal Or by mail to: Free Republic, LLC - PO Box 9771 - Fresno, CA 93794 Thank you very much and God bless you. |
My very first internship in college at a real-time Unix shop, I did an ‘rm -rf’. Spent the rest of the afternoon learning how to use tar to restore from backup.
At least you had a backup.
Why panic, no great loss...
Experience, the absolute best teacher LOL (not laughing at you)
Always change directory to root ‘/’ before running rm -rf. Make sure you have su - root first. Now you are talking unix.
I used to do contract firmware development at Slacker.com and when I had JUST started the firmware build system built the firmware on the server, and you just copied the output file down to your PC and flashed it to your device. Part of this build process involved creating a root filesystem in a directory and adding files to it. The first step cd’d to that directory, and then ran an “rm -rf *” to clean the old root filesystem before it started copying the new files to it. Well, IT had borked my permissions, so the “cd” failed, but the rm -rf * worked. Deleting all the files on the server. This happens, I see what has happened, and I trot over to IT. “I just wiped the server”. (condescendingly) “No you didn’t.”. (Tries to log in, can’t. Does some other checks, sees that server is blank). “What did you do?” I tell him the command I issued, tell him that the “cd” failed, looked like a permissions error. They pull out the tapes, and start a restore. It was very soon after this event that the build process changed, and we were now building our code locally. (Which we should have been doing all along).
They should have upgraded to Windows.
What happens when you run
sudo rm -rf —no-preserve-root /
https://www.youtube.com/watch?v=nkPpwH2HTOU
What happens? You get fired.
In that case, fortunately, it was a virtual machine
We solved this by never deleting anything. Eventually, I started creating .gz versions of older log and data files we would never use again and deleting the files. Then I downloaded the compressed files to the NAS. When I retired, that was the end of that.
In the early 80’s, I was working with a consulting firm in NYC. I came in early and say an analyst crying by her computer terminal. She spent the weekend entering data for a utility rate case. The TECO text editor had a function -KT- Which saves all your work. KY, however, saved the current ‘buffer’ of work, erases all the other data, and overwrites the file without saving. Sad.
And don’t forget to give everyone you hire, including interns, access to your production servers on day 1
I don’t buy into this story. It has threads of plausibility, but I’m highly skeptical that all of this is true. There would have been multiple backups in multiple media formats, and works in progress would be saved to hundreds of different machines and backups. It may have been a pain in the ass to recover, but there was no way they lost everything from one command - as the result clearly shows.
Exactly. This is how I maintain all my systems too. It’s standard operating procedure for all IT engineers and we teach it to all new joiners on day one.
Not in 1998.
Why not? I found backups were far more likely to exist then than now because people were more disciplined about it and it was less automated.
Now backups are so automated and people are so used to not thinking about it there is a tendency to forget about them. Or test that you actually can do a restore.
Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.