My very first internship in college at a real-time Unix shop, I did an ‘rm -rf’. Spent the rest of the afternoon learning how to use tar to restore from backup.
At least you had a backup.
Experience, the absolute best teacher LOL (not laughing at you)
We solved this by never deleting anything. Eventually, I started creating .gz versions of older log and data files we would never use again and deleting the files. Then I downloaded the compressed files to the NAS. When I retired, that was the end of that.