10 things I wish I'd known before becoming a Linux sysadmin

Red Hat ^ | 12 February 2020 | by Ken Hess

[ShadowAce]

I love being a Linux system administrator, but there are things about it that I don't love. No job is perfect, but someone should at least warn the newcomers of the dangers that lie ahead. Well, you've come to the right place to find out everything you wanted to know, and probably a little extra, about being a Linux system administrator. These are my experiences and might not reflect the greater system administration universe. I make no claims, promises, or guarantees by presenting these ten things I wish I'd know before becoming a Linux system administrator to you. They are in no particular order.

1. Users are going to be a major pain. Users are a necessary evil, however, because, without users, we really have no purpose. Still, users do tend to work my nerves at times. They always claim to have changed nothing, although a shallow probe proves otherwise. And the ones with a little knowledge are the worst because they believe that they should have root access so that they can, "Fix their own problems." Yeah, that could happen. 
    
2. Systems always break when you go on vacation. It's inevitable, isn't it? You baby your systems, treat them with kid gloves, and then the very day that you arrive on the beach, BOOM, someone from the office calls and says, "Hey, sorry to bother you because I know you're on vacation, but,"...and the ensuing tale, as heartbreaking as it is, doesn't completely kill my margarita-induced buzz. Sure, I fix it, but I'm angry and I say a lot of bad words to relieve my frustration, but I fix it and then try to restore my sweet vacation bliss.
    
3. Things break at the least convenient times. The end of the month, late at night, just before a security audit, in the middle of a presentation, during holidays, and while you're on vacation (See #2 above) are all perfect times for major outages. Outages never happen when you're fully staffed, wide awake, and in a good mood. Just once, I'd love for an outage to occur Tuesday morning at 10:30 when everyone is standing around talking about where we want to go to lunch, so I can say, "Oh, that's no problem, the whole team's here. We're on it."
    
4. Everyone who worked on the systems before you seems incompetent. It has never failed to be true on every new gig or job I've ever taken. I've never figured out what the people before me did at the same job I now have. They didn't patch. They didn't maintain the hardware. They didn't remove old users. They never upgraded anything. And they documented nothing. On one job, the Linux systems were so out of date, it took hours to update them. And I don't want to even tell you how many reboots and update cycles it required to also patch the Windows servers. 
    
5. Your peers get mad at you when you mention rebooting the servers. Apparently, long uptimes are more impressive to the geekier among us than they are to those of us who dread what happens when a system with a 1,200-day uptime is finally rebooted. If I see an uptime of 90 days or more, I reboot the system. I have a lot of stories of bad things happening because systems haven't been rebooted. Just reboot at least once a quarter to make me happy, if for no other reason.
    
6. There's always a legacy system to support. And that legacy system is the most important system on the whole network and you somehow must keep it healthy, patched, and running. Every job I've ever had contained one of those systems. Every client site. Every data center. Every one. From a 10-year-old Xenix system to many old SCO Unix systems to a Solaris 2.5 computer to an "old Linux system under a desk", I've had to support the unsupportable. For some reason, I'm either the lucky one or the stupid one who'll actually dig in and fix what's broken on a system that even the original vendor can't fix anymore. My eyes are almost worn out from rolling and I have no sighs left in me.
    
7. Backups fail. I'm not convinced that backups ever work. Backups seem to be the weakest link in every company. And somehow I've taken the blame more than once for failed backups, although in an enterprise, it wasn't my responsibility. My best example is when I took on support for a gaggle of systems that hadn't had a good backup in more than three years. However, on one fine day about three months into my new gig, we needed to restore a system that another sysadmin fat-fingered to death. I was somehow blamed for "not checking the backups," although we had a fully staffed backup and restore (BUR) team. See #4 above for no one ever having checked the backups before I arrived on the scene. Don't trust backups. Ever.
    
8. Printing is the most important thing in the world. Although we've lived in a paperless world for at least the last 15 years, some people still believe that they must print things onto paper. It's true. Sure, some things need to be printed, but for that, you only need about two printers for a large office. One for the HR and accounting folks that are not accessible by everyone and then another for everyone else. Almost no one needs to print onto paper. But, if one of the 15 available printers isn't working perfectly, it's a major emergency to get it fixed ASAP. It couldn't possibly be user error such as trying to print the same document six times, so just fix the printer. No, apparently they've never heard of PDFs.
    
9. You take criticism for keeping systems secure. Although it is your job and part of company policy to secure systems, everyone wants to be an exception to the rule. Someone doesn't want to use two-factor authentication. You have a VIP who wants to keep the same password forever. And you take all the heat during the post-mortem after a failed security audit. It's a vicious cycle and you can't win. Everyone wants and demands security but they also can't be inconvenienced in any way to comply with that security. 
    
10. You're perpetually at the bottom of the food chain. You support the infrastructure. The infrastructure is seen as the weakest link in the support chain. You are responsible for fixing every problem that exists, whether within your control or not. Remember, no matter what the problem is, "It's not a network problem." You must find the root cause and you can't point the finger at anyone else during the process. When someone else breaks it, it just broke. When you break it, your job and your reputation are both on the line. I had a sysadmin coworker who had a proverb written on his cubicle wall that read, "Only the person who washes dishes ever breaks any." This means that if you do anything, you're going to make mistakes. People who do nothing never make mistakes.

It's my goal to paint a realistic and somewhat humorous picture of what being a sysadmin is. I don't want to make you believe that there aren't rewards because there are. Sysadmins can make a decent living. There is often a lot of freedom to work remotely. You can often work whatever hours you please and not necessarily eight all in a row. There are some perks in that some people appreciate you for the computing genius that you are. Sometimes kind people will bring you goodies or even take you to lunch because of your awesomeness. Every job has its downsides and its upsides and system administration is no different.

[Rurudyne]

In the Dilbert TV show it was revealed that Wally was the only guy who knew how to make the system behind all systems function.

[SteveH]

so naughty

[wally_bert]

Especially printing.

[rlmorel]

I fail to see how this is a Linux Sysadmin list!

Anyone who works with and has responsibility for any systems (most especially high availability systems) has exactly the same issues.

It is a standard list...all of them.

[SteveH]

you also

[central_va]

Unix is evil. Good parent processes always kill off their children before they die so their children don’t become zombies.

[IndispensableDestiny]

11. All IT staff lie to the users.

[reg45]

I think that most of those ten points apply no matter which OS you are using.

[redcatcherb412]

#2 is very true.
Wife and I on the plane, after it landed the stewardess stopped me at the door telling me pilot advised her my company (another airline) had radioed the pilot and requested I call as soon as we landed. Computer problem at the HQ.
Honest to God real.

[SteveH]

the key to being a successful sysadmin is always acting annoyed. that way, no one bothers you, and then you can go back to playing xtrek with your other sysadmin buddies on the net.

[Bommer]

They have to because of rule #1!
8^)

[rlmorel]

My brother worked at DEC back in the early days, and they had a VIP customer who had a system failure, and in trying to fix it, seemed to bring in more and more people to go down more and more obscure remediation alleys, until they had a large group of people (including the VIP customer and DEC bigwigs) watching, and when they ran a command, after a long pause, the terminal said “System go bye-bye”.

Some coder somewhere thought they had sense of humor, and probably thought that if they coded that little fun in, that branch would be so obscure nobody would ever get to it (much less the customer) but in front of that important crowd, it popped up on the screen.

Apparently, the VIP customer AND the DEC bigwigs went ballistic, and they got so angry they tracked down the guy who wrote the code and fired him.

[SgtHooper]

LMAO! Someone who is realistic!

[SteveH]

never heard that one (but then, i don’t know all the stories). wonder if it was VMS...

[Skywise]

I was going to say the same thing

[Joe Brower]

Hah! Spot on. I love it.

[thoughtomator]

If you’ve never heard of the Bastard Operator from Hell you should look it up.

[VastRWCon]

#6 we still have programs in use that were originally written in 1988. Now that’s a legacy!

[zeugma]

Unix is evil. Good parent processes always kill off their children before they die so their children don’t become zombies.

Of course, you can always disown your children, but sometimes that gets messy. Zombies are the worst.

[MarkL]

Linux is user friendly.

It’s just very particular about who its friends are...

Mark