10 things I wish I'd known before becoming a Linux sysadmin

Red Hat ^ | 12 February 2020 | by Ken Hess

[ShadowAce]

I love being a Linux system administrator, but there are things about it that I don't love. No job is perfect, but someone should at least warn the newcomers of the dangers that lie ahead. Well, you've come to the right place to find out everything you wanted to know, and probably a little extra, about being a Linux system administrator. These are my experiences and might not reflect the greater system administration universe. I make no claims, promises, or guarantees by presenting these ten things I wish I'd know before becoming a Linux system administrator to you. They are in no particular order.

1. Users are going to be a major pain. Users are a necessary evil, however, because, without users, we really have no purpose. Still, users do tend to work my nerves at times. They always claim to have changed nothing, although a shallow probe proves otherwise. And the ones with a little knowledge are the worst because they believe that they should have root access so that they can, "Fix their own problems." Yeah, that could happen. 
    
2. Systems always break when you go on vacation. It's inevitable, isn't it? You baby your systems, treat them with kid gloves, and then the very day that you arrive on the beach, BOOM, someone from the office calls and says, "Hey, sorry to bother you because I know you're on vacation, but,"...and the ensuing tale, as heartbreaking as it is, doesn't completely kill my margarita-induced buzz. Sure, I fix it, but I'm angry and I say a lot of bad words to relieve my frustration, but I fix it and then try to restore my sweet vacation bliss.
    
3. Things break at the least convenient times. The end of the month, late at night, just before a security audit, in the middle of a presentation, during holidays, and while you're on vacation (See #2 above) are all perfect times for major outages. Outages never happen when you're fully staffed, wide awake, and in a good mood. Just once, I'd love for an outage to occur Tuesday morning at 10:30 when everyone is standing around talking about where we want to go to lunch, so I can say, "Oh, that's no problem, the whole team's here. We're on it."
    
4. Everyone who worked on the systems before you seems incompetent. It has never failed to be true on every new gig or job I've ever taken. I've never figured out what the people before me did at the same job I now have. They didn't patch. They didn't maintain the hardware. They didn't remove old users. They never upgraded anything. And they documented nothing. On one job, the Linux systems were so out of date, it took hours to update them. And I don't want to even tell you how many reboots and update cycles it required to also patch the Windows servers. 
    
5. Your peers get mad at you when you mention rebooting the servers. Apparently, long uptimes are more impressive to the geekier among us than they are to those of us who dread what happens when a system with a 1,200-day uptime is finally rebooted. If I see an uptime of 90 days or more, I reboot the system. I have a lot of stories of bad things happening because systems haven't been rebooted. Just reboot at least once a quarter to make me happy, if for no other reason.
    
6. There's always a legacy system to support. And that legacy system is the most important system on the whole network and you somehow must keep it healthy, patched, and running. Every job I've ever had contained one of those systems. Every client site. Every data center. Every one. From a 10-year-old Xenix system to many old SCO Unix systems to a Solaris 2.5 computer to an "old Linux system under a desk", I've had to support the unsupportable. For some reason, I'm either the lucky one or the stupid one who'll actually dig in and fix what's broken on a system that even the original vendor can't fix anymore. My eyes are almost worn out from rolling and I have no sighs left in me.
    
7. Backups fail. I'm not convinced that backups ever work. Backups seem to be the weakest link in every company. And somehow I've taken the blame more than once for failed backups, although in an enterprise, it wasn't my responsibility. My best example is when I took on support for a gaggle of systems that hadn't had a good backup in more than three years. However, on one fine day about three months into my new gig, we needed to restore a system that another sysadmin fat-fingered to death. I was somehow blamed for "not checking the backups," although we had a fully staffed backup and restore (BUR) team. See #4 above for no one ever having checked the backups before I arrived on the scene. Don't trust backups. Ever.
    
8. Printing is the most important thing in the world. Although we've lived in a paperless world for at least the last 15 years, some people still believe that they must print things onto paper. It's true. Sure, some things need to be printed, but for that, you only need about two printers for a large office. One for the HR and accounting folks that are not accessible by everyone and then another for everyone else. Almost no one needs to print onto paper. But, if one of the 15 available printers isn't working perfectly, it's a major emergency to get it fixed ASAP. It couldn't possibly be user error such as trying to print the same document six times, so just fix the printer. No, apparently they've never heard of PDFs.
    
9. You take criticism for keeping systems secure. Although it is your job and part of company policy to secure systems, everyone wants to be an exception to the rule. Someone doesn't want to use two-factor authentication. You have a VIP who wants to keep the same password forever. And you take all the heat during the post-mortem after a failed security audit. It's a vicious cycle and you can't win. Everyone wants and demands security but they also can't be inconvenienced in any way to comply with that security. 
    
10. You're perpetually at the bottom of the food chain. You support the infrastructure. The infrastructure is seen as the weakest link in the support chain. You are responsible for fixing every problem that exists, whether within your control or not. Remember, no matter what the problem is, "It's not a network problem." You must find the root cause and you can't point the finger at anyone else during the process. When someone else breaks it, it just broke. When you break it, your job and your reputation are both on the line. I had a sysadmin coworker who had a proverb written on his cubicle wall that read, "Only the person who washes dishes ever breaks any." This means that if you do anything, you're going to make mistakes. People who do nothing never make mistakes.

It's my goal to paint a realistic and somewhat humorous picture of what being a sysadmin is. I don't want to make you believe that there aren't rewards because there are. Sysadmins can make a decent living. There is often a lot of freedom to work remotely. You can often work whatever hours you please and not necessarily eight all in a row. There are some perks in that some people appreciate you for the computing genius that you are. Sometimes kind people will bring you goodies or even take you to lunch because of your awesomeness. Every job has its downsides and its upsides and system administration is no different.

[ShadowAce]

Tech Ping

[bigbob]

11. You can’t fix everything with sudo

[ShadowAce]

Every single point above seems to describe my life.

[central_va]

Try this:

#su - root
#cd /
#rm -rf *

Your problems are over!

[ClearCase_guy]

IT is a bit of a thankless job. The money can be OK (you’re generally not going to get rich — Bill Gates didn’t get rich because he was really good at IT. He made business deals.) But age discrimination is a huge problem, outsourcing and H1Bs are a huge problem, and IT is the whipping boy for most organizations. People only pay attention to you when things go badly. So pretty much 100% of the time, when people think of you, it is unkindly.

[Billthedrill]

It’s all true. When I was interviewed for my last job I was asked the cringeworthy “Do you consider yourself a people person?” I told my new boss later that if I’d told the truth it would have been, “No! I’m a sysadmin! I hate my users and I want them all to die!!” “I know,” he sez, “that’s why you got the job.”

[ShadowAce]

lol--here's a better one:

:(){ :|:& };:

[rightwingcrazy]

“It has never failed to be true on every new gig or job I’ve ever taken. I’ve never figured out what the people before me did at the same job I now have. They didn’t patch. They didn’t maintain the hardware. They didn’t remove old users. They never upgraded anything. And they documented nothing.”

Stick around long enough, and you’ll burn out and become one of them.

[SaveFerris]

Correct on all points

[ARGLOCKGUY]

One good thing about being a sysadmin is that no one can ever accuse you of being bigoted,sexist, or homophobic. The reason is you end up hating everyone equally.

[CatOwner]

I saw some of this on our Sun Solaris UNIX systems (as a user) with fellow co-workers. They had very little sympathy or understanding of the work sysadmins go through to secure a system for a defense contractor.

Apparently, I knew enough and was patient enough that I eventually became something of a backup sysadmin for some tasks. There were times when the sysadmins were unavailable to enter our classified compartment (and networks were isolated to the compartment), so a few basic root tasks I was allowed to perform. For something a bit more complicated, they were able to talk me through it.

Yes, I had the root password for about 12 years as a user, but not once did I screw up any of the UNIX systems.

[Moss]

I equate it to the pecking order in a chicken coup. The hens figure out which is the weakest hen and begin to peck it to death, but, the farmer will not remove it. As soon as he does, the hens find a new weakling to peck on. So as long as there is a pecking order there will be a System Administrator.

[chrisser]

My favorite.

You can’t convince management to update the servers and infrastructure even though they’re past their forecast and budgeted usability window.

But then they want to buy bleeding edge toys/gadgets, bring them into the office outside of the normal procurement/testing process and are then ticked off when you can’t get them to integrate into the antique (in IT terms) systems you’re forced to keep supporting.

[sauropod]

.

[miliantnutcase]

This isn’t just Linux admin, this is really any job in IT.

[cartan]

11. You can’t fix everything with sudo

For anything else, there is always suid…

[glorgau]

IT is like plumbing. Nobody cares unless there is a stench.

[dayglored]

> :(){ :|:& };:

That's a perennial favorite fork bomb.

Article is a great read, thanks for posting!

[OrangeHoof]

The real job is being on the other end of the phone from Jazmit Slamalamadingdong in Mumbai on his cheap cell phone (in speakerphone mode) explaining how to solve the problem by formatting the hard drive then rebooting. In other words, you’re on your own, jack.