Free Republic
Browse · Search
General/Chat
Topics · Post Article

Skip to comments.

10 things I wish I'd known before becoming a Linux sysadmin
Red Hat ^ | 12 February 2020 | by Ken Hess

Posted on 02/13/2020 11:30:17 AM PST by ShadowAce

I love being a Linux system administrator, but there are things about it that I don't love. No job is perfect, but someone should at least warn the newcomers of the dangers that lie ahead. Well, you've come to the right place to find out everything you wanted to know, and probably a little extra, about being a Linux system administrator. These are my experiences and might not reflect the greater system administration universe. I make no claims, promises, or guarantees by presenting these ten things I wish I'd know before becoming a Linux system administrator to you. They are in no particular order.

  1. Users are going to be a major pain. Users are a necessary evil, however, because, without users, we really have no purpose. Still, users do tend to work my nerves at times. They always claim to have changed nothing, although a shallow probe proves otherwise. And the ones with a little knowledge are the worst because they believe that they should have root access so that they can, "Fix their own problems." Yeah, that could happen. 
     
  2. Systems always break when you go on vacation. It's inevitable, isn't it? You baby your systems, treat them with kid gloves, and then the very day that you arrive on the beach, BOOM, someone from the office calls and says, "Hey, sorry to bother you because I know you're on vacation, but,"...and the ensuing tale, as heartbreaking as it is, doesn't completely kill my margarita-induced buzz. Sure, I fix it, but I'm angry and I say a lot of bad words to relieve my frustration, but I fix it and then try to restore my sweet vacation bliss.
     
  3. Things break at the least convenient times. The end of the month, late at night, just before a security audit, in the middle of a presentation, during holidays, and while you're on vacation (See #2 above) are all perfect times for major outages. Outages never happen when you're fully staffed, wide awake, and in a good mood. Just once, I'd love for an outage to occur Tuesday morning at 10:30 when everyone is standing around talking about where we want to go to lunch, so I can say, "Oh, that's no problem, the whole team's here. We're on it."
     
  4. Everyone who worked on the systems before you seems incompetent. It has never failed to be true on every new gig or job I've ever taken. I've never figured out what the people before me did at the same job I now have. They didn't patch. They didn't maintain the hardware. They didn't remove old users. They never upgraded anything. And they documented nothing. On one job, the Linux systems were so out of date, it took hours to update them. And I don't want to even tell you how many reboots and update cycles it required to also patch the Windows servers. 
     
  5. Your peers get mad at you when you mention rebooting the servers. Apparently, long uptimes are more impressive to the geekier among us than they are to those of us who dread what happens when a system with a 1,200-day uptime is finally rebooted. If I see an uptime of 90 days or more, I reboot the system. I have a lot of stories of bad things happening because systems haven't been rebooted. Just reboot at least once a quarter to make me happy, if for no other reason.
     
  6. There's always a legacy system to support. And that legacy system is the most important system on the whole network and you somehow must keep it healthy, patched, and running. Every job I've ever had contained one of those systems. Every client site. Every data center. Every one. From a 10-year-old Xenix system to many old SCO Unix systems to a Solaris 2.5 computer to an "old Linux system under a desk", I've had to support the unsupportable. For some reason, I'm either the lucky one or the stupid one who'll actually dig in and fix what's broken on a system that even the original vendor can't fix anymore. My eyes are almost worn out from rolling and I have no sighs left in me.
     
  7. Backups fail. I'm not convinced that backups ever work. Backups seem to be the weakest link in every company. And somehow I've taken the blame more than once for failed backups, although in an enterprise, it wasn't my responsibility. My best example is when I took on support for a gaggle of systems that hadn't had a good backup in more than three years. However, on one fine day about three months into my new gig, we needed to restore a system that another sysadmin fat-fingered to death. I was somehow blamed for "not checking the backups," although we had a fully staffed backup and restore (BUR) team. See #4 above for no one ever having checked the backups before I arrived on the scene. Don't trust backups. Ever.
     
  8. Printing is the most important thing in the world. Although we've lived in a paperless world for at least the last 15 years, some people still believe that they must print things onto paper. It's true. Sure, some things need to be printed, but for that, you only need about two printers for a large office. One for the HR and accounting folks that are not accessible by everyone and then another for everyone else. Almost no one needs to print onto paper. But, if one of the 15 available printers isn't working perfectly, it's a major emergency to get it fixed ASAP. It couldn't possibly be user error such as trying to print the same document six times, so just fix the printer. No, apparently they've never heard of PDFs.
     
  9. You take criticism for keeping systems secure. Although it is your job and part of company policy to secure systems, everyone wants to be an exception to the rule. Someone doesn't want to use two-factor authentication. You have a VIP who wants to keep the same password forever. And you take all the heat during the post-mortem after a failed security audit. It's a vicious cycle and you can't win. Everyone wants and demands security but they also can't be inconvenienced in any way to comply with that security. 
     
  10. You're perpetually at the bottom of the food chain. You support the infrastructure. The infrastructure is seen as the weakest link in the support chain. You are responsible for fixing every problem that exists, whether within your control or not. Remember, no matter what the problem is, "It's not a network problem." You must find the root cause and you can't point the finger at anyone else during the process. When someone else breaks it, it just broke. When you break it, your job and your reputation are both on the line. I had a sysadmin coworker who had a proverb written on his cubicle wall that read, "Only the person who washes dishes ever breaks any." This means that if you do anything, you're going to make mistakes. People who do nothing never make mistakes.

It's my goal to paint a realistic and somewhat humorous picture of what being a sysadmin is. I don't want to make you believe that there aren't rewards because there are. Sysadmins can make a decent living. There is often a lot of freedom to work remotely. You can often work whatever hours you please and not necessarily eight all in a row. There are some perks in that some people appreciate you for the computing genius that you are. Sometimes kind people will bring you goodies or even take you to lunch because of your awesomeness. Every job has its downsides and its upsides and system administration is no different.


TOPICS: Computers/Internet
KEYWORDS: linux
Navigation: use the links below to view more comments.
first previous 1-2021-4041-6061-72 next last
To: ShadowAce

Systems, I can relate to and communicate with just fine. I always know what I’m up against. I’m always in control.

Humans on the other hand, I never know what I’m dealing with.


41 posted on 02/13/2020 1:06:57 PM PST by CodeJockey (Dum Spiro, Pugno)
[ Post Reply | Private Reply | To 1 | View Replies]

To: ClearCase_guy

On the AS/400, you had the option to put annoying people of the “slow user queue.”

I really liked that!


42 posted on 02/13/2020 1:08:54 PM PST by ConservativeMind (Trump: Befuddling Democrats, Republicans, and the Media for the benefit of the US and all mankind.)
[ Post Reply | Private Reply | To 6 | View Replies]

To: ShadowAce

Hill.
Sh1t.
Gravity.
Bucket.
Regardless of OS


43 posted on 02/13/2020 1:28:43 PM PST by LakeEffectLad (American's are Dreamers, too!!)
[ Post Reply | Private Reply | To 1 | View Replies]

To: thoughtomator

I read that all the time—I’m caught up, in fact.


44 posted on 02/13/2020 1:29:03 PM PST by ShadowAce (Linux - The Ultimate Windows Service Pack)
[ Post Reply | Private Reply | To 37 | View Replies]

To: LakeEffectLad

Regardless of career


45 posted on 02/13/2020 1:35:40 PM PST by Secret Agent Man (Gone Galt; Not Averse to Going Bronson.)
[ Post Reply | Private Reply | To 43 | View Replies]

To: ShadowAce

I don’t need to read past number 1 to see what the problem is.


46 posted on 02/13/2020 1:36:14 PM PST by Arones (When Leftists are in a minority, then they look for other ways to win.)
[ Post Reply | Private Reply | To 1 | View Replies]

To: ShadowAce

So the IT guy is troubleshooting someone’s workstation and says, “Hmmm...that’s interesting.” This is IT code used when in a mixed environment. What it translates to is “What did you f***ing do to your workstation this time!”

I’ve been out of the business for over 16 years but I got hired at a bank and inherited a Novel network with Lotus Notes for email, Window’s NT workstations and an AT&T voice mail system. Had to sysadmin all of these and then had the whole Y2K non-issue to contend with and don’t get me started when every banker “had” to have a shiny new BlackBerry and wanted their email to forward to it.

Still, the best IT job I had. Software, hardware, pulling wire, punch downs, back-ups, restores, purchasing, office moves, comm room build-outs, account creations, assigning rights. No way to get bored, there was always something different to do, and then end users always found new and interesting ways to keep you occupied.


47 posted on 02/13/2020 1:37:47 PM PST by Sergio (An object at rest cannot be stopped! - The Evil Midnight Bomber What Bombs at Midnight)
[ Post Reply | Private Reply | To 1 | View Replies]

To: SteveH

It was told to me secondhand, so take that for what it is worth.

When I was in the USN on my second Med deployment, I worked with a DEC mainframe on a special project that was recording all the flight information on our A-7 Corsairs. I was a jet mechanic, but they had me working with a rep from Detroit Diesel Allison, and one of my jobs was to read the flight recorder tapes and plot out the flight on a graph showing RPM, Temp, vibration, throttle position, etc.

All this is De Rigueur now, but back then, warplanes didn’t have it.

Anyway, when working on plotting the tape info on the PDP-11, you had to enter the variables just right, and if you didn’t it would spit back “Error-try again”.

Well, I kept making the error of some kind, and after the fifth error, instead of saying “Error-try again” it said something like “Are you in the Marines or something-try again”

When it errored again, it said “Pathetic-try again” and I just began making deliberate errors until it said “What do you have on the ends of your hands, Sh*t for fingers?”

I thought it was pretty funny, and then it went back to the standard dialog!


48 posted on 02/13/2020 1:52:58 PM PST by rlmorel (Finding middle ground with tyranny or evil makes you either a tyrant or evil. Often both.)
[ Post Reply | Private Reply | To 34 | View Replies]

To: ShadowAce
Anyone out there who is or has ever been a unix sysadmin should be familiar with The Bastard Operator from Hell series. If not, you've missed something really funny...
49 posted on 02/13/2020 2:10:43 PM PST by zeugma (I sure wish I lived in a country where the rule of law actually applied to those in power.)
[ Post Reply | Private Reply | To 1 | View Replies]

To: Secret Agent Man

Good point


50 posted on 02/13/2020 2:18:25 PM PST by LakeEffectLad (American's are Dreamers, too!!)
[ Post Reply | Private Reply | To 45 | View Replies]

To: zeugma
The Bastard Operator From Hell - Genesis (Striped Irregular Bucket #1)

I'm really bored. You know how bored you get when work's going on and on and on, and nothing interesting is happening, and you're listening to a radio that picks up ONE station on FM, and it's always the station with the least records in the city, about 5, and one of them is "You're so Vain" which wasn't too bad a song until you hear it about 3 times a day for a year, and *EVERY* time it plays, the announcer tells you it's about Warren Beatty and who he's currently poking, someone you'll never sniff the toe-jam of, let alone meet, let alone get amorous with. And EVERY time someone mentions Warren Beatty, someone says that he used to go out with Madonna too, and have you seen "In Bed With.."

AND THEN, someone ELSE will say "It wasn't really about Warren Beatty, it was James Taylor" and the first person will say "What, `In bed with Madonna?'", and they laugh and everyone else laughs, and I slip out the Magnum from under the desk where I keep it in case someone laughs at a joke that's so dry it's got a built in water-fountain, and blow the lot of them away as a community Service. I figure that I'll get time off my sentence if I ever kill someone by accident who's got a life.

So visitors are getting pretty thin at the moment, and the Quick-Lime Pits are filling up rapidly, and all I've got to do is the full backups and maybe I can go home.

So, to relieve the boredom, I get some iron filings and pour them into the back of my Terminal until it fizzes out (Which doesn't take all that long, surprisingly enough), then call our maintenance contractors and log a fault on the device. Sometimes they'll send someone who knows what they're doing, but it's a lot more fun when they don't - which is about 98% of the time.

So they maintenance guy comes in, and I can tell he's NEW because the photo on his ID actually LOOKS like him, not like the head engineer, whose photo's a black and white tin-type (he's that old).

Maintenance Contractors always dress up nice, with a tie and everything because they believe that a customer will trust a nicely dressed guy with their million dollar equipment *just* because he's got a nice tie..

Because he's NEW and ALONE, he's what you call an appeasement engineer, the new guy they send so they respond within the 4 hour guaranteed response period. (Things are getting better and better) Your average appeasement engineer is about as clued-up on computers as the average computer "hacker" is about B.O, and their main job is to make sure the power plug is in and switched on, then call back to the office for "PARTS". The really keen ones will sometimes even take a cover off the equipment and pretend that they see this stuff all the time. I wonder what sort today's is...

"You got a dud terminal?" he asks pleasantly

I tell him yeah, and bring him into the control room.

"Which one is it?" he asks, confused by the fact that only one of them is smoking.

"It's the Model Three" I say, giving NOTHING away.

"Ah, the old model three!" he says knowingly, without a clue what a model three is, or which one of the three terminals it is, which isn't surprising, as I just made it up.

"We get a lot of Model Three problems" he says nodding "So what actually happened?"

Sneaky, but not good enough. I'm not going to point it out to him.

"It just went dead" I say, in luser mode.

"I see. Could you just recreate what you were doing so I can check the unit out when it's ready for operation?"

Very Sneaky. I decide to let him off the hook.

"Look, I've got to go to the toilet, there it is over there" I say, pointing at our Waffle-Iron.

"But that's a Wa..." He says, then stops. He's a beginner, and it's just possible that the company has a line of terminals that look like waffle irons. He bites.

"Sorry" he says, smiling again "for a minute there I thought it was a Model 2!"

A reasonably good save, but it won't save him.

I leave, which means he's got to take it to bits, otherwise he knows I won't believe he's worked on it. I give him a couple of minutes to get the element exposed then wander back in.

"So how does it look?" I ask, concerned-like.

"Well, I think we could have a processor problem.." he says concentrating on prying the element up.

..concentrating so much that he doesn't notice me plugging the iron in.

"Shouldn't you be wearing an earthing strap?" I ask innocently.

When he thinks I can't see, he creeps his hand over to the wiring frame and says "Well, It's just as easy to hold onto earth like this"

"But what about the risk of a cross-the-body shock with no resistor in series with you?" I ask ever-so-more-innocently

"Oh, it's ok" he says "the unit's unplug..."

*click* *BZZZZZZZEEERRT!* *clunk!*

I ring the maintenance help-desk again...

It's Rhonda

"Hey Ronda!, Ah, I'm going to need another engineer and a new Waffle Iron over here; for some reason your engineer opened up my Waffle Iron without switching it off." I say

Rhonda knows me. It's the third call and the third appeasement engineer. "You're a real prick" she says, annoyed

"Tell ya what Rhonda, why don't you come and fix it; it's a Model Three..."

51 posted on 02/13/2020 2:24:03 PM PST by zeugma (I sure wish I lived in a country where the rule of law actually applied to those in power.)
[ Post Reply | Private Reply | To 49 | View Replies]

To: Bommer
They have to because of rule #1!

When my work computer got trashed by a network pushed update to Windows 10 (from 7), I took it to our helpdesk for them to fix the video drivers. They didn't know my background in managing a big IT project.

The kid says my laptop was due for a "tech refresh" meaning a new machine. I wasn't in the mood, but asked what the new computer would be. Turns out the central buy my organization did was for crippled machines. I asked him to just fix the one I had (only two years old with lots of RAM and a decent processor).

While browsing to the enterprise folder containing the correct drivers he tells me there is no Windows 10 driver for the video. He's staring at the correct install file while telling me this. I asked him to double click "there" pointing at the executable. He says it is for something else.

I eventually surrendered and took the new laptop. I never learned why he would install the correct driver. I was on our enterprise Windows 10 "golden disk" distribution.

52 posted on 02/13/2020 2:39:04 PM PST by IndispensableDestiny
[ Post Reply | Private Reply | To 31 | View Replies]

To: ShadowAce

dating myself, but IIRC I found BOFH when he was up to issue #4


53 posted on 02/13/2020 2:52:11 PM PST by thoughtomator (... this has made a lot of people very angry and been widely regarded as a bad move.)
[ Post Reply | Private Reply | To 44 | View Replies]

To: ShadowAce

If you’re in IT, you should have known this going in...

2 & 3) Computers and networks were originally designed by some guy named “Murphy!”

6) At my last job, there was a business critical application that would only run using Microsoft Java and Windows 98, and ran on a Compaq Deskpro 386. The company that wrote the software was long out of business. They were still using it in 2017.

7) Nobody ever gets fired for bad backups. It’s the “bad restores” that’ll boot you out the door!

Mark


54 posted on 02/13/2020 2:55:10 PM PST by MarkL (Do I really look like a guy with a plan?)
[ Post Reply | Private Reply | To 1 | View Replies]

To: ShadowAce
There's always a legacy system to support. And that legacy system is the most important system on the whole network and you somehow must keep it healthy, patched, and running. Every job I've ever had contained one of those systems. Every client site. Every data center. Every one. From a 10-year-old Xenix system to many old SCO Unix systems to a Solaris 2.5 computer to an "old Linux system under a desk", I've had to support the unsupportable. For some reason, I'm either the lucky one or the stupid one who'll actually dig in and fix what's broken on a system that even the original vendor can't fix anymore. My eyes are almost worn out from rolling and I have no sighs left in me.

I am trying to arrange secret service protection for you, and for downloading your expertise in case of death or nuclear attack. Your employers should be thanking God for your industrious diligence, skill and knowledge.

55 posted on 02/13/2020 2:56:32 PM PST by daniel1212 ( Trust the risen Lord Jesus to save you as a damned and destitute sinner + be baptized + follow Him)
[ Post Reply | Private Reply | To 1 | View Replies]

To: ShadowAce

Been in IT a long time...my first rule when things break

“The harder it is to find the problem... the dumber the cause for the problem”


56 posted on 02/13/2020 3:38:04 PM PST by tophat9000 (Tophat9000)
[ Post Reply | Private Reply | To 2 | View Replies]

To: MarkL

Nobody ever gets fired for bad backups. It’s the “bad restores” that’ll boot you out the door!


That is called a “resume generating” event


57 posted on 02/13/2020 3:41:48 PM PST by tophat9000 (Tophat9000)
[ Post Reply | Private Reply | To 54 | View Replies]

To: Moss

The problem with the “pecking order”... is you always work for the “head pecker”


58 posted on 02/13/2020 3:47:01 PM PST by tophat9000 (Tophat9000)
[ Post Reply | Private Reply | To 13 | View Replies]

To: central_va

Unix is evil...

No Unix was just reversed engineer from the crashed UFO at Roswell in 47


59 posted on 02/13/2020 3:53:51 PM PST by tophat9000 (Tophat9000)
[ Post Reply | Private Reply | To 26 | View Replies]

To: ShadowAce

“Backups fail.”

we used to phrase it as follows: “Sooner or later, you’ll be down to your very last working backup”


60 posted on 02/13/2020 4:14:10 PM PST by catnipman (Cat Nipman: Vote Republican in 2012 and only be called racist one more time!)
[ Post Reply | Private Reply | To 1 | View Replies]


Navigation: use the links below to view more comments.
first previous 1-2021-4041-6061-72 next last

Disclaimer: Opinions posted on Free Republic are those of the individual posters and do not necessarily represent the opinion of Free Republic or its management. All materials posted herein are protected by copyright law and the exemption for fair use of copyrighted works.

Free Republic
Browse · Search
General/Chat
Topics · Post Article

FreeRepublic, LLC, PO BOX 9771, FRESNO, CA 93794
FreeRepublic.com is powered by software copyright 2000-2008 John Robinson