Command-Line Tip: Put Down the Pipe

Linux Journal ^ | 22 January 2019 | Kyle Rankin

[ShadowAce]

Learn a few techniques for avoiding the pipe and making your command-line commands more efficient.

Anyone who uses the command line would acknowledge how powerful the pipe is. Because of the pipe, you can take the output from one command and feed it to another command as input. What's more, you can chain one command after another until you have exactly the output you want.

Pipes are powerful, but people also tend to overuse them. Although it's not necessarily wrong to do so, and it may not even be less efficient, it does make your commands more complicated. More important though, it also wastes keystrokes! Here I highlight a few examples where pipes are commonly used but aren't necessary.

Stop Putting Your Cat in Your Pipe

One of the most common overuses of the pipe is in conjunction with cat. The cat command concatenates multiple files from input into a single output, but it has become the overworked workhorse for piped commands. You often will find people using cat just to output the contents of a single file so they can feed it into a pipe. Here's the most common example:

cat file | grep "foo"

Far too often, if people want to find out whether a file contains a particular pattern, they'll cat the file piped into a grep command. This works, but grep can take a filename as an argument directly, so you can replace the above command with:

grep "foo" file

The next most common overuse of cat is when you want to sort the output from one or more files:

cat file1 file2 | sort | uniq

Like with grep, sort supports multiple files as arguments, so you can replace the above with:

sort file1 file2 | uniq

In general, every time you find yourself catting a file into a pipe, re-examine the piped command and see whether it can accept files directly as input first either as direct arguments or as STDIN redirection. For instance, both sort and grep can accept files as arguments as you saw earlier, but if they couldn't, you could achieve the same thing with redirection:

sort < file1 file2 | uniq
grep "foo" < file

Remove Files without xargs

The xargs command is very powerful on the command line—in particular, when piped to from the find command. Often you'll use the find command to pick out files that have a certain criteria. Once you have identified those files, you naturally want to pipe that output to some command to operate on them. What you'll eventually discover is that commands often have upper limits on the number of arguments they can accept.

So for instance, if you wanted to perform the somewhat dangerous operation of finding and removing all of the files under a directory that match a certain pattern (say, all mp3s), you might be tempted to do something like this:

find ./ -name "*.mp3" -type f -print0 | rm -f

Of course, you should never directly pipe a find command to remove. First, you should always pipe to echo to ensure that the files you are about to delete are the ones you want to delete:

find ./ -name "*.mp3" -type f -print0 | echo

If you have a lot of files that match the pattern, you'll probably get an error about the number of arguments on the command line, and this is where xargs normally comes in:

find ./ -name "*.mp3" -type f -print0 | xargs echo
find ./ -name "*.mp3" -type f -print0 | xargs rm -f

This is better, but if you want to delete files, you don't need to use a pipe at all. Instead, first just use the find command without a piped command to see what files would be deleted:

find ./ -name '*.mp3" -type f

Then take advantage of find's -delete argument to delete them without piping to another command:

find ./ -name '*.mp3" -type f -delete

So next time you find your pinky finger stretching for the pipe key, pause for a second and think about whether you can combine two commands into one. Your efficiency and poor overworked pinky finger (whoever thought it made sense for the pinky to have the heaviest workload on a keyboard?) will thank you.

[zeugma]

This is still one of my favorite lines in any of my scripts:

sed 's/#/ /g' $FILE | awk '{print $4}' | sort | uniq -ic | sort -nr | sed 's/://g' > $FILE.client

[Calvin Locke]

Back in the day, doing development on an 11/70 running Xenix, the prankster at the company would hit the new hires, changing their own various commands with his own.

So, the first time one would print off something during the day, it would get piped into "rev" first. The routine would rename itself, and schedule the replacement command again at midnight, for instance.

[ShadowAce]

Here's an informal poll:

Should I try to put together a very basic introduction/tutorial for the shell?

If "yes" do you have a suggestion for a source?

If "no" why not?

[LaRueLaDue]

“I am surprised they didn’t recommend running each command in its own Docker container !”

Precisely... lol!

[zeugma]

Shell stuff is always good. One of the biggest problems with newbies is they tend to be fairly mystified by it, even if they are somewhat familiar with the DOS prompt, because the commands are mostly different from what they expect. It is easy though, to go too far and scare folks with too much info at once.

I'd actually suggest starting with basics, kinda like what you had in the previous article concerning files, paths, and directories, though perhaps not to that detail. Maybe something that has a DOS command, and the BASH equivalent.

Need something on mounting and unmounting filesystems, even though most of that is done via GUI these days. The 'df' command is pretty critical. Even when using a GUI to mount/unmount a device, I always double-check with 'df' before pulling a drive. Might be worthwhile to go over a file manager and compare it to 'windows explorer'. Which one really doesn't matter that much, as they are all similar by virtue of function, though some have some extra features, such as Dolphin's ability to parse "fish://" URLs.

A document about processes would be helpful. (ps, top, htop, kill)

Perhaps a brief tutorial on how to write simple batch files.

Heck, I could write most of this, but I tend to go off on tangents and be distracted by shiny things.

[BraveMan]

What zeugma said . . .

I’m familiar with DOS command line stuff; Linux, not so much . . .

[Openurmind]

Honestly... I’m along for the ride to the drive-in with a bag of popcorn. While I like to learn everything I can, I have very little desire to actually see what’s under the hood. I have enough on my hands teaching myself to code websites.

As for an OS to get me to my server files, I just want an easy to use, stable GUI OS that always starts and runs, and the headlights and turn signals work. Like most, I only need to know how to drive it to get me there.

Online servers are where my interests are once I get there. I just need a stable reliable vehicle to get me there. Hence Linux. I consider myself among the greater percentage of the average noobs with minimal knowledge about how the turney thingy drives it down the road. lol

But maybe I will learn how to change the oil myself. Never turn down free knowledge. :)

[BitWielder1]

When I started to learn Unix, when one Unix guru taught me something, the next guru would inevitably come along and say, "Why do you do it THAT way? All you have to do is blablabla…"
And the new way was no simpler.

[notdownwidems]

Here's an informal poll: Should I try to put together a very basic introduction/tutorial for the shell?

Yes! I would especially like to know the differences between the shells and why, for instance, I would want to use, say the Korn shell rather than the C shell. Not sure about a source however. Thanks!~

[notdownwidems]

Thanks for that! Do grep or other pattern-matching commands have - delete as well?

[ShadowAce]

grep is a command that prints lines that contain a matching pattern.

It does not edit, change, insert, or delete.