2.7. Advanced shell commands

Now that you’ve learned how to run basic commands, it’s time to go a bit deeper into one of the basic elements of Unix : redirection and pipes.

2.7.1. Redirection

You will have noticed that most Unix programs, such as date, cal, find etc. have ASCII text as output. Normally this output, called the standard output, is written to the screen. However, using the > redirection symbol in the shell, it’s also possible to write the output to a file. For example:

$ date > the_date.txt

We will say the standard output of date has been redirected to the file the_date.txt.

Exercise 2.94

Try the redirection above, and verify with less that things worked well. Remove the file after you’re done.

In the example above, the file the_date.txt is overwritten. Sometimes, you might want to append text to a certain file. In that case, use the >> redirection symbol:

$ date >> date_list.txt

Standard output is one of the three standard files that most Unix commands work with. These three are:

• Standard input

• Standard error

• Standard output

Standard error is used to write error messages to. If you don’t specify anything, these will also be written to the screen. However, you can also redirect it to go together with the standard output, using >& (or &>> to append).

Exercise 2.95

Try date -d "yesterday" > date.txt and inspect the output. Now remove date.txt and try again with the >& notation. After you’re done, remove date.txt.

Standard input is a very useful feature. For example, you used the notation less myfile.txt to view the file myfile.txt. However, to feed myfile.txt to less as standard input, you can use the < redirection sign:

$ less < myfile.txt

Exercise 2.96

Of course, you can use both standard input and output at the same time, e.g.:

$ wc < myfile.txt > myfile.stats

Here we use the nifty wc utility that you encountered before. Verify the results in myfile.stats and remove the file.

2.7.2. Pipes

Fig. 2.9 Connecting programs with redirection.

While redirection is useful, it has a serious limitation, as illustrated in Fig. 2.9. If you want to apply a sequence of operations to a text file, you would have to create a number of intermediary files. For example, take the command wc -l, which counts the number of lines in a file. To find out how many people are logged on at the moment, you might do this:

$ who > tmp.txt
$ wc -l tmp.txt
$ rm tmp.txt

It works, but it’s a bit clumsy. To solve this problem, you can connect the standard output of one command directly to the standard input of another command using pipes. Pipes (symbol |) work in memory, so no intermediate files are created. Fig. 2.10 illustrates this

Fig. 2.10 Connecting programs with pipes.

The previous example becomes:

$ who | wc -l

or, if you would like to store the number of people logged on,

$ who | wc -l > count.txt

Some more examples:

• ls -l /usr/bin | less allows you to view the output of ls one screen at a time.

• ls -l /usr/bin | wc -l counts the number of files in /usr/bin.

• cat /etc/group | cut -d":" -f4 | grep "wcaarls" | wc -l displays the number of groups the user wcaarls is a member of (have a look at the man page of cut).

At some point in the pipe, you might want to “siphon off” the data to see intermediate results, or maybe store them. The command that allows you to do this is tee (for T-connection). If tee is called with the -a option, it will append rather than overwrite a file. For example:

$ cat /etc/passwd | cut -d":" -f1 | tee -a result.txt | grep "d$"`.

This line will append to an intermediate file called result.txt. What do you think this file will contain? Verify.

Pipes are what makes the Unix shell very versatile and flexible. We already discussed the Unix philosophy: make lots of tools, each of which does one thing and does it well. Pipes form the glue between these tools.

Finally, when discussing pipes, it’s useful to recall the cat command that you encountered before. It’s a bit more versatile than you may have thought:

• you can supply a number of file names, which cat will show, one after another. For example, try:

  $ cat myfile*txt
  

• if you don’t specify a file, cat will use standard input. This is useful for quickly creating small text files. Try:

  $ cat > short.txt
  

  You will notice that you don’t get a prompt. Instead, everything you type is stored in short.txt. To stop entering text, press Ctrl-D.

• the cat command is often used as the beginning of a pipe, for example:

  $ cat /etc/passwd | grep "false" | sort | less
  

  Try this line. What do you think it does?

2.7.3. Command substitution

As we have seen above, with a | pipe you can take the standard output of one program and provide it as standard input of another program. Sometimes, however, you want to take the output of a command, and use the result as arguments for a next command, as if you would type it in the shell.

Taking the output of one command, and use it as part of the next command is called command substitution, and Bash has a special syntax for this: $([COMMAND]). Here is a simple example:

$ echo "It was $(date) when I was learning about command substitution"

From the output, you should see that bash has substituted the $(date) part by the output of the date command command. The command within the brackets can be more complex than just running a single program, for example:

$ echo "The weekday tomorrow be $(date -d "tomorrow" | cut -d ' ' -f1)"

It is even possible to nest substitutions $( ), such as

$ echo My terminal is $(grep $(whoami) /etc/passwd | cut -d":" -f7)

When you study such a command to understand what it does, it may make sense to test the commands in order of execution, inspecting their output at each step. In this case, you can construct the full command from the individual commands as follows:

$ whoami
$ grep $(whoami) /etc/passwd
$ grep $(whoami) /etc/passwd | cut -d":" -f7
$ echo My terminal is $(grep $(whoami) /etc/passwd | cut -d":" -f7)

Exercise 2.97

What do you think the following command will do?

$ wc -l $(find . -name "*.txt")

Try it out.

You can use the output of a command as a parameter by using the “backtick notation”, i.e. using the single backward quotes.

Note

There is also an alternative syntax for command substitution using backticks to demarcate the substitution. Here are some examples that do the same as what you saw before:

$ echo "It was `date` when I was learning about command substitution"
$ wc -l `find . -name "*.txt"`

You still find this backtick notation sometimes in scripts and documentation, but it is considered deprecated. One benefit of $( ) over backticks as that this recommended notation can be nested, as we saw before. It is also easier to spot in a command than the more subtle backtick characters.

2.7.4. The command line interface

You’ve already used the command line interface quite a bit, so here are simply some remarks:

• If you specify arguments to commands, protect them using quotes. Protection means that you do not want the shell to start interpreting meta-characters. There are two types of quotes. The double quotes (e.g. grep "some thing" file) just protect against simple meta-characters, such as space. The single quotes protect quite a bit more. For example, try the difference between:

  $ grep "!" myfile.txt
  

  and

  $ grep '!' myfile.txt
  

  (the ! metacharacter has to do with the command history; you will learn about the history below).

• Even with single quotes, the shell still tries to interpret some meta-characters. You will have to “escape” these by prepending a \.

• If you want to run more than one command, but there is no need for pipes, use a semi-colon. For example,

  $ cd
  $ ls
  

  is the same as:

  $ cd; ls
  

• If you want to group two commands together, use parentheses. For example, try the difference between:

  $ (cd /bin; pwd); ls
  

  and

  $ cd /bin; pwd; ls
  

• Most shells feature command completion. In bash, for example, you just enter part of a file name and press the Tab key, the shell will find the full filename. If there is more than one file that starts with the same character(s), it will show a list of matches.

Note that the shell tries to find out when you’re looking for a program or a general file. For example, it will not complete myfil, but it will complete cat myfil to cat myfile.txt. It even knows how to complete the options of many programs!

2.7.5. History

A useful part of any shell is the history. The shell keeps a list of all lines you typed in since the shell was started. Try it:

$ history

The easiest way of using the history is moving through it using Up and Down. You can also repeat specific lines by using the ! shell built-in command, like this:

• !! will repeat the last line.

• !13 will repeat line 13.

• !-5 will repeat the line you entered 5 lines ago.

• !mo will repeat the last command you entered that started with the string mo.

A helpful shortcut is Ctrl-R, which allows you to search in the history. Just start typing the beginning of a command, and it will complete to the last command that matches. Hitting Ctrl-R again cycles through all matches.

Exercise 2.98

Play around with the history commands.

2.7.6. Aliases

A very useful property of the shell is aliasing. This means that you can create your own “short hand” notation for commands you often use. An alias has the form

$ alias name='command'

in which name is the short hand you want to give the command.

To get an overview of all active aliases, you can just enter alias without arguments. To get rid of aliases, use unalias.

Exercise 2.99

Create an alias for find . -name; call it f and try it using f "*.txt". Also, put this line in your .profile file, so you can use it whenever you log on.