Advanced shell commands

2.7. Advanced shell commands#

Now that you’ve learned how to run basic commands, it’s time to go a bit deeper into one of the basic elements of Unix : redirection and pipes.

You will have noticed that most Unix programs, such as date, cal, find etc. have ASCII text as output. Normally this output, called the standard output, is written to the screen.

Standard output is one of the three standard streams that most Unix commands work with. These three are:

Standard input (a.k.a. stdin): used by programs to read input from.
Standard output (stdout): used by programs to write the regular output to.
Standard error (stderr): used by programs to write error messages to.

By default, when you run a program in the shell, the input and output streams are connected to the terminal. So, any output or errors generated by the program will be shown in the terminal, as you have seen in all the previous exercises of this manual. Many text processing programs that you’ve seen, such as grep, cat, wc, etc., can also read input from an input stream if they are not given a file path as argument to work on.

To illustrate the most basic setup of a program taking input from stdin, doing some processing, and producing output on stdout, we can take a look at cat. In its simplest form this command reads all input from stdin and does no processing, but simply copies the input to stdout.

Exercise 2.146

Run plain cat, with no filename argument at all. Type a line of text, then press Ctrl-D to signal the end of your input.

Without a filename, cat reads from stdin instead – in this case, whatever you type at the terminal. Notice that cat still writes to stdout exactly as it would with a file argument: since nothing here is redirected, stdout is still connected to the terminal screen, so cat simply repeats back the text you typed.

(In this browser-based terminal, what you have typed will only appear after you press Ctrl-D. In a real terminal, cat processes the input line by line, and it would repeat each of your lines immediately after you pressed Enter. More on that later.)

{
  "checks": [
    { "type": "commandSucceeded", "pattern": "^cat\\s*$",
      "desc": "should run cat with no filename argument at all" },
    { "type": "commandEvent", "eventType": "coreutil.cat",
      "desc": "should use cat" }
  ]
}

You’ve probably already seen commands generate error messages before throughout this manual and its exercises. Nevertheless, for completeness, let’s also use cat to illustrate how messages appear on the stderr error stream.

Exercise 2.147

First run ls to verify your home directory is empty. Then run cat nofile.txt, a file that doesn’t exist.

cat generates an error message that the file doesn’t exist, written to the stderr stream. Notice that this error is still shown on the terminal, even though it is not the content of nofile.txt that we would expect from cat on stdout. Of course, stderr is by default also connected to the terminal.

{
  "filesystem": { },
  "checks": [
    { "type": "commandEvent", "eventType": "coreutil.ls",
      "desc": "should first run ls to check the directory is empty" },
    { "type": "commandEvent", "eventType": "coreutil.cat", "match": { "absPaths": ["/home/student/nofile.txt"] },
      "desc": "should try to cat nofile.txt" },
    { "type": "stderrContains", "text": "No such file or directory", "pattern": "^cat\\s+nofile\\.txt",
      "desc": "cat nofile.txt should print an error to stderr, since the file doesn't exist" }
  ]
}

2.7.1. Redirecting standard output#

Sometimes you don’t want to see the output of a command in the terminal, but you want to save it in a file. You could try to select the terminal output with the mouse, copy it, and then paste it into an opened text editor. However, the bash shell provides a much easier method using the > redirection symbol after a command, combined with a target file name. This instructs bash to save the text from the command’s standard output stream in the given file path. For example:

$ date > the_date.txt

We will say the standard output of date has been redirected to the file the_date.txt.

Exercise 2.148

Try the to redirect the output of date into a file called the_date.txt, and verify with cat that this file was indeed created and contains the output of the date command.

{
  "checks": [
    { "type": "commandEvent", "eventType": "coreutil.date",
      "desc": "should run date" },
    { "type": "commandEvent", "eventType": "redirect.write", "match": { "absPath": "/home/student/the_date.txt" },
      "desc": "should redirect the output of date into the_date.txt with >" },
    { "type": "commandEvent", "eventType": "coreutil.cat", "match": { "absPaths": ["/home/student/the_date.txt"] },
      "desc": "should inspect the_date.txt with cat" }
  ]
}

In the example above with the > redirection, if file the_date.txt already exists, its content would be completely replaced by the output of the date command. Sometimes, you instead want to append text to a pre-existing file, extending the file’s existing content. In that case, use the >> redirection symbol. For example, the following would ensure the file journal.txt has two lines:

$ echo The current date is: > journal.txt
$ date >> journal.txt

Exercise 2.149

Run date >> date_list.txt. Then repeat the exact same command two more times, but instead of retyping it, use !! to re-run the previous command. Afterwards, date_list.txt should contain three lines, one per invocation. Afterwards, inspect date_list.txt with cat to confirm it contains the content of multiple lines.

{
  "checks": [
    { "type": "commandEvent", "eventType": "history.expansion",
      "desc": "should use !! to re-run the previous command, instead of retyping it" },
    { "type": "commandEvent", "eventType": "redirect.append", "match": { "absPath": "/home/student/date_list.txt" },
      "desc": "should append into date_list.txt with >>, not overwrite it with >" },
    { "type": "fileMatches", "path": "/home/student/date_list.txt", "pattern": "^(.+\\n){3}",
      "desc": "date_list.txt should end up with (at least) three lines, one per date invocation" },
    { "type": "commandEvent", "eventType": "coreutil.cat", "match": { "absPaths": ["/home/student/date_list.txt"] },
      "desc": "should inspect date_list.txt with cat" }
  ]
}

Exercise 2.150

Suppose date_list.txt does not exist yet. What happens if you run date >> date_list.txt?

It fails, since >> requires the file to already exist.
It creates date_list.txt and writes the date into it. For a brand new file, >> behaves exactly like >.
Nothing happens, since there is nothing to append to yet.
It creates an empty date_list.txt, but the date output is discarded.

Exercise 2.151

Run the following three commands, one after another:

$ echo "first line" > README.txt
$ echo "second line" >> README.txt
$ echo "third line" > README.txt

Before you run the last one, guess what README.txt will contain afterwards. Then check it with less or cat.

{
  "checks": [
    { "type": "commandEvent", "eventType": "redirect.write", "match": { "absPath": "/home/student/README.txt" },
      "desc": "should use > at least once (creating, then later overwriting, README.txt)" },
    { "type": "commandEvent", "eventType": "redirect.append", "match": { "absPath": "/home/student/README.txt" },
      "desc": "should use >> at least once (appending to README.txt)" },
    { "type": "fileContains", "path": "/home/student/README.txt", "text": "third line",
      "desc": "README.txt should contain the last line written with >" },
    { "type": "fileContains", "path": "/home/student/README.txt", "text": "first line", "negate": true,
      "desc": "the final > should have erased the earlier content, including \"first line\"" },
    { "type": "fileContains", "path": "/home/student/README.txt", "text": "second line", "negate": true,
      "desc": "the final > should have erased the earlier content, including \"second line\"" }
  ]
}

Exercise 2.152

The file notes.txt already contains some old text. Using only echo and redirection (no mv, rm, or nano) overwrite its content so it ends up containing exactly three lines, in this order: line one, line two, line three. None of the original content should remain. You’ll need to combine > (to start fresh) and >> (to add the rest).

{
  "filesystem": { "/home/student/notes.txt": "This is some old junk text.\nDelete me please.\n" },
  "checks": [
    { "type": "commandEvent", "eventType": "redirect.write", "match": { "absPath": "/home/student/notes.txt" },
      "desc": "should use > at least once, to overwrite the old content" },
    { "type": "commandEvent", "eventType": "redirect.append", "match": { "absPath": "/home/student/notes.txt" },
      "desc": "should use >> at least once, to add the remaining lines" },
    { "type": "commandRan", "pattern": "^mv\\b", "negate": true,
      "desc": "should not use mv" },
    { "type": "commandRan", "pattern": "^rm\\b", "negate": true,
      "desc": "should not use rm" },
    { "type": "commandRan", "pattern": "^nano\\b", "negate": true,
      "desc": "should not use nano" },
    { "type": "fileMatches", "path": "/home/student/notes.txt", "pattern": "^line one\\nline two\\nline three\\n$",
      "desc": "notes.txt should contain exactly the three given lines, in order, with none of the original content left" }
  ]
}

2.7.2. Redirecting standard error#

If you don’t specify anything, any error messages on the standard output stream will also be written to the screen, even if you redirect the standard output somewhere else. This is intentional: if you want to capture the output of a program in a file, then you often do not want any error or warning messages intermixed with the output.

However, there are also cases where you do want to redirect the standard error to a file, for example when you want to errors into a separate error log file. In such cases, you can also instruct bash to redirect the standard error to a file using 2> (the “2” here refers to the 2nd output stream, i.e. the error stream. For example, to capture any errors generated by the date command:

$ date -d "ysterday" 2> error_log.txt

(the misspelling of “yesterday” in this simple example is intentional to make date generate an error. As it doesn’t recognize the word ysterday, it prints an error message to stderr instead of the actual date to stdout. This way, you can actually observe whether that error message ends up in error_log.txt or not.)

Exercise 2.153

Try date -d "ysterday" > date.txt (yes, still misspelled on purpose) and inspect date.txt. Notice it is empty, since the error message went to the screen, not to the file. Now try again, redirecting standard error into the same file using 2> instead (remember you can press Up to easily adjust the command without retyping everything). This time date.txt should actually contain the error message, verify this by inspect it with cat.

{
  "filesystem": { "/home/student/stuff/todo.txt": "buy milk\n" },
  "checks": [
    { "type": "commandEvent", "eventType": "redirect.write",
      "match": { "absPath": "/home/student/date.txt", "mergeStderr": false },
      "desc": "the first attempt (plain >, no 2>) should leave date.txt without the error message" },
    { "type": "commandRan", "pattern": "2>\\s*date\\.txt",
      "desc": "should also try the stderr redirection with 2>" },
    { "type": "commandEvent", "eventType": "redirect.write",
      "match": { "absPath": "/home/student/date.txt", "stream": "stderr" },
      "desc": "the 2> attempt should redirect the error message into date.txt" },
    { "type": "commandEvent", "eventType": "coreutil.cat", "match": { "absPaths": ["/home/student/date.txt"] },
    "desc": "should inspect date.txt with cat" }
  ]
}

Sometimes a command is expected to produce a lot of warnings and errors on stderr which can be safely ignored. To avoid all these messages cluttering the output, it is common to redirect the error stream to a special file of the Linux operating system, called /dev/null. You can write data to it as in any other file, but any data written to it will be completely discarded by the operating system.

For example, searching for a file with find in the filesystem root directory for directories with the same name as the user can produce a lot of warnings on stderr that some directories can not be accessed. An easy way to ignore the stderr messages, and still see the regular matches on stdout, would then be:

$ find / -name $USER 2> /dev/null

Exercise 2.154

Try this in a terminal on your real Ubuntu Linux computer. Search the filesystem for files or directories called opt, starting in the filesystem root. Try to see the differnce with and without appending 2> /dev/null to ignore errors from stderr.

Which of the following opt directories (among others) do exist in your Ubuntu Linux system?

Finally, perhaps you want to capture errors together with the standard output into the same file, just like they would appear in your terminal without redirecting. The portable way to do this, which works the same in every shell, is to redirect standard output as usual with >, and afterwards instruct bash to redirect standard error to “wherever standard output currently goes” using 2>&1:

$ date -d "ysterday" > date.txt 2>&1

Bash also offers the shorthand >& for the same thing, but this only works in bash itself, not in every shell.

Note: in this manual’s browser-based terminal emulator, the 2>&1 and >& notation do not work, but you can try it yourself on a real terminal.

2.7.3. Redirecting standard input#

Standard input can also be redirected to get its content from a file. For example, you used the notation less myfile.txt to view the file myfile.txt, providing less with the filename as an argument. However, to feed myfile.txt to less as standard input, you can use the < redirection sign:

$ less < myfile.txt

This makes the shell read the content of the file after <, and provide that text as input on stdin for the program before the <. So, the command above will send the file content of myfile.txt directly to less, whereas less myfile.txt relies on the less command to interpret its argument as a file path that it should read itself. For this simple example, both variations will have basically the same result.

Exercise 2.155

Of course, you can use both standard input and output at the same time, e.g.:

$ wc < myfile.txt > myfile.stats

Here we use the nifty wc utility that you encountered before. Verify the results in myfile.stats with cat.

{
  "filesystem": { "/home/student/myfile.txt": "hello world\nthis is myfile\n" },
  "checks": [
    { "type": "commandEvent", "eventType": "redirect.read", "match": { "absPath": "/home/student/myfile.txt" },
      "desc": "should feed myfile.txt into wc using <" },
    { "type": "commandEvent", "eventType": "coreutil.wc",
      "desc": "should use wc" },
    { "type": "commandEvent", "eventType": "redirect.write", "match": { "absPath": "/home/student/myfile.stats" },
      "desc": "should redirect the wc output into myfile.stats with >" },
    { "type": "fileMatches", "path": "/home/student/myfile.stats", "pattern": "2\\s+5\\s+27",
      "desc": "myfile.stats should contain the line/word/byte counts of myfile.txt" },
    { "type": "commandEvent", "eventType": "coreutil.cat", "match": { "absPaths": ["/home/student/myfile.stats"] },
    "desc": "should cat myfile.stats" },
    { "type": "tabCompletionHint" }
  ]
}

Exercise 2.156

Running wc myfile.txt and wc < myfile.txt produce nearly the same output, except for a subtle difference. What is that difference?

wc myfile.txt reports the filename in the output; wc < myfile.txt does not as it does not see the filename.
wc myfile.txt by default reports the number of lines, unlike wc < myfile.txt.
wc < myfile.txt prints its output to stderr, while wc myfile.txt use stdout.
wc < myfile.txt will not produce an error message if myfile.txt does not exist, unlike wc myfile.txt.

2.7.5. Command substitution#

As we have seen above, with a | pipe you can take the standard output of one program and provide it as standard input of another program. Sometimes, however, you want to take the output of a command, and use the result as arguments for a next command, as if you would type it in the shell.

Taking the output of one command, and use it as part of the next command is called command substitution, and Bash has a special syntax for this: $([COMMAND]). Here is a simple example:

$ echo "It was $(date) when I was learning about command substitution"

From the output, you should see that bash has substituted the $(date) part by the output of the date command command. The command within the brackets can be more complex than just running a single program, for example:

$ echo "The weekday tomorrow be $(date -d "tomorrow" | cut -d ' ' -f1)"

It is even possible to nest substitutions $( ), such as

$ echo My terminal is $(grep $(whoami) /etc/passwd | cut -d":" -f7)

When you study such a command to understand what it does, it may make sense to test the commands in order of execution, inspecting their output at each step. In this case, you can construct the full command from the individual commands as follows:

$ whoami
$ grep $(whoami) /etc/passwd
$ grep $(whoami) /etc/passwd | cut -d":" -f7
$ echo My terminal is $(grep $(whoami) /etc/passwd | cut -d":" -f7)

Exercise 2.165

The goal of this exercise is to run the word count command wc on all .txt files that exist in the home directory, or any of its subdirectories. To do this, first run find as a standalone command to see in the terminal the relative paths of all .txt files:

$ find . -name "*.txt"

Now, let wc count the number of lines -l of the files, by using command substitution with this find command to list all file paths as arguments for wc.

{
  "filesystem": {
    "/home/student/a.txt": "one\ntwo\n",
    "/home/student/notes/b.txt": "uno\ndos\ntres\n",
    "/home/student/projects/ideas/project.md": "# My new project\nSee README.txt\n",
    "/home/student/projects/ideas/README.txt": "still planning\n",
    "/home/student/readme.md": "not a txt file\n"
  },
  "checks": [
    { "type": "commandEvent", "eventType": "coreutil.find", "match": { "namePattern": "*.txt" },
      "desc": "use find to identify all .txt files" },
    { "type": "commandEvent", "eventType": "coreutil.wc", "match": {"flags": ["l"]},
      "desc": "should use wc -l" },
    { "type": "commandEvent", "eventType": "syntax.cmdsubst",
      "desc": "should use $(...) command substitution" },
    { "type": "compareOutput", "referenceCommand": "wc -l $(find . -name \"*.txt\")",
      "studentPattern": "^wc\\s+-l\\s+\\$\\(find",
      "desc": "the printed line counts should list the number of lines all txt files" }
  ]
}

There is also an alternative syntax for command substitution using backticks to demarcate the substitution. Here are some examples that do the same as what you saw before:

$ echo "It was `date` when I was learning about command substitution"
$ wc -l `find . -name "*.txt"`

Note

You still sometimes find this backtick notation in older scripts and documentation, but it is considered deprecated.

Exercise 2.166

Why is $( ) generally preferred over the backtick notation for command substitution?

Backticks are not supported in bash, only in sh.
$( ) can be nested directly (e.g. $(cmd1 $(cmd2))), while nesting backticks requires escaping the inner ones, and $( ) is visually easier to spot in a command than backticks.
$( ) runs faster, because it avoids starting a subshell.
Backticks can only be used with echo, not any other command.
People like to type more.

2.7.6. Multiple commands on one line#

You can use a semi-colon to provide the shell with multiple commands that should be executed one after the other, without intervention. Unlike using pipes, with ; the shell does not execute the commands in parallel but one after the other, and no output is redirected.

For example, instead of this:

$ sleep 3
$ echo ready

You can achieve the same as a one-liner as follows:

$ sleep 3; echo done

The difference is that with the one-liner you don’t have to wait for the sleep 3 command to finish (after 3 seconds) to enter the next echo done command. The shell will just immediately start the next command itself once the sleep 3 command terminated.

You can also group multiple commands together using parentheses. The parentheses act like a temporary subshell, where you can change directories without affecting the working directory afterwards.

For example, try the difference between:

$ cd ~
$ (cd /bin; pwd); pwd

and

$ cd ~
$ cd /bin; pwd; pwd

Note that the final pwd still shows your working directory is your home directory in the first case, because the cd /bin was executed within the parentheses, and thus only changed the working directory for the first pwd command.

Exercise 2.167

There is a directory stuff/ in your home directory containing a file todo.txt. Then run (cd stuff; pwd); ls. Does the final ls show the contents of your home directory, or the stuff/ subdirectory?

{
  "filesystem": { "/home/student/stuff/todo.txt": "buy milk\n" },
  "checks": [
    { "type": "commandEvent", "eventType": "syntax.subshell",
      "desc": "should run (cd stuff; pwd) as a subshell" },
    { "type": "commandEvent", "eventType": "builtin.cd", "match": { "absPath": "/home/student/stuff" },
      "desc": "should cd into stuff/ inside the subshell" },
    { "type": "stdoutContains", "text": "/home/student/stuff", "pattern": "^\\(cd\\s+stuff",
      "desc": "pwd inside the subshell should report /home/student/stuff" },
    { "type": "stdoutContains", "text": "stuff", "pattern": "^\\(cd\\s+stuff.*\\)\\s*;\\s*ls",
      "desc": "the ls after the subshell should still list your home directory (showing the stuff/ folder itself), since cd inside () didn't affect the parent shell" }
  ]
}

Exercise 2.168

After running (cd stuff; pwd); ls, what is the current working directory of your shell?

stuff/, since cd inside the parentheses changed it.
Still your home directory, since (...) runs the cd inside a subshell which doesn’t affect the parent shell.
It depends on whether stuff/ exists.
/, since parentheses reset the shell to the root directory.

2.7.7. Aliases#

Another useful property of the shell is aliasing. This means that you can create your own “short hand” notation for commands you often use, saving you more typing work. An alias is setup in bash using a syntax as follows:

$ alias name='command'

in which name is the short hand you want to give the command. For example, ll is a common alias to quickly run ls with additional switches for detailed listings of directory content:

$ alias ll='ls -alF'

The alias can be executed without with additional arguments:

$ ll            # like running ls -alF
$ ll *.txt     # like running ls -alF *.txt
$ ll /usr/bin  # like running ls -alF /usr/bin

To get an overview of all active aliases, you can just enter alias without arguments.

Exercise 2.169

Create an alias for find . -name; call it f and try it using f "*.txt".

{
  "filesystem": {
    "/home/student/notes.txt": "todo\n",
    "/home/student/readme.md": "not txt\n"
  },
  "checks": [
    { "type": "commandRan", "pattern": "^alias\\s+f=",
      "desc": "should create an alias called f for find . -name" },
    { "type": "commandEvent", "eventType": "coreutil.find", "match": { "namePattern": "*.txt" },
      "desc": "using f \"*.txt\" should actually run find . -name \"*.txt\"" },
    { "type": "stdoutContains", "text": "notes.txt",
      "desc": "f \"*.txt\" should find notes.txt" }
  ]
}

Exercise 2.170

Try this on your real Ubuntu Linux computer. Open two terminals.

In one of the terminals, define an alias, and confirm that it works in that terminal.

Now check in the other terminal, which is running a separate bash process, if your custom alias is also available there.

Now close all terminals, and open a new one. Again check if the alias is available.

Aliases only exist for the shell where they were created in. They do not persist and are not shared between sessions.
Aliases only exist as long as the shell where they were created in is active. While active, the alias is available in other terminals too.
Setting an alias permanently changes your account settings; it is immediately available in all other existing and later created terminals.
Setting an alias permanently changes your account settings, but it will only be available in new shell processes started after setting the alias.

As you may notice, if you want an aliases to reliably persist, it will need to be defined in each bash processes you start separately. Rather than retyping the alias every time, a more convenient approach is to define your aliases in a startup file like ~/.bashrc, which will be executed automatically every time you start a bash process in a new terminal. This will be discussed later in Section 2.10.7.

Advanced shell commands

Contents

2.7. Advanced shell commands#

2.7.1. Redirecting standard output#

2.7.2. Redirecting standard error#

2.7.3. Redirecting standard input#

2.7.4. Pipes#

2.7.5. Command substitution#

2.7.6. Multiple commands on one line#

2.7.7. Aliases#