Shell scripts

2.10. Shell scripts#

Although the shell looks like a place where you can just type in commands, itā€™s actually much more than that. Besides being a command line interface, itā€™s also a command language interpreter. Rather than the user typing commands one by one in the terminal, a shell process can also be used to read a sequence of commands from a simple text file and execute these in order (as if the user typed them) without intervention. Such files are called shell scripts, and they are very useful as they allow you to quickly automate things.

Linux systems often use shell scripts to manage its subsystems and ā€œglueā€ its different parts together, for example to configure and startup networking subsystems, update software packages, etc. But shell scripts are also a common tool for regular users, who want to simplify executing sequences of commands that they often rely on, for example to build some code and run some tests.

2.10.1. Creating a script#

A shell script is simply a plain text file with instructions that should be executed in a shell. Just as you would need to press Enter to execute a command in the command line interface, subsequent commands can be separated in the text file by placing them on new lines.

Using nano (see Section 2.5.3.1), create a new text file list containing just the following lines of text:

ls -l
echo LISTING DONE

You can also add comments to the file, since the shell ignores anything after # until the end of the line. So the following file content would functionally be the same:

# ----------------------
# This is a test script
# ----------------------

ls -l               # list the current directory
echo LISTING DONE   # inform user

Now there are several ways how we may want to execute this script:

  1. We might want to run this in an interactive shell, just as if we literally wrote the commands ourselves. This allows the script to affect the configuration of our current interactive shell. We call this sourcing the script.

  2. We might want to run it as a standalone process, which prevents the script from affecting the configuration of our current interactive shell. For this we have to execute the script.

We will explore both options below, which will also clarify what we mean with ā€œthe configurationā€ of the shell.

2.10.2. Source a script with source#

First, letā€™s take a look at executing the shell script in the current shell. This can be done with the source [FILENAME] instruction in bash. So, assuming your current working directory contains the list script, then you can simply run

$ source list

This should show you the output of ls -l and then the line LISTING DONE, just as if you would have typed these commands yourself in the terminal one by one.

Note

Sourcing can also be done with a special syntax in bash, namely by writing . instead of source as the command. So the source example above can also be written as:

$ . list

This notation can be confusing if you are not used to it, because it looks similar to other uses of . in the shell, such as representing the current directory (as in ls .) or as the beginning of a file name of a hidden file. In this manual we will prefer to use the more explicit source command, but it is good to be aware of this notation in case you encounter it in other documentation.

Since sourcing a file means all lines are executed in your current shell, without starting any new process, the script can access and set both regular and environment variables just as if you would have types the commands yourself. This makes sourcing suitable for scripts that are intended to setup one or more environment variables for later use.

Exercise 2.196

Make a new text file testvars, and use to print and set some regular and environment variables,

echo "Starting testvars"
echo "ENV_VAR     : $ENV_VAR"        # environment variable
echo "REGULAR_VAR : $REGULAR_VAR"    # non-environment variable

# update the variables
REGULAR_VAR="Updated $REGULAR_VAR"
export ENV_VAR="Updated $ENV_VAR"

Now run source testvars multiple times, what do you see in the output?

Afterwards, use echo and env to verify that sourcing the script has updated the variables in your current shell.

2.10.3. Running a script as a separate process#

Now leave the editor and enter in the terminal:

$ sh list

This tells the command line interpreter you are currently running (a bash process) to start another interpreter process (sh). By giving the sh interpreter the file name of the script list, this new process will open the file and execute its instructions.

If everything is correct, you should see in the terminal first the result of ls -l, displaying the content of the local working directory, and then the text LISTING DONE. Your script worked!

Once the script has no more instructions, the created sh process automatically terminates, and control is handed back to your interactive bash shell.

Exercise 2.197

Make a new script (give it a different filename, weā€™ll keep using the list script below), which when executed does the following:

  1. prints Hello world! to the terminal (Hint: use echo)

  2. prints You are here:

  3. prints the current working directory

Run the script to confirm that it works.

Exercise 2.198

What happens when we try to access regular and environment variables in a child process? Test this by executing the testvars script from Exercise 2.196 several times,

sh testvars

Also inspect the values of the variables with echo and env after executing the script again. Draw conclusions on the following:

  1. Can the exeuted script access all variables? Why (not)?

  2. Is the script able to update the variables in your shell? Why (not)?

Exercise 2.199

The sh shell used here is shell program like bash, but it is more basic. Although bash might not be installed on every Linux distribution, there should always be a basic shell called sh.

Letā€™s confirm that sh is a shell, by starting a new sh process in your interactive prompt

$ sh

You should see a new prompt, probably more basic and less colorful than the default bash prompt. Test that you can run simple commands in this shell just as in bash, e.g. ls, pwd, etc.

Next, run pstree to look at the process tree, and confirm that the sh process is a separate child process of the bash shell you were working in.

Once you are done, and want to stop the sh shell to return back to your bash shell, simply type

$ exit

which tells the current shell (sh) that it should quit. You should now see your normal bash prompt again.

Finally, confirm once more with pstree that the sh child process has terminated.

2.10.4. Making the script executable#

Executing a script can be done even easier, without the need to start a new shell process manually. Re-open the file list in nano and add #!/bin/sh as the very first line of the file. The file content should now look like this:

#!/bin/sh
ls -l
echo LISTING DONE

Save the changes to the list file and leave nano.

Next, we have to change the permissions on the list file such that the fileā€™s owner (you) can execute it. Remember you can use chmod to change file permissions:

$ chmod +x list

Once the file is marked as executable, you should be able to run the script directly, using ./list:

$ ./list

It should work just as before.

Exercise 2.200

Why do we need to mark the script as executable with chmod first?

Use chmod to remove the executable permission from the list script. Then, try to run the script again with ./list. What do you see in the terminal output?

Donā€™t forget to make the file executable again before continuing after this exercise.

So why did this work? When you try to execute a file, and bash notices that the file is a simple text file, it will look if the first line starts with #! (this character combination is called a hashbang or shebang). If so, if look take the remainder of that line and treat it as an interpreter to process the remainder of the file. In this case, bash will start the sh interpreter since /bin/sh is simply the full path to that interpreter program.

Note

The same mechanism can also be used turn scripts of other interpreted languages into executable files. For example, if our text file would contain Python code, you could add a first line #!/usr/bin/python3 (assuming Python 3 is installed in /usr/bin/python3) and mark the file as executable. Then, executing the python file in your shell would result in it automatically starting a Python process to run the code.

2.10.5. Handling optional arguments in scripts#

Ok, till now our list shell script provides a simple way to list files in the current working directory. But when we run the ls -l command in the terminal with an additional argument, like ls -l /usr/bin, then it would list the content of that directory (/usr/bin) instead of the current working directory. Letā€™s see if we can similarly give our script an optional argument to list a specific directory:

Exercise 2.201

Try listing the /usr/bin directory with your script:

$ ./list /usr/bin

Does it show the content of that directory? Why (not)?

To change this behaviour, we can use the arguments the user specifies within the shell script. These arguments are in the shell script available as variables called $1, $2, etc., where $1 is the first argument and so on. You can also get a list of all arguments using $*. Finally, the command name itself (here, list) is stored in $0.

Exercise 2.202

Re-write your shell script list to contain the following:

#!/bin/sh

ls -l $*
echo LISTING DONE

Now try again to use ./list to list the files in /usr/bin.

2.10.6. The PATH environment variable#

You probably noticed that weā€™ve been calling your shell script using ./list, not just list. This is necessary because the shell doesnā€™t automatically search the current working directory for commands: it is not in the path variable. The shell uses many variables to control its behaviour (which will be discussed in more detail in Section 2.9.1). The variable PATH contains a list of directories which the shell will search for programs (each directory is separated by a :). When you try to execute a file, your shell will search these directories in order to find the file name.

For example, if you would create your own ls script file and put it in a directory that occurs before /bin in the PATH variable, it will be used instead of the original system version of ls. Alternatively, you could update your PATH to include new directories, or to change their order.

Exercise 2.203

Look at your PATH shell variable using the env command (it can be helpful to search for PATH using a pipe to grep, see Section 2.7.4). How many directories are included in your PATH?

Exercise 2.204

Create a ~/bin directory and move your script there. The bin directory in your home will be automatically added to your PATH if it exists at the moment you login into your Ubuntu account.

This updating of the PATH variable is done by a file called .profile, as will be described in a moment in section Section 2.10.7. For now, force the shell reread the .profile file with the command source .profile, and then try running list again (not ./list). Now list should work.

Exercise 2.205

Now create a script cdl in ~/bin which:

  • changes the directory to the directory specified as an argument;

  • shows a list of files (including permissions) and pipes the output through less.

Exercise 2.206

Create an executable shell script ~/bin/aboutme that prints your username and home directory.

2.10.7. Startup files#

Till now, the scripts we looked at had to be explicitly sourced or executed by you. However, bash will also automatically source a few specific files for you, at specific moments, without you ever having to call source yourself. Such files are called startup files, and they are the standard place to set up things that should be available in every shell you use afterwards, such as custom environment variables, like your PATH, or aliases.

There are two startup files you will most commonly encounter, and they are read at different moments:

  • ~/.bashrc is automatically sourced every time you start a new interactive, non-login shell. This is the common case: it covers opening a new terminal window in most desktop Linux environments (including the browser-based terminal used throughout this manual), as well as starting a nested bash process from within an existing shell, e.g. by simply typing bash.

  • ~/.profile is automatically sourced only for a login shell. A login shell is the shell you would get immediately after logging into a computer, for example by logging into remote server or robot over ssh. Note that on your regular Ubuntu Linux desktop installation, you typically login via a graphical interface provided by the desktop environment, not via a terminal, so you will then not encounter a login shell directly. Nevertheless, on many Linux systems, like the default Ubuntu setup, the .profile will be sourced in the background when you perform the login via the graphical login manager. So, any exported variables will be available in any terminal you start afterwards. However, any changes to those exported variables will not be reflected until you fully logout of your account, and login again (or of course, when you restart your computer).

Tip

To avoid the two files getting out of sync, it is common for a ~/.profile file (including Ubuntuā€™s default one) to simply source the ~/.bashrc if it exists. This way, whether you start a login or a non-login shell, you still end up with the same, consistent setup that is defined in ~/.bashrc.

Since ~/.bashrc is commonly executed every time you open a new terminal with a new bash process, this is the conventional place to define things you want available in every interactive shell, such as aliases.

~/.profile, on the other hand, is more often used for things that only need to be set up once per login, such as extending your PATH. For example, if you inspect the content .profile file in your home directory on a real Ubuntu Linux system, you can see it automatically adds a $HOME/bin directory to your path (if it exists). This is the mechanism that was used in Exercise 2.204.

Exercise 2.207

In the terminal below, create a .bashrc file in your home directory (it does not exist yet). Add the following to this .bashrc

  • Setup an environment variable FOO, you can assign it any value.

  • Setup an alias ll, as a shorthand for ls -laF

  • Echo a statement ready, so that you can clearly see in the terminal when the file is (automatically) sourced.

You can create or edit the .bashrc file anyway you like (e.g. via echo + >>, nano, etc.).

Once the .bashrc is completed, start a new child shell process by simply typing bash. Verify that this child process will automatically source the .bashrc by checking the terminal output for the word ready. You can also check with env and alias that the startup file was executed.

{
  "checks": [
    { "type": "fileExists", "path": "/home/student/.bashrc",
      "desc": "should create ~/.bashrc" },
    { "type": "commandEvent", "eventType": "process.start", "match": { "command": "bash" },
      "desc": "should start a child bash process" },
    { "type": "varSet", "name": "FOO",
      "desc": "FOO should be set (and still be set once inside the child bash process)" },
    { "type": "varExported", "name": "FOO",
      "desc": "FOO should be exported, so the child process actually inherits it" },
    { "type": "commandEvent", "eventType": "builtin.alias", "match": { "name": "ll", "value": "ls -laF" },
      "desc": "should define an alias ll for ls -laF" },
    { "type": "commandEvent", "eventType": "builtin.echo", "match": { "text": "ready" },
      "desc": "the .bashrc should echo ready, and it should actually get sourced when bash starts" }
  ]
}

Exercise 2.208

Why does a .bashrc or .profile not start with a line #!/bin/sh ?

  • The OS will add this line automatically.

  • It is pointless, because these scripts are intended to be sourced, not executed.

  • The files have already been marked as executable (via chmod +x), so that line is not necessary.

  • Such a line is only needed for human readers to recognize shell scripts, but these files are well known.