Searching files

2.5.2. Searching files#

Now that you know how you can inspect files in a directory and what the output of ls -l means, it’s time to learn how to search for specific files. There are a large number of files on a typical Unix system, so finding that one file you want can be hard.

Here we present to general strategies:

  • Searching for particular files by their name, extension, or some other properties.

  • Searching in files, to find files that contain some specific text.

We’ll discuss useful commands for each strategy separately.

2.5.2.1. Searching for particular files#

If you know what file or what kind of file you are looking for, there are various useful commands that you could use:

  • ls -R: using ls with the -R (for recursive) option will show you all files in all subdirectories of the directory you’re listing. This can quickly give a lot of information!

    Exercise 2.84

    Try this in the root directory:

    $ ls -R /

    What do you think ls -lR will do? Try it.

  • which: the output of which program is the full path to the program program. Use this if you want to find out which version of a certain program you’re actually using.

  • whereis: does more or less the same as which, but it only looks for programs in the standard places where Unix stores its programs. It will not find programs you made yourself. When it finds a program, it also tries to print the location of the man page.

    Exercise 2.85

    Use which and whereis to find out where the ls, which and whereis commands are located.

  • find: the Swiss army knife of search tools. This command has a lot of options. The most often used one is -name, for example:

    $ find -name test.me    # list all files under working directory called `test.me`
$ find / -name "*.jpg"  # all files under the root directory that end with `.jpg` (Note: use quotes because of *)

    This instructs find to look at all file and directory names in the root directory /, or any of its subdirectories (so it searches recursively through the directory tree, similar to ls -R), and reports only the names which end in .jpg. If you want to exclude the possibility that find reports on any directory name that ends with .jpg (which is uncommon, but possible), then you could explicitly specify to only report on files with the -type f argument.

    Exercise 2.86

    Find all files below /usr/share which end in .txt. Also find all files below /etc.

    Exercise 2.87

    Using find’s -name option,

    1. list all files called exactly README in the root directory /

    2. list all the files whose name contains with the string report (this should match names like old_report.txt, report.bin, etc.) in the root directory /. Hint: use quotes when using * patterns as command arguments.

    {
  "filesystem": {
    "/opt/science_project/logo.dat": { "base64": "iVBORw0KGgoAAAAN" },
    "/opt/science_project/report.bin": "%PDF-1.4\n%%EOF\n",
    "/opt/science_project/deploy": { "content": "#!/bin/bash\necho deploying\n", "mode": "755" },
    "/opt/science_project/README": {"content": "Events will be logged in engine.dat", "mtime": "2021-03-04T05:07"},
    "/opt/science_project/engine.dat": {"content": "", "mtime": "2021-03-04T05:07"},
    "/home/student/vacation.txt": { "base64": "/9j/4AAQSkZJRgAB" },
    "/home/student/.thrash/report_placeholder": "Lorem ipsum",
    "/home/student/.thrash/README": "My first project",
    "/home/student/License.txt": "None yet\n",
    "/home/student/old_report.txt": { "content": "Q1 numbers, ancient history by now.\n", "mtime": "2023-01-01T00:00:00Z" },
    "/home/student/new_report.txt": "Q4 numbers, fresh off the press.\n"
  },
  "checks": [
    { "type": "commandEvent", "eventType": "coreutil.find",
      "match": { "absPaths": ["/"], "namePattern": "README" },
      "desc": "should run find on the root directory, using `-name` to list all files called `README`" },
    { "type": "commandEvent", "eventType": "coreutil.find",
      "match": { "absPaths": ["/"], "namePattern": "*report*" },
      "desc": "should run find on the root directory, using `-name` to list all files with `report` in the filename" }
  ]
}

    Other useful options of find allow you to just select files newer or older than a certain date, files of certain types, sizes, files with certain permissions, files belonging to a certain user or group, etc. See man find for the complete overview. For example:

    $ find . -size +80k -ctime +100

    Recursively finds all files in the current directory larger than 80 kilobytes, and which are older than 100 days.

    Note

    This last example uses -ctime, and find also supports combining searches with -exec (see below). Both are too involved to fully support in this book’s in-browser terminal. If you try them here, find will tell you so and point you to a real terminal instead.

    Exercise 2.88

    1. Use find to list only the directories under /.

    2. Use find to list files larger than 30 kilobytes in /usr/share.

    3. Use find to list all empty files under the home directory.

    4. Use find to list files under the home directory that were modified more than 20 days ago.

    5. Use find to list empty files under the home directory that were modified more than 1 year ago.

    {
  "filesystemZip": "bash-exercise-fs/fs-usr-share-common-licenses.zip",
  "filesystem": {
    "/home/student/vacation.txt": { "base64": "/9j/4AAQSkZJRgAB" },
    "/home/student/projects/logo.dat": { "base64": "iVBORw0KGgoAAAAN" },
    "/home/student/projects/report.bin": "%PDF-1.4\n%%EOF\n",
    "/home/student/projects/deploy": { "content": "#!/bin/bash\necho deploying\n", "mode": "755" },
    "/home/student/projects/README": {"content": "Events will be logged in engine.dat", "mtime": "2021-03-04T05:07"},
    "/home/student/projects/engine.dat": {"content": "", "mtime": "2021-03-04T05:07"},
    "/home/student/placeholder": "",
    "/home/student/License.txt": "None yet\n",
    "/home/student/old_report.txt": { "content": "Q1 numbers, ancient history by now.\n", "mtime": "2023-01-01T00:00:00Z" },
    "/home/student/new_report.txt": "Q4 numbers, fresh off the press.\n"
  },
  "checks": [
    { "type": "commandEvent", "eventType": "coreutil.find",
      "match": { "absPaths": ["/"], "typeFilter": "d" },
      "desc": "should use find to list directories under /" },
    { "type": "commandEvent", "eventType": "coreutil.find",
      "match": { "absPaths": ["/usr/share"], "sizeSpec": "+30k" },
      "desc": "should use find to list files larger than 30 kilobytes under /usr/share" },
    { "type": "commandEvent", "eventType": "coreutil.find",
      "match": { "absPaths": ["/home/student"], "empty": true },
      "desc": "should use find to list empty files in the home directory" },
    { "type": "commandEvent", "eventType": "coreutil.find",
      "match": { "absPaths": ["/home/student"], "mtimeSpec": "+20" },
      "desc": "should use find to list files modified more than 20 days ago in the home directory" },
    { "type": "commandEvent", "eventType": "coreutil.find",
      "match": { "absPaths": ["/home/student"], "mtimeSpec": "+365", "empty": true },
      "desc": "should use find to list empty files modified more than 365 days ago in the home directory" }
  ]
}

    Warning

    find also supports a -delete action, which removes every matched file or directory. On a real system this is permanent, and there is no undo. Before you ever add -delete to a find command, always run the exact same command without -delete first, and carefully check the list of matches. Only add -delete once you’re sure that list is exactly what you want removed, and nothing more.

    This sandbox is a safe place to get comfortable with -delete, since nothing here is real. Resetting the exercise restores everything.

    Exercise 2.89

    Using find and the -delete action, remove all empty files under your home directory. Don’t delete projects/ itself, or any file that actually has content.

    {
  "filesystem": {
    "/home/student/placeholder": "",
    "/home/student/projects/old_version/engine.dat": "",
    "/home/student/projects/README": "Read this first.\n",
    "/home/student/notes.txt": "Meeting notes.\n"
  },
  "checks": [
    { "type": "commandEvent", "eventType": "coreutil.find",
      "match": { "absPaths": ["/home/student"], "empty": true, "delete": false },
      "desc": "first use find -empty in the home directory without -delete, to list relevant files" },
    { "type": "commandEvent", "eventType": "coreutil.find",
      "match": { "absPaths": ["/home/student"], "empty": true, "delete": true },
      "desc": "should use find with -delete in the home directory to remove the empty files" },
    { "type": "fileExists", "path": "/home/student/placeholder", "negate": true,
      "desc": "placeholder should be deleted" },
    { "type": "fileExists", "path": "/home/student/projects/old_version/engine.dat", "negate": true,
      "desc": "projects/old_version/engine.dat should be deleted" },
    { "type": "fileExists", "path": "/home/student/projects/README",
      "desc": "projects/README should not be deleted" },
    { "type": "fileExists", "path": "/home/student/notes.txt",
      "desc": "notes.txt should not be deleted" },
    { "type": "dirExists", "path": "/home/student/projects",
      "desc": "the projects directory itself should still exist" }
  ]
}

Note: locate is another tool for finding a file regardless of where it is on the system. Instead of actually searching the disk, it consults a database that is updated daily. This is much faster then find on the root directory, but it won’t show recently added files that are not indexed in its database yet. Ubuntu nowadays does not come with locate installed by default, but you could install it yourself with

sudo apt install plocate

The locate database would be rebuild regularly in the background, but if you want to force building the database immediately, you would need to run sudo updatedb. Afterwards, you can rapidly find any name x on your filesystem with locate x. This can be useful on a powerful desktop or laptop where you often use locate, but you would not want such a database update to occur regularly on a robot with limited computing power.

2.5.2.2. Identifying what a file is#

Just by the name, it’s usually very hard to judge what a file contains. It might be a program, some text, some program source code, a figure, an image, a sound etc. Often, the extension in a filename (like jpg in DSC0005.jpg, or txt in README.txt) is a good indication that a file is an image (extensions jpg, jpeg, png, tiff, etc.), or text (txt for plain text, md for markdown, etc.), or something else. However, this is just a convention and not a hard requirement, a file could lack any extension.

A more robust way of identifying what type of data a file contains is by inspecting it, and try to recognize the data format inside it. Does it contain image information? Some binary data? Perhaps it is the name of a directory, or just simple text? You don’t have to do this inspection and guessing yourself, instead you can rely on the file command. The basic usage is as follows:

file somefile

The file command would inspect the content of the file somefile, and based on various rules and heuristics give a description of the type of data it contains.

Exercise 2.90

Try this in a terminal on your real Ubuntu Linux computer. Run file on all the names (files and directories) in the directory /usr/share/backgrounds, e.g.

$ file /usr/share/backgrounds/*

What does this directory contain?

  • File descriptors of every background process running on your computer

  • Log files for services running in the background (daemons)

  • Shell scripts that launch programs with & to run them in the background

  • Configuration files for background system tasks like cron jobs

  • Compiled binaries for background daemons

  • A list of process IDs (PIDs) for currently backgrounded jobs

  • Symbolic links to the current desktop theme’s asset folder

  • Cached thumbnails generated by the file manager

  • Desktop wallpaper images

You can use file to find out whether you can view a file on the screen or not. File types you can show on the screen are: ASCII text, C++ code files, Bourne shell script text, Python scripts, etc. File types you cannot view this easy are: data, directory, symbolic link to 
, executable, etc.

Exercise 2.91

The directory ~/mystery already exists, full of files with deliberately unhelpful names. file doesn’t look at a name or extension at all, it looks at the actual bytes inside a file. Use file mystery/* (or cd mystery then file *) there to see for yourself, and select all of the following that are true.

{
  "filesystem": {
    "/home/student/mystery/vacation.txt": { "base64": "/9j/4AAQSkZJRgAB" },
    "/home/student/mystery/logo.dat": { "base64": "iVBORw0KGgoAAAAN" },
    "/home/student/mystery/report.bin": "%PDF-1.4\n%%EOF\n",
    "/home/student/mystery/deploy": { "content": "#!/bin/bash\necho deploying\n", "mode": "755" },
    "/home/student/mystery/README": "Read this first.\n",
    "/home/student/mystery/engine.dat": "#include <vector>\ntemplate<typename T>\nclass Engine {};\n",
    "/home/student/mystery/shortcut": { "symlink": "README" },
    "/home/student/mystery/License.txt": "None yet\n",
    "/home/student/mystery/projects/placeholder.txt": "I should start one\n"
  }
}

  • vacation.txt is actually JPEG image data, despite its .txt extension.

  • logo.dat is a plain text file.

  • report.bin is a PDF document.

  • deploy is a bash shell script, and is executable.

  • README is a directory.

  • License.txt is a directory, despite its .txt extension.

  • engine.dat is a directory.

  • engine.dat is C++ source code, despite its .dat extension.

  • shortcut is a symbolic link to README.

  • shortcut is a regular text file.

  • projects is a directory.

  • projects is a symbolic link to License.txt.

2.5.2.3. Searching through files#

Besides looking for particular files, you may also want to find files containing particular pieces of text. In this case, the command to use is grep. This command takes as parameters a string to search for and one or more files:

$ grep "hello world" myfile.txt

The search string can be just a piece of text, in this case the string hello world, and grep reports all lines containing the string. It’s a good idea to put this string between quotes ("), otherwise the shell would interpret the search string as being just hello, and the word world to be a new (unknown) grep parameter.

It is also possible to search in multiple files in a directory at once, simply by using a wildcard for the filename, such as:

$ grep -i error *.txt

Here the -i flag indicates a case insensitive search, so it matches lines with the word error, but also Error or ERROR.

Exercise 2.92

Using grep

  1. Perform a case insensitive search for all lines containing the string yoyo in all licenses in /usr/share/common-licenses.

  2. Find all lines in the file /usr/share/common-licenses/GPL which do not contain a space (" "), and make grep also print the line numbers for the matched lines. Hint: use grep --help to find the relevant command switches.

  3. Report all filenames in the directory /usr/share/common-licenses/ of files that contain the word Mozilla.

{
  "filesystemZip": "bash-exercise-fs/fs-usr-share-common-licenses.zip",
  "checks": [
    { "type": "commandEvent", "eventType": "coreutil.grep",
      "match": { "pattern": "yoyo", "flags": ["i"], "matchCount": 4 },
      "desc": "grep for insensitive search yoyo across all the license files, matching exactly 4 sentences" },
    { "type": "commandEvent", "eventType": "coreutil.grep",
      "match": { "pattern": " ", "flags": ["n", "v"], "absPaths": ["/usr/share/common-licenses/GPL"] },
      "desc": "grep for lines with no empty space in the GPL file, also printing line numbers" },
    { "type": "commandEvent", "eventType": "coreutil.grep",
      "match": { "pattern": "Mozilla", "flags": ["l"], "matchCount": 2 },
      "desc": "grep to print all filenames in /usr/share/common-licenses containing Mozilla" }
  ]
}

However, you can also supply grep with a regular expression. In fact, the name grep stands for global regular expression print. A regular expression is a pattern describing how to match various strings according to a number of rules, which it does using so-called meta characters:

  • "^string" matches string only when it is located at the start of a line.

  • "string$" matches string only when it is located at the end of the line.

  • "." matches any character.

  • "[abc]" matches an a, b or c.

  • "[0123456789]" or "[0-9]" matches any single digit number.

  • "[^abc]" matches anything but an a, b or c.

  • "[abc]*" matches zero or more occurrences of any of the characters a, b or c.

  • "[abc]\?" matches zero or one occurrence of any of the characters a, b or c.

  • "[abc]\+" matches one or more occurrences of any of the characters a, b or c.

  • "[abc]\{n\}" matches exactly n occurrences of any of the characters a, b or c.

Unlike *, the ?, + and {n} repeat-operators above only work as repeat-operators if you put a \ in front of them. Without the backslash, they are just ordinary, literal characters to grep. So, "[abc]?" (no backslash) really means “an a, b or c, followed by a literal question mark”, not “zero or one”. This is the reverse of what you might expect: normally a \ in front of a character makes it literal (see below); for ?, + and {n} specifically, it is the backslash that turns them into repeat-operators.

To search for any of the characters used as meta characters (such as ., *, [ or ]) so that they are matched literally instead, you put a \ in front of them. For example, to look for some text between square brackets, use grep "\[.*\]".

Exercise 2.93

Use grep to find the following expressions in /usr/share/common-licenses/GPL. In each case, try to predict what will be the output before running it:

  1. "[A-Z]"

  2. " [A-Z] "

  3. " [A-Z][a-z]*"

  4. "U[a-z]* "

  5. "U[a-z]\{3\} "

{
  "filesystemZip": "bash-exercise-fs/fs-usr-share-common-licenses.zip",
  "checks": [
    { "type": "commandEvent", "eventType": "coreutil.grep",
      "match": { "pattern": "[A-Z]", "absPaths": ["/usr/share/common-licenses/GPL"], "matchCount": 308 },
      "desc": "grep \"[A-Z]\" GPL should match 308 lines" },
    { "type": "commandEvent", "eventType": "coreutil.grep",
      "match": { "pattern": " [A-Z] ", "absPaths": ["/usr/share/common-licenses/GPL"], "matchCount": 12 },
      "desc": "grep \" [A-Z] \" GPL should match 12 lines" },
    { "type": "commandEvent", "eventType": "coreutil.grep",
      "match": { "pattern": " [A-Z][a-z]*", "absPaths": ["/usr/share/common-licenses/GPL"], "matchCount": 293 },
      "desc": "grep \" [A-Z][a-z]*\" GPL should match 293 lines" },
    { "type": "commandEvent", "eventType": "coreutil.grep",
      "match": { "pattern": "U[a-z]* ", "absPaths": ["/usr/share/common-licenses/GPL"], "matchCount": 29 },
      "desc": "grep \"U[a-z]* \" GPL should match 29 lines" },
    { "type": "commandEvent", "eventType": "coreutil.grep",
      "match": { "pattern": "U[a-z]\\{3\\} ", "absPaths": ["/usr/share/common-licenses/GPL"], "matchCount": 7 },
      "desc": "grep \"U[a-z]\\{3\\} \" GPL should match 7 lines" }
  ]
}

We can combine finding files with searching in them using a clever find option, -exec, which makes it execute a command for every file it finds. The command to execute is specified by additional arguments after -exec, up to a final \; argument which indicates that the command started by -exec has been completely specified. Within this command, find will replace each {} argument by the path it found.

For example, we can run a grep command separately on all files found by find like this:

$ find /bin/ -type f -exec grep -H "# Copyright" {} \;
/bin/zmore:# Copyright (C) 2001, 2002, 2007 Free Software Foundation
/bin/zmore:# Copyright (C) 1992, 1993 Jean-loup Gailly
/bin/zdiff:# Copyright (C) 1998, 2002, 2006-2007, 2009-2022 Free Software Foundation, Inc.
/bin/zdiff:# Copyright (C) 1993 Jean-loup Gailly

(This probably produces many more results than shown here)

So in this example, find /bin/ -type f will find in a list of file paths, such as /bin/zmore, /bin/zdiff, etcetera. As a result, find will execute grep -H "# Copyright" /bin/zmore, then grep -H "# Copyright" /bin/zdiff, etcetera (the -H option of grep will make grep also print the filename for every matched line).