Recording data with ROS bags

6.3. Recording data with ROS bags#

ros2 bag is a command line tool for recording data published on topics and services in your ROS 2 system, and also to play back such recorded data. If you use it to record, it will accumulate the data passed on any number of topics and services, and save it in a database. Such a recording is simply called a “rosbag”. The ability to record and replay rosbags can have various uses, such as:

  • To reproduce the results of your tests and experiments.

  • To share your work and allow others to recreate it.

  • To record sensor data for data collection

  • To record any messages being exchanged between the nodes in the ROS graph for offline debugging.

ROS bags are like log files that let you store data along with messages. ROS systems can generate a lot of data, so when you bag data you must select which topics you want. Bags are a great tool for testing and debugging your application and a great tool for building robust unit tests.

Let’s take a look at the root ROS Bag command by typing ros2 bag --help into the terminal.

$ ros2 bag -h
usage: ros2 bag [-h] Call `ros2 bag <command> -h` for more detailed usage. ...

Various rosbag related sub-commands

Commands:
  info    ros2 bag info
  play    ros2 bag play
  record  ros2 bag record

As you can see there are three sub commands, record, play, and info. With these commands you can record a bag file, play/replay a file you’ve recorded, and find information about a bag file.

6.3.1. Setup a ROS graph and choose which topic(s) to record#

Let’s try recording our first bag file. To do this we’ll need three terminals all running ROS.

The first terminal should have turtlesim running. If it isn’t running you can restart it with ros2 run turtlesim turtlesim_node. In the second terminal you’ll need to start the turtle_teleop_key demo again to control the default turtle with the keyboard. To do this run ros2 run turtlesim turtle_teleop_key.

ros2 bag can record data from messages published to one or more topics. To see the list of your system’s topics, run in the third terminal the command:

$ ros2 topic list
/parameter_events
/rosout
/turtle1/cmd_vel
/turtle1/color_sensor
/turtle1/pose

Recall that the /turtle_teleop node publishes commands on the /turtle1/cmd_vel topic to make the turtle move in turtlesim.

Recall from the ROS topics section that we can see the data that /turtle1/cmd_vel is publishing via the command:

$ ros2 topic echo /turtle1/cmd_vel

Nothing will show up at first because no data is being published by the teleop. Return to the terminal where you ran the teleop and select it so it’s active. Use the arrow keys to move the turtle around, and you will see data being published on the terminal running ros2 topic echo.

linear:
  x: 2.0
  y: 0.0
  z: 0.0
angular:
  x: 0.0
  y: 0.0
  z: 0.0
  ---

Ok, now we know that /turtle1/cmd_vel is publishing messages. Let’s try to record those messages.

6.3.2. Record a ROS bag#

Let’s first look at the “record” sub command by running in the third terminal ros2 bag record -h

$ ros2 bag record -h
usage: ros2 bag record [-h] [-a] [-o OUTPUT] [-s STORAGE]
                       [-f SERIALIZATION_FORMAT] [--no-discovery]
                       [-p POLLING_INTERVAL] [-b MAX_BAG_SIZE]
                       [topics [topics ...]]

ros2 bag record

positional arguments:
  topics                topics to be recorded

optional arguments:
  -h, --help            show this help message and exit
  -a, --all             recording all topics, required if no topics are listed
                        explicitly.
  -o OUTPUT, --output OUTPUT
                        destination of the bagfile to create, defaults to a
                        timestamped folder in the current directory
  -s STORAGE, --storage STORAGE
                        storage identifier to be used, defaults to "sqlite3"
  -f SERIALIZATION_FORMAT, --serialization-format SERIALIZATION_FORMAT
                        rmw serialization format in which the messages are
                        saved, defaults to the rmw currently in use
  --no-discovery        disables topic auto discovery during recording: only
                        topics present at startup will be recorded
  -p POLLING_INTERVAL, --polling-interval POLLING_INTERVAL
                        time in ms to wait between querying available topics
                        for recording. It has no effect if --no-discovery is
                        enabled.
  -b MAX_BAG_SIZE, --max-bag-size MAX_BAG_SIZE
                        maximum size in bytes before the bagfile will be
                        split. Default it is zero, recording written in single
                        bagfile and splitting is disabled.

We can see from the help file that the syntax for recording a bag is to simply give the sub command a list of topics to record. Most of the other arguments are for more advanced users to help configure how and when data is stored. It is worth noting that there is a -a, --all command that records all the data. You can also specify the output bag file with the -o, --output command.

Note

A good practice is to store your bag files in a seprate directory from your other files. To do so, simply make a new directory to store our saved recordings, e.g.,

$ mkdir bag_files
$ cd bag_files

By default, using ros2 bag to record a rosbag will store it in the current working directory, so be sure to cd into your recordings directory first. Likewise, if you use the tool to playback recorded bags later, it will look for filenames in the current working diretory too.

Let’s go ahead and run our bag command, and let’s “bag” the data on the /turtle1/cmd_vel topic. Run the command:

$ ros2 bag record /turtle1/cmd_vel

Warning

Be aware that the program will continue bagging data until you hit CTRL-C, so give the command a good 30 seconds to collect data before you kill it.

Also, ROS bags can become very large if you record many topics with high-frequency sensor data, so be careful that you don’t fill up your free disk space while recording, because your system will become unresponsive!

You will see the following messages in the terminal (the date and time will be different):

[INFO] [rosbag2_storage]: Opened database 'rosbag2_2019_10_11-05_18_45'.
[INFO] [rosbag2_transport]: Listening for topics...
[INFO] [rosbag2_transport]: Subscribed to topic '/turtle1/cmd_vel'
[INFO] [rosbag2_transport]: All requested topics are subscribed. Stopping discovery...
^C[INFO] [rclcpp]: signal_handler(signal_value=2)

Now ros2 bag is recording the data published on the /turtle1/cmd_vel topic. Return to the teleop terminal and move the turtle around again. The movements don’t matter, but try to make a recognizable pattern to see when you replay the data later.

../_images/record1.png

Press Ctrl+C to stop recording.

If we don’t explicitly give ros2 bag record a name for the new recording with the -o flag, the data will by default be accumulated in a new bag directory with a name in the pattern of rosbag2_YEAR_MONTH_DAY-HOUR_MINUTE_SECOND. The recording directory will contain a metadata.yaml along with the bag file in the recorded format.

6.3.3. Record multiple topics#

You can also record multiple topics, as well as change the name of the file ros2 bag saves to. Run the following command:

$ ros2 bag record -o subset /turtle1/cmd_vel /turtle1/pose

The -o option allows you to choose a unique name for your bag file. The following string, in this case subset, is the file name for the BAG file.

To record more than one topic at a time, simply list each topic separated by a space.

Note

There is another option you can add to the command, -a, which records all the topics on your system.

You will see the following message, confirming that both topics are being recorded.

[INFO] [rosbag2_storage]: Opened database 'subset'.
[INFO] [rosbag2_transport]: Listening for topics...
[INFO] [rosbag2_transport]: Subscribed to topic '/turtle1/cmd_vel'
[INFO] [rosbag2_transport]: Subscribed to topic '/turtle1/pose'
[INFO] [rosbag2_transport]: All requested topics are subscribed. Stopping discovery...
^C[INFO] [rclcpp]: signal_handler(signal_value=2)

You can move the turtle around, and don’t forget to press Ctrl+C when you’re finished.

6.3.4. Inspecting the contents of a ROS Bag#

Now that we collected our data let’s inspect our bag file. You can introspect any bag file using the ros2 bag info command. This command will list informative properties, such as:

  • the file size,

  • the total number of recorded messages,

  • the duration of file,

  • the recording date and time,

  • the included topic names, and number of messages per topic

For example:

$ ros2 bag info subset
Files:             subset.mcap
Bag size:          228.5 KiB
Storage id:        mcap
Duration:          48.47s
Start:             Oct 11 2019 06:09:09.12 (1570799349.12)
End                Oct 11 2019 06:09:57.60 (1570799397.60)
Messages:          3013
Topic information: Topic: /turtle1/cmd_vel | Type: geometry_msgs/msg/Twist | Count: 9 | Serialization Format: cdr
                 Topic: /turtle1/pose | Type: turtlesim/msg/Pose | Count: 3004 | Serialization Format: cdr

6.3.5. Replaying a ROS Bag#

Once you have collected a bag file you can replay the file just like a running system. Bags are a great tool for debugging and testing. You can treat a ROS bag like a recording of a running ROS system. When you play a bag file you can use the regular ros2 CLI tools to inspect the recorded topics.

Before replaying the bag file, enter Ctrl+C in the terminal where the teleop is running. Then make sure your turtlesim window is visible so you can see the bag file in action.

Note

Think about it, why would you want to stop the teleop node before replaying the rosbag?

Now in the third terminal replay the bag file using the following command:

$ ros2 bag play subset
[INFO] [rosbag2_storage]: Opened database 'subset'.

Nothing should happen visibly in this third terminal, but a lot is happening under the hood. To see what is happening go to a second terminal. Your turtle will follow roughly the same path you entered while recording

../_images/playback.png

Because the subset file recorded the /turtle1/pose topic, the ros2 bag play command won’t quit for as long as you had turtlesim running, even if you weren’t moving. This is because as long as the /turtlesim node is active, it publishes data on the /turtle1/pose topic at regular intervals.

Warning

You may notice that the replay will not produce 100% exactly the same turtle path 100% as before!

In practice, like any control system, turtlesim is sensitive to small changes in the system’s timing of its control commands. As you are replaying velocity commands, small differences in the timing between how your original teleop node published the messages, and how the ros bag player node publishes the same messages, will result in accumulating differences in behavior.

This is not a bug in ros bag player, but an inherent property of the asynchronous communication between nodes, and of nodes being independent processes with all their own message processing loop.

Tip

In the terminal where you are running ros2 bag play, while the bag is playing, you can pause or unpause playback by pressing SPACE-bar. When you pause, you should see a line in the terminal like

INFO] [1726354133.858323972] [rosbag2_player]: Pausing play.

The bag playback command has several useful options that can come in very handy when inspecting a bag. Try running

$ ros2 bag play --help

Among the options in the output, you should see the following (among many more options):

  -r RATE, --rate RATE  rate at which to play back messages. Valid range > 0.0.
  --topics TOPICS [TOPICS ...]
                        topics to replay, separated by space. If none specified, all topics will be replayed.
  -l, --loop            enables loop playback when playing a bagfile: it starts back at the beginning on reaching the end and plays indefinitely.
  --remap REMAP [REMAP ...], -m REMAP [REMAP ...]
                        list of topics to be remapped: in the form "old_topic1:=new_topic1 old_topic2:=new_topic2 etc."
  --clock [CLOCK]       Publish to /clock at a specific frequency in Hz, to act as a ROS Time Source. Value must be positive. Defaults to not publishing.
  -p, --start-paused    Start the playback player in a paused state.

For example, if you want to replay your bag continuously and at half speed, you could use:

$ ros2 bag play subset -l -r 0.5

Note that with the loop option -l you must stop playback with CTRL-C, while without loop the ros2 bag play command will stop once the end of the bag has been reached.

6.3.6. Inspecting the messages during playback#

The playback sub command just creates a temporary ROS node that publishes all of the recorded topics. So, just as when a running any regular ROS node, you should be able to run CLI tools while the ROS bag tool is publishing to inspect the recorded topics.

$ ros2 topic list
/parameter_events
/rosout
/turtle1/cmd_vel
/turtle1/pose

Notice in the output of ros2 bag info before that /turtle1/pose has a Count value of over 3000; while we were recording, data was published on that topic 3000 times. To get an idea of how often position data is published, you can now run the command:

$ ros2 topic hz /turtle1/pose

If you need a more thorough explanation of ros2 bag, you can check out the README here.

6.3.7. “Now” is not always now: ROS Time vs Wall Clock time#

Many robotic subsystems need to consider how much time has passed between different messages or between a message and the “current time”. The time at which a message was (first) sent can often be found in its Header’s time stamp (if it has a header field).

Accurately assessing the time and age of messages is therefore crucial. For example:

  • If a message is too old, it may be considered irrelevant and should not be processed.

  • If time difference between measurements is reported incorrectly, the robot’s control mechanism could overcompensate.

Important

When you want to use ROS to process data generated by a simulation or by replaying a rosbag, the nodes in your graph should not consider the current date and time reported by your computer’s regular clock since then the age of a message could be interpreted incorrectly, resulting in unexpected and unintended behavior. We therefore distinguish the following two concepts of time:

  • wall clock time (a.k.a. “real” time) is the time you can currently see on a normal clock (hanging on your wall), and is also the time normally reported by your computer.

  • ROS time is the time that you want ROS nodes to perceive.

When you use ROS to operate an actual robot, these two time concepts are the same: the ROS time is by default set to the wall clock time. The age of a message can then be determined by comparing its time stamp to the computer’s regular clock.

However, when you run a robot simulation or replay a rosbag, the ROS time could run faster or slower than the wall clock time, and could even represent a completely different date, e.g., when replaying a recording from a year ago. In this case all nodes should listen to a different clock, which is what we’d call ROS time.

To make a ROS node adjust its internal ROS time to your intended time, two things need to happen:

  1. The special /clock topic should publish messages indicating the intended ROS time.

  2. You need to instruct all nodes to listen to this /clock topic instead of your computer’s wall clock by setting a node parameter called use_sim_time to true.

For step 1, how do you obtain the /clock topic? When running a simulation, your simulator’s ROS integration should publish the simulated time on /clock. How fast it ticks typically depends on the physics engine, the complexity of the simulation, and the computational power of the computer running the simulation. Consult your simulator’s documentation on how to do this.

When replaying a rosbag, you should explicitly instruct the rosbag player to report its internal playback time representing the time of recording on /clock. This can be done using the --clock argument:

$ ros2 bag play <bag_name> --clock

The --clock argument can be combined with all the other arguments, such as -r 0.5 to reduce playback rate by two, effectively making time pass twice as slow.

For step 2, how do we enable use_sim_time for each node? Unfortunately, we need to set the use_sim_time parameter for each node individually when starting each node. When starting nodes one by one via the CLI, this can be done using the syntax described in the section on setting parameters on node startup, i.e.,

$ ros2 run <package_name> <node_executable> --ros-args -p use_sim_time:=true

In real-world robotic applications, we would set the intended startup value for use_sim_time for all nodes by properly configuring the launch files of our robot, which start and configure whole groups of nodes at once.

Exercise 6.2

Replay a rosbag with the --clock argument and verify that the /clock topic is now being published to with

$ ros2 topic echo /clock

As you can see, the clock reports time in seconds and nanoseconds. This time format is Unix time, and represents the number of seconds and nanoseconds that have elapsed since January 1st, 1970, 00:00:00 UTC.

Take the reported number of seconds on /clock from the rosbag (just stop the topic echo and use the last reported value). Can you convert this number to a more human-readable date? You can use any online “UNIX time converter” website.

Verify with ros2 bag info <bag_name> that your converted time is within the Start and End time of the bag.

6.3.8. Summary#

You can record data passed on topics in your ROS 2 system using the ros2 bag command. Whether you’re sharing your work with others or introspecting your own experiments, it’s a great tool to know about.