Debugging

This guide contains advice about debugging software issues with the components of the Librem 5 software stack and applications.

Reading the System Journal

Errors and warnings that are logged to the system journal can be read by running the journalctl tool as the root user, or using sudo. It can be useful to pass the -e option to skip to the end of the log, and specify -x to give context to messages in the log output:

sudo journalctl -xe

To see messages from a specific boot, pass the -b option with an offset that corresponds to the collection of messages for that boot. The collections are referenced chronologically using positive indices starting from 1, but they can also be referenced in reverse chronological order using negative indices starting from 0 and counting down for earlier boots.

For example, messages from the last recorded boot can be obtained with this command:

sudo journalctl -b 0

See the journalctl man page for detailed information about the different ways to access and format journals.

Finding Core Dumps

A core dump is an image of running process. It’s created when either the application terminates on a fatal error or the operating systems terminates the appliation (e.g. due to an out of bounds memory access).

Core dumps can be found in the journal and extracted by running the coredumpctl tool. Running the tool with the list command will produce a list of core dumps:

coredumpctl list

To list only the last core dump, pass the -1 option:

coredumpctl -1 list

It is typically most useful to obtain information about the last core dump for a particular executable. This can be done by using the info command; for example, the following command shows information for the last core dump associated with the /usr/bin/phoc executable:

coredumpctl info /usr/bin/phoc

The core dump itself can be extracted using the dump command, which in this example will be written to a file named core:

coredumpctl dump -o core /usr/bin/phoc

If the core dump does not need to be extracted for archiving purposes, it can be more convenient to start debugging directly by using the debug command, as in this example:

coredumpctl debug /usr/bin/phoc

This run the default debugger – typically gdb – on the dump. Other debuggers can be specified with the --debugger option. This can be used to generate so called backtraces that allow developers to get an idea what’s going on. This is done using the bt command:

coredumpctl debug /usr/bin/phoc bt

For these backtraces to become useful you need to have debug packages installed (see below):

Installing debug packages

The needed debug packages are dependent on the application but this is a good start:

# Add debug package repositories
cat <<EOF > /etc/apt/sources.list.d/debug.list
deb https://repo.pureos.net/pureos-debug/ byzantium-debug main
deb https://repo.pureos.net/pureos-debug/ landing-debug main
deb http://debug.mirrors.debian.org/debian-debug/ bullseye-debug main
EOF

# Install debug packages and debugging tools
apt update
apt install systemd-coredump gdb calls-dbgsym chatty-dbgsym libglib2.0-0-dbgsym libgtk-3-0-dbgsym phoc-dbgsym phosh-dbgsym squeekboard-dbgsym

See also this issue for further details.