This guide contains advice about debugging software issues with the components of the Librem 5 software stack and applications.
Reading the System Journal¶
Errors and warnings that are logged to the system journal can be read by running the
journalctl tool as the root user, or using
sudo. It can be useful to pass the
-e option to skip to the end of the log, and specify
-x to give context to messages in the log output:
sudo journalctl -xe
To see messages from a specific boot, pass the
-b option with an offset that corresponds to the collection of messages for that boot. The collections are referenced chronologically using positive indices starting from 1, but they can also be referenced in reverse chronological order using negative indices starting from 0 and counting down for earlier boots.
For example, messages from the last recorded boot can be obtained with this command:
sudo journalctl -b 0
See the journalctl man page for detailed information about the different ways to access and format journals.
Finding Core Dumps¶
A core dump is an image of running process. It’s created when either the application terminates on a fatal error or the operating systems terminates the appliation (e.g. due to an out of bounds memory access).
Core dumps can be found in the journal and extracted by running the
coredumpctl tool. Running the tool with the
list command will produce a list of core dumps:
To list only the last core dump, pass the
coredumpctl -1 list
It is typically most useful to obtain information about the last core dump for a particular executable. This can be done by using the
info command; for example, the following command shows information for the last core dump associated with the
coredumpctl info /usr/bin/phoc
The core dump itself can be extracted using the
dump command, which in this example will be written to a file named
coredumpctl dump -o core /usr/bin/phoc
If the core dump does not need to be extracted for archiving purposes, it can be more convenient to start debugging directly by using the
debug command, as in this example:
coredumpctl debug /usr/bin/phoc
This run the default debugger – typically
gdb – on the dump. Other debuggers can be specified with the
This can be used to generate so called backtraces that allow developers to get an idea what’s going on. This is done using the bt command:
coredumpctl debug /usr/bin/phoc bt
For these backtraces to become useful you need to have debug packages installed (see below):
Installing debug packages¶
The needed debug packages are dependent on the application but this is a good start:
# Add debug package repositories cat <<EOF > /etc/apt/sources.list.d/debug.list deb https://repo.pureos.net/pureos-debug/ byzantium-debug main deb https://repo.pureos.net/pureos-debug/ landing-debug main deb http://debug.mirrors.debian.org/debian-debug/ bullseye-debug main EOF # Install debug packages and debugging tools apt update apt install systemd-coredump gdb calls-dbgsym chatty-dbgsym libglib2.0-0-dbgsym libgtk-3-0-dbgsym phoc-dbgsym phosh-dbgsym squeekboard-dbgsym
See also this issue for further details.