systemd/docs/HACKING.md

title	category	layout	SPDX-License-Identifier
Hacking on systemd	Contributing	default	LGPL-2.1-or-later

Hacking on systemd

We welcome all contributions to systemd. If you notice a bug or a missing feature, please feel invited to fix it, and submit your work as a GitHub Pull Request (PR).

Please make sure to follow our Coding Style when submitting patches. Also have a look at our Contribution Guidelines.

When adding new functionality, tests should be added. For shared functionality (in src/basic/ and src/shared/) unit tests should be sufficient. The general policy is to keep tests in matching files underneath src/test/, e.g. src/test/test-path-util.c contains tests for any functions in src/basic/path-util.c. If adding a new source file, consider adding a matching test executable. For features at a higher level, tests in src/test/ are very strongly recommended. If that is not possible, integration tests in test/ are encouraged.

Please always test your work before submitting a PR. For many of the components of systemd testing is straightforward as you can simply compile systemd and run the relevant tool from the build directory.

For some components (most importantly, systemd/PID 1 itself) this is not possible, however. In order to simplify testing for cases like this we provide a set of mkosi config files directly in the source tree. mkosi is a tool for building clean OS images from an upstream distribution in combination with a fresh build of the project in the local working directory. To make use of this, please install mkosi from the GitHub repository. mkosi will build an image for the host distro by default. First, run mkosi genkey to generate a key and certificate to be used for secure boot and verity signing. After that is done, it is sufficient to type mkosi in the systemd project directory to generate a disk image you can boot either in systemd-nspawn or in a UEFI-capable VM:

$ sudo mkosi boot # nspawn still needs sudo for now

or:

$ mkosi qemu

By default, the tools from your host system are used to build the image. Sometimes we start using mkosi features that rely on functionality in systemd tools that's not in an official release yet. In that case, you'll need to build systemd from source on the host and configure mkosi to use the tools from the systemd build directory.

To do a local build, most distributions provide very simple and convenient ways to install most development packages necessary to build systemd:

# Fedora
$ sudo dnf builddep systemd
# Debian/Ubuntu
$ sudo apt-get build-dep systemd
# Arch
$ sudo pacman -S devtools
$ pkgctl repo clone --protocol=https systemd
$ cd systemd
$ makepkg -seoc

After installing the development packages, systemd can be built from source as follows:

$ meson setup build <options>
$ ninja -C build
$ meson test -C build

To have mkosi use the systemd tools from the build/ directory, add the following to mkosi.local.conf:

[Host]
ExtraSearchPaths=build/

And if you want mkosi to build a tools image and use the tools from there instead of looking for tools on the host, add the following to mkosi.local.conf:

[Host]
ToolsTree=default

Every time you rerun the mkosi command a fresh image is built, incorporating all current changes you made to the project tree. To build the latest changes and re-install after booting the image, run one of the following commands in another terminal on your host (choose the right one depending on the distribution of the container or virtual machine):

mkosi -t none && mkosi ssh dnf upgrade --disablerepo="*" --assumeyes "/work/build/*.rpm"             # CentOS/Fedora
mkosi -t none && mkosi ssh apt-get install "/work/build/*.deb"                                       # Debian/Ubuntu
mkosi -t none && mkosi ssh pacman --upgrade --needed --noconfirm "/work/build/*.pkg.tar"             # Arch Linux
mkosi -t none && mkosi ssh zypper --non-interactive install --allow-unsigned-rpm "/work/build/*.rpm" # OpenSUSE

and optionally restart the daemon(s) you're working on using systemctl restart <units> or systemctl daemon-reexec if you're working on pid1 or systemctl soft-reboot to restart everything.

Putting this all together, here's a series of commands for preparing a patch for systemd:

$ git clone https://github.com/systemd/mkosi.git
$ ln -s $PWD/mkosi/bin/mkosi /usr/local/bin/mkosi
$ git clone https://github.com/systemd/systemd.git
$ cd systemd
$ git checkout -b <BRANCH>        # where BRANCH is the name of the branch
$ vim src/core/main.c             # or wherever you'd like to make your changes
$ mkosi -f qemu                   # (re-)build and boot up the test image in qemu
$ mkosi -t none                   # Build new packages without rebuilding the image
$ git add -p                      # interactively put together your patch
$ git commit                      # commit it
$ git push -u <REMOTE>            # where REMOTE is your "fork" on GitHub

And after that, head over to your repo on GitHub and click "Compare & pull request"

Happy hacking!

Building distribution packages with mkosi

To build distribution packages for a specific distribution and release without building an actual image, the following command can be used:

mkosi -d <distribution> -r <release> -t none

Afterwards the distribution packages will be located in build/mkosi.builddir/<distribution>~<release>~<architecture>/. To also build debuginfo packages, the following command can be used:

mkosi -d <distribution> -r <release> -E WITH_DEBUG=1 -t none

To upgrade the systemd packages on the host system to the newer versions built by mkosi, run the following:

dnf upgrade build/mkosi.builddir/<distribution>~<release>~<architecture>/*.rpm                                           # Fedora/CentOS
apt-get install build/mkosi.builddir/<distribution>~<release>~<architecture>/*.deb                                       # Debian/Ubuntu
pacman --upgrade --needed --noconfirm build/mkosi.builddir/<distribution>~<release>~<architecture>/*.pkg.tar             # Arch Linux
zypper --non-interactive install --allow-unsigned-rpm build/mkosi.builddir/<distribution>~<release>~<architecture>/*.rpm # OpenSUSE

To downgrade back to the old version shipped by the distribution, run the following:

dnf downgrade "systemd*" # Fedora/CentOS
# TODO: Other distributions

Additionally, for each pull request, the built distribution packages are attached as CI artifacts to the pull request CI jobs, which means that users can download and install them to test out if a pull request fixes the issue that they reported. To download the packages from a pull request, click on the Checks tab. Then click on the mkosi workflow in the list of workflows on the left of the Checks page. Finally, scroll down to find the list of CI artifacts. In this list of artifacts you can find artifacts containing distribution packages. To install these, download the artifact which is a zip archive, extract the zip archive to access the individual packages, and install them with your package manager in the same way as described above for packages that were built locally.

Templating engines in .in files

Some source files are generated during build. We use two templating engines:

• meson's configure_file() directive uses syntax with @VARIABLE@.

See the Meson docs for configure_file() for details.

{% raw %}

• most files are rendered using jinja2, with {{VARIABLE}} and {% if … %}, {% elif … %}, {% else … %}, {% endif … %} blocks. {# … #} is a jinja2 comment, i.e. that block will not be visible in the rendered output. {% raw %} … {% endraw %}{{ '{' }}{{ '% endraw %' }}} creates a block where jinja2 syntax is not interpreted.

See the Jinja Template Designer Documentation for details.

Please note that files for both template engines use the .in extension.

Developer and release modes

In the default meson configuration (-Dmode=developer), certain checks are enabled that are suitable when hacking on systemd (such as internal documentation consistency checks). Those are not useful when compiling for distribution and can be disabled by setting -Dmode=release.

Sanitizers in mkosi

See Testing systemd using sanitizers for more information on how to build with sanitizers enabled in mkosi.

Fuzzers

systemd includes fuzzers in src/fuzz/ that use libFuzzer and are automatically run by OSS-Fuzz with sanitizers. To add a fuzz target, create a new src/fuzz/fuzz-foo.c file with a LLVMFuzzerTestOneInput function and add it to the list in src/fuzz/meson.build.

Whenever possible, a seed corpus and a dictionary should also be added with new fuzz targets. The dictionary should be named src/fuzz/fuzz-foo.dict and the seed corpus should be built and exported as $OUT/fuzz-foo_seed_corpus.zip in tools/oss-fuzz.sh.

The fuzzers can be built locally if you have libFuzzer installed by running tools/oss-fuzz.sh, or by running:

CC=clang CXX=clang++ \
meson setup build-libfuzz -Dllvm-fuzz=true -Db_sanitize=address,undefined -Db_lundef=false \
-Dc_args='-fno-omit-frame-pointer -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION'
ninja -C build-libfuzz fuzzers

Each fuzzer then can be then run manually together with a directory containing the initial corpus:

export UBSAN_OPTIONS=print_stacktrace=1:print_summary=1:halt_on_error=1
build-libfuzz/fuzz-varlink-idl test/fuzz/fuzz-varlink-idl/

Note: the halt_on_error=1 UBSan option is especially important, otherwise the fuzzer won't crash when undefined behavior is triggered.

You should also confirm that the fuzzers can be built and run using the OSS-Fuzz toolchain:

path_to_systemd=...

git clone --depth=1 https://github.com/google/oss-fuzz
cd oss-fuzz

for sanitizer in address undefined memory; do
for engine in libfuzzer afl honggfuzz; do
./infra/helper.py build_fuzzers --sanitizer "$sanitizer" --engine "$engine" \
--clean systemd "$path_to_systemd"

./infra/helper.py check_build --sanitizer "$sanitizer" --engine "$engine" \
-e ALLOWED_BROKEN_TARGETS_PERCENTAGE=0 systemd
done
done

./infra/helper.py build_fuzzers --clean --architecture i386 systemd "$path_to_systemd"
./infra/helper.py check_build --architecture i386 -e ALLOWED_BROKEN_TARGETS_PERCENTAGE=0 systemd

./infra/helper.py build_fuzzers --clean --sanitizer coverage systemd "$path_to_systemd"
./infra/helper.py coverage --no-corpus-download systemd

If you find a bug that impacts the security of systemd, please follow the guidance in CONTRIBUTING.md on how to report a security vulnerability.

For more details on building fuzzers and integrating with OSS-Fuzz, visit:

• Setting up a new project - OSS-Fuzz
• Tutorials - OSS-Fuzz

Debugging binaries that need to run as root in vscode

When trying to debug binaries that need to run as root, we need to do some custom configuration in vscode to have it try to run the applications as root and to ask the user for the root password when trying to start the binary. To achieve this, we'll use a custom debugger path which points to a script that starts gdb as root using pkexec. pkexec will prompt the user for their root password via a graphical interface. This guide assumes the C/C++ extension is used for debugging.

First, create a file sgdb in the root of the systemd repository with the following contents and make it executable:

#!/bin/sh
exec pkexec gdb "$@"

Then, open launch.json in vscode, and set miDebuggerPath to ${workspaceFolder}/sgdb for the corresponding debug configuration. Now, whenever you try to debug the application, vscode will try to start gdb as root via pkexec which will prompt you for your password via a graphical interface. After entering your password, vscode should be able to start debugging the application.

For more information on how to set up a debug configuration for C binaries, please refer to the official vscode documentation here

Debugging systemd with mkosi + vscode

To simplify debugging systemd when testing changes using mkosi, we're going to show how to attach VSCode's debugger to an instance of systemd running in a mkosi image using QEMU.

To allow VSCode's debugger to attach to systemd running in a mkosi image, we have to make sure it can access the virtual machine spawned by mkosi where systemd is running. After booting the image with mkosi qemu, you should now be able to connect to it by running mkosi ssh from the same directory in another terminal window.

Now we need to configure VSCode. First, make sure the C/C++ extension is installed. If you're already using a different extension for code completion and other IDE features for C in VSCode, make sure to disable the corresponding parts of the C/C++ extension in your VSCode user settings by adding the following entries:

"C_Cpp.formatting": "Disabled",
"C_Cpp.intelliSenseEngine": "Disabled",
"C_Cpp.enhancedColorization": "Disabled",
"C_Cpp.suggestSnippets": false,

With the extension set up, we can create the launch.json file in the .vscode/ directory to tell the VSCode debugger how to attach to the systemd instance running in our mkosi container/VM. Create the file, and possibly the directory, and add the following contents:

{
    "version": "0.2.0",
    "configurations": [
        {
            "type": "cppdbg",
            "program": "/usr/lib/systemd/systemd",
            "processId": "${command:pickRemoteProcess}",
            "request": "attach",
            "name": "systemd",
            "pipeTransport": {
                "pipeProgram": "mkosi",
                "pipeArgs": ["-C", "${workspaceFolder}", "ssh"],
                "debuggerPath": "/usr/bin/gdb"
            },
            "MIMode": "gdb",
            "sourceFileMap": {
                "/work/src": {
                    "editorPath": "${workspaceFolder}",
                    "useForBreakpoints": false
                },
            }
        }
    ]
}

Now that the debugger knows how to connect to our process in the container/VM and we've set up the necessary source mappings, go to the "Run and Debug" window and run the "systemd" debug configuration. If everything goes well, the debugger should now be attached to the systemd instance running in the container/VM. You can attach breakpoints from the editor and enjoy all the other features of VSCode's debugger.

To debug systemd components other than PID 1, set "program" to the full path of the component you want to debug and set "processId" to "${command:pickProcess}". Now, when starting the debugger, VSCode will ask you the PID of the process you want to debug. Run systemctl show --property MainPID --value <component> in the container to figure out the PID and enter it when asked and VSCode will attach to that process instead.

Debugging systemd-boot

During boot, systemd-boot and the stub loader will output messages like systemd-boot@0x0A and systemd-stub@0x0B, providing the base of the loaded code. This location can then be used to attach to a QEMU session (provided it was run with -s). See debug-sd-boot.sh script in the tools folder which automates this processes.

If the debugger is too slow to attach to examine an early boot code passage, the call to DEFINE_EFI_MAIN_FUNCTION() can be modified to enable waiting. As soon as the debugger has control, we can then run set variable wait = 0 or return to continue. Once the debugger has attached, setting breakpoints will work like usual.

To debug systemd-boot in an IDE such as VSCode we can use a launch configuration like this:

{
    "name": "systemd-boot",
    "type": "cppdbg",
    "request": "launch",
    "program": "${workspaceFolder}/build/src/boot/efi/systemd-bootx64.efi",
    "cwd": "${workspaceFolder}",
    "MIMode": "gdb",
    "miDebuggerServerAddress": ":1234",
    "setupCommands": [
        { "text": "shell mkfifo /tmp/sdboot.{in,out}" },
        { "text": "shell qemu-system-x86_64 [...] -s -serial pipe:/tmp/sdboot" },
        { "text": "shell ${workspaceFolder}/tools/debug-sd-boot.sh ${workspaceFolder}/build/src/boot/efi/systemd-bootx64.efi /tmp/sdboot.out systemd-boot.gdb" },
        { "text": "source /tmp/systemd-boot.gdb" },
    ]
}

mkosi + clangd

clangd is a language server that provides code completion, diagnostics and more right in your editor of choice (with the right plugin installed). When using mkosi, we can run clangd in the mkosi build container to avoid needing to build systemd on the host machine just to make clangd work.

All that is required is to run mkosi once to make sure cached images are available and to modify the path of the clangd binary used by your editor to the mkosi.clangd script included in the systemd repository. For example, for VScode, you'd have to add the following to the VSCode workspace settings of the systemd repository:

{
    "clangd.path": "<path-to-systemd-repository>/mkosi.clangd",
}