mirror of
https://github.com/systemd/systemd.git
synced 2024-12-04 15:53:41 +08:00
437 lines
22 KiB
XML
437 lines
22 KiB
XML
<?xml version='1.0'?>
|
|
<!DOCTYPE refentry PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN"
|
|
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
|
|
<!-- SPDX-License-Identifier: LGPL-2.1-or-later -->
|
|
|
|
<refentry id="systemd-coredump" conditional='ENABLE_COREDUMP'
|
|
xmlns:xi="http://www.w3.org/2001/XInclude">
|
|
|
|
<refentryinfo>
|
|
<title>systemd-coredump</title>
|
|
<productname>systemd</productname>
|
|
</refentryinfo>
|
|
|
|
<refmeta>
|
|
<refentrytitle>systemd-coredump</refentrytitle>
|
|
<manvolnum>8</manvolnum>
|
|
</refmeta>
|
|
|
|
<refnamediv>
|
|
<refname>systemd-coredump</refname>
|
|
<refname>systemd-coredump.socket</refname>
|
|
<refname>systemd-coredump@.service</refname>
|
|
<refpurpose>Acquire, save and process core dumps</refpurpose>
|
|
</refnamediv>
|
|
|
|
<refsynopsisdiv>
|
|
<para><filename>/usr/lib/systemd/systemd-coredump</filename></para>
|
|
<para><filename>/usr/lib/systemd/systemd-coredump</filename> <option>--backtrace</option></para>
|
|
<para><filename>systemd-coredump@.service</filename></para>
|
|
<para><filename>systemd-coredump.socket</filename></para>
|
|
</refsynopsisdiv>
|
|
|
|
<refsect1>
|
|
<title>Description</title>
|
|
<para><filename>systemd-coredump@.service</filename> is a system service to process core dumps. It will
|
|
log a summary of the event to
|
|
<citerefentry><refentrytitle>systemd-journald.service</refentrytitle><manvolnum>8</manvolnum></citerefentry>,
|
|
including information about the process identifier, owner, the signal that killed the process, and the
|
|
stack trace if possible. It may also save the core dump for later processing. See the "Information about
|
|
the crashed process" section below.</para>
|
|
|
|
<para>The behavior of a specific program upon reception of a signal is governed by a few
|
|
factors which are described in detail in
|
|
<citerefentry project='man-pages'><refentrytitle>core</refentrytitle><manvolnum>5</manvolnum></citerefentry>.
|
|
In particular, the core dump will only be processed when the related resource limits are sufficient.
|
|
</para>
|
|
|
|
<para>Core dumps can be written to the journal or saved as a file. In both cases, they can be retrieved
|
|
for further processing, for example in
|
|
<citerefentry project='man-pages'><refentrytitle>gdb</refentrytitle><manvolnum>1</manvolnum></citerefentry>.
|
|
See <citerefentry><refentrytitle>coredumpctl</refentrytitle><manvolnum>1</manvolnum></citerefentry>,
|
|
in particular the <command>list</command> and <command>debug</command> verbs.</para>
|
|
|
|
<para>By default, <command>systemd-coredump</command> will log the core dump to the journal, including a
|
|
backtrace if possible, and store the core dump (an image of the memory contents of the process) itself in
|
|
an external file in <filename>/var/lib/systemd/coredump</filename>. These core dumps are deleted after a
|
|
few days by default; see <filename>/usr/lib/tmpfiles.d/systemd.conf</filename> for details. Note that the
|
|
removal of core files from the file system and the purging of journal entries are independent, and the
|
|
core file may be present without the journal entry, and journal entries may point to since-removed core
|
|
files. Some metadata is attached to core files in the form of extended attributes, so the core files are
|
|
useful for some purposes even without the full metadata available in the journal entry.</para>
|
|
|
|
<refsect2>
|
|
<title>Invocation of <command>systemd-coredump</command></title>
|
|
|
|
<para>The <command>systemd-coredump</command> executable does the actual work. It is invoked twice:
|
|
once as the handler by the kernel, and the second time in the
|
|
<filename>systemd-coredump@.service</filename> to actually write the data to the journal and process
|
|
and save the core file.</para>
|
|
|
|
<para>When the kernel invokes <command>systemd-coredump</command> to handle a core dump, it runs in
|
|
privileged mode, and will connect to the socket created by the
|
|
<filename>systemd-coredump.socket</filename> unit, which in turn will spawn an unprivileged
|
|
<filename>systemd-coredump@.service</filename> instance to process the core dump. Hence
|
|
<filename>systemd-coredump.socket</filename> and <filename>systemd-coredump@.service</filename> are
|
|
helper units which do the actual processing of core dumps and are subject to normal service
|
|
management.</para>
|
|
|
|
<para>It is also possible to invoke <command>systemd-coredump</command> with
|
|
<option>--backtrace</option> option. In this case, <command>systemd-coredump</command> expects a
|
|
journal entry in the journal
|
|
<ulink url="https://systemd.io/JOURNAL_EXPORT_FORMATS#journal-export-format">Journal Export Format</ulink>
|
|
on standard input. The entry should contain a <varname>MESSAGE=</varname> field and any additional
|
|
metadata fields the caller deems reasonable. <command>systemd-coredump</command> will append additional
|
|
metadata fields in the same way it does for core dumps received from the kernel. In this mode, no core
|
|
dump is stored in the journal.</para>
|
|
</refsect2>
|
|
</refsect1>
|
|
|
|
<refsect1>
|
|
<title>Configuration</title>
|
|
<para>For programs started by <command>systemd</command>, process resource limits can be set by directive
|
|
<varname>LimitCORE=</varname>, see
|
|
<citerefentry><refentrytitle>systemd.exec</refentrytitle><manvolnum>5</manvolnum></citerefentry>.
|
|
</para>
|
|
|
|
<para>In order to be used by the kernel to handle core dumps,
|
|
<command>systemd-coredump</command> must be configured in
|
|
<citerefentry project='man-pages'><refentrytitle>sysctl</refentrytitle><manvolnum>8</manvolnum></citerefentry>
|
|
parameter <varname>kernel.core_pattern</varname>. The syntax of this parameter is explained in
|
|
<citerefentry project='man-pages'><refentrytitle>core</refentrytitle><manvolnum>5</manvolnum></citerefentry>.
|
|
systemd installs the file <filename>/usr/lib/sysctl.d/50-coredump.conf</filename> which configures
|
|
<varname>kernel.core_pattern</varname> accordingly. This file may be masked or overridden to use a different
|
|
setting following normal
|
|
<citerefentry><refentrytitle>sysctl.d</refentrytitle><manvolnum>5</manvolnum></citerefentry>
|
|
rules. If the sysctl configuration is modified, it must be updated in the kernel before it
|
|
takes effect, see
|
|
<citerefentry project='man-pages'><refentrytitle>sysctl</refentrytitle><manvolnum>8</manvolnum></citerefentry>
|
|
and
|
|
<citerefentry><refentrytitle>systemd-sysctl</refentrytitle><manvolnum>8</manvolnum></citerefentry>.
|
|
</para>
|
|
|
|
<para>In order to be used in the <option>--backtrace</option> mode, an appropriate backtrace
|
|
handler must be installed on the sender side. For example, in case of
|
|
<citerefentry project='die-net'><refentrytitle>python</refentrytitle><manvolnum>1</manvolnum></citerefentry>, this
|
|
means a <varname>sys.excepthook</varname> must be installed, see
|
|
<ulink url="https://github.com/systemd/systemd-coredump-python">systemd-coredump-python</ulink>.
|
|
</para>
|
|
|
|
<para>The behavior of <command>systemd-coredump</command> itself is configured through the configuration file
|
|
<filename>/etc/systemd/coredump.conf</filename> and corresponding snippets
|
|
<filename>/etc/systemd/coredump.conf.d/*.conf</filename>, see
|
|
<citerefentry><refentrytitle>coredump.conf</refentrytitle><manvolnum>5</manvolnum></citerefentry>. A new
|
|
instance of <command>systemd-coredump</command> is invoked upon receiving every core dump. Therefore, changes
|
|
in these files will take effect the next time a core dump is received.</para>
|
|
|
|
<para>Resources used by core dump files are restricted in two ways. Parameters like maximum size of acquired
|
|
core dumps and files can be set in files <filename>/etc/systemd/coredump.conf</filename> and snippets mentioned
|
|
above. In addition the storage time of core dump files is restricted by <command>systemd-tmpfiles</command>,
|
|
corresponding settings are by default in <filename>/usr/lib/tmpfiles.d/systemd.conf</filename>. The default is
|
|
to delete core dumps after a few days; see the above file for details.</para>
|
|
|
|
<refsect2>
|
|
<title>Disabling coredump processing</title>
|
|
|
|
<para>To disable potentially resource-intensive processing by <command>systemd-coredump</command>, set
|
|
<programlisting>Storage=none ProcessSizeMax=0</programlisting> in
|
|
<citerefentry><refentrytitle>coredump.conf</refentrytitle><manvolnum>5</manvolnum></citerefentry>.
|
|
</para>
|
|
</refsect2>
|
|
</refsect1>
|
|
|
|
<refsect1>
|
|
<title>Information about the crashed process</title>
|
|
|
|
<para><citerefentry><refentrytitle>coredumpctl</refentrytitle><manvolnum>1</manvolnum></citerefentry> can
|
|
be used to retrieve saved core dumps independently of their location, to display information, and to
|
|
process them e.g. by passing to the GNU debugger (gdb).</para>
|
|
|
|
<para>Data stored in the journal can be also viewed with
|
|
<citerefentry><refentrytitle>journalctl</refentrytitle><manvolnum>1</manvolnum></citerefentry> as usual
|
|
(or from any other process, using the
|
|
<citerefentry><refentrytitle>sd-journal</refentrytitle><manvolnum>3</manvolnum></citerefentry> API).
|
|
The relevant messages have <constant>MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1</constant>:</para>
|
|
<programlisting>$ journalctl MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1 -o verbose
|
|
…
|
|
MESSAGE_ID=fc2e22bc6ee647b6b90729ab34a250b1
|
|
COREDUMP_PID=552351
|
|
COREDUMP_UID=1000
|
|
COREDUMP_GID=1000
|
|
COREDUMP_SIGNAL_NAME=SIGSEGV
|
|
COREDUMP_SIGNAL=11
|
|
COREDUMP_TIMESTAMP=1614342930000000
|
|
COREDUMP_COMM=Web Content
|
|
COREDUMP_EXE=/usr/lib64/firefox/firefox
|
|
COREDUMP_USER_UNIT=app-gnome-firefox-552136.scope
|
|
COREDUMP_CMDLINE=/usr/lib64/firefox/firefox -contentproc -childID 5 -isForBrowser …
|
|
COREDUMP_CGROUP=/user.slice/user-1000.slice/user@1000.service/app.slice/app-….scope
|
|
COREDUMP_FILENAME=/var/lib/systemd/coredump/core.Web….552351.….zst
|
|
…
|
|
</programlisting>
|
|
|
|
<para>The following fields are saved (if known) with the journal entry</para>
|
|
|
|
<variablelist class='journal-directives'>
|
|
<varlistentry>
|
|
<term><varname>COREDUMP_UID=</varname></term>
|
|
<term><varname>COREDUMP_PID=</varname></term>
|
|
<term><varname>COREDUMP_GID=</varname></term>
|
|
<listitem><para>The process number (PID), owner user number (UID), and group number (GID) of the
|
|
crashed process.</para>
|
|
|
|
<para>When the crashed process was part of a container (or in a process or user namespace in
|
|
general), those are the values as seen <emphasis>outside</emphasis>, in the namespace where
|
|
<filename>systemd-coredump</filename> is running.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><varname>COREDUMP_TIMESTAMP=</varname></term>
|
|
<listitem><para>The time of the crash as reported by the kernel (in µs since the epoch).</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><varname>COREDUMP_RLIMIT=</varname></term>
|
|
<listitem><para>The core file size soft resource limit, see
|
|
<citerefentry project='man-pages'><refentrytitle>getrlimit</refentrytitle><manvolnum>2</manvolnum></citerefentry>.
|
|
</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><varname>COREDUMP_UNIT=</varname></term>
|
|
<term><varname>COREDUMP_SLICE=</varname></term>
|
|
<listitem><para>The system unit and slice names.</para>
|
|
|
|
<para>When the crashed process was in container, those are the units names
|
|
<emphasis>outside</emphasis>, in the main system manager.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><varname>COREDUMP_CGROUP=</varname></term>
|
|
<listitem><para>Control group information in the format used in
|
|
<filename>/proc/self/cgroup</filename>. On systems with the unified cgroup hierarchy, this is a
|
|
single path prefixed with <literal>0::</literal>, and multiple paths prefixed with controller numbers
|
|
on legacy systems.</para>
|
|
|
|
<para>When the crashed process was in a container, this is the full path, as seen outside of the
|
|
container.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><varname>COREDUMP_OWNER_UID=</varname></term>
|
|
<term><varname>COREDUMP_USER_UNIT=</varname></term>
|
|
<listitem><para>The numerical UID of the user owning the login session or systemd user unit of the
|
|
crashed process, and the user manager unit. Both fields are only present for user processes.
|
|
</para>
|
|
|
|
<para>When the crashed process was in container, those are the values <emphasis>outside</emphasis>,
|
|
in the main system.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><varname>COREDUMP_SIGNAL_NAME=</varname></term>
|
|
<term><varname>COREDUMP_SIGNAL=</varname></term>
|
|
|
|
<listitem><para>The terminating signal name (with the <literal>SIG</literal> prefix
|
|
<footnote><para><citerefentry
|
|
project='man-pages'><refentrytitle>kill</refentrytitle><manvolnum>1</manvolnum></citerefentry>
|
|
expects signal names <emphasis>without</emphasis> the prefix; <citerefentry
|
|
project='man-pages'><refentrytitle>kill</refentrytitle><manvolnum>2</manvolnum></citerefentry> uses
|
|
the prefix; all systemd tools accept signal names both with and without the prefix.
|
|
</para></footnote>) and numerical value. (Both are included because signal numbers vary by
|
|
architecture.)</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><varname>COREDUMP_CWD=</varname></term>
|
|
<term><varname>COREDUMP_ROOT=</varname></term>
|
|
|
|
<listitem><para>The current working directory and root directory of the crashed process.</para>
|
|
|
|
<para>When the crashed process is in a container, those paths are relative to the root of the
|
|
container's mount namespace.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><varname>COREDUMP_OPEN_FDS=</varname></term>
|
|
|
|
<listitem><para>Information about open file descriptors, in the following format:</para>
|
|
<programlisting><replaceable>fd</replaceable>:<replaceable>/path/to/file</replaceable>
|
|
pos: ...
|
|
flags: ...
|
|
...
|
|
|
|
<replaceable>fd</replaceable>:<replaceable>/path/to/file</replaceable>
|
|
pos: ...
|
|
flags: ...
|
|
...
|
|
</programlisting>
|
|
|
|
<para>The first line contains the file descriptor number <replaceable>fd</replaceable> and the path,
|
|
while subsequent lines show the contents of
|
|
<filename>/proc/<replaceable>pid</replaceable>/fdinfo/<replaceable>fd</replaceable></filename>.
|
|
</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><varname>COREDUMP_EXE=</varname></term>
|
|
|
|
<listitem><para>The destination of the <filename>/proc/<replaceable>pid</replaceable>/exe</filename>
|
|
symlink.</para>
|
|
|
|
<para>When the crashed process is in a container, that path is relative to the root of the
|
|
container's mount namespace.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><varname>COREDUMP_COMM=</varname></term>
|
|
<term><varname>COREDUMP_PROC_STATUS=</varname></term>
|
|
<term><varname>COREDUMP_PROC_MAPS=</varname></term>
|
|
<term><varname>COREDUMP_PROC_LIMITS=</varname></term>
|
|
<term><varname>COREDUMP_PROC_MOUNTINFO=</varname></term>
|
|
<term><varname>COREDUMP_ENVIRON=</varname></term>
|
|
|
|
<listitem><para>Fields that map the per-process entries in the <filename>/proc/</filename>
|
|
filesystem: <filename>/proc/<replaceable>pid</replaceable>/comm</filename> (the command name
|
|
associated with the process), <filename>/proc/<replaceable>pid</replaceable>/exe</filename> (the
|
|
filename of the executed command), <filename>/proc/<replaceable>pid</replaceable>/status</filename>
|
|
(various metadata about the process), <filename>/proc/<replaceable>pid</replaceable>/maps</filename>
|
|
(memory regions visible to the process and their access permissions),
|
|
<filename>/proc/<replaceable>pid</replaceable>/limits</filename> (the soft and hard resource limits),
|
|
<filename>/proc/<replaceable>pid</replaceable>/mountinfo</filename> (mount points in the process's
|
|
mount namespace), <filename>/proc/<replaceable>pid</replaceable>/environ</filename>
|
|
(the environment block of the crashed process).</para>
|
|
|
|
<para>See
|
|
<citerefentry project='man-pages'><refentrytitle>proc</refentrytitle><manvolnum>5</manvolnum></citerefentry>
|
|
for more information.</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><varname>COREDUMP_HOSTNAME=</varname></term>
|
|
|
|
<listitem><para>The system hostname.</para>
|
|
|
|
<para>When the crashed process was in container, this is the container hostname.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><varname>COREDUMP_CONTAINER_CMDLINE=</varname></term>
|
|
|
|
<listitem><para>For processes running in a container, the commandline of the process spawning the
|
|
container (the first parent process with a different mount namespace).</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><varname>COREDUMP=</varname></term>
|
|
|
|
<listitem><para>When the core is stored in the journal, the core image itself.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><varname>COREDUMP_FILENAME=</varname></term>
|
|
|
|
<listitem><para>When the core is stored externally, the path to the core file.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><varname>COREDUMP_TRUNCATED=</varname></term>
|
|
|
|
<listitem><para>Set to <literal>1</literal> when the saved coredump was truncated. (A partial core
|
|
image may still be processed by some tools, though obviously not all information is available.)
|
|
</para></listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><varname>COREDUMP_PACKAGE_NAME=</varname></term>
|
|
<term><varname>COREDUMP_PACKAGE_VERSION=</varname></term>
|
|
<term><varname>COREDUMP_PACKAGE_JSON=</varname></term>
|
|
|
|
<listitem><para>If the executable contained .package metadata ELF notes, they will be
|
|
parsed and attached. The <varname>package</varname> and <varname>packageVersion</varname>
|
|
of the 'main' ELF module (ie: the executable) will be appended individually. The
|
|
JSON-formatted content of all modules will be appended as a single JSON object, each with
|
|
the module name as the key. For more information about this metadata format and content, see
|
|
<ulink url="https://systemd.io/COREDUMP_PACKAGE_METADATA/">the coredump metadata spec</ulink>.</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
|
|
<varlistentry>
|
|
<term><varname>MESSAGE=</varname></term>
|
|
|
|
<listitem><para>The message generated by <command>systemd-coredump</command> that includes the
|
|
backtrace if it was successfully generated. When <command>systemd-coredump</command> is invoked with
|
|
<option>--backtrace</option>, this field is provided by the caller.</para></listitem>
|
|
</varlistentry>
|
|
</variablelist>
|
|
|
|
<para>Various other fields exist in the journal entry, but pertain to the logging process,
|
|
i.e. <command>systemd-coredump</command>, not the crashed process. See
|
|
<citerefentry><refentrytitle>systemd.journal-fields</refentrytitle><manvolnum>7</manvolnum></citerefentry>.
|
|
</para>
|
|
|
|
<para>The following fields are saved (if known) with the external file listed in
|
|
<varname>COREDUMP_FILENAME=</varname> as extended attributes:</para>
|
|
|
|
<variablelist>
|
|
<varlistentry>
|
|
<term><varname>user.coredump.pid</varname></term>
|
|
<term><varname>user.coredump.uid</varname></term>
|
|
<term><varname>user.coredump.gid</varname></term>
|
|
<term><varname>user.coredump.signal</varname></term>
|
|
<term><varname>user.coredump.timestamp</varname></term>
|
|
<term><varname>user.coredump.rlimit</varname></term>
|
|
<term><varname>user.coredump.hostname</varname></term>
|
|
<term><varname>user.coredump.comm</varname></term>
|
|
<term><varname>user.coredump.exe</varname></term>
|
|
|
|
<listitem><para>Those are the same as <varname>COREDUMP_PID=</varname>,
|
|
<varname>COREDUMP_UID=</varname>, <varname>COREDUMP_GID=</varname>,
|
|
<varname>COREDUMP_SIGNAL=</varname>, <varname>COREDUMP_TIMESTAMP=</varname>,
|
|
<varname>COREDUMP_RLIMIT=</varname>, <varname>COREDUMP_HOSTNAME=</varname>,
|
|
<varname>COREDUMP_COMM=</varname>, and <varname>COREDUMP_EXE=</varname>, described above.
|
|
</para></listitem>
|
|
</varlistentry>
|
|
</variablelist>
|
|
|
|
<para>Those can be viewed using
|
|
<citerefentry project='man-pages'><refentrytitle>getfattr</refentrytitle><manvolnum>1</manvolnum></citerefentry>.
|
|
For the core file described in the journal entry shown above:
|
|
<programlisting>$ getfattr --absolute-names -d /var/lib/systemd/coredump/core.Web….552351.….zst
|
|
# file: /var/lib/systemd/coredump/core.Web….552351.….zst
|
|
user.coredump.pid="552351"
|
|
user.coredump.uid="1000"
|
|
user.coredump.gid="1000"
|
|
user.coredump.signal="11"
|
|
user.coredump.timestamp="1614342930000000"
|
|
user.coredump.comm="Web Content"
|
|
user.coredump.exe="/usr/lib64/firefox/firefox"
|
|
…
|
|
</programlisting>
|
|
</para>
|
|
</refsect1>
|
|
|
|
<refsect1>
|
|
<title>See Also</title>
|
|
<para>
|
|
<citerefentry><refentrytitle>coredump.conf</refentrytitle><manvolnum>5</manvolnum></citerefentry>,
|
|
<citerefentry><refentrytitle>coredumpctl</refentrytitle><manvolnum>1</manvolnum></citerefentry>,
|
|
<citerefentry><refentrytitle>systemd-journald.service</refentrytitle><manvolnum>8</manvolnum></citerefentry>,
|
|
<citerefentry><refentrytitle>systemd-tmpfiles</refentrytitle><manvolnum>8</manvolnum></citerefentry>,
|
|
<citerefentry project='man-pages'><refentrytitle>core</refentrytitle><manvolnum>5</manvolnum></citerefentry>,
|
|
<citerefentry><refentrytitle>sysctl.d</refentrytitle><manvolnum>5</manvolnum></citerefentry>,
|
|
<citerefentry><refentrytitle>systemd-sysctl.service</refentrytitle><manvolnum>8</manvolnum></citerefentry>.
|
|
</para>
|
|
</refsect1>
|
|
</refentry>
|