mirror of
https://mirrors.bfsu.edu.cn/git/linux.git
synced 2024-11-27 22:24:11 +08:00
4fbd0a81a0
* KDB: improved searching
* No longer enter debug core on panic if panic timeout is set
KGDB/KDB regressions / cleanups
* fix pdf doc build errors
* prevent junk characters on kdb console from printk levels
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
iQIcBAABAgAGBQJU5qxHAAoJEIciOldedpOjliYP/izuoNZ/EtjjeihOL44ic0o0
cmvdSc/ovR/mO4fbDpftMB0nhzclgRyAvr+VTPd3Bp5Poh+wJ0ZKu1R7f+ioSN73
Y4ek9PJqPSBQr+JdfPK80N56Choeni48bsC6up12i3BTfXobj81zlu4Sj0SMOoHq
IkFkB7soRuiFoc5IkKMvf3N3T9j1PnEULmHteNDRr0hTmGipEzkD3zocc/bRFV/l
JTZRzIMGBNGnF01DPLDcuvbu0wGBh6ADMBLtx5v1UrhV32ypfJq2bgxFvgM/AXn2
3VG4HcRbVsGmlBOahFR6X0DE/WAplw01yu1EabR2EWUePyz41cnSkxl4nR/NNhwz
qMbr3uzu1iWUTTz5ySRcWxSuRRCihVQqNk6p+y21N/jY/5cr2jI03qJm0zZ/ObqL
VUcPE7CfdcriCDXoepgXZE4XfX65Jf5tUiyiCj+1ds05ab5qHELIwKOZdjU2ON1b
pb2ElPngGSEEoU/eSDgP2RVJ9Mk/k5s2TxaPXVJNkeWGNxPU5HtCytZpVI5hckbP
/NZWTtyUDZ85is8cWUkHEdjnQ+CdzaA/FwJEqnB0or2is91mo8uBxP5BvdqPnPL0
QdPPnVgD72dumXfJpH2HY3DdUs24LaP0vgSO8ELKgfA67nprS+5xztNSd8ekNnhF
4wMhZbuAhB68E6bA0X7G
=TH0R
-----END PGP SIGNATURE-----
Merge tag 'for_linux-3.20-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/jwessel/kgdb
Pull kgdb/kdb updates from Jason Wessel:
"KGDB/KDB New:
- KDB: improved searching
- No longer enter debug core on panic if panic timeout is set
KGDB/KDB regressions / cleanups
- fix pdf doc build errors
- prevent junk characters on kdb console from printk levels"
* tag 'for_linux-3.20-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/jwessel/kgdb:
kgdb, docs: Fix <para> pdfdocs build errors
debug: prevent entering debug mode on panic/exception.
kdb: Const qualifier for kdb_getstr's prompt argument
kdb: Provide forward search at more prompt
kdb: Fix a prompt management bug when using | grep
kdb: Remove stack dump when entering kgdb due to NMI
kdb: Avoid printing KERN_ levels to consoles
kdb: Fix off by one error in kdb_cpu()
kdb: fix incorrect counts in KDB summary command output
919 lines
42 KiB
XML
919 lines
42 KiB
XML
<?xml version="1.0" encoding="UTF-8"?>
|
|
<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.1.2//EN"
|
|
"http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd" []>
|
|
|
|
<book id="kgdbOnLinux">
|
|
<bookinfo>
|
|
<title>Using kgdb, kdb and the kernel debugger internals</title>
|
|
|
|
<authorgroup>
|
|
<author>
|
|
<firstname>Jason</firstname>
|
|
<surname>Wessel</surname>
|
|
<affiliation>
|
|
<address>
|
|
<email>jason.wessel@windriver.com</email>
|
|
</address>
|
|
</affiliation>
|
|
</author>
|
|
</authorgroup>
|
|
<copyright>
|
|
<year>2008,2010</year>
|
|
<holder>Wind River Systems, Inc.</holder>
|
|
</copyright>
|
|
<copyright>
|
|
<year>2004-2005</year>
|
|
<holder>MontaVista Software, Inc.</holder>
|
|
</copyright>
|
|
<copyright>
|
|
<year>2004</year>
|
|
<holder>Amit S. Kale</holder>
|
|
</copyright>
|
|
|
|
<legalnotice>
|
|
<para>
|
|
This file is licensed under the terms of the GNU General Public License
|
|
version 2. This program is licensed "as is" without any warranty of any
|
|
kind, whether express or implied.
|
|
</para>
|
|
|
|
</legalnotice>
|
|
</bookinfo>
|
|
|
|
<toc></toc>
|
|
<chapter id="Introduction">
|
|
<title>Introduction</title>
|
|
<para>
|
|
The kernel has two different debugger front ends (kdb and kgdb)
|
|
which interface to the debug core. It is possible to use either
|
|
of the debugger front ends and dynamically transition between them
|
|
if you configure the kernel properly at compile and runtime.
|
|
</para>
|
|
<para>
|
|
Kdb is simplistic shell-style interface which you can use on a
|
|
system console with a keyboard or serial console. You can use it
|
|
to inspect memory, registers, process lists, dmesg, and even set
|
|
breakpoints to stop in a certain location. Kdb is not a source
|
|
level debugger, although you can set breakpoints and execute some
|
|
basic kernel run control. Kdb is mainly aimed at doing some
|
|
analysis to aid in development or diagnosing kernel problems. You
|
|
can access some symbols by name in kernel built-ins or in kernel
|
|
modules if the code was built
|
|
with <symbol>CONFIG_KALLSYMS</symbol>.
|
|
</para>
|
|
<para>
|
|
Kgdb is intended to be used as a source level debugger for the
|
|
Linux kernel. It is used along with gdb to debug a Linux kernel.
|
|
The expectation is that gdb can be used to "break in" to the
|
|
kernel to inspect memory, variables and look through call stack
|
|
information similar to the way an application developer would use
|
|
gdb to debug an application. It is possible to place breakpoints
|
|
in kernel code and perform some limited execution stepping.
|
|
</para>
|
|
<para>
|
|
Two machines are required for using kgdb. One of these machines is
|
|
a development machine and the other is the target machine. The
|
|
kernel to be debugged runs on the target machine. The development
|
|
machine runs an instance of gdb against the vmlinux file which
|
|
contains the symbols (not a boot image such as bzImage, zImage,
|
|
uImage...). In gdb the developer specifies the connection
|
|
parameters and connects to kgdb. The type of connection a
|
|
developer makes with gdb depends on the availability of kgdb I/O
|
|
modules compiled as built-ins or loadable kernel modules in the test
|
|
machine's kernel.
|
|
</para>
|
|
</chapter>
|
|
<chapter id="CompilingAKernel">
|
|
<title>Compiling a kernel</title>
|
|
<para>
|
|
<itemizedlist>
|
|
<listitem><para>In order to enable compilation of kdb, you must first enable kgdb.</para></listitem>
|
|
<listitem><para>The kgdb test compile options are described in the kgdb test suite chapter.</para></listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
<sect1 id="CompileKGDB">
|
|
<title>Kernel config options for kgdb</title>
|
|
<para>
|
|
To enable <symbol>CONFIG_KGDB</symbol> you should look under
|
|
"Kernel hacking" / "Kernel debugging" and select "KGDB: kernel debugger".
|
|
</para>
|
|
<para>
|
|
While it is not a hard requirement that you have symbols in your
|
|
vmlinux file, gdb tends not to be very useful without the symbolic
|
|
data, so you will want to turn
|
|
on <symbol>CONFIG_DEBUG_INFO</symbol> which is called "Compile the
|
|
kernel with debug info" in the config menu.
|
|
</para>
|
|
<para>
|
|
It is advised, but not required, that you turn on the
|
|
<symbol>CONFIG_FRAME_POINTER</symbol> kernel option which is called "Compile the
|
|
kernel with frame pointers" in the config menu. This option
|
|
inserts code to into the compiled executable which saves the frame
|
|
information in registers or on the stack at different points which
|
|
allows a debugger such as gdb to more accurately construct
|
|
stack back traces while debugging the kernel.
|
|
</para>
|
|
<para>
|
|
If the architecture that you are using supports the kernel option
|
|
CONFIG_DEBUG_RODATA, you should consider turning it off. This
|
|
option will prevent the use of software breakpoints because it
|
|
marks certain regions of the kernel's memory space as read-only.
|
|
If kgdb supports it for the architecture you are using, you can
|
|
use hardware breakpoints if you desire to run with the
|
|
CONFIG_DEBUG_RODATA option turned on, else you need to turn off
|
|
this option.
|
|
</para>
|
|
<para>
|
|
Next you should choose one of more I/O drivers to interconnect
|
|
debugging host and debugged target. Early boot debugging requires
|
|
a KGDB I/O driver that supports early debugging and the driver
|
|
must be built into the kernel directly. Kgdb I/O driver
|
|
configuration takes place via kernel or module parameters which
|
|
you can learn more about in the in the section that describes the
|
|
parameter "kgdboc".
|
|
</para>
|
|
<para>Here is an example set of .config symbols to enable or
|
|
disable for kgdb:
|
|
<itemizedlist>
|
|
<listitem><para># CONFIG_DEBUG_RODATA is not set</para></listitem>
|
|
<listitem><para>CONFIG_FRAME_POINTER=y</para></listitem>
|
|
<listitem><para>CONFIG_KGDB=y</para></listitem>
|
|
<listitem><para>CONFIG_KGDB_SERIAL_CONSOLE=y</para></listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
</sect1>
|
|
<sect1 id="CompileKDB">
|
|
<title>Kernel config options for kdb</title>
|
|
<para>Kdb is quite a bit more complex than the simple gdbstub
|
|
sitting on top of the kernel's debug core. Kdb must implement a
|
|
shell, and also adds some helper functions in other parts of the
|
|
kernel, responsible for printing out interesting data such as what
|
|
you would see if you ran "lsmod", or "ps". In order to build kdb
|
|
into the kernel you follow the same steps as you would for kgdb.
|
|
</para>
|
|
<para>The main config option for kdb
|
|
is <symbol>CONFIG_KGDB_KDB</symbol> which is called "KGDB_KDB:
|
|
include kdb frontend for kgdb" in the config menu. In theory you
|
|
would have already also selected an I/O driver such as the
|
|
CONFIG_KGDB_SERIAL_CONSOLE interface if you plan on using kdb on a
|
|
serial port, when you were configuring kgdb.
|
|
</para>
|
|
<para>If you want to use a PS/2-style keyboard with kdb, you would
|
|
select CONFIG_KDB_KEYBOARD which is called "KGDB_KDB: keyboard as
|
|
input device" in the config menu. The CONFIG_KDB_KEYBOARD option
|
|
is not used for anything in the gdb interface to kgdb. The
|
|
CONFIG_KDB_KEYBOARD option only works with kdb.
|
|
</para>
|
|
<para>Here is an example set of .config symbols to enable/disable kdb:
|
|
<itemizedlist>
|
|
<listitem><para># CONFIG_DEBUG_RODATA is not set</para></listitem>
|
|
<listitem><para>CONFIG_FRAME_POINTER=y</para></listitem>
|
|
<listitem><para>CONFIG_KGDB=y</para></listitem>
|
|
<listitem><para>CONFIG_KGDB_SERIAL_CONSOLE=y</para></listitem>
|
|
<listitem><para>CONFIG_KGDB_KDB=y</para></listitem>
|
|
<listitem><para>CONFIG_KDB_KEYBOARD=y</para></listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
</sect1>
|
|
</chapter>
|
|
<chapter id="kgdbKernelArgs">
|
|
<title>Kernel Debugger Boot Arguments</title>
|
|
<para>This section describes the various runtime kernel
|
|
parameters that affect the configuration of the kernel debugger.
|
|
The following chapter covers using kdb and kgdb as well as
|
|
providing some examples of the configuration parameters.</para>
|
|
<sect1 id="kgdboc">
|
|
<title>Kernel parameter: kgdboc</title>
|
|
<para>The kgdboc driver was originally an abbreviation meant to
|
|
stand for "kgdb over console". Today it is the primary mechanism
|
|
to configure how to communicate from gdb to kgdb as well as the
|
|
devices you want to use to interact with the kdb shell.
|
|
</para>
|
|
<para>For kgdb/gdb, kgdboc is designed to work with a single serial
|
|
port. It is intended to cover the circumstance where you want to
|
|
use a serial console as your primary console as well as using it to
|
|
perform kernel debugging. It is also possible to use kgdb on a
|
|
serial port which is not designated as a system console. Kgdboc
|
|
may be configured as a kernel built-in or a kernel loadable module.
|
|
You can only make use of <constant>kgdbwait</constant> and early
|
|
debugging if you build kgdboc into the kernel as a built-in.
|
|
</para>
|
|
<para>Optionally you can elect to activate kms (Kernel Mode
|
|
Setting) integration. When you use kms with kgdboc and you have a
|
|
video driver that has atomic mode setting hooks, it is possible to
|
|
enter the debugger on the graphics console. When the kernel
|
|
execution is resumed, the previous graphics mode will be restored.
|
|
This integration can serve as a useful tool to aid in diagnosing
|
|
crashes or doing analysis of memory with kdb while allowing the
|
|
full graphics console applications to run.
|
|
</para>
|
|
<sect2 id="kgdbocArgs">
|
|
<title>kgdboc arguments</title>
|
|
<para>Usage: <constant>kgdboc=[kms][[,]kbd][[,]serial_device][,baud]</constant></para>
|
|
<para>The order listed above must be observed if you use any of the
|
|
optional configurations together.
|
|
</para>
|
|
<para>Abbreviations:
|
|
<itemizedlist>
|
|
<listitem><para>kms = Kernel Mode Setting</para></listitem>
|
|
<listitem><para>kbd = Keyboard</para></listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
<para>You can configure kgdboc to use the keyboard, and/or a serial
|
|
device depending on if you are using kdb and/or kgdb, in one of the
|
|
following scenarios. The order listed above must be observed if
|
|
you use any of the optional configurations together. Using kms +
|
|
only gdb is generally not a useful combination.</para>
|
|
<sect3 id="kgdbocArgs1">
|
|
<title>Using loadable module or built-in</title>
|
|
<para>
|
|
<orderedlist>
|
|
<listitem><para>As a kernel built-in:</para>
|
|
<para>Use the kernel boot argument: <constant>kgdboc=<tty-device>,[baud]</constant></para></listitem>
|
|
<listitem>
|
|
<para>As a kernel loadable module:</para>
|
|
<para>Use the command: <constant>modprobe kgdboc kgdboc=<tty-device>,[baud]</constant></para>
|
|
<para>Here are two examples of how you might format the kgdboc
|
|
string. The first is for an x86 target using the first serial port.
|
|
The second example is for the ARM Versatile AB using the second
|
|
serial port.
|
|
<orderedlist>
|
|
<listitem><para><constant>kgdboc=ttyS0,115200</constant></para></listitem>
|
|
<listitem><para><constant>kgdboc=ttyAMA1,115200</constant></para></listitem>
|
|
</orderedlist>
|
|
</para>
|
|
</listitem>
|
|
</orderedlist></para>
|
|
</sect3>
|
|
<sect3 id="kgdbocArgs2">
|
|
<title>Configure kgdboc at runtime with sysfs</title>
|
|
<para>At run time you can enable or disable kgdboc by echoing a
|
|
parameters into the sysfs. Here are two examples:</para>
|
|
<orderedlist>
|
|
<listitem><para>Enable kgdboc on ttyS0</para>
|
|
<para><constant>echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc</constant></para></listitem>
|
|
<listitem><para>Disable kgdboc</para>
|
|
<para><constant>echo "" > /sys/module/kgdboc/parameters/kgdboc</constant></para></listitem>
|
|
</orderedlist>
|
|
<para>NOTE: You do not need to specify the baud if you are
|
|
configuring the console on tty which is already configured or
|
|
open.</para>
|
|
</sect3>
|
|
<sect3 id="kgdbocArgs3">
|
|
<title>More examples</title>
|
|
<para>You can configure kgdboc to use the keyboard, and/or a serial device
|
|
depending on if you are using kdb and/or kgdb, in one of the
|
|
following scenarios.
|
|
<orderedlist>
|
|
<listitem><para>kdb and kgdb over only a serial port</para>
|
|
<para><constant>kgdboc=<serial_device>[,baud]</constant></para>
|
|
<para>Example: <constant>kgdboc=ttyS0,115200</constant></para>
|
|
</listitem>
|
|
<listitem><para>kdb and kgdb with keyboard and a serial port</para>
|
|
<para><constant>kgdboc=kbd,<serial_device>[,baud]</constant></para>
|
|
<para>Example: <constant>kgdboc=kbd,ttyS0,115200</constant></para>
|
|
</listitem>
|
|
<listitem><para>kdb with a keyboard</para>
|
|
<para><constant>kgdboc=kbd</constant></para>
|
|
</listitem>
|
|
<listitem><para>kdb with kernel mode setting</para>
|
|
<para><constant>kgdboc=kms,kbd</constant></para>
|
|
</listitem>
|
|
<listitem><para>kdb with kernel mode setting and kgdb over a serial port</para>
|
|
<para><constant>kgdboc=kms,kbd,ttyS0,115200</constant></para>
|
|
</listitem>
|
|
</orderedlist>
|
|
</para>
|
|
<para>NOTE: Kgdboc does not support interrupting the target via the
|
|
gdb remote protocol. You must manually send a sysrq-g unless you
|
|
have a proxy that splits console output to a terminal program.
|
|
A console proxy has a separate TCP port for the debugger and a separate
|
|
TCP port for the "human" console. The proxy can take care of sending
|
|
the sysrq-g for you.
|
|
</para>
|
|
<para>When using kgdboc with no debugger proxy, you can end up
|
|
connecting the debugger at one of two entry points. If an
|
|
exception occurs after you have loaded kgdboc, a message should
|
|
print on the console stating it is waiting for the debugger. In
|
|
this case you disconnect your terminal program and then connect the
|
|
debugger in its place. If you want to interrupt the target system
|
|
and forcibly enter a debug session you have to issue a Sysrq
|
|
sequence and then type the letter <constant>g</constant>. Then
|
|
you disconnect the terminal session and connect gdb. Your options
|
|
if you don't like this are to hack gdb to send the sysrq-g for you
|
|
as well as on the initial connect, or to use a debugger proxy that
|
|
allows an unmodified gdb to do the debugging.
|
|
</para>
|
|
</sect3>
|
|
</sect2>
|
|
</sect1>
|
|
<sect1 id="kgdbwait">
|
|
<title>Kernel parameter: kgdbwait</title>
|
|
<para>
|
|
The Kernel command line option <constant>kgdbwait</constant> makes
|
|
kgdb wait for a debugger connection during booting of a kernel. You
|
|
can only use this option if you compiled a kgdb I/O driver into the
|
|
kernel and you specified the I/O driver configuration as a kernel
|
|
command line option. The kgdbwait parameter should always follow the
|
|
configuration parameter for the kgdb I/O driver in the kernel
|
|
command line else the I/O driver will not be configured prior to
|
|
asking the kernel to use it to wait.
|
|
</para>
|
|
<para>
|
|
The kernel will stop and wait as early as the I/O driver and
|
|
architecture allows when you use this option. If you build the
|
|
kgdb I/O driver as a loadable kernel module kgdbwait will not do
|
|
anything.
|
|
</para>
|
|
</sect1>
|
|
<sect1 id="kgdbcon">
|
|
<title>Kernel parameter: kgdbcon</title>
|
|
<para> The kgdbcon feature allows you to see printk() messages
|
|
inside gdb while gdb is connected to the kernel. Kdb does not make
|
|
use of the kgdbcon feature.
|
|
</para>
|
|
<para>Kgdb supports using the gdb serial protocol to send console
|
|
messages to the debugger when the debugger is connected and running.
|
|
There are two ways to activate this feature.
|
|
<orderedlist>
|
|
<listitem><para>Activate with the kernel command line option:</para>
|
|
<para><constant>kgdbcon</constant></para>
|
|
</listitem>
|
|
<listitem><para>Use sysfs before configuring an I/O driver</para>
|
|
<para>
|
|
<constant>echo 1 > /sys/module/kgdb/parameters/kgdb_use_con</constant>
|
|
</para>
|
|
<para>
|
|
NOTE: If you do this after you configure the kgdb I/O driver, the
|
|
setting will not take effect until the next point the I/O is
|
|
reconfigured.
|
|
</para>
|
|
</listitem>
|
|
</orderedlist>
|
|
</para>
|
|
<para>IMPORTANT NOTE: You cannot use kgdboc + kgdbcon on a tty that is an
|
|
active system console. An example of incorrect usage is <constant>console=ttyS0,115200 kgdboc=ttyS0 kgdbcon</constant>
|
|
</para>
|
|
<para>It is possible to use this option with kgdboc on a tty that is not a system console.
|
|
</para>
|
|
</sect1>
|
|
<sect1 id="kgdbreboot">
|
|
<title>Run time parameter: kgdbreboot</title>
|
|
<para> The kgdbreboot feature allows you to change how the debugger
|
|
deals with the reboot notification. You have 3 choices for the
|
|
behavior. The default behavior is always set to 0.</para>
|
|
<orderedlist>
|
|
<listitem><para>echo -1 > /sys/module/debug_core/parameters/kgdbreboot</para>
|
|
<para>Ignore the reboot notification entirely.</para>
|
|
</listitem>
|
|
<listitem><para>echo 0 > /sys/module/debug_core/parameters/kgdbreboot</para>
|
|
<para>Send the detach message to any attached debugger client.</para>
|
|
</listitem>
|
|
<listitem><para>echo 1 > /sys/module/debug_core/parameters/kgdbreboot</para>
|
|
<para>Enter the debugger on reboot notify.</para>
|
|
</listitem>
|
|
</orderedlist>
|
|
</sect1>
|
|
</chapter>
|
|
<chapter id="usingKDB">
|
|
<title>Using kdb</title>
|
|
<para>
|
|
</para>
|
|
<sect1 id="quickKDBserial">
|
|
<title>Quick start for kdb on a serial port</title>
|
|
<para>This is a quick example of how to use kdb.</para>
|
|
<para><orderedlist>
|
|
<listitem><para>Configure kgdboc at boot using kernel parameters:
|
|
<itemizedlist>
|
|
<listitem><para><constant>console=ttyS0,115200 kgdboc=ttyS0,115200</constant></para></listitem>
|
|
</itemizedlist></para>
|
|
<para>OR</para>
|
|
<para>Configure kgdboc after the kernel has booted; assuming you are using a serial port console:
|
|
<itemizedlist>
|
|
<listitem><para><constant>echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc</constant></para></listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
</listitem>
|
|
<listitem><para>Enter the kernel debugger manually or by waiting for an oops or fault. There are several ways you can enter the kernel debugger manually; all involve using the sysrq-g, which means you must have enabled CONFIG_MAGIC_SYSRQ=y in your kernel config.</para>
|
|
<itemizedlist>
|
|
<listitem><para>When logged in as root or with a super user session you can run:</para>
|
|
<para><constant>echo g > /proc/sysrq-trigger</constant></para></listitem>
|
|
<listitem><para>Example using minicom 2.2</para>
|
|
<para>Press: <constant>Control-a</constant></para>
|
|
<para>Press: <constant>f</constant></para>
|
|
<para>Press: <constant>g</constant></para>
|
|
</listitem>
|
|
<listitem><para>When you have telneted to a terminal server that supports sending a remote break</para>
|
|
<para>Press: <constant>Control-]</constant></para>
|
|
<para>Type in:<constant>send break</constant></para>
|
|
<para>Press: <constant>Enter</constant></para>
|
|
<para>Press: <constant>g</constant></para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</listitem>
|
|
<listitem><para>From the kdb prompt you can run the "help" command to see a complete list of the commands that are available.</para>
|
|
<para>Some useful commands in kdb include:
|
|
<itemizedlist>
|
|
<listitem><para>lsmod -- Shows where kernel modules are loaded</para></listitem>
|
|
<listitem><para>ps -- Displays only the active processes</para></listitem>
|
|
<listitem><para>ps A -- Shows all the processes</para></listitem>
|
|
<listitem><para>summary -- Shows kernel version info and memory usage</para></listitem>
|
|
<listitem><para>bt -- Get a backtrace of the current process using dump_stack()</para></listitem>
|
|
<listitem><para>dmesg -- View the kernel syslog buffer</para></listitem>
|
|
<listitem><para>go -- Continue the system</para></listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>When you are done using kdb you need to consider rebooting the
|
|
system or using the "go" command to resuming normal kernel
|
|
execution. If you have paused the kernel for a lengthy period of
|
|
time, applications that rely on timely networking or anything to do
|
|
with real wall clock time could be adversely affected, so you
|
|
should take this into consideration when using the kernel
|
|
debugger.</para>
|
|
</listitem>
|
|
</orderedlist></para>
|
|
</sect1>
|
|
<sect1 id="quickKDBkeyboard">
|
|
<title>Quick start for kdb using a keyboard connected console</title>
|
|
<para>This is a quick example of how to use kdb with a keyboard.</para>
|
|
<para><orderedlist>
|
|
<listitem><para>Configure kgdboc at boot using kernel parameters:
|
|
<itemizedlist>
|
|
<listitem><para><constant>kgdboc=kbd</constant></para></listitem>
|
|
</itemizedlist></para>
|
|
<para>OR</para>
|
|
<para>Configure kgdboc after the kernel has booted:
|
|
<itemizedlist>
|
|
<listitem><para><constant>echo kbd > /sys/module/kgdboc/parameters/kgdboc</constant></para></listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
</listitem>
|
|
<listitem><para>Enter the kernel debugger manually or by waiting for an oops or fault. There are several ways you can enter the kernel debugger manually; all involve using the sysrq-g, which means you must have enabled CONFIG_MAGIC_SYSRQ=y in your kernel config.</para>
|
|
<itemizedlist>
|
|
<listitem><para>When logged in as root or with a super user session you can run:</para>
|
|
<para><constant>echo g > /proc/sysrq-trigger</constant></para></listitem>
|
|
<listitem><para>Example using a laptop keyboard</para>
|
|
<para>Press and hold down: <constant>Alt</constant></para>
|
|
<para>Press and hold down: <constant>Fn</constant></para>
|
|
<para>Press and release the key with the label: <constant>SysRq</constant></para>
|
|
<para>Release: <constant>Fn</constant></para>
|
|
<para>Press and release: <constant>g</constant></para>
|
|
<para>Release: <constant>Alt</constant></para>
|
|
</listitem>
|
|
<listitem><para>Example using a PS/2 101-key keyboard</para>
|
|
<para>Press and hold down: <constant>Alt</constant></para>
|
|
<para>Press and release the key with the label: <constant>SysRq</constant></para>
|
|
<para>Press and release: <constant>g</constant></para>
|
|
<para>Release: <constant>Alt</constant></para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</listitem>
|
|
<listitem>
|
|
<para>Now type in a kdb command such as "help", "dmesg", "bt" or "go" to continue kernel execution.</para>
|
|
</listitem>
|
|
</orderedlist></para>
|
|
</sect1>
|
|
</chapter>
|
|
<chapter id="EnableKGDB">
|
|
<title>Using kgdb / gdb</title>
|
|
<para>In order to use kgdb you must activate it by passing
|
|
configuration information to one of the kgdb I/O drivers. If you
|
|
do not pass any configuration information kgdb will not do anything
|
|
at all. Kgdb will only actively hook up to the kernel trap hooks
|
|
if a kgdb I/O driver is loaded and configured. If you unconfigure
|
|
a kgdb I/O driver, kgdb will unregister all the kernel hook points.
|
|
</para>
|
|
<para> All kgdb I/O drivers can be reconfigured at run time, if
|
|
<symbol>CONFIG_SYSFS</symbol> and <symbol>CONFIG_MODULES</symbol>
|
|
are enabled, by echo'ing a new config string to
|
|
<constant>/sys/module/<driver>/parameter/<option></constant>.
|
|
The driver can be unconfigured by passing an empty string. You cannot
|
|
change the configuration while the debugger is attached. Make sure
|
|
to detach the debugger with the <constant>detach</constant> command
|
|
prior to trying to unconfigure a kgdb I/O driver.
|
|
</para>
|
|
<sect1 id="ConnectingGDB">
|
|
<title>Connecting with gdb to a serial port</title>
|
|
<orderedlist>
|
|
<listitem><para>Configure kgdboc</para>
|
|
<para>Configure kgdboc at boot using kernel parameters:
|
|
<itemizedlist>
|
|
<listitem><para><constant>kgdboc=ttyS0,115200</constant></para></listitem>
|
|
</itemizedlist></para>
|
|
<para>OR</para>
|
|
<para>Configure kgdboc after the kernel has booted:
|
|
<itemizedlist>
|
|
<listitem><para><constant>echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc</constant></para></listitem>
|
|
</itemizedlist></para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>Stop kernel execution (break into the debugger)</para>
|
|
<para>In order to connect to gdb via kgdboc, the kernel must
|
|
first be stopped. There are several ways to stop the kernel which
|
|
include using kgdbwait as a boot argument, via a sysrq-g, or running
|
|
the kernel until it takes an exception where it waits for the
|
|
debugger to attach.
|
|
<itemizedlist>
|
|
<listitem><para>When logged in as root or with a super user session you can run:</para>
|
|
<para><constant>echo g > /proc/sysrq-trigger</constant></para></listitem>
|
|
<listitem><para>Example using minicom 2.2</para>
|
|
<para>Press: <constant>Control-a</constant></para>
|
|
<para>Press: <constant>f</constant></para>
|
|
<para>Press: <constant>g</constant></para>
|
|
</listitem>
|
|
<listitem><para>When you have telneted to a terminal server that supports sending a remote break</para>
|
|
<para>Press: <constant>Control-]</constant></para>
|
|
<para>Type in:<constant>send break</constant></para>
|
|
<para>Press: <constant>Enter</constant></para>
|
|
<para>Press: <constant>g</constant></para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>Connect from gdb</para>
|
|
<para>
|
|
Example (using a directly connected port):
|
|
</para>
|
|
<programlisting>
|
|
% gdb ./vmlinux
|
|
(gdb) set remotebaud 115200
|
|
(gdb) target remote /dev/ttyS0
|
|
</programlisting>
|
|
<para>
|
|
Example (kgdb to a terminal server on TCP port 2012):
|
|
</para>
|
|
<programlisting>
|
|
% gdb ./vmlinux
|
|
(gdb) target remote 192.168.2.2:2012
|
|
</programlisting>
|
|
<para>
|
|
Once connected, you can debug a kernel the way you would debug an
|
|
application program.
|
|
</para>
|
|
<para>
|
|
If you are having problems connecting or something is going
|
|
seriously wrong while debugging, it will most often be the case
|
|
that you want to enable gdb to be verbose about its target
|
|
communications. You do this prior to issuing the <constant>target
|
|
remote</constant> command by typing in: <constant>set debug remote 1</constant>
|
|
</para>
|
|
</listitem>
|
|
</orderedlist>
|
|
<para>Remember if you continue in gdb, and need to "break in" again,
|
|
you need to issue an other sysrq-g. It is easy to create a simple
|
|
entry point by putting a breakpoint at <constant>sys_sync</constant>
|
|
and then you can run "sync" from a shell or script to break into the
|
|
debugger.</para>
|
|
</sect1>
|
|
</chapter>
|
|
<chapter id="switchKdbKgdb">
|
|
<title>kgdb and kdb interoperability</title>
|
|
<para>It is possible to transition between kdb and kgdb dynamically.
|
|
The debug core will remember which you used the last time and
|
|
automatically start in the same mode.</para>
|
|
<sect1>
|
|
<title>Switching between kdb and kgdb</title>
|
|
<sect2>
|
|
<title>Switching from kgdb to kdb</title>
|
|
<para>
|
|
There are two ways to switch from kgdb to kdb: you can use gdb to
|
|
issue a maintenance packet, or you can blindly type the command $3#33.
|
|
Whenever the kernel debugger stops in kgdb mode it will print the
|
|
message <constant>KGDB or $3#33 for KDB</constant>. It is important
|
|
to note that you have to type the sequence correctly in one pass.
|
|
You cannot type a backspace or delete because kgdb will interpret
|
|
that as part of the debug stream.
|
|
<orderedlist>
|
|
<listitem><para>Change from kgdb to kdb by blindly typing:</para>
|
|
<para><constant>$3#33</constant></para></listitem>
|
|
<listitem><para>Change from kgdb to kdb with gdb</para>
|
|
<para><constant>maintenance packet 3</constant></para>
|
|
<para>NOTE: Now you must kill gdb. Typically you press control-z and
|
|
issue the command: kill -9 %</para></listitem>
|
|
</orderedlist>
|
|
</para>
|
|
</sect2>
|
|
<sect2>
|
|
<title>Change from kdb to kgdb</title>
|
|
<para>There are two ways you can change from kdb to kgdb. You can
|
|
manually enter kgdb mode by issuing the kgdb command from the kdb
|
|
shell prompt, or you can connect gdb while the kdb shell prompt is
|
|
active. The kdb shell looks for the typical first commands that gdb
|
|
would issue with the gdb remote protocol and if it sees one of those
|
|
commands it automatically changes into kgdb mode.</para>
|
|
<orderedlist>
|
|
<listitem><para>From kdb issue the command:</para>
|
|
<para><constant>kgdb</constant></para>
|
|
<para>Now disconnect your terminal program and connect gdb in its place</para></listitem>
|
|
<listitem><para>At the kdb prompt, disconnect the terminal program and connect gdb in its place.</para></listitem>
|
|
</orderedlist>
|
|
</sect2>
|
|
</sect1>
|
|
<sect1>
|
|
<title>Running kdb commands from gdb</title>
|
|
<para>It is possible to run a limited set of kdb commands from gdb,
|
|
using the gdb monitor command. You don't want to execute any of the
|
|
run control or breakpoint operations, because it can disrupt the
|
|
state of the kernel debugger. You should be using gdb for
|
|
breakpoints and run control operations if you have gdb connected.
|
|
The more useful commands to run are things like lsmod, dmesg, ps or
|
|
possibly some of the memory information commands. To see all the kdb
|
|
commands you can run <constant>monitor help</constant>.</para>
|
|
<para>Example:
|
|
<informalexample><programlisting>
|
|
(gdb) monitor ps
|
|
1 idle process (state I) and
|
|
27 sleeping system daemon (state M) processes suppressed,
|
|
use 'ps A' to see all.
|
|
Task Addr Pid Parent [*] cpu State Thread Command
|
|
|
|
0xc78291d0 1 0 0 0 S 0xc7829404 init
|
|
0xc7954150 942 1 0 0 S 0xc7954384 dropbear
|
|
0xc78789c0 944 1 0 0 S 0xc7878bf4 sh
|
|
(gdb)
|
|
</programlisting></informalexample>
|
|
</para>
|
|
</sect1>
|
|
</chapter>
|
|
<chapter id="KGDBTestSuite">
|
|
<title>kgdb Test Suite</title>
|
|
<para>
|
|
When kgdb is enabled in the kernel config you can also elect to
|
|
enable the config parameter KGDB_TESTS. Turning this on will
|
|
enable a special kgdb I/O module which is designed to test the
|
|
kgdb internal functions.
|
|
</para>
|
|
<para>
|
|
The kgdb tests are mainly intended for developers to test the kgdb
|
|
internals as well as a tool for developing a new kgdb architecture
|
|
specific implementation. These tests are not really for end users
|
|
of the Linux kernel. The primary source of documentation would be
|
|
to look in the drivers/misc/kgdbts.c file.
|
|
</para>
|
|
<para>
|
|
The kgdb test suite can also be configured at compile time to run
|
|
the core set of tests by setting the kernel config parameter
|
|
KGDB_TESTS_ON_BOOT. This particular option is aimed at automated
|
|
regression testing and does not require modifying the kernel boot
|
|
config arguments. If this is turned on, the kgdb test suite can
|
|
be disabled by specifying "kgdbts=" as a kernel boot argument.
|
|
</para>
|
|
</chapter>
|
|
<chapter id="CommonBackEndReq">
|
|
<title>Kernel Debugger Internals</title>
|
|
<sect1 id="kgdbArchitecture">
|
|
<title>Architecture Specifics</title>
|
|
<para>
|
|
The kernel debugger is organized into a number of components:
|
|
<orderedlist>
|
|
<listitem><para>The debug core</para>
|
|
<para>
|
|
The debug core is found in kernel/debugger/debug_core.c. It contains:
|
|
<itemizedlist>
|
|
<listitem><para>A generic OS exception handler which includes
|
|
sync'ing the processors into a stopped state on an multi-CPU
|
|
system.</para></listitem>
|
|
<listitem><para>The API to talk to the kgdb I/O drivers</para></listitem>
|
|
<listitem><para>The API to make calls to the arch-specific kgdb implementation</para></listitem>
|
|
<listitem><para>The logic to perform safe memory reads and writes to memory while using the debugger</para></listitem>
|
|
<listitem><para>A full implementation for software breakpoints unless overridden by the arch</para></listitem>
|
|
<listitem><para>The API to invoke either the kdb or kgdb frontend to the debug core.</para></listitem>
|
|
<listitem><para>The structures and callback API for atomic kernel mode setting.</para>
|
|
<para>NOTE: kgdboc is where the kms callbacks are invoked.</para></listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
</listitem>
|
|
<listitem><para>kgdb arch-specific implementation</para>
|
|
<para>
|
|
This implementation is generally found in arch/*/kernel/kgdb.c.
|
|
As an example, arch/x86/kernel/kgdb.c contains the specifics to
|
|
implement HW breakpoint as well as the initialization to
|
|
dynamically register and unregister for the trap handlers on
|
|
this architecture. The arch-specific portion implements:
|
|
<itemizedlist>
|
|
<listitem><para>contains an arch-specific trap catcher which
|
|
invokes kgdb_handle_exception() to start kgdb about doing its
|
|
work</para></listitem>
|
|
<listitem><para>translation to and from gdb specific packet format to pt_regs</para></listitem>
|
|
<listitem><para>Registration and unregistration of architecture specific trap hooks</para></listitem>
|
|
<listitem><para>Any special exception handling and cleanup</para></listitem>
|
|
<listitem><para>NMI exception handling and cleanup</para></listitem>
|
|
<listitem><para>(optional) HW breakpoints</para></listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
</listitem>
|
|
<listitem><para>gdbstub frontend (aka kgdb)</para>
|
|
<para>The gdbstub is located in kernel/debug/gdbstub.c. It contains:</para>
|
|
<itemizedlist>
|
|
<listitem><para>All the logic to implement the gdb serial protocol</para></listitem>
|
|
</itemizedlist>
|
|
</listitem>
|
|
<listitem><para>kdb frontend</para>
|
|
<para>The kdb debugger shell is broken down into a number of
|
|
components. The kdb core is located in kernel/debug/kdb. There
|
|
are a number of helper functions in some of the other kernel
|
|
components to make it possible for kdb to examine and report
|
|
information about the kernel without taking locks that could
|
|
cause a kernel deadlock. The kdb core contains implements the following functionality.</para>
|
|
<itemizedlist>
|
|
<listitem><para>A simple shell</para></listitem>
|
|
<listitem><para>The kdb core command set</para></listitem>
|
|
<listitem><para>A registration API to register additional kdb shell commands.</para>
|
|
<itemizedlist>
|
|
<listitem><para>A good example of a self-contained kdb module
|
|
is the "ftdump" command for dumping the ftrace buffer. See:
|
|
kernel/trace/trace_kdb.c</para></listitem>
|
|
<listitem><para>For an example of how to dynamically register
|
|
a new kdb command you can build the kdb_hello.ko kernel module
|
|
from samples/kdb/kdb_hello.c. To build this example you can
|
|
set CONFIG_SAMPLES=y and CONFIG_SAMPLE_KDB=m in your kernel
|
|
config. Later run "modprobe kdb_hello" and the next time you
|
|
enter the kdb shell, you can run the "hello"
|
|
command.</para></listitem>
|
|
</itemizedlist></listitem>
|
|
<listitem><para>The implementation for kdb_printf() which
|
|
emits messages directly to I/O drivers, bypassing the kernel
|
|
log.</para></listitem>
|
|
<listitem><para>SW / HW breakpoint management for the kdb shell</para></listitem>
|
|
</itemizedlist>
|
|
</listitem>
|
|
<listitem><para>kgdb I/O driver</para>
|
|
<para>
|
|
Each kgdb I/O driver has to provide an implementation for the following:
|
|
<itemizedlist>
|
|
<listitem><para>configuration via built-in or module</para></listitem>
|
|
<listitem><para>dynamic configuration and kgdb hook registration calls</para></listitem>
|
|
<listitem><para>read and write character interface</para></listitem>
|
|
<listitem><para>A cleanup handler for unconfiguring from the kgdb core</para></listitem>
|
|
<listitem><para>(optional) Early debug methodology</para></listitem>
|
|
</itemizedlist>
|
|
Any given kgdb I/O driver has to operate very closely with the
|
|
hardware and must do it in such a way that does not enable
|
|
interrupts or change other parts of the system context without
|
|
completely restoring them. The kgdb core will repeatedly "poll"
|
|
a kgdb I/O driver for characters when it needs input. The I/O
|
|
driver is expected to return immediately if there is no data
|
|
available. Doing so allows for the future possibility to touch
|
|
watchdog hardware in such a way as to have a target system not
|
|
reset when these are enabled.
|
|
</para>
|
|
</listitem>
|
|
</orderedlist>
|
|
</para>
|
|
<para>
|
|
If you are intent on adding kgdb architecture specific support
|
|
for a new architecture, the architecture should define
|
|
<constant>HAVE_ARCH_KGDB</constant> in the architecture specific
|
|
Kconfig file. This will enable kgdb for the architecture, and
|
|
at that point you must create an architecture specific kgdb
|
|
implementation.
|
|
</para>
|
|
<para>
|
|
There are a few flags which must be set on every architecture in
|
|
their <asm/kgdb.h> file. These are:
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>
|
|
NUMREGBYTES: The size in bytes of all of the registers, so
|
|
that we can ensure they will all fit into a packet.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
BUFMAX: The size in bytes of the buffer GDB will read into.
|
|
This must be larger than NUMREGBYTES.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
CACHE_FLUSH_IS_SAFE: Set to 1 if it is always safe to call
|
|
flush_cache_range or flush_icache_range. On some architectures,
|
|
these functions may not be safe to call on SMP since we keep other
|
|
CPUs in a holding pattern.
|
|
</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
<para>
|
|
There are also the following functions for the common backend,
|
|
found in kernel/kgdb.c, that must be supplied by the
|
|
architecture-specific backend unless marked as (optional), in
|
|
which case a default function maybe used if the architecture
|
|
does not need to provide a specific implementation.
|
|
</para>
|
|
!Iinclude/linux/kgdb.h
|
|
</sect1>
|
|
<sect1 id="kgdbocDesign">
|
|
<title>kgdboc internals</title>
|
|
<sect2>
|
|
<title>kgdboc and uarts</title>
|
|
<para>
|
|
The kgdboc driver is actually a very thin driver that relies on the
|
|
underlying low level to the hardware driver having "polling hooks"
|
|
to which the tty driver is attached. In the initial
|
|
implementation of kgdboc the serial_core was changed to expose a
|
|
low level UART hook for doing polled mode reading and writing of a
|
|
single character while in an atomic context. When kgdb makes an I/O
|
|
request to the debugger, kgdboc invokes a callback in the serial
|
|
core which in turn uses the callback in the UART driver.</para>
|
|
<para>
|
|
When using kgdboc with a UART, the UART driver must implement two callbacks in the <constant>struct uart_ops</constant>. Example from drivers/8250.c:<programlisting>
|
|
#ifdef CONFIG_CONSOLE_POLL
|
|
.poll_get_char = serial8250_get_poll_char,
|
|
.poll_put_char = serial8250_put_poll_char,
|
|
#endif
|
|
</programlisting>
|
|
Any implementation specifics around creating a polling driver use the
|
|
<constant>#ifdef CONFIG_CONSOLE_POLL</constant>, as shown above.
|
|
Keep in mind that polling hooks have to be implemented in such a way
|
|
that they can be called from an atomic context and have to restore
|
|
the state of the UART chip on return such that the system can return
|
|
to normal when the debugger detaches. You need to be very careful
|
|
with any kind of lock you consider, because failing here is most likely
|
|
going to mean pressing the reset button.
|
|
</para>
|
|
</sect2>
|
|
<sect2 id="kgdbocKbd">
|
|
<title>kgdboc and keyboards</title>
|
|
<para>The kgdboc driver contains logic to configure communications
|
|
with an attached keyboard. The keyboard infrastructure is only
|
|
compiled into the kernel when CONFIG_KDB_KEYBOARD=y is set in the
|
|
kernel configuration.</para>
|
|
<para>The core polled keyboard driver driver for PS/2 type keyboards
|
|
is in drivers/char/kdb_keyboard.c. This driver is hooked into the
|
|
debug core when kgdboc populates the callback in the array
|
|
called <constant>kdb_poll_funcs[]</constant>. The
|
|
kdb_get_kbd_char() is the top-level function which polls hardware
|
|
for single character input.
|
|
</para>
|
|
</sect2>
|
|
<sect2 id="kgdbocKms">
|
|
<title>kgdboc and kms</title>
|
|
<para>The kgdboc driver contains logic to request the graphics
|
|
display to switch to a text context when you are using
|
|
"kgdboc=kms,kbd", provided that you have a video driver which has a
|
|
frame buffer console and atomic kernel mode setting support.</para>
|
|
<para>
|
|
Every time the kernel
|
|
debugger is entered it calls kgdboc_pre_exp_handler() which in turn
|
|
calls con_debug_enter() in the virtual console layer. On resuming kernel
|
|
execution, the kernel debugger calls kgdboc_post_exp_handler() which
|
|
in turn calls con_debug_leave().</para>
|
|
<para>Any video driver that wants to be compatible with the kernel
|
|
debugger and the atomic kms callbacks must implement the
|
|
mode_set_base_atomic, fb_debug_enter and fb_debug_leave operations.
|
|
For the fb_debug_enter and fb_debug_leave the option exists to use
|
|
the generic drm fb helper functions or implement something custom for
|
|
the hardware. The following example shows the initialization of the
|
|
.mode_set_base_atomic operation in
|
|
drivers/gpu/drm/i915/intel_display.c:
|
|
<informalexample>
|
|
<programlisting>
|
|
static const struct drm_crtc_helper_funcs intel_helper_funcs = {
|
|
[...]
|
|
.mode_set_base_atomic = intel_pipe_set_base_atomic,
|
|
[...]
|
|
};
|
|
</programlisting>
|
|
</informalexample>
|
|
</para>
|
|
<para>Here is an example of how the i915 driver initializes the fb_debug_enter and fb_debug_leave functions to use the generic drm helpers in
|
|
drivers/gpu/drm/i915/intel_fb.c:
|
|
<informalexample>
|
|
<programlisting>
|
|
static struct fb_ops intelfb_ops = {
|
|
[...]
|
|
.fb_debug_enter = drm_fb_helper_debug_enter,
|
|
.fb_debug_leave = drm_fb_helper_debug_leave,
|
|
[...]
|
|
};
|
|
</programlisting>
|
|
</informalexample>
|
|
</para>
|
|
</sect2>
|
|
</sect1>
|
|
</chapter>
|
|
<chapter id="credits">
|
|
<title>Credits</title>
|
|
<para>
|
|
The following people have contributed to this document:
|
|
<orderedlist>
|
|
<listitem><para>Amit Kale<email>amitkale@linsyssoft.com</email></para></listitem>
|
|
<listitem><para>Tom Rini<email>trini@kernel.crashing.org</email></para></listitem>
|
|
</orderedlist>
|
|
In March 2008 this document was completely rewritten by:
|
|
<itemizedlist>
|
|
<listitem><para>Jason Wessel<email>jason.wessel@windriver.com</email></para></listitem>
|
|
</itemizedlist>
|
|
In Jan 2010 this document was updated to include kdb.
|
|
<itemizedlist>
|
|
<listitem><para>Jason Wessel<email>jason.wessel@windriver.com</email></para></listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
</chapter>
|
|
</book>
|
|
|