I think the help message of the -d option is somewhat misleading.
Path to the kernel source directory to search (defaults to the working directory)
The part "kernel source directory" is the source of the confusion.
Some people misunderstand as if this script did not support separate
output directories.
Actually, this script also works for out-of-tree builds. You can
use the -d option to point to the object output directory, not to
the source directory. It should match to the O= option used in the
previous kernel build, and then appears in the "directory" field of
compile_commands.json.
Reword the help message.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
The tools/ directory uses a different build system, and the format of
.cmd files is different because the tools builds run in a different
work directory.
Supporting two formats compilicates the script.
The only loss by this change is objtool.
Also, rename the confusing variable 'relative_path' because it is
not necessarily a relative path. When the output directory is not
the direct child of the source tree (e.g. O=foo/bar), it is an
absolute path. Rename it to 'file_path'.
os.path.join(root_directory, file_path) works whether the file_path
is relative or not. If file_path is already absolute, it returns it
as-is.
I used os.path.abspath() to normalize file paths. If you run this
script against the kernel built with O=foo option, the file_path
contains '../' patterns. os.path.abspath() fixes up 'foo/bar/../baz'
into 'foo/baz', and produces a cleaner commands_database.json.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
Use 'choices' to check if the given parameter is valid.
I also simplified the help message because, with 'choices', --help
shows the list of valid parameters:
--log_level {DEBUG,INFO,WARNING,ERROR,CRITICAL}
I started the help message with a lower case, "the level of log ..."
in order to be consistent with the -h option:
-h, --help show this help message and exit
The message "show this help ..." comes from the ArgumentParser library
code, and I do not know how to change it. So, I changed our code.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
After the allmodconfig build, this script takes about 5 sec on my
machine. Most of the run-time is consumed for needless regex matching.
We know the format of .*.cmd file; the first line is the build command.
There is no need to parse the rest.
With this optimization, now it runs 4 times faster.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
Tested-by: Nick Desaulniers <ndesaulniers@google.com>
Running gen_compile_commands.py after building the kernel with
allnoconfig gave this:
$ ./scripts/gen_compile_commands.py
WARNING: Found 449 entries. Have you compiled the kernel?
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
The LLVM/Clang project provides many tools for analyzing C source code.
Many of these tools are based on LibTooling
(https://clang.llvm.org/docs/LibTooling.html), which depends on a
database of compiler flags. The standard container for this database is
compile_commands.json, which consists of a list of JSON objects, each
with "directory", "file", and "command" fields.
Some build systems, like cmake or bazel, produce this compilation
information directly. Naturally, Makefiles don't. However, the kernel
makefiles already create .<target>.o.cmd files that contain all the
information needed to build a compile_commands.json file.
So, this commit adds scripts/gen_compile_commands.py, which recursively
searches through a directory for .<target>.o.cmd files and extracts
appropriate compile commands from them. It writes a
compile_commands.json file that LibTooling-based tools can use.
By default, gen_compile_commands.py starts its search in its working
directory and (over)writes compile_commands.json in the working
directory. However, it also supports --output and --directory flags for
out-of-tree use.
Note that while gen_compile_commands.py enables the use of clang-based
tools, it does not require the kernel to be compiled with clang. E.g.,
the following sequence of commands produces a compile_commands.json file
that works correctly with LibTooling.
make defconfig
make
scripts/gen_compile_commands.py
Also note that this script is written to work correctly in both Python 2
and Python 3, so it does not specify the Python version in its first
line.
For an example of the utility of this script: after running
gen_compile_commands.json on the latest kernel version, I was able to
use Vim + the YouCompleteMe pluging + clangd to automatically jump to
definitions and declarations. Obviously, cscope and ctags provide some
of this functionality; the advantage of supporting LibTooling is that it
opens the door to many other clang-based tools that understand the code
directly and do not rely on regular expressions and heuristics.
Tested: Built several recent kernel versions and ran the script against
them, testing tools like clangd (for editor/LSP support) and clang-check
(for static analysis). Also extracted some test .cmd files from a kernel
build and wrote a test script to check that the script behaved correctly
with all permutations of the --output and --directory flags.
Signed-off-by: Tom Roeder <tmroeder@google.com>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
