systemd/coccinelle/run-coccinelle.sh

86 lines
3.2 KiB
Bash
Raw Normal View History

#!/usr/bin/env bash
# SPDX-License-Identifier: LGPL-2.1-or-later
set -e
# Exclude following paths from the Coccinelle transformations
EXCLUDED_PATHS=(
"src/boot/efi/*"
"src/shared/linux/*"
"src/basic/linux/*"
# Symlinked to test-bus-vtable-cc.cc, which causes issues with the IN_SET macro
"src/libsystemd/sd-bus/test-bus-vtable.c"
"src/libsystemd/sd-journal/lookup3.c"
coccinelle: rework how we run the Coccinelle transformations Turns out that the original way we did things was quite broken, as it skipped a _lot_ of code. This was because we just threw everything into one pile and tried to spatch it, but this made Coccinelle sad, like when man page examples redefined some of our macros, causing typedef conflicts. For example, with a minimal reproducer that defines a cleanup macro in two source files, Coccinelle has no issues when spatch-ing each one separately: $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci main.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: main.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments SPECIAL NAMES: adding _cleanup_free_ as a attribute $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci logcontrol-example.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: logcontrol-example.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments But when you try to spatch both of them at once, Coccinelle starts complaining and skipping the "bad" code: $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci main.c logcontrol-example.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: main.c logcontrol-example.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments SPECIAL NAMES: adding _cleanup_free_ as a attribute remapping: _cleanup_ to an ident in macro name ERROR-RECOV: found sync end of #define, line 44 parsing pass2: try again ERROR-RECOV: found sync end of #define, line 44 parse error = File "logcontrol-example.c", line 44, column 21, charpos = 1719 around = '__attribute__', whole content = #define _cleanup_(f) __attribute__((cleanup(f))) badcount: 2 bad: #include <systemd/sd-journal.h> bad: BAD:!!!!! #define _cleanup_(f) __attribute__((cleanup(f))) This was, unfortunately, hidden as it is visible only with --verbose-parsing (or --parse-error-msg). Another issue was how we handled includes. The original way of throwing them into the pile of source files doesn't really work, leading up to similar issues as above. The better way is to let Coccinelle properly resolve all includes by telling it where to find our own include files (basically the same thing we already do during compilation). After fixing all this, Coccinelle now has a chance to process much more of our code (there are still some issues in more complex macros, but that requires further investigation). However, there's a huge downside from all of this - doing a _proper_ code analysis is surprisingly time and resource heavy; meaning that processing just one Coccinelle rule now takes 15 - 30 minutes. To make this slightly less painful, Coccinelle supports caching the generated ASTs, which actually helps a lot - it gets the runtime of one rule from 15 - 30 minutes down to ~1 minute. It, of course, has its own downside - the cache is _really_ big (ATTOW the cache takes ~15 GiB). However, even with the aggressive AST caching you're still looking at ~1 hour for one full Coccinelle run, which is a bit annoying, but I guess that's the price of doing things _properly_ (but I'll definitely look into ways of further optimizing this).
2023-12-25 18:28:26 +08:00
# Ignore man examples, as they redefine some macros we use internally, which makes Coccinelle complain
# and ignore code that tries to use the redefined stuff
"man/*"
)
2021-09-30 01:55:24 +08:00
TOP_DIR="$(git rev-parse --show-toplevel)"
coccinelle: rework how we run the Coccinelle transformations Turns out that the original way we did things was quite broken, as it skipped a _lot_ of code. This was because we just threw everything into one pile and tried to spatch it, but this made Coccinelle sad, like when man page examples redefined some of our macros, causing typedef conflicts. For example, with a minimal reproducer that defines a cleanup macro in two source files, Coccinelle has no issues when spatch-ing each one separately: $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci main.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: main.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments SPECIAL NAMES: adding _cleanup_free_ as a attribute $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci logcontrol-example.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: logcontrol-example.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments But when you try to spatch both of them at once, Coccinelle starts complaining and skipping the "bad" code: $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci main.c logcontrol-example.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: main.c logcontrol-example.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments SPECIAL NAMES: adding _cleanup_free_ as a attribute remapping: _cleanup_ to an ident in macro name ERROR-RECOV: found sync end of #define, line 44 parsing pass2: try again ERROR-RECOV: found sync end of #define, line 44 parse error = File "logcontrol-example.c", line 44, column 21, charpos = 1719 around = '__attribute__', whole content = #define _cleanup_(f) __attribute__((cleanup(f))) badcount: 2 bad: #include <systemd/sd-journal.h> bad: BAD:!!!!! #define _cleanup_(f) __attribute__((cleanup(f))) This was, unfortunately, hidden as it is visible only with --verbose-parsing (or --parse-error-msg). Another issue was how we handled includes. The original way of throwing them into the pile of source files doesn't really work, leading up to similar issues as above. The better way is to let Coccinelle properly resolve all includes by telling it where to find our own include files (basically the same thing we already do during compilation). After fixing all this, Coccinelle now has a chance to process much more of our code (there are still some issues in more complex macros, but that requires further investigation). However, there's a huge downside from all of this - doing a _proper_ code analysis is surprisingly time and resource heavy; meaning that processing just one Coccinelle rule now takes 15 - 30 minutes. To make this slightly less painful, Coccinelle supports caching the generated ASTs, which actually helps a lot - it gets the runtime of one rule from 15 - 30 minutes down to ~1 minute. It, of course, has its own downside - the cache is _really_ big (ATTOW the cache takes ~15 GiB). However, even with the aggressive AST caching you're still looking at ~1 hour for one full Coccinelle run, which is a bit annoying, but I guess that's the price of doing things _properly_ (but I'll definitely look into ways of further optimizing this).
2023-12-25 18:28:26 +08:00
CACHE_DIR="$(dirname "$0")/.coccinelle-cache"
2021-09-30 01:55:24 +08:00
ARGS=()
# Create an array from files tracked by git...
coccinelle: rework how we run the Coccinelle transformations Turns out that the original way we did things was quite broken, as it skipped a _lot_ of code. This was because we just threw everything into one pile and tried to spatch it, but this made Coccinelle sad, like when man page examples redefined some of our macros, causing typedef conflicts. For example, with a minimal reproducer that defines a cleanup macro in two source files, Coccinelle has no issues when spatch-ing each one separately: $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci main.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: main.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments SPECIAL NAMES: adding _cleanup_free_ as a attribute $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci logcontrol-example.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: logcontrol-example.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments But when you try to spatch both of them at once, Coccinelle starts complaining and skipping the "bad" code: $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci main.c logcontrol-example.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: main.c logcontrol-example.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments SPECIAL NAMES: adding _cleanup_free_ as a attribute remapping: _cleanup_ to an ident in macro name ERROR-RECOV: found sync end of #define, line 44 parsing pass2: try again ERROR-RECOV: found sync end of #define, line 44 parse error = File "logcontrol-example.c", line 44, column 21, charpos = 1719 around = '__attribute__', whole content = #define _cleanup_(f) __attribute__((cleanup(f))) badcount: 2 bad: #include <systemd/sd-journal.h> bad: BAD:!!!!! #define _cleanup_(f) __attribute__((cleanup(f))) This was, unfortunately, hidden as it is visible only with --verbose-parsing (or --parse-error-msg). Another issue was how we handled includes. The original way of throwing them into the pile of source files doesn't really work, leading up to similar issues as above. The better way is to let Coccinelle properly resolve all includes by telling it where to find our own include files (basically the same thing we already do during compilation). After fixing all this, Coccinelle now has a chance to process much more of our code (there are still some issues in more complex macros, but that requires further investigation). However, there's a huge downside from all of this - doing a _proper_ code analysis is surprisingly time and resource heavy; meaning that processing just one Coccinelle rule now takes 15 - 30 minutes. To make this slightly less painful, Coccinelle supports caching the generated ASTs, which actually helps a lot - it gets the runtime of one rule from 15 - 30 minutes down to ~1 minute. It, of course, has its own downside - the cache is _really_ big (ATTOW the cache takes ~15 GiB). However, even with the aggressive AST caching you're still looking at ~1 hour for one full Coccinelle run, which is a bit annoying, but I guess that's the price of doing things _properly_ (but I'll definitely look into ways of further optimizing this).
2023-12-25 18:28:26 +08:00
mapfile -t FILES < <(git ls-files ':/*.c')
# ...and filter everything that matches patterns from EXCLUDED_PATHS
for excl in "${EXCLUDED_PATHS[@]}"; do
2021-09-30 01:55:24 +08:00
# shellcheck disable=SC2206
FILES=(${FILES[@]//$excl})
done
case "$1" in
-i)
2021-09-30 01:55:24 +08:00
ARGS+=(--in-place)
shift
;;
esac
if ! parallel -h >/dev/null; then
echo 'Please install GNU parallel (package "parallel")'
exit 1
fi
2021-09-30 01:55:24 +08:00
[[ ${#@} -ne 0 ]] && SCRIPTS=("$@") || SCRIPTS=("$TOP_DIR"/coccinelle/*.cocci)
coccinelle: rework how we run the Coccinelle transformations Turns out that the original way we did things was quite broken, as it skipped a _lot_ of code. This was because we just threw everything into one pile and tried to spatch it, but this made Coccinelle sad, like when man page examples redefined some of our macros, causing typedef conflicts. For example, with a minimal reproducer that defines a cleanup macro in two source files, Coccinelle has no issues when spatch-ing each one separately: $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci main.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: main.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments SPECIAL NAMES: adding _cleanup_free_ as a attribute $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci logcontrol-example.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: logcontrol-example.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments But when you try to spatch both of them at once, Coccinelle starts complaining and skipping the "bad" code: $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci main.c logcontrol-example.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: main.c logcontrol-example.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments SPECIAL NAMES: adding _cleanup_free_ as a attribute remapping: _cleanup_ to an ident in macro name ERROR-RECOV: found sync end of #define, line 44 parsing pass2: try again ERROR-RECOV: found sync end of #define, line 44 parse error = File "logcontrol-example.c", line 44, column 21, charpos = 1719 around = '__attribute__', whole content = #define _cleanup_(f) __attribute__((cleanup(f))) badcount: 2 bad: #include <systemd/sd-journal.h> bad: BAD:!!!!! #define _cleanup_(f) __attribute__((cleanup(f))) This was, unfortunately, hidden as it is visible only with --verbose-parsing (or --parse-error-msg). Another issue was how we handled includes. The original way of throwing them into the pile of source files doesn't really work, leading up to similar issues as above. The better way is to let Coccinelle properly resolve all includes by telling it where to find our own include files (basically the same thing we already do during compilation). After fixing all this, Coccinelle now has a chance to process much more of our code (there are still some issues in more complex macros, but that requires further investigation). However, there's a huge downside from all of this - doing a _proper_ code analysis is surprisingly time and resource heavy; meaning that processing just one Coccinelle rule now takes 15 - 30 minutes. To make this slightly less painful, Coccinelle supports caching the generated ASTs, which actually helps a lot - it gets the runtime of one rule from 15 - 30 minutes down to ~1 minute. It, of course, has its own downside - the cache is _really_ big (ATTOW the cache takes ~15 GiB). However, even with the aggressive AST caching you're still looking at ~1 hour for one full Coccinelle run, which is a bit annoying, but I guess that's the price of doing things _properly_ (but I'll definitely look into ways of further optimizing this).
2023-12-25 18:28:26 +08:00
mkdir -p "$CACHE_DIR"
echo "--x-- Using Coccinelle cache directory: $CACHE_DIR"
echo
2021-09-30 01:55:24 +08:00
for script in "${SCRIPTS[@]}"; do
echo "--x-- Processing $script --x--"
TMPFILE="$(mktemp)"
echo "+ spatch --sp-file $script ${ARGS[*]} ..."
coccinelle: rework how we run the Coccinelle transformations Turns out that the original way we did things was quite broken, as it skipped a _lot_ of code. This was because we just threw everything into one pile and tried to spatch it, but this made Coccinelle sad, like when man page examples redefined some of our macros, causing typedef conflicts. For example, with a minimal reproducer that defines a cleanup macro in two source files, Coccinelle has no issues when spatch-ing each one separately: $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci main.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: main.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments SPECIAL NAMES: adding _cleanup_free_ as a attribute $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci logcontrol-example.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: logcontrol-example.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments But when you try to spatch both of them at once, Coccinelle starts complaining and skipping the "bad" code: $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci main.c logcontrol-example.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: main.c logcontrol-example.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments SPECIAL NAMES: adding _cleanup_free_ as a attribute remapping: _cleanup_ to an ident in macro name ERROR-RECOV: found sync end of #define, line 44 parsing pass2: try again ERROR-RECOV: found sync end of #define, line 44 parse error = File "logcontrol-example.c", line 44, column 21, charpos = 1719 around = '__attribute__', whole content = #define _cleanup_(f) __attribute__((cleanup(f))) badcount: 2 bad: #include <systemd/sd-journal.h> bad: BAD:!!!!! #define _cleanup_(f) __attribute__((cleanup(f))) This was, unfortunately, hidden as it is visible only with --verbose-parsing (or --parse-error-msg). Another issue was how we handled includes. The original way of throwing them into the pile of source files doesn't really work, leading up to similar issues as above. The better way is to let Coccinelle properly resolve all includes by telling it where to find our own include files (basically the same thing we already do during compilation). After fixing all this, Coccinelle now has a chance to process much more of our code (there are still some issues in more complex macros, but that requires further investigation). However, there's a huge downside from all of this - doing a _proper_ code analysis is surprisingly time and resource heavy; meaning that processing just one Coccinelle rule now takes 15 - 30 minutes. To make this slightly less painful, Coccinelle supports caching the generated ASTs, which actually helps a lot - it gets the runtime of one rule from 15 - 30 minutes down to ~1 minute. It, of course, has its own downside - the cache is _really_ big (ATTOW the cache takes ~15 GiB). However, even with the aggressive AST caching you're still looking at ~1 hour for one full Coccinelle run, which is a bit annoying, but I guess that's the price of doing things _properly_ (but I'll definitely look into ways of further optimizing this).
2023-12-25 18:28:26 +08:00
# A couple of notes:
#
# 1) Limit this to 10 files at once, as processing the ASTs is _very_ memory hungry - e.g. with 20 files
2023-12-29 03:14:55 +08:00
# at once one spatch process can take around 2.5 GiB of RAM, which can easily eat up all available RAM
# when paired together with parallel
coccinelle: rework how we run the Coccinelle transformations Turns out that the original way we did things was quite broken, as it skipped a _lot_ of code. This was because we just threw everything into one pile and tried to spatch it, but this made Coccinelle sad, like when man page examples redefined some of our macros, causing typedef conflicts. For example, with a minimal reproducer that defines a cleanup macro in two source files, Coccinelle has no issues when spatch-ing each one separately: $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci main.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: main.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments SPECIAL NAMES: adding _cleanup_free_ as a attribute $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci logcontrol-example.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: logcontrol-example.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments But when you try to spatch both of them at once, Coccinelle starts complaining and skipping the "bad" code: $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci main.c logcontrol-example.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: main.c logcontrol-example.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments SPECIAL NAMES: adding _cleanup_free_ as a attribute remapping: _cleanup_ to an ident in macro name ERROR-RECOV: found sync end of #define, line 44 parsing pass2: try again ERROR-RECOV: found sync end of #define, line 44 parse error = File "logcontrol-example.c", line 44, column 21, charpos = 1719 around = '__attribute__', whole content = #define _cleanup_(f) __attribute__((cleanup(f))) badcount: 2 bad: #include <systemd/sd-journal.h> bad: BAD:!!!!! #define _cleanup_(f) __attribute__((cleanup(f))) This was, unfortunately, hidden as it is visible only with --verbose-parsing (or --parse-error-msg). Another issue was how we handled includes. The original way of throwing them into the pile of source files doesn't really work, leading up to similar issues as above. The better way is to let Coccinelle properly resolve all includes by telling it where to find our own include files (basically the same thing we already do during compilation). After fixing all this, Coccinelle now has a chance to process much more of our code (there are still some issues in more complex macros, but that requires further investigation). However, there's a huge downside from all of this - doing a _proper_ code analysis is surprisingly time and resource heavy; meaning that processing just one Coccinelle rule now takes 15 - 30 minutes. To make this slightly less painful, Coccinelle supports caching the generated ASTs, which actually helps a lot - it gets the runtime of one rule from 15 - 30 minutes down to ~1 minute. It, of course, has its own downside - the cache is _really_ big (ATTOW the cache takes ~15 GiB). However, even with the aggressive AST caching you're still looking at ~1 hour for one full Coccinelle run, which is a bit annoying, but I guess that's the price of doing things _properly_ (but I'll definitely look into ways of further optimizing this).
2023-12-25 18:28:26 +08:00
#
# 2) Make sure spatch can find our includes via -I <dir>, similarly as we do when compiling stuff.
# Also, include the system include path as well, since we're not kernel and we make use of the stdlib
# (and other libraries).
coccinelle: rework how we run the Coccinelle transformations Turns out that the original way we did things was quite broken, as it skipped a _lot_ of code. This was because we just threw everything into one pile and tried to spatch it, but this made Coccinelle sad, like when man page examples redefined some of our macros, causing typedef conflicts. For example, with a minimal reproducer that defines a cleanup macro in two source files, Coccinelle has no issues when spatch-ing each one separately: $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci main.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: main.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments SPECIAL NAMES: adding _cleanup_free_ as a attribute $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci logcontrol-example.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: logcontrol-example.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments But when you try to spatch both of them at once, Coccinelle starts complaining and skipping the "bad" code: $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci main.c logcontrol-example.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: main.c logcontrol-example.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments SPECIAL NAMES: adding _cleanup_free_ as a attribute remapping: _cleanup_ to an ident in macro name ERROR-RECOV: found sync end of #define, line 44 parsing pass2: try again ERROR-RECOV: found sync end of #define, line 44 parse error = File "logcontrol-example.c", line 44, column 21, charpos = 1719 around = '__attribute__', whole content = #define _cleanup_(f) __attribute__((cleanup(f))) badcount: 2 bad: #include <systemd/sd-journal.h> bad: BAD:!!!!! #define _cleanup_(f) __attribute__((cleanup(f))) This was, unfortunately, hidden as it is visible only with --verbose-parsing (or --parse-error-msg). Another issue was how we handled includes. The original way of throwing them into the pile of source files doesn't really work, leading up to similar issues as above. The better way is to let Coccinelle properly resolve all includes by telling it where to find our own include files (basically the same thing we already do during compilation). After fixing all this, Coccinelle now has a chance to process much more of our code (there are still some issues in more complex macros, but that requires further investigation). However, there's a huge downside from all of this - doing a _proper_ code analysis is surprisingly time and resource heavy; meaning that processing just one Coccinelle rule now takes 15 - 30 minutes. To make this slightly less painful, Coccinelle supports caching the generated ASTs, which actually helps a lot - it gets the runtime of one rule from 15 - 30 minutes down to ~1 minute. It, of course, has its own downside - the cache is _really_ big (ATTOW the cache takes ~15 GiB). However, even with the aggressive AST caching you're still looking at ~1 hour for one full Coccinelle run, which is a bit annoying, but I guess that's the price of doing things _properly_ (but I'll definitely look into ways of further optimizing this).
2023-12-25 18:28:26 +08:00
#
# 3) Make sure to include includes from includes (--recursive-includes), but use them only to get type
2023-12-29 03:14:55 +08:00
# definitions (--include-headers-for-types) - otherwise we'd start formatting them as well, which might
# be unwanted, especially for includes we fetch verbatim from third-parties
coccinelle: rework how we run the Coccinelle transformations Turns out that the original way we did things was quite broken, as it skipped a _lot_ of code. This was because we just threw everything into one pile and tried to spatch it, but this made Coccinelle sad, like when man page examples redefined some of our macros, causing typedef conflicts. For example, with a minimal reproducer that defines a cleanup macro in two source files, Coccinelle has no issues when spatch-ing each one separately: $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci main.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: main.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments SPECIAL NAMES: adding _cleanup_free_ as a attribute $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci logcontrol-example.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: logcontrol-example.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments But when you try to spatch both of them at once, Coccinelle starts complaining and skipping the "bad" code: $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci main.c logcontrol-example.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: main.c logcontrol-example.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments SPECIAL NAMES: adding _cleanup_free_ as a attribute remapping: _cleanup_ to an ident in macro name ERROR-RECOV: found sync end of #define, line 44 parsing pass2: try again ERROR-RECOV: found sync end of #define, line 44 parse error = File "logcontrol-example.c", line 44, column 21, charpos = 1719 around = '__attribute__', whole content = #define _cleanup_(f) __attribute__((cleanup(f))) badcount: 2 bad: #include <systemd/sd-journal.h> bad: BAD:!!!!! #define _cleanup_(f) __attribute__((cleanup(f))) This was, unfortunately, hidden as it is visible only with --verbose-parsing (or --parse-error-msg). Another issue was how we handled includes. The original way of throwing them into the pile of source files doesn't really work, leading up to similar issues as above. The better way is to let Coccinelle properly resolve all includes by telling it where to find our own include files (basically the same thing we already do during compilation). After fixing all this, Coccinelle now has a chance to process much more of our code (there are still some issues in more complex macros, but that requires further investigation). However, there's a huge downside from all of this - doing a _proper_ code analysis is surprisingly time and resource heavy; meaning that processing just one Coccinelle rule now takes 15 - 30 minutes. To make this slightly less painful, Coccinelle supports caching the generated ASTs, which actually helps a lot - it gets the runtime of one rule from 15 - 30 minutes down to ~1 minute. It, of course, has its own downside - the cache is _really_ big (ATTOW the cache takes ~15 GiB). However, even with the aggressive AST caching you're still looking at ~1 hour for one full Coccinelle run, which is a bit annoying, but I guess that's the price of doing things _properly_ (but I'll definitely look into ways of further optimizing this).
2023-12-25 18:28:26 +08:00
#
# 4) Explicitly undefine the SD_BOOT symbol, so Coccinelle ignores includes guarded by #if SD_BOOT
#
# 5) Use cache, since generating the full AST is expensive. With cache we can do that only once and then
# reuse the cached ASTs for other rules. This cuts down the time needed to run each rule by ~60%.
coccinelle: rework how we run the Coccinelle transformations Turns out that the original way we did things was quite broken, as it skipped a _lot_ of code. This was because we just threw everything into one pile and tried to spatch it, but this made Coccinelle sad, like when man page examples redefined some of our macros, causing typedef conflicts. For example, with a minimal reproducer that defines a cleanup macro in two source files, Coccinelle has no issues when spatch-ing each one separately: $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci main.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: main.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments SPECIAL NAMES: adding _cleanup_free_ as a attribute $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci logcontrol-example.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: logcontrol-example.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments But when you try to spatch both of them at once, Coccinelle starts complaining and skipping the "bad" code: $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci main.c logcontrol-example.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: main.c logcontrol-example.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments SPECIAL NAMES: adding _cleanup_free_ as a attribute remapping: _cleanup_ to an ident in macro name ERROR-RECOV: found sync end of #define, line 44 parsing pass2: try again ERROR-RECOV: found sync end of #define, line 44 parse error = File "logcontrol-example.c", line 44, column 21, charpos = 1719 around = '__attribute__', whole content = #define _cleanup_(f) __attribute__((cleanup(f))) badcount: 2 bad: #include <systemd/sd-journal.h> bad: BAD:!!!!! #define _cleanup_(f) __attribute__((cleanup(f))) This was, unfortunately, hidden as it is visible only with --verbose-parsing (or --parse-error-msg). Another issue was how we handled includes. The original way of throwing them into the pile of source files doesn't really work, leading up to similar issues as above. The better way is to let Coccinelle properly resolve all includes by telling it where to find our own include files (basically the same thing we already do during compilation). After fixing all this, Coccinelle now has a chance to process much more of our code (there are still some issues in more complex macros, but that requires further investigation). However, there's a huge downside from all of this - doing a _proper_ code analysis is surprisingly time and resource heavy; meaning that processing just one Coccinelle rule now takes 15 - 30 minutes. To make this slightly less painful, Coccinelle supports caching the generated ASTs, which actually helps a lot - it gets the runtime of one rule from 15 - 30 minutes down to ~1 minute. It, of course, has its own downside - the cache is _really_ big (ATTOW the cache takes ~15 GiB). However, even with the aggressive AST caching you're still looking at ~1 hour for one full Coccinelle run, which is a bit annoying, but I guess that's the price of doing things _properly_ (but I'll definitely look into ways of further optimizing this).
2023-12-25 18:28:26 +08:00
parallel --halt now,fail=1 --keep-order --noswap --max-args=10 \
spatch --cache-prefix "$CACHE_DIR" \
-I src \
-I /usr/include \
coccinelle: rework how we run the Coccinelle transformations Turns out that the original way we did things was quite broken, as it skipped a _lot_ of code. This was because we just threw everything into one pile and tried to spatch it, but this made Coccinelle sad, like when man page examples redefined some of our macros, causing typedef conflicts. For example, with a minimal reproducer that defines a cleanup macro in two source files, Coccinelle has no issues when spatch-ing each one separately: $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci main.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: main.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments SPECIAL NAMES: adding _cleanup_free_ as a attribute $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci logcontrol-example.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: logcontrol-example.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments But when you try to spatch both of them at once, Coccinelle starts complaining and skipping the "bad" code: $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci main.c logcontrol-example.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: main.c logcontrol-example.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments SPECIAL NAMES: adding _cleanup_free_ as a attribute remapping: _cleanup_ to an ident in macro name ERROR-RECOV: found sync end of #define, line 44 parsing pass2: try again ERROR-RECOV: found sync end of #define, line 44 parse error = File "logcontrol-example.c", line 44, column 21, charpos = 1719 around = '__attribute__', whole content = #define _cleanup_(f) __attribute__((cleanup(f))) badcount: 2 bad: #include <systemd/sd-journal.h> bad: BAD:!!!!! #define _cleanup_(f) __attribute__((cleanup(f))) This was, unfortunately, hidden as it is visible only with --verbose-parsing (or --parse-error-msg). Another issue was how we handled includes. The original way of throwing them into the pile of source files doesn't really work, leading up to similar issues as above. The better way is to let Coccinelle properly resolve all includes by telling it where to find our own include files (basically the same thing we already do during compilation). After fixing all this, Coccinelle now has a chance to process much more of our code (there are still some issues in more complex macros, but that requires further investigation). However, there's a huge downside from all of this - doing a _proper_ code analysis is surprisingly time and resource heavy; meaning that processing just one Coccinelle rule now takes 15 - 30 minutes. To make this slightly less painful, Coccinelle supports caching the generated ASTs, which actually helps a lot - it gets the runtime of one rule from 15 - 30 minutes down to ~1 minute. It, of course, has its own downside - the cache is _really_ big (ATTOW the cache takes ~15 GiB). However, even with the aggressive AST caching you're still looking at ~1 hour for one full Coccinelle run, which is a bit annoying, but I guess that's the price of doing things _properly_ (but I'll definitely look into ways of further optimizing this).
2023-12-25 18:28:26 +08:00
--recursive-includes \
--include-headers-for-types \
--undefined SD_BOOT \
--undefined ENABLE_DEBUG_HASHMAP \
--macro-file-builtins "coccinelle/parsing_hacks.h" \
coccinelle: rework how we run the Coccinelle transformations Turns out that the original way we did things was quite broken, as it skipped a _lot_ of code. This was because we just threw everything into one pile and tried to spatch it, but this made Coccinelle sad, like when man page examples redefined some of our macros, causing typedef conflicts. For example, with a minimal reproducer that defines a cleanup macro in two source files, Coccinelle has no issues when spatch-ing each one separately: $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci main.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: main.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments SPECIAL NAMES: adding _cleanup_free_ as a attribute $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci logcontrol-example.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: logcontrol-example.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments But when you try to spatch both of them at once, Coccinelle starts complaining and skipping the "bad" code: $ spatch --verbose-parsing --sp-file zz-drop-braces.cocci main.c logcontrol-example.c init_defs_builtins: /usr/lib64/coccinelle/standard.h HANDLING: main.c logcontrol-example.c SPECIAL NAMES: adding _cleanup_ as a attribute with arguments SPECIAL NAMES: adding _cleanup_free_ as a attribute remapping: _cleanup_ to an ident in macro name ERROR-RECOV: found sync end of #define, line 44 parsing pass2: try again ERROR-RECOV: found sync end of #define, line 44 parse error = File "logcontrol-example.c", line 44, column 21, charpos = 1719 around = '__attribute__', whole content = #define _cleanup_(f) __attribute__((cleanup(f))) badcount: 2 bad: #include <systemd/sd-journal.h> bad: BAD:!!!!! #define _cleanup_(f) __attribute__((cleanup(f))) This was, unfortunately, hidden as it is visible only with --verbose-parsing (or --parse-error-msg). Another issue was how we handled includes. The original way of throwing them into the pile of source files doesn't really work, leading up to similar issues as above. The better way is to let Coccinelle properly resolve all includes by telling it where to find our own include files (basically the same thing we already do during compilation). After fixing all this, Coccinelle now has a chance to process much more of our code (there are still some issues in more complex macros, but that requires further investigation). However, there's a huge downside from all of this - doing a _proper_ code analysis is surprisingly time and resource heavy; meaning that processing just one Coccinelle rule now takes 15 - 30 minutes. To make this slightly less painful, Coccinelle supports caching the generated ASTs, which actually helps a lot - it gets the runtime of one rule from 15 - 30 minutes down to ~1 minute. It, of course, has its own downside - the cache is _really_ big (ATTOW the cache takes ~15 GiB). However, even with the aggressive AST caching you're still looking at ~1 hour for one full Coccinelle run, which is a bit annoying, but I guess that's the price of doing things _properly_ (but I'll definitely look into ways of further optimizing this).
2023-12-25 18:28:26 +08:00
--smpl-spacing \
--sp-file "$script" \
"${ARGS[@]}" ::: "${FILES[@]}" \
2>"$TMPFILE" || cat "$TMPFILE"
rm -f "$TMPFILE"
2021-09-30 01:55:24 +08:00
echo -e "--x-- Processed $script --x--\n"
done