linux/scripts
Douglas Anderson 3298b690b2 kbuild: Add a cache for generated variables
While timing a "no-op" build of the kernel (incrementally building the
kernel even though nothing changed) in the Chrome OS build system I
found that it was much slower than I expected.

Digging into things a bit, I found that quite a bit of the time was
spent invoking the C compiler even though we weren't actually building
anything.  Currently in the Chrome OS build system the C compiler is
called through a number of wrappers (one of which is written in
python!) and can take upwards of 100 ms to invoke even if we're not
doing anything difficult, so these invocations of the compiler were
taking a lot of time.  Worse the invocations couldn't seem to take
advantage of the multiple cores on my system.

Certainly it seems like we could make the compiler invocations in the
Chrome OS build system faster, but only to a point.  Inherently
invoking a program as big as a C compiler is a fairly heavy
operation.  Thus even if we can speed the compiler calls it made sense
to track down what was happening.

It turned out that all the compiler invocations were coming from
usages like this in the kernel's Makefile:

KBUILD_CFLAGS += $(call cc-option,-fno-delete-null-pointer-checks,)

Due to the way cc-option and similar statements work the above
contains an implicit call to the C compiler.  ...and due to the fact
that we're storing the result in KBUILD_CFLAGS, a simply expanded
variable, the call will happen every time the Makefile is parsed, even
if there are no users of KBUILD_CFLAGS.

Rather than redoing this computation every time, it makes a lot of
sense to cache the result of all of the Makefile's compiler calls just
like we do when we compile a ".c" file to a ".o" file.  Conceptually
this is quite a simple idea.  ...and since the calls to invoke the
compiler and similar tools are centrally located in the Kbuild.include
file this doesn't even need to be super invasive.

Implementing the cache in a simple-to-use and efficient way is not
quite as simple as it first sounds, though.  To get maximum speed we
really want the cache in a format that make can natively understand
and make doesn't really have an ability to load/parse files. ...but
make _can_ import other Makefiles, so the solution is to store the
cache in Makefile format.  This requires coming up with a valid/unique
Makefile variable name for each value to be cached, but that's
solvable with some cleverness.

After this change, we'll automatically create a ".cache.mk" file that
will contain our cached variables.  We'll load this on each invocation
of make and will avoid recomputing anything that's already in our
cache.  The cache is stored in a format that it shouldn't need any
invalidation since anything that might change should affect the "key"
and any old cached value won't be used.

Signed-off-by: Douglas Anderson <dianders@chromium.org>
Tested-by: Ingo Molnar <mingo@kernel.org>
Tested-by: Guenter Roeck <linux@roeck-us.net>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
2017-11-13 22:54:23 +09:00
..
basic fixdep: trivial: typo fix and correction 2017-08-10 01:01:03 +09:00
coccinelle drm: Introduce drm_property_blob_{get,put}() 2017-02-28 16:16:46 +01:00
dtc scripts/dtc: dtx_diff - 2nd update of include dts paths to match build 2017-09-20 10:13:05 -05:00
gcc-plugins Merge branch 'for-next/gcc-plugin/structleak' into for-next/gcc-plugins 2017-08-07 13:29:04 -07:00
gdb kbuild: use $(abspath ...) instead of $(shell cd ... && /bin/pwd) 2017-09-01 08:50:32 +09:00
genksyms genksyms: fix gperf removal conversion 2017-09-08 14:32:34 -07:00
kconfig Remove gperf usage from toolchain 2017-08-19 11:02:53 -07:00
ksymoops Fix dead URLs to ftp.kernel.org 2017-03-28 16:16:52 +02:00
mod Modules updates for v4.14 2017-09-13 11:28:19 -07:00
package kbuild: rpm-pkg: fix version number handling 2017-09-21 00:06:03 +09:00
selinux selinux: genheaders should fail if too many permissions are defined 2017-07-31 19:03:02 -04:00
tracing ftrace/scripts: Add helper script to bisect function tracing problem functions 2016-09-21 13:56:55 -04:00
.gitignore Docs: clean up some DocBook loose ends 2017-06-23 14:17:38 -06:00
adjust_autoksyms.sh kbuild: fix scripts/adjust_autoksyms.sh* for the no modules case 2016-12-11 22:30:10 +01:00
asn1_compiler.c ASN.1: fix open failure check on headername 2016-04-06 14:06:48 +01:00
bloat-o-meter scripts/bloat-o-meter: compile .NUMBER regex 2016-12-12 18:55:06 -08:00
bootgraph.pl scripts: Switch to more portable Perl shebang 2017-05-14 11:20:44 +09:00
check_00index.sh scripts: add a script to check if Documentation/00-INDEX is sane 2016-12-05 14:21:24 -07:00
check_extable.sh scripts: add check_extable.sh script. 2015-04-13 21:03:02 +09:30
checkincludes.pl scripts: Switch to more portable Perl shebang 2017-05-14 11:20:44 +09:00
checkkconfigsymbols.py checkkconfigsymbols.py: support Kconfig's 'imply' statement 2017-02-03 11:49:06 +01:00
checkpatch.pl checkpatch: add 6 missing types to --list-types 2017-09-08 18:26:50 -07:00
checkstack.pl scripts: Switch to more portable Perl shebang 2017-05-14 11:20:44 +09:00
checksyscalls.sh kbuild: speed up checksyscalls.sh 2017-06-10 01:15:36 +09:00
checkversion.pl scripts: Switch to more portable Perl shebang 2017-05-14 11:20:44 +09:00
cleanfile scripts: Switch to more portable Perl shebang 2017-05-14 11:20:44 +09:00
cleanpatch scripts: Switch to more portable Perl shebang 2017-05-14 11:20:44 +09:00
coccicheck scripts/coccicheck: Update reference for the corresponding documentation 2016-10-11 09:50:43 +02:00
config
conmakehash.c
const_structs.checkpatch const_structs.checkpatch: add frequently used from Julia Lawall's list 2016-10-11 15:06:30 -07:00
decode_stacktrace.sh scripts/decode_stacktrace.sh: Fix address line detection on x86 2016-11-29 14:19:50 +01:00
decodecode
depmod.sh
diffconfig vfs,ext2: remove CONFIG_EXT2_FS_XIP and rename CONFIG_FS_XIP to CONFIG_FS_DAX 2015-02-16 17:56:04 -08:00
export_report.pl scripts: Switch to more portable Perl shebang 2017-05-14 11:20:44 +09:00
extract_xc3028.pl scripts: Switch to more portable Perl shebang 2017-05-14 11:20:44 +09:00
extract-cert.c KEYS: Remove unnecessary header #inclusions from extract-cert.c 2015-09-25 16:31:45 +01:00
extract-ikconfig scripts/extract-ikconfig: Support LZ4-compressed images. 2015-04-15 14:01:12 +02:00
extract-module-sig.pl scripts: Switch to more portable Perl shebang 2017-05-14 11:20:44 +09:00
extract-sys-certs.pl scripts: Switch to more portable Perl shebang 2017-05-14 11:20:44 +09:00
extract-vmlinux
faddr2line scripts/faddr2line: Fix "size mismatch" error 2016-10-25 18:40:37 +02:00
gcc-goto.sh
gcc-ld
gcc-plugin.sh gcc-plugins: abort builds cleanly when not supported 2016-08-08 17:49:05 -07:00
gcc-version.sh
gcc-x86_32-has-stack-protector.sh
gcc-x86_64-has-stack-protector.sh scripts/has-stack-protector: add -fno-PIE 2016-11-09 22:28:05 +01:00
gen_initramfs_list.sh Kbuild misc updates for 4.13 2017-07-07 15:09:09 -07:00
get_dvb_firmware scripts: Switch to more portable Perl shebang 2017-05-14 11:20:44 +09:00
get_maintainer.pl get_maintainer: Prepare for separate MAINTAINERS files 2017-08-08 11:09:31 -07:00
gfp-translate
headerdep.pl scripts: Switch to more portable Perl shebang 2017-05-14 11:20:44 +09:00
headers_check.pl scripts: Switch to more portable Perl shebang 2017-05-14 11:20:44 +09:00
headers_install.sh Merge branch 'kbuild' of git://git.kernel.org/pub/scm/linux/kernel/git/mmarek/kbuild 2014-10-14 09:22:26 +02:00
headers.sh CRIS: Fix headers_install 2014-12-20 00:01:12 +01:00
insert-sys-cert.c KEYS: Reserve an extra certificate symbol for inserting without recompiling 2016-02-26 15:30:20 +00:00
kallsyms.c kbuild: modversions: add infrastructure for emitting relative CRCs 2017-02-03 08:28:25 -08:00
Kbuild.include kbuild: Add a cache for generated variables 2017-11-13 22:54:23 +09:00
kernel-doc kernel-doc parser mishandles declarations split into lines 2017-08-30 16:18:16 -06:00
ld-version.sh ld-version: Drop the 4th and 5th version components 2016-05-13 14:02:05 +02:00
Lindent scripts/Lindent: clean up and optimize 2017-02-22 16:41:26 -08:00
link-vmlinux.sh kbuild: link-vmlinux.sh: simplify .version increment 2017-10-09 23:28:46 +09:00
Makefile Docs: clean up some DocBook loose ends 2017-06-23 14:17:38 -06:00
Makefile.asm-generic kbuild: add forward declaration of default target to Makefile.asm-generic 2017-11-13 22:54:17 +09:00
Makefile.build objtool: Skip unreachable warnings for GCC 4.4 and older 2017-09-28 07:23:02 +02:00
Makefile.clean Shared library support 2016-06-07 22:57:10 +02:00
Makefile.dtbinst kbuild: Do not use hyphen in exported variable name 2017-08-21 09:06:00 +09:00
Makefile.extrawarn Kbuild: enable -Wunused-macros warning for "make W=2" 2017-09-01 08:53:17 +09:00
Makefile.gcc-plugins gcc-plugins: structleak: add option to init all vars used as byref args 2017-08-07 11:20:57 -07:00
Makefile.headersinst kbuild: replace $(hdr-arch) with $(SRCARCH) 2017-10-10 10:00:47 +09:00
Makefile.help
Makefile.host kbuild: Add __cc-option macro 2017-06-25 12:47:55 +09:00
Makefile.kasan kasan: turn on -fsanitize-address-use-after-scope 2016-12-12 18:55:09 -08:00
Makefile.lib kbuild: remove KBUILD_SUBDIR_ASFLAGS and KBUILD_SUBDIR_CCFLAGS 2017-10-31 00:39:03 +09:00
Makefile.modbuiltin
Makefile.modinst modsign: Abort modules_install when signing fails 2015-08-07 16:26:13 +01:00
Makefile.modpost kbuild: add arch specific post-link Makefile 2016-09-09 10:47:00 +02:00
Makefile.modsign
Makefile.ubsan Kbuild: enable -Wmaybe-uninitialized warning for "make W=1" 2016-11-11 08:45:08 -08:00
makelst
markup_oops.pl scripts: Switch to more portable Perl shebang 2017-05-14 11:20:44 +09:00
mkcompile_h kbuild: mkcompile_h: do not create .version 2017-10-09 23:28:46 +09:00
mkmakefile
mksysmap mksysmap: Add h8300 local symbol pattern 2015-06-23 13:35:47 +09:00
mkuboot.sh
module-common.lds module: set .init_array alignment to 8 2017-03-13 09:40:28 -07:00
namespace.pl scripts: Switch to more portable Perl shebang 2017-05-14 11:20:44 +09:00
objdiff scripts: objdiff: Ignore debug info when comparing 2017-03-11 11:13:38 +09:00
parse-maintainers.pl parse-maintainers: Move matching sections from MAINTAINERS 2017-08-08 11:16:14 -07:00
patch-kernel
pnmtologo.c
profile2linkerlist.pl scripts: Switch to more portable Perl shebang 2017-05-14 11:20:44 +09:00
prune-kernel scripts: add "prune-kernel" script to clean up old kernel images 2016-02-09 10:09:52 -08:00
recordmcount.c ftrace: Allow for function tracing to record init functions on boot up 2017-03-24 20:51:49 -04:00
recordmcount.h recordmcount: Fix endianness handling bug for nop_mcount 2015-11-03 10:45:26 -05:00
recordmcount.pl scripts: Switch to more portable Perl shebang 2017-05-14 11:20:44 +09:00
setlocalversion kbuild: setlocalversion: print error to STDERR 2016-06-20 17:37:45 +02:00
show_delta
sign-file.c sign-file: fix build error in sign-file.c with libressl 2017-02-10 12:43:47 +11:00
sortextable.c powerpc: Build-time sort the exception table 2016-11-14 11:11:51 +11:00
sortextable.h scripts/sortextable: suppress warning: `relocs_size' may be used uninitialized 2014-10-14 02:18:23 +02:00
spelling.txt scripts/spelling.txt: add a bunch more spelling mistakes 2017-07-06 16:24:30 -07:00
sphinx-pre-install docs-rst: don't require adjustbox anymore 2017-09-08 10:02:55 -06:00
stackdelta scripts: Switch to more portable Perl shebang 2017-05-14 11:20:44 +09:00
stackusage scripts: add stackusage script 2015-08-28 17:04:40 +02:00
tags.sh tags: honor COMPILED_SOURCE with apart output directory 2017-06-06 08:42:34 +09:00
unifdef.c
ver_linux ver_linux: Use /usr/bin/awk instead of /bin/awk 2017-04-08 18:03:42 +02:00
xen-hypercalls.sh xen: build infrastructure for generating hypercall depending symbols 2015-03-16 14:49:13 +00:00
xz_wrap.sh