2013-06-06 07:53:30 +08:00
|
|
|
################################################################################
|
2012-04-17 22:45:20 +08:00
|
|
|
#
|
|
|
|
|
# This file contains various utility functions used by the package
|
|
|
|
|
# infrastructure, or by the packages themselves.
|
|
|
|
|
#
|
2013-06-06 07:53:30 +08:00
|
|
|
################################################################################
|
2012-04-17 22:45:20 +08:00
|
|
|
|
|
|
|
|
#
|
|
|
|
|
# Manipulation of .config files based on the Kconfig
|
2014-05-31 15:55:35 +08:00
|
|
|
# infrastructure. Used by the BusyBox package, the Linux kernel
|
2012-04-17 22:45:20 +08:00
|
|
|
# package, and more.
|
|
|
|
|
#
|
|
|
|
|
|
package/pkg-utils.mk: kconfig mangling defaults to current package's .config
The kconfig mangling macros currently operate on the caller-supplied
.config file, on the assumption that the caller will always know what
file to mangle.
This was correct so far, as packages would indeed only mangle their
own .config files.
However, the Linux kernel does its mangling based on whether some
other packages are enabled or not. That list of conditional mangling
is getting bigger and bigger with each new package that needs such
mangling, culminating with the pending firewalld one [0]. Furthermore,
this mangling is not accessible to packages in br2-external trees. So
we'll want to have packages provide the mangling commands.
So we'll want the mangling to be done on the Linux' .config file in
the expanding package context, not in the package calling the macros.
But packages do not, and should not have knowledge about where the
.config file is, nor how it is named.
So we make the parameter to specify the .config file to mangle
optional. If it is set, this is what the macros will mangle; if it is
not set, the expanding package's .config file will be used.
This has the added benefit that we do not have to repeat in the
expanding package context the knowledge of how the .config file is
named:
FOO_KCONFIG_DOTCONFIG = .config
define FOO_KCONFIG_FIXUPS_CMDS
$(call KCONFIG_ENABLE_OPT,BLA)
endef
[0] http://lists.busybox.net/pipermail/buildroot/2020-March/278683.html
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-04-04 20:10:14 +08:00
|
|
|
# KCONFIG_DOT_CONFIG ([file])
|
|
|
|
|
# Returns the path to the .config file that should be used, which will
|
|
|
|
|
# be $(1) if provided, or the current package .config file otherwise.
|
|
|
|
|
KCONFIG_DOT_CONFIG = $(strip \
|
|
|
|
|
$(if $(strip $(1)), $(1), \
|
|
|
|
|
$($(PKG)_BUILDDIR)/$($(PKG)_KCONFIG_DOTCONFIG) \
|
|
|
|
|
) \
|
|
|
|
|
)
|
|
|
|
|
|
|
|
|
|
# KCONFIG_MUNGE_DOT_CONFIG (option, newline [, file])
|
2020-04-04 20:10:13 +08:00
|
|
|
define KCONFIG_MUNGE_DOT_CONFIG
|
2022-07-27 22:08:22 +08:00
|
|
|
$(SED) '/^\(# \)\?$(strip $(1))\>/d' $(call KCONFIG_DOT_CONFIG,$(3)) && \
|
package/pkg-utils.mk: kconfig mangling defaults to current package's .config
The kconfig mangling macros currently operate on the caller-supplied
.config file, on the assumption that the caller will always know what
file to mangle.
This was correct so far, as packages would indeed only mangle their
own .config files.
However, the Linux kernel does its mangling based on whether some
other packages are enabled or not. That list of conditional mangling
is getting bigger and bigger with each new package that needs such
mangling, culminating with the pending firewalld one [0]. Furthermore,
this mangling is not accessible to packages in br2-external trees. So
we'll want to have packages provide the mangling commands.
So we'll want the mangling to be done on the Linux' .config file in
the expanding package context, not in the package calling the macros.
But packages do not, and should not have knowledge about where the
.config file is, nor how it is named.
So we make the parameter to specify the .config file to mangle
optional. If it is set, this is what the macros will mangle; if it is
not set, the expanding package's .config file will be used.
This has the added benefit that we do not have to repeat in the
expanding package context the knowledge of how the .config file is
named:
FOO_KCONFIG_DOTCONFIG = .config
define FOO_KCONFIG_FIXUPS_CMDS
$(call KCONFIG_ENABLE_OPT,BLA)
endef
[0] http://lists.busybox.net/pipermail/buildroot/2020-March/278683.html
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-04-04 20:10:14 +08:00
|
|
|
echo '$(strip $(2))' >> $(call KCONFIG_DOT_CONFIG,$(3))
|
2012-04-17 22:45:19 +08:00
|
|
|
endef
|
|
|
|
|
|
package/pkg-utils.mk: kconfig mangling defaults to current package's .config
The kconfig mangling macros currently operate on the caller-supplied
.config file, on the assumption that the caller will always know what
file to mangle.
This was correct so far, as packages would indeed only mangle their
own .config files.
However, the Linux kernel does its mangling based on whether some
other packages are enabled or not. That list of conditional mangling
is getting bigger and bigger with each new package that needs such
mangling, culminating with the pending firewalld one [0]. Furthermore,
this mangling is not accessible to packages in br2-external trees. So
we'll want to have packages provide the mangling commands.
So we'll want the mangling to be done on the Linux' .config file in
the expanding package context, not in the package calling the macros.
But packages do not, and should not have knowledge about where the
.config file is, nor how it is named.
So we make the parameter to specify the .config file to mangle
optional. If it is set, this is what the macros will mangle; if it is
not set, the expanding package's .config file will be used.
This has the added benefit that we do not have to repeat in the
expanding package context the knowledge of how the .config file is
named:
FOO_KCONFIG_DOTCONFIG = .config
define FOO_KCONFIG_FIXUPS_CMDS
$(call KCONFIG_ENABLE_OPT,BLA)
endef
[0] http://lists.busybox.net/pipermail/buildroot/2020-March/278683.html
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-04-04 20:10:14 +08:00
|
|
|
# KCONFIG_ENABLE_OPT (option [, file])
|
2022-07-25 20:09:27 +08:00
|
|
|
# If the option is already set to =m or =y, ignore.
|
|
|
|
|
define KCONFIG_ENABLE_OPT
|
|
|
|
|
$(Q)if ! grep -q '^$(strip $(1))=[my]' $(call KCONFIG_DOT_CONFIG,$(2)); then \
|
|
|
|
|
$(call KCONFIG_MUNGE_DOT_CONFIG, $(1), $(1)=y, $(2)); \
|
|
|
|
|
fi
|
|
|
|
|
endef
|
package/pkg-utils.mk: kconfig mangling defaults to current package's .config
The kconfig mangling macros currently operate on the caller-supplied
.config file, on the assumption that the caller will always know what
file to mangle.
This was correct so far, as packages would indeed only mangle their
own .config files.
However, the Linux kernel does its mangling based on whether some
other packages are enabled or not. That list of conditional mangling
is getting bigger and bigger with each new package that needs such
mangling, culminating with the pending firewalld one [0]. Furthermore,
this mangling is not accessible to packages in br2-external trees. So
we'll want to have packages provide the mangling commands.
So we'll want the mangling to be done on the Linux' .config file in
the expanding package context, not in the package calling the macros.
But packages do not, and should not have knowledge about where the
.config file is, nor how it is named.
So we make the parameter to specify the .config file to mangle
optional. If it is set, this is what the macros will mangle; if it is
not set, the expanding package's .config file will be used.
This has the added benefit that we do not have to repeat in the
expanding package context the knowledge of how the .config file is
named:
FOO_KCONFIG_DOTCONFIG = .config
define FOO_KCONFIG_FIXUPS_CMDS
$(call KCONFIG_ENABLE_OPT,BLA)
endef
[0] http://lists.busybox.net/pipermail/buildroot/2020-March/278683.html
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-04-04 20:10:14 +08:00
|
|
|
# KCONFIG_SET_OPT (option, value [, file])
|
2023-05-05 11:43:17 +08:00
|
|
|
KCONFIG_SET_OPT = $(Q)$(call KCONFIG_MUNGE_DOT_CONFIG, $(1), $(1)=$(2), $(3))
|
package/pkg-utils.mk: kconfig mangling defaults to current package's .config
The kconfig mangling macros currently operate on the caller-supplied
.config file, on the assumption that the caller will always know what
file to mangle.
This was correct so far, as packages would indeed only mangle their
own .config files.
However, the Linux kernel does its mangling based on whether some
other packages are enabled or not. That list of conditional mangling
is getting bigger and bigger with each new package that needs such
mangling, culminating with the pending firewalld one [0]. Furthermore,
this mangling is not accessible to packages in br2-external trees. So
we'll want to have packages provide the mangling commands.
So we'll want the mangling to be done on the Linux' .config file in
the expanding package context, not in the package calling the macros.
But packages do not, and should not have knowledge about where the
.config file is, nor how it is named.
So we make the parameter to specify the .config file to mangle
optional. If it is set, this is what the macros will mangle; if it is
not set, the expanding package's .config file will be used.
This has the added benefit that we do not have to repeat in the
expanding package context the knowledge of how the .config file is
named:
FOO_KCONFIG_DOTCONFIG = .config
define FOO_KCONFIG_FIXUPS_CMDS
$(call KCONFIG_ENABLE_OPT,BLA)
endef
[0] http://lists.busybox.net/pipermail/buildroot/2020-March/278683.html
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-04-04 20:10:14 +08:00
|
|
|
# KCONFIG_DISABLE_OPT (option [, file])
|
2023-05-05 11:43:17 +08:00
|
|
|
KCONFIG_DISABLE_OPT = $(Q)$(call KCONFIG_MUNGE_DOT_CONFIG, $(1), $(SHARP_SIGN) $(1) is not set, $(2))
|
2012-04-17 22:45:19 +08:00
|
|
|
|
|
|
|
|
# Helper functions to determine the name of a package and its
|
|
|
|
|
# directory from its makefile directory, using the $(MAKEFILE_LIST)
|
2015-04-13 17:53:21 +08:00
|
|
|
# variable provided by make. This is used by the *-package macros to
|
2014-02-05 17:44:04 +08:00
|
|
|
# automagically find where the package is located.
|
.mk files: bulk aligment and whitespace cleanup of assignments
The Buildroot coding style defines one space around make assignments and
does not align the assignment symbols.
This patch does a bulk fix of offending packages. The package
infrastructures (or more in general assignments to calculated variable
names, like $(2)_FOO) are not touched.
Alignment of line continuation characters (\) is kept as-is.
The sed command used to do this replacement is:
find * -name "*.mk" | xargs sed -i \
-e 's#^\([A-Z0-9a-z_]\+\)\s*\([?:+]\?=\)\s*$#\1 \2#'
-e 's#^\([A-Z0-9a-z_]\+\)\s*\([?:+]\?=\)\s*\([^\\]\+\)$#\1 \2 \3#'
-e 's#^\([A-Z0-9a-z_]\+\)\s*\([?:+]\?=\)\s*\([^\\ \t]\+\s*\\\)\s*$#\1 \2 \3#'
-e 's#^\([A-Z0-9a-z_]\+\)\s*\([?:+]\?=\)\(\s*\\\)#\1 \2\3#'
Brief explanation of this command:
^\([A-Z0-9a-z_]\+\) a regular variable at the beginning of the line
\([?:+]\?=\) any assignment character =, :=, ?=, +=
\([^\\]\+\) any string not containing a line continuation
\([^\\ \t]\+\s*\\\) string, optional whitespace, followed by a
line continuation character
\(\s*\\\) optional whitespace, followed by a line
continuation character
Hence, the first subexpression handles empty assignments, the second
handles regular assignments, the third handles regular assignments with
line continuation, and the fourth empty assignments with line
continuation.
This expression was tested on following test text: (initial tab not
included)
FOO = spaces before
FOO = spaces before and after
FOO = tab before
FOO = tab and spaces before
FOO = tab after
FOO = tab and spaces after
FOO = spaces and tab after
FOO = \
FOO = bar \
FOO = bar space \
FOO = \
GENIMAGE_DEPENDENCIES = host-pkgconf libconfuse
FOO += spaces before
FOO ?= spaces before and after
FOO :=
FOO =
FOO =
FOO =
FOO =
$(MAKE1) CROSS_COMPILE=$(TARGET_CROSS) -C
AT91BOOTSTRAP3_DEFCONFIG = \
AXEL_DISABLE_I18N=--i18n=0
After this bulk change, following manual fixups were done:
- fix line continuation alignment in cegui06 and spice (the sed
expression leaves the number of whitespace between the value and line
continuation character intact, but the whitespace before that could have
changed, causing misalignment.
- qt5base was reverted, as this package uses extensive alignment which
actually makes the code more readable.
Finally, the end result was manually reviewed.
Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
Cc: Yann E. Morin <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2014-10-07 15:06:03 +08:00
|
|
|
pkgdir = $(dir $(lastword $(MAKEFILE_LIST)))
|
|
|
|
|
pkgname = $(lastword $(subst /, ,$(pkgdir)))
|
2012-04-17 22:45:19 +08:00
|
|
|
|
2019-03-26 03:59:20 +08:00
|
|
|
# Helper to build the extension for a package archive, based on various
|
|
|
|
|
# conditions.
|
|
|
|
|
# $(1): upper-case package name
|
2020-12-13 21:59:28 +08:00
|
|
|
pkg_source_ext = $(BR_FMT_VERSION_$($(1)_SITE_METHOD)).tar.gz
|
2019-03-26 03:59:20 +08:00
|
|
|
|
2012-04-17 22:45:19 +08:00
|
|
|
# Define extractors for different archive suffixes
|
|
|
|
|
INFLATE.bz2 = $(BZCAT)
|
|
|
|
|
INFLATE.gz = $(ZCAT)
|
2017-02-13 04:15:39 +08:00
|
|
|
INFLATE.lz = $(LZCAT)
|
2015-06-16 06:57:21 +08:00
|
|
|
INFLATE.lzma = $(XZCAT)
|
2012-04-17 22:45:19 +08:00
|
|
|
INFLATE.tbz = $(BZCAT)
|
|
|
|
|
INFLATE.tbz2 = $(BZCAT)
|
|
|
|
|
INFLATE.tgz = $(ZCAT)
|
|
|
|
|
INFLATE.xz = $(XZCAT)
|
|
|
|
|
INFLATE.tar = cat
|
2013-08-02 17:05:14 +08:00
|
|
|
# suitable-extractor(filename): returns extractor based on suffix
|
|
|
|
|
suitable-extractor = $(INFLATE$(suffix $(1)))
|
2012-04-17 22:45:19 +08:00
|
|
|
|
2019-12-16 23:31:14 +08:00
|
|
|
EXTRACTOR_PKG_DEPENDENCY.lzma = $(BR2_XZCAT_HOST_DEPENDENCY)
|
|
|
|
|
EXTRACTOR_PKG_DEPENDENCY.xz = $(BR2_XZCAT_HOST_DEPENDENCY)
|
|
|
|
|
EXTRACTOR_PKG_DEPENDENCY.lz = $(BR2_LZIP_HOST_DEPENDENCY)
|
|
|
|
|
|
|
|
|
|
# extractor-pkg-dependency(filename): returns a Buildroot package
|
|
|
|
|
# dependency needed to extract file based on suffix
|
|
|
|
|
extractor-pkg-dependency = $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1)))
|
|
|
|
|
|
2019-12-16 23:31:13 +08:00
|
|
|
# extractor-system-dependency(filename): returns the name of the tool
|
|
|
|
|
# needed to extract 'filename', and is meant to be used with
|
|
|
|
|
# DL_TOOLS_DEPENDENCIES, in order to check that the necesary tool is
|
|
|
|
|
# provided by the system Buildroot runs on.
|
|
|
|
|
#
|
|
|
|
|
# $(firstword) is used here because the extractor can have arguments,
|
|
|
|
|
# like ZCAT="gzip -d -c", and to check for the dependency we only want
|
|
|
|
|
# 'gzip'.
|
2019-12-16 23:31:15 +08:00
|
|
|
extractor-system-dependency = $(if $(EXTRACTOR_PKG_DEPENDENCY$(suffix $(1))),,\
|
|
|
|
|
$(firstword $(INFLATE$(suffix $(1)))))
|
2017-02-13 04:15:38 +08:00
|
|
|
|
2014-10-05 15:58:31 +08:00
|
|
|
# check-deprecated-variable -- throw an error on deprecated variables
|
|
|
|
|
# example:
|
|
|
|
|
# $(eval $(call check-deprecated-variable,FOO_MAKE_OPT,FOO_MAKE_OPTS))
|
|
|
|
|
define check-deprecated-variable # (deprecated var, new var)
|
|
|
|
|
ifneq ($$(origin $(1)),undefined)
|
|
|
|
|
$$(error Package error: use $(2) instead of $(1). Please fix your .mk file)
|
|
|
|
|
endif
|
|
|
|
|
endef
|
|
|
|
|
|
2019-08-06 19:02:03 +08:00
|
|
|
# $(1): YES or NO
|
|
|
|
|
define yesno-to-bool
|
|
|
|
|
$(subst NO,false,$(subst YES,true,$(1)))
|
|
|
|
|
endef
|
|
|
|
|
|
core: introduce new global show-info
Users are increasingly trying to extract information about packages. For
example, they might need to get the list of URIs, or the dependencies of
a package.
Although we do have a bunch of rules to generate some of that, this is
done in ad-hoc way, with most of the output formats just ad-hoc, raw,
unformatted blurbs, mostly internal data dumped as-is.
Introduce a new rule, show-info, that provides a properly formatted
output of all the meta-information about packages: name, type, version,
licenses, dependencies...
We choose to use JSON as the output format, because it is pretty
versatile, has parsers in virtually all languages, has tools to parse
from the shell (jq). It also closely matches Python data structure,
which makes it easy to use with our own internal tools as well. Finally,
JSON being a key-value store, allows for easy expanding the output
without requiring existing consumers to be updated; new, unknown keys
are simply ignored by those (as long as they are true JSON parsers).
The complex part of this change was the conditional output of parts of
the data: virtual packages have no source, version, license or
downloads, unlike non-virtual packages. Same goes for filesystems. We
use a wrapper macro, show-info, that de-multiplexes unto either the
package-related- or filesystem-related macros, and for packages, we also
use a detailed macro for non-virtual packages.
It is non-trivial to properly output correct JSON blurbs, especially
when trying to output an array of objects, like so, where the last item
shall not be followed by a comma: [ { ... }, { ... } ]
So, we use a trick (as sugegsted by Arnout), to $(subst) any pair of
",}" or ", }" or ",]" or ", ]" with only the respective closing symbol,
"}" or "]".
The whole stuff is $(strip)ed to make it a somewhat-minified JSON blurb
that fits on a single line with all spaces squashed (but still with
spaces, as it is not possible to differentiate spaces between JSON
elements from spaces inside JSON strings).
Reported-by: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-04-16 03:47:31 +08:00
|
|
|
# json-info -- return package or filesystem metadata formatted as an entry
|
|
|
|
|
# of a JSON dictionnary
|
|
|
|
|
# $(1): upper-case package or filesystem name
|
|
|
|
|
define json-info
|
|
|
|
|
"$($(1)_NAME)": {
|
2021-11-13 21:28:23 +08:00
|
|
|
"type": $(call mk-json-str,$($(1)_TYPE)),
|
core: introduce new global show-info
Users are increasingly trying to extract information about packages. For
example, they might need to get the list of URIs, or the dependencies of
a package.
Although we do have a bunch of rules to generate some of that, this is
done in ad-hoc way, with most of the output formats just ad-hoc, raw,
unformatted blurbs, mostly internal data dumped as-is.
Introduce a new rule, show-info, that provides a properly formatted
output of all the meta-information about packages: name, type, version,
licenses, dependencies...
We choose to use JSON as the output format, because it is pretty
versatile, has parsers in virtually all languages, has tools to parse
from the shell (jq). It also closely matches Python data structure,
which makes it easy to use with our own internal tools as well. Finally,
JSON being a key-value store, allows for easy expanding the output
without requiring existing consumers to be updated; new, unknown keys
are simply ignored by those (as long as they are true JSON parsers).
The complex part of this change was the conditional output of parts of
the data: virtual packages have no source, version, license or
downloads, unlike non-virtual packages. Same goes for filesystems. We
use a wrapper macro, show-info, that de-multiplexes unto either the
package-related- or filesystem-related macros, and for packages, we also
use a detailed macro for non-virtual packages.
It is non-trivial to properly output correct JSON blurbs, especially
when trying to output an array of objects, like so, where the last item
shall not be followed by a comma: [ { ... }, { ... } ]
So, we use a trick (as sugegsted by Arnout), to $(subst) any pair of
",}" or ", }" or ",]" or ", ]" with only the respective closing symbol,
"}" or "]".
The whole stuff is $(strip)ed to make it a somewhat-minified JSON blurb
that fits on a single line with all spaces squashed (but still with
spaces, as it is not possible to differentiate spaces between JSON
elements from spaces inside JSON strings).
Reported-by: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-04-16 03:47:31 +08:00
|
|
|
$(if $(filter rootfs,$($(1)_TYPE)), \
|
|
|
|
|
$(call _json-info-fs,$(1)), \
|
|
|
|
|
$(call _json-info-pkg,$(1)), \
|
|
|
|
|
)
|
|
|
|
|
}
|
|
|
|
|
endef
|
|
|
|
|
|
|
|
|
|
# _json-info-pkg, _json-info-pkg-details, _json-info-fs: private helpers
|
|
|
|
|
# for json-info, above
|
|
|
|
|
define _json-info-pkg
|
2021-11-13 21:28:23 +08:00
|
|
|
"name": $(call mk-json-str,$($(1)_RAWNAME)),
|
core: introduce new global show-info
Users are increasingly trying to extract information about packages. For
example, they might need to get the list of URIs, or the dependencies of
a package.
Although we do have a bunch of rules to generate some of that, this is
done in ad-hoc way, with most of the output formats just ad-hoc, raw,
unformatted blurbs, mostly internal data dumped as-is.
Introduce a new rule, show-info, that provides a properly formatted
output of all the meta-information about packages: name, type, version,
licenses, dependencies...
We choose to use JSON as the output format, because it is pretty
versatile, has parsers in virtually all languages, has tools to parse
from the shell (jq). It also closely matches Python data structure,
which makes it easy to use with our own internal tools as well. Finally,
JSON being a key-value store, allows for easy expanding the output
without requiring existing consumers to be updated; new, unknown keys
are simply ignored by those (as long as they are true JSON parsers).
The complex part of this change was the conditional output of parts of
the data: virtual packages have no source, version, license or
downloads, unlike non-virtual packages. Same goes for filesystems. We
use a wrapper macro, show-info, that de-multiplexes unto either the
package-related- or filesystem-related macros, and for packages, we also
use a detailed macro for non-virtual packages.
It is non-trivial to properly output correct JSON blurbs, especially
when trying to output an array of objects, like so, where the last item
shall not be followed by a comma: [ { ... }, { ... } ]
So, we use a trick (as sugegsted by Arnout), to $(subst) any pair of
",}" or ", }" or ",]" or ", ]" with only the respective closing symbol,
"}" or "]".
The whole stuff is $(strip)ed to make it a somewhat-minified JSON blurb
that fits on a single line with all spaces squashed (but still with
spaces, as it is not possible to differentiate spaces between JSON
elements from spaces inside JSON strings).
Reported-by: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-04-16 03:47:31 +08:00
|
|
|
$(if $($(1)_IS_VIRTUAL), \
|
|
|
|
|
"virtual": true$(comma),
|
|
|
|
|
"virtual": false$(comma)
|
|
|
|
|
$(call _json-info-pkg-details,$(1)) \
|
|
|
|
|
)
|
2021-11-13 21:28:25 +08:00
|
|
|
"stamp_dir": $(call mk-json-str,$(patsubst $(CONFIG_DIR)/%,%,$($(1)_DIR))),
|
|
|
|
|
"source_dir": $(call mk-json-str,$(patsubst $(CONFIG_DIR)/%,%,$($(1)_DIR))),
|
2021-11-13 21:28:23 +08:00
|
|
|
"build_dir": $(call mk-json-str,$(patsubst $(CONFIG_DIR)/%,%,$($(1)_BUILDDIR))),
|
2020-04-11 16:12:28 +08:00
|
|
|
$(if $(filter target,$($(1)_TYPE)), \
|
|
|
|
|
"install_target": $(call yesno-to-bool,$($(1)_INSTALL_TARGET))$(comma) \
|
|
|
|
|
"install_staging": $(call yesno-to-bool,$($(1)_INSTALL_STAGING))$(comma) \
|
|
|
|
|
"install_images": $(call yesno-to-bool,$($(1)_INSTALL_IMAGES))$(comma) \
|
|
|
|
|
)
|
core: introduce new global show-info
Users are increasingly trying to extract information about packages. For
example, they might need to get the list of URIs, or the dependencies of
a package.
Although we do have a bunch of rules to generate some of that, this is
done in ad-hoc way, with most of the output formats just ad-hoc, raw,
unformatted blurbs, mostly internal data dumped as-is.
Introduce a new rule, show-info, that provides a properly formatted
output of all the meta-information about packages: name, type, version,
licenses, dependencies...
We choose to use JSON as the output format, because it is pretty
versatile, has parsers in virtually all languages, has tools to parse
from the shell (jq). It also closely matches Python data structure,
which makes it easy to use with our own internal tools as well. Finally,
JSON being a key-value store, allows for easy expanding the output
without requiring existing consumers to be updated; new, unknown keys
are simply ignored by those (as long as they are true JSON parsers).
The complex part of this change was the conditional output of parts of
the data: virtual packages have no source, version, license or
downloads, unlike non-virtual packages. Same goes for filesystems. We
use a wrapper macro, show-info, that de-multiplexes unto either the
package-related- or filesystem-related macros, and for packages, we also
use a detailed macro for non-virtual packages.
It is non-trivial to properly output correct JSON blurbs, especially
when trying to output an array of objects, like so, where the last item
shall not be followed by a comma: [ { ... }, { ... } ]
So, we use a trick (as sugegsted by Arnout), to $(subst) any pair of
",}" or ", }" or ",]" or ", ]" with only the respective closing symbol,
"}" or "]".
The whole stuff is $(strip)ed to make it a somewhat-minified JSON blurb
that fits on a single line with all spaces squashed (but still with
spaces, as it is not possible to differentiate spaces between JSON
elements from spaces inside JSON strings).
Reported-by: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-04-16 03:47:31 +08:00
|
|
|
"dependencies": [
|
2021-11-13 21:28:23 +08:00
|
|
|
$(call make-comma-list, \
|
|
|
|
|
$(foreach dep,$(sort $($(1)_FINAL_ALL_DEPENDENCIES)), \
|
|
|
|
|
$(call mk-json-str,$(dep)) \
|
|
|
|
|
) \
|
|
|
|
|
)
|
core: introduce new global show-info
Users are increasingly trying to extract information about packages. For
example, they might need to get the list of URIs, or the dependencies of
a package.
Although we do have a bunch of rules to generate some of that, this is
done in ad-hoc way, with most of the output formats just ad-hoc, raw,
unformatted blurbs, mostly internal data dumped as-is.
Introduce a new rule, show-info, that provides a properly formatted
output of all the meta-information about packages: name, type, version,
licenses, dependencies...
We choose to use JSON as the output format, because it is pretty
versatile, has parsers in virtually all languages, has tools to parse
from the shell (jq). It also closely matches Python data structure,
which makes it easy to use with our own internal tools as well. Finally,
JSON being a key-value store, allows for easy expanding the output
without requiring existing consumers to be updated; new, unknown keys
are simply ignored by those (as long as they are true JSON parsers).
The complex part of this change was the conditional output of parts of
the data: virtual packages have no source, version, license or
downloads, unlike non-virtual packages. Same goes for filesystems. We
use a wrapper macro, show-info, that de-multiplexes unto either the
package-related- or filesystem-related macros, and for packages, we also
use a detailed macro for non-virtual packages.
It is non-trivial to properly output correct JSON blurbs, especially
when trying to output an array of objects, like so, where the last item
shall not be followed by a comma: [ { ... }, { ... } ]
So, we use a trick (as sugegsted by Arnout), to $(subst) any pair of
",}" or ", }" or ",]" or ", ]" with only the respective closing symbol,
"}" or "]".
The whole stuff is $(strip)ed to make it a somewhat-minified JSON blurb
that fits on a single line with all spaces squashed (but still with
spaces, as it is not possible to differentiate spaces between JSON
elements from spaces inside JSON strings).
Reported-by: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-04-16 03:47:31 +08:00
|
|
|
],
|
|
|
|
|
"reverse_dependencies": [
|
2021-11-13 21:28:23 +08:00
|
|
|
$(call make-comma-list, \
|
|
|
|
|
$(foreach dep,$(sort $($(1)_RDEPENDENCIES)), \
|
|
|
|
|
$(call mk-json-str,$(dep)) \
|
|
|
|
|
) \
|
|
|
|
|
)
|
core: introduce new global show-info
Users are increasingly trying to extract information about packages. For
example, they might need to get the list of URIs, or the dependencies of
a package.
Although we do have a bunch of rules to generate some of that, this is
done in ad-hoc way, with most of the output formats just ad-hoc, raw,
unformatted blurbs, mostly internal data dumped as-is.
Introduce a new rule, show-info, that provides a properly formatted
output of all the meta-information about packages: name, type, version,
licenses, dependencies...
We choose to use JSON as the output format, because it is pretty
versatile, has parsers in virtually all languages, has tools to parse
from the shell (jq). It also closely matches Python data structure,
which makes it easy to use with our own internal tools as well. Finally,
JSON being a key-value store, allows for easy expanding the output
without requiring existing consumers to be updated; new, unknown keys
are simply ignored by those (as long as they are true JSON parsers).
The complex part of this change was the conditional output of parts of
the data: virtual packages have no source, version, license or
downloads, unlike non-virtual packages. Same goes for filesystems. We
use a wrapper macro, show-info, that de-multiplexes unto either the
package-related- or filesystem-related macros, and for packages, we also
use a detailed macro for non-virtual packages.
It is non-trivial to properly output correct JSON blurbs, especially
when trying to output an array of objects, like so, where the last item
shall not be followed by a comma: [ { ... }, { ... } ]
So, we use a trick (as sugegsted by Arnout), to $(subst) any pair of
",}" or ", }" or ",]" or ", ]" with only the respective closing symbol,
"}" or "]".
The whole stuff is $(strip)ed to make it a somewhat-minified JSON blurb
that fits on a single line with all spaces squashed (but still with
spaces, as it is not possible to differentiate spaces between JSON
elements from spaces inside JSON strings).
Reported-by: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-04-16 03:47:31 +08:00
|
|
|
]
|
2020-11-04 22:51:39 +08:00
|
|
|
$(if $($(1)_CPE_ID_VALID), \
|
2021-11-13 21:28:23 +08:00
|
|
|
$(comma) "cpe-id": $(call mk-json-str,$($(1)_CPE_ID)) \
|
2020-11-04 22:51:39 +08:00
|
|
|
)
|
2020-07-24 23:43:51 +08:00
|
|
|
$(if $($(1)_IGNORE_CVES),
|
|
|
|
|
$(comma) "ignore_cves": [
|
2021-11-13 21:28:23 +08:00
|
|
|
$(call make-comma-list, \
|
|
|
|
|
$(foreach cve,$(sort $($(1)_IGNORE_CVES)), \
|
|
|
|
|
$(call mk-json-str,$(cve)) \
|
|
|
|
|
) \
|
|
|
|
|
)
|
2020-07-24 23:43:51 +08:00
|
|
|
]
|
|
|
|
|
)
|
core: introduce new global show-info
Users are increasingly trying to extract information about packages. For
example, they might need to get the list of URIs, or the dependencies of
a package.
Although we do have a bunch of rules to generate some of that, this is
done in ad-hoc way, with most of the output formats just ad-hoc, raw,
unformatted blurbs, mostly internal data dumped as-is.
Introduce a new rule, show-info, that provides a properly formatted
output of all the meta-information about packages: name, type, version,
licenses, dependencies...
We choose to use JSON as the output format, because it is pretty
versatile, has parsers in virtually all languages, has tools to parse
from the shell (jq). It also closely matches Python data structure,
which makes it easy to use with our own internal tools as well. Finally,
JSON being a key-value store, allows for easy expanding the output
without requiring existing consumers to be updated; new, unknown keys
are simply ignored by those (as long as they are true JSON parsers).
The complex part of this change was the conditional output of parts of
the data: virtual packages have no source, version, license or
downloads, unlike non-virtual packages. Same goes for filesystems. We
use a wrapper macro, show-info, that de-multiplexes unto either the
package-related- or filesystem-related macros, and for packages, we also
use a detailed macro for non-virtual packages.
It is non-trivial to properly output correct JSON blurbs, especially
when trying to output an array of objects, like so, where the last item
shall not be followed by a comma: [ { ... }, { ... } ]
So, we use a trick (as sugegsted by Arnout), to $(subst) any pair of
",}" or ", }" or ",]" or ", ]" with only the respective closing symbol,
"}" or "]".
The whole stuff is $(strip)ed to make it a somewhat-minified JSON blurb
that fits on a single line with all spaces squashed (but still with
spaces, as it is not possible to differentiate spaces between JSON
elements from spaces inside JSON strings).
Reported-by: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-04-16 03:47:31 +08:00
|
|
|
endef
|
|
|
|
|
|
|
|
|
|
define _json-info-pkg-details
|
2021-11-13 21:28:23 +08:00
|
|
|
"version": $(call mk-json-str,$($(1)_DL_VERSION)),
|
|
|
|
|
"licenses": $(call mk-json-str,$($(1)_LICENSE)),
|
|
|
|
|
"dl_dir": $(call mk-json-str,$($(1)_DL_SUBDIR)),
|
core: introduce new global show-info
Users are increasingly trying to extract information about packages. For
example, they might need to get the list of URIs, or the dependencies of
a package.
Although we do have a bunch of rules to generate some of that, this is
done in ad-hoc way, with most of the output formats just ad-hoc, raw,
unformatted blurbs, mostly internal data dumped as-is.
Introduce a new rule, show-info, that provides a properly formatted
output of all the meta-information about packages: name, type, version,
licenses, dependencies...
We choose to use JSON as the output format, because it is pretty
versatile, has parsers in virtually all languages, has tools to parse
from the shell (jq). It also closely matches Python data structure,
which makes it easy to use with our own internal tools as well. Finally,
JSON being a key-value store, allows for easy expanding the output
without requiring existing consumers to be updated; new, unknown keys
are simply ignored by those (as long as they are true JSON parsers).
The complex part of this change was the conditional output of parts of
the data: virtual packages have no source, version, license or
downloads, unlike non-virtual packages. Same goes for filesystems. We
use a wrapper macro, show-info, that de-multiplexes unto either the
package-related- or filesystem-related macros, and for packages, we also
use a detailed macro for non-virtual packages.
It is non-trivial to properly output correct JSON blurbs, especially
when trying to output an array of objects, like so, where the last item
shall not be followed by a comma: [ { ... }, { ... } ]
So, we use a trick (as sugegsted by Arnout), to $(subst) any pair of
",}" or ", }" or ",]" or ", ]" with only the respective closing symbol,
"}" or "]".
The whole stuff is $(strip)ed to make it a somewhat-minified JSON blurb
that fits on a single line with all spaces squashed (but still with
spaces, as it is not possible to differentiate spaces between JSON
elements from spaces inside JSON strings).
Reported-by: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-04-16 03:47:31 +08:00
|
|
|
"downloads": [
|
|
|
|
|
$(foreach dl,$(sort $($(1)_ALL_DOWNLOADS)),
|
|
|
|
|
{
|
2021-11-13 21:28:23 +08:00
|
|
|
"source": $(call mk-json-str,$(notdir $(dl))),
|
core: introduce new global show-info
Users are increasingly trying to extract information about packages. For
example, they might need to get the list of URIs, or the dependencies of
a package.
Although we do have a bunch of rules to generate some of that, this is
done in ad-hoc way, with most of the output formats just ad-hoc, raw,
unformatted blurbs, mostly internal data dumped as-is.
Introduce a new rule, show-info, that provides a properly formatted
output of all the meta-information about packages: name, type, version,
licenses, dependencies...
We choose to use JSON as the output format, because it is pretty
versatile, has parsers in virtually all languages, has tools to parse
from the shell (jq). It also closely matches Python data structure,
which makes it easy to use with our own internal tools as well. Finally,
JSON being a key-value store, allows for easy expanding the output
without requiring existing consumers to be updated; new, unknown keys
are simply ignored by those (as long as they are true JSON parsers).
The complex part of this change was the conditional output of parts of
the data: virtual packages have no source, version, license or
downloads, unlike non-virtual packages. Same goes for filesystems. We
use a wrapper macro, show-info, that de-multiplexes unto either the
package-related- or filesystem-related macros, and for packages, we also
use a detailed macro for non-virtual packages.
It is non-trivial to properly output correct JSON blurbs, especially
when trying to output an array of objects, like so, where the last item
shall not be followed by a comma: [ { ... }, { ... } ]
So, we use a trick (as sugegsted by Arnout), to $(subst) any pair of
",}" or ", }" or ",]" or ", ]" with only the respective closing symbol,
"}" or "]".
The whole stuff is $(strip)ed to make it a somewhat-minified JSON blurb
that fits on a single line with all spaces squashed (but still with
spaces, as it is not possible to differentiate spaces between JSON
elements from spaces inside JSON strings).
Reported-by: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-04-16 03:47:31 +08:00
|
|
|
"uris": [
|
2021-11-13 21:28:23 +08:00
|
|
|
$(call make-comma-list, \
|
|
|
|
|
$(foreach uri,$(call DOWNLOAD_URIS,$(dl),$(1)), \
|
|
|
|
|
$(call mk-json-str,$(subst \|,|,$(uri))) \
|
|
|
|
|
) \
|
core: introduce new global show-info
Users are increasingly trying to extract information about packages. For
example, they might need to get the list of URIs, or the dependencies of
a package.
Although we do have a bunch of rules to generate some of that, this is
done in ad-hoc way, with most of the output formats just ad-hoc, raw,
unformatted blurbs, mostly internal data dumped as-is.
Introduce a new rule, show-info, that provides a properly formatted
output of all the meta-information about packages: name, type, version,
licenses, dependencies...
We choose to use JSON as the output format, because it is pretty
versatile, has parsers in virtually all languages, has tools to parse
from the shell (jq). It also closely matches Python data structure,
which makes it easy to use with our own internal tools as well. Finally,
JSON being a key-value store, allows for easy expanding the output
without requiring existing consumers to be updated; new, unknown keys
are simply ignored by those (as long as they are true JSON parsers).
The complex part of this change was the conditional output of parts of
the data: virtual packages have no source, version, license or
downloads, unlike non-virtual packages. Same goes for filesystems. We
use a wrapper macro, show-info, that de-multiplexes unto either the
package-related- or filesystem-related macros, and for packages, we also
use a detailed macro for non-virtual packages.
It is non-trivial to properly output correct JSON blurbs, especially
when trying to output an array of objects, like so, where the last item
shall not be followed by a comma: [ { ... }, { ... } ]
So, we use a trick (as sugegsted by Arnout), to $(subst) any pair of
",}" or ", }" or ",]" or ", ]" with only the respective closing symbol,
"}" or "]".
The whole stuff is $(strip)ed to make it a somewhat-minified JSON blurb
that fits on a single line with all spaces squashed (but still with
spaces, as it is not possible to differentiate spaces between JSON
elements from spaces inside JSON strings).
Reported-by: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-04-16 03:47:31 +08:00
|
|
|
)
|
|
|
|
|
]
|
|
|
|
|
},
|
|
|
|
|
)
|
|
|
|
|
],
|
|
|
|
|
endef
|
|
|
|
|
|
|
|
|
|
define _json-info-fs
|
2020-04-11 16:12:33 +08:00
|
|
|
"image_name": $(if $($(1)_FINAL_IMAGE_NAME), \
|
2021-11-13 21:28:23 +08:00
|
|
|
$(call mk-json-str,$($(1)_FINAL_IMAGE_NAME)), \
|
2020-04-11 16:12:33 +08:00
|
|
|
null \
|
|
|
|
|
),
|
core: introduce new global show-info
Users are increasingly trying to extract information about packages. For
example, they might need to get the list of URIs, or the dependencies of
a package.
Although we do have a bunch of rules to generate some of that, this is
done in ad-hoc way, with most of the output formats just ad-hoc, raw,
unformatted blurbs, mostly internal data dumped as-is.
Introduce a new rule, show-info, that provides a properly formatted
output of all the meta-information about packages: name, type, version,
licenses, dependencies...
We choose to use JSON as the output format, because it is pretty
versatile, has parsers in virtually all languages, has tools to parse
from the shell (jq). It also closely matches Python data structure,
which makes it easy to use with our own internal tools as well. Finally,
JSON being a key-value store, allows for easy expanding the output
without requiring existing consumers to be updated; new, unknown keys
are simply ignored by those (as long as they are true JSON parsers).
The complex part of this change was the conditional output of parts of
the data: virtual packages have no source, version, license or
downloads, unlike non-virtual packages. Same goes for filesystems. We
use a wrapper macro, show-info, that de-multiplexes unto either the
package-related- or filesystem-related macros, and for packages, we also
use a detailed macro for non-virtual packages.
It is non-trivial to properly output correct JSON blurbs, especially
when trying to output an array of objects, like so, where the last item
shall not be followed by a comma: [ { ... }, { ... } ]
So, we use a trick (as sugegsted by Arnout), to $(subst) any pair of
",}" or ", }" or ",]" or ", ]" with only the respective closing symbol,
"}" or "]".
The whole stuff is $(strip)ed to make it a somewhat-minified JSON blurb
that fits on a single line with all spaces squashed (but still with
spaces, as it is not possible to differentiate spaces between JSON
elements from spaces inside JSON strings).
Reported-by: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-04-16 03:47:31 +08:00
|
|
|
"dependencies": [
|
2021-11-13 21:28:23 +08:00
|
|
|
$(call make-comma-list, \
|
|
|
|
|
$(foreach dep,$(sort $($(1)_DEPENDENCIES)), \
|
|
|
|
|
$(call mk-json-str,$(dep)) \
|
|
|
|
|
) \
|
|
|
|
|
)
|
core: introduce new global show-info
Users are increasingly trying to extract information about packages. For
example, they might need to get the list of URIs, or the dependencies of
a package.
Although we do have a bunch of rules to generate some of that, this is
done in ad-hoc way, with most of the output formats just ad-hoc, raw,
unformatted blurbs, mostly internal data dumped as-is.
Introduce a new rule, show-info, that provides a properly formatted
output of all the meta-information about packages: name, type, version,
licenses, dependencies...
We choose to use JSON as the output format, because it is pretty
versatile, has parsers in virtually all languages, has tools to parse
from the shell (jq). It also closely matches Python data structure,
which makes it easy to use with our own internal tools as well. Finally,
JSON being a key-value store, allows for easy expanding the output
without requiring existing consumers to be updated; new, unknown keys
are simply ignored by those (as long as they are true JSON parsers).
The complex part of this change was the conditional output of parts of
the data: virtual packages have no source, version, license or
downloads, unlike non-virtual packages. Same goes for filesystems. We
use a wrapper macro, show-info, that de-multiplexes unto either the
package-related- or filesystem-related macros, and for packages, we also
use a detailed macro for non-virtual packages.
It is non-trivial to properly output correct JSON blurbs, especially
when trying to output an array of objects, like so, where the last item
shall not be followed by a comma: [ { ... }, { ... } ]
So, we use a trick (as sugegsted by Arnout), to $(subst) any pair of
",}" or ", }" or ",]" or ", ]" with only the respective closing symbol,
"}" or "]".
The whole stuff is $(strip)ed to make it a somewhat-minified JSON blurb
that fits on a single line with all spaces squashed (but still with
spaces, as it is not possible to differentiate spaces between JSON
elements from spaces inside JSON strings).
Reported-by: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-04-16 03:47:31 +08:00
|
|
|
]
|
|
|
|
|
endef
|
|
|
|
|
|
|
|
|
|
# clean-json -- cleanup pseudo-json into clean json:
|
|
|
|
|
# - remove commas before closing ] and }
|
|
|
|
|
# - minify with $(strip)
|
|
|
|
|
clean-json = $(strip \
|
|
|
|
|
$(subst $(comma)},}, $(subst $(comma)$(space)},$(space)}, \
|
|
|
|
|
$(subst $(comma)],], $(subst $(comma)$(space)],$(space)], \
|
|
|
|
|
$(strip $(1)) \
|
2021-11-13 21:28:23 +08:00
|
|
|
)))) \
|
core: introduce new global show-info
Users are increasingly trying to extract information about packages. For
example, they might need to get the list of URIs, or the dependencies of
a package.
Although we do have a bunch of rules to generate some of that, this is
done in ad-hoc way, with most of the output formats just ad-hoc, raw,
unformatted blurbs, mostly internal data dumped as-is.
Introduce a new rule, show-info, that provides a properly formatted
output of all the meta-information about packages: name, type, version,
licenses, dependencies...
We choose to use JSON as the output format, because it is pretty
versatile, has parsers in virtually all languages, has tools to parse
from the shell (jq). It also closely matches Python data structure,
which makes it easy to use with our own internal tools as well. Finally,
JSON being a key-value store, allows for easy expanding the output
without requiring existing consumers to be updated; new, unknown keys
are simply ignored by those (as long as they are true JSON parsers).
The complex part of this change was the conditional output of parts of
the data: virtual packages have no source, version, license or
downloads, unlike non-virtual packages. Same goes for filesystems. We
use a wrapper macro, show-info, that de-multiplexes unto either the
package-related- or filesystem-related macros, and for packages, we also
use a detailed macro for non-virtual packages.
It is non-trivial to properly output correct JSON blurbs, especially
when trying to output an array of objects, like so, where the last item
shall not be followed by a comma: [ { ... }, { ... } ]
So, we use a trick (as sugegsted by Arnout), to $(subst) any pair of
",}" or ", }" or ",]" or ", ]" with only the respective closing symbol,
"}" or "]".
The whole stuff is $(strip)ed to make it a somewhat-minified JSON blurb
that fits on a single line with all spaces squashed (but still with
spaces, as it is not possible to differentiate spaces between JSON
elements from spaces inside JSON strings).
Reported-by: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: Thomas De Schampheleire <patrickdepinguin@gmail.com>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
2019-04-16 03:47:31 +08:00
|
|
|
)
|
|
|
|
|
|
package/pkg-utils: introduce helper to properly json-escape a string
In quite a few places, we need to generate string that are proper JSON
values or keys.
However, JSON is very strict on what constitute a string, and most JSON
parsers (like jq or python3's json module) are very picky when parsing a
string; any deviation from the spec is immediately sanctioned by a hard
error (jq aborts, python3's json module raise an exception).
Introduce a macro that properly prepares a Makefile value into a valid
JSON string:
- backslash '\' must be escaped;
- double-quotes need to be escaped of course, as they are the string
delimiter in JSON;
- anything in the range [0x00..0x1F] must be escaped; in practice, we
only ever need to escape \n, \t, and ESC (we could add more in the
future if need be);
- finally, we also escape the space, \x20, so that we can call
$(strip) on a JSON blurb (like we do for example do build a
comma-separated list, or when we sanitise the JSON) without losing
multiple spaces where they make sense.
It would have been nice if we had been able to split the macro on
multiple lines, but spaces creep in from everywhere in that case, and
getting rid of them is getting quite nasty... We could introduce
intermediate macros, but meh...
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2021-11-13 21:28:22 +08:00
|
|
|
# mk-json-str -- escape and double-quote a string to make it a valid json string
|
|
|
|
|
# - escape \
|
|
|
|
|
# - escape "
|
|
|
|
|
# - escape \n
|
|
|
|
|
# - escape \t
|
|
|
|
|
# - escape ESC
|
|
|
|
|
# - escape SPACE (so that we can $(strip) a JSON blurb without squashing multiple spaces)
|
|
|
|
|
# This unfortunately has to be on a single line...
|
|
|
|
|
mk-json-str = "$(subst $(space),\u0020,$(subst $(escape),\u001b,$(subst $(tab),\t,$(subst $(sep),\n,$(subst ",\",$(subst \,\\,$(1)))))))"
|
|
|
|
|
# )))))" # Syntax colouring
|
|
|
|
|
|
core: implement per-package SDK and target
This commit implements the core of the move to per-package SDK and
target directories. The main idea is that instead of having a global
output/host and output/target in which all packages install files, we
switch to per-package host and target directories, that only contain
their explicit dependencies.
There are two main benefits:
- Packages will now see only the dependencies they explicitly list in
their <pkg>_DEPENDENCIES variable, and the recursive dependencies
thereof.
- We can support top-level parallel build properly, because a package
only "sees" its own host directory and target directory, isolated
from the build of other packages that can happen in parallel.
It works as follows:
- A new output/per-package/ directory is created, which will contain
one sub-directory per package, and inside it, a "host" directory
and a "target" directory:
output/per-package/busybox/target
output/per-package/busybox/host
output/per-package/host-fakeroot/target
output/per-package/host-fakeroot/host
This output/per-package/ directory is PER_PACKAGE_DIR.
- The global TARGET_DIR and HOST_DIR variable now automatically point
to the per-package directory when PKG is defined. So whenever a
package references $(HOST_DIR) or $(TARGET_DIR) in its build
process, it effectively references the per-package host/target
directories. Note that STAGING_DIR is a sub-dir of HOST_DIR, so it
is handled as well.
- Of course, packages have dependencies, so those dependencies must
be installed in the per-package host and target directories. To do
so, we simply rsync (using hard links to save space and time) the
host and target directories of the direct dependencies of the
package to the current package host and target directories.
We only need to take care of direct dependencies (and not
recursively all dependencies), because we accumulate into those
per-package host and target directories the files installed by the
dependencies. Note that this only works because we make the
assumption that one package does *not* overwrite files installed by
another package.
This is done for "extract dependencies" at the beginning of the
extract step, and for "normal dependencies" at the beginning of the
configure step.
This is basically enough to make per-package SDK and target work. The
only gotcha is that at the end of the build, output/target and
output/host are empty, which means that:
- The filesystem image creation code cannot work.
- We don't have a SDK to build code outside of Buildroot.
In order to fix this, this commit extends the target-finalize step so
that it starts by populating output/target and output/host by
rsync-ing into them the target and host directories of all packages
listed in the $(PACKAGES) variable. It is necessary to do this
sequentially in the target-finalize step and not in each
package. Doing it in package installation means that it can be done in
parallel. In that case, there is a chance that two rsyncs are creating
the same hardlink or directory at the same time, which makes one of
them fail.
This change to per-package directories has an impact on the RPATH
built into the host binaries, as those RPATH now point to various
per-package host directories, and no longer to the global host
directory. We do not try to rewrite such RPATHs during the build as
having such RPATHs is perfectly fine, but we still need to handle two
fallouts from this change:
- The check-host-rpath script, which verifies at the end of each
package installation that it has the appropriate RPATH, is modified
to understand that a RPATH to $(PER_PACKAGE_DIR)/<pkg>/host/lib is
a correct RPAT.
- The fix-rpath script, which mungles the RPATH mainly for the SDK
preparation, is modified to rewrite the RPATH to not point to
per-package directories. Indeed the patchelf --make-rpath-relative
call only works if the RPATH points to the ROOTDIR passed as
argument, and this ROOTDIR is the global host directory. Rewriting
the RPATH to not point to per-package host directories prior to
this is an easy solution to this issue.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-11-06 00:46:40 +08:00
|
|
|
ifeq ($(BR2_PER_PACKAGE_DIRECTORIES),y)
|
|
|
|
|
# rsync the contents of per-package directories
|
|
|
|
|
# $1: space-separated list of packages to rsync from
|
|
|
|
|
# $2: 'host' or 'target'
|
|
|
|
|
# $3: destination directory
|
2023-10-18 05:01:20 +08:00
|
|
|
# $4: literal "copy" or "hardlink" to copy or hardlink files from src to dest
|
core: implement per-package SDK and target
This commit implements the core of the move to per-package SDK and
target directories. The main idea is that instead of having a global
output/host and output/target in which all packages install files, we
switch to per-package host and target directories, that only contain
their explicit dependencies.
There are two main benefits:
- Packages will now see only the dependencies they explicitly list in
their <pkg>_DEPENDENCIES variable, and the recursive dependencies
thereof.
- We can support top-level parallel build properly, because a package
only "sees" its own host directory and target directory, isolated
from the build of other packages that can happen in parallel.
It works as follows:
- A new output/per-package/ directory is created, which will contain
one sub-directory per package, and inside it, a "host" directory
and a "target" directory:
output/per-package/busybox/target
output/per-package/busybox/host
output/per-package/host-fakeroot/target
output/per-package/host-fakeroot/host
This output/per-package/ directory is PER_PACKAGE_DIR.
- The global TARGET_DIR and HOST_DIR variable now automatically point
to the per-package directory when PKG is defined. So whenever a
package references $(HOST_DIR) or $(TARGET_DIR) in its build
process, it effectively references the per-package host/target
directories. Note that STAGING_DIR is a sub-dir of HOST_DIR, so it
is handled as well.
- Of course, packages have dependencies, so those dependencies must
be installed in the per-package host and target directories. To do
so, we simply rsync (using hard links to save space and time) the
host and target directories of the direct dependencies of the
package to the current package host and target directories.
We only need to take care of direct dependencies (and not
recursively all dependencies), because we accumulate into those
per-package host and target directories the files installed by the
dependencies. Note that this only works because we make the
assumption that one package does *not* overwrite files installed by
another package.
This is done for "extract dependencies" at the beginning of the
extract step, and for "normal dependencies" at the beginning of the
configure step.
This is basically enough to make per-package SDK and target work. The
only gotcha is that at the end of the build, output/target and
output/host are empty, which means that:
- The filesystem image creation code cannot work.
- We don't have a SDK to build code outside of Buildroot.
In order to fix this, this commit extends the target-finalize step so
that it starts by populating output/target and output/host by
rsync-ing into them the target and host directories of all packages
listed in the $(PACKAGES) variable. It is necessary to do this
sequentially in the target-finalize step and not in each
package. Doing it in package installation means that it can be done in
parallel. In that case, there is a chance that two rsyncs are creating
the same hardlink or directory at the same time, which makes one of
them fail.
This change to per-package directories has an impact on the RPATH
built into the host binaries, as those RPATH now point to various
per-package host directories, and no longer to the global host
directory. We do not try to rewrite such RPATHs during the build as
having such RPATHs is perfectly fine, but we still need to handle two
fallouts from this change:
- The check-host-rpath script, which verifies at the end of each
package installation that it has the appropriate RPATH, is modified
to understand that a RPATH to $(PER_PACKAGE_DIR)/<pkg>/host/lib is
a correct RPAT.
- The fix-rpath script, which mungles the RPATH mainly for the SDK
preparation, is modified to rewrite the RPATH to not point to
per-package directories. Indeed the patchelf --make-rpath-relative
call only works if the RPATH points to the ROOTDIR passed as
argument, and this ROOTDIR is the global host directory. Rewriting
the RPATH to not point to per-package host directories prior to
this is an easy solution to this issue.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-11-06 00:46:40 +08:00
|
|
|
define per-package-rsync
|
Revert "package/pkg-generic.mk: create directories upfront in the configure step"
This _partially_ reverts commit 2dbb0660fb4d89f8eac86fcff08e9b3d0a705917.
Indeed, before 2dbb0660fb, the per-package ../host/ directory was
explicitly created right before it was populated. With 2dbb0660fb, it
is now created only at the beginning of the configure step.
However, some packages may have download or extract host dependencies,
and when we are trying to rsync them, the destination .../host/
directory does not yet exist at the download or extract steps, thus the
build fails. E.g., with wget which needs host-lzip if the system does
not have it yet:
rsync -a --link-dest=/home/ymorin/dev/buildroot/O/per-package/host-lzip/host/ /home/ymorin/dev/buildroot/O/per-package/host-lzip/host/ /home/ymorin/dev/buildroot/O/per-package/wget/host
rsync: mkdir "/home/ymorin/dev/buildroot/O/per-package/wget/host" failed: No such file or directory (2)
rsync error: error in file IO (code 11) at main.c(682) [Receiver=3.1.3]
make[1]: *** [package/pkg-generic.mk:170: /home/ymorin/dev/buildroot/O/build/wget-1.20.3/.stamp_extracted] Error 11
Kudos to Raphael for pin-pointing the culprit commit.
Fixes:
http://autobuild.buildroot.org/results/4c8/4c895c7ac0a635d38f5015b5df0393581ad7b508/
and a slew of similar failures
Notes:
An alternative solution would be to move the creation of the directories
from the configure step to the download step *and* to the rsync step,
but this is not so nice in the end... Let's just ensure they are created
where needed: at the configure step, and when populating them.
Reported-by: Raphael Jacob <r.jacob2002@gmail.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2020-08-01 19:44:16 +08:00
|
|
|
mkdir -p $(3)
|
core: implement per-package SDK and target
This commit implements the core of the move to per-package SDK and
target directories. The main idea is that instead of having a global
output/host and output/target in which all packages install files, we
switch to per-package host and target directories, that only contain
their explicit dependencies.
There are two main benefits:
- Packages will now see only the dependencies they explicitly list in
their <pkg>_DEPENDENCIES variable, and the recursive dependencies
thereof.
- We can support top-level parallel build properly, because a package
only "sees" its own host directory and target directory, isolated
from the build of other packages that can happen in parallel.
It works as follows:
- A new output/per-package/ directory is created, which will contain
one sub-directory per package, and inside it, a "host" directory
and a "target" directory:
output/per-package/busybox/target
output/per-package/busybox/host
output/per-package/host-fakeroot/target
output/per-package/host-fakeroot/host
This output/per-package/ directory is PER_PACKAGE_DIR.
- The global TARGET_DIR and HOST_DIR variable now automatically point
to the per-package directory when PKG is defined. So whenever a
package references $(HOST_DIR) or $(TARGET_DIR) in its build
process, it effectively references the per-package host/target
directories. Note that STAGING_DIR is a sub-dir of HOST_DIR, so it
is handled as well.
- Of course, packages have dependencies, so those dependencies must
be installed in the per-package host and target directories. To do
so, we simply rsync (using hard links to save space and time) the
host and target directories of the direct dependencies of the
package to the current package host and target directories.
We only need to take care of direct dependencies (and not
recursively all dependencies), because we accumulate into those
per-package host and target directories the files installed by the
dependencies. Note that this only works because we make the
assumption that one package does *not* overwrite files installed by
another package.
This is done for "extract dependencies" at the beginning of the
extract step, and for "normal dependencies" at the beginning of the
configure step.
This is basically enough to make per-package SDK and target work. The
only gotcha is that at the end of the build, output/target and
output/host are empty, which means that:
- The filesystem image creation code cannot work.
- We don't have a SDK to build code outside of Buildroot.
In order to fix this, this commit extends the target-finalize step so
that it starts by populating output/target and output/host by
rsync-ing into them the target and host directories of all packages
listed in the $(PACKAGES) variable. It is necessary to do this
sequentially in the target-finalize step and not in each
package. Doing it in package installation means that it can be done in
parallel. In that case, there is a chance that two rsyncs are creating
the same hardlink or directory at the same time, which makes one of
them fail.
This change to per-package directories has an impact on the RPATH
built into the host binaries, as those RPATH now point to various
per-package host directories, and no longer to the global host
directory. We do not try to rewrite such RPATHs during the build as
having such RPATHs is perfectly fine, but we still need to handle two
fallouts from this change:
- The check-host-rpath script, which verifies at the end of each
package installation that it has the appropriate RPATH, is modified
to understand that a RPATH to $(PER_PACKAGE_DIR)/<pkg>/host/lib is
a correct RPAT.
- The fix-rpath script, which mungles the RPATH mainly for the SDK
preparation, is modified to rewrite the RPATH to not point to
per-package directories. Indeed the patchelf --make-rpath-relative
call only works if the RPATH points to the ROOTDIR passed as
argument, and this ROOTDIR is the global host directory. Rewriting
the RPATH to not point to per-package host directories prior to
this is an easy solution to this issue.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-11-06 00:46:40 +08:00
|
|
|
$(foreach pkg,$(1),\
|
2023-10-18 05:01:20 +08:00
|
|
|
rsync -a \
|
|
|
|
|
--hard-links \
|
|
|
|
|
$(if $(filter hardlink,$(4)), \
|
|
|
|
|
--link-dest=$(PER_PACKAGE_DIR)/$(pkg)/$(2)/, \
|
|
|
|
|
$(if $(filter copy,$(4)), \
|
|
|
|
|
$(empty), \
|
|
|
|
|
$(error per-package-rsync can only "copy" or "hardlink", not "$(4)") \
|
|
|
|
|
) \
|
|
|
|
|
) \
|
2022-01-09 01:35:40 +08:00
|
|
|
$(PER_PACKAGE_DIR)/$(pkg)/$(2)/ \
|
2022-01-09 14:44:06 +08:00
|
|
|
$(3)$(sep))
|
core: implement per-package SDK and target
This commit implements the core of the move to per-package SDK and
target directories. The main idea is that instead of having a global
output/host and output/target in which all packages install files, we
switch to per-package host and target directories, that only contain
their explicit dependencies.
There are two main benefits:
- Packages will now see only the dependencies they explicitly list in
their <pkg>_DEPENDENCIES variable, and the recursive dependencies
thereof.
- We can support top-level parallel build properly, because a package
only "sees" its own host directory and target directory, isolated
from the build of other packages that can happen in parallel.
It works as follows:
- A new output/per-package/ directory is created, which will contain
one sub-directory per package, and inside it, a "host" directory
and a "target" directory:
output/per-package/busybox/target
output/per-package/busybox/host
output/per-package/host-fakeroot/target
output/per-package/host-fakeroot/host
This output/per-package/ directory is PER_PACKAGE_DIR.
- The global TARGET_DIR and HOST_DIR variable now automatically point
to the per-package directory when PKG is defined. So whenever a
package references $(HOST_DIR) or $(TARGET_DIR) in its build
process, it effectively references the per-package host/target
directories. Note that STAGING_DIR is a sub-dir of HOST_DIR, so it
is handled as well.
- Of course, packages have dependencies, so those dependencies must
be installed in the per-package host and target directories. To do
so, we simply rsync (using hard links to save space and time) the
host and target directories of the direct dependencies of the
package to the current package host and target directories.
We only need to take care of direct dependencies (and not
recursively all dependencies), because we accumulate into those
per-package host and target directories the files installed by the
dependencies. Note that this only works because we make the
assumption that one package does *not* overwrite files installed by
another package.
This is done for "extract dependencies" at the beginning of the
extract step, and for "normal dependencies" at the beginning of the
configure step.
This is basically enough to make per-package SDK and target work. The
only gotcha is that at the end of the build, output/target and
output/host are empty, which means that:
- The filesystem image creation code cannot work.
- We don't have a SDK to build code outside of Buildroot.
In order to fix this, this commit extends the target-finalize step so
that it starts by populating output/target and output/host by
rsync-ing into them the target and host directories of all packages
listed in the $(PACKAGES) variable. It is necessary to do this
sequentially in the target-finalize step and not in each
package. Doing it in package installation means that it can be done in
parallel. In that case, there is a chance that two rsyncs are creating
the same hardlink or directory at the same time, which makes one of
them fail.
This change to per-package directories has an impact on the RPATH
built into the host binaries, as those RPATH now point to various
per-package host directories, and no longer to the global host
directory. We do not try to rewrite such RPATHs during the build as
having such RPATHs is perfectly fine, but we still need to handle two
fallouts from this change:
- The check-host-rpath script, which verifies at the end of each
package installation that it has the appropriate RPATH, is modified
to understand that a RPATH to $(PER_PACKAGE_DIR)/<pkg>/host/lib is
a correct RPAT.
- The fix-rpath script, which mungles the RPATH mainly for the SDK
preparation, is modified to rewrite the RPATH to not point to
per-package directories. Indeed the patchelf --make-rpath-relative
call only works if the RPATH points to the ROOTDIR passed as
argument, and this ROOTDIR is the global host directory. Rewriting
the RPATH to not point to per-package host directories prior to
this is an easy solution to this issue.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-11-06 00:46:40 +08:00
|
|
|
endef
|
|
|
|
|
|
|
|
|
|
# prepares the per-package HOST_DIR and TARGET_DIR of the current
|
|
|
|
|
# package, by rsync the host and target directories of the
|
|
|
|
|
# dependencies of this package. The list of dependencies is passed as
|
|
|
|
|
# argument, so that this function can be used to prepare with
|
|
|
|
|
# different set of dependencies (download, extract, configure, etc.)
|
|
|
|
|
#
|
|
|
|
|
# $1: space-separated list of packages to rsync from
|
|
|
|
|
define prepare-per-package-directory
|
2023-10-18 05:01:20 +08:00
|
|
|
$(call per-package-rsync,$(1),host,$(HOST_DIR),hardlink)
|
|
|
|
|
$(call per-package-rsync,$(1),target,$(TARGET_DIR),hardlink)
|
core: implement per-package SDK and target
This commit implements the core of the move to per-package SDK and
target directories. The main idea is that instead of having a global
output/host and output/target in which all packages install files, we
switch to per-package host and target directories, that only contain
their explicit dependencies.
There are two main benefits:
- Packages will now see only the dependencies they explicitly list in
their <pkg>_DEPENDENCIES variable, and the recursive dependencies
thereof.
- We can support top-level parallel build properly, because a package
only "sees" its own host directory and target directory, isolated
from the build of other packages that can happen in parallel.
It works as follows:
- A new output/per-package/ directory is created, which will contain
one sub-directory per package, and inside it, a "host" directory
and a "target" directory:
output/per-package/busybox/target
output/per-package/busybox/host
output/per-package/host-fakeroot/target
output/per-package/host-fakeroot/host
This output/per-package/ directory is PER_PACKAGE_DIR.
- The global TARGET_DIR and HOST_DIR variable now automatically point
to the per-package directory when PKG is defined. So whenever a
package references $(HOST_DIR) or $(TARGET_DIR) in its build
process, it effectively references the per-package host/target
directories. Note that STAGING_DIR is a sub-dir of HOST_DIR, so it
is handled as well.
- Of course, packages have dependencies, so those dependencies must
be installed in the per-package host and target directories. To do
so, we simply rsync (using hard links to save space and time) the
host and target directories of the direct dependencies of the
package to the current package host and target directories.
We only need to take care of direct dependencies (and not
recursively all dependencies), because we accumulate into those
per-package host and target directories the files installed by the
dependencies. Note that this only works because we make the
assumption that one package does *not* overwrite files installed by
another package.
This is done for "extract dependencies" at the beginning of the
extract step, and for "normal dependencies" at the beginning of the
configure step.
This is basically enough to make per-package SDK and target work. The
only gotcha is that at the end of the build, output/target and
output/host are empty, which means that:
- The filesystem image creation code cannot work.
- We don't have a SDK to build code outside of Buildroot.
In order to fix this, this commit extends the target-finalize step so
that it starts by populating output/target and output/host by
rsync-ing into them the target and host directories of all packages
listed in the $(PACKAGES) variable. It is necessary to do this
sequentially in the target-finalize step and not in each
package. Doing it in package installation means that it can be done in
parallel. In that case, there is a chance that two rsyncs are creating
the same hardlink or directory at the same time, which makes one of
them fail.
This change to per-package directories has an impact on the RPATH
built into the host binaries, as those RPATH now point to various
per-package host directories, and no longer to the global host
directory. We do not try to rewrite such RPATHs during the build as
having such RPATHs is perfectly fine, but we still need to handle two
fallouts from this change:
- The check-host-rpath script, which verifies at the end of each
package installation that it has the appropriate RPATH, is modified
to understand that a RPATH to $(PER_PACKAGE_DIR)/<pkg>/host/lib is
a correct RPAT.
- The fix-rpath script, which mungles the RPATH mainly for the SDK
preparation, is modified to rewrite the RPATH to not point to
per-package directories. Indeed the patchelf --make-rpath-relative
call only works if the RPATH points to the ROOTDIR passed as
argument, and this ROOTDIR is the global host directory. Rewriting
the RPATH to not point to per-package host directories prior to
this is an easy solution to this issue.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-11-06 00:46:40 +08:00
|
|
|
endef
|
2022-12-27 23:43:00 +08:00
|
|
|
|
|
|
|
|
# Ensure files like .la, .pc, .pri, .cmake, and so on, point to the
|
|
|
|
|
# proper staging and host directories for the current package: find
|
|
|
|
|
# all text files that contain the PPD root, and replace it with the
|
|
|
|
|
# current package's PPD.
|
|
|
|
|
# $1: destination root directory containing host and staging
|
|
|
|
|
define ppd-fixup-paths
|
|
|
|
|
$(Q)grep --binary-files=without-match -lrZ '$(PER_PACKAGE_DIR)/[^/]\+/' $(HOST_DIR) \
|
|
|
|
|
|while read -d '' f; do \
|
|
|
|
|
file -b --mime-type "$${f}" | grep -q '^text/' || continue; \
|
|
|
|
|
printf '%s\0' "$${f}"; \
|
|
|
|
|
done \
|
|
|
|
|
|xargs -0 --no-run-if-empty \
|
|
|
|
|
$(SED) 's:$(PER_PACKAGE_DIR)/[^/]\+/:$(1)/:g'
|
|
|
|
|
endef
|
core: implement per-package SDK and target
This commit implements the core of the move to per-package SDK and
target directories. The main idea is that instead of having a global
output/host and output/target in which all packages install files, we
switch to per-package host and target directories, that only contain
their explicit dependencies.
There are two main benefits:
- Packages will now see only the dependencies they explicitly list in
their <pkg>_DEPENDENCIES variable, and the recursive dependencies
thereof.
- We can support top-level parallel build properly, because a package
only "sees" its own host directory and target directory, isolated
from the build of other packages that can happen in parallel.
It works as follows:
- A new output/per-package/ directory is created, which will contain
one sub-directory per package, and inside it, a "host" directory
and a "target" directory:
output/per-package/busybox/target
output/per-package/busybox/host
output/per-package/host-fakeroot/target
output/per-package/host-fakeroot/host
This output/per-package/ directory is PER_PACKAGE_DIR.
- The global TARGET_DIR and HOST_DIR variable now automatically point
to the per-package directory when PKG is defined. So whenever a
package references $(HOST_DIR) or $(TARGET_DIR) in its build
process, it effectively references the per-package host/target
directories. Note that STAGING_DIR is a sub-dir of HOST_DIR, so it
is handled as well.
- Of course, packages have dependencies, so those dependencies must
be installed in the per-package host and target directories. To do
so, we simply rsync (using hard links to save space and time) the
host and target directories of the direct dependencies of the
package to the current package host and target directories.
We only need to take care of direct dependencies (and not
recursively all dependencies), because we accumulate into those
per-package host and target directories the files installed by the
dependencies. Note that this only works because we make the
assumption that one package does *not* overwrite files installed by
another package.
This is done for "extract dependencies" at the beginning of the
extract step, and for "normal dependencies" at the beginning of the
configure step.
This is basically enough to make per-package SDK and target work. The
only gotcha is that at the end of the build, output/target and
output/host are empty, which means that:
- The filesystem image creation code cannot work.
- We don't have a SDK to build code outside of Buildroot.
In order to fix this, this commit extends the target-finalize step so
that it starts by populating output/target and output/host by
rsync-ing into them the target and host directories of all packages
listed in the $(PACKAGES) variable. It is necessary to do this
sequentially in the target-finalize step and not in each
package. Doing it in package installation means that it can be done in
parallel. In that case, there is a chance that two rsyncs are creating
the same hardlink or directory at the same time, which makes one of
them fail.
This change to per-package directories has an impact on the RPATH
built into the host binaries, as those RPATH now point to various
per-package host directories, and no longer to the global host
directory. We do not try to rewrite such RPATHs during the build as
having such RPATHs is perfectly fine, but we still need to handle two
fallouts from this change:
- The check-host-rpath script, which verifies at the end of each
package installation that it has the appropriate RPATH, is modified
to understand that a RPATH to $(PER_PACKAGE_DIR)/<pkg>/host/lib is
a correct RPAT.
- The fix-rpath script, which mungles the RPATH mainly for the SDK
preparation, is modified to rewrite the RPATH to not point to
per-package directories. Indeed the patchelf --make-rpath-relative
call only works if the RPATH points to the ROOTDIR passed as
argument, and this ROOTDIR is the global host directory. Rewriting
the RPATH to not point to per-package host directories prior to
this is an easy solution to this issue.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2019-11-06 00:46:40 +08:00
|
|
|
endif
|
|
|
|
|
|
2012-05-18 01:33:00 +08:00
|
|
|
#
|
|
|
|
|
# legal-info helper functions
|
|
|
|
|
#
|
.mk files: bulk aligment and whitespace cleanup of assignments
The Buildroot coding style defines one space around make assignments and
does not align the assignment symbols.
This patch does a bulk fix of offending packages. The package
infrastructures (or more in general assignments to calculated variable
names, like $(2)_FOO) are not touched.
Alignment of line continuation characters (\) is kept as-is.
The sed command used to do this replacement is:
find * -name "*.mk" | xargs sed -i \
-e 's#^\([A-Z0-9a-z_]\+\)\s*\([?:+]\?=\)\s*$#\1 \2#'
-e 's#^\([A-Z0-9a-z_]\+\)\s*\([?:+]\?=\)\s*\([^\\]\+\)$#\1 \2 \3#'
-e 's#^\([A-Z0-9a-z_]\+\)\s*\([?:+]\?=\)\s*\([^\\ \t]\+\s*\\\)\s*$#\1 \2 \3#'
-e 's#^\([A-Z0-9a-z_]\+\)\s*\([?:+]\?=\)\(\s*\\\)#\1 \2\3#'
Brief explanation of this command:
^\([A-Z0-9a-z_]\+\) a regular variable at the beginning of the line
\([?:+]\?=\) any assignment character =, :=, ?=, +=
\([^\\]\+\) any string not containing a line continuation
\([^\\ \t]\+\s*\\\) string, optional whitespace, followed by a
line continuation character
\(\s*\\\) optional whitespace, followed by a line
continuation character
Hence, the first subexpression handles empty assignments, the second
handles regular assignments, the third handles regular assignments with
line continuation, and the fourth empty assignments with line
continuation.
This expression was tested on following test text: (initial tab not
included)
FOO = spaces before
FOO = spaces before and after
FOO = tab before
FOO = tab and spaces before
FOO = tab after
FOO = tab and spaces after
FOO = spaces and tab after
FOO = \
FOO = bar \
FOO = bar space \
FOO = \
GENIMAGE_DEPENDENCIES = host-pkgconf libconfuse
FOO += spaces before
FOO ?= spaces before and after
FOO :=
FOO =
FOO =
FOO =
FOO =
$(MAKE1) CROSS_COMPILE=$(TARGET_CROSS) -C
AT91BOOTSTRAP3_DEFCONFIG = \
AXEL_DISABLE_I18N=--i18n=0
After this bulk change, following manual fixups were done:
- fix line continuation alignment in cegui06 and spice (the sed
expression leaves the number of whitespace between the value and line
continuation character intact, but the whitespace before that could have
changed, causing misalignment.
- qt5base was reverted, as this package uses extensive alignment which
actually makes the code more readable.
Finally, the end result was manually reviewed.
Signed-off-by: Thomas De Schampheleire <thomas.de.schampheleire@gmail.com>
Cc: Yann E. Morin <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2014-10-07 15:06:03 +08:00
|
|
|
LEGAL_INFO_SEPARATOR = "::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::"
|
2013-11-12 16:48:00 +08:00
|
|
|
|
|
|
|
|
define legal-warning # text
|
|
|
|
|
echo "WARNING: $(1)" >>$(LEGAL_WARNINGS)
|
|
|
|
|
endef
|
|
|
|
|
|
|
|
|
|
define legal-warning-pkg # pkg, text
|
|
|
|
|
echo "WARNING: $(1): $(2)" >>$(LEGAL_WARNINGS)
|
|
|
|
|
endef
|
|
|
|
|
|
2014-06-22 20:41:15 +08:00
|
|
|
define legal-warning-nosource # pkg, {local|override}
|
|
|
|
|
$(call legal-warning-pkg,$(1),sources not saved ($(2) packages not handled))
|
2012-05-18 01:33:00 +08:00
|
|
|
endef
|
2013-11-12 16:48:00 +08:00
|
|
|
|
2018-10-21 19:59:24 +08:00
|
|
|
define legal-manifest # {HOST|TARGET}, pkg, version, license, license-files, source, url, dependencies
|
|
|
|
|
echo '"$(2)","$(3)","$(4)","$(5)","$(6)","$(7)","$(8)"' >>$(LEGAL_MANIFEST_CSV_$(1))
|
2013-11-12 16:48:00 +08:00
|
|
|
endef
|
|
|
|
|
|
package/pkg-download: lookup hash files in global-patch-dir
Currently, we expect and only use hash files that lie within the package
directory, alongside the .mk file. Those hash files are thus bundled
with Buildroot.
This implies that only what's known to Buildroot can ever get into those
hash files. For packages where the version is fixed (or a static
choice), then we can carry hashes for those known versions.
However, we do have a few packages for which the version is a free-form
entry, where the user can provide a custom location and/or version. like
a custom VCS tree and revision, or a custom tarball URL. This means that
Buildroot has no way to be able to cary hashes for such custom versions.
This means that there is no integrity check that what was downloaded is
what was expected. For a sha1 in a git tree, this is a minor issue,
because the sha1 by itself is already a hash of the expected content.
But for custom tarballs URLs, or for a tag in a VCS, there is indeed no
integrity check.
Buildroot can't provide such hashes, but interested users may want to
provide those, and currently there is no (easy) way to do so.
We leverage the existing global-patch-dir mechanism to look for extra
hash files. We use the same heuristic that is used for bundled hash
files, and for each global patch directory <dir>, we use the first file
to exist among:
1. look into <dir>/<package>/<version>/<package>.hash
2. look into <dir>/<package>/<package>.hash
Reported-by: "Martin Zeiser (mzeiser)" <mzeiser@cisco.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2023-11-07 03:09:13 +08:00
|
|
|
define legal-license-file # {HOST|TARGET}, pkgname, pkgname-pkgver, filename, file-fullpath, pkg-hashfiles
|
support/download: teach dl-wrapper to handle more than one hash file
Currently, we expect and only use hash files that lie within the package
directory, alongside the .mk file. Those hash files are thus bundled
with Buildroot.
This implies that only what's known to Buildroot can ever get into those
hash files. For packages where the version is fixed (or a static
choice), then we can carry hashes for those known versions.
However, we do have a few packages for which the version is a free-form
entry, where the user can provide a custom location and/or version. like
a custom VCS tree and revision, or a custom tarball URL. This means that
Buildroot has no way to be able to cary hashes for such custom versions.
This means that there is no integrity check that what was downloaded is
what was expected. For a sha1 in a git tree, this is a minor issue,
because the sha1 by itself is already a hash of the expected content.
But for custom tarballs URLs, or for a tag in a VCS, there is indeed no
integrity check.
Buildroot can't provide such hashes, but interested users may want to
provide those, and currently there is no (easy) way to do so.
So, we need our download helpers to be able to accept more than one hash
file to lookup for hashes.
Extend the dl-wrapper and the check-hash helpers thusly, and update the
legal-info accordingly.
Note that, to be able to pass more than one hash file, we also need to
re-order the arguments passed to support/download/check-hash, which also
impies some shuffling in the three places it is called:
- 2 in dl-wrapper
- 1 in the legal-info infra
That in turn also requires that the legal-license-file macro args get
re-ordered to have the hash file last; we take the opportunity to also
move the HOST/TARGET arg to be first, like in the other legal-info
macros.
Reported-by: "Martin Zeiser (mzeiser)" <mzeiser@cisco.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2023-11-07 03:09:12 +08:00
|
|
|
mkdir -p $(LICENSE_FILES_DIR_$(1))/$(3)/$(dir $(4)) && \
|
2017-06-26 06:03:40 +08:00
|
|
|
{ \
|
support/download: teach dl-wrapper to handle more than one hash file
Currently, we expect and only use hash files that lie within the package
directory, alongside the .mk file. Those hash files are thus bundled
with Buildroot.
This implies that only what's known to Buildroot can ever get into those
hash files. For packages where the version is fixed (or a static
choice), then we can carry hashes for those known versions.
However, we do have a few packages for which the version is a free-form
entry, where the user can provide a custom location and/or version. like
a custom VCS tree and revision, or a custom tarball URL. This means that
Buildroot has no way to be able to cary hashes for such custom versions.
This means that there is no integrity check that what was downloaded is
what was expected. For a sha1 in a git tree, this is a minor issue,
because the sha1 by itself is already a hash of the expected content.
But for custom tarballs URLs, or for a tag in a VCS, there is indeed no
integrity check.
Buildroot can't provide such hashes, but interested users may want to
provide those, and currently there is no (easy) way to do so.
So, we need our download helpers to be able to accept more than one hash
file to lookup for hashes.
Extend the dl-wrapper and the check-hash helpers thusly, and update the
legal-info accordingly.
Note that, to be able to pass more than one hash file, we also need to
re-order the arguments passed to support/download/check-hash, which also
impies some shuffling in the three places it is called:
- 2 in dl-wrapper
- 1 in the legal-info infra
That in turn also requires that the legal-license-file macro args get
re-ordered to have the hash file last; we take the opportunity to also
move the HOST/TARGET arg to be first, like in the other legal-info
macros.
Reported-by: "Martin Zeiser (mzeiser)" <mzeiser@cisco.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2023-11-07 03:09:12 +08:00
|
|
|
support/download/check-hash $(5) $(4) $(6); \
|
2017-07-17 01:22:39 +08:00
|
|
|
case $${?} in (0|3) ;; (*) exit 1;; esac; \
|
2017-06-26 06:03:40 +08:00
|
|
|
} && \
|
support/download: teach dl-wrapper to handle more than one hash file
Currently, we expect and only use hash files that lie within the package
directory, alongside the .mk file. Those hash files are thus bundled
with Buildroot.
This implies that only what's known to Buildroot can ever get into those
hash files. For packages where the version is fixed (or a static
choice), then we can carry hashes for those known versions.
However, we do have a few packages for which the version is a free-form
entry, where the user can provide a custom location and/or version. like
a custom VCS tree and revision, or a custom tarball URL. This means that
Buildroot has no way to be able to cary hashes for such custom versions.
This means that there is no integrity check that what was downloaded is
what was expected. For a sha1 in a git tree, this is a minor issue,
because the sha1 by itself is already a hash of the expected content.
But for custom tarballs URLs, or for a tag in a VCS, there is indeed no
integrity check.
Buildroot can't provide such hashes, but interested users may want to
provide those, and currently there is no (easy) way to do so.
So, we need our download helpers to be able to accept more than one hash
file to lookup for hashes.
Extend the dl-wrapper and the check-hash helpers thusly, and update the
legal-info accordingly.
Note that, to be able to pass more than one hash file, we also need to
re-order the arguments passed to support/download/check-hash, which also
impies some shuffling in the three places it is called:
- 2 in dl-wrapper
- 1 in the legal-info infra
That in turn also requires that the legal-license-file macro args get
re-ordered to have the hash file last; we take the opportunity to also
move the HOST/TARGET arg to be first, like in the other legal-info
macros.
Reported-by: "Martin Zeiser (mzeiser)" <mzeiser@cisco.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Peter Korsgaard <peter@korsgaard.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2023-11-07 03:09:12 +08:00
|
|
|
cp $(5) $(LICENSE_FILES_DIR_$(1))/$(3)/$(4)
|
2012-05-18 01:33:00 +08:00
|
|
|
endef
|
2018-10-21 19:59:24 +08:00
|
|
|
|
|
|
|
|
non-virtual-deps = $(foreach p,$(1),$(if $($(call UPPERCASE,$(p))_IS_VIRTUAL),,$(p)))
|
|
|
|
|
|
|
|
|
|
# Returns the list of recursive dependencies and their licensing terms
|
|
|
|
|
# for the package specified in parameter (in lowercase). If that
|
|
|
|
|
# package is a target package, remove host packages from the list.
|
|
|
|
|
legal-deps = \
|
|
|
|
|
$(foreach p,\
|
|
|
|
|
$(filter-out $(if $(1:host-%=),host-%),\
|
|
|
|
|
$(call non-virtual-deps,\
|
|
|
|
|
$($(call UPPERCASE,$(1))_FINAL_RECURSIVE_DEPENDENCIES))),$(p) [$($(call UPPERCASE,$(p))_LICENSE)])
|
2022-08-16 00:56:30 +08:00
|
|
|
|
2022-08-16 00:56:31 +08:00
|
|
|
# Helper for self-extracting binaries distributed by NXP, and
|
|
|
|
|
# formerlly Freescale.
|
2022-08-16 00:56:30 +08:00
|
|
|
#
|
|
|
|
|
# The --force option makes sure it doesn't fail if the source
|
|
|
|
|
# directory already exists. The --auto-accept skips the license check,
|
|
|
|
|
# as it is not needed in Buildroot because we have legal-info. Since
|
|
|
|
|
# there's a EULA in the binary file, we extract it in this macro, and
|
|
|
|
|
# it should therefore be added to the LICENSE_FILES variable of
|
|
|
|
|
# packages using this macro. Also, remember to set REDISTRIBUTE to
|
|
|
|
|
# "NO". Indeed, this is a legal minefield: the EULA specifies that the
|
|
|
|
|
# Board Support Package includes software and hardware (sic!) for
|
|
|
|
|
# which a separate license is needed...
|
|
|
|
|
#
|
|
|
|
|
# $(1): full path to the archive file
|
|
|
|
|
#
|
2022-08-16 00:56:31 +08:00
|
|
|
define NXP_EXTRACT_HELPER
|
2022-08-16 00:56:30 +08:00
|
|
|
awk 'BEGIN { start = 0; } \
|
|
|
|
|
/^EOEULA/ { start = 0; } \
|
|
|
|
|
{ if (start) print; } \
|
|
|
|
|
/<<EOEULA/ { start = 1; }' \
|
|
|
|
|
$(1) > $(@D)/EULA
|
|
|
|
|
cd $(@D) && sh $(1) --force --auto-accept
|
|
|
|
|
find $(@D)/$(basename $(notdir $(1))) -mindepth 1 -maxdepth 1 -exec mv {} $(@D) \;
|
|
|
|
|
rmdir $(@D)/$(basename $(notdir $(1)))
|
|
|
|
|
endef
|
