Add comments to build system makefile functions and variables to help
developers in understanding build system internals and ease the
development process.
This patch adds some documentation examples with proposed doxygen-like
syntax. Hopefully, this would start the discussion and result in
generation of the makefile documentation guidelines.
Signed-off-by: Sergey Matsievskiy <matsievskiysv@gmail.com>
Link: https://github.com/openwrt/openwrt/pull/16888
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Permit to overwrite PACKAGE_DIR and PACKAGE_DIR_ALL variables in
rules.mk.
This is to handle a special case with the ImageBuilder where these
variable are overwrite.
The main problem is that any include calling rules.mk again (example
image.mk) will set these variables again dropping the modified value.
To keep the modified value, set the PACKAGE_DIR and PACKAGE_DIR_ALL only
if not already set. This permits the ImageBuilder to use custom
directory instead of the default one defined in rules.mk.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Usage of g++ should mimic the usage of gcc,
otherwise, part of a binary or library may have
optimizations and part of it may not,
unbeknownst to the users or developers working on a build.
This can lead to some features like, for example, FORTIFY_SOURCE,
to have less of an effect or even cause a build error on some hosts.
Therefore, let HOST_CXXFLAGS default to HOST_CFLAGS.
Fixes: 87d489f67 ("build: add HOST_CXXFLAGS for host build")
Signed-off-by: Michael Pratt <mcpratt@pm.me>
Link: https://github.com/openwrt/openwrt/pull/15853
Signed-off-by: Robert Marko <robimarko@gmail.com>
Several GNU tools such as tar, coreutils, and findutils
now build with support for 64-bit time by default
and otherwise require reconfiguring with a flag
--disable-year2038 in order to build without 64-bit time.
Some standard C libraries, for example,
certain older versions of glibc such as 2.31
have large file support but not long time bits support:
checking for ... option to enable large file support... -D_FILE_OFFSET_BITS=64
checking for ... option for timestamps after 2038... support not detected
This test using C code taken from largefile.m4 in gnulib
uses math and casting to check for overflow
with a macro and array pair that can only be defined
when 64-bit time support is present, and otherwise errors.
It is the exact same code used to test for 64-bit time
during the configure stage of building these tools,
so the results of this test before configure takes place
will always be in concordance with the results of
the test that takes place during the configure script.
Based on the test, the configure flag --disable-year2038
is added to every host tool build depending on the host system.
When the year 2038 problem finally comes around,
the effect of the test can be converted
from the toggling of a configure option into a build prerequisite,
requiring it to pass in order to continue building.
Signed-off-by: Michael Pratt <mcpratt@pm.me>
Link: https://github.com/openwrt/openwrt/pull/15799
Signed-off-by: Robert Marko <robimarko@gmail.com>
A new option called `USE_APK` is added which generated APK packages
(.apk) instead of OPKG packages (.ipk).
Some features like fstools `snapshot` command are not yet ported
Signed-off-by: Paul Spooren <mail@aparcar.org>
Currently when external toolchain is used, a info.mk is created (with
to-be-filled values) but is never actually filled with real values and
is never actually used in rules.mk.
This info.mk is used down the code with special packages like libgcc to
reference info like gcc version and package special library.
To mimic what is done with internal toolchain, add the include in
rules.mk for info.mk also for external library and in fix
toolchain/wrapper to fill the staging_dir info.mk.
The logic is to check if the external toolchain provide an info.mk and
if it doesn't we at least fill the GCC_VERSION with the value set in the
.config file.
With this special library like libgcc correctly reference and have set the
GCC_VERSION variable.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Make toolchain dirs define more consistent between internal and external
toolchains.
Make use of specific dirs also for intenral toolchain and generilize
include and lib inclusion.
Also set TOOLCHAIN_ROOT_DIR for internal toolchain as this is what
packages should use to reference staging toolchain directory.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
The current minimum OS requirement for OpenWrt is Ubuntu 18.04, which
includes 7 and 8. 8 is necessary for ccache.
gcc and g+++ are now symlinked to staging_dir, similar to Python.
Signed-off-by: Rosen Penev <rosenp@gmail.com>
sstrip only has one functional arg. Make that a bool option, which can
easily depend on other knobs then.
This is required to be disabled for the mold linker.
Signed-off-by: Andre Heider <a.heider@gmail.com>
Add a variable that stores the original value of $PATH
in the host system's shell, before Make alters it.
This can be useful for when it is necessary
to ignore symlinks and programs made by the build system.
Define this new variable before all instances of
'export PATH:=' or similar.
Signed-off-by: Michael Pratt <mcpratt@pm.me>
By having a local copy of gnulib, we can:
import the latest macro fixes into any package,
get rid of some statically stored macros that were otherwise missing,
bootstrap GNU tools with the latest relevant source
without having to wait for a release or rely on git submodules,
and possibly more...
The patch assists in bootstrapping by ignoring
the building of po files using gettext,
and also to allow a user-defined path to a program
to include parameters.
Signed-off-by: Michael Pratt <mcpratt@pm.me>
Some uses cases, like with autotools, need a path for 'true'
if we use them to force something to skip.
This will work by default on Linux hosts,
and require MacOS hosts to get coreutils,
which is currently installed in the CI
for 'macos-latest' host.
In the future, prereq stage can be reworked
to search for the actual binary
instead of relying on env.
Signed-off-by: Michael Pratt <mcpratt@pm.me>
Some packages using Libtool have the possibility
of using this environment variable if set,
instead of the direct command for "file" using PATH,
or the need to patch a hard-coded path.
This is a new feature of Libtool 2.4.7
Ref: bf261073d ("tools/libtool: bump to 2.4.7")
Signed-off-by: Michael Pratt <mcpratt@pm.me>
These wrappers are not needed as CC doesn't need to be a single word.
a53b084e49 which introduced the wrappers
doesn't explain why they were really needed and why only for the target
and not for the host.
Moreover, name of the wrappers breaks a ccache assumption: since
v4.0-3-g6a92b4cd3a67 it has special handling for "chained" invocation
such as "ccache ccache gcc" where it skips all the "ccache*" names in
the middle and proceeds to run as if it was started as "ccache
gcc"[1][2].
This becomes important when a build system sees ccache in the PATH and
automatically enables it by prepending to CC. An example of such a
system would be autosetup as used by jimtcl. With the wrappers it breaks
as the command line ends up being just "ccache -Os..." because
"ccache_cc" gets skipped as it starts with "ccache".
[1] https://github.com/ccache/ccache/blob/master/src/ccache.cpp#L2105
[2] https://github.com/ccache/ccache/blob/master/src/Util.cpp#L802
Reported-by: Karl Palsson <karlp@etactica.com>
Signed-off-by: Paul Fertser <fercerpav@gmail.com>
STAGING_DIR may be provided from command line. We currently hardcoded
STAGING_DIR_HOST and STAGING_DIR_HOSTPKG to the default location but we
currently have some relocatable patch that derive the path from
STAGING_DIR.
Fix this and correctly derive STAGING_DIR_HOST and STAGING_DIR_HOSTPKG
from STAGING_DIR.
The intention is to fix inconsistency from the relocatable patch and the
use of STAGING_DIR_HOST that is always hardcoded.
This with a wrong configuration may end up in broken state with some
host tools expecing a PATH from STAGING_DIR and others using library
from the default staging_dir/host path.
To save downstream project the original implementation is saved while
fixing the inconsistency between patch and .mk.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
We currently enable -Wno-error=unused-but-set-variable and
-Wno-error=unused-result by default on every compile package.
While this is (relatively) unharmful, we should follow other project
direction and starts enforcing good code quality. For example the linux
kernel recently started to enforce Wall by default and clean code is
mandatory for inclusion.
Drop for good these flags and and make it mandatory to correctly handle
return values at least with a warning log if they are not strictly error
condition.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
To generate commitcount we use grep --max-count. This is not present on
alpine grep and cause wrong generation. Use -m as it's just the short
version of --max-count and more portable.
Fixes: #11200
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Move DOWNLOAD_CHECK_CERTIFICATE to include/download.mk as it's a better
place than exporting it in the global rules.mk makefile.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Some packages download a single raw file or set of raw files
instead of an archive.
The filenames of these download files have names and version numbers
that do not directly correspond to the name and version of the package
as it is defined in its Makefile.
To better organize downloaded files, this adds the option of
creating a subdirectory in the download directory for these files
that can be named with package specific variables.
DL_DIR is now a recursively expanded variable
since the value now depends on whether DL_SUBDIR is defined.
Signed-off-by: Michael Pratt <mcpratt@pm.me>
Remove flags from wget and curl instructing them to ignore bad server
certificates. Although other mechanisms can protect against malicious
modifications of downloads, other vectors of attack may be available
to an adversary.
TLS certificate verification can be disabled by turning oof the
"Enable TLS certificate verification during package download" option
enabled by default in the "Global build settings" in "make menuconfig"
Signed-off-by: Josh Roys <roysjosh@gmail.com>
[ add additional info on how to disable this option ]
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Don't add wrapped bin to the TARGET_PATH as it does cause compilation
error.
cmake.mk will use the "command -v" and will use the wrapped bin instead
of the external toolchain bin as they have the same name and command
will select the first result.
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
When using the OpenWrt toolchain as an external toolchain the build
failed due to missing LTO support. By choosing the GCC wrappers of
the tools this commit makes sure that the LTO-enabled executables
are being used.
Signed-off-by: Vincent Wiemann <vincent.wiemann@ironai.com>
[ wrap the commit description to 72 char ]
Signed-off-by: Christian Marangi <ansuelsmth@gmail.com>
Compiler option -no-plt will break kernel builds on some architectures
eg. (x86) Filter this option from the recently introduced handling of
KCFLAGS vs EXTRA_OPTIMISATION
Fixes: 1d42af720c ("kernel: use KCFLAGS for passing EXTRA_OPTIMIZATION flags")
Suggested-by: Felix Fietkau <nbd@nbd.name>
Signed-off-by: Kevin Darbyshire-Bryant <ldir@darbyshire-bryant.me.uk>
On recent macOS, /usr/bin/python3 is a wrapper that finds the right python executable
It checks argv[0] to determine if python2 or python3 should be called. Always execute
it as python3 to ensure it calls the right version
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Added HOST_CXXFLAGS to specify CXXFLAGS during host-compile
(e.g. to specify c++ standard: HOST_CXXFLAGS += -std=c++11)
Signed-off-by: Sergey V. Lobanov <sergey@lobanov.in>
The development branch is now on version 10, we shouldn't drag to many
old versions and therefore drop at least 7.x.
Signed-off-by: Paul Spooren <mail@aparcar.org>
Acked-by: Rosen Penev <rosenp@gmail.com>
Acked-by: Rui Salvaterra <rsalvaterra@gmail.com>
Acked-by: Hauke Mehrtens <hauke@hauke-m.de>
Acked-by: Aleksander Jan Bajkowski <A.Bajkowski@stud.elka.pw.edu.pl>
ninja is faster at building cmake packages than make, and according to reports
also more reliable at handling parallel builds
This commit includes a patch that adds GNU make jobserver support, in order to
allow more precise control over the number of parallel tasks
Enable parallel build by default for packages using ninja
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Signed-off-by: Rosen Penev <rosenp@gmail.com>
Before this commit, it was assumed that mkhash is in the PATH. While
this was fine for the normal build workflow, this led to some issues if
make TOPDIR="$(pwd)" -C "$pkgdir" compile
was called manually. In most of the cases, I just saw warnings like this:
make: Entering directory '/home/.../package/gluon-status-page'
bash: line 1: mkhash: command not found
bash: line 1: mkhash: command not found
bash: line 1: mkhash: command not found
bash: line 1: mkhash: command not found
bash: line 1: mkhash: command not found
bash: line 1: mkhash: command not found
bash: line 1: mkhash: command not found
bash: line 1: mkhash: command not found
[...]
While these were only warnings and the package still compiled sucessfully,
I also observed that some package even fail to build because of this.
After applying this commit, the variable $(MKHASH) is introduced. This
variable points to $(STAGING_DIR_HOST)/bin/mkhash, which is always the
correct path.
Signed-off-by: Leonardo Mörlein <me@irrelefant.net>
Fixes issue openwrt/packages#14921, whereby inline ASM wasn't getting
built as PIC; look at gmp-6.2.1/mpn/x86/pentium/popcount.asm for
example:
ifdef(`PIC',`
...
for a routine that exists in both PIC and non-PIC versions.
Make sure that wherever $(FPIC) gets passed as a variable expansion
that it gets quoted where necessary (such as setting environment
variables in shell commands).
Signed-off-by: Philip Prindeville <philipp@redfish-solutions.com>
This reverts commit b12288fa69.
The patchelf approach is too fragile, and the only users of this have been
converted to make patching unnecessary
Leave the abi_version_str variable in place in rules.mk
Signed-off-by: Felix Fietkau <nbd@nbd.name>
This makes it possible to declare a package ABI_VERSION independent from the
upstream soname by setting PKG_ABI_VERSION in the package makefile.
The library filename is fixed up for files installed to packages and to the
staging dir. References to the original from executables within the same
package are also fixed up
Signed-off-by: Felix Fietkau <nbd@nbd.name>
The license folder is a core part of OpenWrt and all GPL-2.0 licensed.
Use SPDX license tags to allow machines to check licenses.
Signed-off-by: Paul Spooren <mail@aparcar.org>
[rebase, keep some Copyright lines, sharpen commit message]
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
Packages that are in-tree only often lack a PKG_VERSION and only use the
PKG_RELEASE to mark changes. Using COMMITCOUNT/AUTORELEASE variables
causes an issue as both variables are empty during the metadata DUMP
phase.
Instead of leaving these variables empty and causing an error message
like below, set the variables to 0 during dumping. On actual building
the variable is evaluated causing in a value above 0.
ERROR: please fix package/utils/px5g-wolfssl/Makefile - \
see logs/package/utils/px5g-wolfssl/dump.txt for details
Makefile:48: *** Package/px5g-wolfssl is missing the VERSION field. Stop.
Reported-by: Daniel Golle <daniel@makrotopia.org>
Reported-by: Stijn Segers <foss@volatilesystems.org>
Reported-by: Stijn Tintel <stijn@linux-ipv6.be>
Signed-off-by: Paul Spooren <mail@aparcar.org>
The newly added $(COMMITCOUNT) variable was wrongly increased by plus
one. The addition should have been only added to $(AUTORELEASE) as
OpenWrt traditionally starts counting at one rather than zero.
$(AUTORELEASE) counts the commits since the last bump, which is zero on
the version bump commit itself.
This commit increases $(AUTORELEASE) by one while leaving $(COMMITCOUNT)
as is.
The base-files package is the only package using $(COMMITCOUNT) so far
and requires a pseudo commit to keep the PKG_RELEASE correct. A non
functional change (Copyright bump) is done in the next commit.
Signed-off-by: Paul Spooren <mail@aparcar.org>
The lack of bumped PKG_RELEASE variables is a recurring theme on the
mailing list and in GitHub comments. This costs precious review time,
a rare good within the OpenWrt project.
Instead of relying on a manually set PKG_RELEASE this commit adds a
`commitcount` function that uses the number of Git commits to determine
the release. The function is called via the variables `$(AUTORELEASE)`
or `$(COMMITCOUNT)`. The `PKG_RELEASE` variable can be set to either of
the two.
- $(AUTORELEASE):
Release is automagically set to the number of commits since the last
commit containing either ": update to " or ": bump to ".
Example below:
$ git log packages/foobar/
foobar: fixup file location
foobar: disable docs
foobar: bump to 5.3.2
foobar: fixup copyright
Resulting package name: foobar_5.3.2-3_all.ipk, two package changes
since the last upstream version change, using a 1 based counter.
- $(COMMITCOUNT):
For non-traditional versioning (x.y.z), most prominent `base-files`,
this variable contains the total number of package commits.
The new functionality can also be used by other feeds like packages.git.
In case no build information is available, e.g. when using release
tarballs, the SOURCE_DATE_EPOCH is used to have a reproducible release
identifier.
Suggested-by: Daniel Golle <daniel@makrotopia.org>
Signed-off-by: Paul Spooren <mail@aparcar.org>
Some packages fail to build on arm64 when PKG_ASLR_PIE_ALL=y, due to
machine-specific size restrictions on the global offset table.
While the manual instructs to recompile with -fPIC if it fails with
-fpic, by doing this per package, there is still a risk of random
breakage due to version bumps or other changes, so let's use -fPIC on
arm64 by default.
While comparing the sizes of 141 packages built with -fpic vs -fPIC,
most packages are either equal or smaller in size. Only 9 of the
compared packages turned out slightly larger.
Signed-off-by: Stijn Tintel <stijn@linux-ipv6.be>
Commit f98878e4c1 ("cmake.mk: set C/CXX compiler for host builds as
well") has introduced regression as it didn't taken usage of ccache into
the account so fix it by handling ccache use cases as well.
In order to get this working we need to export HOSTCXX_NOCACHE in
rules.mk as well.
Fixes: f98878e4c1 ("cmake.mk: set C/CXX compiler for host builds as well")
Reported-by: Ansuel Smith <ansuelsmth@gmail.com>
Tested-by: Ansuel Smith <ansuelsmth@gmail.com>
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Drop our local sstrip copy and use the current ELFKickers upstream
version.
Patch the original makefile in order to avoid building elftoc, since it
fails with musl's elf.h. This is fine, since we only need sstrip anyway.
Finally, add the possibility to pass additional arguments to sstrip and
pass -z (remove trailing zeros) by default, which matches the behaviour
of the previous version.
Signed-off-by: Rui Salvaterra <rsalvaterra@gmail.com>
[shorten long commit msg lines]
Signed-off-by: Paul Spooren <mail@aparcar.org>
Since fakeroot is patched to discover related ressources relative to the
STAGING_DIR_HOST environment variable, there is no need to pass the path
to faked or the preload library manually anymore.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
SELinux support requires setting the appropriate SELinux security context
to files and directories, which needs to happen at build time in order
to support read-only root filesystem scenarios. In order to create these
security contexts, we will have to run some SELinux-specific tools on
the host machine, but that requires root access. This adds support for
fakeroot, which the build process will use to run the SELinux security
context creation and the image creation.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Apply to current master, and adjust commit message
Thomas' original work is available at
http://lists.infradead.org/pipermail/openwrt-devel/2019-November/025976.html.
Signed-off-by: W. Michael Petullo <mike@flyn.org>
[add rules.mk FAKEROOT variable]
Signed-off-by: Paul Spooren <mail@aparcar.org>
[update, fix macos build]
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Remapping the local build path in debug information makes debugging
using ./scripts/remote-gdb harder, because files no longer refer to the full
path on the build host.
For local builds, debug information does not need to be reproducible,
since it will be stripped out of packages anyway.
For buildbot builds, it makes sense to keep debug information reproducible,
since the full path is not needed (nor desired) anywhere.
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Set CCACHE_DIR to $(TOPDIR)/.ccache and CCACHE_BASEDIR to $(TOPDIR).
This allows to do clean and dirclean. Cache hit rate for test build
after dirclean is ~65%.
If CCACHE is enabled stats are printed out at the end of building process.
CCACHE_DIR config variable allows to override default, which could be useful
when sharing cache with many builds.
cacheclean make target allows to clean the cache.
Changes from v1:
- remove ccache directory using CCACHE_DIR variable
- remove ccache leftovers from sdk and toolchain make files
- introduce CONFIG_CCACHE_DIR variable
- introduce cacheclean make target
Signed-off-by: Roman Yeryomin <roman@advem.lv>
"$(STAGING_DIR)/include" was carried over from buildroot-ng to OpenWrt
in commit 60c1f0f64d. buildroot has
dropped this directory a long time ago.
In OpenWrt the directory is still created by the PrepareStaging macro
and is part of the default TARGET_CPPFLAGS. But nothing at all installs
headers into this directory, nor should anything be installed under this
path.
Removing this directory from TARGET_CPPFLAGS will cut down the log noise
a bit. Not only will CPPFLAGS be shorter, there will be less warnings
set off by "-Wmissing-include-dirs" (or even failures when paired with
"-Werror"). After all the directory does not even _exist_ in the SDKs,
which are used on the build bots when building packages (see [1] and
[2]).
make[8]: Entering directory '/builder/shared-workdir/build/sdk/build_dir/target-aarch64_generic_musl/libmbim-1.20.0/src/common'
CC libmbim_common_la-mbim-common.lo
cc1: error: /builder/shared-workdir/build/sdk/staging_dir/target-aarch64_generic_musl/include: No such file or directory [-Werror=missing-include-dirs]
cc1: all warnings being treated as errors
[1] https://github.com/openwrt/packages/issues/10377
[2] https://github.com/openwrt/packages/pull/10378
Signed-off-by: Sebastian Kemper <sebastian_ml@gmx.net>
Acked-by: Jo-Philipp Wich <jo@mein.io>
Acked-by: Rosen Penev <rosenp@gmail.com>
-ffile-prefix-map=OLD=NEW is an alias for both -fdebug-prefix-map and
-fmacro-prefix-map and is available since GCC 8.
Co-Developed-by: Andre Heider <a.heider@gmail.com>
Signed-off-by: Andre Heider <a.heider@gmail.com>
Signed-off-by: Paul Spooren <mail@aparcar.org>
[refactored into separate commit]
Signed-off-by: Petr Štetiar <ynezz@true.cz>
This speeds up the packing of the imagebuilder a lot:
imagebuilder-T0.tar.xz real 0m25.199s user 2m45.967s sys 0m1.218s
imagebuilder-T1.tar.xz real 2m02.543s user 2m02.418s sys 0m1.653s
imagebuilder-T2.tar.xz real 1m03.684s user 1m59.931s sys 0m0.587s
imagebuilder-T3.tar.xz real 0m48.033s user 2m02.904s sys 0m0.637s
imagebuilder-T4.tar.xz real 0m38.963s user 2m15.521s sys 0m0.783s
imagebuilder-T5.tar.xz real 0m37.994s user 2m21.461s sys 0m0.919s
imagebuilder-T6.tar.xz real 0m39.524s user 2m48.115s sys 0m1.279s
imagebuilder-T7.tar.xz real 0m34.061s user 2m45.097s sys 0m1.174s
imagebuilder-T8.tar.xz real 0m27.286s user 2m55.449s sys 0m1.329s
imagebuilder-T9.tar.xz real 0m25.205s user 2m44.894s sys 0m1.208s
To keep the output reproducible in any case, we enforce a minimum amount
of 2 threads.
Signed-off-by: Martin Schiller <ms@dev.tdt.de>
[refactored into reusable NPROC var, more verbose commit message]
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Add option BUILD_LOG_DIR to menuconfig to change log destination.
The mix-up of *DIR* and *FOLDER* is confusing however.
Signed-off-by: Paul Spooren <mail@aparcar.org>
