Commit Graph

357 Commits

Author SHA1 Message Date
Yann E. MORIN
32b4912a75 utils/checkpackagelib: instruct shellcheck to follow included files
We do have a few shell scripts that have source directives (either with
'source' or with '.'), and they all are currently either globally
ignored in .checkpackageignore, or have shellcheck directives to ignore
the source statement (SC1090).

So, we can safely instruct shellcheck where to lookup for sourced files;
that does not trigger any new error.

This will alow fixing some shellcheck errors in later commits.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@datacom.com.br>
[Arnout: use long option]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2024-07-14 10:09:07 +02:00
Yann E. MORIN
fc2dedcfc6 utils/check-package: add option to run the test-suite
Currently, how to run the check-package test-suite is thoroughly under-
documented. There is one hint in the commit log for commit fc254881e6
(utils/checkpackagelib: add unit tests), and another in commit
242e9d72e7 (utils/docker-run: new script); the former is hard to find,
and the latter is about an unrelated script, so harder yet to find...

Add a new option to check-package, that will make it run its test-suite.
Since pytest is only needed for the test-suite, only import it in that
case.

pytest will be default create a cache of the tests (not sure what it is
about, though), so instruct it not to (esp. since the tree can be
read-only).

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@datacom.com.br>
[Arnout: move the code into checkpackagelib.base]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2024-07-14 10:09:06 +02:00
Yann E. MORIN
5836b79762 utils/checkpackagelib: extend hint about unprefixed variables
User may get confused when they see the current hint, and take that as
the proper replacement, while we're only reporting the stem of the
variable name:

    .../foo.mk:16: possible typo: BLA -> *FOO*

There is usually no easy way to actually suggest the proper variable
name, though, so let's make it a little bit more obvious that we meant
the variable was improperly prefixed:

    .../foo.mk:16: possible typo, variable not properly prefixed: BLA -> *FOO_XXX*

And while at it, throw in the URL to the corresponding manual entry.

Adapt the test accordingly.

Reported-by: "Frager, Neal" <neal.frager@amd.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@datacom.com.br>
Reviewed-by: Neal Frager <neal.frager@amd.com>
[Arnout: also update new test, scoped -> prefixed]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2024-07-14 08:53:40 +02:00
Yann E. MORIN
b0964df557 utils/checkpackagelib: really check variable name
Currently, when a package defines an unprefixed variable, but its value
contains a properly prefixed expansion (or even just the name of a
variable), there is not error reported (e.g. with the recently fixed
composer issue):

    BASE_SITE = https://getcomposer.org/download/$(COMPOSER_VERSION)/composer.phar

The reason is that he check is done on the whole line, rather than on
the variable that is being set.

We fix that by really looking at the variable we found, instead of
looking in the whole line.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@datacom.com.br>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2024-07-14 08:53:36 +02:00
Ricardo Martincoski
ccc12c0f24 utils/check-package: check linux-tools
Each linux tool uses a fragment of a .mk file, named, for instance:
package/linux-tools/linux-tool-cpupower.mk.in
So currently check-package does not check these files.

Add the support in check-package script.
At the same time, factor out a function to derive package prefix from
the filename being checked, so the fix (calling os.path.splitext twice)
can be applied in a single place.

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
[Arnout: add docstring and explain double splitext to
         get_package_prefix_from_filename]
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2024-07-12 20:30:40 +02:00
Ricardo Martincoski
cca315f814 utils/check-package: fix check for <pkg>_REDISTRIBUTE
Commit "8e87d76c12 check-package: check *.mk files" added a typo that
prevents check-package from checking <PKG>_REDISTRIBUTE default value.

Fix the typo: _INSTALL_REDISTRIBUTE -> _REDISTRIBUTE

Signed-off-by: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Arnout Vandecappelle <arnout@mind.be>
2024-07-12 20:30:36 +02:00
Yann E. MORIN
a316aeb7da utils/genrandconfig: do not check certificates with curl
genrandconfig is used in autobuilders, and some autobuilders are running
on old distributions that are lacking the most recent CAs, causing build
failures because package sources can't be retrieved.

Do for the curl backend what we already did a while back for the wget
backend, with commit 0866a280e4 (utils/genrandconfig: use
--no-check-certificate in wget by default); in curl, the equivalent
would be --insecure, and applies to the ftps transport.

The integrity of the downloads are validated against our bundled hashes
so there is no risk of corruption of the downloaded files. The only
issue would be that an MITM could inspect the transaction, the same way
as for the wget --no-check-certificate in 0866a280e4, but this is not
considered a high-level issue (we're anyway talking FTPS here, that's a
legacy protocol that has other issues).

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-07-12 14:36:10 +02:00
Yann E. MORIN
881a8f1346 utils/genrandconfig: stop passing --passive-ftp to wget
Since we no longer use WGET to retrieve FTP-hosted files, we can drop
the --passive-ftp option from genrandconfig, as it would cause
problems on systems that use wget2.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-07-12 14:34:57 +02:00
James Hilliard
0fa7343b6b utils/update-rust: use pathlib with relative paths
This avoids the requirement of running update-rust from TOPDIR and
is slightly cleaner.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-07-10 23:03:42 +02:00
James Hilliard
8436a5141b utils/update-rust: use urllib.request from stdlib instead of requests
We can use urllib.request without adding any addition lines of code
compared with requests which is not part of the stdlib.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-07-10 22:40:52 +02:00
Vincent Fazio
76cd14167f package/python3: bump python to 3.12.4
Along with the version bump, a number of the patches have been dropped
or reworked.

Compiled shared module builds are now disabled via py_cv_module_* = n/a
when possible (see PY_STDLIB_MOD_SET_NA). Leveraging this method allows
dropping numerous patches.

Note: this method does not work when access to a compiled module is
wrapped by a python-based module such as sqlite, tkinter, etc. In these
cases, configure knobs are still required to disable these modules and
their wrappers.

Both setup.py and distutils are no longer utilized upstream, so patches
have been dropped or adjusted accordingly.

Patch 0007 has been dropped, the Xtensa architecture can avoid compile
errors by using a toolchain with kernel headers newer than 3.17 [0].
That would require a new, non-trivial dependency to be added, but it
would also need to be propagated to all packages that select python3; as
we consider xtensa with old headers to be a corner case, we decide to
not address the issue, and rather add an exclusion in genrandconfig to
avoid the issue in autobuilders.

Patch 0026 has been dropped, the issue is fixed in uClibc v1.0.37+ [1].
Note: uClibc is not generally supported by upstream CPython.

While we're making assumptions about uClibc, ac_cv_func_wcsftime=no is
no longer set as it was fixed in v1.0.0 [2].

Co-developed-by: Adam Duskett <adam.duskett@amarulasolutions.com>
Co-developed-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[See 36e635d2d5]

Co-developed-by: Roy Kollen Svendsen <roy.kollen.svendsen@akersolutions.com>
[See 687b96db4d]

[0]: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=f61bf8e7d19e0a3456a7a9ed97c399e4353698dc
[1]: https://cgit.uclibc-ng.org/cgi/cgit/uclibc-ng.git/commit/libcrypt/crypt.c?id=2f6076cdaada2132b670b5ef25ad80c76a916c5a
[2]: https://lists.uclibc.org/pipermail/uclibc/2014-November/048723.html

Signed-off-by: Vincent Fazio <vfazio@gmail.com>
[yann.morin.1998@free.fr;
  - extend the commit log for the xtensa headers case
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-06-21 21:10:37 +02:00
Yann E. MORIN
2ab21855bb package/rust: update to 1.79.0
In rust 1.75.0, the follwong targets were dropped [0]:
    mips-unknown-linux-musl
    mips64-unknown-linux-muslabi64
    mips64el-unknown-linux-muslabi64
    mipsel-unknown-linux-musl

[0] 03870154f4

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-06-21 21:10:33 +02:00
Yann E. MORIN
df899041fb utils/update-rust: add script to automatically update the rust version
When we update the rust version, we need to update a large list of
hashes for rust-bin, and update the hash for rust (source). This is very
tedious and very error-prone.

Add a helper script that generates the hash files automatically, and
also iupdate the _VERSION in the .mk.

We decided not to carry the hint about the verification of the tarballs
against the upstream GPG signature for three reasons:

  - it requires that all the tarballs be downloaded, which can take
    quite some time;

  - the hash files are available for all the needed archives;

  - the hash files are downloaded over https, so if someone managed to
    get a hold of the rust server to provide backdoored archives, they
    can also change the hash files;

  - properly verifying the archives would require a chain of trust
    between the person running the upsate script, and the rust GPG key,
    which is not a given, and verifying sigantures using an unverified
    key is not providing much security, if at all.

Regenerate the hash files with that script.

Note (for the future, maybe): there are manifest files for each release,
https:/static.rust-lang.org/dist/channel-rust-VERSION.toml, that we
could use to generate the RUST_HOSTS and RUST_TARGETS list. However,
adding new hosts or new targets implies updating the corresponding
_ARCH_SUPPORTS and accompanying symbols, so better leave that to a
manual operation, at least for now.

Notes: Many thanks to James for providing an initial script with the
same purpose. Given the feedback from previous reviews, it was simpler
to rewrite it from scratch; it should now be much, much simpler.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: James Hilliard <james.hilliard1@gmail.com>
Tested-by: James Hilliard <james.hilliard1@gmail.com>
2024-06-21 21:10:31 +02:00
Peter Korsgaard
876d24578c Merge branch 'next' 2024-06-13 11:16:07 +02:00
Yann E. MORIN
8b8f5e3366 utils/genrandconfig: fix matching for a few lines
The matching in genrandconfig is idiomatically done by matching whole
lines, i.e. with the terminating \n but a few places are missing that.

Those are only matching against '=y', a boolean symbol, so it is in
practice not causing any issue. Still, for consistency, fix those.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-06-10 21:54:21 +02:00
James Hilliard
929a491f40 utils/genrandconfig: drop glibc Linaro toolchain version check
It has been well over 10 years since glibc 2.14 was released; the last
Debian version that had an earlier glibc was Wheezy, which Freexian
stopped to maintain as an ELTS in June 2020, 4 years ago, while the
oldest still maintained Ubuntu has glibc 2.21. It is now safe to assume
glibc 2.14 on all major, relevant distributions nowadays.

The distutils module is no longer bundled with python 3.12 so this
eliminates the need to install additional python modules under python
3.12.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
[yann.morin.1998@free.fr: add Debian and Ubuntu references]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-06-03 22:27:37 +02:00
Julien Olivain
f97e6293de utils/docker-run: set podman userns option in environment
Commit 9a629f5 "utils/docker-run: allow running with Podman" added an
option on system providing the podman command. This case is mainly
for Fedora systems.

Fedora repositories have a podman-docker package, that provides the
docker command for compatibility. See [1]. In that case, invoking
"docker" redirects to podman.

When this package is installed on a Fedora system, both the docker and
podman commands are available. Since the docker command is checked
before podman, the --userns option is not passed in that case. This
brings "permission denied" errors.

Other cases are also possible, like a host system providing the real
Docker alongside a podman installation. In such a case, to avoid
unexpected change of behavior of the docker-run script, the original
search order is preserved (search for "docker" first, then "podman").

This commit changes the way the podman user namespace mode is set.
Rather than adding the "--userns=keep-id" command line option only in
the podman case, it is globally set using the PODMAN_USERNS=keep-id
environment variable [2].

Doing so makes sure that the variable will be consumed by the "docker"
compatibility command, and just ignored by the real "docker"
implementation.

[1] https://packages.fedoraproject.org/pkgs/podman/podman-docker/
[2] https://docs.podman.io/en/latest/markdown/podman-run.1.html

Signed-off-by: Julien Olivain <ju.o@free.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-06-01 15:51:11 +02:00
James Hilliard
584ebdea6e utils/genrandconfig: switch to async/await format
This requires python 3.5 or newer but is a bit cleaner than the
previous coroutine method.

This should also fix a python3.12 issue:
[Tue, 28 May 2024 13:09:05] INFO: generate the configuration
Traceback (most recent call last):
  File "/home/autobuild/autobuild/instance-0/buildroot/utils/genrandconfig", line 833, in <module>
    ret = asyncio.run(gen_config(args))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/runners.py", line 89, in run
    raise ValueError("a coroutine was expected, got {!r}".format(coro))
ValueError: a coroutine was expected, got <generator object gen_config at 0xffff7bd822c0>
[Tue, 28 May 2024 13:09:06] WARN: failed to generate configuration

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Reviewed-by: Vincent Fazio <vfazio@xes-inc.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-05-31 21:20:03 +02:00
Fiona Klute
9a629f5129 utils/docker-run: allow running with Podman
Podman is command line compatible with Docker, there's no need to
require contributors to install Docker to run checks before sending
patches.

The additional "--userns=keep-id" option is necessary because unlike
Docker Podman creates a user namespace for containers by
default. Without keep-id the repository and pre-existing output files
belong to root inside the container namespace, breaking writes,
certain Git safety checks, and possibly all access (if the user is
using a strict umask).

Signed-off-by: Fiona Klute <fiona.klute+wiwa@gmx.de>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-05-31 14:42:18 +02:00
Brandon Maier
d506e232e7 utils/add-custom-hashes: symlink linux-headers to linux
Most boards use BR2_KERNEL_HEADERS_AS_KERNEL with their custom kernels.
So when creating their custom hash files, the linux-headers.hash is the
same as linux.hash. In this case we symlink linux-headers to linux to
make maintenance easier. Update the add-custom-hashes tool to explicitly
handle this case.

Signed-off-by: Brandon Maier <brandon.maier@collins.com>
[Peter: use cmp -s]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2024-05-29 08:56:47 +02:00
Romain Naour
c6784ea0e1 utils/genrandconfig: remove Sourcery CodeBench Nios-II exclusion
BR2_TOOLCHAIN_EXTERNAL_CODESOURCERY_NIOSII has been removed by [1].

[1] 7fa0159a50

Cc: Giulio Benetti <giulio.benetti@benettiengineering.com>
Signed-off-by: Romain Naour <romain.naour@smile.fr>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-05-01 23:35:15 +02:00
Yann E. MORIN
7931f78c16 utils/check-package: new check for Buildroot's defconfig files
Now that we do have support for checking hashes for custom versions
(for the few packages for which we do support custom versions, like the
kernel, some bootloaders...), we want to ensure that our defconfig
files, when they enable one or more such custom version, do enable
checking the hashes for those versions, and thus we want to require all
our defconfigs do enable BR2_DOWNLOAD_FORCE_CHECK_HASHES.

Add a check for that condition.

We need to be careful that we only check Buildroot's defconfig, whether
in-tree or in a br2-external, and not kernel or other kconfig-based
defconfig files, like those in board/ sub-directories. So we only match
defconfig files that are in a configs/ directory, whether at the
toplevel (for in-tree defconfigs), or not (for br2-external defconfigs).

Since we only have two defconfigs that check hashes for custom versions,
regnerate .checkpackageignore to ignore all so-far broken defconfigs.

Suggested-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@datacom.com.br>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-05-01 21:41:27 +02:00
Yann E. MORIN
ccb4e5db5c utils/check-package: emit library name along with check function name
Currently, when we generate .checkpackageignore, we store, for each
error, only the name of the function that generated that error.

Although we currently do not have two check libs that have same-name
check functions, there is nothing that would prevent that, and there
is no reason why two unrelated libs could not implement checks with
the same name.

If such a situation were to arise, we'd have no way, when parsing the
ignore list (in-tree: .checkpackageignore), to know which of the libs
the exclusion would apply to.

Fix that by storing both the library and function names together. The
leading "checkpackagelib." (with the trailing dot, 16 chars) is removed
for brevity, because it's present in all libs' names.

As a consequence, regenerate .checkpackageignore.

Note: people using that script to validate their br2-external trees will
also have to regenerate their own exclusion list if they have one.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Ricardo Martincoski <ricardo.martincoski@datacom.com.br>
Reviewed-by: Arnout Vandecappelle <arnout@mind.be>
2024-04-01 20:52:26 +02:00
Yann E. MORIN
81bb14a935 utils/check-package: don't ignore check-package
Commit 32934b526b (utils/checkpackagelib: check for Upstream trailers)
introduced a new python module to check Upstream tags in patch files. In
doing so, it introduced a flake8 coding style issue. That was not caught
when applying the change, and neither was it caught by our daily checks,
because the .checkpackagefile was regenerated right just in the next
commit, to apply ignore patterns to existing patch files.

It is a bit sad that one of our checks does not itself passes all our
checks...

Fix that trivial issue now.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-03-31 22:21:52 +02:00
Fabrice Fontaine
6ee61ef95b utils/genrandconfig: fix BR2_TOOLCHAIN_BARE_METAL_BUILDROOT_ARCH
Commit b7232c51dd added default setting
for bare-metal toolchain arch but unfortunately it used configlines.add
instead of configlines.append resulting in the following build failure:

/bin/sh: line 8: /home/autobuild/autobuild/instance-1/output-1/per-package/host-gcc-bare-metal/host/bin/-ar: No such file or directory

While at it, also append /n for consistency

Fixes: b7232c51dd
 - http://autobuild.buildroot.org/results/95ac565653ddb5c14ec71470c32a34ad10b048cb

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2024-03-01 09:13:34 +01:00
Yann E. MORIN
fcf465d04b utils/checkpackagelib: add check for CPE variables set to default values
Now that we can specify that the default values for the CPE_ID variables
are valid, without having to actually set one (or more) to their
default, add a check-package check that validates that the CPE_ID
variables are indeed not set to their default.

It also validates that CPE_ID_VALID is not set when another CPE_ID
variable is set to a non-default value.

Add an anchor in the manual so that we can easily point to it.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Cc: Ricardo Martincoski <ricardo.martincoski@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-02-11 16:30:59 +01:00
Yann E. MORIN
dfed5acb56 utils/check-package: use https for the manual URL
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-02-11 16:28:10 +01:00
James Hilliard
c12e1c7b59 utils/scanpypi: use a set comprehension for dependencies
This ensures that we don't have duplicate dependencies.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-02-07 19:58:28 +01:00
Thomas Petazzoni
b7232c51dd utils/genrandconfig: add default setting for bare-metal toolchain arch
We recently added support for building a bare-metal toolchain, and the
autobuilders are therefore trying to build such toolchains. However,
by default the BR2_TOOLCHAIN_BARE_METAL_BUILDROOT_ARCH option is
empty, causing the target tuple to be empty, and therefore
host-gcc-bare-metal tries to build a toolchain for the target
"x86_64-pc-linux-gnu", which isn't a bare-metal target, and fails
badly.

In order to properly test this bare-metal support, this commit adjusts
the genrandconfig script so that it injects a valid target
architecture tuple.

Fixes:

  http://autobuild.buildroot.net/results/6cb1514b19bfe056fb984d3538bdccdda5e174c1/

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-02-07 10:23:48 +01:00
Neal Frager
7befbaaea7 toolchain/toolchain-bare-metal-buildroot: new toolchain
This patch adds a new virtual package for adding a bare-metal
toolchain to Buildroot. For now, it depends on nothing, so it will not
actually build anything, but it defines some options that will be
needed by the various packages that will be part of this toolchain
build process.

Signed-off-by: Neal Frager <neal.frager@amd.com>
Signed-off-by: Ibai Erkiaga <ibai.erkiaga-elorza@amd.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2024-02-06 17:57:17 +01:00
Marcus Hoffmann
21d89a5b86 utils/readme.txt: document docker-run script
Signed-off-by: Marcus Hoffmann <buildroot@bubu1.eu>
[yann.morin.1998@free.fr: the default is an interactive shell]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-02-06 17:02:03 +01:00
Marcus Hoffmann
fba40a9f39 utils/readme.txt: expand check-package description
check-package does a lot more by now than checking .mk and
Config.in files. Add this to the description.

Signed-off-by: Marcus Hoffmann <buildroot@bubu1.eu>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2024-02-06 17:02:02 +01:00
James Hilliard
e241419490 utils/scanpypi: add flit package support
These packages don't have a setup.py so we instead need to parse their
pyproject.toml file.

It is possible that such a package does not define a project_urls config
item, so ensure we do not choke on it (None.get() would raise).

Note that this currently doesn't handle flit package dependency
resolution.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Reviewed-by: Yegor Yefremov <yegorslists@googlemail.com>
[yann.morin.1998@free.fr:
  - add sentence about the project_urls rewrite
  - fix flake8 errors
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2023-11-25 12:47:10 +01:00
Peter Korsgaard
4984d0f230 utils/add-custom-hashes: add script to manage global patch dir hashes
Add a script to manage the .hash files in the BR2_GLOBAL_PATCH_DIR for
packages using custom versions.

To use it, run in a configured Buildroot directory, E.G.

  make foo_defconfig; ./utils/add-custom-hashes

We support multiple patch directories in BR2_GLOBAL_PATCH_DIR.  If multiple
directories are specified then use the last one as that is likely to be the
most specific one.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
[Peter: silence command -v invocation]
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
2023-11-12 11:59:27 +01:00
Yann E. MORIN
8f60241530 utils/docker-run: propagate user's proxy settings
When dealing with enterprise-grade networks, it is more often than not
the case that the wider internet is unreachable but through proxies.

There is a usual set of variables that users can set in the
environment to point various tools (curl, git...) to use those
proxies.

Propagate those variables inside the container.

Note that there are a few tools (e.g. cvs, svn) that may not recognise
those variables; instead, they require custom setup that is too
complex to handle, so is left as an exercise to interested parties.

Similarly, there are other types of proxy, socks4 or socks5, that also
require custom setup that is not trivial to replicate in a container,
so is also left out as an exercise for interested parties.

In the large majority of cases, those few variables are enough to Make
Things Work™.

Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Cc: Ricardo Martincoski <ricardo.martincoski@datacom.com.br>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-11-04 18:42:48 +01:00
Yann E. MORIN
893bd56974 utils/docker: use host's network setup
Our utils/docker-run wrapper is needed to provide a reproducible build
environment: tools, variables, etc... but is not meant for
isolation. As such, we do not care which network configuration is
used.

In some settings (e.g. enterprise networks), it is often the case that
a VPN is in use, especially in those wonderful times of widespread
remote work.

Letting Docker decide on the network setup will most usually lead to
it creating a private network that is NATed onto the principal network
interface, leading to non-functional network in the container when a
VPN is in use.

As such, always use the host network configuration, and do not let
Docker create a private network for the container.

Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Cc: Ricardo Martincoski <ricardo.martincoski@datacom.com.br>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-11-04 18:42:13 +01:00
Daniel Lang
7bc5ea80c7 utils/getdeveloperlib.py: handle file removal
If a patch only removes files, it is ignored. Meaning, that the
registered developer isn't automatically picked up when calling
get-developer.
Fix this by also checking if the line starts with ---, as a patch
removing a file has a line starting with --- with the name of the
removed file and one started with +++ /dev/null.
A set is used to store the changed files, which doesn't allow
duplicates. Therefore normal patches aren't affected by this change.

Signed-off-by: Daniel Lang <dalang@gmx.at>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-09-11 22:08:22 +02:00
James Knight
2be2ccb487 utils/check-package: cleanup line reading
Cleanup the implementation for reading lines by having files processed
in context managers and utilizing the iterable file object for line
reading (instead of needing to call `readlines()`).

Signed-off-by: James Knight <james.d.knight@live.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-09-02 17:47:48 +02:00
Yann E. MORIN
2423d9f16b Release 2023.08-rc2
-----BEGIN PGP SIGNATURE-----
 
 iG8EABECADAWIQSrB9gG0s50H7iG7lCwJbqLWcNjGQUCZOKHvRIcamFjbWV0QHVj
 bGliYy5vcmcACgkQsCW6i1nDYxn1/QCg2un/vUk0HEIbpn4d1fMRZFBDSlwAmKRp
 iO+4qkBgt1h+2LxZSJmNbPY=
 =nvGJ
 -----END PGP SIGNATURE-----

Merge tag '2023.08-rc2' into next

Conflicts:
  - .checkpackageignore
  - Makefile
  - board/versal/post-image.sh
  - package/sentry-cli/0001-Disable-SSL-support-for-the-curl-module.patch
      => keep version in next

  - Config.in.legacy
      => merge, introduce legacy comment for 2023.11

  - toolchain/toolchain-external/toolchain-external-bootlin/Config.in.options
      => regenerate, drop dependency on inexistant BR2_ARCH_NEEDS_GCC_AT_LEAST_14

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2023-08-21 21:36:17 +02:00
Thomas Petazzoni
80a1e728b0 utils/docker-run: allow IMAGE to be passed in the environment
It is sometimes useful to use docker-run, but with a different image
than the default one. This commit allows to override the image being
used by only defining IMAGE if not already passed in the environment.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr: s/\t/    /g]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2023-08-20 16:20:57 +02:00
Yann E. MORIN
3b877dc7c2 utils/docker-run: make it compatible with SELinux
After switching to a fresh Fedora 38 installation with SELinux disabled,
we noticed that utils/docker-run doesn't work as the applications
running inside the container are not allowed to accept the data mounted
through the bind mount.

Since we do not really need to isolate and confine the build, but rather
to provide a known environment, we don;t really need to enforce any
SELinux confinment in the container.

So, we tell docker to turn off label confinement for the container:

    https://manpages.org/docker-run

    --security-opt=[]
      Security Options
        [...]
        "label=disable"     : Turn off label confinement for the container

Suggested-by: Antoine Tenart <atenart@kernel.org>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr: use Antoine's proposal]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2023-08-20 16:20:54 +02:00
Yann E. MORIN
db777eef13 utils/docker-run: also mount current working directory
Currently, using utils/docker-run expects that the current working
directory is the working copy. This means that it is not possible
to use docker-run with an out-of-tree build (one using O=).

Add the current working directory to the list of mountpoints, and
use that as working directory in the container.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-08-20 16:20:51 +02:00
Nicolas Boichat
e71fb8f71c utils/docker-run: bind mount .git/objects if needed
If buildroot is checked out as part of a 'repo' manifest, docker-run
doesn't fully bind mount the .git directory, leading to commands such
as `utils/docker-run make check-package` to fail.

Signed-off-by: Nicolas Boichat <drinkcat@google.com>
[yann.morin.1998@free.fr: use newly introduced mountpoints list]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2023-08-20 16:20:48 +02:00
Thomas Petazzoni
7fddbe2530 utils/docker-run: mount the download directory if specified
If the user has defined $BR2_DL_DIR in the environment, it would be
nice to have it accessible inside the Docker container, and the
BR2_DL_DIR environment variable set to access it.

This commit does exactly this: it mounts the host $BR2_DL_DIR as /dl
in the container, and sets BR2_DL_DIR=/dl in the container.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
[yann.morin.1998@free.fr: use the new mountpoints list]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2023-08-20 16:20:45 +02:00
Yann E. MORIN
f4b798120b utils/docker-run: introduce sorted list of mountpoints
For now, we only ever mount two mountpoints, the main directory (i.e.
the working copy), and the git directory.

To pave the way for adding new mountpoints, we introduce a list of them,
that we sort to ensure that we never mount a shallower mounpoint after a
deeper one (that would shadow the deeper mountpoint).

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-08-20 16:20:42 +02:00
Fabrice Fontaine
283e6859f1 utils/genrandconfig: add ubi handling
Add a custom case to make sure that a random configuration with an empty
configuration file for ubi doesn't fail.

ubinize: error!: no sections found the ini-file "/home/buildroot/autobuild/instance-2/output-1/build/ubinize.cfg"

Fixes:
 - http://autobuild.buildroot.org/results/f678c17cc4df06fb2737467e769cd8f72a3ea420

Signed-off-by: Fabrice Fontaine <fontaine.fabrice@gmail.com>
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
2023-08-12 22:06:21 +02:00
Yann E. MORIN
a845276757 utils/genradconfig: drop legacy lpc32xxcdl exclusion
Commit 503252d8b0 (boot/lpc32xxcdl: remove package) forgot to remove
the special handling in genrandconfig. Since the package no longer
exists, we don't need to special-case its symbol anymore, so drop it
now.

Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2023-08-10 19:45:19 +02:00
Thomas Petazzoni
503252d8b0 boot/lpc32xxcdl: remove package
This package has dubious licensing conditions (not even documented in
the .mk file), and is a bootloader for very old platforms. The
defconfigs making use of it have been removed in Buildroot in 2014, in
commit c6a410964b ("configs: remove
lpc32xx defconfigs"), so let's get rid of the package.

Signed-off-by: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Acked-by: Alexandre Belloni <alexandre.belloni@bootlin.com>
[yann.morin.1998@free.fr: remove reference in test]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2023-08-10 19:41:42 +02:00
Yann E. MORIN
5b559109ee utils/docker-run: fix running when CWD is not MAIN_DIR
Commit 9079079092 (utils/docker-run: fix support for git-worktrees)
got last-minute changes when it was applied, and the case when the
current working directory is not the top of the current working copy
got broken.

Fix that by duplicating (and thus reinstating) the 'cd MAIN_DIR' to
match what is done when retrieving the git-common-dir.

Fixes: 9079079092

Reported-by: Brandon Maier <Brandon.Maier@collins.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2023-07-31 20:56:57 +02:00
Brandon Maier
9079079092 utils/docker-run: fix support for git-worktrees
The docker-run script attempts to support git-new-workdirs and
git-worktrees by resolving the symlink at '$GIT_DIR/config' to get the
true $GIT_DIR. However this does not work for git-worktrees as they do
not use symlinks, instead they change the $GIT_DIR into a regular file
that contains the path to the real $GIT_DIR. To complicate things
further, we actually want the $GIT_COMMON_DIR which is the superset of a
worktree's $GIT_DIR.

git-rev-parse supports the '--git-common-dir' which will resolve the
$GIT_COMMON_DIR for us. However it does not work for git-new-workdirs,
so we still need to detect and handle them.

'--git-common-dir' also appeared only with git 2.10.0, released in 2016,
so it will not be available in older "enterprise-grade" distributions.
In that case, 'git rev-parse --git-common-dir' would return the option
flag '--git-common-dir' as-is, which is incorrect. So, we instruct it to
never return flags.

'--git-common-dir' also returns just '.git' for the main working copy,
but 'docker run' want an absolute path, so we canonicalise it.

Signed-off-by: Brandon Maier <brandon.maier@collins.com>
[yann.morin.1998@free.fr:
  - support git versions before --git-common-dir was introduced
  - don't mount GIT_DIR if unknown (i.e. not needed)
  - fix expanding MAIN_DIR
]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
2023-07-29 17:22:40 +02:00