Simon Glass <sjg@chromium.org> says:
Labgrid provides access to a hardware lab in an automated way. It is
possible to boot U-Boot on boards in the lab without physically touching
them. It relies on relays, USB UARTs and SD muxes, among other things.
By way of background, about 4 years ago I wrong a thing called Labman[1]
which allowed my lab of about 30 devices to be operated remotely, using
tbot for the console and build integration. While it worked OK and I
used it for many bisects, I didn't take it any further.
It turns out that there was already an existing program, called Labgrid,
which I did not know about at time (thank you Tom for telling me). It is
more rounded than Labman and has a number of advantages:
- does not need udev rules, mostly
- has several existing users who rely on it
- supports multiple machines exporting their devices
It lacks a 'lab check' feature and a few other things, but these can be
remedied.
On and off over the past several weeks I have been experimenting with
Labgrid. I have managed to create an initial U-Boot integration (this
series) by adding various features to Labgrid[2] and the U-Boot test
hooks.
I hope that this might inspire others to set up boards and run tests
automatically, rather than relying on infrequent, manual test. Perhaps
it may even be possible to have a number of labs available.
Included in the integration are a number of simple scripts which make it
easy to connect to boards and run tests:
ub-int <target>
Build and boot on a target, starting an interactive session
ub-cli <target>
Build and boot on a target, ensure U-Boot starts and provide an interactive
session from there
ub-smoke <target>
Smoke test U-Boot to check that it boots to a prompt on a target
ub-bisect <target>
Bisect a git tree to locate a failure on a particular target
ub-pyt <target> <testspec>
Run U-Boot pytests on a target
Some of these help to provide the same tbot[4] workflow which I have
relied on for several years, albeit much simpler versions.
The goal here is to create some sort of script which can collect
patches from the mailing list, apply them and test them on a selection
of boards. I suspect that script already exists, so please let me know
what you suggest.
I hope you find this interesting and take a look!
[1] https://github.com/sjg20/u-boot/tree/lab6a
[2] https://github.com/labgrid-project/labgrid/pull/1411
[3] https://github.com/sjg20/uboot-test-hooks/tree/labgrid
[4] https://tbot.tools/index.html
Link: https://lore.kernel.org/r/20241112141326.643128-1-sjg@chromium.org
[trini: Move the sjg-lab job to prior to world build, to fix pipeline
status]
Signed-off-by: Tom Rini <trini@konsulko.com>
Add a way to run tests on a real hardware lab. This is in the very early
experimental stages. There are only 23 boards and 3 of those are broken!
(bob, ff3399, samus). A fourth fails due to problems with the TPM tests.
To try this, assuming you have gitlab access, set SJG_LAB=1, e.g.:
git push -o ci.variable="SJG_LAB=1" dm HEAD:try
This relies on the two previous series targeted at -next as well as the
bugfix series for -master
Signed-off-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Andrejs Cainikovs <andrejs.cainikovs@toradex.com>
Patrick Rudolph <patrick.rudolph@9elements.com> says:
Based on the existing work done by Simon Glass this series adds
support for booting aarch64 devices using ACPI only.
As first target QEMU SBSA support is added, which relies on ACPI
only to boot an OS. As secondary target the Raspberry Pi4 was used,
which is broadly available and allows easy testing of the proposed
solution.
The series is split into ACPI cleanups and code movements, adding
Arm specific ACPI tables and finally SoC and mainboard related
changes to boot a Linux on the QEMU SBSA and RPi4. Currently only the
mandatory ACPI tables are supported, allowing to boot into Linux
without errors.
The QEMU SBSA support is feature complete and provides the same
functionality as the EDK2 implementation.
The changes were tested on real hardware as well on QEMU v9.0:
qemu-system-aarch64 -machine sbsa-ref -nographic -cpu cortex-a57 \
-pflash secure-world.rom \
-pflash unsecure-world.rom
qemu-system-aarch64 -machine raspi4b -kernel u-boot.bin -cpu cortex-a72 \
-smp 4 -m 2G -drive file=raspbian.img,format=raw,index=0 \
-dtb bcm2711-rpi-4-b.dtb -nographic
Tested against FWTS V24.03.00.
Known issues:
- The QEMU rpi4 support is currently limited as it doesn't emulate PCI,
USB or ethernet devices!
- The SMP bringup doesn't work on RPi4, but works in QEMU (Possibly
cache related).
- PCI on RPI4 isn't working on real hardware since the pcie_brcmstb
Linux kernel module doesn't support ACPI yet.
Link: https://lore.kernel.org/r/20241023132116.970117-1-patrick.rudolph@9elements.com
Add QEMU's SBSA ref board to azure pipelines and gitlab CI to run tests on it.
TEST: Run on Azure pipelines and confirmed that tests succeed.
Signed-off-by: Patrick Rudolph <patrick.rudolph@9elements.com>
Reviewed-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Build and run qemu_arm64_lwip_defconfig in CI. This tests the lightweight
IP (lwIP) implementation of the dhcp, tftpboot and ping commands.
Signed-off-by: Jerome Forissier <jerome.forissier@linaro.org>
When we have platforms being emulated by QEMU we cannot rely on the
"sleep" command running for the expected wall-clock amount of time. Even
with our current allowance for deviation from expected time, it will
still fail from time to time. Exclude the sleep test here.
Signed-off-by: Tom Rini <trini@konsulko.com>
Since MbedTLS is an external repo with its own coding style,
exclude it from Azure and gitlab CI CONFIG checks.
Signed-off-by: Raymond Mao <raymond.mao@linaro.org>
Reviewed-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
Jiaxun Yang <jiaxun.yang@flygoat.com> says:
Hi all,
This series enabled qemu-xtensa board.
For dc232b CPU it needs to be built with toolchain[1].
This is a side product of me investigating architectures
physical address != virtual address in U-Boot. Now we can
get it covered under CI and regular tests.
VirtIO devices are not working as expected, due to U-Boot's
assumption on VA == PA everywhere, I'm going to get this fixed
later.
My Xtensa knowledge is pretty limited, Xtensa people please
feel free to point out if I got anything wrong.
Thanks
[1]: https://github.com/foss-xtensa/toolchain/releases/download/2020.07/x86_64-2020.07-xtensa-dc232b-elf.tar.gz
Both GitLab and Azure (and other CI systems) have native support for
displaying JUnitXML test report results. The pytest framework that we
use can generate these reports. Change our CI tests so that they will
generate these reports and then have the respective CI platform pick
them up. We write to different locations because of where each CI is
(and isn't) able to easily pass things along.
Reviewed-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Since devicetree-rebasing is an external repo with its own coding style,
exclude it from Azure and gitlab CI CONFIG checks.
Reviewed-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Ilias Apalodimas <ilias.apalodimas@linaro.org>
Reviewed-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Sumit Garg <sumit.garg@linaro.org>
Instead of downloading coreboot binaries from a Google drive location,
use the ones we have built ourselves.
Signed-off-by: Tom Rini <trini@konsulko.com>
The primary motivation for having a sandbox without LTO build in CI is
to ensure that we don't have that option break. We now have the ability
to run tests of specific options being enabled/disabled, so drop the
parts of CI that build and test that configuration specifically and add
a build test instead. We still test that "NO_LTO=1" rather than editing
the config file works via the ftrace tests.
Reviewed-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
Use the most recent upstream release of OpenSBI for CI testing.
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Reviewed-by: Leo Yu-Chi Liang <ycliang@andestech.com>
At this point we have all of the defconfigs maintained again, so
re-enable the check to prevent further regressions.
Signed-off-by: Tom Rini <trini@konsulko.com>
This add some basic functions to create images, and a test for said
functions. This is not intended to be a test of the image parsing
functions, but rather a framework for creating minimal images for testing
load methods. That said, it does do an OK job at finding bugs in the image
parsing directly.
Since we have two methods for loading/parsing FIT images, add LOAD_FIT_FULL
as a separate CI run.
Signed-off-by: Sean Anderson <seanga2@gmail.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Out-of-tree users could run an out-of-tree CI with limited coverage,
however it is convenient to be able to run the upstream CI from time
to time. To enable that we would need to change job rules to be able
to run on any GitLab event. Excerpt from GitLab documentation:
> Jobs with no rules default to except: merge_requests
Signed-off-by: Andrejs Cainikovs <andrejs.cainikovs@toradex.com>
Reviewed-by: Tom Rini <trini@konsulko.com>
Test both 32bit and 64bit sandbox boards in CI.
Signed-off-by: Marek Vasut <marek.vasut+renesas@mailbox.org>
Reviewed-by: Tom Rini <trini@konsulko.com>
- We have added more TODO/etc comments since this task was created and
never focused on removing them.
- The output of sloccount isn't preserved or looked at, and if desired
should be in the release stats pages instead somehow.
- The results of cppcheck aren't investigated and require modeling work
to be useful to start with.
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
These jobs are to confirm specific build targets, on a Linux host. We
can safely combine these two build tests, with a make mrproper in
between.
Signed-off-by: Tom Rini <trini@konsulko.com>
This platform is behind on migrations (it is the sole user of the oldest
legacy version of the USB gadget stack and is long overdue for
migration) and with Pali no longer being a maintainer, we remove this
platform.
Signed-off-by: Tom Rini <trini@konsulko.com>
When running the trace test on the sandbox platform, the current size
of 16MiB is no longer large enough for capturing the entire trace
history, and results in truncation. Use a size of 32MiB for the trace
buffer on the sandbox platform while running the trace test.
Signed-off-by: Sughosh Ganu <sughosh.ganu@linaro.org>
Reviewed-by: Simon Glass <sjg@chromium.org>
The latest kernel.org toolchains for gcc are now 13.2.0, so upgrade to
that.
Reviewed-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Tom Rini <trini@konsulko.com>
When running tools for various tests use the tools-only build rather
than sandbox_spl. We used sandbox_spl here for historical reasons that
are no longer true.
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Use a recent coreboot build for this test.
The coreboot commit is:
6f5ead14b4 mb/google/nissa/var/joxer: Update eMMC DLL settings
This is build with default settings, i.e. QEMU x86 i440fx/piix4
Add some documentation as to how to update it next time.
Signed-off-by: Simon Glass <sjg@chromium.org>
Rather than using the -R option to get this report as a side effect, add
a dedicated option for it.
Disable CI for now as there are some missing maintainers, unfortunately.
Signed-off-by: Simon Glass <sjg@chromium.org>
Now that buildman has a requirements.txt file we need to make use of it.
Reviewed-by: Simon Glass <sjg@chromium.org>
[n-francis@ti.com: Adding missing command from .azure-pipelines.yml]
Signed-off-by: Neha Malcom Francis <n-francis@ti.com>
Signed-off-by: Tom Rini <trini@konsulko.com>
It is not uncommon for some of the QEMU-based jobs to fail not because
of a code issue but rather because of a timing issue or similar problem
that is out of our control. Make use of the keywords that Azure and
GitLab provide so that we will automatically re-run these when they fail
2 times. If they fail that often it is likely we have found a real issue
to investigate.
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
As this is the current version of the public cross toolchains we use,
upgrade to this now.
Suggested-by: Alexey Brodkin <Alexey.Brodkin@synopsys.com>
Signed-off-by: Tom Rini <trini@konsulko.com>
Acked-by: Alexey Brodkin <abrodkin@synopsys.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Use the latest OpenSBI v1.2 release binaries for the RISC-V CI.
Signed-off-by: Bin Meng <bmeng@tinylab.org>
Reviewed-by: Leo Yu-Chi Liang <ycliang@andestech.com>
We do not want to merge documentation that produces Sphinx warnings.
scripts/kernel-doc uses environment variable KDOC_WERROR to determine
if warnings should be treated as errors.
Reported-by: Tom Rini <trini@konsulko.com>
Signed-off-by: Heinrich Schuchardt <heinrich.schuchardt@canonical.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Add M5208EVBE board to CI. This does not use default config due to
limitations of QEMU emulation, instead the timer is switched from
DMA timer to PIT timer and RAMBAR accesses are inhibited.
Local QEMU launch command is as follows:
$ qemu-system-m68k -nographic -machine mcf5208evb -cpu m5208 -bios u-boot.bin
Signed-off-by: Marek Vasut <marek.vasut+renesas@mailbox.org>
Acked-by: Angelo Dureghello <angelo@kernel-space.org>
As this is now the stable release, move to using that now for our tests.
Signed-off-by: Tom Rini <trini@konsulko.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
To conserve bandwidth and potentially avoid rate limits, allow a local
mirror of Docker Hub to be specified globally. The default value is
unchanged.
Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>
The default behavior of Gitlab runners is to only run jobs which match
the configured tag, although there is an option to run untagged jobs
[1].
To support running the CI in more complex environments where different
types of runners may be present that support different tags, allow the
DEFAULT_TAG for all jobs in the pipeline to be set globally using an
environment variable. An empty default value is provided to retain
support for untagged runners.
[1] https://docs.gitlab.com/ee/ci/runners/configure_runners.html#use-tags-to-control-which-jobs-a-runner-can-run
Signed-off-by: Peter Hoyes <Peter.Hoyes@arm.com>
Reviewed-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>
Copy build artifacts for all test.py tests, so they show up in
artifacts storage for later inspection. The test.py tests output
in CI is basically useless, but it is far more useful in the html
output for analysis and debugging.
Reviewed-by: Simon Glass <sjg@chromium.org>
Reviewed-by: Tom Rini <trini@konsulko.com>
Suggested-by: Simon Glass <sjg@chromium.org>
Signed-off-by: Marek Vasut <marek.vasut+renesas@mailbox.org>
Now that the Dockerfile creates images which have the binaries we
require included, have CI make symlinks for them and update the existing
script to support this.
Signed-off-by: Tom Rini <trini@konsulko.com>
Signed-off-by: Pali Rohár <pali@kernel.org>
Reviewed-by: Simon Glass <sjg@chromium.org>
