Commit Graph

103944 Commits

Author SHA1 Message Date
Richard Henderson
372b7ec3a8 target/arm: Split gen_add_CC and gen_sub_CC
Split out specific 32-bit and 64-bit functions.
These carry the same signature as tcg_gen_add_i64,
and so will be easier to pass as callbacks.

Retain gen_add_CC and gen_sub_CC during conversion.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Message-id: 20230512144106.3608981-6-peter.maydell@linaro.org
[PMM: rebased]
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2023-05-18 11:28:39 +01:00
Richard Henderson
45fda88ea2 target/arm: Convert PC-rel addressing to decodetree
Convert the ADR and ADRP instructions.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Message-id: 20230512144106.3608981-5-peter.maydell@linaro.org
[PMM: Rebased]
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2023-05-18 11:28:38 +01:00
Peter Maydell
270076d01a target/arm: Pull calls to disas_sve() and disas_sme() out of legacy decoder
The SVE and SME decode is already done by decodetree.  Pull the calls
to these decoders out of the legacy decoder.  This doesn't change
behaviour because all the patterns in sve.decode and sme.decode
already require the bits that the legacy decoder is decoding to have
the correct values.

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-id: 20230512144106.3608981-4-peter.maydell@linaro.org
2023-05-18 11:16:45 +01:00
Peter Maydell
8058c8316f target/arm: Create decodetree skeleton for A64
The A64 translator uses a hand-written decoder for everything except
SVE or SME.  It's fairly well structured, but it's becoming obvious
that it's still more painful to add instructions to than the A32
translator, because putting a new instruction into the right place in
a hand-written decoder is much harder than adding new instruction
patterns to a decodetree file.

As the first step in conversion to decodetree, create the skeleton of
the decodetree decoder; where it does not handle instructions we will
fall back to the legacy decoder (which will be for everything at the
moment, since there are no patterns in a64.decode).

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-id: 20230512144106.3608981-3-peter.maydell@linaro.org
2023-05-18 11:16:45 +01:00
Richard Henderson
8ed24ba17a target/arm: Split out disas_a64_legacy
Split out all of the decode stuff from aarch64_tr_translate_insn.
Call it disas_a64_legacy to indicate it will be replaced.

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Message-id: 20230512144106.3608981-2-peter.maydell@linaro.org
[PMM: Rebased]
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2023-05-18 11:16:45 +01:00
Marcin Juszkiewicz
9162ac6b9e sbsa-ref: use Bochs graphics card instead of VGA
Bochs card is normal PCI Express card so it fits better in system with
PCI Express bus. VGA is simple legacy PCI card.

Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
Reviewed-by: Leif Lindholm <quic_llindhol@quicinc.com>
Message-id: 20230505120936.1097060-1-marcin.juszkiewicz@linaro.org
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2023-05-18 11:13:25 +01:00
Alex Bennée
70a670cadb target/arm: add RAZ/WI handling for DBGDTR[TX|RX]
The commit b3aa2f2128 (target/arm: provide stubs for more external
debug registers) was added to handle HyperV's unconditional usage of
Debug Communications Channel. It turns out that Linux will similarly
break if you enable CONFIG_HVC_DCC "ARM JTAG DCC console".

Extend the registers we RAZ/WI set to avoid this.

Cc: Anders Roxell <anders.roxell@linaro.org>
Cc: Evgeny Iakovlev <eiakovlev@linux.microsoft.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-id: 20230516104420.407912-1-alex.bennee@linaro.org
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2023-05-18 11:09:36 +01:00
Cornelia Huck
b320e21c48 arm/kvm: add support for MTE
Extend the 'mte' property for the virt machine to cover KVM as
well. For KVM, we don't allocate tag memory, but instead enable the
capability.

If MTE has been enabled, we need to disable migration, as we do not
yet have a way to migrate the tags as well. Therefore, MTE will stay
off with KVM unless requested explicitly.

Signed-off-by: Cornelia Huck <cohuck@redhat.com>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-id: 20230428095533.21747-2-cohuck@redhat.com
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2023-05-18 11:08:38 +01:00
Marcin Juszkiewicz
96e6d25fdd Maintainers: add myself as reviewer for sbsa-ref
At Linaro I work on sbsa-ref, know direction it goes.

May not get code details each time.

Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-id: 20230515143753.365591-1-marcin.juszkiewicz@linaro.org
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2023-05-18 11:08:03 +01:00
Richard Henderson
a6771f2f5c target/arm: Fix vd == vm overlap in sve_ldff1_z
If vd == vm, copy vm to scratch, so that we can pre-zero
the output and still access the gather indicies.

Cc: qemu-stable@nongnu.org
Resolves: https://gitlab.com/qemu-project/qemu/-/issues/1612
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Message-id: 20230504104232.1877774-1-richard.henderson@linaro.org
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2023-05-18 10:31:43 +01:00
Marcin Juszkiewicz
1877272bad sbsa-ref: switch default cpu core to Neoverse-N1
The world outside moves to newer and newer cpu cores. Let move SBSA
Reference Platform to something newer as well.

Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
Reviewed-by: Leif Lindholm <quic_llindhol@quicinc.com>
Message-id: 20230506183417.1360427-1-marcin.juszkiewicz@linaro.org
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2023-05-18 10:31:43 +01:00
Paolo Bonzini
911d4caaa2 build: move stack protector flag selection to meson
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:52 +02:00
Paolo Bonzini
6739825aa6 build: move coroutine backend selection to meson
To simplify the code, rename coroutine-win32.c to match the option
passed to configure.

Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:52 +02:00
Paolo Bonzini
721fa5e563 build: move SafeStack tests to meson
This disables the old behavior of detecting SafeStack from environment
CFLAGS.  SafeStack is now enabled purely based on the configure arguments.

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:52 +02:00
Paolo Bonzini
34f983d86f build: move sanitizer tests to meson
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:52 +02:00
Paolo Bonzini
d67212d47f meson: prepare move of QEMU_CFLAGS to meson
Clean up the handling of compiler flags in meson.build, splitting
the general flags that should be included in subprojects as well,
from warning flags that only apply to QEMU itself.  The two were
mixed in both configure tests and meson tests.

This split makes it easier to move the compiler tests piecewise
from configure to Meson.

Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:52 +02:00
Paolo Bonzini
6002711c66 configure, meson: move --enable-modules to Meson
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:52 +02:00
Paolo Bonzini
877c556705 configure: remove pkg-config functions
All uses of pkg-config have been moved to Meson.

Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:52 +02:00
Paolo Bonzini
fc9a809e0d build: move glib detection and workarounds to meson
QEMU adds the path to glib.h to all compilation commands.  This is simpler
due to the pervasive use of static_library, and was grandfathered in from
the previous Make-based build system.  Until Meson 0.63 the only way to
do this was to detect glib in configure and use add_project_arguments,
but now it is possible to use add_project_dependencies instead.

gmodule is detected in a separate variable, with export enabled for
modules and disabled for plugin.

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:52 +02:00
Paolo Bonzini
5328353c05 meson: drop unnecessary declare_dependency()
The libvfio_user_dep variable of subprojects/libvfio-user/lib/meson.build
is already a dependency, so there is no need to wrap it with another
declare_dependency().

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:52 +02:00
Paolo Bonzini
bd3615d68f meson: add more version numbers to the summary
Whenever declare_dependency is used to add some compile flags or dependent
libraries to the outcome of dependency(), the version of the original
dependency is dropped in the summary.  Make sure that declare_dependency()
has a version argument in those cases.

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
Paolo Bonzini
063d5119c1 meson: remove static_kwargs
After static_kwargs has been changed to an empty dictionary, it has
no functional effect and can be removed.

Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
Paolo Bonzini
a0cbd2e849 meson: use prefer_static option
The option is new in Meson 0.63 and removes the need to pass "static:
true" to all dependency and find_library invocation.  Actually cleaning
up the invocations is left for a separate patch.

Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
Paolo Bonzini
3b087f79a4 meson: require 0.63.0
This version allows cleanups in modinfo collection, but they only
work with Ninja 1.9.x and 1.8.x is still supported.  It also supports the
equivalent of QEMU's --static option to configure.

The wheel file is bumped to 0.63.3, the last release in the 0.63 branch.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
Paolo Bonzini
5fbfea6c9d meson: regenerate meson-buildoptions.sh
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
Paolo Bonzini
7b4b98c46c python: bump some of the dependencies
The version of pyflakes that is listed in python/tests/minreqs.txt
breaks on Python 3.8 with the following message:

  AttributeError: 'FlakesChecker' object has no attribute 'CONSTANT'

Now that we do not support EOL'd Python versions anymore, we can
update to newer, fixed versions.  It is a good time to do so, before
Python packages start dropping support for Python 3.7 as well!

The new mypy is also a bit smarter about which packages are actually
being used, so remove the now-unnecessary sections from setup.cfg.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-27-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
Paolo Bonzini
02312f1af1 mkvenv: mark command as required
This is only available in Python 3.7+.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-26-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
John Snow
e46b82a03c configure: Add courtesy hint to Python version failure message
If we begin requiring Python 3.7+, a few platforms are going to need to
install an additional Python interpreter package.

As a courtesy to the user, suggest the optional package they might need
to install. This will hopefully minimize any downtime caused by the
change in Python dependency.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-Id: <20230221012456.2607692-3-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20230511035435.734312-25-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
Paolo Bonzini
5591b74511 Python: Drop support for Python 3.6
Python 3.6 was EOL 2021-12-31. Newer versions of upstream libraries have
begun dropping support for this version and it is becoming more
cumbersome to support. Avocado-framework and qemu.qmp each have their
own reasons for wanting to drop Python 3.6, but won't until QEMU does.

Versions of Python available in our supported build platforms as of today,
with optional versions available in parentheses:

openSUSE Leap 15.4: 3.6.15 (3.9.10, 3.10.2)
CentOS Stream 8:    3.6.8  (3.8.13, 3.9.16)
CentOS Stream 9:    3.9.13
Fedora 36:          3.10
Fedora 37:          3.11
Debian 11:          3.9.2
Alpine 3.14, 3.15:  3.9.16
Alpine 3.16, 3.17:  3.10.10
Ubuntu 20.04 LTS:   3.8.10
Ubuntu 22.04 LTS:   3.10.4
NetBSD 9.3:         3.9.13*
FreeBSD 12.4:       3.9.16
FreeBSD 13.1:       3.9.16
OpenBSD 7.2:        3.9.16

Note: Our VM tests install 3.9 explicitly for FreeBSD and 3.10 for
NetBSD; the default for "python" or "python3" in FreeBSD is
3.9.16. NetBSD does not appear to have a default meta-package, but
offers several options, the lowest of which is 3.7.15. "python39"
appears to be a pre-requisite to one of the other packages we request in
tests/vm/netbsd. pip, ensurepip and other Python essentials are
currently only available for Python 3.10 for NetBSD.

CentOS and OpenSUSE support parallel installation of multiple Python
interpreters, and binaries in /usr/bin will always use Python 3.6.  However,
the newly introduced support for virtual environments ensures that all build
steps that execute QEMU Python code use a single interpreter.

Since it is safe to under our supported platform policy, bump our
minimum supported version of Python to 3.7.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20230511035435.734312-24-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
John Snow
0c5f3dcbb2 configure: add --enable-pypi and --disable-pypi
In the event that there's no vendored source present and no sufficient
version of $package can be found, we will attempt to connect to PyPI to
install the package if '--disable-pypi' was not passed.

This means that PyPI access is "enabled by default", but there are some
subtleties that make this action occur much less frequently than you
might imagine:

(1) While --enable-pypi is the default, vendored source will always be
    preferred when found, making PyPI a fallback. This should ensure
    that configure-time venv building "just works" for almost everyone
    in almost every circumstance.

(2) Because meson source is, at time of writing, vendored directly into
    qemu.git, PyPI will never be used for sourcing meson.

(3) Because Sphinx is an optional dependency, if docs are set to "auto",
    PyPI will not be used to obtain Sphinx source as a fallback and
    instead docs will be disabled. If PyPI sourcing of sphinx is
    desired, --enable-docs should be passed to force the lookup. I chose
    this as the default behavior to avoid adding new internet lookups to
    a "default" invocation of configure.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-23-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
John Snow
6f6652ebc2 configure: bootstrap sphinx with mkvenv
When docs are explicitly requested, require Sphinx>=1.6.0. When docs are
explicitly disabled, don't bother to check for Sphinx at all. If docs
are set to "auto", attempt to locate Sphinx, but continue onward if it
wasn't located.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-22-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
John Snow
6b0cedcdc7 configure: move --enable-docs and --disable-docs back to configure
Move this option back from meson into configure for the purposes of
using the configuration value to bootstrap Sphinx in different ways
based on this value.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-21-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
John Snow
9c6692db55 tests: Use configure-provided pyvenv for tests
This patch changes how the avocado tests are provided, ever so
slightly. Instead of creating a new testing venv, use the
configure-provided 'pyvenv' instead, and install optional packages into
that.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-20-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
John Snow
aab9512756 qemu.git: drop meson git submodule
Now that meson is installed from a vendored wheel, we don't need the git
submodule anymore. Drop it.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-19-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
John Snow
66e2c6cbac configure: use 'mkvenv ensure meson' to bootstrap meson
This commit changes how we detect and install meson. It notably removes
'--meson='.

Currently, configure creates a lightweight Python virtual environment
unconditionally using the user's configured $python that inherits system
packages. Temporarily, we forced the use of meson source present via git
submodule or in the release tarball.

With this patch, we restore the ability to use a system-provided meson:

If Meson is installed in the build venv and meets our minimum version
requirements, we will use that Meson. This includes a system provided
meson, which would be visible via system-site packages inside the venv.

In the event that Meson is installed but *not for the chosen Python
interpreter*, not found, or of insufficient version, we will attempt to
install Meson from vendored source into the newly created Python virtual
environment. This vendored installation replaces both the git submodule
and tarball source mechanisms for sourcing meson.

As a result of this patch, the Python interpreter we use for both our
own build scripts *and* Meson extensions are always known to be the
exact same Python. As a further benefit, there will also be a symlink
available in the build directory that points to the correct, configured
python and can be used by e.g. manual tests to invoke the correct,
configured Python unambiguously.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-18-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
John Snow
e80bdbf283 python/wheels: add vendored meson package
In preference to vendoring meson source, vendor a built distributable
("bdist" in python parlance). This has some benefits:

(1) We can get rid of a git submodule,
(2) Installing built meson into a venv doesn't require any extra
    dependencies (the python "wheel" package, chiefly.)
(3) We don't treat meson any differently than we would any other python
    package (we install it, end of story, done.)
(4) All future tarball *and* developer checkouts will function offline;
    No git or PyPI connection needed to fetch meson.

Note that because mkvenv prefers vendored packages to PyPI, as mkvenv is
currently written we will never consult PyPI for meson. (Do keep in mind
that your distribution's meson will be preferred above the vendored
version, though.)

```
jsnow@scv ~/s/q/python (python-configure-venv)> python3 scripts/vendor.py
pip download --dest /home/jsnow/src/qemu/python/wheels --require-hashes -r /tmp/tmpvo5qav7i
Collecting meson==0.61.5
  Using cached meson-0.61.5-py3-none-any.whl (862 kB)
Saved ./wheels/meson-0.61.5-py3-none-any.whl
Successfully downloaded meson
```

Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-17-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
John Snow
81e2b198a8 configure: create a python venv unconditionally
This patch changes the configure script so that it always creates and
uses a python virtual environment unconditionally.

Meson bootstrapping is temporarily altered to force the use of meson
from git or vendored source (as packaged in our source tarballs). A
subsequent commit restores the use of distribution-vendored Meson.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-16-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
John Snow
2274817f6c python: add vendor.py utility
This is a teeny-tiny script that just downloads any packages we want to
vendor from PyPI and stores them in qemu.git/python/wheels/. If I'm hit
by a meteor, it'll be easy to replicate what I have done in order to
udpate the vendored source.

We don't really care which python runs it; it exists as a meta-utility
with no external dependencies and we won't package or install it. It
will be monitored by the linters/type checkers, though; so it's
guaranteed safe on python 3.6+.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-15-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
John Snow
1dee66c693 tests/vm: add py310-expat to NetBSD
NetBSD cannot successfully run "ensurepip" without access to the pyexpat
module, which NetBSD debundles. Like the Debian patch, it would be
strictly faster long term to install pip/setuptools, and I recommend
developers at their workstations take that approach instead.

For the purposes of a throwaway VM, there's not really a speed
difference for who is responsible for installing pip; us (needs
py310-pip) or Python (needs py310-expat).

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20230511035435.734312-14-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
John Snow
dd48b477e9 tests/vm: Configure netbsd to use Python 3.10
NetBSD removes some packages from the Python stdlib, but only
re-packages them for Python 3.10. Switch to using Python 3.10.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20230511035435.734312-13-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
John Snow
a22a4b29ad tests/docker: add python3-venv dependency
Several debian-based tests need the python3-venv dependency as a
consequence of Debian debundling the "ensurepip" module normally
included with Python.

As mkvenv.py stands as of this commit, Debian requires EITHER:

(A) setuptools and pip, or
(B) ensurepip

mkvenv is a few seconds faster if you have setuptools and pip, so
developers should prefer the first requirement. For the purposes of CI,
the time-save is a wash; it's only a matter of who is responsible for
installing pip and when; the timing is about the same.

Arbitrarily, I chose adding ensurepip to the test configuration because
it is normally part of the Python stdlib, and always having it allows us
a more consistent cross-platform environment.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20230511035435.734312-12-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
John Snow
c804962617 mkvenv: work around broken pip installations on Debian 10
This is a workaround intended for Debian 10, where the debian-patched
pip does not function correctly if accessed from within a virtual
environment.

We don't support Debian 10 as a build platform any longer, though we do
still utilize it for our build-tricore-softmmu CI test. It's also
possible that this bug might appear on other derivative platforms and
this workaround may prove useful.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-11-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
John Snow
f1ad527ff5 mkvenv: avoid ensurepip if pip is installed
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
John Snow
68ea6d17fe mkvenv: use pip's vendored distlib as a fallback
distlib is usually not installed on Linux distribution, but it is vendored
into pip.  Because the virtual environment has pip via ensurepip, we
can piggy-back on pip's vendored version.  This could break if they move
our cheese in the future, but the fix would be simply to require distlib.

If it is debundled, as it is on msys, it is simply available directly.

Signed-off-by: John Snow <jsnow@redhat.com>
[Move to toplevel. - Paolo]
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
John Snow
928348949d mkvenv: add console script entry point generation
When creating a virtual environment that inherits system packages,
script entry points (like "meson", "sphinx-build", etc) are not
re-generated with the correct shebang. When you are *inside* of the
venv, this is not a problem, but if you are *outside* of it, you will
not have a script that engages the virtual environment appropriately.

Add a mechanism that generates new entry points for pre-existing
packages so that we can use these scripts to run "meson",
"sphinx-build", "pip", unambiguously inside the venv.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-9-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
John Snow
4695a22e9a mkvenv: add --diagnose option to explain "ensure" failures
This is a routine that is designed to print some usable info for human
beings back out to the terminal if/when "mkvenv ensure" fails to locate
or install a package during configure time, such as meson or sphinx.

Since we are requiring that "meson" and "sphinx" are installed to the
same Python environment as QEMU is configured to build with, this can
produce some surprising failures when things are mismatched. This method
is here to try and ease that sting by offering some actionable
diagnosis.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-8-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
John Snow
c5538eed12 mkvenv: add ensure subcommand
This command is to be used to add various packages (or ensure they're
already present) into the configure-provided venv in a modular fashion.

Examples:

mkvenv ensure --online --dir "${source_dir}/python/wheels/" "meson>=0.61.5"
mkvenv ensure --online "sphinx>=1.6.0"
mkvenv ensure "qemu.qmp==0.0.2"

It's designed to look for packages in three places, in order:

(1) In system packages, if the version installed is already good
enough. This way your distribution-provided meson, sphinx, etc are
always used as first preference.

(2) In a vendored packages directory. Here I am suggesting
qemu.git/python/wheels/ as that directory. This is intended to serve as
a replacement for vendoring the meson source for QEMU tarballs. It is
also highly likely to be extremely useful for packaging the "qemu.qmp"
package in source distributions for platforms that do not yet package
qemu.qmp separately.

(3) Online, via PyPI, ***only when "--online" is passed***. This is only
ever used as a fallback if the first two sources do not have an
appropriate package that meets the requirement. The ability to build
QEMU and run tests *completely offline* is not impinged.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-7-jsnow@redhat.com>
[Use distlib to lookup distributions. - Paolo]
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
John Snow
dee01b827f mkvenv: add nested venv workaround
Python virtual environments do not typically nest; they may inherit from
the top-level system packages or not at all.

For our purposes, it would be convenient to emulate "nested" virtual
environments to allow callers of the configure script to install
specific versions of python utilities in order to test build system
features, utility version compatibility, etc.

While it is possible to install packages into the system environment
(say, by using the --user flag), it's nicer to install test packages
into a totally isolated environment instead.

As detailed in https://www.qemu.org/2023/03/24/python/, Emulate a nested
venv environment by using .pth files installed into the site-packages
folder that points to the parent environment when appropriate.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-6-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
John Snow
a9dbde71da mkvenv: add better error message for broken or missing ensurepip
Debian debundles ensurepip for python; NetBSD debundles pyexpat but
ensurepip needs pyexpat. Try our best to offer a helpful error message
instead of just failing catastrophically.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-5-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00
John Snow
dd84028ff9 python: add mkvenv.py
This script will be responsible for building a lightweight Python
virtual environment at configure time. It works with Python 3.6 or
newer.

It has been designed to:
- work *offline*, no PyPI required.
- work *quickly*, The fast path is only ~65ms on my machine.
- work *robustly*, with multiple fallbacks to keep things working.
- work *cooperatively*, using system packages where possible.
  (You can use your distro's meson, no problem.)

Due to its unique position in the build chain, it exists outside of the
installable python packages in-tree and *must* be runnable without any
third party dependencies.

Under normal circumstances, the only dependency required to execute this
script is Python 3.6+ itself. The script is *faster* by several seconds
when setuptools and pip are installed in the host environment, which is
probably the case for a typical multi-purpose developer workstation.

In the event that pip/setuptools are missing or not usable, additional
dependencies may be required on some distributions which remove certain
Python stdlib modules to package them separately:

- Debian may require python3-venv to provide "ensurepip"
- NetBSD may require py310-expat to provide "pyexpat" *
  (* Or whichever version is current for NetBSD.)

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Message-Id: <20230511035435.734312-4-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2023-05-18 08:53:51 +02:00