When running on a detached head (eg. checkout of a tag or a specific
commit), there is no active branch, so we can't perform this check; just
skip it and assume the user knows what they're doing.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27578>
This avoids the surprising behaviour where `--target jobname` works for
some jobs but not others, because gitlab adds `X/N` at the end of these
job names.
If the user does specify something like `jobname 1/.*` to only run the
first, the extra `\d+/\d+` is ignored, just like if the job isn't
`parallel:` and therefore doesn't end with `X/N`.
If the user really wants to fail to match parallel jobs (previous
behaviour), they can simply add a `$` at the end of the job name/regex
(but also, I don't see why someone would want that behaviour).
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27530>
This validation code has 2 bugs, the main one being that it is wrong and
is refusing perfectly valid codes. Let's remove this until we come up
with a valid check.
This reverts commit cd8b546205.
Fixes: cd8b546205 ("bin/ci: Add GitLab basic token validation")
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27312>
Validate the token to be able to warn the user about wrong
authentication settings.
Treat the default token argument case, when the token file is not found.
Add some loggings to guide the user.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27206>
Moved the `get_token_from_default_dir` method to a common module for
shared use between `gitlab_gql.py` and `ci_run_n_monitor.py`. This
migration facilitates better code organization and potential future
reuse.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27206>
Fix an issue in `ci_run_n_monitor.py` where the token was not
being correctly propagated to the GitlabGQL abstraction. This addresses
misbehavior in scenarios like running pipelines in a private fork,
ensuring proper functionality.
Also document `find_dependencies` function.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27206>
When generating the list of targets from a script, being able to just
pass `--target "${list[@]}"` is very convenient.
The list of targets is simply converted to an "or" regex, matching any
of the `--target`s given.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/27252>
Generate Gantt chart and post an in thread reply to Marge's messages
with it in html format.
html format is used for being interactive.
Signed-off-by: Helen Koike <helen.koike@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26796>
Use the CustomLogger class and CLI tool to create strutured logs
for poe scripts which are used by broadcom and nouveau jobs.
Renamed stage lint to code-validation and added python-test job
which runs the tests for structured and customer logger to ci.
Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25179>
This commit introduces the CustomLogger class, which provides methods
for updating, creating, and managing dut jobs and phases in a structured
log in below json format.
{
"_timestamp": "2023-10-05T06:16:42.603921",
"dut_job_type": "rpi3",
"farm": "igalia",
"dut_jobs": [
{
"status": "pass",
"submitter_start_time": "2023-10-05T06:16:42.745862",
"dut_start_time": "2023-10-05T06:16:42.819964",
"dut_submit_time": "2023-10-05T06:16:45.866096",
"dut_end_time": "2023-10-05T06:24:13.533394",
"dut_name": "igalia-ci01-rpi3-1gb",
"dut_state": "finished",
"dut_job_phases": [
{
"name": "boot",
"start_time": "2023-10-05T06:16:45.865863",
"end_time": "2023-10-05T06:17:14.801002"
},
{
"name": "test",
"start_time": "2023-10-05T06:17:14.801009",
"end_time": "2023-10-05T06:24:13.610296"
}
],
"submitter_end_time": "2023-10-05T06:24:13.680729"
}
],
"job_combined_status": "pass",
"dut_attempt_counter": 1
}
This class uses the existing StructuredLogger module,
which provides a robust and flexible logging utility supporting multiple
formats. It also includes a command-line tool for updating, creating,
and modifying dut jobs and phases through command-line arguments.
This can be used in ci job scripts to update information in structured logs.
Unit tests also have been added for the new class.
Currently, only LAVA jobs create structured log files, and this will be
extended for other jobs using this tool.
Signed-off-by: Vignesh Raman <vignesh.raman@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25179>
Reading the local root config file and then asking gitlab to evaluate it
in the context of some other version will cause issues if they are not
identical.
Instead, the local document should be a simple include of whatever is
the root config file at that commit.
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/26074>
We already print all the detected target jobs from regex and its
dependencies. But for more complex regexes the list can be cumbersome,
and an aggregate list of dependencies and targets can be more value, so
add these prints as well.
This is what looks like:
```
Running 10 dependency jobs:
alpine/x86_64_lava_ssh_client, clang-format, debian-arm64,
debian-testing, debian/arm64_build, debian/x86_64_build,
debian/x86_64_build-base, kernel+rootfs_arm64, kernel+rootfs_x86_64,
rustfmt
Running 15 target jobs:
a618_gl 1/4, a660_gl 1/2, intel-tgl-skqp, iris-amly-egl, iris-apl-deqp
1/3, iris-cml-deqp 1/4, iris-glk-deqp 1/2, iris-kbl-deqp 1/3,
lima-mali450-deqp:arm64, lima-mali450-piglit:arm64 1/2,
panfrost-g52-gl:arm64 1/3, panfrost-g72-gl:arm64 1/3,
panfrost-t720-gles2:arm64, panfrost-t860-egl:arm64, zink-anv-tgl
```
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25940>
Modify the GraphQL query used to fetch all jobs within a pipeline,
transitioning from fetching data via stage nodes to a direct job
retrieval approach.
The prior method was not paginated, potentially overloading the server
and complicating result parsing due to the structure of stage nodes. The
new approach simplifies data interpretation and handles job lists
exceeding 100 elements by implementing pagination with helper functions
to concatenate paginated results.
- Transitioned from extracting jobs from stage nodes to a direct query
for all jobs in the pipeline, improving data readability and server
performance.
- With the enhanced data clarity from the updated query, removed the
Dag+JobMetadata tuple as it's now redundant. The refined query
provides a more comprehensive job data including job name, stage, and
dependencies.
- The previous graph query relied on a graph node that will (or should)
be paginated anyway.
Closes: #10050
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25940>
Make query support pagination by supplying the paginated key.
In the following toy example, the paginated key is:
["levels", "cars"]
```graphql
query vehicle_store($location: ID!) {
levels {
cars {
pageInfo {
hasNextPage
endCursor
}
...
}
}
}
```
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25940>
Make the caching optional, as it can fail in some complex queries where
the transport middleware uses non-pickle-able objects.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25940>
For some reason AIOHTTPTransport started to use MultiDict after doing
some adjustments in the GraphQL query, which made `filecache` fail
because MultiDict object are not pickle-able.
Changing the transport strategy from AIOHTTPTransport to
RequestsHTTPTransport, which dropped one requirement. We aren't doing
async anyway, all the calls were sync before.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25940>
The function is getting too big, let's add comments, docstrings to the
most important function, new type hints and extract methods from it to
make it easier to read.
Signed-off-by: Guilherme Gallo <guilherme.gallo@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25858>
When running multiple targets, the script was returning if any of the
targets failed.
But sometimes some of the targets are still running and we want to
monitor it until it finishes, even if some of the targets failed.
So check if any targeted jobs are in pending or running before deciding
to finish.
Signed-off-by: Helen Koike <helen.koike@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25826>
Now you don't fail if you're trying to test a mesa/mesa MR pipeline and
gitlab takes more than 10s to create it. And you don't have to wait 10
seconds to get things started (aka see if your regex was right) if you're
testing a user/mesa fork pipeline.
Fixes: 941d92408e ("bin/ci_run_n_monitor: automatically pick MR pipelines when they exist")
Closes: #9894
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25810>
Show currently assigned jobs to Marge and return 0 when it's free.
Useful for combination with ci_run_n_monitor.py .
Signed-off-by: David Heidelberg <david.heidelberg@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/20347>
Allow to monitor pipelines in any <user>/<project> on gitlab.fdo.
If developers want to monitor MR pipelines, they can run with
--project mesa/mesa to force it to be picked from mesa username.
Signed-off-by: Helen Koike <helen.koike@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25473>
leave to enable_job() and cancel_job() to do the right thing according
to the job status.
let target canceled jobs to be re-runned.
Signed-off-by: Helen Koike <helen.koike@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25473>
Python 3.8 is 2 years old by now, I think it's time to allow using its
features, especially for a script that's only used by the release
maintainers.
Signed-off-by: Eric Engestrom <eric@engestrom.ch>
Cc: mesa-stable
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/13664>
Now that we have drm-ci, add --project, so the script can also be used
to linux (and any other projects).
Let the default to "mesa" so it can keep behaving as before when the
option is not given.
Signed-off-by: Helen Koike <helen.koike@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24912>
This isolates the script environment from the rest of the machine,
avoiding missing/incompatible dependencies and avoiding polluting the
rest of the machine with python packages.
Signed-off-by: Eric Engestrom <eric@igalia.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/24367>
