ci/deqp-runner: Add a post-deqp-run filter list for known flakes.

So far, we've been putting our known flakes that intermittently fail CI
into the skips list.  This has two downsides:

1) You don't know when the flakes stop happening and when to delist them
   from skips, unless you go do a bunch of manual runs with the skips list
   cleared.

2) If the flake was because the previous test left some broken state in
   the HW, you may just move your intermittent to a new test.

With this new path, you can list your flakes in the flakes file to keep
them from erroring out people's pipelines.  They still get run and
reported as is.

Reviewed-by: Christian Gmeiner <christian.gmeiner@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/6392>
This commit is contained in:
Eric Anholt 2020-08-19 10:30:12 -07:00 committed by Marge Bot
parent 4bb59fcee9
commit 30da82c74c
3 changed files with 23 additions and 2 deletions

View File

@ -31,6 +31,7 @@ for var in \
DEQP_EXPECTED_FAILS \
DEQP_EXPECTED_RENDERER \
DEQP_NO_SAVE_RESULTS \
DEQP_FLAKES \
DEQP_PARALLEL \
DEQP_RUN_SUFFIX \
DEQP_SKIPS \

View File

@ -291,6 +291,27 @@ if [ $DEQP_EXITCODE -ne 0 ]; then
cp /tmp/deqp_runner.*.txt $RESULTS/
egrep -v ",Pass|,Skip|,ExpectedFail" $RESULTSFILE > $UNEXPECTED_RESULTSFILE
# deqp-runner's flake detection won't perfectly detect all flakes, so
# allow the driver to list some known flakes that won't intermittently
# fail people's pipelines (while still allowing them to run and be
# reported to IRC in the usual flake detection path). If we had some
# fails listed (so this wasn't a total runner failure), then filter out
# the known flakes and see if there are any issues left.
if [ -n "$DEQP_FLAKES" -a -s $UNEXPECTED_RESULTSFILE ]; then
set +x
while read line; do
line=`echo $line | sed 's|#.*||g'`
if [ -n "$line" ]; then
sed -i "/$line/d" $UNEXPECTED_RESULTSFILE
fi
done < $INSTALL/$DEQP_FLAKES
set -x
if [ ! -s $UNEXPECTED_RESULTSFILE ]; then
exit 0
fi
fi
if [ -z "$DEQP_NO_SAVE_RESULTS" ]; then
echo "Some unexpected results found (see cts-runner-results.txt in artifacts for full results):"
head -n 50 $UNEXPECTED_RESULTSFILE

View File

@ -34,8 +34,7 @@ cp -Rp .gitlab-ci/fossils.yml install/
cp -Rp .gitlab-ci/fossils install/
cp -Rp .gitlab-ci/fossilize-runner.sh install/
cp -Rp .gitlab-ci/deqp-runner.sh install/
cp -Rp .gitlab-ci/deqp-*-fails.txt install/
cp -Rp .gitlab-ci/deqp-*-skips.txt install/
cp -Rp .gitlab-ci/deqp-*.txt install/
# Tar up the install dir so that symlinks and hardlinks aren't each
# packed separately in the zip file.