This is an initial attempt to introduce some mechanisms to identify
racy testcases present in our testsuite. As can be seen in previous
discussions, racy tests are really bothersome and cause our BuildBot
to pollute the gdb-testers mailing list with hundreds of
false-positives messages every month. Hopefully, identifying these
racy tests in advance (and automatically) will contribute to the
reduction of noise traffic to gdb-testers, maybe to the point where we
will be able to send the failure messages directly to the authors of
the commits.
I spent some time trying to decide the best way to tackle this
problem, and decided that there is no silver bullet. Racy tests are
tricky and it is difficult to catch them, so the best solution I could
find (for now?) is to run our testsuite a number of times in a row,
and then compare the results (i.e., the gdb.sum files generated during
each run). The more times you run the tests, the more racy tests you
are likely to detect (at the expense of waiting longer and longer).
You can also run the tests in parallel, which makes things faster (and
contribute to catching more racy tests, because your machine will have
less resources for each test and some of them are likely to fail when
this happens). I did some tests in my machine (8-core i7, 16GB RAM),
and running the whole GDB testsuite 5 times using -j6 took 23 minutes.
Not bad.
In order to run the racy test machinery, you need to specify the
RACY_ITER environment variable. You will assign a number to this
variable, which represents the number of times you want to run the
tests. So, for example, if you want to run the whole testsuite 3
times in parallel (using 2 cores), you will do:
make check RACY_ITER=3 -j2
It is also possible to use the TESTS variable and specify which tests
you want to run:
make check TEST='gdb.base/default.exp' RACY_ITER=3 -j2
And so on. The output files will be put at the directory
gdb/testsuite/racy_outputs/.
After make invokes the necessary rules to run the tests, it finally
runs a Python script that will analyze the resulting gdb.sum files.
This Python script will read each file, and construct a series of sets
based on the results of the tests (one set for FAIL's, one for
PASS'es, one for KFAIL's, etc.). It will then do some set operations
and come up with a list of unique, sorted testcases that are racy.
The algorithm behind this is:
for state in PASS, FAIL, XFAIL, XPASS...; do
if a test's state in every sumfile is $state; then
it is not racy
else
it is racy
(The algorithm is actually a bit more complex than that, because it
takes into account other things in order to decide whether the test
should be ignored or not).
IOW, a test must have the same state in every sumfile.
After processing everything, the script prints the racy tests it could
identify on stdout. I am redirecting this to a file named racy.sum.
Something else that I wasn't sure how to deal with was non-unique
messages in our testsuite. I decided to do the same thing I do in our
BuildBot: include a unique identifier in the end of message, like:
gdb.base/xyz.exp: non-unique message
gdb.base/xyz.exp: non-unique message <<2>>
This means that you will have to be careful about them when you use
the racy.sum file.
I ran the script several times here, and it did a good job catching
some well-known racy tests. Overall, I am satisfied with this
approach and I think it will be helpful to have it upstream'ed. I
also intend to extend our BuildBot and create new, specialized
builders that will be responsible for detecting the racy tests every X
number of days.
2016-03-05 Sergio Durigan Junior <sergiodj@redhat.com>
* Makefile.in (DEFAULT_RACY_ITER): New variable.
(CHECK_TARGET_TMP): Likewise.
(check-single-racy): New rule.
(check-parallel-racy): Likewise.
(TEST_TARGETS): Adjust rule to account for RACY_ITER.
(do-check-parallel-racy): New rule.
(check-racy/%.exp): Likewise.
* README (Racy testcases): New section.
* analyze-racy-logs.py: New file.
Currently, you can cd to the gdb/testsuite/ dir and use
make check-parallel, instead of using FORCE_PARALLEL:
$ make -j8 check-parallel RUNTESTFLAGS="--target_board=native-gdbserver"
$ make -j8 check RUNTESTFLAGS="--target_board=native-gdbserver" FORCE_PARALLEL=1
But you can't do that in the build/gdb/ dir:
$ make check-parallel RUNTESTFLAGS="--target_board=native-gdbserver"
make: *** No rule to make target `check-parallel'. Stop.
I find check-parallel a bit more convenient, and more typo-proof, so
this patch makes it work from the gdb build dir too.
While documenting this in testsuite/README, I found that the parallel
testing mode would better be pulled out to its own section and
extended.
gdb/ChangeLog:
2016-02-11 Pedro Alves <palves@redhat.com>
* Makefile.in (check-parallel): New rule.
gdb/testsuite/ChangeLog:
2016-02-11 Pedro Alves <palves@redhat.com>
* README (Parallel testing): New section.
(GDB_PARALLEL): Rewrite.
(FORCE_PARALLEL): Document.
I couldn't find a test that verified the thread name functionality, so I
created a new one.
A target board can define gdb,no_thread_names if it doesn't support thread
names and wants to skip the tests that uses them.
This test has been made with Linux in mind. Not all platforms use
pthread_setname_np to set the thread name, but some #ifdefs can be added
later in order to support other platforms.
Tested on x86-64 Ubuntu 14.04, native and remote.
gdb/testsuite/ChangeLog:
* gdb.threads/names.exp: New file.
* gdb.threads/names.c: New file.
* README: Mention gdb,no_thread_names.
This integrates Jan Kratochvil's nice race reproducer from PR
testsuite/12649 into the testsuite infrustructure directly.
With this, one only has to do either 'make check-read1' or 'make check
READ1="1"' to preload the read1.so library into expect.
Currently only enabled for glibc/GNU systems, and if
build==host==target.
gdb/testsuite/ChangeLog:
* Makefile.in (EXTRA_RULES, CC): New variables, get from
configure.
(EXPECT): Handle READ1 being set.
(all): Depend on EXTRA_RULES.
(check-read1, expect-read1, read1.so, read1): New rules.
* README (Testsuite Parameters): Document the READ1 make variable.
(Race detection): New section.
* configure: Regenerate.
* configure.ac: If build==host==target, and running under a
GNU/glibc system, add read1 to the extra Makefile rules.
(EXTRA_RULES): AC_SUBST it.
* lib/read1.c: New file.
gdb/ChangeLog:
* Makefile.in (check-read1): New rule.
Right now we provide a board info entry, `gdb_init_command', that allows
one to send a single command to GDB before the program to be debugged is
started. This is useful e.g. for slow remote targets to change the
default "remotetimeout" setting. Occasionally I found a need to send
multiple commands instead, however this cannot be achieved with
`gdb_init_command'.
This change therefore extends the mechanism by adding a TCL list of GDB
commands to send, via a board info entry called `gdb_init_commands'.
There is no limit as to the number of commands put there. The old
`gdb_init_command' mechanism remains supported for compatibility with
existing people's environments.
* lib/gdb-utils.exp: New file.
* lib/gdb.exp (gdb_run_cmd): Call gdb_init_commands, replacing
inline `gdb_init_command' processing.
(gdb_start_cmd): Likewise.
* lib/mi-support.exp (mi_run_cmd): Likewise.
* README: Document `gdb_init_command' and `gdb_init_commands'.
Hi,
This patch is to add a new board setting gdb_reverse_timeout, which is
used to set timeout for all gdb.reverse test cases, which are usually
very slow and cause some TIMEOUT failures, for example, on some arm
boards. We have some alternatives to this approach, but I am not
satisfied with them:
- Increase the timeout value. This is the global change, and it may
cause some delay where actual failures happen.
- Set timeout by gdb_reverse_timeout in every gdb.reverse/*.exp.
Then, we have to touch every file under gdb.reverse.
In this patch, we choose a central place to set timeout for all tests
in gdb.reverse, which is convenient.
gdb/testsuite:
2014-05-20 Yao Qi <yao@codesourcery.com>
* lib/gdb.exp (gdb_init): Set timeout if test file is under
gdb.reverse directory and gdb_reverse_timeout exists in board
setting.
* README: Document gdb_reverse_timeout.
* Makefile.in (TESTS): New variable.
(expanded_tests, expanded_tests_or_none): New variables
(check-single): Pass $(expanded_tests_or_none) to runtest.
(check-parallel): Only run tests in $(TESTS) if non-empty.
(check/no-matching-tests-found): New rule.
* README: Document TESTS makefile variable.
Running catch-syscall.exp against a gdbserver that actually supports
it, we get:
FAIL: gdb.base/catch-syscall.exp: continue until exit (the program exited)
FAIL: gdb.base/catch-syscall.exp: continue until exit (the program exited)
FAIL: gdb.base/catch-syscall.exp: continue until exit (the program exited)
FAIL: gdb.base/catch-syscall.exp: continue until exit at catch syscall with unused syscall (mlock) (the program exited)
FAIL: gdb.base/catch-syscall.exp: continue until exit (the program exited)
The fail pattern is:
Catchpoint 2 (call to syscall exit_group), 0x000000323d4baa29 in _exit () from /lib64/libc.so.6
(gdb) PASS: gdb.base/catch-syscall.exp: program has called exit_group
delete breakpoints
Delete all breakpoints? (y or n) y
(gdb) info breakpoints
No breakpoints or watchpoints.
(gdb) break exit
Breakpoint 3 at 0x323d438bf0
(gdb) continue
Continuing.
[Inferior 1 (process 21081) exited normally]
That "break exit" + "continue" comes from:
> # gdb_continue_to_end:
> # The case where the target uses stubs has to be handled specially. If a
> # stub is used, we set a breakpoint at exit because we cannot rely on
> # exit() behavior of a remote target.
> #
The native-gdbserver.exp board, used to test against gdbserver in
"target remote" mode, triggers that case ($use_gdb_stub is true). So
gdb_continue_to_end doesn't work for catch-syscall.exp as here we
catch the exit_group and continue from that, expecting to see a real
program exit. I was about to post a patch that changes
catch-syscall.exp to call a new function that just always does what
gdb_continue_to_end does in the !$use_gdb_stub case. But, since
GDBserver doesn't really need this, in the end I thought it better to
teach the testsuite that there are stubs that know how to report
program exits, by adding a new "exit_is_reliable" board variable that
then gdb_continue_to_end checks.
Tested on x86_64 Fedora 17, native and gdbserver.
gdb/testsuite/
2013-10-02 Pedro Alves <palves@redhat.com>
* README (Board Settings): Document "exit_is_reliable".
* lib/gdb.exp (gdb_continue_to_end): Check whether the board says
running to exit reliably reports program exits.
* boards/native-gdbserver.exp: Set exit_is_reliable in the board
info.
* boards/native-stdio-gdbserver.exp: Likewise.
