cpython/Lib/test
Tim Peters 7ea39b135a New test "+sort", tacking 10 random floats on to the end of a sorted
array.  Our samplesort special-cases the snot out of this, running about
12x faster than *sort.  The experimental mergesort runs it about 8x
faster than *sort without special-casing, but should really do better
than that (when merging runs of different lengths, right now it only
does something clever about finding where the second run begins in
the first and where the first run ends in the second, and that's more
of a temp-memory optimization).
2002-07-21 17:37:03 +00:00
..
data New files which test the corners of multipart/message and 2002-07-09 02:36:10 +00:00
output Follow PyXML: Remove all prints from successful tests. This means we can 2002-07-19 22:16:41 +00:00
__init__.py Add __init__.py to test package. 1997-11-06 15:41:23 +00:00
audiotest.au The encoding type was wrong, I think. 1999-02-05 20:57:44 +00:00
autotest.py Rehabilitate autotest.py. 2000-08-23 05:28:45 +00:00
badsyntax_future3.py Fix compileall.py so that it fails on SyntaxErrors 2001-04-18 01:19:28 +00:00
badsyntax_future4.py Fix compileall.py so that it fails on SyntaxErrors 2001-04-18 01:19:28 +00:00
badsyntax_future5.py Fix compileall.py so that it fails on SyntaxErrors 2001-04-18 01:19:28 +00:00
badsyntax_future6.py Fix compileall.py so that it fails on SyntaxErrors 2001-04-18 01:19:28 +00:00
badsyntax_future7.py Fix compileall.py so that it fails on SyntaxErrors 2001-04-18 01:19:28 +00:00
badsyntax_nocaret.py Fix compileall.py so that it fails on SyntaxErrors 2001-04-18 01:19:28 +00:00
double_const.py This is a test showing SF bug 422177. It won't trigger until I check in 2001-05-08 03:58:01 +00:00
greyrgb.uue Use uuencoded test images. 1997-04-16 00:30:45 +00:00
pickletester.py Whitespace normalization. 2002-04-16 01:38:40 +00:00
pydocfodder.py SF bug [#472347] pydoc and properties. 2001-10-18 19:56:17 +00:00
pystone.py Make reindent.py happy (convert everything to 4-space indents!). 2000-10-23 17:22:08 +00:00
re_tests.py bug #133283, #477728, #483789, #490573 2001-12-09 16:13:15 +00:00
README Rewrote the PyUnit description so that it now recommends to use 2002-04-23 21:39:00 +00:00
regex_tests.py Make reindent.py happy (convert everything to 4-space indents!). 2000-10-23 17:22:08 +00:00
regrtest.py Add missing comma. 2002-07-17 15:56:55 +00:00
reperf.py Measure performance of sub(), split(), findall(). 1998-07-17 21:10:42 +00:00
sortperf.py New test "+sort", tacking 10 random floats on to the end of a sorted 2002-07-21 17:37:03 +00:00
string_tests.py SF bug # 493951 string.{starts,ends}with vs slices 2002-06-14 00:50:42 +00:00
test___all__.py Tighten up some warning filters, and break some dependencies on the 2002-04-16 01:27:44 +00:00
test___future__.py Expose the CO_xxx flags via the "new" module (re-solving a problem "the 2001-08-18 20:18:49 +00:00
test_al.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_anydbm.py initial test_anydbm module - based on test_dumbdbm.py 2002-03-18 03:00:37 +00:00
test_array.py Fix the bug described in 2002-06-19 15:44:15 +00:00
test_asynchat.py Set the SO_REUSEADDR socket option in the server thread -- this seems 2001-04-15 00:42:13 +00:00
test_atexit.py Use sys.executable to run Python, as suggested by Neal Norwitz. 2002-07-17 00:34:26 +00:00
test_audioop.py The first batch of changes recommended by the fixdiv tool. These are 2001-09-04 19:14:14 +00:00
test_augassign.py Fiddle test_augassign so it passes under -Qnew. 2001-12-11 19:20:15 +00:00
test_b1.py Whitespace normalization. 2002-07-16 21:35:23 +00:00
test_b2.py Skip Montanaro's patch, SF 559833, exposing xrange type in builtins. 2002-06-05 23:12:45 +00:00
test_base64.py Whitespace normalization. 2002-05-23 15:15:30 +00:00
test_bastion.py add test of bastion and rexec to std regression test suite 2001-01-19 03:22:48 +00:00
test_binascii.py Whitespace normalization. 2002-05-23 15:15:30 +00:00
test_binhex.py Change the PyUnit-based tests to use the test_main() approach. This 2001-09-20 21:33:42 +00:00
test_binop.py Change the PyUnit-based tests to use the test_main() approach. This 2001-09-20 21:33:42 +00:00
test_bisect.py Fred, THIS NEEDS DOCS! The function docstrings tell the tale. 2000-12-29 02:06:45 +00:00
test_bool.py This is my nearly two year old patch 2002-06-11 10:55:12 +00:00
test_bsddb.py SF patch [ 545523 ] patch for 514433 bsddb.dbopen (NULL) 2002-04-23 02:11:05 +00:00
test_bufio.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_builtin.py Initial revision 1992-01-27 17:00:37 +00:00
test_calendar.py SF bug 533234: tm_isdst > 1 Passed to strftime. 2002-03-23 03:26:53 +00:00
test_call.py Change the PyUnit-based tests to use the test_main() approach. This 2001-09-20 21:33:42 +00:00
test_capi.py Fix typo in comment (the module is now called _testcapi, not _test). 2001-04-13 17:03:04 +00:00
test_cd.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_cfgparser.py Whitespace normalization. 2002-04-16 01:38:40 +00:00
test_cgi.py This patch removes all uses of "assert" in the regression test suite 2001-01-17 19:11:13 +00:00
test_charmapcodec.py Committing and closing SF patch #441348 to help Jython to pass this test. 2001-08-23 18:57:01 +00:00
test_cl.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_class.py Fix for SF bug 532646. This is a little simpler than what Neal 2002-06-13 21:32:51 +00:00
test_cmath.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_codecs.py Change the PyUnit-based tests to use the test_main() approach. This 2001-09-20 21:33:42 +00:00
test_codeop.py Change the PyUnit-based tests to use the test_main() approach. This 2001-09-20 21:33:42 +00:00
test_coercion.py Tighten up some warning filters, and break some dependencies on the 2002-04-16 01:27:44 +00:00
test_commands.py Patch #551911: Escape . properly. 2002-06-06 09:52:49 +00:00
test_compare.py The first batch of changes recommended by the fixdiv tool. These are 2001-09-04 19:14:14 +00:00
test_compile.py Fix SF Bug 564931: compile() traceback must include filename. 2002-07-09 09:23:27 +00:00
test_complex.py test_support: add a docstring to vereq(). 2001-12-29 17:34:57 +00:00
test_contains.py Patch #445762: Support --disable-unicode 2001-08-17 18:39:25 +00:00
test_cookie.py Teach regrtest how to pass on doctest failure msgs. This is done via a 2001-09-09 06:12:01 +00:00
test_copy_reg.py Change the PyUnit-based tests to use the test_main() approach. This 2001-09-20 21:33:42 +00:00
test_cpickle.py Merge of the release22 branch changes back into the trunk. 2001-12-21 20:04:22 +00:00
test_crypt.py Whitespace normalization. Leaving tokenize_tests.py alone for now. 2001-01-18 02:22:22 +00:00
test_curses.py Patch #473187: Add a test script that exercises most of the functions in 2001-10-22 15:26:09 +00:00
test_dbm.py Check that f.keys() == [] right after creation -- this prevents bugs 2001-03-22 00:40:23 +00:00
test_descr.py subtype_resurrection(): Removed unused import. 2002-07-11 18:39:56 +00:00
test_descrtut.py SF 560736. Optimize list iteration by filling the tp_iter slot. 2002-05-31 21:40:38 +00:00
test_difflib.py Teach regrtest how to pass on doctest failure msgs. This is done via a 2001-09-09 06:12:01 +00:00
test_dircache.py Change the PyUnit-based tests to use the test_main() approach. This 2001-09-20 21:33:42 +00:00
test_dl.py Patch #404680: disables the nis module and enables the dl module when 2001-02-27 20:54:23 +00:00
test_doctest2.py SF bug [#467331] ClassType.__doc__ always None. 2001-10-04 05:27:00 +00:00
test_doctest.py Whitespace normalization. 2001-09-18 02:26:39 +00:00
test_dospath.py Change the PyUnit-based tests to use the test_main() approach. This 2001-09-20 21:33:42 +00:00
test_dumbdbm.py restructure a bit to not rely on test case execution ordering 2002-03-17 23:03:42 +00:00
test_email_codecs.py Shut the test up and add a missing import 2002-07-19 22:44:23 +00:00
test_email.py Shut the test up and add a missing import 2002-07-19 22:44:23 +00:00
test_enumerate.py Removed the generator future-stmt -- not needed for 2.3. 2002-06-20 14:52:37 +00:00
test_errno.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_exceptions.py Tighten up some warning filters, and break some dependencies on the 2002-04-16 01:27:44 +00:00
test_extcall.py Quick and dirty fix for test_extcall failures trigged by Guido's 2001-08-24 19:11:57 +00:00
test_fcntl.py Patch #488073: AtheOS port. 2002-06-11 06:22:31 +00:00
test_file.py Patch #488073: AtheOS port. 2002-06-11 06:22:31 +00:00
test_fileinput.py SF patch #440170: Tests for fileinput module. 2001-07-11 22:21:17 +00:00
test_fnmatch.py Change the PyUnit-based tests to use the test_main() approach. This 2001-09-20 21:33:42 +00:00
test_fork1.py Unixware 7 support by Billy G. Allie (SF patch 413011) 2001-04-11 20:58:20 +00:00
test_format.py Use raw-unicode-escape for the tests that require it. 2001-08-17 22:08:34 +00:00
test_fpformat.py Change the PyUnit-based tests to use the test_main() approach. This 2001-09-20 21:33:42 +00:00
test_frozen.py Fix the frozen bytecode for __hello__ (betcha didn't know that existed 2001-10-18 18:49:37 +00:00
test_funcattrs.py Fix for SF bug #472940: can't getattr() attribute shown by dir() 2001-10-22 02:00:09 +00:00
test_future1.py Import the nested_scopes feature twice, to exercise the patch introduced 2001-03-10 02:18:47 +00:00
test_future2.py Add regression test for future statements. This adds eight files, but 2001-02-28 17:48:06 +00:00
test_future3.py Add test case to cover multiple future statements on separate lines of 2001-08-20 21:18:56 +00:00
test_future.py Fix SF bug [ #450245 ] Error in parsing future stmts 2001-08-20 20:33:42 +00:00
test_gc.py test_trashcan() and supporting class Ouch(): Jeremy noted that this test 2002-07-11 19:07:45 +00:00
test_gdbm.py Check that f.keys() == [] right after creation -- this prevents bugs 2001-03-22 00:40:23 +00:00
test_generators.py Whitespace normalization. 2002-07-16 21:35:23 +00:00
test_getargs.py Guess what? --disable-unicode proofing. 2002-05-20 14:54:17 +00:00
test_getopt.py Patch 473512: add GNU style scanning as gnu_getopt. 2002-06-06 10:58:36 +00:00
test_gettext.py Move the setting of os.environ['LANGUAGE'] to setup(), and reset it to 2002-07-20 00:36:38 +00:00
test_gl.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_glob.py Change the PyUnit-based tests to use the test_main() approach. This 2001-09-20 21:33:42 +00:00
test_global.py test_global was broken by some recent checkin. Repairing. 2001-03-02 01:48:16 +00:00
test_grammar.py The initial patch #468662 was not applied quite verbatim. This should one 2001-12-09 09:12:34 +00:00
test_grp.py Change the PyUnit-based tests to use the test_main() approach. This 2001-09-20 21:33:42 +00:00
test_gzip.py force gzip module to open files using 'b'inary mode. 2002-05-23 01:43:05 +00:00
test_hash.py Change the PyUnit-based tests to use the test_main() approach. This 2001-09-20 21:33:42 +00:00
test_hmac.py Use attributes appropriately 2002-04-01 19:00:50 +00:00
test_hotshot.py Gave hotshot.LogReader a close() method, to allow users to close the 2002-07-18 14:54:28 +00:00
test_htmllib.py Non-failing test for SF bug #467059. 2001-10-04 20:05:10 +00:00
test_htmlparser.py Add a test that makes sure unclosed entity references are handled consitently. 2001-12-03 16:44:09 +00:00
test_httplib.py Whitespace normalization. 2002-07-16 21:35:23 +00:00
test_imageop.py SF patch #474590 -- RISC OS support 2001-10-24 20:42:55 +00:00
test_imaplib.py Alter text test arg to obey new rule, also include inverse test to make time-zone independant 2002-06-17 07:06:24 +00:00
test_imgfile.py joinfields -> join. 2001-02-09 17:05:53 +00:00
test_import.py Got rid of special case for Macintosh realloc slowdown: Tim fixed the problem. 2002-07-08 10:07:25 +00:00
test_inspect.py Remove obsolete __dynamic__ distinction. 2001-10-15 22:53:29 +00:00
test_isinstance.py Whitespace normalization. 2002-05-23 15:15:30 +00:00
test_iter.py Bunch of tests to make sure that StopIteration is a sink state. 2002-07-16 21:48:11 +00:00
test_largefile.py Change Windows file.truncate() to (a) restore the original file position, 2002-03-12 03:04:44 +00:00
test_linuxaudiodev.py don't fail if the audio device is busy, just skip. 2002-04-23 02:20:46 +00:00
test_locale.py Whitespace normalization. 2001-05-02 05:54:44 +00:00
test_long_future.py Fix typo in error reporting. This doesn't need to go into the release 2001-09-07 00:47:00 +00:00
test_long.py Format strings (tuples,) appropriately 2002-04-01 19:01:39 +00:00
test_longexp.py Tim_one's change to aggressively overallocate nodes when adding child 2002-07-15 12:03:19 +00:00
test_mailbox.py SF patch #474590 -- RISC OS support 2001-10-24 20:42:55 +00:00
test_marshal.py marshal.c r_long64: When reading a TYPE_INT64 value on a box with 32-bit 2001-08-29 02:28:42 +00:00
test_math.py Whitespace normalization. 2002-05-23 15:15:30 +00:00
test_md5.py Update the code to better reflect recommended style: 2000-12-12 23:11:42 +00:00
test_mhlib.py Patch #488073: AtheOS port. 2002-06-11 06:22:31 +00:00
test_mimetools.py Use string.ascii_letters instead of string.letters (SF bug #226706). 2001-07-20 19:05:50 +00:00
test_mimetypes.py Whitespace normalization. 2001-10-30 05:56:40 +00:00
test_MimeWriter.py Make reindent.py happy (convert everything to 4-space indents!). 2000-10-23 17:22:08 +00:00
test_minidom.py Follow PyXML: Remove all prints from successful tests. This means we can 2002-07-19 22:16:41 +00:00
test_mmap.py test_mmap started breaking on Windows, only when run after test_bsddb. 2002-04-23 23:07:28 +00:00
test_module.py Repair the test (adding a docstring to the module type changed the 2002-06-04 06:06:54 +00:00
test_mpz.py Whitespace normalization. 2002-04-16 01:38:40 +00:00
test_mutants.py lookdict: stop more insane core-dump mutating comparison cases. Should 2001-06-03 04:54:32 +00:00
test_netrc.py Whitespace normalization. 2002-04-16 01:38:40 +00:00
test_new.py Extend function() to support an optional closure argument. 2002-07-11 18:30:27 +00:00
test_nis.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_ntpath.py Only test ntpath.abspath() on Windows. This allows the rest of the module to 2002-01-15 03:46:43 +00:00
test_opcodes.py The one thing I love more then writing code is deleting code. 2001-01-29 06:21:17 +00:00
test_openpty.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_operations.py BadDictKey test: The output file expected "raising error" to be printed 2001-05-29 21:14:32 +00:00
test_operator.py Change the PyUnit-based tests to use the test_main() approach. This 2001-09-20 21:33:42 +00:00
test_os.py Patch #488073: AtheOS port. 2002-06-11 06:22:31 +00:00
test_parser.py Remove some now-obsolete generator future statements. 2002-04-01 00:28:59 +00:00
test_pep247.py Whitespace normalization. 2001-11-13 21:51:26 +00:00
test_pickle.py Fix the test control support for the pickle & cPickle tests so the tests run 2001-12-19 16:42:15 +00:00
test_pkg.py SF patch #474590 -- RISC OS support 2001-10-24 20:42:55 +00:00
test_pkgimport.py SF patch #474590 -- RISC OS support 2001-10-24 20:42:55 +00:00
test_poll.py This patch removes all uses of "assert" in the regression test suite 2001-01-17 19:11:13 +00:00
test_popen2.py Patch #488073: AtheOS port. 2002-06-11 06:22:31 +00:00
test_popen.py Get popen test to work even if python is not in the path 2002-07-20 20:35:13 +00:00
test_posixpath.py String method conversion. 2001-02-09 12:00:47 +00:00
test_pow.py Format strings (tuples,) appropriately 2002-04-01 19:01:39 +00:00
test_pprint.py Add an experimental mechanism to support extending the pprint formatting. 2002-04-02 05:08:35 +00:00
test_profile.py Whitespace normalization. 2001-10-04 05:36:56 +00:00
test_profilehooks.py Remove some now-obsolete generator future statements. 2002-04-01 00:28:59 +00:00
test_pty.py Fix the second reincarnation of SF #456395 -- failure on IRIX. This 2001-09-11 14:24:35 +00:00
test_pwd.py String method conversion. 2001-02-09 12:00:47 +00:00
test_pyclbr.py Remove httplib from tested modules. 2002-07-12 15:54:37 +00:00
test_pyexpat.py Added character data buffering to pyexpat parser objects. 2002-06-28 22:56:48 +00:00
test_queue.py Fix bug 544473 - "Queue module can deadlock". 2002-04-19 00:11:32 +00:00
test_quopri.py Patch #462190, patch #464070: Support quoted printable in the binascii module. 2001-09-30 20:32:11 +00:00
test_random.py random.gauss() uses a piece of hidden state used by nothing else, 2002-05-05 20:40:00 +00:00
test_re.py Check in a testcase for SF bug #449000: re.sub(r'\n', ...) broke. 2001-08-10 14:52:48 +00:00
test_regex.py Tighten up some warning filters, and break some dependencies on the 2002-04-16 01:27:44 +00:00
test_repr.py Remove all tests that rely on deprecated-in-2.2 features of xrange objects. 2002-05-02 18:40:31 +00:00
test_resource.py Add tests for the recent resource module change. 2002-04-23 20:21:22 +00:00
test_rfc822.py SF bug 558179. 2002-06-05 19:07:39 +00:00
test_rgbimg.py SF patch #474590 -- RISC OS support 2001-10-24 20:42:55 +00:00
test_richcmp.py SF patch #421922: Implement rich comparison for dicts. 2001-05-08 04:38:29 +00:00
test_robotparser.py Whitespace normalization. 2002-04-16 01:38:40 +00:00
test_rotor.py Update the code to better reflect recommended style: 2000-12-12 23:11:42 +00:00
test_sax.py SF patch #474590 -- RISC OS support 2001-10-24 20:42:55 +00:00
test_scope.py Add test for eval() w/ free variables. 2002-04-20 04:51:39 +00:00
test_select.py Test exceptional condition in select() 2002-06-13 22:23:47 +00:00
test_sgmllib.py Added several new tests to check the behavior with respect to doctype 2001-09-24 20:22:09 +00:00
test_sha.py Change the PyUnit-based tests to use the test_main() approach. This 2001-09-20 21:33:42 +00:00
test_signal.py Whitespace normalization. 2002-07-16 21:35:23 +00:00
test_slice.py A few days ago, Guido said (in the thread "[Python-Dev] Python 2002-07-19 15:47:06 +00:00
test_socket_ssl.py Whitespace normalization. 2001-10-18 21:57:37 +00:00
test_socket.py Remove a few lines that aren't used and cause problems on platforms 2002-07-19 19:23:54 +00:00
test_socketserver.py Use sendall() in the stream test instead of send(). 2001-10-29 07:18:02 +00:00
test_softspace.py SF bug 480215: softspace confused in nested print 2002-03-24 19:25:00 +00:00
test_sre.py bug #133283, #477728, #483789, #490573 2001-12-09 16:13:15 +00:00
test_strftime.py The first batch of changes recommended by the fixdiv tool. These are 2001-09-04 19:14:14 +00:00
test_string.py Whitespace normalization. 2002-05-23 15:15:30 +00:00
test_StringIO.py Make StringIO work in --disable-unicode builds... 2002-05-13 09:42:16 +00:00
test_strop.py Tighten up some warning filters, and break some dependencies on the 2002-04-16 01:27:44 +00:00
test_strptime.py Pure Python strptime implementation by Brett Cannon. See SF patch 474274. 2002-07-19 17:04:46 +00:00
test_struct.py The 'p' (Pascal string) pack code acts unreasonably when the string size 2001-09-15 02:35:15 +00:00
test_structseq.py Test for 2002-03-06 17:18:15 +00:00
test_sunaudiodev.py If the sunaudiodev module is available but we cannot find an audio 2001-04-14 03:10:12 +00:00
test_sundry.py Tighten up some warning filters, and break some dependencies on the 2002-04-16 01:27:44 +00:00
test_support.py test_support: add a docstring to vereq(). 2001-12-29 17:34:57 +00:00
test_symtable.py Whitespace normalization. 2001-02-15 23:56:39 +00:00
test_syntax.py Fix SF bug [ 561825 ] Confusing error for "del f()" 2002-05-31 14:08:29 +00:00
test_tempfile.py SF bug #476138: tempfile behavior across platforms 2001-10-29 21:46:08 +00:00
test_thread.py SF bug #516372: test_thread: unhandled exc. in thread 2002-02-16 07:26:27 +00:00
test_threaded_import.py SF bug #516372: test_thread: unhandled exc. in thread 2002-02-16 07:26:27 +00:00
test_threadedtempfile.py Add "import thread" at the top of the module; this prevents us from 2001-04-14 14:35:43 +00:00
test_threading.py Test failed because these was no expected-output file, but always printed 2001-08-20 21:45:19 +00:00
test_time.py Pure Python strptime implementation by Brett Cannon. See SF patch 474274. 2002-07-19 17:04:46 +00:00
test_timeout.py Temporarily disable the timeout and socket tests. 2002-06-13 20:24:17 +00:00
test_timing.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_tokenize.py SF patch #474590 -- RISC OS support 2001-10-24 20:42:55 +00:00
test_traceback.py Change the PyUnit-based tests to use the test_main() approach. This 2001-09-20 21:33:42 +00:00
test_types.py Whitespace normalization. 2002-07-16 21:35:23 +00:00
test_ucn.py Move uchhash functionality into unicodedata (after the recent 2001-01-24 07:59:11 +00:00
test_unary.py tolerate --disable-unicode... 2002-05-20 14:24:53 +00:00
test_unicode_file.py Disable code intended for PEP 277. 2002-01-15 21:25:51 +00:00
test_unicode.py Whitespace normalization. 2002-05-23 15:15:30 +00:00
test_unicodedata.py This patch removes all uses of "assert" in the regression test suite 2001-01-17 19:11:13 +00:00
test_univnewlines.py Assorted code cleanups for readability. Greatly boosted the size of the 2002-04-21 06:12:02 +00:00
test_unpack.py Update the code to better reflect recommended style: 2000-12-12 23:11:42 +00:00
test_urllib2.py SF patch #474590 -- RISC OS support 2001-10-24 20:42:55 +00:00
test_urllib.py added several more urlencode test cases - part of patch 103391 2001-01-28 21:12:22 +00:00
test_urlparse.py Whitespace normalization. 2002-04-16 01:38:40 +00:00
test_userdict.py Add regression test for SF bug #476616 -- make sure copy of a derived class 2001-11-05 17:41:48 +00:00
test_userlist.py a bold attempt to fix things broken by MAL's verify patch: import 2001-01-17 21:51:36 +00:00
test_userstring.py I am mad. This test never worked! 2002-04-12 16:25:39 +00:00
test_uu.py Test that uu.py will not override an existing file if out_file isn't 2001-08-17 20:00:11 +00:00
test_wave.py Use binary mode to open "wave" files. 2000-12-19 06:32:57 +00:00
test_weakref.py SF patch 564549 (Erik Andersén). 2002-06-10 20:00:52 +00:00
test_winreg.py Patch #445762: Support --disable-unicode 2001-08-17 18:39:25 +00:00
test_winsound.py Make reindent.py happy (convert everything to 4-space indents!). 2000-10-23 17:22:08 +00:00
test_xmllib.py Tighten up some warning filters, and break some dependencies on the 2002-04-16 01:27:44 +00:00
test_xmlrpc.py added tests for long ints and ints where they are > 32 bits. 2001-10-19 16:06:52 +00:00
test_xreadline.py Whitespace normalization. Leaving tokenize_tests.py alone for now. 2001-01-18 02:22:22 +00:00
test_zipfile.py SF patch #474590 -- RISC OS support 2001-10-24 20:42:55 +00:00
test_zlib.py Fix SF #544995 (zlib crash on second flush call) 2002-04-19 14:37:07 +00:00
test.xml Added minidom tests. 2000-07-01 14:54:16 +00:00
test.xml.out Added necessary test input file for test_sax.py 2000-09-24 18:57:26 +00:00
testall.py Restore old behavior of autotest and testall, using regrtest. 1997-03-07 21:05:43 +00:00
testcodec.py Whitespace normalization. Leaving tokenize_tests.py alone for now. 2001-01-18 02:22:22 +00:00
testimg.uue Use uuencoded test images. 1997-04-09 21:25:01 +00:00
testimgr.uue Use uuencoded test images. 1997-04-09 21:25:01 +00:00
testrgb.uue Use uuencoded test images. 1997-04-09 21:25:01 +00:00
tokenize_tests.py There's no need for the tokenize tests to include a SyntaxError. 2001-04-13 14:36:51 +00:00
xmltests.py Suppress the variable verbose output from test.xmltests; the inclusion of 2002-06-26 15:16:16 +00:00

                      Writing Python Regression Tests
                      -------------------------------
                               Skip Montanaro
                              (skip@mojam.com)


Introduction

If you add a new module to Python or modify the functionality of an existing
module, you should write one or more test cases to exercise that new
functionality.  There are different ways to do this within the regression
testing facility provided with Python; any particular test should use only
one of these options.  Each option requires writing a test module using the
conventions of the the selected option:

    - PyUnit based tests
    - doctest based tests
    - "traditional" Python test modules

Regardless of the mechanics of the testing approach you choose,
you will be writing unit tests (isolated tests of functions and objects
defined by the module) using white box techniques.  Unlike black box
testing, where you only have the external interfaces to guide your test case
writing, in white box testing you can see the code being tested and tailor
your test cases to exercise it more completely.  In particular, you will be
able to refer to the C and Python code in the CVS repository when writing
your regression test cases.


PyUnit based tests

The PyUnit framework is based on the ideas of unit testing as espoused
by Kent Beck and the Extreme Programming (XP) movement.  The specific
interface provided by the framework is tightly based on the JUnit
Java implementation of Beck's original SmallTalk test framework.  Please
see the documentation of the unittest module for detailed information on
the interface and general guidelines on writing PyUnit based tests.

The test_support helper module provides a two functions for use by
PyUnit based tests in the Python regression testing framework:
run_unittest() takes a unittest.TestCase derived class as a parameter
and runs the tests defined in that class, and run_suite() takes a
populated TestSuite instance and runs the tests.  run_suite() is
preferred because unittest files typically grow multiple test classes,
and you might as well be prepared.

All test methods in the Python regression framework have names that
start with "test_" and use lower-case names with words separated with
underscores.

All PyUnit-based tests in the Python test suite use boilerplate that
looks like this:

    import unittest
    import test_support

    class MyTestCase1(unittest.TestCase):
        # define test methods here...

    class MyTestCase2(unittest.TestCase):
        # define more test methods here...

    def test_main():
	suite = unittest.TestSuite()
	suite.addTest(unittest.makeSuite(MyTestCase1))
	suite.addTest(unittest.makeSuite(MyTestCase2))
        test_support.run_suite(suite)

    if __name__ == "__main__":
        test_main()

This has the advantage that it allows the unittest module to be used
as a script to run individual tests as well as working well with the
regrtest framework.


doctest based tests

Tests written to use doctest are actually part of the docstrings for
the module being tested.  Each test is written as a display of an
interactive session, including the Python prompts, statements that would
be typed by the user, and the output of those statements (including
tracebacks, although only the exception msg needs to be retained then).
The module in the test package is simply a wrapper that causes doctest
to run over the tests in the module.  The test for the difflib module
provides a convenient example:

    import difflib, test_support
    test_support.run_doctest(difflib)

If the test is successful, nothing is written to stdout (so you should not
create a corresponding output/test_difflib file), but running regrtest
with -v will give a detailed report, the same as if passing -v to doctest.

A second argument can be passed to run_doctest to tell doctest to search
sys.argv for -v instead of using test_support's idea of verbosity.  This
is useful for writing doctest-based tests that aren't simply running a
doctest'ed Lib module, but contain the doctests themselves.  Then at
times you may want to run such a test directly as a doctest, independent
of the regrtest framework.  The tail end of test_descrtut.py is a good
example:

    def test_main(verbose=None):
        import test_support, test.test_descrtut
        test_support.run_doctest(test.test_descrtut, verbose)

    if __name__ == "__main__":
        test_main(1)

If run via regrtest, test_main() is called (by regrtest) without specifying
verbose, and then test_support's idea of verbosity is used.  But when
run directly, test_main(1) is called, and then doctest's idea of verbosity
is used.

See the documentation for the doctest module for information on
writing tests using the doctest framework.


"traditional" Python test modules

The mechanics of how the "traditional" test system operates are fairly
straightforward.  When a test case is run, the output is compared with the
expected output that is stored in .../Lib/test/output.  If the test runs to
completion and the actual and expected outputs match, the test succeeds, if
not, it fails.  If an ImportError or test_support.TestSkipped error is
raised, the test is not run.


Executing Test Cases

If you are writing test cases for module spam, you need to create a file
in .../Lib/test named test_spam.py.  In addition, if the tests are expected
to write to stdout during a successful run, you also need to create an
expected output file in .../Lib/test/output named test_spam ("..."
represents the top-level directory in the Python source tree, the directory
containing the configure script).  If needed, generate the initial version
of the test output file by executing:

    ./python Lib/test/regrtest.py -g test_spam.py

from the top-level directory.

Any time you modify test_spam.py you need to generate a new expected
output file.  Don't forget to desk check the generated output to make sure
it's really what you expected to find!  All in all it's usually better
not to have an expected-out file (note that doctest- and unittest-based
tests do not).

To run a single test after modifying a module, simply run regrtest.py
without the -g flag:

    ./python Lib/test/regrtest.py test_spam.py

While debugging a regression test, you can of course execute it
independently of the regression testing framework and see what it prints:

    ./python Lib/test/test_spam.py

To run the entire test suite:

[UNIX, + other platforms where "make" works] Make the "test" target at the
top level:

    make test

[WINDOWS] Run rt.bat from your PCBuild directory.  Read the comments at
the top of rt.bat for the use of special -d, -O and -q options processed
by rt.bat.

[OTHER] You can simply execute the two runs of regrtest (optimized and
non-optimized) directly:

    ./python Lib/test/regrtest.py
    ./python -O Lib/test/regrtest.py

But note that this way picks up whatever .pyc and .pyo files happen to be
around.  The makefile and rt.bat ways run the tests twice, the first time
removing all .pyc and .pyo files from the subtree rooted at Lib/.

Test cases generate output based upon values computed by the test code.
When executed, regrtest.py compares the actual output generated by executing
the test case with the expected output and reports success or failure.  It
stands to reason that if the actual and expected outputs are to match, they
must not contain any machine dependencies.  This means your test cases
should not print out absolute machine addresses (e.g. the return value of
the id() builtin function) or floating point numbers with large numbers of
significant digits (unless you understand what you are doing!).


Test Case Writing Tips

Writing good test cases is a skilled task and is too complex to discuss in
detail in this short document.  Many books have been written on the subject.
I'll show my age by suggesting that Glenford Myers' "The Art of Software
Testing", published in 1979, is still the best introduction to the subject
available.  It is short (177 pages), easy to read, and discusses the major
elements of software testing, though its publication predates the
object-oriented software revolution, so doesn't cover that subject at all.
Unfortunately, it is very expensive (about $100 new).  If you can borrow it
or find it used (around $20), I strongly urge you to pick up a copy.

The most important goal when writing test cases is to break things.  A test
case that doesn't uncover a bug is much less valuable than one that does.
In designing test cases you should pay attention to the following:

    * Your test cases should exercise all the functions and objects defined
      in the module, not just the ones meant to be called by users of your
      module.  This may require you to write test code that uses the module
      in ways you don't expect (explicitly calling internal functions, for
      example - see test_atexit.py).

    * You should consider any boundary values that may tickle exceptional
      conditions (e.g. if you were writing regression tests for division,
      you might well want to generate tests with numerators and denominators
      at the limits of floating point and integer numbers on the machine
      performing the tests as well as a denominator of zero).

    * You should exercise as many paths through the code as possible.  This
      may not always be possible, but is a goal to strive for.  In
      particular, when considering if statements (or their equivalent), you
      want to create test cases that exercise both the true and false
      branches.  For loops, you should create test cases that exercise the
      loop zero, one and multiple times.

    * You should test with obviously invalid input.  If you know that a
      function requires an integer input, try calling it with other types of
      objects to see how it responds.

    * You should test with obviously out-of-range input.  If the domain of a
      function is only defined for positive integers, try calling it with a
      negative integer.

    * If you are going to fix a bug that wasn't uncovered by an existing
      test, try to write a test case that exposes the bug (preferably before
      fixing it).

    * If you need to create a temporary file, you can use the filename in
      test_support.TESTFN to do so.  It is important to remove the file
      when done; other tests should be able to use the name without cleaning
      up after your test.


Regression Test Writing Rules

Each test case is different.  There is no "standard" form for a Python
regression test case, though there are some general rules (note that
these mostly apply only to the "classic" tests; unittest- and doctest-
based tests should follow the conventions natural to those frameworks):

    * If your test case detects a failure, raise TestFailed (found in
      test_support).

    * Import everything you'll need as early as possible.

    * If you'll be importing objects from a module that is at least
      partially platform-dependent, only import those objects you need for
      the current test case to avoid spurious ImportError exceptions that
      prevent the test from running to completion.

    * Print all your test case results using the print statement.  For
      non-fatal errors, print an error message (or omit a successful
      completion print) to indicate the failure, but proceed instead of
      raising TestFailed.

    * Use "assert" sparingly, if at all.  It's usually better to just print
      what you got, and rely on regrtest's got-vs-expected comparison to
      catch deviations from what you expect.  assert statements aren't
      executed at all when regrtest is run in -O mode; and, because they
      cause the test to stop immediately, can lead to a long & tedious
      test-fix, test-fix, test-fix, ... cycle when things are badly broken
      (and note that "badly broken" often includes running the test suite
      for the first time on new platforms or under new implementations of
      the language).


Miscellaneous

There is a test_support module you can import from your test case.  It
provides the following useful objects:

    * TestFailed - raise this exception when your regression test detects a
      failure.

    * TestSkipped - raise this if the test could not be run because the
      platform doesn't offer all the required facilities (like large
      file support), even if all the required modules are available.

    * verbose - you can use this variable to control print output.  Many
      modules use it.  Search for "verbose" in the test_*.py files to see
      lots of examples.

    * verify(condition, reason='test failed').  Use this instead of

          assert condition[, reason]

      verify() has two advantages over assert:  it works even in -O mode,
      and it raises TestFailed on failure instead of AssertionError.

    * TESTFN - a string that should always be used as the filename when you
      need to create a temp file.  Also use try/finally to ensure that your
      temp files are deleted before your test completes.  Note that you
      cannot unlink an open file on all operating systems, so also be sure
      to close temp files before trying to unlink them.

    * sortdict(dict) - acts like repr(dict.items()), but sorts the items
      first.  This is important when printing a dict value, because the
      order of items produced by dict.items() is not defined by the
      language.

    * findfile(file) - you can call this function to locate a file somewhere
      along sys.path or in the Lib/test tree - see test_linuxaudiodev.py for
      an example of its use.

    * use_large_resources - true iff tests requiring large time or space
      should be run.

    * fcmp(x,y) - you can call this function to compare two floating point
      numbers when you expect them to only be approximately equal withing a
      fuzz factor (test_support.FUZZ, which defaults to 1e-6).

NOTE:  Always import something from test_support like so:

    from test_support import verbose

or like so:

    import test_support
    ... use test_support.verbose in the code ...

Never import anything from test_support like this:

    from test.test_support import verbose

"test" is a package already, so can refer to modules it contains without
"test." qualification.  If you do an explicit "test.xxx" qualification, that
can fool Python into believing test.xxx is a module distinct from the xxx
in the current package, and you can end up importing two distinct copies of
xxx.  This is especially bad if xxx=test_support, as regrtest.py can (and
routinely does) overwrite its "verbose" and "use_large_resources"
attributes:  if you get a second copy of test_support loaded, it may not
have the same values for those as regrtest intended.


Python and C statement coverage results are currently available at

    http://www.musi-cal.com/~skip/python/Python/dist/src/

As of this writing (July, 2000) these results are being generated nightly.
You can refer to the summaries and the test coverage output files to see
where coverage is adequate or lacking and write test cases to beef up the
coverage.


Some Non-Obvious regrtest Features

    * Automagic test detection:  When you create a new test file
      test_spam.py, you do not need to modify regrtest (or anything else)
      to advertise its existence.  regrtest searches for and runs all
      modules in the test directory with names of the form test_xxx.py.

    * Miranda output:  If, when running test_spam.py, regrtest does not
      find an expected-output file test/output/test_spam, regrtest
      pretends that it did find one, containing the single line

      test_spam

      This allows new tests that don't expect to print anything to stdout
      to not bother creating expected-output files.

    * Two-stage testing:  To run test_spam.py, regrtest imports test_spam
      as a module.  Most tests run to completion as a side-effect of
      getting imported.  After importing test_spam, regrtest also executes
      test_spam.test_main(), if test_spam has a "test_main" attribute.
      This is rarely required with the "traditional" Python tests, and
      you shouldn't create a module global with name test_main unless
      you're specifically exploiting this gimmick.  This usage does
      prove useful with PyUnit-based tests as well, however; defining
      a test_main() which is run by regrtest and a script-stub in the
      test module ("if __name__ == '__main__': test_main()") allows
      the test to be used like any other Python test and also work
      with the unittest.py-as-a-script approach, allowing a developer
      to run specific tests from the command line.