zstd/TESTING.md

Testing
=======

Zstandard CI testing is split up into three sections:
short, medium, and long tests.

Short Tests
-----------
Short tests run on CircleCI for new commits on every branch and pull request.
They consist of the following tests:
- Compilation on all supported targets (x86, x86_64, ARM, AArch64, PowerPC, and PowerPC64)
- Compilation on various versions of gcc, clang, and g++
- `tests/playTests.sh` on x86_64, without the tests on long data (CLI tests)
- Small tests (`tests/legacy.c`, `tests/longmatch.c`) on x64_64

Medium Tests
------------
Medium tests run on every commit and pull request to `dev` branch, on TravisCI.
They consist of the following tests:
- The following tests run with UBsan and Asan on x86_64 and x86, as well as with
  Msan on x86_64
  - `tests/playTests.sh --test-large-data`
  - Fuzzer tests: `tests/fuzzer.c`, `tests/zstreamtest.c`, and `tests/decodecorpus.c`
- `tests/zstreamtest.c` under Tsan (streaming mode, including multithreaded mode)
- Valgrind Test (`make -C tests test-valgrind`) (testing CLI and fuzzer under `valgrind`)
- Fuzzer tests (see above) on ARM, AArch64, PowerPC, and PowerPC64

Long Tests
----------
Long tests run on all commits to `release` branch,
and once a day on the current version of `dev` branch,
on TravisCI.
They consist of the following tests:
- Entire test suite (including fuzzers and some other specialized tests) on:
  - x86_64 and x86 with UBsan and Asan
  - x86_64 with Msan
  - ARM, AArch64, PowerPC, and PowerPC64
- Streaming mode fuzzer with Tsan (for the `zstdmt` testing)
- ZlibWrapper tests, including under valgrind
- Versions test (ensuring `zstd` can decode files from all previous versions)
- `pzstd` with asan and tsan, as well as in 32-bits mode
- Testing `zstd` with legacy mode off
- Entire test suite and make install on macOS
Add testing description 2017-03-10 05:59:26 +08:00			`Testing`
			`=======`

			`Zstandard CI testing is split up into three sections:`
			`short, medium, and long tests.`

			`Short Tests`
			`-----------`
			`Short tests run on CircleCI for new commits on every branch and pull request.`
			`They consist of the following tests:`
			`- Compilation on all supported targets (x86, x86_64, ARM, AArch64, PowerPC, and PowerPC64)`
			`- Compilation on various versions of gcc, clang, and g++`
			- `tests/playTests.sh` on x86_64, without the tests on long data (CLI tests)
Removing symbols already in unit tests and adding some new unit tests for missing symbols (#1985) * Removing symbols that are not being tested * Removing symbols used in zstdcli, fileio, dibio and benchzstd * Removing symbols used in zbuff and add test-zbuff to travis * Removing remaining symbols and adding unit tests instead * Removing symbols test entirely 2020-02-06 08:55:00 +08:00			- Small tests (`tests/legacy.c`, `tests/longmatch.c`) on x64_64
Add testing description 2017-03-10 05:59:26 +08:00
			`Medium Tests`
			`------------`
			Medium tests run on every commit and pull request to `dev` branch, on TravisCI.
			`They consist of the following tests:`
			`- The following tests run with UBsan and Asan on x86_64 and x86, as well as with`
			`Msan on x86_64`
Fix testing documentation typo 2020-01-30 23:31:00 +08:00			- `tests/playTests.sh --test-large-data`
Add testing description 2017-03-10 05:59:26 +08:00			- Fuzzer tests: `tests/fuzzer.c`, `tests/zstreamtest.c`, and `tests/decodecorpus.c`
			- `tests/zstreamtest.c` under Tsan (streaming mode, including multithreaded mode)
streamline `make clean` list maintenance When creating a new `Makefile` target to build, it's also necessary to update the `clean` target, which purpose is to remove built targets when they are present. This process is simple, but it's also easy to forget : since there is a large distance between the position in the `Makefile` where the new built target is added, and the place where the list of files to `clean` is defined. Moreover, the list of files becomes pretty long over time, hence it's difficult to visually ensure that all built targets are present there, or that no old target (no longer produced) is no longer in the list This PR tries to improve this process by adding a CLEAN variable. Now, when a new built target is added to the `Makefile`, it should preceded by : ``` CLEAN += newTarget newTarget: <TAB> ...recipe... ``` This new requirement is somewhat similar to `.PHONY: newTarget` for non-built targets. This new method offers the advantage of locality : there is no separate place in the file to maintain a list of files to clean. This makes maintenance of `make clean` easier. 2022-09-08 07:10:13 +08:00			- Valgrind Test (`make -C tests test-valgrind`) (testing CLI and fuzzer under `valgrind`)
Add testing description 2017-03-10 05:59:26 +08:00			`- Fuzzer tests (see above) on ARM, AArch64, PowerPC, and PowerPC64`

			`Long Tests`
			`----------`
moving all references to `release` branch was previously `master` 2020-12-17 15:00:35 +08:00			Long tests run on all commits to `release` branch,
Add testing description 2017-03-10 05:59:26 +08:00			and once a day on the current version of `dev` branch,
			`on TravisCI.`
			`They consist of the following tests:`
			`- Entire test suite (including fuzzers and some other specialized tests) on:`
			`- x86_64 and x86 with UBsan and Asan`
			`- x86_64 with Msan`
			`- ARM, AArch64, PowerPC, and PowerPC64`
			- Streaming mode fuzzer with Tsan (for the `zstdmt` testing)
			`- ZlibWrapper tests, including under valgrind`
			- Versions test (ensuring `zstd` can decode files from all previous versions)
			- `pzstd` with asan and tsan, as well as in 32-bits mode
			- Testing `zstd` with legacy mode off
Fix name of macOS 2018-06-10 03:31:17 +08:00			`- Entire test suite and make install on macOS`