php-src/README.TESTING

[IMPORTANT NOTICE]
------------------
 Do _not_ ask to developers why some or all tests are failed under
your environment! Let us know if you find why it fails. Thank you.


[Testing Basics] 
----------------
 To execute test scripts, you must build PHP with some SAPI, then you
type "make test" to execute all or some test scripts saved under
"tests" directory under source root directory.

Usage:
make test

 "make test" basically executes "run-tests.php" script
under the source root (parallel builds will not work). Therefore you
can execute the script as follows:

TEST_PHP_EXECUTABLE=sapi/cli/php \
sapi/cli/php [-c /path/to/php.ini] run-tests.php [ext/foo/tests/GLOB]


[Which "php" executable "make test" look for]
---------------------------------------------
 You must use TEST_PHP_EXECUTABLE environment variable to explicitly
select the php executable to be used to run the tests. That can either
be the CLI or CGI executable.

 "make test" executes "run-tests.php" script with "php" binary.  Some
test scripts such as session must be executed by CGI SAPI. Therefore,
you must build PHP with CGI SAPI to perform all tests.

NOTE: PHP binary executing "run-tests.php" and php binary used for
executing test scripts may differ. If you use different PHP binary for
executing "run-tests.php" script, you may get errors.


[Which php.ini is used]
-----------------------
 "make test" uses the same php.ini file as it would once installed.
The tests have been written to be independent of that php.ini file,
so if you find a test that is affected by a setting, please report
this, so we can address the issue.


[Which test scripts are executed]
---------------------------------
 "run-tests.php" ("make test"), without any arguments executes all
test scripts by extracting all directories named "tests"
from the source root and any subdirectories below. If there are files,
which have a "phpt" extension, "run-tests.php" looks at the sections
in these files, determines whether it should run it, by evaluating
the 'SKIP' section. If the test is eligible for execution, the 'FILE'
section is extracted into a ".php" file, with a unique name and
executed.
When an argument is given or TESTS environment variable is set, the
GLOB is expanded by the shell and any file with extension "*.phpt" is
regarded as a test file.

 Tester can easily execute tests selectively with as follows.

Examples:
./sapi/cli/php run-tests.php ext/mbstring/*
./sapi/cli/php run-tests.php ext/mbstring/020.phpt


[Test results]
--------------
 Test results are printed to standard output. If there is a failed test, 
the "run-tests.php" script saves the result, the expected result and the
code executed to the test script directory. For example, if 
ext/myext/tests/myext.phpt fails to pass, the following files are created:

ext/myext/tests/myext.php   - actual test file executed
ext/myext/tests/myext.log   - log of test execution (L)
ext/myext/tests/myext.exp   - expected output (E)
ext/myext/tests/myext.out   - output from test script (O)
ext/myext/tests/myext.diff  - diff of .out and .exp (D)

 Failed tests are always bugs. Either the test is bugged or not considering
factors applying to the tester's environment, or there is a bug in PHP.
If this is a known bug, we strive to provide bug numbers, in either the
test name or the file name. You can check the status of such a bug, by
going to: http://bugs.php.net/12345 where 12345 is the bug number.
For clarity and automated processing, bug numbers are prefixed by a hash
sign '#' in test names and/or test cases are named bug12345.phpt.

NOTE: The files generated by tests can be selected by setting the
environment variable TEST_PHP_LOG_FORMAT. For each file you want to be
generated use the character in brackets as shown above (default is LEOD).
The php file will be generated always.

NOTE: You can set environment variable TEST_PHP_DETAILED to enable
detailed test information.

[Automated testing]
 If you like to keep up to speed, with latest developments and quality
assurance, setting the environment variable NO_INTERACTION to 1, will not
prompt the tester for any user input.

Normally, the exit status of "make test" is zero, regardless of the results
of independent tests. Set the environment variable REPORT_EXIT_STATUS to 1,
and "make test" will set the exit status ("$?") to non-zero, when an
individual test has failed.

Example script to be run by cron(1):
========== qa-test.sh =============
#!/bin/sh

CO_DIR=$HOME/cvs/php4
MYMAIL=qa-test@domain.com
TMPDIR=/var/tmp
TODAY=`date +"%Y%m%d"`

# Make sure compilation enviroment is correct
CONFIGURE_OPTS='--disable-all --enable-cli --with-pcre'
export MAKE=gmake
export CC=gcc

# Set test environment
export NO_INTERACTIVITY=1
export REPORT_EXIT_STATUS=1

cd $CO_DIR
cvs update . >>$TMPDIR/phpqatest.$TODAY
./cvsclean ; ./buildconf ; ./configure $CONFIGURE_OPTS ; $MAKE
$MAKE test >>$TMPDIR/phpqatest.$TODAY 2>&1
if test $? -gt 0
then
        cat $TMPDIR/phpqatest.$TODAY | mail -s"PHP-QA Test Failed for $TODAY" $MYMAIL
fi
========== end of qa-test.sh =============

NOTE: the exit status of run-tests.php will be 1 when
REPORT_EXIT_STATUS is set. The result of "make test" may be higher
than that. At present, gmake 3.79.1 returns 2, so it is
advised to test for non-zero, rather then a specific value.


[Creating new test files]
-------------------------
 Writing test file is very easy if you are used to PHP. Test file
has following format. Here is a actual test file from iconv module.

===== ext/iconv/002.phpt =======
--TEST--
UCS4BE to ASCII
--SKIPIF--
<?php include('skipif.inc'); ?>
--FILE--
<?php include('002.inc'); ?>
--EXPECT--
&#97;&#98;&#99;&#100;
abcd
===== end of ext/iconv/002.phpt =======

 As you can see the file has the following sections:

"--TEST--"    is title of the test.
"--SKIPIF--"  is condition when to skip this test (optional).
"--POST--"    is POST variable passed to test script (optional).
"--GET--"     is GET variable passed to test script (optional).
"--INI--"     each line contains an ini setting e.g. foo=bar (optional).
"--FILE--"    is the test script.
"--EXPECT--"  is the expected output from the test script.
"--EXPECTF--" this is the alternate form of --EXPECT--. The difference 
              is is that this form uses sscanf for output validation.

ext/iconv/002.phpt uses 2 files. "skipif.inc" is used to skip
test when test cannot be performed such as iconv module is not
compiled or loaded. 

==== ext/iconv/skipif.inc ====
<?php
// This script prints "skip" if condition does not meet.

if (!extension_loaded("iconv") && ini_get("enable_dl")) {
  $dlext = (substr(PHP_OS, 0, 3) == "WIN") ? ".dll" : ".so";
  @dl("iconv$dlext");
}
if (!extension_loaded("iconv")) {
    die("skip\n");
}
?>
==== end of ext/iconv/skipif.inc ====

ext/inconv/002.inc is the test script. "run-tests.php" script
executes test script with CGI executable.

==== ext/iconv/002.inc ===
<?php
/*
Expected output:
&#97;&#98;&#99;&#100;
abcd
*/

   $s = unpack("V*", iconv("ascii","UCS-4LE", "abcd"));
   foreach($s as $c) { print "&#$c;"; } print "\n";

   $s = pack("NNNN", 97, 98, 99, 100);
   $q = iconv("UCS-4BE", "ascii", $s);
   print $q; print "\n";
?>
=== end of ext/iconv/002.inc ===

Test script and SKIPIF code should be directly written into *.phpt.
However, it is recommended to use include files when more test scripts
depend on the same SKIPIF code or when certain test files need the same
values for some input.


/ext/standard/tests/strings 003.phpt is a good example for using EXPECTF 
instead of EXPECT. From time to time the algorithm used for shuffle 
changed and sometimes the machine used to execute the code has influence 
on the result of shuffle. But it always returns a three character string 
detectable by %s. Other scan-able forms are %i for integers, %f for 
floating point values and %x for hexadecimal values.

==== /ext/standard/tests/strings 003.phpt ===
--TEST--
Testing str_shuffle.
--FILE--
<?php
$s = '123';
var_dump(str_shuffle($s));
var_dump($s);
?>
--EXPECTF--
string(3) %s
string(3) "123"
==== end of /ext/standard/tests/strings 003.phpt ===

[How to help us]
----------------
 If you find bug in PHP, you can submit bug report AND test script 
for us. You don't have to write complete script, just give us test
script with following format. Please test the script and make sure
you write the correct ACTUAL OUTPUT and EXPECTED OUTPUT before you
submit.

<?php
/* 
Bug #12345
substr() bug. Do not return expected string.

ACTUAL OUTPUT
XYXA

EXPECTED OUTPUT
ABCD
*/

$str = "XYZABCD";
echo substr($str,3,7);

?>