**************************************************************************
* _ _ ____ _
* Project ___| | | | _ \| |
* / __| | | | |_) | |
* | (__| |_| | _ <| |___
* \___|\___/|_| \_\_____|
*
* Copyright (C) 1998 - 2020, Daniel Stenberg, <daniel@haxx.se>, et al.
*
* This software is licensed as described in the file COPYING, which
* you should have received as part of this distribution. The terms
* are also available at https://curl.haxx.se/docs/copyright.html.
*
* You may opt to use, copy, modify, merge, publish, distribute and/or sell
* copies of the Software, and permit persons to whom the Software is
* furnished to do so, under the terms of the COPYING file.
*
* This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY
* KIND, either express or implied.
*
**************************************************************************

runtests.pl - run one or more test cases runtests.pl [options] [tests] runtests.pl runs one, several or all the existing test cases in curl's test suite. It is often called from the root Makefile of the curl package with 'make test'. Specify which test(s) to run by specifying test numbers or keywords. If no test number or keyword is given, all existing tests that the script can find will be considered for running. You can specify single test cases to run by specifying test numbers space-separated, like "1 3 5 7 11", and you can specify a range of tests like "45 to 67". Specify tests to not run with a leading exclamation point, like "!66", which runs all available tests except number 66. Prefix a test number with a tilde (~) to still run it, but ignore the results. It is also possible to specify tests based on a keyword describing the test(s) to run, like "FTPS". The keywords are strings used in the individual tests. You can also specify keywords with a leading exclamation point and the keyword or phrase, like "!HTTP NTLM auth" to run all tests except those using this keyword. Remember that the exclamation marks and spaces will need to be quoted somehow when entered at many command shells. Prefix a keyword with a tilde (~) to still run it, but ignore the results. Continue running the rest of the test cases even if one test fails. By default, the test script stops as soon as an error is detected. Provide a path to a custom curl binary to run the tests with. Default is the curl executable in the build tree. Enable protocol debug: have the servers display protocol output. Run the test event-based (if possible). This will make runtests invoke curl with --test-event option. This option only works if both curl and libcurl were built debug-enabled. Run the given test(s) with gdb. This is best used on a single test case and curl built --disable-shared. This then fires up gdb with command line set to run the specified test case. Simply (set a break-point and) type 'run' to start. Displays a help text about this program's command line options. Keep output and log files in log/ after a test run, even if no error was detected. Useful for debugging. Lists all test case names. Disable the check for and use of valgrind. Prints out all files in "log/" to stdout when a test case fails. Very practical when used in the automated and distributed tests since then the people checking the failures and the reasons for them might not have physical access to the machine and logs. Run the tests in a scrambled, or randomized, order instead of sequentially. The random seed initially set for this is fixed per month and can be set with --seed. Display run time statistics. (Requires Perl Time::HiRes module) Display full run time statistics. (Requires Perl Time::HiRes module) This will repeat the given set of test numbers this many times. If no test numbers are given, it will repeat ALL tests this many times. It iteratively adds the new sequence at the end of the initially given one. If -R is also used, the scrambling is done after the repeats have extended the test sequence. Shorter output. Speaks less than default. When using --shallow or -R\rP that random certain aspects of the behavior, this option can set the initial seed. If not set, the random seed will be set based on the currently set local year and month and the first line of the "curl -V" output. Used together with -t. This limits the number of tests to fail in torture mode to no more than 'num' per test case. If this reduces the amount, the script will randomly discard entries to fail until the amount is 'num'. The random seed initially set for this is fixed per month and can be set with --seed. Selects a torture test for the given tests. This makes runtests.pl first run the tests once and count the number of memory allocations made. It then reruns the test that number of times, each time forcing one of the allocations to fail until all allocs have been tested. By setting num you can force the allocation with that number to be set to fail at once instead of looping through everyone, which is very handy when debugging and then often in combination with -g. Enable verbose output. Speaks more than default. Provide a path to a custom curl binary to run when verifying that the servers running are indeed our test servers. Default is the curl executable in the build tree. Many tests have conditions that must be met before the test case can run fine. They could depend on built-in features in libcurl or features present in the operating system or even in third-party libraries that curl may or may not use.

The test script checks most of these by itself to determine when it is safe to attempt to run each test. Those which cannot be run due to failed requirements will simply be skipped and listed at the completion of all test cases. In some unusual configurations, the test script cannot make the correct determination for all tests. In these cases, the problematic tests can be skipped using the "!keyword" skip feature documented earlier.

The simplest way to write test cases is to start with a similar existing test, save it with a new number and then adjust it to fit. There's an attempt to document the test case file format in the tests/FILEFORMAT.