README.md
1# ANGLE Test Harness
2
3The ANGLE test harness is a harness around GoogleTest that provides functionality similar to the
4[Chromium test harness][BaseTest]. It features:
5
6 * splitting a test set into shards
7 * catching and reporting crashes and timeouts
8 * outputting to the Chromium [JSON test results format][JSONFormat]
9 * multi-process execution
10
11## Command-Line Arguments
12
13The ANGLE test harness accepts all standard GoogleTest arguments. The harness also accepts the
14following additional command-line arguments:
15
16 * `--batch-size` limits the number of tests to run in each batch
17 * `--batch-timeout` limits the amount of time spent in each batch
18 * `--bot-mode` enables multi-process execution and test batching
19 * `--debug-test-groups` dumps the test config categories when using `bot-mode`
20 * `--filter-file` allows passing a larger `gtest_filter` via a file
21 * `--histogram-json-file` outputs a [formatted JSON file][HistogramSet] for perf dashboards
22 * `--max-processes` limits the number of simuntaneous processes
23 * `--results-directory` specifies a directory to write test results to
24 * `--results-file` specifies a location for the JSON test result output
25 * `--shard-count` and `--shard-index` control the test sharding
26 * `--test-timeout` limits the amount of time spent in each test
27 * `--flaky-retries` allows for tests to fail a fixed number of times and still pass
28 * `--disable-crash-handler` forces off OS-level crash handling
29 * `--isolated-outdir` specifies a test artifacts directory
30 * `--max-failures` specifies a count of failures after which the harness early exits.
31
32`--isolated-script-test-output` and `--isolated-script-perf-test-output` mirror `--results-file`
33and `--histogram-json-file` respectively.
34
35As well as the custom command-line arguments we support a few standard GoogleTest arguments:
36
37 * `gtest_filter` works as it normally does with GoogleTest
38 * `gtest_also_run_disabled_tests` works as it normally does as well
39
40Other GoogleTest arguments are not supported although they may work.
41
42## Implementation Notes
43
44 * The test harness only requires `angle_common` and `angle_util`.
45 * It does not depend on any Chromium browser code. This allows us to compile on other non-Clang platforms.
46 * It uses rapidjson to read and write JSON files.
47 * Test execution is not currently deterministic in multi-process mode.
48
49## Normal Mode vs Bot Mode
50
51The test runner has two main modes of operation: normal and *bot mode*.
52
53During normal mode:
54
55 * Tests are executed single-process and single-thread.
56 * The test runner executes the GoogleTest Run function.
57 * We use a `TestEventListener` to record test results for our output JSON file.
58 * A *watchdog thread* will force a fast exit if no test results get recorded after a timeout.
59 * Crashes are handled via ANGLE's test crash handling code.
60
61During bot mode:
62
63 * Tests are run in multiple processes depending on the system processor count.
64 * A server process records the child processes' stdout and stderr.
65 * The server terminates a child process if there's no progress after a timeout.
66 * The server sorts work into batches according to the back-end configuration.
67 * This prevents driver errors from using multiple back-ends in the same process.
68 * Batches are striped to help split up slow groups of tests.
69 * The server passes test batches to child processes via a `gtest_filter` file.
70 * Bot mode does not work on Android or Fuchsia.
71
72See the source code for more details: [TestSuite.h](TestSuite.h) and [TestSuite.cpp](TestSuite.cpp).
73
74## Potential Areas of Improvement
75
76 * Deterministic test execution.
77 * Using sockets to communicate with test children. Similar to dEQP's test harness.
78 * Closer integration with ANGLE's test expectations and system config libraries.
79 * Supporting a GoogleTest-free integration.
80
81[BaseTest]: https://chromium.googlesource.com/chromium/src/+/refs/heads/master/base/test/
82[JSONFormat]: https://chromium.googlesource.com/chromium/src/+/master/docs/testing/json_test_results_format.md
83[HistogramSet]: https://chromium.googlesource.com/catapult/+/HEAD/docs/histogram-set-json-format.md
84