README.md
1# ANGLE Performance Tests
2
3`angle_perftests` is a standalone testing suite that contains targeted tests for OpenGL, Vulkan and ANGLE internal classes. The tests currently run on the Chromium ANGLE infrastructure and report results to the [Chromium perf dashboard](https://chromeperf.appspot.com/report).
4
5You can also build your own dashboards. For example, a comparison of ANGLE's back-end draw call performance on Windows NVIDIA can be found [at this link](https://chromeperf.appspot.com/report?sid=1fdf94a308f52b6bf02c08f6f36e87ca0d0075e2d2eefc61e6cf90c919c1643a&start_rev=577814&end_rev=582136). Note that this link is not kept current.
6
7## Running the Tests
8
9You can follow the usual instructions to [check out and build ANGLE](../../../doc/DevSetup.md). Build the `angle_perftests` target. Note that all test scores are higher-is-better. You should also ensure `is_debug=false` in your build. Running with `angle_assert_always_on` or debug validation enabled is not recommended.
10
11Variance can be a problem when benchmarking. We have a test harness to run a single test in an infinite loop and print some statistics to help mitigate variance. See [`scripts/perf_test_runner.py`](https://chromium.googlesource.com/angle/angle/+/master/scripts/perf_test_runner.py). To use the script first compile `angle_perftests` into a folder with the word `Release` in it. Then provide the name of the test as the argument to the script. The script will automatically pick up the most current `angle_perftests` and run in an infinite loop.
12
13### Choosing the Test to Run
14
15You can choose individual tests to run with `--gtest_filter=*TestName*`. To select a particular ANGLE back-end, add the name of the back-end to the test filter. For example: `DrawCallPerfBenchmark.Run/gl` or `DrawCallPerfBenchmark.Run/d3d11`. Many tests have sub-tests that run slightly different code paths. You might need to experiment to find the right sub-test and its name.
16
17### Null/No-op Configurations
18
19ANGLE implements a no-op driver for OpenGL, D3D11 and Vulkan. To run on these configurations use the `gl_null`, `d3d11_null` or `vulkan_null` test configurations. These null drivers will not do any GPU work. They will skip the driver entirely. These null configs are useful for diagnosing performance overhead in ANGLE code.
20
21### Command-line Arguments
22
23Several command-line arguments control how the tests run:
24
25* `--one-frame-only`: Runs tests once and quickly exits. Used as a quick smoke test.
26* `--enable-trace`: Write a JSON event log that can be loaded in Chrome.
27* `--trace-file file`: Name of the JSON event log for `--enable-trace`.
28* `--calibration`: Prints the number of steps a test runs in a fixed time. Used by `perf_test_runner.py`.
29* `--steps-per-trial x`: Fixed number of steps to run for each test trial.
30* `--max-steps-performed x`: Upper maximum on total number of steps for the entire test run.
31* `--screenshot-dir dir`: Directory to store test screenshots. Only implemented in `TracePerfTest`.
32* `--screenshot-frame <frame>`: Which frame to capture a screenshot of. Defaults to first frame (1). Only implemented in `TracePerfTest`.
33* `--render-test-output-dir=dir`: Equivalent to `--screenshot-dir dir`.
34* `--verbose`: Print extra timing information.
35* `--warmup-loops x`: Number of times to warm up the test before starting timing. Defaults to 3.
36* `--no-warmup`: Skip warming up the tests. Equivalent to `--warmup-steps 0`.
37* `--calibration-time`: Run each test calibration step in a fixed time. Defaults to 1 second.
38* `--max-trial-time x`: Run each test trial under this max time. Defaults to 10 seconds.
39* `--fixed-test-time x`: Run the tests until this much time has elapsed.
40* `--trials`: Number of times to repeat testing. Defaults to 3.
41* `--no-finish`: Don't call glFinish after each test trial.
42* `--enable-all-trace-tests`: Offscreen and vsync-limited trace tests are disabled by default to reduce test time.
43* `--minimize-gpu-work`: Modify API calls so that GPU work is reduced to minimum.
44
45For example, for an endless run with no warmup, run:
46
47`angle_perftests --gtest_filter=TracePerfTest.Run/vulkan_trex_200 --steps 1000000 --no-warmup`
48
49The command line arguments implementations are located in [`ANGLEPerfTestArgs.cpp`](ANGLEPerfTestArgs.cpp).
50
51## Test Breakdown
52
53* [`DrawCallPerfBenchmark`](DrawCallPerf.cpp): Runs a tight loop around DrawArarys calls.
54 * `validation_only`: Skips all rendering.
55 * `render_to_texture`: Render to a user Framebuffer instead of the default FBO.
56 * `vbo_change`: Applies a Vertex Array change between each draw.
57 * `tex_change`: Applies a Texture change between each draw.
58* [`UniformsBenchmark`](UniformsPerf.cpp): Tests performance of updating various uniforms counts followed by a DrawArrays call.
59 * `vec4`: Tests `vec4` Uniforms.
60 * `matrix`: Tests using Matrix uniforms instead of `vec4`.
61 * `multiprogram`: Tests switching Programs between updates and draws.
62 * `repeating`: Skip the update of uniforms before each draw call.
63* [`DrawElementsPerfBenchmark`](DrawElementsPerf.cpp): Similar to `DrawCallPerfBenchmark` but for indexed DrawElements calls.
64* [`BindingsBenchmark`](BindingPerf.cpp): Tests Buffer binding performance. Does no draw call operations.
65 * `100_objects_allocated_every_iteration`: Tests repeated glBindBuffer with new buffers allocated each iteration.
66 * `100_objects_allocated_at_initialization`: Tests repeated glBindBuffer the same objects each iteration.
67* [`TexSubImageBenchmark`](TexSubImage.cpp): Tests `glTexSubImage` update performance.
68* [`BufferSubDataBenchmark`](BufferSubData.cpp): Tests `glBufferSubData` update performance.
69* [`TextureSamplingBenchmark`](TextureSampling.cpp): Tests Texture sampling performance.
70* [`TextureBenchmark`](TexturesPerf.cpp): Tests Texture state change performance.
71* [`LinkProgramBenchmark`](LinkProgramPerfTest.cpp): Tests performance of `glLinkProgram`.
72* [`glmark2`](glmark2.cpp): Runs the glmark2 benchmark.
73* [`TracePerfTest`](TracePerfTest.cpp): Runs replays of restricted traces, not available publicly. To enable, read more in [`RestrictedTraceTests`](../restricted_traces/README.md)
74
75Many other tests can be found that have documentation in their classes.
76