1# ANGLE Testing Infrastructure
2
3ANGLE runs hundreds of thousands of tests on every change before it lands in
4the tree. We scale our pre-commit and post-commit testing to many machines
5using [Chromium Swarming][Swarming]. Our testing setup heavily leverages
6existing work in Chromium. We also run compile-only
7[Standalone Testing][Standalone] that does not depend on a Chromium checkout.
8
9Also see the documentation on [ANGLE Wrangling][Wrangling] for more info.
10
11## Pre-Commit Testing
12
13See the pre-commit try waterfall here:
14
15[`https://ci.chromium.org/p/chromium/g/tryserver.chromium.angle/builders`](https://ci.chromium.org/p/chromium/g/tryserver.chromium.angle/builders)
16
17We currently run pre-commit tests on:
18
19 * Windows 32-bit AMD and Windows 64-bit Intel and NVIDIA GPUs
20 * Linux 64-bit NVIDIA and Intel GPUs
21 * Mac NVIDIA, Intel and AMD GPUs
22 * Pixel 2 and Nexus 5X
23 * Fuchsia testing in a VM
24
25Looking at an example build shows how tests are split up between machines. See for example:
26
27[`https://ci.chromium.org/ui/p/chromium/builders/try/mac-angle-try/454/overview`](https://ci.chromium.org/ui/p/chromium/builders/try/mac-angle-try/454/overview)
28
29This build ran 81 test steps across 3 GPU families. In some cases (e.g.
30`angle_deqp_gles3_metal_tests`) the test is split up between multiple machines to
31run faster (in this case 4 different machines at once). This build took 15
32minutes to complete 1 hour of real automated testing.
33
34For more details on running and working with our test sets see the docs in [Contributing Code][Contrib].
35
36[Swarming]: https://chromium-swarm.appspot.com/
37[Standalone]: #ANGLE-Standalone-Testing
38[Contrib]: ../doc/ContributingCode.md#Testing
39[Wrangling]: ANGLEWrangling.md
40
41## ANGLE Standalone Testing
42
43In addition to the ANGLE try bots using Chrome, and the GPU.FYI bots, ANGLE
44has standalone testing on the Chrome infrastructure. Currently these tests are
45compile-only. This page is for maintaining the configurations that don't use
46Chromium. Also see the main instructions for [ANGLE Wrangling](ANGLEWrangling.md).
47
48It's the ANGLE team's responsibility for maintaining this testing
49infrastructure. The bot configurations live in four different repos and six
50branches.
51
52## Info Consoles
53
54Continuous builders for every ANGLE revision are found on the CI console:
55
56[https://ci.chromium.org/p/angle/g/ci/console](https://ci.chromium.org/p/angle/g/ci/console)
57
58Try jobs from pre-commit builds are found on the builders console:
59
60[https://ci.chromium.org/p/angle/g/try/builders](https://ci.chromium.org/p/angle/g/try/builders)
61
62## How to add a new build configuration
63
64 1. [`bugs.chromium.org/p/chromium/issues/entry?template=Build+Infrastructure`](http://bugs.chromium.org/p/chromium/issues/entry?template=Build+Infrastructure):
65
66    * If adding a Mac bot, request new slaves by filing an infra issue.
67
68 1. [`chrome-internal.googlesource.com/infradata/config`](http://chrome-internal.googlesource.com/infradata/config):
69
70    * Update **`configs/chromium-swarm/starlark/bots/angle.star`** with either Mac slaves requested in the previous step or increase the amount of Windows or Linux GCEs.
71
72 1. [`chromium.googlesource.com/chromium/tools/build`](https://chromium.googlesource.com/chromium/tools/build):
73
74    * Update **`scripts/slave/recipes/angle.py`** with new the config.
75    * The recipe code requires 100% code coverage through mock bots, so add mock bot config to GenTests.
76    * Maybe run `./scripts/slave/recipes.py test train` to update checked-in golden files. This might no longer be necessary.
77
78 1. [`chromium.googlesource.com/angle/angle`](http://chromium.googlesource.com/angle/angle):
79
80    * Update **`infra/config/global/cr-buildbucket.cfg`** to add the new builder (to ci and try), and set the new config option.
81    * Update **`infra/config/global/luci-milo.cfg`** to make the builders show up on the ci and try waterfalls.
82    * Update **`infra/config/global/luci-scheduler.cfg`** to make the builders trigger on new commits or try jobs respectively.
83    * Update **`infra/config/global/commit-queue.cfg`** to add the builder to the default CQ jobs (if desired).
84
85## Other Configuration
86
87There are other places where configuration for ANGLE infra lives. These are files that we shouldn't need to modify very often:
88
89 1. [`chrome-internal.googlesource.com/infradata/config`](http://chrome-internal.googlesource.com/infradata/config):
90
91    * **`configs/luci-token-server/service_accounts.cfg`** (service account names)
92    * **`configs/chromium-swarm/pools.cfg`** (swarming pools)
93
94 1. [`chromium.googlesource.com/chromium/tools/depot_tools`](http://chromium.googlesource.com/chromium/tools/depot_tools):
95
96    * **`recipes/recipe_modules/gclient/config.py`** (gclient config)
97