1# Running ART Tests with Atest / Trade Federation
2
3ART Testing has early support for execution in the [Trade
4Federation](https://source.android.com/devices/tech/test_infra/tradefed)
5("TradeFed") test harness, in particular via the
6[Atest](https://source.android.com/compatibility/tests/development/atest)
7command line tool.
8
9Atest conveniently takes care of building tests and their dependencies (using
10Soong, the Android build system) and executing them using Trade Federation.
11
12See also [README.md](README.md) for a general introduction to ART run-tests and
13gtests.
14
15## ART run-tests
16
17### Running ART run-tests on device
18
19ART run-tests are defined in sub-directories of `test/` starting with a number
20(e.g. `test/001-HelloWorld`). Each ART run-test is identified in the build
21system by a Soong module name following the `art-run-test-`*`<test-directory>`*
22format (e.g. `art-run-test-001-HelloWorld`).
23
24You can run a specific ART run-test on device by passing its Soong module name
25to Atest:
26```bash
27atest art-run-test-001-HelloWorld
28```
29
30To run all ART run-tests in a single command, the currently recommended way is
31to use [test mapping](#test-mapping) (see below).
32
33You can nonetheless run all supported ART run-tests with a single Atest command,
34using its support for wildcards:
35```bash
36atest art-run-test-\*
37```
38
39Note: Many ART run-tests are failing with the TradeFed harness as of March 2021,
40so the above Atest command will likely report many tests failures. The ART team
41is actively working on this issue.
42
43## ART gtests
44
45### Running ART gtests on device
46
47Because of current build- and link-related limitations, ART gtests can only run
48as part of the Testing ART APEX (`com.android.art.testing.apex`) on device,
49i.e. they have to be part of the ART APEX package itself to be able to build and
50run properly. This means that it is not possible to test the ART APEX presently
51residing on a device (either the original one, located in the "system"
52partition, or an updated package, present in the "data" partition).
53
54There are two ways to run ART gtests on device:
55* by installing the Testing ART APEX (i.e. manually "updating" the ART APEX on
56  device); or
57* by setting up a `chroot` environment on the device, and "activating" the
58  Testing ART APEX in that environment.
59
60### Running ART gtests on device by installing the Testing ART APEX
61
62You can run ART gtests on device with the ART APEX installation strategy by
63using the following `atest` command:
64
65```bash
66atest ArtGtestsTargetInstallApex
67```
68
69This command:
701. builds the Testing ART APEX from the Android source tree (including the ART
71   gtests);
722. installs the Testing ART APEX using `adb install`;
733. reboots the device;
744. runs the tests; and
755. uninstalls the module.
76
77You can run the tests of a single ART gtest C++ class using the
78`ArtGtestsTargetInstallApex:`*`<art-gtest-c++-class>`* syntax, e.g.:
79```bash
80atest ArtGtestsTargetInstallApex:JniInternalTest
81```
82
83This syntax also supports the use of wildcards, e.g.:
84```bash
85atest ArtGtestsTargetInstallApex:*Test*
86```
87
88You can also use Trade Federation options to run a subset of ART gtests, e.g.:
89```bash
90atest ArtGtestsTargetInstallApex -- \
91  --module ArtGtestsTargetInstallApex --test '*JniInternalTest*'
92```
93
94You can also pass option `--gtest_filter` to the gtest binary to achieve a
95similar effect:
96```bash
97atest ArtGtestsTargetInstallApex -- \
98  --test-arg com.android.tradefed.testtype.GTest:native-test-flag:"--gtest_filter=*JniInternalTest*"
99```
100
101### Running ART gtests on device using a `chroot` environment
102
103You can run ART gtests on device with the chroot-based strategy by using the
104following `atest` command:
105
106```bash
107atest ArtGtestsTargetChroot
108```
109
110This command:
1111. builds the Testing ART APEX from the Android source tree (including the ART
112   gtests) and all the necessary dependencies for the `chroot` environment;
1132. sets up a `chroot` environment on the device;
1143. "activates" the Testing ART APEX (and other APEXes that it depends on) in the
115   `chroot` environment;
1164. runs the tests within the `chroot` environment; and
1175. cleans up the environment (deactivates the APEXes and removes the `chroot`
118   environment).
119
120## Test Mapping
121
122ART Testing supports the execution of tests via [Test
123Mapping](https://source.android.com/compatibility/tests/development/test-mapping).
124The tests declared in ART's [TEST_MAPPING](../TEST_MAPPING) file are executed
125during pre-submit testing (when an ART changelist in Gerrit is verified by
126Treehugger) and/or post-submit testing (when a given change is merged in the
127Android code base), depending on the "test group" where a test is declared.
128
129### Running tests via Test Mapping with Atest
130
131It is possible to run tests via test mapping locally using Atest.
132
133To run all the tests declared in ART's `TEST_MAPPING` file, use the following
134command from the Android source tree top-level directory:
135```bash
136atest --test-mapping art:all
137```
138In the previous command, `art` is the (relative) path to the directory
139containing the `TEST_MAPPING` file listing the tests to run, while `all` means
140that tests declared in all [test
141groups](https://source.android.com/compatibility/tests/development/test-mapping#defining_test_groups)
142shall be run.
143
144To only run tests executed during pre-submit testing, use:
145```bash
146atest --test-mapping art:presubmit
147```
148
149To only run tests executed during post-submit testing, use:
150```bash
151atest --test-mapping art:postsubmit
152```
153