1# Running ART Tests with Atest / Trade Federation 2 3ART Testing has early support for execution in the [Trade 4Federation](https://source.android.com/devices/tech/test_infra/tradefed) 5("TradeFed") test harness, in particular via the 6[Atest](https://source.android.com/compatibility/tests/development/atest) 7command line tool. 8 9Atest conveniently takes care of building tests and their dependencies (using 10Soong, the Android build system) and executing them using Trade Federation. 11 12See also [README.md](README.md) for a general introduction to ART run-tests and 13gtests. 14 15## ART run-tests 16 17### Running ART run-tests on device 18 19ART run-tests are defined in sub-directories of `test/` starting with a number 20(e.g. `test/001-HelloWorld`). Each ART run-test is identified in the build 21system by a Soong module name following the `art-run-test-`*`<test-directory>`* 22format (e.g. `art-run-test-001-HelloWorld`). 23 24You can run a specific ART run-test on device by passing its Soong module name 25to Atest: 26```bash 27atest art-run-test-001-HelloWorld 28``` 29 30To run all ART run-tests in a single command, the currently recommended way is 31to use [test mapping](#test-mapping) (see below). 32 33You can nonetheless run all supported ART run-tests with a single Atest command, 34using its support for wildcards: 35```bash 36atest art-run-test-\* 37``` 38 39Note: Many ART run-tests are failing with the TradeFed harness as of March 2021, 40so the above Atest command will likely report many tests failures. The ART team 41is actively working on this issue. 42 43## ART gtests 44 45### Running ART gtests on device 46 47Because of current build- and link-related limitations, ART gtests can only run 48as part of the Testing ART APEX (`com.android.art.testing.apex`) on device, 49i.e. they have to be part of the ART APEX package itself to be able to build and 50run properly. This means that it is not possible to test the ART APEX presently 51residing on a device (either the original one, located in the "system" 52partition, or an updated package, present in the "data" partition). 53 54There are two ways to run ART gtests on device: 55* by installing the Testing ART APEX (i.e. manually "updating" the ART APEX on 56 device); or 57* by setting up a `chroot` environment on the device, and "activating" the 58 Testing ART APEX in that environment. 59 60### Running ART gtests on device by installing the Testing ART APEX 61 62You can run ART gtests on device with the ART APEX installation strategy by 63using the following `atest` command: 64 65```bash 66atest ArtGtestsTargetInstallApex 67``` 68 69This command: 701. builds the Testing ART APEX from the Android source tree (including the ART 71 gtests); 722. installs the Testing ART APEX using `adb install`; 733. reboots the device; 744. runs the tests; and 755. uninstalls the module. 76 77You can run the tests of a single ART gtest C++ class using the 78`ArtGtestsTargetInstallApex:`*`<art-gtest-c++-class>`* syntax, e.g.: 79```bash 80atest ArtGtestsTargetInstallApex:JniInternalTest 81``` 82 83This syntax also supports the use of wildcards, e.g.: 84```bash 85atest ArtGtestsTargetInstallApex:*Test* 86``` 87 88You can also use Trade Federation options to run a subset of ART gtests, e.g.: 89```bash 90atest ArtGtestsTargetInstallApex -- \ 91 --module ArtGtestsTargetInstallApex --test '*JniInternalTest*' 92``` 93 94You can also pass option `--gtest_filter` to the gtest binary to achieve a 95similar effect: 96```bash 97atest ArtGtestsTargetInstallApex -- \ 98 --test-arg com.android.tradefed.testtype.GTest:native-test-flag:"--gtest_filter=*JniInternalTest*" 99``` 100 101### Running ART gtests on device using a `chroot` environment 102 103You can run ART gtests on device with the chroot-based strategy by using the 104following `atest` command: 105 106```bash 107atest ArtGtestsTargetChroot 108``` 109 110This command: 1111. builds the Testing ART APEX from the Android source tree (including the ART 112 gtests) and all the necessary dependencies for the `chroot` environment; 1132. sets up a `chroot` environment on the device; 1143. "activates" the Testing ART APEX (and other APEXes that it depends on) in the 115 `chroot` environment; 1164. runs the tests within the `chroot` environment; and 1175. cleans up the environment (deactivates the APEXes and removes the `chroot` 118 environment). 119 120## Test Mapping 121 122ART Testing supports the execution of tests via [Test 123Mapping](https://source.android.com/compatibility/tests/development/test-mapping). 124The tests declared in ART's [TEST_MAPPING](../TEST_MAPPING) file are executed 125during pre-submit testing (when an ART changelist in Gerrit is verified by 126Treehugger) and/or post-submit testing (when a given change is merged in the 127Android code base), depending on the "test group" where a test is declared. 128 129### Running tests via Test Mapping with Atest 130 131It is possible to run tests via test mapping locally using Atest. 132 133To run all the tests declared in ART's `TEST_MAPPING` file, use the following 134command from the Android source tree top-level directory: 135```bash 136atest --test-mapping art:all 137``` 138In the previous command, `art` is the (relative) path to the directory 139containing the `TEST_MAPPING` file listing the tests to run, while `all` means 140that tests declared in all [test 141groups](https://source.android.com/compatibility/tests/development/test-mapping#defining_test_groups) 142shall be run. 143 144To only run tests executed during pre-submit testing, use: 145```bash 146atest --test-mapping art:presubmit 147``` 148 149To only run tests executed during post-submit testing, use: 150```bash 151atest --test-mapping art:postsubmit 152``` 153