1# ART Testing 2 3There are two suites of tests in the Android Runtime (ART): 4* _ART run-tests_: Tests of the ART runtime using Dex bytecode (mostly written 5 in Java). 6* _ART gtests_: C++ tests exercising various aspects of ART. 7 8## ART run-tests 9 10ART run-tests are tests exercising the runtime using Dex bytecode. They are 11written in Java and/or [Smali](https://github.com/JesusFreke/smali) 12(compiled/assembled as Dex bytecode) and sometimes native code (written as C/C++ 13testing libraries). Some tests also make use of the 14[Jasmin](http://jasmin.sourceforge.net/) assembler or the 15[ASM](https://asm.ow2.io/) bytecode manipulation tool. Run-tests are 16executed on the ART runtime (`dalvikvm`), possibly preceded by a 17pre-optimization of the Dex code (using `dex2oat`). 18 19The run-tests are identified by directories in this `test` directory, named with 20a numeric prefix and containing an `info.txt` file. For most run tests, the 21sources are in the `src` subdirectory. Sources found in the `src2` directory are 22compiled separately but to the same output directory; this can be used to 23exercise "API mismatch" situations by replacing class files created in the first 24pass. The `src-ex` directory is built separately, and is intended for exercising 25class loaders. Resources can be stored in the `res` directory, which is 26distributed together with the executable files. 27 28The run-tests logic lives in the `test/run-test` Bash script. The execution of a 29run-test has three main parts: building the test, running the test, and checking 30the test's output. By default, these three steps are implemented by three Bash 31scripts located in the `test/etc` directory (`default-build`, `default-run`, and 32`default-check`). These scripts rely on environment variables set by 33`test/run-test`. 34 35The default logic for all of these these steps (build, run, check) is overridden 36if the test's directory contains a Bash script named after the step 37(i.e. `build`, `run`, or `check`). Note that the default logic of the "run" step 38is actually implemented in the "JAR runner" (`test/etc/run-test-jar`), invoked 39by `test/etc/default-run`. 40 41After the execution of a run-test, the check step's default behavior 42(implemented in `test/etc/default-check`) is to respectively compare its 43standard output and standard error with the contents of the 44`expected-stdout.txt` and `expected-stderr.txt` files contained in the test's 45directory; any mismatch triggers a test failure. 46 47The `test/run-test` script handles the execution of a single run-test in a given 48configuration. The Python script `test/testrunner/testrunner.py` is a convenient 49script handling the construction and execution of multiple tests in one 50configuration or more. 51 52To see the invocation options supported by `run-test` and `testrunner.py`, run 53these commands from the Android source top-level directory: 54```sh 55art/test/run-test --help 56``` 57```sh 58art/test/testrunner/testrunner.py --help 59``` 60 61## ART gtests 62 63ART gtests are written in C++ using the [Google 64Test](https://github.com/google/googletest) framework. These tests exercise 65various aspects of the runtime (the logic in `libart`, `libart-compiler`, etc.) 66and its binaries (`dalvikvm`, `dex2oat`, `oatdump`, etc.). Some of them are used 67as unit tests to verify a particular construct in ART. These tests may depend on 68some test Dex files and core images. 69 70ART gtests are defined in various directories within the ART project (usually in 71the same directory as the code they exercise). Their source files usually end 72with the suffix `_test.cc`. The construction logic of these tests is implemented 73in ART's build system (`Android.bp` and `Android*.mk` files). On host, these 74gtests can be run by executing `m test-art-host-gtest`. On device, the 75recommended approach is to run these tests in a chroot environment (see 76`README.chroot.md` in this directory). 77 78 79# Test execution 80 81All tests in either suite can be run using the `art/test.py` 82script. Additionally, run-tests can be run individually. All of the tests can be 83run on the build host, on a USB-attached device, or using the build host 84"reference implementation". 85 86ART also supports running target (device) tests in a chroot environment (see 87`README.chroot.md` in this directory). This is currently the recommended way to 88run tests on target (rather than using `art/test.py --target`). 89 90To see command flags run: 91 92```sh 93$ art/test.py -h 94``` 95 96## Building tests 97 98In general all tests require some dependencies to be built before they can be run. 99In general you can pass the `--build-dependencies` flag (also available as short 100option -b) to `art/test.py` program to automatically build required dependencies. 101One can also directly use the various `test-art-...-dependencies` targets listed 102below. 103 104## Running all tests on the build host 105 106```sh 107$ # Build test files 108$ m test-art-host-run-test-dependencies 109$ # Run the tests 110$ art/test.py --host 111``` 112 113Or: 114 115``` 116$ art/test.py -b --host 117``` 118 119## Running all tests on the target device 120 121```sh 122$ # Build test files 123$ m test-art-target-run-test-dependencies 124$ # Run the tests 125$ art/test.py --target 126``` 127 128Or: 129 130``` 131$ art/test.py -b --target 132``` 133 134## Running all gtests on the build host 135 136```sh 137$ art/test.py --host -g 138``` 139 140## Running all gtests on the target device 141 142```sh 143$ art/test.py --target -g 144``` 145 146## Running all run-tests on the build host 147 148```sh 149$ # Build test files 150$ m test-art-host-run-test-dependencies 151$ art/test.py --host -r 152``` 153 154Or: 155 156``` 157$ art/test.py -b --host -r 158``` 159 160## Running all run-tests on the target device 161 162```sh 163$ art/test.py --target -r 164``` 165 166## Running one run-test on the build host 167 168```sh 169$ # Build test files 170$ m test-art-host-run-test-dependencies 171$ # Run the tests 172$ art/test.py --host -r -t 001-HelloWorld 173``` 174 175Or: 176 177``` 178$ art/test.py -b --host -r -t 001-HelloWorld 179``` 180 181## Running one run-test on the target device 182 183```sh 184$ art/test.py --target -r -t 001-HelloWorld 185``` 186 187## Running one gtest on the build host 188 189```sh 190$ find out/host/ -type f -name art_runtime_tests # Find the path of the test. 191$ out/host/linux-x86/nativetest/art_runtime_tests/art_runtime_tests 192``` 193 194Add "--no_isolate" to run the tests one by one in single process (disable forking). 195Add "--gtest_filter=..." to select specific sub-test(s) to run. 196Prefix by "gdb --args " to run the test in gdb. 197 198# ART Continuous Integration 199 200Both ART run-tests and gtests are run continuously as part of [ART's continuous 201integration](https://ci.chromium.org/p/art/g/luci/console). In addition, two 202other test suites are run continuously on this service: Libcore tests and JDWP 203tests. 204