1# ART Testing
2
3There are two suites of tests in the Android Runtime (ART):
4* _ART run-tests_: Tests of the ART runtime using Dex bytecode (mostly written
5  in Java).
6* _ART gtests_: C++ tests exercising various aspects of ART.
7
8## ART run-tests
9
10ART run-tests are tests exercising the runtime using Dex bytecode. They are
11written in Java and/or [Smali](https://github.com/JesusFreke/smali)
12(compiled/assembled as Dex bytecode) and sometimes native code (written as C/C++
13testing libraries). Some tests also make use of the
14[Jasmin](http://jasmin.sourceforge.net/) assembler or the
15[ASM](https://asm.ow2.io/) bytecode manipulation tool. Run-tests are
16executed on the ART runtime (`dalvikvm`), possibly preceded by a
17pre-optimization of the Dex code (using `dex2oat`).
18
19The run-tests are identified by directories in this `test` directory, named with
20a numeric prefix and containing an `info.txt` file. For most run tests, the
21sources are in the `src` subdirectory. Sources found in the `src2` directory are
22compiled separately but to the same output directory; this can be used to
23exercise "API mismatch" situations by replacing class files created in the first
24pass. The `src-ex` directory is built separately, and is intended for exercising
25class loaders.  Resources can be stored in the `res` directory, which is
26distributed together with the executable files.
27
28The run-tests logic lives in the `test/run-test` Bash script. The execution of a
29run-test has three main parts: building the test, running the test, and checking
30the test's output. By default, these three steps are implemented by three Bash
31scripts located in the `test/etc` directory (`default-build`, `default-run`, and
32`default-check`). These scripts rely on environment variables set by
33`test/run-test`.
34
35The default logic for all of these these steps (build, run, check) is overridden
36if the test's directory contains a Bash script named after the step
37(i.e. `build`, `run`, or `check`). Note that the default logic of the "run" step
38is actually implemented in the "JAR runner" (`test/etc/run-test-jar`), invoked
39by `test/etc/default-run`.
40
41After the execution of a run-test, the check step's default behavior
42(implemented in `test/etc/default-check`) is to compare its standard output with
43the contents of the `expected.txt` file contained in the test's directory; any
44mismatch triggers a test failure.
45
46The `test/run-test` script handles the execution of a single run-test in a given
47configuration. The Python script `test/testrunner/testrunner.py` is a convenient
48script handling the construction and execution of multiple tests in one
49configuration or more.
50
51To see the invocation options supported by `run-test` and `testrunner.py`, run
52these commands from the Android source top-level directory:
53```sh
54art/test/run-test --help
55```
56```sh
57art/test/testrunner/testrunner.py --help
58```
59
60## ART gtests
61
62ART gtests are written in C++ using the [Google
63Test](https://github.com/google/googletest) framework. These tests exercise
64various aspects of the runtime (the logic in `libart`, `libart-compiler`, etc.)
65and its binaries (`dalvikvm`, `dex2oat`, `oatdump`, etc.). Some of them are used
66as unit tests to verify a particular construct in ART. These tests may depend on
67some test Dex files and core images.
68
69ART gtests are defined in various directories within the ART project (usually in
70the same directory as the code they exercise). Their source files usually end
71with the suffix `_test.cc`. The construction logic of these tests is implemented
72in ART's build system (`Android.bp` and `Android*.mk` files). On host, these
73gtests can be run by executing `m test-art-host-gtest`. On device, the
74recommended approach is to run these tests in a chroot environment (see
75`README.chroot.md` in this directory).
76
77
78# Test execution
79
80All tests in either suite can be run using the `art/test.py`
81script. Additionally, run-tests can be run individually. All of the tests can be
82run on the build host, on a USB-attached device, or using the build host
83"reference implementation".
84
85ART also supports running target (device) tests in a chroot environment (see
86`README.chroot.md` in this directory). This is currently the recommended way to
87run tests on target (rather than using `art/test.py --target`).
88
89To see command flags run:
90
91```sh
92$ art/test.py -h
93```
94
95## Running all tests on the build host
96
97```sh
98$ art/test.py --host
99```
100
101## Running all tests on the target device
102
103```sh
104$ art/test.py --target
105```
106
107## Running all gtests on the build host
108
109```sh
110$ art/test.py --host -g
111```
112
113## Running all gtests on the target device
114
115```sh
116$ art/test.py --target -g
117```
118
119## Running all run-tests on the build host
120
121```sh
122$ art/test.py --host -r
123```
124
125## Running all run-tests on the target device
126
127```sh
128$ art/test.py --target -r
129```
130
131## Running one run-test on the build host
132
133```sh
134$ art/test.py --host -r -t 001-HelloWorld
135```
136
137## Running one run-test on the target device
138
139```sh
140$ art/test.py --target -r -t 001-HelloWorld
141```
142
143
144# ART Continuous Integration
145
146Both ART run-tests and gtests are run continuously as part of [ART's continuous
147integration](https://ci.chromium.org/p/art/g/luci/console). In addition, two
148other test suites are run continuously on this service: Libcore tests and JDWP
149tests.
150