1# ART Testing
2
3There are two suites of tests in the Android Runtime (ART):
4* _ART run-tests_: Tests of the ART runtime using Dex bytecode (mostly written
5  in Java).
6* _ART gtests_: C++ tests exercising various aspects of ART.
7
8## ART run-tests
9
10ART run-tests are tests exercising the runtime using Dex bytecode. They are
11written in Java and/or [Smali](https://github.com/JesusFreke/smali)
12(compiled/assembled as Dex bytecode) and sometimes native code (written as C/C++
13testing libraries). Some tests also make use of the
14[Jasmin](http://jasmin.sourceforge.net/) assembler or the
15[ASM](https://asm.ow2.io/) bytecode manipulation tool. Run-tests are
16executed on the ART runtime (`dalvikvm`), possibly preceded by a
17pre-optimization of the Dex code (using `dex2oat`).
18
19The run-tests are identified by directories in this `test` directory, named with
20a numeric prefix and containing an `info.txt` file. For most run tests, the
21sources are in the `src` subdirectory. Sources found in the `src2` directory are
22compiled separately but to the same output directory; this can be used to
23exercise "API mismatch" situations by replacing class files created in the first
24pass. The `src-ex` directory is built separately, and is intended for exercising
25class loaders.  Resources can be stored in the `res` directory, which is
26distributed together with the executable files.
27
28The run-tests logic lives in the `test/run-test` Bash script. The execution of a
29run-test has three main parts: building the test, running the test, and checking
30the test's output. By default, these three steps are implemented by three Bash
31scripts located in the `test/etc` directory (`default-build`, `default-run`, and
32`default-check`). These scripts rely on environment variables set by
33`test/run-test`.
34
35The default logic for all of these these steps (build, run, check) is overridden
36if the test's directory contains a Bash script named after the step
37(i.e. `build`, `run`, or `check`). Note that the default logic of the "run" step
38is actually implemented in the "JAR runner" (`test/etc/run-test-jar`), invoked
39by `test/etc/default-run`.
40
41After the execution of a run-test, the check step's default behavior
42(implemented in `test/etc/default-check`) is to respectively compare its
43standard output and standard error with the contents of the
44`expected-stdout.txt` and `expected-stderr.txt` files contained in the test's
45directory; any mismatch triggers a test failure.
46
47The `test/run-test` script handles the execution of a single run-test in a given
48configuration. The Python script `test/testrunner/testrunner.py` is a convenient
49script handling the construction and execution of multiple tests in one
50configuration or more.
51
52To see the invocation options supported by `run-test` and `testrunner.py`, run
53these commands from the Android source top-level directory:
54```sh
55art/test/run-test --help
56```
57```sh
58art/test/testrunner/testrunner.py --help
59```
60
61## ART gtests
62
63ART gtests are written in C++ using the [Google
64Test](https://github.com/google/googletest) framework. These tests exercise
65various aspects of the runtime (the logic in `libart`, `libart-compiler`, etc.)
66and its binaries (`dalvikvm`, `dex2oat`, `oatdump`, etc.). Some of them are used
67as unit tests to verify a particular construct in ART. These tests may depend on
68some test Dex files and core images.
69
70ART gtests are defined in various directories within the ART project (usually in
71the same directory as the code they exercise). Their source files usually end
72with the suffix `_test.cc`. The construction logic of these tests is implemented
73in ART's build system (`Android.bp` and `Android*.mk` files). On host, these
74gtests can be run by executing `m test-art-host-gtest`. On device, the
75recommended approach is to run these tests in a chroot environment (see
76`README.chroot.md` in this directory).
77
78
79# Test execution
80
81All tests in either suite can be run using the `art/test.py`
82script. Additionally, run-tests can be run individually. All of the tests can be
83run on the build host, on a USB-attached device, or using the build host
84"reference implementation".
85
86ART also supports running target (device) tests in a chroot environment (see
87`README.chroot.md` in this directory). This is currently the recommended way to
88run tests on target (rather than using `art/test.py --target`).
89
90To see command flags run:
91
92```sh
93$ art/test.py -h
94```
95
96## Building tests
97
98In general all tests require some dependencies to be built before they can be run.
99In general you can pass the `--build-dependencies` flag (also available as short
100option -b) to `art/test.py` program to automatically build required dependencies.
101One can also directly use the various `test-art-...-dependencies` targets listed
102below.
103
104## Running all tests on the build host
105
106```sh
107$ # Build test files
108$ m test-art-host-run-test-dependencies
109$ # Run the tests
110$ art/test.py --host
111```
112
113Or:
114
115```
116$ art/test.py -b --host
117```
118
119## Running all tests on the target device
120
121```sh
122$ # Build test files
123$ m test-art-target-run-test-dependencies
124$ # Run the tests
125$ art/test.py --target
126```
127
128Or:
129
130```
131$ art/test.py -b --target
132```
133
134## Running all gtests on the build host
135
136```sh
137$ art/test.py --host -g
138```
139
140## Running all gtests on the target device
141
142```sh
143$ art/test.py --target -g
144```
145
146## Running all run-tests on the build host
147
148```sh
149$ # Build test files
150$ m test-art-host-run-test-dependencies
151$ art/test.py --host -r
152```
153
154Or:
155
156```
157$ art/test.py -b --host -r
158```
159
160## Running all run-tests on the target device
161
162```sh
163$ art/test.py --target -r
164```
165
166## Running one run-test on the build host
167
168```sh
169$ # Build test files
170$ m test-art-host-run-test-dependencies
171$ # Run the tests
172$ art/test.py --host -r -t 001-HelloWorld
173```
174
175Or:
176
177```
178$ art/test.py -b --host -r -t 001-HelloWorld
179```
180
181## Running one run-test on the target device
182
183```sh
184$ art/test.py --target -r -t 001-HelloWorld
185```
186
187## Running one gtest on the build host
188
189```sh
190$ find out/host/ -type f -name art_runtime_tests  # Find the path of the test.
191$ out/host/linux-x86/nativetest/art_runtime_tests/art_runtime_tests
192```
193
194Add "--no_isolate" to run the tests one by one in single process (disable forking).
195Add "--gtest_filter=..." to select specific sub-test(s) to run.
196Prefix by "gdb --args " to run the test in gdb.
197
198# ART Continuous Integration
199
200Both ART run-tests and gtests are run continuously as part of [ART's continuous
201integration](https://ci.chromium.org/p/art/g/luci/console). In addition, two
202other test suites are run continuously on this service: Libcore tests and JDWP
203tests.
204