1# Atest 2 3Atest is a command line tool that allows users to build, install and run Android tests locally. 4This markdown will explain how to use atest on the commandline to run android tests.<br> 5 6**For instructions on writing tests [go here](https://android.googlesource.com/platform/platform_testing/+/master/docs/index.md).** 7Importantly, when writing your test's build script file (Android.mk), make sure to include 8the variable `LOCAL_COMPATIBILITY_SUITE`. A good default to use for it is `device-test`. 9 10Curious about how atest works? Want to add a feature but not sure where to begin? [Go here.](./docs/developer_workflow.md) 11Just want to learn about the overall structure? [Go here.](./docs/atest_structure.md) 12 13##### Table of Contents 141. [Environment Setup](#environment-setup) 152. [Basic Usage](#basic-usage) 163. [Identifying Tests](#identifying-tests) 174. [Specifying Steps: Build, Install or Run](#specifying-steps:-build,-install-or-run) 185. [Running Specific Methods](#running-specific-methods) 196. [Running Multiple Classes](#running-multiple-classes) 207. [Detecting Metrics Regression](#detecting-metrics-regression) 21 22 23## <a name="environment-setup">Environment Setup</a> 24 25Before you can run atest, you first have to setup your environment. 26 27##### 1. Run envsetup.sh 28From the root of the android source checkout, run the following command: 29 30`$ source build/envsetup.sh` 31 32##### 2. Run lunch 33 34Run the `$ lunch` command to bring up a "menu" of supported devices. Find the device that matches 35the device you have connected locally and run that command. 36 37For instance, if you have an ARM device connected, you would run the following command: 38 39`$ lunch aosp_arm64-eng` 40 41This will set various environment variables necessary for running atest. It will also add the 42atest command to your $PATH. 43 44## <a name="basic-usage">Basic Usage</a> 45 46Atest commands take the following form: 47 48 **atest \<optional arguments> \<tests to run>** 49 50#### \<optional arguments> 51 52<table> 53<tr><td> -b</td><td>--build</td> 54 <td>Build test targets.</td></tr> 55<tr><td> -i</td><td>--install</td> 56 <td>Install test artifacts (APKs) on device.</td></tr> 57<tr><td> -t</td><td>--test</td> 58 <td>Run the tests.</td></tr> 59<tr><td> -s</td><td>--serial</td> 60 <td>Run the tests on the specified device. Currently, one device can be tested at a time.</td></tr> 61<tr><td> -d</td><td>--disable-teardown</td> 62 <td>Disables test teardown and cleanup.</td></tr> 63<tr><td> -m</td><td>--rebuild-module-info</td> 64 <td>Forces a rebuild of the module-info.json file.</td></tr> 65<tr><td> -w</td><td>--wait-for-debugger</td> 66 <td>Only for instrumentation tests. Waits for debugger prior to execution.</td></tr> 67<tr><td> -v</td><td>--verbose</td> 68 <td>Display DEBUG level logging.</td></tr> 69<tr><td> </td><td>--generate-baseline</td> 70 <td>Generate baseline metrics, run 5 iterations by default.</td></tr> 71<tr><td> </td><td>--generate-new-metrics</td> 72 <td>Generate new metrics, run 5 iterations by default.</td></tr> 73<tr><td> </td><td>--detect-regression</td> 74 <td>Run regression detection algorithm.</td></tr> 75<tr><td> </td><td>--[CUSTOM_ARGS]</td> 76 <td>Specify custom args for the test runners.</td></tr> 77<tr><td> -h</td><td>--help</td> 78 <td>Show this help message and exit</td></tr> 79</table> 80 81More information about **-b**, **-i** and **-t** can be found below under [Specifying Steps: Build, Install or Run.](#specifying-steps:-build,-install-or-run) 82 83#### \<tests to run> 84 85 The positional argument **\<tests to run>** should be a reference to one or more 86 of the tests you'd like to run. Multiple tests can be run by separating test 87 references with spaces.<br> 88 89 Usage template: `atest <reference_to_test_1> <reference_to_test_2>` 90 91 Here are some simple examples: 92 93 `atest FrameworksServicesTests`<br> 94 `atest example/reboot`<br> 95 `atest FrameworksServicesTests CtsJankDeviceTestCases`<br> 96 `atest FrameworksServicesTests:ScreenDecorWindowTests`<br> 97 98 More information on how to reference a test can be found under [Identifying Tests.](#identifying-tests) 99 100 101## <a name="identifying-tests">Identifying Tests</a> 102 103 A \<reference_to_test> can be satisfied by the test's **Module Name**, 104**Module:Class**, **Class Name**, **TF Integration Test**, **File Path** 105or **Package Name**. 106 107 #### Module Name 108 109 Identifying a test by its module name will run the entire module. Input 110 the name as it appears in the `LOCAL_MODULE` or `LOCAL_PACKAGE_NAME` 111 variables in that test's **Android.mk** or **Android.bp** file. 112 113 Note: Use **TF Integration Test** to run non-module tests integrated directly 114into TradeFed. 115 116 Examples:<br> 117 `atest FrameworksServicesTests`<br> 118 `atest CtsJankDeviceTestCases`<br> 119 120 121 #### Module:Class 122 123 Identifying a test by its class name will run just the tests in that 124 class and not the whole module. **Module:Class** is the preferred way to run 125 a single class. **Module** is the same as described above. **Class** is the 126 name of the test class in the .java file. It can either be the fully 127 qualified class name or just the basic name. 128 129 Examples:<br> 130 131 `atest FrameworksServicesTests:ScreenDecorWindowTests`<br> 132 133 `atest FrameworksServicesTests:com.android.server.wm.ScreenDecorWindowTests`<br> 134 135 `atest CtsJankDeviceTestCases:CtsDeviceJankUi`<br> 136 137 138 #### Class Name 139 140 A single class can also be run by referencing the class name without 141 the module name. 142 143 Examples:<br> 144 `atest ScreenDecorWindowTests`<br> 145 `atest CtsDeviceJankUi`<br> 146 147 However, this will take more time than the equivalent **Module:Class** 148 reference, so we suggest using a **Module:Class** reference whenever possible. 149 Examples below are ordered by performance from the fastest to the slowest: 150 151 Examples:<br> 152 153 `atest FrameworksServicesTests:com.android.server.wm.ScreenDecorWindowTests`<br> 154 155 `atest FrameworksServicesTests:ScreenDecorWindowTests`<br> 156 157 `atest ScreenDecorWindowTests`<br> 158 159 #### TF Integration Test 160 161 To run tests that are integrated directly into TradeFed (non-modules), 162 input the name as it appears in the output of the "tradefed.sh list 163 configs" cmd. 164 165 Example:<br> 166 `atest example/reboot` (runs [this test](https://android.googlesource.com/platform/tools/tradefederation/contrib/+/master/res/config/example/reboot.xml))<br> 167 `atest native-benchmark` (runs [this test](https://android.googlesource.com/platform/tools/tradefederation/+/master/res/config/native-benchmark.xml))<br> 168 169 170 #### File Path 171 172 Both module-based tests and integration-based tests can be run by 173 inputting the path to their test file or dir as appropriate. A single 174 class can also be run by inputting the path to the class's java file. 175 Both relative and absolute paths are supported. 176 177 Example - 2 ways to run the `CtsJankDeviceTestCases` module via path:<br> 178 1. run module from android \<repo root>: `atest cts/tests/jank` 179 2. from android \<repo root>/cts/tests/jank: `atest .` 180 181 182 Example - run a specific class within `CtsJankDeviceTestCases` module via path:<br> 183 From android \<repo root>: 184 `atest cts/tests/jank/src/android/jank/cts/ui/CtsDeviceJankUi.java` 185 186 Example - run an integration test via path:<br> 187 From android \<repo root>: 188 `atest tools/tradefederation/contrib/res/config/example/reboot.xml` 189 190 191 #### Package Name 192 193 Atest supports searching tests from package name as well. 194 195 Examples:<br> 196 `atest com.android.server.wm`<br> 197 `atest android.jank.cts`<br> 198 199## <a name="specifying-steps:-build,-install-or-run">Specifying Steps: Build, Install or Run</a> 200 201The **-b**, **-i** and **-t** options allow you to specify which steps you want 202to run. If none of those options are given, then all steps are run. If any of 203these options are provided then only the listed steps are run. 204 205Note: **-i** alone is not currently support and can only be included with **-t**. 206Both **-b** and **-t** can be run alone. 207 208 209- Build targets only: `atest -b <test>` 210- Run tests only: `atest -t <test> ` 211- Install apk and run tests: `atest -it <test> ` 212- Build and run, but don't install: `atest -bt <test> ` 213 214 215Atest now has the ability to force a test to skip its cleanup/teardown step. 216Many tests, e.g. CTS, cleanup the device after the test is run, so trying 217to rerun your test with -t will fail without having the **--disable-teardown** 218parameter. Use **-d** before -t to skip the test clean up step and test 219iteratively. 220 221 `atest -d <test>`<br/> 222 `atest -t <test>` 223 224Note that -t disables both **setup/install** and **teardown/cleanup** of the device. 225So you can continue to rerun your test with just `atest -t <test>` as many times as you want. 226 227 228## <a name="running-specific-methods">Running Specific Methods</a> 229 230It is possible to run only specific methods within a test class. While the whole module will 231still need to be built, this will greatly speed up the time needed to run the tests. To run only 232specific methods, identify the class in any of the ways supported for identifying a class 233(MODULE:CLASS, FILE PATH, etc) and then append the name of the method or method using the following 234template: 235 236 `<reference_to_class>#<method1>` 237 238Multiple methods can be specified with commas: 239 240 `<reference_to_class>#<method1>,<method2>,<method3>` 241 242Examples:<br> 243 244`atest com.android.server.wm.ScreenDecorWindowTests#testMultipleDecors`<br> 245 246`atest FrameworksServicesTests:ScreenDecorWindowTests#testFlagChange,testRemoval` 247 248Here are the two preferred ways to run a single method, we're specifying the `testFlagChange` method. 249These are preferred over just the class name, because specifying the module or the java file location 250allows atest to find the test much faster: 251 2521. `atest FrameworksServicesTests:ScreenDecorWindowTests#testFlagChange` 2532. From android \<repo root>:<br/> 254`atest frameworks/base/services/tests/servicestests/src/com/android/server/wm/ScreenDecorWindowTests.java#testFlagChange` 255 256Multiple methods can be ran from different classes and modules:<br> 257 258`atest FrameworksServicesTests:ScreenDecorWindowTests#testFlagChange,testRemoval ScreenDecorWindowTests#testMultipleDecors` 259 260## <a name="running-multiple-classes">Running Multiple Classes</a> 261 262To run multiple classes, deliminate them with spaces just like you would when running multiple tests. 263Atest will handle building and running classes in the most efficient way possible, so specifying 264a subset of classes in a module will improve performance over running the whole module. 265 266Examples:<br> 267- two classes in same module:<br> 268 269`atest FrameworksServicesTests:ScreenDecorWindowTests FrameworksServicesTests:DimmerTests` 270 271- two classes, different modules:<br> 272 273`atest FrameworksServicesTests:ScreenDecorWindowTests CtsJankDeviceTestCases:CtsDeviceJankUi` 274 275 276## <a name="detecting-metrics-regression">Detecting Metrics Regression</a> 277 278Generate pre-patch or post-patch metrics without running regression detection: 279 280Examples:<br> 281 `atest <test> --generate-baseline <optional iter>` <br> 282 `atest <test> --generate-new-metrics <optional iter>` 283 284Local regression detection can be run in three options: 285 2861) Provide a folder containing baseline (pre-patch) metrics (generated previously). 287 Atest will run the tests n (default 5) iterations, generate a new set of post-patch metrics, and compare those against existing metrics. 288 289 Example:<br> 290 291 `atest <test> --detect-regression </path/to/baseline> --generate-new-metrics <optional iter>` 292 2932) Provide a folder containing post-patch metrics (generated previously). 294 Atest will run the tests n (default 5) iterations, generate a new set of pre-patch 295 metrics, and compare those against those provided. Note: the developer needs to 296 revert the device/tests to pre-patch state to generate baseline metrics. 297 298 Example:<br> 299 300 `atest <test> --detect-regression </path/to/new> --generate-baseline <optional iter>` 301 3023) Provide 2 folders containing both pre-patch and post-patch metrics. Atest will run no tests but 303 the regression detection algorithm. 304 305 Example:<br> 306 307 `atest --detect-regression </path/to/baseline> </path/to/new>` 308 309 310