1# Autotest for Chromium OS developers 2 3[TOC] 4 5## Useful documents 6 7[Autotest documentation on GitHub](https://github.com/autotest/autotest/wiki/AutotestApi): 8This would be a good read if you want to familiarize yourself with the basic 9Autotest concepts. 10 11[Gentoo Portage ebuild/eclass Information](http://www.gentoo.org/proj/en/devrel/handbook/handbook.xml?part=2): 12Getting to know the package build system we use. 13 14[ChromiumOS specific Portage FAQ](http://www.chromium.org/chromium-os/how-tos-and-troubleshooting/portage-build-faq): 15Learning something about the way we use portage. 16 17## Autotest and ebuild workflow 18 19To familiarize with autotest concepts, you should start with the upstream 20Autotest documentation at: https://github.com/autotest/autotest/wiki/AutotestApi 21 22The rest of this document is going to use some terms and only explain them 23vaguely. 24 25### Overview 26 27At a high level, tests are organized in test cases, each test case being either 28server or client, with one main .py file named the same as the test case, and 29one or more control files. In order to be able to perform all tasks on a given 30test, autotest expects tests to be placed in a monolithic file structure 31of: 32 33- `/client/tests/` 34- `/client/site_tests/` 35- `/server/tests/` 36- `/server/site_tests/` 37 38Each test directory has to have at least a control file, but typically also has 39a main job module (named the same as the test case). Furthermore, if it needs 40any additional files checked in, they are typically placed in a `files/` 41directory, and separate projects that can be built with a Makefile inside the 42`src/` directory. 43 44Due to structural limitations in Chromium OS, it is not possible to store all 45test cases in this structure in a single large source repository as upstream 46autotest source would (placed at `third_party/autotest/files/` in Chromium OS). 47In particular, the following has been required in the past: 48 49- Having confidential (publicly inaccessible) tests or generally per-test ACLs 50 for sharing only with a particular partner only. 51- Storing test cases along with the project they wrap around, because the test 52 requires binaries built as a by-product of the project’s own build system. 53 (e.g. chrome or tpm tests) 54 55Furthermore, it has been desired to generally build everything that is not 56strongly ordered in parallel, significantly decreasing build times. That, 57however, requires proper dependency tree declaration and being able to specify 58which test cases require what dependencies, in addition to being able to 59process different "independent" parts of a single source repository in 60parallel. 61 62This leads to the ebuild workflow, which generally allows compositing any 63number of sources in any format into a single monolithic tree, whose contents 64depend on build parameters. 65 66 67 68This allows using standard autotest workflow without any change, however, 69unlike what upstream does, the tests aren’t run directly from the source 70repository, rather from a staging read-only install location. This leads to 71certain differences in workflow: 72 73- Source may live in an arbitrary location or can be generated on the fly. 74 Anything that can be created as an ebuild (shell script) can be a test source. 75 (cros-workon may be utilised, introducing a fairly standard Chromium OS 76 project workflow) 77- The staging location (`/build/${board}/usr/local/autotest/`) may not be 78 modified; if one wants to modify it, they have to find the source to it 79 (using other tools, see FAQ). 80- Propagating source changes requires an emerge step. 81 82### Ebuild setup, autotest eclass 83 84**NOTE**: This assumes some basic knowledge of how ebuilds in Chromium OS work. 85Further documentation is available at http://www.chromium.org/chromium-os/how-tos-and-troubleshooting/portage-build-faq 86 87An **autotest ebuild** is an ebuild that produces test cases and installs them into 88the staging area. It has three general tasks: 89 90- Obtain the source - This is generally (but not necessarily) provided by 91 ‘cros-workon’ eclass. It could also work with the more standard tarball 92 SRC_URI pathway or generally any shell code executed in `src_unpack()`. 93- Prepare test cases - This includes, but is not limited to preprocessing any 94 source, copying source files or intermediate binaries into the expected 95 locations, where they will be taken over by autotest code, specifically the 96 `setup()` function of the appropriate test. Typically, this is not needed. 97- Call autotest to "build" all sources and subsequently install them - This 98 should be done exclusively by inheriting the **autotest eclass**, which 99 bundles up all the necessary code to install into the intermediate location. 100 101**Autotest eclass** is inherited by all autotest ebuilds, only requires a 102number of variables specified and works by itself otherwise. Most variables 103describe the locations and listings of work that needs to be done: 104 105- Location variables define the paths to directories containing the test 106files: 107 108 - `AUTOTEST_{CLIENT,SERVER}_{TESTS,SITE_TESTS}` 109 - `AUTOTEST_{DEPS,PROFILERS,CONFIG}` 110 111 These typically only need to be specified if they differ from the defaults 112 (which follow the upstream directory structure) 113 114- List variables (`AUTOTEST_*_LIST`) define the list of deps, profilers, 115 configs that should be handled by this ebuild. 116- IUSE test list specification TESTS=, is a USE_EXPANDed specification of 117 tests managed by the given ebuild. By virtue of being an IUSE variable, all 118 of the options are visible as USE flag toggles while building the ebuild, 119 unlike with list variables which are a given and the ebuild has to be 120 modified for those to change. 121 122Each ebuild usually operates on a single source repository. That does not 123always have to hold true, however, and in case of autotest, many ebuilds check 124out the sources of the same source repository (*autotest.git*). Invariably, this 125means that they have to be careful to not install the same files and split the 126sources between themselves to avoid file install collisions. 127If more than one autotest ebuild operates on the same source repository, they 128**have to** use the above variables to define mutually exclusive slices in order 129to not collide during installation. Generally, if we have a source repository 130with client site_tests A and B, you can have either: 131 132- one ebuild with IUSE_TESTS="+tests_A +tests_B" 133- two different ebuilds, one with IUSE_TESTS="+tests_A", the other with 134 IUSE_TESTS="+tests_B" 135 136As soon as an overlap between ebuilds happens, either an outside mechanism has 137to ensure the overlapping tests are never enabled at the same time, or file 138collisions happen. 139 140 141## Building tests 142 143Fundamentally, a test has two main phases: 144 145- `run_*()` - This is is the main part that performs all testing and is 146 invoked by the control files, once or repeatedly. 147- `setup()` - This function, present in the test case’s main .py file is 148 supposed to prepare the test for running. This includes building any 149 binaries, initializing data, etc. 150 151During building using emerge, autotest will call a `setup()` function of all 152test cases/deps involved. This is supposed to prepare everything. Typically, 153this will invoke make on a Makefile present in the test’s src/ directory, but 154can involve any other transformation of sources (also be empty if there’s 155nothing to build). 156**Note**, however, that `setup()` is implicitly called many times as test 157initialization even during `run_*()` step, so it should be a noop on reentry 158that merely verifies everything is in order. 159 160Unlike `run_*()` functions, `setup()` gets called both during the prepare phase 161which happens on the **host and target alike**. This creates a problem with 162code that is being depended on or directly executed during `setup()`. Python 163modules that are imported in any pathway leading to `setup()` are needed both 164in the host chroot and on the target board to properly support the test. Any 165binaries would need to be compiled using the host compiler and either ensured 166that they will be skipped on the target (incremental `setup()` runs) or 167cross-compiled again and dynamically chosen while running on target. 168 169**More importantly**, in Chromium OS scenario, doing any write operations 170inside the `setup()` function will lead to **access denied failures**, because 171tests are being run from the intermediate read-only location. 172 173Given the above, building is as easy as **emerge**-ing the autotest ebuild that 174contains our test. 175``` 176$ emerge-${board} ${test_ebuild} 177``` 178 179*Currently, tests are organized within these notable ebuilds*: (see 180[FAQ](#Q1_What-autotest-ebuilds-are-out-there_) full list): 181 182- chromeos-base/autotest-tests - The main ebuild handling most of autotest.git 183 repository and its client and server tests. 184- chromeos-base/autotest-tests-* - Various ebuilds that build other parts of 185 autotest.git 186- chromeos-base/chromeos-chrome - chrome tests; the tests that are part of 187 chrome 188 189### Building tests selectively 190 191Test cases built by ebuilds generally come in large bundles. Sometimes, only a 192subset, or generally a different set of the tests provided by a given ebuild is 193desired. That is achieved using a 194[USE_EXPANDed](http://devmanual.gentoo.org/general-concepts/use-flags/index.html) 195flag called TESTS. 196 197All USE flags (and therefore tests) have a default state, either enabled (+) or 198disabled (-), specified directly in the ebuild, that can be manually overridden 199from the commandline. There are two ways to do that. 200 201- Non-Incremental - Simply override the default selection by an entirely new 202 selection, ignoring the defaults. This is useful if you develop a single 203 test and don’t want to waste time building the others. 204 205 $ TESTS="test1 test2" emerge-${board} ${ebuild} 206 207- Incremental - All USE_EXPAND flags are also accessible as USE flags, with 208 the appropriate prefix, and can be used incrementally to selectively 209 enable/disable tests in addition to the defaults. This can be useful if you 210 aim to enable a test that is disabled by default and want to test locally. 211 212 $ USE="test_to_be_enabled -test_to_be_disabled" emerge-${board} \ 213 ${ebuild} 214 215For operations across all tests, following incremental USE wildcard is 216supported by portage: "tests_*" to select all tests at once (or - to 217de-select). 218 219**NOTE**: Both Incremental and Non-Incremental methods can be set/overriden by 220(in this order): the ebuild (default values), make.profile, make.conf, 221/etc/portage, commandline (see above). That means that any settings provided on 222the emerge commandline override everything else. 223 224## Running tests 225 226**NOTE**: In order to run tests on your device, it needs to have a 227[test-enabled image](#W4_Create-and-run-a-test-enabled-image-on-your-device). 228 229When running tests, fundamentally, you want to either: 230 231- Run sets of tests manually - Use case: Developing test cases 232 233 Take your local test sources, modify them, and then attempt to run them on a 234 target machine using autotest. You are generally responsible for making sure 235 that the machine is imaged to a test image, and the image contains all the 236 dependencies needed to support your tests. 237 238- Verify a given image - Use case: Developing the projects subject to testing 239 240 Take an image, re-image the target device and run a test suite on it. This 241 requires either use of build-time autotest artifacts or their reproduction 242 by not modifying or resyncing your sources after an image has been built. 243 244### Running tests on a machine 245 246Autotests are run with a tool called 247[test_that](https://chromium.googlesource.com/chromiumos/third_party/autotest/+/refs/heads/master/docs/test-that.md). 248 249### Running tests in a VM - cros_run_vm_tests 250 251VM tests are conveniently wrapped into a script `cros_run_vm_tests` that sets up 252the VM using a given image and then calls `test_that`. This is run by builders 253to test using the Smoke suite. 254 255If you want to run your tests in a VM (see 256[here](https://chromium.googlesource.com/chromiumos/docs/+/master/cros_vm.md#Run-an-autotest-in-the-VM) 257 258- `cros_run_vm_test` starts up a VM and runs autotests using the port 259- specified (defaults to 9222). As an example: 260 261 $ cros_run_vm_test --autotest=suite:smoke \ 262 --image-path=<my_image_to_start or don't set to use most recent build> \ 263 --board=amd64-generic 264 265- The emulator command line redirects localhost port 9222 to the emulated 266 machine's port 22 to allow you to ssh into the emulator. For Chromium OS to 267 actually listen on this port you must append the `--test_image` parameter 268 when you run the `./image_to_vm.sh` script, or perhaps run the 269 `mod_image_for_test.sh` script instead. 270- You can then run tests on the correct ssh port with something like 271 272 $ test_that --board=x86-generic localhost:9222 'f:.*platform_BootPerf/control' 273 274- To set the sudo password run set_shared_user_password. Then within the 275 emulator you can press Ctrl-Alt-T to get a terminal, and sudo using this 276 password. This will also allow you to ssh into the unit with, e.g. 277 278 $ ssh -p 9222 root@localhost 279 280- Warning: After 281 [crbug/710629](https://bugs.chromium.org/p/chromium/issues/detail?id=710629), 282 'betty' is the only board regularly run through pre-CQ and CQ VMTest and so 283 is the most likely to work at ToT. 'betty' is based on 'amd64-generic', 284 so 'amd64-generic' is likely to also work for most (non-ARC) tests. 285 286 287## Result log layout structure 288 289For information regarding the layout structure please refer to the following: 290[autotest-results-logs](https://www.chromium.org/chromium-os/testing/test-code-labs/autotest-client-tests/autotest-results-logs) 291 292### Interpreting test results 293 294Running autotest will result in a lot of information going by which is probably 295not too informative if you have not used autotest before. At the end of the 296`test_that` run, you will see a summary of pass/failure status, along with 297performance results: 298 299``` 30022:44:30 INFO | Using installation dir /home/autotest 30122:44:30 ERROR| Could not install autotest from repos 30222:44:32 INFO | Installation of autotest completed 30322:44:32 INFO | GOOD ---- Autotest.install timestamp=1263509072 localtime=Jan 14 22:44:32 30422:44:33 INFO | Executing /home/autotest/bin/autotest /home/autotest/control phase 0 30522:44:36 INFO | START ---- ---- timestamp=1263509075 localtime=Jan 14 14:44:35 30622:44:36 INFO | START sleeptest sleeptest timestamp=1263509076 localtime=Jan 14 14:44:36 30722:44:36 INFO | Bundling /usr/local/autotest/client/tests/sleeptest into test-sleeptest.tar.bz2 30822:44:40 INFO | GOOD sleeptest sleeptest timestamp=1263509079 localtime=Jan 14 14:44:39 completed successfully 30922:44:40 INFO | END GOOD sleeptest sleeptest timestamp=1263509079 localtime=Jan 14 14:44:39 31022:44:42 INFO | END GOOD ---- ---- timestamp=1263509082 localtime=Jan 14 14:44:42 31122:44:44 INFO | Client complete 31222:44:45 INFO | Finished processing control file 313``` 314 315`test_that` will leave around a temp directory populated with diagnostic information: 316 317``` 318Finished running tests. Results can be found in /tmp/test_that_results_j8GoWH or /tmp/test_that_latest 319``` 320 321This directory will contain a directory per test run. Each directory contains 322the logs pertaining to that test run. 323 324In that directory some interesting files are: 325 326${TEST}/debug/client.DEBUG - the most detailed output from running the 327client-side test 328 329### Running tests automatically, Suites 330 331Suites provide a mechanism to group tests together in test groups. They also 332serve as hooks for automated runs of tests verifying various builds. Most 333importantly, that is the BVT (board verification tests) and Smoke (a subset of 334BVT that can run in a VM). 335 336Please refer to the [suites documentation](https://www.chromium.org/chromium-os/testing/test-suites). 337 338## Writing and developing tests 339 340### Writing a test 341 342For understanding and writing the actual python code for autotest, please refer 343to the [Developer FAQ](http://www.chromium.org/chromium-os/testing/autotest-developer-faq#TOC-Writing-Autotests) 344 345Currently, all code should be placed in a standard layout inside the 346autotest.git repository, unless otherwise is necessary for technical reasons. 347Regardless, the following text assumes that code is placed in generally any 348repository. 349 350For a test to be fully functional in Chromium OS, it has to be associated with 351an ebuild. It is generally possible to run tests without an ebuild using 352`test_that` but discouraged, as the same will not function with other parts of 353the system. 354 355### Making a new test work with ebuilds 356 357The choice of ebuild depends on the location of its sources. Structuring tests 358into more smaller ebuilds (as opposed to one ebuild per source repository) 359serves two purposes: 360 361- Categorisation - Grouping similar tests together, possibly with deps they 362 use exclusively. 363- Parallelisation - Multiple independent ebuilds can build entirely in 364 parallel. 365- Dependency tracking - Larger bundles of tests depend on more system 366 packages without proper resolution which dependency belongs to which test. 367 This also increases paralellism. 368 369Current ebuild structure is largely a result of breaking off the biggest 370blockers for parallelism, ie. tests depending on chrome or similar packages, 371and as such, using any of the current ebuilds should be sufficient. (see FAQ 372for listing of ebuilds) 373 374After choosing the proper ebuild to add your test into, the test (in the form 375“+tests_<testname>”) needs to be added to IUSE_TESTS list that all autotest 376ebuilds have. Failing to do so will simply make ebuilds ignore your tests 377entirely. As with all USE flags, prepending it with + means the test will be 378enabled by default, and should be the default, unless you want to keep the test 379experimental for your own use, or turn the USE flag on explicitly by other 380means, eg. in a config for a particular board only. 381 382Should a **new ebuild** be started, it should be added to 383**chromeos-base/autotest-all** package, which is a meta-ebuild depending on all 384autotest ebuild packages that can be built. autotest-all is used by the build 385system to automatically build all tests that we have and therefore keep them 386from randomly breaking. 387 388### Deps 389 390Autotest uses deps to provide a de-facto dependencies into the ecosystem. A dep 391is a directory in ‘**client/deps**’ with a structure similar to a test case 392without a control file. A test case that depends on a dep will invoke the dep’s 393`setup()` function in its own `setup()` function and will be able to access the 394files provided by the dep. Note that autotest deps have nothing to do with 395system dependencies. 396 397As the calls to a dep are internal autotest code, it is not possible to 398automatically detect these and make them an inter-package dependencies on the 399ebuild level. For that reason, deps should either be 400[provided](#Ebuild-setup_autotest-eclass) by the same ebuild that builds test 401that consume them, or ebuild dependencies need to be declared manually between 402the dep ebuild and the test ebuild that uses it. An **autotest-deponly** 403eclass exists to provide solution for ebuilds that build only deps and no 404tests. A number of deponly ebuilds already exist. 405 406Common deps are: 407 408- chrome_test - Intending to use any of the test binaries produced by chrome. 409- pyauto_dep - Using pyauto for your code. 410 411### Test naming conventions 412 413Generally, the naming convention runs like this: 414 415\<component>\_\<TestName\> 416 417That convention names the directory containing the test code. It also names 418the .py file containing the test code, and the class of the Autotest test. 419 420If there's only one control file, it's named control. The test's NAME in the 421control file is \<component\>_\<TestName\>, like the directory and .py 422file. 423 424If there are multiple control files for a test, they are named 425control.\<testcase\>. These tests' NAMEs are then 426\<component\>_\<TestName\>.\<testcase\>. 427 428## Common workflows 429 430### W1. Develop and iterate on a test 431 4321. Set up the environment. 433 434 $ cd ~/trunk/src/third_party/autotest/files/ 435 $ export TESTS=”<the test cases to iterate on>” 436 $ EBUILD=<the ebuild that contains TEST> 437 $ board=<the board on which to develop> 438 4392. Ensure cros_workon is started 440 441 $ cros_workon --board=${board} start ${EBUILD} 442 $ repo sync # Necessary only if you use minilayout. 443 4443. Make modifications (on first run, you may want to just do 3,4 to verify 445 everything works before you touch it \& break it) 446 447 $ ... 448 4494. Build test (TESTS= is not necessary if you exported it before) 450 451 $ emerge-$board $EBUILD 452 4535. Run test to make sure it works before you touch it 454 455 $ test_that <machine IP> ${TESTS} 456 4576. Go to 2) to iterate 4587. Clean up environment 459 460 $ cros_workon --board=${board} stop ${EBUILD} 461 $ unset TESTS 462 463### W2. Creating a test - steps and checklist 464 465When creating a test, the following steps should be done/verified. 466 4671. Create the actual test directory, main test files/sources, at least one 468 control file 4692. Find the appropriate ebuild package and start working on it: 470 471 $ cros_workon --board=${board} start <package> 472 4733. Add the new test into the IUSE_TESTS list of 9999 ebuild 4744. Try building: (make sure it’s the 9999 version being built) 475 476 $ TESTS=<test> emerge-$board <package> 477 4785. Try running: 479 480 $ test_that <IP> <test> 481 4826. Iterate on 4,5 and modify source until happy with the initial version. 4837. Commit test source first, when it is safely in, commit the 9999 ebuild 484 version change. 4858. Cleanup 486 487 $ cros_workon --board=${board} stop <package> 488 489### W3. Splitting autotest ebuild into two 490 491Removing a test from one ebuild and adding to another in the same revision 492causes portage file collisions unless counter-measures are taken. Generally, 493some things routinely go wrong in this process, so this checklist should serve 494to help that. 495 4961. We have ebuild **foo-0.0.1-r100** with **test** and would like to split 497 that test off into ebuild **bar-0.0.1-r1**. 498 Assume that: 499 - both ebuilds are using cros-workon (because it’s likely the case). 500 - foo is used globally (eg. autotest-all depends on it), rather than just 501 some personal ebuild 5022. Remove **test** from foo-{0.0.1-r100,9999}; uprev foo-0.0.1-r100 (to -r101) 5033. Create bar-9999 (making a copy of foo and replacing IUSE_TESTS may be a good 504 start), with IUSE_TESTS containing just the entry for **test** 5054. Verify package dependencies of test. Make bar-9999 only depend on what is 506 needed for test, remove the dependencies from foo-9999, unless they are 507 needed by tests that remained. 5085. Add a blocker. Since bar installs files owned by foo-0.0.1-r100 and earlier, 509 the blocker’s format will be: 510 511 RDEPEND="!<=foo-0.0.1-r100" 512 5136. Add a dependency to the new version of bar into 514 chromeos-base/autotest-all-0.0.1 515 516 RDEPEND="bar" 517 5187. Change the dependency of foo in chromeos-base/autotest-all-0.0.1 to be 519 version locked to the new rev: 520 521 RDEPEND=">foo-0.0.1-r100" 522 5238. Uprev (move) autotest-all-0.0.1-rX symlink by one. 5249. Publish all as the same change list, have it reviewed, push. 525 526### W4. Create and run a test-enabled image on your device 527 5281. Choose which board you want to build for (we'll refer to this as ${BOARD}, 529 which is for example "x86-generic"). 5302. Set up a proper portage build chroot setup. Go through the normal process 531 of setup_board if you haven't already. 532 533 $ ./build_packages --board=${BOARD} 534 5353. Build test image. 536 537 $ ./build_image --board=${BOARD} test 538 5394. Install the Chromium OS testing image to your target machine. This is 540 through the standard mechanisms: either USB, or by reimaging a device 541 currently running a previously built Chromium OS image modded for test, or 542 by entering the shell on the machine and forcing an auto update to your 543 machine when it's running a dev server. For clarity we'll walk through two 544 common ways below, but if you already know about this, just do what you 545 normally do. 546 547 - If you choose to use a USB boot, you first put the image on USB and run 548 this from outside the chroot. 549 550 $ ./image_to_usb.sh --to /dev/sdX --board=${BOARD} \ 551 --image_name=chromiumos_test_image.bin 552 553 - Alternatively, if you happen to already have a machine running an image 554 modified for test and you know its IP address (${REMOTE_IP}), you can 555 avoid using a USB key and reimage it with a freshly built image by 556 running this from outside the chroot: 557 558 $ ./image_to_live.sh --remote=${REMOTE_IP} \ 559 --image=`./get_latest_image.sh \ 560 --board=${BOARD}`/chromiumos_test_image.bin 561 562This will automatically start dev server, ssh to your machine, cause it to 563update to from that dev server using memento_updater, reboot, wait for reboot, 564print out the new version updated to, and shut down your dev server. 565 566## Troubleshooting/FAQ 567 568### Q1: What autotest ebuilds are out there? 569 570Note that the list of ebuilds may differ per board, as each board has 571potentially different list of overlays. To find all autotest ebuilds for board 572foo, you can run: 573``` 574$ board=foo 575$ for dir in $(portageq-${board} envvar PORTDIR_OVERLAY); do 576 find . -name '*.ebuild' | xargs grep "inherit.*autotest" | grep "9999" | \ 577 cut -f1 -d: | \ 578 sed -e 's/.*\/\([^/]*\)\/\([^/]*\)\/.*\.ebuild/\1\/\2/' 579 done 580``` 581(Getting: "WARNING: 'portageq envvar PORTDIR_OVERLAY' is deprecated. Use 582'portageq repositories_configuration' instead." Please fix documentation.) 583 584### Q2: I see a test of the name ‘greattests_TestsEverything’ in build output/logs/whatever! How do I find which ebuild builds it? 585 586All ebuilds have lists of tests exported as **USE_EXPANDed** lists called 587**TESTS**. An 588expanded use can be searched for in the same way as other use flags, but with 589the appropriate prefix, in this case, you would search for 590**tests_greattests_TestsEverything**’: 591``` 592$ use_search=tests_greattests_TestsEverything 593$ equery-$board hasuse $use_search 594 * Searching for USE flag tests_greattests_TestsEverything ... 595 * [I-O] [ ] some_ebuild_package_name:0 596``` 597 598This will however only work on ebuilds which are **already installed**, ie. 599their potentially outdated versions. 600**Alternatively**, you can run a pretended emerge (emerge -p) of all autotest 601ebuilds and scan the output. 602``` 603$ emerge -p ${all_ebuilds_from_Q1} |grep -C 10 “${use_search}” 604``` 605 606### Q3: I have an ebuild ‘foo’, where are its sources? 607 608Generally speaking, one has to look at the ebuild source to figure that 609question out (and it shouldn’t be hard). However, all present autotest ebuilds 610(at the time of this writing) are also ‘cros-workon’, and for those, this 611should always work: 612``` 613$ ebuild_search=foo 614$ ebuild $(equery-$board which $ebuild_search) info 615CROS_WORKON_SRCDIR=”/home/you/trunk/src/third_party/foo” 616CROS_WORKON_PROJECT=”chromiumos/third_party/foo” 617``` 618 619### Q4: I have an ebuild, what tests does it build? 620 621You can run a pretended emerge on the ebuild and observe the ‘TESTS=’ 622statement: 623``` 624$ ebuild_name=foo 625$ emerge-$board -pv ${ebuild_name} 626These are the packages that would be merged, in order: 627 628Calculating dependencies... done! 629[ebuild R ] foo-foo_version to /build/$board/ USE="autox hardened tpmtools 630xset -buildcheck -opengles" TESTS="enabled_test1 enabled_test2 ... enabled_testN 631-disabled_test1 ...disabled_testN" 0 kB [1] 632``` 633 634Alternately, you can use equery, which will list tests with the USE_EXPAND 635prefix: 636``` 637$ equery-$board uses ${ebuild_name} 638[ Legend : U - final flag setting for installation] 639[ : I - package is installed with flag ] 640[ Colors : set, unset ] 641 * Found these USE flags for chromeos-base/autotest-tests-9999: 642 U I 643 + + autotest : <unknown> 644 + + autotest : <unknown> 645 + + autox : <unknown> 646 + + buildcheck : <unknown> 647 + + hardened : activate default security enhancements for toolchain (gcc, glibc, binutils) 648 - - opengles : <unknown> 649 + + tests_enabled_test : <unknown> 650 - - tests_disabled_test : <unknown> 651``` 652 653### Q5: I’m working on some test sources, how do I know which ebuilds to cros_workon start in order to properly propagate? 654 655You should ‘workon’ and always cros_workon start all ebuilds that have files 656that you touched. If you’re interested in a particular file/directory, that 657is installed in `/build/$board/usr/local/autotest/` and would like know which 658package has provided that file, you can use equery: 659 660``` 661$ equery-$board belongs /build/${board}/usr/local/autotest/client/site_tests/foo_bar/foo_bar.py 662 * Searching for <filename> ... 663chromeos-base/autotest-tests-9999 (<filename>) 664``` 665 666DON’T forget to do equery-$board. Just equery will also work, only never 667return anything useful. 668 669As a rule of thumb, if you work on anything from the core autotest framework or 670shared libraries (anything besides 671{server,client}/{test,site_tests,deps,profilers,config}), it belongs to 672chromeos-base/autotest. Individual test case will each belong to a particular 673ebuild, see Q2. 674 675It is important to cros_workon start every ebuild involved. 676 677### Q6: I created a test, added it into ebuild, emerged it, and I’m getting access denied failures. What did I do wrong? 678 679Your test’s `setup()` function (which runs on the host before being uploaded) is 680probably trying to write into the read-only intermediate location. See 681[explanation](#Building-tests). 682 683
