1# Codelab: Finding UI elements using chrome.automation API 2 3A common task in autotests is to make hardware changes and verify that UI gets 4updated or interact with UI elements and verify that hardware is updated. We can 5use the [chrome.automation] API to help with both of these tasks. 6 7[TOC] 8 9## Getting familiar with chrome.automation API 10 11Detailed information about chrome.automation API can be found at 12https://developer.chrome.com/extensions/automation. 13 14In short, the API is a wrapper around Chrome's hierarchy of accessibility nodes 15that describe Chrome UI elements. The most important attributes of accessibility 16nodes are **role** and **name**. See the section on [Accessibility Attributes] 17of the accessiblity overview. 18 19## Setup 20 21Follow the steps in [Loading autotest extension on 22device](loading-autotest-extension-on-device.md). Loading the AutotestPrivate 23extension will give you access to chrome.automation API as well. 24 25## To find a specific UI element 26 27Load a js console connected to the autotest extension's background page. See the 28previous section for steps on how to connect to the extension's background page. 29 30**NOTE**: The following steps are meant to be run interactively in the console 31and will not work if used in a real test. Section [Using chrome.automation in 32autotests](#Using-chrome_automation-in-autotests) shows how to use the API 33in a real test. 34 35Let's start by grabbing a reference to the root node of the accessibility tree. 36 37``` js 38var root; 39chrome.automation.getDesktop(r => root = r); 40``` 41 42### Finding a button in the hierarchy 43 44Let's demonstrate how to simulate a click on the launcher button in the system 45shelf. 46 47We'll start by listing all buttons visible in the tree. 48 49``` js 50root.findAll({attributes: {role: "button"}}).map(node => node.name); 51``` 52 53After typing that into the console you should get a response such as this: 54 55``` js 56> (7) ["Back", "Launcher", "Chromium", "Stylus tools", "Status tray, time 4:21 57PM, Battery is 22% full.", "Connected to Ethernet", "Battery is 22% full. Time 58 left until battery is empty, 1 hour and 39 minutes"] 59``` 60 61**NOTE**: Names will change depending on the locale of the device. We currently 62don't have a locale independent way of identifying UI nodes. 63 64Just by looking at button names we can easily guess that the button named 65"Launcher" is the one we're looking for. 66 67Finally, to simulate a click on our button: 68 69``` js 70var launcher = root.find({attributes: {role: "button", name: "Launcher"}}); 71launcher.doDefault(); 72``` 73 74The `doDefault` method performs an action based on the node's *role*, which for 75buttons is a button click. 76 77The `find` method supports multiple attributes filters. It returns UI elements 78that satisfy all conditions. 79 80## Important roles 81 82The API supports interactions with many types of UI elements. 83 84The following table contains chrome.automation roles for common UI elements: 85 86| views class | chorme.automation role | 87|-----------------:|------------------------| 88| views::Button | button | 89| views::Label | staticText | 90| views::ImageView | image | 91| views::TextField | textField | 92 93## Finding name and role of a view subclass 94 95View subclasses override the `GetAccessibleNodeData` method to provide role and 96name information. 97 98For example, look at [views::Button::GetAccessibleNodeData]. 99 100## Using chrome.automation in autotests 101 102chrome.automation extension can be accessed through the autotest extension. 103 104``` python 105with Chrome.chrome(autotest_ext=True) as cr: 106 ext = cr.autotest_ext 107 ext.ExecuteJavaScript(""" 108 chrome.automation.getDesktop(root => { 109 var launcher = root.find({attributes: {role: 'button', name: 'Launcher'}}); 110 launcher.doDefault(); 111 }); 112 """) 113``` 114 115[chrome.automation]: https://developer.chrome.com/extensions/automation 116[Accessibility Attributes]: https://chromium.googlesource.com/chromium/src/+/HEAD/docs/accessibility/overview.md#the-accessibility-tree-and-accessibility-attributes 117[views::Button::GetAccessibleNodeData]: https://cs.chromium.org/search/?q=views::Button::GetAccessibleNodeData 118[Getting to a command prompt]: https://www.chromium.org/chromium-os/poking-around-your-chrome-os-device#TOC-Getting-to-a-command-prompt 119[Run Chromium with flags]: https://www.chromium.org/developers/how-tos/run-chromium-with-flags 120