1# Codelab: Finding UI elements using chrome.automation API
2
3A common task in autotests is to make hardware changes and verify that UI gets
4updated or interact with UI elements and verify that hardware is updated. We can
5use the [chrome.automation] API to help with both of these tasks.
6
7[TOC]
8
9## Getting familiar with chrome.automation API
10
11Detailed information about chrome.automation API can be found at
12https://developer.chrome.com/extensions/automation.
13
14In short, the API is a wrapper around Chrome's hierarchy of accessibility nodes
15that describe Chrome UI elements. The most important attributes of accessibility
16nodes are **role** and **name**. See the section on [Accessibility Attributes]
17of the accessiblity overview.
18
19## Setup
20
21Follow the steps in [Loading autotest extension on
22device](loading-autotest-extension-on-device.md). Loading the AutotestPrivate
23extension will give you access to chrome.automation API as well.
24
25## To find a specific UI element
26
27Load a js console connected to the autotest extension's background page. See the
28previous section for steps on how to connect to the extension's background page.
29
30**NOTE**: The following steps are meant to be run interactively in the console
31and will not work if used in a real test. Section [Using chrome.automation in
32autotests](#Using-chrome_automation-in-autotests) shows how to use the API
33in a real test.
34
35Let's start by grabbing a reference to the root node of the accessibility tree.
36
37``` js
38var root;
39chrome.automation.getDesktop(r => root = r);
40```
41
42### Finding a button in the hierarchy
43
44Let's demonstrate how to simulate a click on the launcher button in the system
45shelf.
46
47We'll start by listing all buttons visible in the tree.
48
49``` js
50root.findAll({attributes: {role: "button"}}).map(node => node.name);
51```
52
53After typing that into the console you should get a response such as this:
54
55``` js
56> (7) ["Back", "Launcher", "Chromium", "Stylus tools", "Status tray, time 4:21
57PM, Battery is 22% full.", "Connected to Ethernet", "Battery is 22% full. Time
58 left until battery is empty, 1 hour and 39 minutes"]
59```
60
61**NOTE**: Names will change depending on the locale of the device. We currently
62don't have a locale independent way of identifying UI nodes.
63
64Just by looking at button names we can easily guess that the button named
65"Launcher" is the one we're looking for.
66
67Finally, to simulate a click on our button:
68
69``` js
70var launcher = root.find({attributes: {role: "button", name: "Launcher"}});
71launcher.doDefault();
72```
73
74The `doDefault` method performs an action based on the node's *role*, which for
75buttons is a button click.
76
77The `find` method supports multiple attributes filters. It returns UI elements
78that satisfy all conditions.
79
80## Important roles
81
82The API supports interactions with many types of UI elements.
83
84The following table contains chrome.automation roles for common UI elements:
85
86| views class      | chorme.automation role |
87|-----------------:|------------------------|
88| views::Button    | button                 |
89| views::Label     | staticText             |
90| views::ImageView | image                  |
91| views::TextField | textField              |
92
93## Finding name and role of a view subclass
94
95View subclasses override the `GetAccessibleNodeData` method to provide role and
96name information.
97
98For example, look at [views::Button::GetAccessibleNodeData].
99
100## Using chrome.automation in autotests
101
102chrome.automation extension can be accessed through the autotest extension.
103
104``` python
105with Chrome.chrome(autotest_ext=True) as cr:
106    ext = cr.autotest_ext
107    ext.ExecuteJavaScript("""
108        chrome.automation.getDesktop(root => {
109            var launcher = root.find({attributes: {role: 'button', name: 'Launcher'}});
110            launcher.doDefault();
111        });
112    """)
113```
114
115[chrome.automation]: https://developer.chrome.com/extensions/automation
116[Accessibility Attributes]: https://chromium.googlesource.com/chromium/src/+/HEAD/docs/accessibility/overview.md#the-accessibility-tree-and-accessibility-attributes
117[views::Button::GetAccessibleNodeData]: https://cs.chromium.org/search/?q=views::Button::GetAccessibleNodeData
118[Getting to a command prompt]: https://www.chromium.org/chromium-os/poking-around-your-chrome-os-device#TOC-Getting-to-a-command-prompt
119[Run Chromium with flags]: https://www.chromium.org/developers/how-tos/run-chromium-with-flags
120