1Infra Trooper Documentation
2===========================
3
4### Contents ###
5
6*   [What does an Infra trooper do?](#what_is_a_trooper)
7*   [View current and upcoming troopers](#view_current_upcoming_troopers)
8*   [How to swap trooper shifts](#how_to_swap)
9*   [Tips for troopers](#tips)
10
11
12<a name="what_is_a_trooper"></a>
13What does an Infra trooper do?
14------------------------------
15
16The trooper has two main jobs:
17
181) Keep an eye on Infra alerts emails (sent to infra-alerts@skia.org). The alerts are also available [here](https://alerts.skia.org/infra).
19
202) Resolve the above alerts as they come in.
21
22<a name="view_current_upcoming_troopers"></a>
23View current and upcoming troopers
24----------------------------------
25
26The list of troopers is specified in the [skia-tree-status web app](http://skia-tree-status.appspot.com/trooper). The current trooper is highlighted in green.
27The banner on the top of the [status page](https://status.skia.org) also displays the current trooper.
28
29
30<a name="how_to_swap"></a>
31How to swap trooper shifts
32--------------------------
33
34If you need to swap shifts with someone (because you are out sick or on vacation), please get approval from the person you want to swap with. Then send an email to skiabot@google.com and cc rmistry@.
35
36
37<a name="tips"></a>
38Tips for troopers
39-----------------
40
41- Make sure you are a member of
42  [MDB group chrome-skia-ninja](https://ganpati.corp.google.com/#Group_Info?name=chrome-skia-ninja@prod.google.com).
43  Valentine passwords and Chrome Golo access are based on membership in this
44  group.
45
46- These alerts generally auto-dismiss once the criteria for the alert is no
47  longer met:
48  - Monitoring alerts, including prober, collectd, and others
49  - Disconnected build slaves
50
51- These alerts generally do not auto-dismiss ([issue here](https://bug.skia.org/4292)):
52  - Build slaves that failed a step
53  - Disconnected devices (these are detected as the "wait for device" step failing)
54
55- "Failed to execute query" may show a different query than the failing one;
56  dismiss the alert to get a new alert showing the query that is actually
57  failing. (All "failed to execute query" alerts are lumped into a single alert,
58  which is why the failed query which initially triggered the alert may not be
59  failing any more but the alert is still active because another query is
60  failing.)
61
62- Where machines are located:
63  - Machine name like "skia-vm-NNN" -> GCE
64  - Machine name ends with "a3", "a4", "m3" -> Chrome Golo
65  - Machine name starts with "skiabot-" -> Chapel Hill lab
66  - Machine name starts with "win8" -> Chapel Hill lab (Windows machine
67    names can't be very long, so the "skiabot-shuttle-" prefix is dropped.)
68  - slave11-c3 is a Chrome infra GCE machine (not to be confused with the Skia
69    Buildbots GCE, which we refer to as simply "GCE")
70
71- The [chrome-infra IRC channel](https://comlink.googleplex.com/chrome-infra) is
72  useful for questions regarding bots managed by the Chrome Infra team and to
73  get visibility into upstream failures that cause problems for us.
74
75- To log in to a Linux buildbot in GCE, use `gcloud compute ssh default@<machine
76  name>`. Choose the zone listed for the
77  [GCE VM](https://pantheon.corp.google.com/project/31977622648/compute/instances)
78  (or specify it using the `--zone` command-line flag).
79
80- To log in to a Windows buildbot in GCE, use
81  [Chrome RDP Extension](https://chrome.google.com/webstore/detail/chrome-rdp/cbkkbcmdlboombapidmoeolnmdacpkch?hl=en-US)
82  with the
83  [IP address of the GCE VM](https://pantheon.corp.google.com/project/31977622648/compute/instances)
84  shown on the [host info page](https://status.skia.org/hosts) for that bot. The
85  username is chrome-bot and the password can be found on
86  [Valentine](https://valentine.corp.google.com/) as "chrome-bot (Win GCE)".
87
88- If there is a problem with a bot in the Chrome Golo or Chrome infra GCE, the
89  best course of action is to
90  [file a bug](https://code.google.com/p/chromium/issues/entry?template=Build%20Infrastructure)
91  with the Chrome infra team. But if you know what you're doing:
92  - To access bots in the Chrome Golo,
93    [follow these instructions](https://chrome-internal.googlesource.com/infra/infra_internal/+/master/doc/ssh.md).
94    - Machine name ends with "a3" or "a4" -> ssh command looks like `ssh
95      build3-a3.chrome`
96    - Machine name ends with "m3" -> ssh command looks like `ssh build5-m3.golo`
97    - For MacOS and Windows bots, you will be prompted for a password, which is
98      stored on [Valentine](https://valentine.corp.google.com/) as "Chrome Golo,
99      Perf, GPU bots - chrome-bot".
100  - To access bots in the Chrome infra GCE -> command looks like `gcutil
101    --project=google.com:chromecompute ssh --ssh_user=default slave11-c3` (or
102    use the ccompute ssh script from the infra_internal repo).
103
104- Read over the [SkiaLab documentation](../testing/skialab) for more detail on
105  dealing with device alerts.
106
107- To stop a buildslave for a device, log in to the host for that device, `cd
108  ~/buildbot/<slave name>/build/slave; make stop`. To start it again,
109  `TESTING_SLAVENAME=<slave name> make start`.
110
111- Buildslaves can be slow to come up after reboot, but if the buildslave remains
112  disconnected, you may need to start it manually. On Mac and Linux, check using
113  `ps aux | grep python` that neither buildbot nor gclient are running, then run
114  `~/skiabot-slave-start-on-boot.sh`.
115