1# Hypervisor & guest tracing
2
3## Hypervisor tracing
4
5Starting with android14-5.15 kernel it is possible to get traces from the hypervisor.
6
7### User space interface
8
9The user space hypervisor tracing interface is located either at /sys/kernel/tracing/hyp or at
10/sys/kernel/debug/tracing/hyp. On the Android phones it will usually be /sys/kernel/tracing/hyp,
11while on QEMU it will be /sys/kernel/debug/tracing/hyp.
12
13The user space interface is very similar to the ftrace user space interface, however there are some
14differences, e.g.:
15
16* Only boot clock is supported, and there is no way for user space to change the tracing_clock.
17* Hypervisor tracing periodically polls the data from the hypervisor, this is different from the
18  regular ftrace instance which pushes the events into the ring buffer.
19* Resetting ring buffers (by clearing the trace file) is only supported when there are no active
20  readers. If the trace file is cleared while there are active readers, then the ring buffers will
21  be cleared after the last reader disconnects.
22* Changing the size of the ring buffer while the tracing session is active is also not supported.
23
24Note: the list above is not exhaustive.
25
26### Perfetto integration
27
28[Perfetto](https://perfetto.dev/docs/) is an open-source stack for performance instrumentation and
29trace analysis widely used in  Android. Perfetto supports capturing and visualizing hypervisor
30traces.
31
32#### Capturing hypervisor traces on Android
33
34Consider first familiarizing yourself with Perfetto documentation for recording traces on Android:
35https://perfetto.dev/docs/quickstart/android-tracing.
36
37The [record_android_trace](
38https://cs.android.com/android/platform/superproject/+/master:external/perfetto/tools/record_android_trace)
39script supports a shortcut to capture all hypervisor events that are  known to Perfetto:
40
41```shell
42external/perfetto/tools/record_android_trace hyp -t 15s -b 32mb -o /tmp/hyp.pftrace
43```
44
45Alternatively you can use full trace config to capture hypervisor. Example usage:
46
47```shell
48cat<<EOF>config.pbtx
49duration_ms: 10000
50
51buffers: {
52    size_kb: 8960
53    fill_policy: DISCARD
54}
55
56data_sources: {
57    config {
58        name: "linux.ftrace"
59        ftrace_config {
60            instance_name: "hyp"
61            ftrace_events: "hyp/hyp_enter"
62            ftrace_events: "hyp/hyp_exit"
63        }
64    }
65}
66EOF
67
68./record_android_trace -c config.pbtx -o trace_file.perfetto-trace
69```
70
71If you have an Android tree checked out, then record_android_trace helper script can be located at
72${REPO_ROOT}/external/perfetto/tools/record_android_traces. Otherwise, you can download the script
73by following steps outlined in the [Perfetto docs](
74https://perfetto.dev/docs/quickstart/android-tracing#recording-a-trace-through-the-cmdline)
75
76#### Capturing hypervisor traces on QEMU
77
78Perfetto supports capturing traces on Linux: https://perfetto.dev/docs/quickstart/linux-tracing.
79However, since pKVM hypervisor is only supported on arm64, you will need to cross-compile Perfetto
80binaries for linux-arm64 (unless you have an arm64 workstation).
81
821. Checkout Perfetto repository: https://perfetto.dev/docs/contributing/getting-started
832. Follow https://perfetto.dev/docs/contributing/build-instructions#cross-compiling-for-linux-arm-64
84  to compile Perfetto binaries for arm64 architecture.
853. Copy the tracebox binary to QEMU
864. Run `tracebox` binary on QEMU to capture traces, it's interface is very similar to the
87`record_android_trace` binary. E.g. to capture all hypervisor events run:
88```shell
89tracebox -t 15s -b 32mb hyp
90```
91
92### Analysing traces using SQL
93
94On top of visualisation, Perfetto also provides a SQL interface to analyse traces. More
95documentation is available at https://perfetto.dev/docs/quickstart/trace-analysis and
96https://perfetto.dev/docs/analysis/trace-processor.
97
98Hypervisor events can be queried via `pkvm_hypervisor_events` SQL view. You can load that view by
99calling `SELECT IMPORT("pkvm.hypervisor");`, e.g.:
100
101```sql
102SELECT IMPORT("pkvm.hypervisor");
103SELECT * FROM pkvm_hypervisor_events limit 5;
104```
105
106Below are some SQL queries that might be useful when analysing hypervisor traces.
107
108**What is the longest time CPU spent in hypervisor, grouped by the reason to enter hypervisor**
109```sql
110SELECT IMPORT("pkvm.hypervisor");
111
112SELECT
113  cpu,
114  reason,
115  ts,
116  dur
117FROM pkvm_hypervisor_events
118JOIN (
119  SELECT
120    MAX(dur) as dur2,
121    cpu as cpu2,
122    reason as reason2
123  FROM pkvm_hypervisor_events
124  GROUP BY 2, 3) AS sc
125ON
126  cpu = sc.cpu2
127  AND dur = sc.dur2
128  AND (reason = sc.reason2 OR (reason IS NULL AND sc.reason2 IS NULL))
129ORDER BY dur desc;
130```
131
132**What are the 10 longest times CPU spent in hypervisor because of host_mem_abort**
133```sql
134SELECT
135  hyp.dur as dur,
136  hyp.ts as ts,
137  EXTRACT_ARG(slices.arg_set_id, 'esr') as esr,
138  EXTRACT_ARG(slices.arg_set_id, 'addr') as addr
139FROM pkvm_hypervisor_events as hyp
140JOIN slices
141ON hyp.slice_id = slices.id
142WHERE hyp.reason = 'host_mem_abort'
143ORDER BY dur desc
144LIMIT 10;
145```
146
147## Microdroid VM tracing
148
149IMPORTANT: Tracing is only supported for debuggable Microdroid VMs.
150
151### Capturing trace in Microdroid
152
153Starting with Android U, Microdroid contains Perfetto tracing binaries, which makes it possible to
154capture traces inside Microdroid VM using Perfetto stack. The commands used to capture traces on
155Android should work for Microdroid VM as well, with a difference that Perfetto's tracing binaries
156are not enabled in Microdroid by default, so you need to manually start them by setting
157`persist.traced.enable` system property to `1`.
158
159Here is a quick example on how trace Microdroid VM:
160
1611. First start your VM. For this example we are going to use
162`adb shell /apex/com.android.virt/bin/vm run-microdroid`.
163
1642. Set up an adb connection with the running VM:
165```shell
166adb shell forward tcp:9876 vsock:${CID}:5555
167adb connect localhost:9876
168adb -s localhost:9876 root
169```
170Where `${CID}` corresponds to the running Microdroid VM that you want to establish adb connection
171with. List of running VMs can be obtained by running `adb shell /apex/com.android.virt/bin/vm list`.
172Alternatively you can use `vm_shell` utility to connect to a running VM, i.e.: `vm_shell connect`.
173
1743. Start Perfetto daemons and capture trace
175```shell
176adb -s localhost:9876 shell setprop persist.traced.enable 1
177${ANDROID_BULD_TOP}/external/perfetto/tools/record_android_trace \
178  -s localhost:9876 \
179  -o /tmp/microdroid-trace-file.pftrace \
180  -t 10s \
181  -b 32mb \
182  sched/sched_switch task/task_newtask sched/sched_process_exit
183```
184
185If you don't have Android repo checked out, then you can download the record_android_trace script by
186following the following [instructions](
187https://perfetto.dev/docs/quickstart/android-tracing#recording-a-trace-through-the-cmdline)
188
189More documentation on Perfetto's tracing on Android is available here:
190https://perfetto.dev/docs/quickstart/android-tracing
191
192### Capturing Microdroid boot trace
193
194TODO(b/271412868): Stay tuned, more docs are coming soon!
195