1# Getting started with device assignment
2
3Device assignment allows a VM to have direct access to HW without host/hyp
4intervention. AVF uses `vfio-platform` for device assignment, and host kernel
5support is required.
6
7This document explains how to setup and launch VM with device assignments.
8
9## VM device assignment DTBO (a.k.a. VM DTBO)
10
11For device assignment, a VM device assignment DTBO (a.k.a. VM DTBO) is required.
12VM DTBO is a device tree overlay which describes all assignable devices
13information. Information includes physical reg, IOMMU, device properties, and
14dependencies.
15
16VM DTBO allows to pass extra properties of assignable platform
17devices to the VM (which can't be discovered from the HW) while keeping the VMM
18device-agnostic.
19
20When the host boots, the bootloader provides VM DTBO to both Android and pvmfw.
21
22When a VM boots, the VMM selectively applies the DTBO based from provided
23labels, describing the assigned devices.
24
25## Prepare VM DTBO
26
27VM DTBO should be included in the dtbo partition. It should be in its own
28entry, and not together with any host OS's. See [DTB/DTBO Paritions] for
29partition format.
30
31[DTB/DTBO Paritions]: https://source.android.com/docs/core/architecture/dto/partitions
32
33### Write VM DTS for VM DTBO
34
35DTBO is compiled from device tree source (DTS) with `dtc` tool. [DTBO syntax]
36explains basic syntax of DTS.
37
38[DTBO syntax]: https://source.android.com/docs/core/architecture/dto/syntax
39
40Here are details and requirements:
41
42#### Describe assignable devices
43
44VM DTBO should describe assignable devices and their labels.
45
46* VM DTBO should have assignable devices in the `&{/}`, so it can be
47  overlaid onto VM DT. Assignable devices should be backed by physical device.
48  * We only support overlaying onto root node (i.e. `&{/}`) to prevent
49    unexpected modification of VM DT.
50* VM DTBO should have labels for assignable devices, so AVF can recognize
51  assignable device list. Labels should point to valid 'overlayable' nodes.
52  * Overlayable node is a node that would be applied to the base device tree
53    when DTBO is applied.
54
55#### Describe physical devices and physical IOMMUs
56
57VM DTBO should describe a `/host` node which describes physical devices and
58physical IOMMUs. The `/host` node only describes information for verification of
59assigned devices, and wouldn't be applied to VM DT. Here are details:
60
61* Physical IOMMU nodes
62  * IOMMU nodes must have a phandle to be referenced by a physical device node.
63  * IOMMU nodes must have `<android,pvmfw,token>` property. The property
64    describes the IOMMU token. An IOMMU token is a hypervisor-specific `<u64>`
65    which uniquely identifies a physical IOMMU. IOMMU token must be constant
66    across the VM boot for provisioning by pvmfw remains valid. The token must
67    be kept up-to-date across hypervisor updates.
68  * IOMMU nodes should be multi-master IOMMUs. (i.e. `#iommu-cells = <1>`)
69    * Other `#iommu-cells` values aren't supported for now.
70    * See: [Device tree binding for IOMMUs][IOMMU]
71* Physical device nodes
72  * Physical device nodes must have a `<android,pvmfw,target>` property that
73    references an overlayable node. The overlayable node contains the properties
74    that would be included in VM DT.
75  * Physical device nodes must have `<reg>` property to provide physical
76    regions.
77  * Physical device nodes can optionally contain `<iommus>` property. The
78    property is a prop-encoded-array and contains a number of
79    (iommu phandle, SID) pairs.
80    * IOMMU can be shared among devices, but should use distinct SIDs. Sharing
81      the same IOMMU-SID pair among multiple devices isn't supported for now.
82
83[IOMMU]: https://www.kernel.org/doc/Documentation/devicetree/bindings/iommu/iommu.txt
84
85#### Describe dependencies
86
87VM DTBO may have dependencies via phandle references. When a device node is
88assigned, dependencies of the node are also applied to VM DT.
89
90When dependencies are applied, siblings or children nodes of dependencies are
91ignored unless explicitly referenced.
92
93#### VM DTBO example
94
95Here's a simple example device tree source with four assignable devices nodes.
96
97```dts
98/dts-v1/;
99/plugin/;
100
101/ {
102    // host node describes physical devices and IOMMUs, and wouldn't be applied to VM DT
103    host {
104        #address-cells = <0x2>;
105        #size-cells = <0x1>;
106        rng {
107            reg = <0x0 0x12f00000 0x1000>;
108            iommus = <&iommu0 0x3>;
109            android,pvmfw,target = <&rng>;
110        };
111        light {
112            reg = <0x0 0x00f00000 0x1000>, <0x0 0x00f10000 0x1000>;
113            iommus = <&iommu1 0x4>, <&iommu2 0x5>;
114            android,pvmfw,target = <&light>;
115        };
116        led {
117            reg = <0x0 0x12000000 0x1000>;
118            iommus = <&iommu1 0x3>;
119            android,pvmfw,target = <&led>;
120        };
121        bus0 {
122            #address-cells = <0x1>;
123            #size-cells = <0x1>;
124            backlight {
125                reg = <0x300 0x100>;
126                android,pvmfw,target = <&backlight>;
127            };
128        };
129        iommu0: iommu0 {
130            #iommu-cells = <0x1>;
131            android,pvmfw,token = <0x0 0x12e40000>;
132        };
133        iommu1: iommu1 {
134            #iommu-cells = <0x1>;
135            android,pvmfw,token = <0x0 0x40000>;
136        };
137        iommu2: iommu2 {
138            #iommu-cells = <0x1>;
139            android,pvmfw,token = <0x0 0x50000>;
140        };
141    };
142};
143
144// Beginning of the assignable devices. Assigned devices would be applied to VM DT
145&{/} {  // We only allows to overlay to root node
146    rng: rng {
147        compatible = "android,rng";
148        android,rng,ignore-gctrl-reset;
149    };
150    light: light {
151        compatible = "android,light";
152        version = <0x1 0x2>;
153    };
154    led: led {
155        compatible = "android,led";
156        prop = <0x555>;
157    };
158    bus0 {
159        backlight: backlight {
160            compatible = "android,backlight";
161            android,backlight,ignore-gctrl-reset;
162        };
163    };
164};
165```
166
167If you compile the above with `dtc -@`, then you'll get `__symbols__` for free.
168`__symbol__` has label of nodes, and it's required for the next step.
169
170```dts
171    // generated __symbols__
172    __symbols__ {
173        iommu0 = "/host/iommu0";
174        iommu1 = "/host/iommu1";
175        iommu2 = "/host/iommu2";
176        rng = "/fragment@rng/__overlay__/rng";
177        light = "/fragment@sensor/__overlay__/light";
178        led = "/fragment@led/__overlay__/led";
179        backlight = "/fragment@backlight/__overlay__/bus0/backlight";
180    };
181```
182
183## Prepare AVF assignable devices XML
184
185AVF requires assignable device information to unbind from the host device driver
186and bind to VFIO driver. The information should be provided in an XML file at
187`/vendor/etc/avf/assignable_devices.xml`.
188
189Here's example.
190
191```xml
192<devices>
193    <device>
194        <kind>sensor</kind>
195        <dtbo_label>light</dtbo_label>
196        <sysfs_path>/sys/bus/platform/devices/16d00000.light</sysfs_path>
197    </device>
198</devices>
199```
200
201* `<kind>`: Device kind. Currently only used for debugging purposes and not used
202  for device assignment.
203* `<dtbo_label>`: Label in the VM DTBO (i.e. symbol in `__symbols__`). Must be
204  non-empty and unique in the XML.
205* `<sysfs_path>`: Sysfs path of the device in host, used to bind to the VFIO
206  driver. Must be non-empty and unique in the XML.
207
208## Boot with VM DTBO
209
210Bootloader should provide VM DTBO to both Android and pvmfw.
211
212### Provide VM DTBO index in dtbo.img
213
214Bootloader should provide the VM DTBO index with sysprop
215`ro.boot.hypervisor.vm_dtbo_idx.`. DTBO index represents DTBO location in
216dtbo.img.
217
218### Provide VM DTBO in the pvmfw config
219
220For protected VM, bootloader must provide VM DTBO to the pvmfw. pvmfw sanitizes
221incoming device tree with the VM DTBO.
222
223For more detail about providing VM DTBO in pvmfw,
224see: [pvmfw/README.md](../pvmfw/README.md#configuration-data-format)
225
226
227## Launch VM with device assignment
228
229We don't support client API yet in Android V, but you can use CLI to test device
230assignment. Note that host kernel support is required.
231
232Specify `--devices ${sysfs_path}` when booting VM. The parameter can be repeated
233multiple times for specifying multiple devices.
234
235Here's an example:
236
237```sh
238adb shell /apex/com.android.virt/bin/vm run-microdroid --devices /sys/bus/platform/devices/16d00000.light
239```