1# Vendor Extensions
2
3Neural Networks API (NNAPI) vendor extensions, introduced in Android 10, are
4collections of vendor-defined operations and data types. On devices running NN
5HAL 1.2 or higher, drivers can provide custom hardware-accelerated operations by
6supporting corresponding vendor extensions. Vendor extensions don't modify the
7behavior of existing operations.
8
9Vendor extensions provide a more structured alternative to OEM operation and
10data types, which were deprecated in Android 10.
11For more information, see the last section.
12
13## Extensions usage allowlist
14
15Vendor extensions can only be used by explicitly specified Android apps and
16native binaries on the `/product`, `/vendor`, `/odm`, and `/data` partitions.
17Apps and native binaries located on the `/system` partition can't use vendor
18extensions.
19
20A list of Android apps and binaries permitted to use NNAPI vendor extensions is
21stored in `/vendor/etc/nnapi_extensions_app_allowlist`. Each line of the file
22contains a new entry. An entry can be a native binary path that is prefixed with
23a slash (/), for example, `/data/foo`, or a name of an Android app package, for
24example, `com.foo.bar`.
25
26The allowlist is enforced from the NNAPI runtime shared library. This library
27protects against accidental usage but not against deliberate circumvention by
28an app directly using the NNAPI driver HAL interface.
29
30## Vendor extension definition
31
32The vendor creates and maintains a header file with the extension definition. A
33complete example of an extension definition can be found in
34`example/fibonacci/FibonacciExtension.h`.
35
36Each extension must have a unique name that starts with the reverse domain name
37of the vendor.
38
39Note: Names of extension operation and operand types must be qualified with a
40vendor name. The vendor name is represented by `EXAMPLE` in the code samples on
41this page.
42
43```
44const char EXAMPLE_EXTENSION_NAME[] = "com.example.my_extension";
45```
46
47The name acts as a namespace for operations and data types. The NNAPI uses this
48name to distinguish between vendor extensions.
49
50Operations and data types are declared in a way similar to those in
51`runtime/include/NeuralNetworks.h`.
52
53```
54enum {
55    /**
56     * A custom scalar type.
57     */
58    EXAMPLE_SCALAR = 0,
59
60    /**
61     * A custom tensor type.
62     *
63     * Attached to this tensor is {@link ExampleTensorParams}.
64     */
65    EXAMPLE_TENSOR = 1,
66};
67
68enum {
69    /**
70     * Computes example function.
71     *
72     * Inputs:
73     * * 0: A scalar of {@link EXAMPLE_SCALAR}.
74     *
75     * Outputs:
76     * * 0: A tensor of {@link EXAMPLE_TENSOR}.
77     */
78    EXAMPLE_FUNCTION = 0,
79};
80```
81
82An extension operation can use any operand type, including nonextension operand
83types and operand types from other extensions. When using an operand type from
84another extension, the driver must support the other extension.
85
86Extensions can also declare custom structures to accompany extension operands.
87
88```
89/**
90 * Quantization parameters for {@link EXAMPLE_TENSOR}.
91 */
92typedef struct ExampleTensorParams {
93    double scale;
94    int64_t zeroPoint;
95} ExampleTensorParams;
96```
97
98## Using extensions in NNAPI clients
99
100The `runtime/include/NeuralNetworksExtensions.h` (C API) file provides runtime
101extension support. This section provides an overview of the C API.
102
103To check whether a device supports an extension, use
104`ANeuralNetworksDevice_getExtensionSupport`.
105
106```
107bool isExtensionSupported;
108CHECK_EQ(ANeuralNetworksDevice_getExtensionSupport(device, EXAMPLE_EXTENSION_NAME, &isExtensionSupported),
109         ANEURALNETWORKS_NO_ERROR);
110if (isExtensionSupported) {
111    // The device supports the extension.
112    ...
113}
114```
115
116To build a model with an extension operand, use `ANeuralNetworksModel_getExtensionOperandType`
117to obtain the operand type and call `ANeuralNetworksModel_addOperand`.
118
119```
120int32_t type;
121CHECK_EQ(ANeuralNetworksModel_getExtensionOperandType(model, EXAMPLE_EXTENSION_NAME, EXAMPLE_TENSOR, &type),
122         ANEURALNETWORKS_NO_ERROR);
123ANeuralNetworksOperandType operandType{
124        .type = type,
125        .dimensionCount = dimensionCount,
126        .dimensions = dimensions,
127};
128CHECK_EQ(ANeuralNetworksModel_addOperand(model, &operandType), ANEURALNETWORKS_NO_ERROR);
129```
130
131Optionally, use
132`ANeuralNetworksModel_setOperandExtensionData`
133to associate additional data with an extension operand.
134
135```
136ExampleTensorParams params{
137        .scale = 0.5,
138        .zeroPoint = 128,
139};
140CHECK_EQ(ANeuralNetworksModel_setOperandExtensionData(model, operandIndex, &params, sizeof(params)),
141         ANEURALNETWORKS_NO_ERROR);
142```
143
144To build a model with an extension operation, use
145`ANeuralNetworksModel_getExtensionOperationType`
146to obtain the operation type and call
147`ANeuralNetworksModel_addOperation`.
148
149```
150ANeuralNetworksOperationType type;
151CHECK_EQ(ANeuralNetworksModel_getExtensionOperationType(model, EXAMPLE_EXTENSION_NAME, EXAMPLE_FUNCTION, &type),
152         ANEURALNETWORKS_NO_ERROR);
153CHECK_EQ(ANeuralNetworksModel_addOperation(model, type, inputCount, inputs, outputCount, outputs),
154         ANEURALNETWORKS_NO_ERROR);
155```
156
157## Adding extension support to an NNAPI driver
158
159Drivers report supported extensions through the `IDevice::getSupportedExtensions`
160method. The returned list must contain an entry describing each supported
161extension.
162
163```
164Extension {
165    .name = EXAMPLE_EXTENSION_NAME,
166    .operandTypes = {
167        {
168            .type = EXAMPLE_SCALAR,
169            .isTensor = false,
170            .byteSize = 8,
171        },
172        {
173            .type = EXAMPLE_TENSOR,
174            .isTensor = true,
175            .byteSize = 8,
176        },
177    },
178}
179```
180
181Of the 32 bits used to identify types and operations, the high
182`ExtensionTypeEncoding::HIGH_BITS_PREFIX` bits is the extension
183_prefix_ and the low `ExtensionTypeEncoding::LOW_BITS_TYPE`
184bits represents the type or operation of the extension.
185
186When handling an operation or operand type, the driver must check the extension
187prefix. If the extension prefix has a nonzero value, the operation or operand
188type is an extension type. If the value is `0`, the operation or operand type
189isn't an extension type.
190
191To map the prefix to an extension name, look it up in `model.extensionNameToPrefix`.
192The mapping from the prefix to the extension name is a one-to-one correspondence
193(bijection) for a given model. Different prefix values might correspond to the
194same extension name in different models.
195
196The driver must validate extension operations and data types because the NNAPI
197runtime can't validate particular extension operations and data types.
198
199Extension operands can have associated data in `operand.extraParams.extension`,
200which the runtime treats as a raw data blob of arbitrary size.
201
202## OEM operation and data types
203
204Note: OEM operation and data types have been deprecated. For devices running
205Android 10 or higher, use vendor extensions instead.
206
207NNAPI has an OEM operation and OEM data types to allow
208device manufacturers to provide custom, driver-specific functionality. These
209operation and data types are only used by OEM applications. The semantics of OEM
210operation and data types are OEM-specific and can change at any time. The OEM
211operation and data types are encoded using `OperationType::OEM_OPERATION`,
212`OperandType::OEM`, and `OperandType::TENSOR_OEM_BYTE`.
213