1# NNAPI vendor extensions
2
3In Android Q, Neural Networks API introduced vendor extensions -- a better,
4more structured alternative to the OEM operation and data types.
5An extension is a collection of vendor-defined operations and data types.
6A driver can provide custom hardware-accelerated operations via NNAPI 1.2+
7by supporting corresponding vendor extensions.
8
9Note that extensions do not modify behavior of existing operations.
10
11This document explains how to create and use extensions.
12
13## Extensions usage allowlist
14
15Vendor extensions can only be used by explicitly specified Android apps and
16native binaries on the /product, /vendor, /odm, and /data partitions. It's not possible to
17specify an app or a native binary located on the /system partition.
18
19The allowlist is stored in `/vendor/etc/nnapi_extensions_app_allowlist`, and contains
20a list of Android apps and binaries permitted to use NNAPI vendor extensions.
21Each line of the file contains a new entry. If an entry is prefixed by '/',
22then it's a native binary path (e.g. '/data/foo'). If not, it's a name of an Android
23app package (e.g. 'com.foo.bar').
24
25Allowlist is enforced from the NNAPI runtime shared library. It protects
26against accidental usage, but not against deliberate circumvention by directly
27using the NNAPI driver HAL interface.
28
29## Vendor extension definition
30
31The vendor is expected to create and maintain a header file with the
32extension definition. A complete example is provided in
33`test_vendor/fibonacci/FibonacciExtension.h`.
34
35Each extension must have a unique name that starts with the reverse domain name
36of the vendor:
37```c
38const char MY_EXTENSION_NAME[] = "com.example.my_extension";
39```
40
41This name acts as a namespace for operations and data types.
42NNAPI uses this name to distinguish between extensions.
43
44Operations and data types are declared in a way similar to
45`../runtime/include/NeuralNetworks.h`:
46```c
47enum {
48    /**
49     * A custom scalar type.
50     */
51    MY_SCALAR = 0,
52
53    /**
54     * A custom tensor type.
55     *
56     * Attached to this tensor is {@link MyTensorParams}.
57     */
58    MY_TENSOR = 1,
59};
60
61enum {
62    /**
63     * Computes my function.
64     *
65     * Inputs:
66     * * 0: A scalar of {@link MY_SCALAR}.
67     *
68     * Outputs:
69     * * 0: A tensor of {@link MY_TENSOR}.
70     */
71    MY_FUNCTION = 0,
72};
73```
74
75An extension operation may use any operand types, including non-extension
76operand types and operand types from other extensions. In the latter case,
77the driver must support those other extensions in order to support the
78extension.
79
80Extensions may also declare custom structures to accompany extension operands:
81```c
82/**
83 * Quantization parameters for {@link MY_TENSOR}.
84 */
85typedef struct MyTensorParams {
86    double scale;
87    int64_t zeroPoint;
88} MyTensorParams;
89```
90
91## Using extensions in NNAPI clients
92
93Runtime extension support is provided by
94`../runtime/include/NeuralNetworksExtensions.h` (C API) and
95`../runtime/include/NeuralNetworksWrapperExtensions.h` (C++ API).
96This section provides an overview of the former.
97
98Use `ANeuralNetworksDevice_getExtensionSupport` to check whether a device
99supports an extension:
100```c
101bool isExtensionSupported;
102CHECK_EQ(ANeuralNetworksDevice_getExtensionSupport(device, MY_EXTENSION_NAME,
103                                                   &isExtensionSupported),
104         ANEURALNETWORKS_NO_ERROR);
105if (isExtensionSupported) {
106    // The device supports the extension.
107    ...
108}
109```
110
111To build a model with an extension operand, use
112`ANeuralNetworksModel_getExtensionOperandType` to obtain the operand type.
113Then call `ANeuralNetworksModel_addOperand` as usual:
114```c
115int32_t type;
116CHECK_EQ(ANeuralNetworksModel_getExtensionOperandType(model, MY_EXTENSION_NAME, MY_TENSOR, &type),
117         ANEURALNETWORKS_NO_ERROR);
118ANeuralNetworksOperandType operandType{
119        .type = type,
120        .dimensionCount = dimensionCount,
121        .dimensions = dimensions,
122};
123CHECK_EQ(ANeuralNetworksModel_addOperand(model, &operandType), ANEURALNETWORKS_NO_ERROR);
124```
125
126Optionally, use `ANeuralNetworksModel_setOperandExtensionData` to
127associate additional data with an extension operand.
128```c
129MyTensorParams params{
130        .scale = 0.5,
131        .zeroPoint = 128,
132};
133CHECK_EQ(ANeuralNetworksModel_setOperandExtensionData(model, operandIndex, &params, sizeof(params)),
134         ANEURALNETWORKS_NO_ERROR);
135```
136
137To build a model with an extension operation, use
138`ANeuralNetworksModel_getExtensionOperationType` to obtain the operation type.
139Then call `ANeuralNetworksModel_addOperation` as usual:
140```c
141ANeuralNetworksOperationType type;
142CHECK_EQ(ANeuralNetworksModel_getExtensionOperationType(model, MY_EXTENSION_NAME, MY_FUNCTION,
143                                                        &type),
144         ANEURALNETWORKS_NO_ERROR);
145CHECK_EQ(ANeuralNetworksModel_addOperation(model, type, inputCount, inputs, outputCount, outputs),
146         ANEURALNETWORKS_NO_ERROR);
147```
148
149## Adding extension support to an NNAPI driver
150
151The driver reports supported extensions via the
152`IDevice::getSupportedExtensions()` method.
153For each supported extension, the returned list must contain an entry
154describing it:
155```c++
156Extension {
157    .name = MY_EXTENSION_NAME,
158    .operandTypes = {
159        {
160            .type = MY_SCALAR,
161            .isTensor = false,
162            .byteSize = 8,
163        },
164        {
165            .type = MY_TENSOR,
166            .isTensor = true,
167            .byteSize = 8,
168        },
169    },
170}
171```
172
173When handling operation and operand types, the driver must check the
174`Model::ExtensionTypeEncoding::HIGH_BITS_PREFIX` high bits of the type.
175These bits constitute the extension _prefix_. A zero prefix means no extension,
176whereas a non-zero prefix maps uniquely within a model to an extension name via
177`model.extensionNameToPrefix`.
178The low `Model::ExtensionTypeEncoding::LOW_BITS_TYPE` bits of the type
179correspond to the type within the extension.
180
181The driver must validate extension operations and data types, as the NNAPI
182runtime does not know how to validate particular extension operations and data
183types.
184
185Extension operands may have associated data in `operand.extraParams.extension`,
186which the runtime treats as a raw data blob of arbitrary size.
187