• Home
  • History
  • Annotate
  • Line#
  • Scopes#
  • Navigate#
  • Raw
  • Download
1# Using the NN-API Test Generator
2
3## Prerequisites
4
5- Python3
6- Numpy
7
8## Writing a Test Specification
9
10You should create new test specs in `nn/runtime/test/specs/<version>/` and name it with `.mod.py` suffix, so that other tools can automatically update the unit tests.
11
12### Specifying Operands
13
14#### Syntax
15
16```
17OperandType(name, (type, shape, <optional scale, zero point>), <optional initializer>)
18```
19
20For example,
21
22```Python
23# p1 is a 2-by-2 fp matrix parameter, with value [1, 2; 3, 4]
24p1 = Parameter("param", ("TENSOR_FLOAT32", [2, 2]), [1, 2, 3, 4])
25
26# i1 is a quantized input of shape (2, 256, 256, 3), with scale = 0.5, zero point = 128
27i1 = Input("input", ("TENSOR_QUANT8_ASYMM", [2, 256, 256, 3], 0.5, 128))
28
29# p2 is an Int32 scalar with value 1
30p2 = Int32Scalar("act", 1)
31```
32
33#### OperandType
34
35There are currently 10 operand types supported by the test generator.
36
37- Input
38- Output
39    * IgnoredOutput, will not compare results in the test
40- Parameter
41    * Int32Scalar, shorthand for parameter with type INT32
42    * Float32Scalar, shorthand for parameter with type FLOAT32
43    * Int32Vector, shorthand for 1-D TENSOR_INT32 parameter
44    * Float32Vector, shorthand for 1-D TENSOR_FLOAT32 parameter
45- Internal, for model with multiple operations
46
47### Specifying Models
48
49#### Instantiate a model
50
51```Python
52# Instantiate a model
53model = Model()
54
55# Instantiate a model with a name
56model2 = Model("model_name")
57```
58
59#### Add an operation
60
61```
62model.Operation(optype, i1, i2, ...).To(o1, o2, ...)
63```
64
65For example,
66
67```Python
68model.Operation("ADD", i1, i2, act).To(o1)
69```
70
71#### Use implicit operands
72
73Simple scalar and 1-D vector parameters can now be directly passed to Operation constructor, and test generator will deduce the operand type from the value provided.
74
75```Python
76model.Operation("MEAN", i1, [1], 0) # axis = [1], keep_dims = 0
77```
78
79Note that, for fp values, the initializer should all be Python fp numbers, e.g. use `1.0` or `1.` instead of `1` for implicit fp operands.
80
81### Specifying Inputs and Expected Outputs
82
83The combination of inputs and expected outputs is called an example for a given model. An example is defined like
84
85```Python
86# Example 1, separate dictionary for inputs and outputs
87input1 = {
88    i1: [1, 2],
89    i2: [3, 4]
90}
91output1 = {o1: [4, 6]}
92
93# Example 2, combined dictionary
94example2_values = {
95    i1: [5, 6],
96    i2: [7, 8],
97    o1: [12, 14]
98}
99
100# Instantiate an example
101Example((input1, output1), example2_values)
102```
103
104By default, examples will be attached to the most recent instantiated model. You can explicitly specify the target model, and optionally, the example name by
105
106```Python
107Example((input1, output1), example2_values, model=model, name="example_name")
108```
109
110### Specifying Variations
111
112You can add variations to the example so that the test generator can automatically create multiple tests. Currently, 6 types of variation are supported:
113
114- DefaultVariation, i.e. no variation
115- DataTypeConverter
116- DataLayoutConverter
117- AxisConverter
118- RelaxedModeConverter
119- ParameterAsInputConverter
120- ActivationConverter
121
122#### DataTypeConverter
123
124Convert input/parameter/output to the specified type, e.g. float32 -> quant8. The target data type for each operand to transform has to be explicitly specified. It is the spec writer's responsibility to ensure such conversion is valid.
125
126```Python
127converter = DataTypeConverter(name="variation_name").Identify({
128    op1: (target_type, target_scale, target_zero_point),
129    op2: (target_type, target_scale, target_zero_point),
130    ...
131})
132```
133
134#### DataLayoutConverter
135
136Convert input/parameter/output between NHWC and NCHW. The caller need to provide a list of target operands to transform, and also the data layout parameter to set.
137
138```Python
139converter = DataLayoutConverter(target_data_layout, name="variation_name").Identify(
140    [op1, op2, ..., layout_parameter]
141)
142```
143
144#### AxisConverter
145
146Transpose a certain axis in input/output to target position, and optionally remove some axis. The caller need to provide a list of target operands to transform, and also the axis parameter to set.
147
148```Python
149converter = AxisConverter(originalAxis, targetAxis, dimension, drop=[], name="variation_name").Identify(
150    [op1, op2, ..., axis_parameter]
151)
152```
153
154This model variation is for ops that apply calculation along certain axis, such as L2_NORMALIZATION, SOFTMAX, and CHANNEL_SHUFFLE. For example, consider L2_NORMALIZATION with input of shape [2, 3, 4, 5] along the last axis, i.e. axis = -1. The output shape would be the same as input. We can create a new model which will do the calculation along axis 0 by transposing input and output shape to [5, 2, 3, 4] and modify the axis parameter to 0. Such converter can be defined as
155
156```Python
157toAxis0 = AxisConverter(-1, 0, 4).Identify([input, output, axis])
158```
159
160The target axis can also be negative to test the negative indexing
161
162```Python
163toAxis0 = AxisConverter(-1, -4, 4).Identify([input, output, axis])
164```
165
166Consider the same L2_NORMALIZATION example, we can also create a new model with input/output of 2D shape [4, 5] by removing the first two dimension. This is essentially doing `new_input = input[0,0,:,:]` in numpy. Such converter can be defined as
167
168```Python
169toDim2 = AxisConverter(-1, -1, 4, drop=[0, 1]).Identify([input, output, axis])
170```
171
172If transposition and removal are specified at the same time, the converter will do transposition first and then remove the axis. For example, the following converter will result in shape [5, 4] and axis 0.
173
174```Python
175toDim2Axis0 = AxisConverter(-1, 2, 4, drop=[0, 1]).Identify([input, output, axis])
176```
177
178#### RelaxedModeConverter
179
180Convert the model to enable/disable relaxed computation.
181
182```Python
183converter = RelaxedModeConverter(is_relaxed, name="variation_name")
184```
185
186#### ParameterAsInputConverter
187
188Convert a certain parameter to model input, e.g. weight in CONV_2D. The caller need to provide a list of target operands to convert.
189
190```Python
191converter = ParameterAsInputConverter(name="variation_name").Identify(
192    [op1, op2, ...]
193)
194```
195
196#### ActivationConverter
197
198Convert the output by certain activation, the original activation is assumed to be NONE. The caller need to provide a list of target operands to transform,  and also the activation parameter to set.
199
200```Python
201converter = ActivationConverter(name="variation_name").Identify(
202    [op1, op2, ..., act_parameter]
203)
204```
205
206#### Add variation to example
207
208Each example can have multiple groups of variations, and if so, will take the cartesian product of the groups. For example, suppose we declare a model with two groups, and each group has two variations: `[[default, nchw], [default, relaxed, quant8]]`. This will result in 6 examples: `[default, default], [default, relaxed], [default, quant8], [nchw, default], [nchw, relaxed], [nchw, quant8]`.
209
210Use `AddVariations` to add a group of variations to the example
211
212```Python
213# Add two groups of variations [default, nchw] and [default, relaxed, quant8]
214example.AddVariations(nchw).AddVariations(relaxed, quant8)
215```
216
217By default, when you add a group of variation, a unnamed default variation will be automatically included in the list. You can name the default variation by
218
219```Python
220example.AddVariations(nchw, defaultName="nhwc").AddVariations(relaxed, quant8)
221```
222
223Also, you can choose not to include default by
224
225```Python
226# Add two groups of variations [nchw] and [default, relaxed, quant8]
227example.AddVariations(nchw, includeDefault=False).AddVariations(relaxed, quant8)
228```
229
230The example above will result in 3 examples: `[nchw, default], [nchw, relaxed], [nchw, quant8]`.
231
232#### Some helper functions
233
234The test generator provides several helper functions or shorthands to add commonly used group of variations.
235
236```Python
237# Each following group of statements are equivalent
238
239# DataTypeConverter
240example.AddVariations(DataTypeConverter().Identify({op1: "TENSOR_FLOAT16", ...}))
241example.AddVariations("float16")    # will apply to every TENSOR_FLOAT32 operands
242
243example.AddVariations(DataTypeConverter().Identify({op1: "TENSOR_INT32", ...}))
244example.AddVariations("int32")      # will apply to every TENSOR_FLOAT32 operands
245
246# DataLayoutConverter
247example.AddVariations(DataLayoutConverter("nchw").Identify(op_list))
248example.AddVariations(("nchw", op_list))
249example.AddNchw(*op_list)
250
251# AxisConverter
252# original axis and dim are deduced from the op_list
253example.AddVariations(*[AxisConverter(origin, t, dim).Identify(op_list) for t in targets])
254example.AddAxis(targets, *op_list)
255
256example.AddVariations(*[
257        AxisConverter(origin, t, dim).Identify(op_list) for t in range(dim)
258    ], includeDefault=False)
259example.AddAllPositiveAxis(*op_list)
260
261example.AddVariations(*[
262        AxisConverter(origin, t, dim).Identify(op_list) for t in range(-dim, dim)
263    ], includeDefault=False)
264example.AddAllAxis(*op_list)
265
266drop = list(range(dim))
267drop.pop(origin)
268example.AddVariations(*[
269    AxisConverter(origin, origin, dim, drop[0:(dim-i)]).Identify(op_list) for i in dims])
270example.AddDims(dims, *op_list)
271
272example.AddVariations(*[
273    AxisConverter(origin, origin, dim, drop[0:i]).Identify(op_list) for i in range(dim)])
274example.AddAllDims(dims, *op_list)
275
276example.AddVariations(*[
277        AxisConverter(origin, j, dim, range(i)).Identify(op_list) \
278                for i in range(dim) for j in range(i, dim)
279    ], includeDefault=False)
280example.AddAllDimsAndPositiveAxis(dims, *op_list)
281
282example.AddVariations(*[
283        AxisConverter(origin, k, dim, range(i)).Identify(op_list) \
284                for i in range(dim) for j in range(i, dim) for k in [j, j - dim]
285    ], includeDefault=False)
286example.AddAllDimsAndAxis(dims, *op_list)
287
288# ParameterAsInputConverter
289example.AddVariations(ParameterAsInputConverter().Identify(op_list))
290example.AddVariations(("as_input", op_list))
291example.AddInput(*op_list)
292
293# RelaxedModeConverter
294example.Addvariations(RelaxedModeConverter(True))
295example.AddVariations("relaxed")
296example.AddRelaxed()
297
298# ActivationConverter
299example.AddVariations(ActivationConverter("relu").Identify(op_list))
300example.AddVariations(("relu", op_list))
301example.AddRelu(*op_list)
302
303example.AddVariations(
304    ActivationConverter("relu").Identify(op_list),
305    ActivationConverter("relu1").Identify(op_list),
306    ActivationConverter("relu6").Identify(op_list))
307example.AddVariations(
308    ("relu", op_list),
309    ("relu1", op_list),
310    ("relu6", op_list))
311example.AddAllActivations(*op_list)
312```
313
314### Specifying the Model Version
315
316If not explicitly specified, the minimal required HAL version will be inferred from the path, e.g. the models defined in `nn/runtime/test/specs/V1_0/add.mod.py` will all have version `V1_0`. However there are several exceptions that a certain operation is under-tested in previous version and more tests are added in a later version. In this case, two methods are provided to set the version manually.
317
318#### Set the version when creating the model
319
320Use `IntroducedIn` to set the version of a model. All variations of the model will have the same version.
321
322```Python
323model_V1_0 = Model().IntroducedIn("V1_0")
324...
325# All variations of model_V1_0 will have the same version V1_0.
326Example(example, model=model_V1_0).AddVariations(var0, var1, ...)
327```
328
329#### Set the version overrides
330
331Use `Example.SetVersion` to override the model version for specific tests. The target tests are specified by names. This method can also override the version specified by `IntroducedIn`.
332
333```Python
334Example.SetVersion(<version>, testName0, testName1, ...)
335```
336
337This is useful when only a subset of variations has a different version.
338
339### A Complete Example
340
341```Python
342# Declare input, output, and parameters
343i1 = Input("op1", "TENSOR_FLOAT32", "{1, 3, 4, 1}")
344f1 = Parameter("op2", "TENSOR_FLOAT32", "{1, 3, 3, 1}", [1, 4, 7, 2, 5, 8, 3, 6, 9])
345b1 = Parameter("op3", "TENSOR_FLOAT32", "{1}", [-200])
346act = Int32Scalar("act", 0)
347o1 = Output("op4", "TENSOR_FLOAT32", "{1, 3, 4, 1}")
348
349# Instantiate a model and add CONV_2D operation
350# Use implicit parameter for implicit padding and strides
351Model().Operation("CONV_2D", i1, f1, b1, 1, 1, 1, act, layout).To(o1)
352
353# Additional data type
354quant8 = DataTypeConverter().Identify({
355    i1: ("TENSOR_QUANT8_ASYMM", 0.5, 127),
356    f1: ("TENSOR_QUANT8_ASYMM", 0.5, 127),
357    b1: ("TENSOR_INT32", 0.25, 0),
358    o1: ("TENSOR_QUANT8_ASYMM", 1.0, 50)
359})
360
361# Instantiate an example
362example = Example({
363    i1: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12],
364    o1: [0, 0, 0, 0, 35, 112, 157, 0, 0, 34, 61, 0]
365})
366
367# Only use NCHW data layout
368example.AddNchw(i1, f1, o1, layout, includeDefault=False)
369
370# Add two more groups of variations
371example.AddInput(f1, b1).AddVariations("relaxed", quant8).AddAllActivations(o1, act)
372```
373
374The spec above will result in 24 tests.
375
376## Generate Tests
377
378Once you have your model ready, run
379
380```
381$ANDROID_BUILD_TOP/frameworks/ml/nn/runtime/test/specs/generate_test.sh
382$ANDROID_BUILD_TOP/frameworks/ml/nn/runtime/test/specs/generate_vts_test.sh
383```
384
385It will read and generate all CTS/VTS unit tests based on spec files in `nn/runtime/test/specs/V1_*/*` if needed. CTS test generator is able to identify which spec files are modified since last generation and only regenerate those files to reduce compilation time. To force a regeneration, use `-f` flag.
386
387```
388$ANDROID_BUILD_TOP/frameworks/ml/nn/runtime/test/specs/generate_test.sh -f
389```
390
391If you only want to regenerate a certain set of files, simply append the file names to the end of the command, and optionally, use `-f` flag.
392
393```
394$ANDROID_BUILD_TOP/frameworks/ml/nn/runtime/test/specs/generate_test.sh -f file1.mod.py file2.mod.py ...
395```
396
397Rebuild with mm afterwards.
398