1# Using the NN-API Test Generator 2 3## Prerequisites 4 5- Python3 6- Numpy 7 8## Writing a Test Specification 9 10You should create new test specs in `nn/runtime/test/specs/<version>/` and name it with `.mod.py` suffix, so that other tools can automatically update the unit tests. 11 12### Specifying Operands 13 14#### Syntax 15 16``` 17OperandType(name, (type, shape, <optional scale, zero point>), <optional initializer>) 18``` 19 20For example, 21 22```Python 23# p1 is a 2-by-2 fp matrix parameter, with value [1, 2; 3, 4] 24p1 = Parameter("param", ("TENSOR_FLOAT32", [2, 2]), [1, 2, 3, 4]) 25 26# i1 is a quantized input of shape (2, 256, 256, 3), with scale = 0.5, zero point = 128 27i1 = Input("input", ("TENSOR_QUANT8_ASYMM", [2, 256, 256, 3], 0.5, 128)) 28 29# p2 is an Int32 scalar with value 1 30p2 = Int32Scalar("act", 1) 31``` 32 33#### OperandType 34 35There are currently 10 operand types supported by the test generator. 36 37- Input 38- Output 39 * IgnoredOutput, will not compare results in the test 40- Parameter 41 * Int32Scalar, shorthand for parameter with type INT32 42 * Float32Scalar, shorthand for parameter with type FLOAT32 43 * Int32Vector, shorthand for 1-D TENSOR_INT32 parameter 44 * Float32Vector, shorthand for 1-D TENSOR_FLOAT32 parameter 45- Internal, for model with multiple operations 46 47### Specifying Models 48 49#### Instantiate a model 50 51```Python 52# Instantiate a model 53model = Model() 54 55# Instantiate a model with a name 56model2 = Model("model_name") 57``` 58 59#### Add an operation 60 61``` 62model.Operation(optype, i1, i2, ...).To(o1, o2, ...) 63``` 64 65For example, 66 67```Python 68model.Operation("ADD", i1, i2, act).To(o1) 69``` 70 71#### Use implicit operands 72 73Simple scalar and 1-D vector parameters can now be directly passed to Operation constructor, and test generator will deduce the operand type from the value provided. 74 75```Python 76model.Operation("MEAN", i1, [1], 0) # axis = [1], keep_dims = 0 77``` 78 79Note that, for fp values, the initializer should all be Python fp numbers, e.g. use `1.0` or `1.` instead of `1` for implicit fp operands. 80 81### Specifying Inputs and Expected Outputs 82 83The combination of inputs and expected outputs is called an example for a given model. An example is defined like 84 85```Python 86# Example 1, separate dictionary for inputs and outputs 87input1 = { 88 i1: [1, 2], 89 i2: [3, 4] 90} 91output1 = {o1: [4, 6]} 92 93# Example 2, combined dictionary 94example2_values = { 95 i1: [5, 6], 96 i2: [7, 8], 97 o1: [12, 14] 98} 99 100# Instantiate an example 101Example((input1, output1), example2_values) 102``` 103 104By default, examples will be attached to the most recent instantiated model. You can explicitly specify the target model, and optionally, the example name by 105 106```Python 107Example((input1, output1), example2_values, model=model, name="example_name") 108``` 109 110### Specifying Variations 111 112You can add variations to the example so that the test generator can automatically create multiple tests. Currently, 6 types of variation are supported: 113 114- DefaultVariation, i.e. no variation 115- DataTypeConverter 116- DataLayoutConverter 117- AxisConverter 118- RelaxedModeConverter 119- ParameterAsInputConverter 120- ActivationConverter 121 122#### DataTypeConverter 123 124Convert input/parameter/output to the specified type, e.g. float32 -> quant8. The target data type for each operand to transform has to be explicitly specified. It is the spec writer's responsibility to ensure such conversion is valid. 125 126```Python 127converter = DataTypeConverter(name="variation_name").Identify({ 128 op1: (target_type, target_scale, target_zero_point), 129 op2: (target_type, target_scale, target_zero_point), 130 ... 131}) 132``` 133 134#### DataLayoutConverter 135 136Convert input/parameter/output between NHWC and NCHW. The caller need to provide a list of target operands to transform, and also the data layout parameter to set. 137 138```Python 139converter = DataLayoutConverter(target_data_layout, name="variation_name").Identify( 140 [op1, op2, ..., layout_parameter] 141) 142``` 143 144#### AxisConverter 145 146Transpose a certain axis in input/output to target position, and optionally remove some axis. The caller need to provide a list of target operands to transform, and also the axis parameter to set. 147 148```Python 149converter = AxisConverter(originalAxis, targetAxis, dimension, drop=[], name="variation_name").Identify( 150 [op1, op2, ..., axis_parameter] 151) 152``` 153 154This model variation is for ops that apply calculation along certain axis, such as L2_NORMALIZATION, SOFTMAX, and CHANNEL_SHUFFLE. For example, consider L2_NORMALIZATION with input of shape [2, 3, 4, 5] along the last axis, i.e. axis = -1. The output shape would be the same as input. We can create a new model which will do the calculation along axis 0 by transposing input and output shape to [5, 2, 3, 4] and modify the axis parameter to 0. Such converter can be defined as 155 156```Python 157toAxis0 = AxisConverter(-1, 0, 4).Identify([input, output, axis]) 158``` 159 160The target axis can also be negative to test the negative indexing 161 162```Python 163toAxis0 = AxisConverter(-1, -4, 4).Identify([input, output, axis]) 164``` 165 166Consider the same L2_NORMALIZATION example, we can also create a new model with input/output of 2D shape [4, 5] by removing the first two dimension. This is essentially doing `new_input = input[0,0,:,:]` in numpy. Such converter can be defined as 167 168```Python 169toDim2 = AxisConverter(-1, -1, 4, drop=[0, 1]).Identify([input, output, axis]) 170``` 171 172If transposition and removal are specified at the same time, the converter will do transposition first and then remove the axis. For example, the following converter will result in shape [5, 4] and axis 0. 173 174```Python 175toDim2Axis0 = AxisConverter(-1, 2, 4, drop=[0, 1]).Identify([input, output, axis]) 176``` 177 178#### RelaxedModeConverter 179 180Convert the model to enable/disable relaxed computation. 181 182```Python 183converter = RelaxedModeConverter(is_relaxed, name="variation_name") 184``` 185 186#### ParameterAsInputConverter 187 188Convert a certain parameter to model input, e.g. weight in CONV_2D. The caller need to provide a list of target operands to convert. 189 190```Python 191converter = ParameterAsInputConverter(name="variation_name").Identify( 192 [op1, op2, ...] 193) 194``` 195 196#### ActivationConverter 197 198Convert the output by certain activation, the original activation is assumed to be NONE. The caller need to provide a list of target operands to transform, and also the activation parameter to set. 199 200```Python 201converter = ActivationConverter(name="variation_name").Identify( 202 [op1, op2, ..., act_parameter] 203) 204``` 205 206#### Add variation to example 207 208Each example can have multiple groups of variations, and if so, will take the cartesian product of the groups. For example, suppose we declare a model with two groups, and each group has two variations: `[[default, nchw], [default, relaxed, quant8]]`. This will result in 6 examples: `[default, default], [default, relaxed], [default, quant8], [nchw, default], [nchw, relaxed], [nchw, quant8]`. 209 210Use `AddVariations` to add a group of variations to the example 211 212```Python 213# Add two groups of variations [default, nchw] and [default, relaxed, quant8] 214example.AddVariations(nchw).AddVariations(relaxed, quant8) 215``` 216 217By default, when you add a group of variation, a unnamed default variation will be automatically included in the list. You can name the default variation by 218 219```Python 220example.AddVariations(nchw, defaultName="nhwc").AddVariations(relaxed, quant8) 221``` 222 223Also, you can choose not to include default by 224 225```Python 226# Add two groups of variations [nchw] and [default, relaxed, quant8] 227example.AddVariations(nchw, includeDefault=False).AddVariations(relaxed, quant8) 228``` 229 230The example above will result in 3 examples: `[nchw, default], [nchw, relaxed], [nchw, quant8]`. 231 232#### Some helper functions 233 234The test generator provides several helper functions or shorthands to add commonly used group of variations. 235 236```Python 237# Each following group of statements are equivalent 238 239# DataTypeConverter 240example.AddVariations(DataTypeConverter().Identify({op1: "TENSOR_FLOAT16", ...})) 241example.AddVariations("float16") # will apply to every TENSOR_FLOAT32 operands 242 243example.AddVariations(DataTypeConverter().Identify({op1: "TENSOR_INT32", ...})) 244example.AddVariations("int32") # will apply to every TENSOR_FLOAT32 operands 245 246# DataLayoutConverter 247example.AddVariations(DataLayoutConverter("nchw").Identify(op_list)) 248example.AddVariations(("nchw", op_list)) 249example.AddNchw(*op_list) 250 251# AxisConverter 252# original axis and dim are deduced from the op_list 253example.AddVariations(*[AxisConverter(origin, t, dim).Identify(op_list) for t in targets]) 254example.AddAxis(targets, *op_list) 255 256example.AddVariations(*[ 257 AxisConverter(origin, t, dim).Identify(op_list) for t in range(dim) 258 ], includeDefault=False) 259example.AddAllPositiveAxis(*op_list) 260 261example.AddVariations(*[ 262 AxisConverter(origin, t, dim).Identify(op_list) for t in range(-dim, dim) 263 ], includeDefault=False) 264example.AddAllAxis(*op_list) 265 266drop = list(range(dim)) 267drop.pop(origin) 268example.AddVariations(*[ 269 AxisConverter(origin, origin, dim, drop[0:(dim-i)]).Identify(op_list) for i in dims]) 270example.AddDims(dims, *op_list) 271 272example.AddVariations(*[ 273 AxisConverter(origin, origin, dim, drop[0:i]).Identify(op_list) for i in range(dim)]) 274example.AddAllDims(dims, *op_list) 275 276example.AddVariations(*[ 277 AxisConverter(origin, j, dim, range(i)).Identify(op_list) \ 278 for i in range(dim) for j in range(i, dim) 279 ], includeDefault=False) 280example.AddAllDimsAndPositiveAxis(dims, *op_list) 281 282example.AddVariations(*[ 283 AxisConverter(origin, k, dim, range(i)).Identify(op_list) \ 284 for i in range(dim) for j in range(i, dim) for k in [j, j - dim] 285 ], includeDefault=False) 286example.AddAllDimsAndAxis(dims, *op_list) 287 288# ParameterAsInputConverter 289example.AddVariations(ParameterAsInputConverter().Identify(op_list)) 290example.AddVariations(("as_input", op_list)) 291example.AddInput(*op_list) 292 293# RelaxedModeConverter 294example.Addvariations(RelaxedModeConverter(True)) 295example.AddVariations("relaxed") 296example.AddRelaxed() 297 298# ActivationConverter 299example.AddVariations(ActivationConverter("relu").Identify(op_list)) 300example.AddVariations(("relu", op_list)) 301example.AddRelu(*op_list) 302 303example.AddVariations( 304 ActivationConverter("relu").Identify(op_list), 305 ActivationConverter("relu1").Identify(op_list), 306 ActivationConverter("relu6").Identify(op_list)) 307example.AddVariations( 308 ("relu", op_list), 309 ("relu1", op_list), 310 ("relu6", op_list)) 311example.AddAllActivations(*op_list) 312``` 313 314### Specifying the Model Version 315 316If not explicitly specified, the minimal required HAL version will be inferred from the path, e.g. the models defined in `nn/runtime/test/specs/V1_0/add.mod.py` will all have version `V1_0`. However there are several exceptions that a certain operation is under-tested in previous version and more tests are added in a later version. In this case, two methods are provided to set the version manually. 317 318#### Set the version when creating the model 319 320Use `IntroducedIn` to set the version of a model. All variations of the model will have the same version. 321 322```Python 323model_V1_0 = Model().IntroducedIn("V1_0") 324... 325# All variations of model_V1_0 will have the same version V1_0. 326Example(example, model=model_V1_0).AddVariations(var0, var1, ...) 327``` 328 329#### Set the version overrides 330 331Use `Example.SetVersion` to override the model version for specific tests. The target tests are specified by names. This method can also override the version specified by `IntroducedIn`. 332 333```Python 334Example.SetVersion(<version>, testName0, testName1, ...) 335``` 336 337This is useful when only a subset of variations has a different version. 338 339### A Complete Example 340 341```Python 342# Declare input, output, and parameters 343i1 = Input("op1", "TENSOR_FLOAT32", "{1, 3, 4, 1}") 344f1 = Parameter("op2", "TENSOR_FLOAT32", "{1, 3, 3, 1}", [1, 4, 7, 2, 5, 8, 3, 6, 9]) 345b1 = Parameter("op3", "TENSOR_FLOAT32", "{1}", [-200]) 346act = Int32Scalar("act", 0) 347o1 = Output("op4", "TENSOR_FLOAT32", "{1, 3, 4, 1}") 348 349# Instantiate a model and add CONV_2D operation 350# Use implicit parameter for implicit padding and strides 351Model().Operation("CONV_2D", i1, f1, b1, 1, 1, 1, act, layout).To(o1) 352 353# Additional data type 354quant8 = DataTypeConverter().Identify({ 355 i1: ("TENSOR_QUANT8_ASYMM", 0.5, 127), 356 f1: ("TENSOR_QUANT8_ASYMM", 0.5, 127), 357 b1: ("TENSOR_INT32", 0.25, 0), 358 o1: ("TENSOR_QUANT8_ASYMM", 1.0, 50) 359}) 360 361# Instantiate an example 362example = Example({ 363 i1: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12], 364 o1: [0, 0, 0, 0, 35, 112, 157, 0, 0, 34, 61, 0] 365}) 366 367# Only use NCHW data layout 368example.AddNchw(i1, f1, o1, layout, includeDefault=False) 369 370# Add two more groups of variations 371example.AddInput(f1, b1).AddVariations("relaxed", quant8).AddAllActivations(o1, act) 372``` 373 374The spec above will result in 24 tests. 375 376## Generate Tests 377 378Once you have your model ready, run 379 380``` 381$ANDROID_BUILD_TOP/frameworks/ml/nn/runtime/test/specs/generate_test.sh 382$ANDROID_BUILD_TOP/frameworks/ml/nn/runtime/test/specs/generate_vts_test.sh 383``` 384 385It will read and generate all CTS/VTS unit tests based on spec files in `nn/runtime/test/specs/V1_*/*` if needed. CTS test generator is able to identify which spec files are modified since last generation and only regenerate those files to reduce compilation time. To force a regeneration, use `-f` flag. 386 387``` 388$ANDROID_BUILD_TOP/frameworks/ml/nn/runtime/test/specs/generate_test.sh -f 389``` 390 391If you only want to regenerate a certain set of files, simply append the file names to the end of the command, and optionally, use `-f` flag. 392 393``` 394$ANDROID_BUILD_TOP/frameworks/ml/nn/runtime/test/specs/generate_test.sh -f file1.mod.py file2.mod.py ... 395``` 396 397Rebuild with mm afterwards. 398