1# Integrate object detectors
2
3Object detectors can identify which of a known set of objects might be present
4and provide information about their positions within the given image or a video
5stream. An object detector is trained to detect the presence and location of
6multiple classes of objects. For example, a model might be trained with images
7that contain various pieces of fruit, along with a _label_ that specifies the
8class of fruit they represent (e.g. an apple, a banana, or a strawberry), and
9data specifying where each object appears in the image. See the
10[introduction of object detection](../../models/object_detection/overview.md)
11for more information about object detectors.
12
13Use the Task Library `ObjectDetector` API to deploy your custom object detectors
14or pretrained ones into your model apps.
15
16## Key features of the ObjectDetector API
17
18*   Input image processing, including rotation, resizing, and color space
19    conversion.
20
21*   Label map locale.
22
23*   Score threshold to filter results.
24
25*   Top-k detection results.
26
27*   Label allowlist and denylist.
28
29## Supported object detector models
30
31The following models are guaranteed to be compatible with the `ObjectDetector`
32API.
33
34*   The
35    [pretrained object detection models on TensorFlow Hub](https://tfhub.dev/tensorflow/collections/lite/task-library/object-detector/1).
36
37*   Models created by
38    [AutoML Vision Edge Object Detection](https://cloud.google.com/vision/automl/object-detection/docs).
39
40*   Custom models that meet the
41    [model compatibility requirements](#model-compatibility-requirements).
42
43## Run inference in Java
44
45See the
46[Object Detection reference app](https://github.com/tensorflow/examples/tree/master/lite/examples/object_detection/android/)
47for an example of how to use `ObjectDetector` in an Android app.
48
49### Step 1: Import Gradle dependency and other settings
50
51Copy the `.tflite` model file to the assets directory of the Android module
52where the model will be run. Specify that the file should not be compressed, and
53add the TensorFlow Lite library to the module’s `build.gradle` file:
54
55```java
56android {
57    // Other settings
58
59    // Specify tflite file should not be compressed for the app apk
60    aaptOptions {
61        noCompress "tflite"
62    }
63
64}
65
66dependencies {
67    // Other dependencies
68
69    // Import the Task Vision Library dependency
70    implementation 'org.tensorflow:tensorflow-lite-task-vision:0.1.0'
71}
72```
73
74### Step 2: Using the model
75
76```java
77// Initialization
78ObjectDetectorOptions options = ObjectDetectorOptions.builder().setMaxResults(1).build();
79ObjectDetector objectDetector = ObjectDetector.createFromFileAndOptions(context, modelFile, options);
80
81// Run inference
82List<Detection> results = objectDetector.detect(image);
83```
84
85See the
86[source code and javadoc](https://github.com/tensorflow/tflite-support/blob/master/tensorflow_lite_support/java/src/java/org/tensorflow/lite/task/vision/detector/ObjectDetector.java)
87for more options to configure `ObjectDetector`.
88
89## Run inference in C++
90
91Note: we are working on improving the usability of the C++ Task Library, such as
92providing prebuilt binaries and creating user-friendly workflows to build from
93source code. The C++ API may be subject to change.
94
95```c++
96// Initialization
97ObjectDetectorOptions options;
98options.mutable_model_file_with_metadata()->set_file_name(model_file);
99std::unique_ptr<ObjectDetector> object_detector = ObjectDetector::CreateFromOptions(options).value();
100
101// Run inference
102const DetectionResult result = object_detector->Detect(*frame_buffer).value();
103```
104
105See the
106[source code](https://github.com/tensorflow/tflite-support/blob/master/tensorflow_lite_support/cc/task/vision/object_detector.h)
107for more options to configure `ObjectDetector`.
108
109## Example results
110
111Here is an example of the detection results of
112[ssd mobilenet v1](https://tfhub.dev/tensorflow/lite-model/ssd_mobilenet_v1/1/metadata/1)
113from TensorFlow Hub.
114
115<img src="images/dogs.jpg" alt="dogs" width="50%">
116
117```
118Results:
119 Detection #0 (red):
120  Box: (x: 355, y: 133, w: 190, h: 206)
121  Top-1 class:
122   index       : 17
123   score       : 0.73828
124   class name  : dog
125 Detection #1 (green):
126  Box: (x: 103, y: 15, w: 138, h: 369)
127  Top-1 class:
128   index       : 17
129   score       : 0.73047
130   class name  : dog
131```
132
133Render the bounding boxes onto the input image:
134
135<img src="images/detection-output.png" alt="detection output" width="50%">
136
137Try out the simple
138[CLI demo tool for ObjectDetector](https://github.com/tensorflow/tflite-support/tree/master/tensorflow_lite_support/examples/task/vision/desktop#object-detector)
139with your own model and test data.
140
141## Model compatibility requirements
142
143The `ObjectDetector` API expects a TFLite model with mandatory
144[TFLite Model Metadata](../../convert/metadata.md).
145
146The compatible object detector models should meet the following requirements:
147
148*   Input image tensor: (kTfLiteUInt8/kTfLiteFloat32)
149
150    -   image input of size `[batch x height x width x channels]`.
151    -   batch inference is not supported (`batch` is required to be 1).
152    -   only RGB inputs are supported (`channels` is required to be 3).
153    -   if type is kTfLiteFloat32, NormalizationOptions are required to be
154        attached to the metadata for input normalization.
155
156*   Output tensors must be the 4 outputs of a `DetectionPostProcess` op, i.e:
157
158    -   Locations tensor (kTfLiteFloat32)
159        -   tensor of size `[1 x num_results x 4]`, the inner array representing
160            bounding boxes in the form [top, left, right, bottom].
161        -   BoundingBoxProperties are required to be attached to the metadata
162            and must specify `type=BOUNDARIES` and `coordinate_type=RATIO.
163    -   Classes tensor (kTfLiteFloat32)
164
165        -   tensor of size `[1 x num_results]`, each value representing the
166            integer index of a class.
167        -   optional (but recommended) label map(s) can be attached as
168            AssociatedFile-s with type TENSOR_VALUE_LABELS, containing one label
169            per line. The first such AssociatedFile (if any) is used to fill the
170            `class_name` field of the results. The `display_name` field is
171            filled from the AssociatedFile (if any) whose locale matches the
172            `display_names_locale` field of the `ObjectDetectorOptions` used at
173            creation time ("en" by default, i.e. English). If none of these are
174            available, only the `index` field of the results will be filled.
175
176    -   Scores tensor (kTfLiteFloat32)
177
178        -   tensor of size `[1 x num_results]`, each value representing the
179            score of the detected object.
180
181    -   Number of detection tensor (kTfLiteFloat32)
182
183        -   integer num_results as a tensor of size `[1]`.
184