1# Integrate image segmenters
2
3Image segmenters predict whether each pixel of an image is associated with a
4certain class. This is in contrast to
5<a href="../../models/object_detection/overview.md">object detection</a>, which
6detects objects in rectangular regions, and
7<a href="../../models/image_classification/overview.md">image
8classification</a>, which classifies the overall image. See the
9[introduction of image segmentation](../../models/segmentation/overview.md) for
10more information about image segmenters.
11
12Use the Task Library `ImageSegmenter` API to deploy your custom image segmenters
13or pretrained ones into your model apps.
14
15## Key features of the ImageSegmenter API
16
17*   Input image processing, including rotation, resizing, and color space
18    conversion.
19
20*   Label map locale.
21
22*   Two output types, category mask and confidence masks.
23
24*   Colored label for display purpose.
25
26## Supported image segmenter models
27
28The following models are guaranteed to be compatible with the `ImageSegmenter`
29API.
30
31*   The
32    [pretrained image segmentation models on TensorFlow Hub](https://tfhub.dev/tensorflow/collections/lite/task-library/image-segmenter/1).
33
34*   Custom models that meet the
35    [model compatibility requirements](#model-compatibility-requirements).
36
37## Run inference in Java
38
39See the
40[Image Segmentation reference app](https://github.com/tensorflow/examples/tree/master/lite/examples/image_segmentation/android/)
41for an example of how to use `ImageSegmenter` in an Android app.
42
43### Step 1: Import Gradle dependency and other settings
44
45Copy the `.tflite` model file to the assets directory of the Android module
46where the model will be run. Specify that the file should not be compressed, and
47add the TensorFlow Lite library to the module’s `build.gradle` file:
48
49```java
50android {
51    // Other settings
52
53    // Specify tflite file should not be compressed for the app apk
54    aaptOptions {
55        noCompress "tflite"
56    }
57
58}
59
60dependencies {
61    // Other dependencies
62
63    // Import the Task Vision Library dependency
64    implementation 'org.tensorflow:tensorflow-lite-task-vision:0.1.0'
65}
66```
67
68### Step 2: Using the model
69
70```java
71// Initialization
72ImageSegmenterOptions options = ImageSegmenterOptions.builder().setOutputType(OutputType.CONFIDENCE_MASK).build();
73ImageSegmenter imageSegmenter = ImageSegmenter.createFromFileAndOptions(context, modelFile, options);
74
75// Run inference
76List<Segmentation> results = imageSegmenter.segment(image);
77```
78
79See the
80[source code and javadoc](https://github.com/tensorflow/tflite-support/blob/master/tensorflow_lite_support/java/src/java/org/tensorflow/lite/task/vision/segmenter/ImageSegmenter.java)
81for more options to configure `ImageSegmenter`.
82
83## Run inference in C++
84
85Note: we are working on improving the usability of the C++ Task Library, such as
86providing prebuilt binaries and creating user-friendly workflows to build from
87source code. The C++ API may be subject to change.
88
89```c++
90// Initialization
91ImageSegmenterOptions options;
92options.mutable_model_file_with_metadata()->set_file_name(model_file);
93std::unique_ptr<ImageSegmenter> image_segmenter = ImageSegmenter::CreateFromOptions(options).value();
94
95// Run inference
96const SegmentationResult result = image_segmenter->Segment(*frame_buffer).value();
97```
98
99See the
100[source code](https://github.com/tensorflow/tflite-support/blob/master/tensorflow_lite_support/cc/task/vision/image_segmenter.h)
101for more options to configure `ImageSegmenter`.
102
103## Example results
104
105Here is an example of the segmentation results of
106[deeplab_v3](https://tfhub.dev/tensorflow/lite-model/deeplabv3/1/metadata/1), a
107generic segmentation model available on TensorFlow Hub.
108
109<img src="images/plane.jpg" alt="plane" width="50%">
110
111```
112Color Legend:
113 (r: 000, g: 000, b: 000):
114  index       : 0
115  class name  : background
116 (r: 128, g: 000, b: 000):
117  index       : 1
118  class name  : aeroplane
119
120# (omitting multiple lines for conciseness) ...
121
122 (r: 128, g: 192, b: 000):
123  index       : 19
124  class name  : train
125 (r: 000, g: 064, b: 128):
126  index       : 20
127  class name  : tv
128Tip: use a color picker on the output PNG file to inspect the output mask with
129this legend.
130```
131
132The segmentation category mask should looks like:
133
134<img src="images/segmentation-output.png" alt="segmentation-output" width="30%">
135
136Try out the simple
137[CLI demo tool for ImageSegmenter](https://github.com/tensorflow/tflite-support/tree/master/tensorflow_lite_support/examples/task/vision/desktop#image-segmenter)
138with your own model and test data.
139
140## Model compatibility requirements
141
142The `ImageSegmenter` API expects a TFLite model with mandatory
143[TFLite Model Metadata](../../convert/metadata.md).
144
145*   Input image tensor (kTfLiteUInt8/kTfLiteFloat32)
146
147    -   image input of size `[batch x height x width x channels]`.
148    -   batch inference is not supported (`batch` is required to be 1).
149    -   only RGB inputs are supported (`channels` is required to be 3).
150    -   if type is kTfLiteFloat32, NormalizationOptions are required to be
151        attached to the metadata for input normalization.
152
153*   Output masks tensor: (kTfLiteUInt8/kTfLiteFloat32)
154
155    -   tensor of size `[batch x mask_height x mask_width x num_classes]`, where
156        `batch` is required to be 1, `mask_width` and `mask_height` are the
157        dimensions of the segmentation masks produced by the model, and
158        `num_classes` is the number of classes supported by the model.
159    -   optional (but recommended) label map(s) can be attached as
160        AssociatedFile-s with type TENSOR_AXIS_LABELS, containing one label per
161        line. The first such AssociatedFile (if any) is used to fill the `label`
162        field (named as `class_name` in C++) of the results. The `display_name`
163        field is filled from the AssociatedFile (if any) whose locale matches
164        the `display_names_locale` field of the `ImageSegmenterOptions` used at
165        creation time ("en" by default, i.e. English). If none of these are
166        available, only the `index` field of the results will be filled.
167