1# Segmentation
2
3Image segmentation is the process of partitioning a digital image into multiple
4segments (sets of pixels, also known as image objects). The goal of segmentation
5is to simplify and/or change the representation of an image into something that
6is more meaningful and easier to analyze.
7
8The following image shows the output of the image segmentation model on Android.
9The model will create a mask over the target objects with high accuracy.
10
11<img src="images/segmentation.gif" class="attempt-right" />
12
13## Get started
14
15If you are new to TensorFlow Lite and are working with Android or iOS, it is
16recommended you explore the following example applications that can help you get
17started.
18
19You can leverage the out-of-box API from
20[TensorFlow Lite Task Library](../../inference_with_metadata/task_library/image_segmenter)
21to integrate image segmentation models within just a few lines of code. You can
22also integrate the model using the
23[TensorFlow Lite Interpreter Java API](../../guide/inference#load_and_run_a_model_in_java).
24
25The Android example below demonstrates the implementation for both methods as
26[lib_task_api](https://github.com/tensorflow/examples/tree/master/lite/examples/image_segmentation/android/lib_task_api)
27and
28[lib_interpreter](https://github.com/tensorflow/examples/tree/master/lite/examples/image_segmentation/android/lib_interpreter),
29respectively.
30
31<a class="button button-primary" href="https://github.com/tensorflow/examples/tree/master/lite/examples/image_segmentation/android">View
32Android example</a>
33
34<a class="button button-primary" href="https://github.com/tensorflow/examples/tree/master/lite/examples/image_segmentation/ios">View
35iOS example</a>
36
37If you are using a platform other than Android or iOS, or you are already
38familiar with the
39<a href="https://www.tensorflow.org/api_docs/python/tf/lite">TensorFlow Lite
40APIs</a>, you can download our starter image segmentation model.
41
42<a class="button button-primary" href="https://tfhub.dev/tensorflow/lite-model/deeplabv3/1/metadata/2?lite-format=tflite">Download
43starter model</a>
44
45## Model description
46
47_DeepLab_ is a state-of-art deep learning model for semantic image segmentation,
48where the goal is to assign semantic labels (e.g. person, dog, cat) to every
49pixel in the input image.
50
51### How it works
52
53Semantic image segmentation predicts whether each pixel of an image is
54associated with a certain class. This is in contrast to
55<a href="../object_detection/overview.md">object detection</a>, which detects
56objects in rectangular regions, and
57<a href="../image_classification/overview.md">image classification</a>, which
58classifies the overall image.
59
60The current implementation includes the following features:
61<ol>
62  <li>DeepLabv1: We use atrous convolution to explicitly control the resolution at which feature responses are computed within Deep Convolutional Neural Networks.</li>
63  <li>DeepLabv2: We use atrous spatial pyramid pooling (ASPP) to robustly segment objects at multiple scales with filters at multiple sampling rates and effective fields-of-views.</li>
64  <li>DeepLabv3: We augment the ASPP module with image-level feature [5, 6] to capture longer range information. We also include batch normalization [7] parameters to facilitate the training. In particular, we applying atrous convolution to extract output features at different output strides during training and evaluation, which efficiently enables training BN at output stride = 16 and attains a high performance at output stride = 8 during evaluation.</li>
65  <li>DeepLabv3+: We extend DeepLabv3 to include a simple yet effective decoder module to refine the segmentation results especially along object boundaries. Furthermore, in this encoder-decoder structure one can arbitrarily control the resolution of extracted encoder features by atrous convolution to trade-off precision and runtime.</li>
66</ol>
67
68## Performance benchmarks
69
70Performance benchmark numbers are generated with the tool
71[described here](https://www.tensorflow.org/lite/performance/benchmarks).
72
73<table>
74  <thead>
75    <tr>
76      <th>Model Name</th>
77      <th>Model size </th>
78      <th>Device </th>
79      <th>GPU</th>
80      <th>CPU</th>
81    </tr>
82  </thead>
83  <tr>
84    <td rowspan = 3>
85      <a href="https://tfhub.dev/tensorflow/lite-model/deeplabv3/1/metadata/2?lite-format=tflite">Deeplab v3</a>
86    </td>
87    <td rowspan = 3>
88      2.7 Mb
89    </td>
90    <td>Pixel 3 (Android 10) </td>
91    <td>16ms</td>
92    <td>37ms*</td>
93  </tr>
94   <tr>
95     <td>Pixel 4 (Android 10) </td>
96    <td>20ms</td>
97    <td>23ms*</td>
98  </tr>
99   <tr>
100     <td>iPhone XS (iOS 12.4.1) </td>
101     <td>16ms</td>
102    <td>25ms** </td>
103  </tr>
104</table>
105
106\* 4 threads used.
107
108\*\* 2 threads used on iPhone for the best performance result.
109
110## Further reading and resources
111
112<ul>
113  <li><a href="https://ai.googleblog.com/2018/03/semantic-image-segmentation-with.html">Semantic Image Segmentation with DeepLab in TensorFlow</a></li>
114  <li><a href="https://medium.com/tensorflow/tensorflow-lite-now-faster-with-mobile-gpus-developer-preview-e15797e6dee7">TensorFlow Lite Now Faster with Mobile GPUs (Developer Preview)</a></li>
115  <li><a href="https://github.com/tensorflow/models/tree/master/research/deeplab">DeepLab: Deep Labelling for Semantic Image Segmentation</a></li>
116</ul>
117