1# Tensorflow Lite Core ML delegate
2
3The TensorFlow Lite Core ML delegate enables running TensorFlow Lite models on
4[Core ML framework](https://developer.apple.com/documentation/coreml), which
5results in faster model inference on iOS devices.
6
7Note: This delegate is in experimental (beta) phase. It is available from
8TensorFlow Lite 2.4.0 and latest nightly releases.
9
10Note: Core ML delegate supports Core ML version 2 and later.
11
12**Supported iOS versions and devices:**
13
14*   iOS 12 and later. In the older iOS versions, Core ML delegate will
15    automatically fallback to CPU.
16*   By default, Core ML delegate will only be enabled on devices with A12 SoC
17    and later (iPhone Xs and later) to use Neural Engine for faster inference.
18    If you want to use Core ML delegate also on the older devices, please see
19    [best practices](#best-practices)
20
21**Supported models**
22
23The Core ML delegate currently supports float (FP32 and FP16) models.
24
25## Trying the Core ML delegate on your own model
26
27The Core ML delegate is already included in nightly release of TensorFlow lite
28CocoaPods. To use Core ML delegate, change your TensorFlow lite pod to include
29subspec `CoreML` in your `Podfile`.
30
31Note: If you want to use C API instead of Objective-C API, you can include
32`TensorFlowLiteC/CoreML` pod to do so.
33
34```
35target 'YourProjectName'
36  pod 'TensorFlowLiteSwift/CoreML', '~> 2.4.0'  # Or TensorFlowLiteObjC/CoreML
37```
38
39OR
40
41```
42# Particularily useful when you also want to include 'Metal' subspec.
43target 'YourProjectName'
44  pod 'TensorFlowLiteSwift', '~> 2.4.0', :subspecs => ['CoreML']
45```
46
47Note: Core ML delegate can also use C API for Objective-C code. Prior to
48TensorFlow Lite 2.4.0 release, this was the only option.
49
50<div>
51  <devsite-selector>
52    <section>
53      <h3>Swift</h3>
54      <p><pre class="prettyprint lang-swift">
55    let coreMLDelegate = CoreMLDelegate()
56    var interpreter: Interpreter
57
58    // Core ML delegate will only be created for devices with Neural Engine
59    if coreMLDelegate != nil {
60      interpreter = try Interpreter(modelPath: modelPath,
61                                    delegates: [coreMLDelegate!])
62    } else {
63      interpreter = try Interpreter(modelPath: modelPath)
64    }
65      </pre></p>
66    </section>
67    <section>
68      <h3>Objective-C</h3>
69      <p><pre class="prettyprint lang-objc">
70
71    // Import module when using CocoaPods with module support
72    @import TFLTensorFlowLite;
73
74    // Or import following headers manually
75    # import "tensorflow/lite/objc/apis/TFLCoreMLDelegate.h"
76    # import "tensorflow/lite/objc/apis/TFLTensorFlowLite.h"
77
78    // Initialize Core ML delegate
79    TFLCoreMLDelegate* coreMLDelegate = [[TFLCoreMLDelegate alloc] init];
80
81    // Initialize interpreter with model path and Core ML delegate
82    TFLInterpreterOptions* options = [[TFLInterpreterOptions alloc] init];
83    NSError* error = nil;
84    TFLInterpreter* interpreter = [[TFLInterpreter alloc]
85                                    initWithModelPath:modelPath
86                                              options:options
87                                            delegates:@[ coreMLDelegate ]
88                                                error:&amp;error];
89    if (error != nil) { /* Error handling... */ }
90
91    if (![interpreter allocateTensorsWithError:&amp;error]) { /* Error handling... */ }
92    if (error != nil) { /* Error handling... */ }
93
94    // Run inference ...
95      </pre></p>
96    </section>
97    <section>
98      <h3>C (Until 2.3.0)</h3>
99      <p><pre class="prettyprint lang-c">
100    #include "tensorflow/lite/delegates/coreml/coreml_delegate.h"
101
102    // Initialize interpreter with model
103    TfLiteModel* model = TfLiteModelCreateFromFile(model_path);
104
105    // Initialize interpreter with Core ML delegate
106    TfLiteInterpreterOptions* options = TfLiteInterpreterOptionsCreate();
107    TfLiteDelegate* delegate = TfLiteCoreMlDelegateCreate(NULL);  // default config
108    TfLiteInterpreterOptionsAddDelegate(options, delegate);
109    TfLiteInterpreterOptionsDelete(options);
110
111    TfLiteInterpreter* interpreter = TfLiteInterpreterCreate(model, options);
112
113    TfLiteInterpreterAllocateTensors(interpreter);
114
115    // Run inference ...
116
117    /* ... */
118
119    // Dispose resources when it is no longer used.
120    // Add following code to the section where you dispose of the delegate
121    // (e.g. `dealloc` of class).
122
123    TfLiteInterpreterDelete(interpreter);
124    TfLiteCoreMlDelegateDelete(delegate);
125    TfLiteModelDelete(model);
126      </pre></p>
127    </section>
128  </devsite-selector>
129</div>
130
131## Best practices
132
133### Using Core ML delegate on devices without Neural Engine
134
135By default, Core ML delegate will only be created if the device has Neural
136Engine, and will return `null` if the delegate is not created. If you want to
137run Core ML delegate on other environments (for example, simulator), pass `.all`
138as an option while creating delegate in Swift. On C++ (and Objective-C), you can
139pass `TfLiteCoreMlDelegateAllDevices`. Following example shows how to do this:
140
141<div>
142  <devsite-selector>
143    <section>
144      <h3>Swift</h3>
145      <p><pre class="prettyprint lang-swift">
146    var options = CoreMLDelegate.Options()
147    options.enabledDevices = .all
148    let coreMLDelegate = CoreMLDelegate(options: options)!
149    let interpreter = try Interpreter(modelPath: modelPath,
150                                      delegates: [coreMLDelegate])
151      </pre></p>
152    </section>
153    <section>
154      <h3>Objective-C</h3>
155      <p><pre class="prettyprint lang-objc">
156    TFLCoreMLDelegateOptions* coreMLOptions = [[TFLCoreMLDelegateOptions alloc] init];
157    coreMLOptions.enabledDevices = TFLCoreMLDelegateEnabledDevicesAll;
158    TFLCoreMLDelegate* coreMLDelegate = [[TFLCoreMLDelegate alloc]
159                                          initWithOptions:coreMLOptions];
160
161    // Initialize interpreter with delegate
162      </pre></p>
163    </section>
164    <section>
165      <h3>C</h3>
166      <p><pre class="prettyprint lang-c">
167    TfLiteCoreMlDelegateOptions options;
168    options.enabled_devices = TfLiteCoreMlDelegateAllDevices;
169    TfLiteDelegate* delegate = TfLiteCoreMlDelegateCreate(&amp;options);
170    // Initialize interpreter with delegate
171      </pre></p>
172    </section>
173  </devsite-selector>
174</div>
175
176### Using Metal(GPU) delegate as a fallback.
177
178When the Core ML delegate is not created, alternatively you can still use
179[Metal delegate](https://www.tensorflow.org/lite/performance/gpu#ios) to get
180performance benefits. Following example shows how to do this:
181
182<div>
183  <devsite-selector>
184    <section>
185      <h3>Swift</h3>
186      <p><pre class="prettyprint lang-swift">
187    var delegate = CoreMLDelegate()
188    if delegate == nil {
189      delegate = MetalDelegate()  // Add Metal delegate options if necessary.
190    }
191
192    let interpreter = try Interpreter(modelPath: modelPath,
193                                      delegates: [delegate!])
194      </pre></p>
195    </section>
196    <section>
197      <h3>Objective-C</h3>
198      <p><pre class="prettyprint lang-objc">
199    TFLDelegate* delegate = [[TFLCoreMLDelegate alloc] init];
200    if (!delegate) {
201      // Add Metal delegate options if necessary
202      delegate = [[TFLMetalDelegate alloc] init];
203    }
204    // Initialize interpreter with delegate
205      </pre></p>
206    </section>
207    <section>
208      <h3>C</h3>
209      <p><pre class="prettyprint lang-c">
210    TfLiteCoreMlDelegateOptions options = {};
211    delegate = TfLiteCoreMlDelegateCreate(&amp;options);
212    if (delegate == NULL) {
213      // Add Metal delegate options if necessary
214      delegate = TFLGpuDelegateCreate(NULL);
215    }
216    // Initialize interpreter with delegate
217      </pre></p>
218    </section>
219  </devsite-selector>
220</div>
221
222The delegate creation logic reads device's machine id (e.g. iPhone11,1) to
223determine its Neural Engine availability. See the
224[code](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/delegates/coreml/coreml_delegate.mm)
225for more detail. Alternatively, you can implement your own set of denylist
226devices using other libraries such as
227[DeviceKit](https://github.com/devicekit/DeviceKit).
228
229### Using older Core ML version
230
231Although iOS 13 supports Core ML 3, the model might work better when it is
232converted with Core ML 2 model specification. The target conversion version is
233set to the latest version by default, but you can change this by setting
234`coreMLVersion` (in Swift, `coreml_version` in C API) in the delegate option to
235older version.
236
237## Supported ops
238
239Following ops are supported by the Core ML delegate.
240
241*   Add
242    *   Only certain shapes are broadcastable. In Core ML tensor layout,
243        following tensor shapes are broadcastable. `[B, C, H, W]`, `[B, C, 1,
244        1]`, `[B, 1, H, W]`, `[B, 1, 1, 1]`.
245*   AveragePool2D
246*   Concat
247    *   Concatenation should be done along the channel axis.
248*   Conv2D
249    *   Weights and bias should be constant.
250*   DepthwiseConv2D
251    *   Weights and bias should be constant.
252*   FullyConnected (aka Dense or InnerProduct)
253    *   Weights and bias (if present) should be constant.
254    *   Only supports single-batch case. Input dimensions should be 1, except
255        the last dimension.
256*   Hardswish
257*   Logistic (aka Sigmoid)
258*   MaxPool2D
259*   MirrorPad
260    *   Only 4D input with `REFLECT` mode is supported. Padding should be
261        constant, and is only allowed for H and W dimensions.
262*   Mul
263    *   Only certain shapes are broadcastable. In Core ML tensor layout,
264        following tensor shapes are broadcastable. `[B, C, H, W]`, `[B, C, 1,
265        1]`, `[B, 1, H, W]`, `[B, 1, 1, 1]`.
266*   Pad and PadV2
267    *   Only 4D input is supported. Padding should be constant, and is only
268        allowed for H and W dimensions.
269*   Relu
270*   ReluN1To1
271*   Relu6
272*   Reshape
273    *   Only supported when target Core ML version is 2, not supported when
274        targeting Core ML 3.
275*   ResizeBilinear
276*   SoftMax
277*   Tanh
278*   TransposeConv
279    *   Weights should be constant.
280
281## Feedback
282
283For issues, please create a
284[GitHub](https://github.com/tensorflow/tensorflow/issues/new?template=50-other-issues.md)
285issue with all the necessary details to reproduce.
286
287## FAQ
288
289* Does CoreML delegate support fallback to CPU if a graph contains unsupported
290  ops?
291  * Yes
292* Does CoreML delegate work on iOS Simulator?
293  * Yes. The library includes x86 and x86_64 targets so it can run on
294    a simulator, but you will not see performance boost over CPU.
295* Does TensorFlow Lite and CoreML delegate support MacOS?
296  * TensorFlow Lite is only tested on iOS but not MacOS.
297* Is custom TF Lite ops supported?
298  * No, CoreML delegate does not support custom ops and they will fallback to
299    CPU.
300
301## APIs
302
303*   [Core ML delegate Swift API](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/swift/Sources/CoreMLDelegate.swift)
304*   [Core ML delegate C API](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/delegates/coreml/coreml_delegate.h)
305    *   This can be used for Objective-C codes. ~~~
306