“Core ML is awesome! There’s so much ideas that I can do with it!” You might exclaim following WWDC 2017. “But I only have one mac. I can’t just install a beta OS on my main machine — It’ll probably brick and I’ll lose work”
Fortunately you don’t have to go to the High Sierras (pun intended) just to “have a go” on machine learning with Core ML. All thanks to the iOS Simulator.
Yes, iOS 11 Simulator has Core ML too. You don’t need to install a pre-production macOS nor install a buggy OS to your phone either. Just run the beta iOS Simulator with a production macOS alongside your regular Xcode and iOS Simulators.
What’s also great is that both iOS Simulator and Core ML works on Swift Playgrounds — making it easier for you for exploratory testing of ML models (and ensuring that it survived conversion). Well, sort-of for Core ML. You will need the “compiled” Core ML model and make some modifications on the corresponding generated stub sources. Hats off to Andrew C Bancroft who describes a solution to a similar issue with Core Data.
Here’s what you need to do to get your hands dirty on Core ML. You will need Xcode 9 Beta for this, but you can run it on current production macOS 10.12 — beta macOS optional.
- Get a Core ML Model.
- Compile the model using a dummy project. This will create a compiled
.mlmodelc
model and a corresponding Swift source file containing stub classes for the model. - Use Xcode 9 (beta) to create a Swift Playground with the platform set to iOS
- Copy the generated
.mlmodelc
folder into the Playground’sResources
folder. - Copy the generated Swift stub into the playground’s
Sources
model. - Modify the generated Swift stub file and make the classes therein
public
as well as any other properties that you might use in your main Playground file. - Start playing with Core ML!
Getting a Core ML model
The easiest way to obtain an ML model is to download a pre-made one from Apple’s machine learning overview page. If any model suffice — just for getting your hands dirty — taking one from their Core ML sample code would be better. Otherwise you’ll need to create one yourself and convert it to Core ML’s format using the provided tools.
Compiling the model.
The easiest way right now to compile a Core ML model and generate Swift files is via a dummy project. Create a blank Swift project and set the deployment target to iOS 11. Then copy the .mlmodel
file into the project and add it to the main target. Start a build for iOS Simulator and then you should get the model file compiled.
If you’d like to play around with the command line, there is the mlkitc
command line program inside Xcode’s application package under Contents/Developer/usr/bin
. However I was only able to generate Objective-C stub sources and couldn’t find how to make it generate Swift files. I found these by guessing and there’s no help text on how to use it as of Xcode 9 beta 3.
Use the word compile
as the first parameter of mlkitc
followed by the source .mlmodel
file and a target where to place the compiled models.
./mlkitc compile {source-model} {target-folder}
For example, the following command would compile VGG16.mlmodel
from the Downloads
folder and then create a folder VGG16.mlmodelc
containing the compiled model data under the Downloads/VGG
folder.
./mlkitc compile ~/Downloads/VGG16.mlmodel ~/Downloads/VGG
To create Objective-C stub classes for easy access to the model, use the word generate
as the first parameter to mlkitc
. This would create a pair of .h
and .m
files that contains three classes — one for the ML model itself, an input class for specifying input parameters and an output class that contains the model’s inference result.
./mlkitc generate {source-model} {target-folder}
For example, the following command would take the VGG16.mlmodel
model file and create the stub Objective-C sources in the Downloads/VGG
folder.
./mlkitc generate ~/Downloads/VGG16.mlmodel ~/Downloads/VGG
In case you’re curious, VGG16.mlmodel
is Visual Geometry Group’s convolutional model to identify the dominant object of an image. The Core ML model file was downloaded from Apple’s Machine Learning overview page
Locating the generated files
After building the target, the Core ML stub sources should be present inside your project’s derived data folder. Try looking for a folder named CoreMLGenerated
from the derived data folder and generated the stub files should be two level down from that, enclosed by a folder derived from the model file name.
Another way easiest way to find the stub sources is by referencing the model stub class from one of your own sources and let Xcode show you where it was defined.
For example, write code to instantiate the model stub from one of your view controllers like so:
let model = MarsHabitatPricer()
Build the target then command-click on the word MarsHabitatPricer
and select Jump to Definition
to find out where it came from. With that file open, ctrl-click on a black area in the source file and select Show in Finder
to open a Finder window that highlights its location. Then you can easily copy it out to the playground.
However the compiled Core ML model would be found inside the target application bundle itself, since it’s meant to be shipped as part of the application. The easiest way to find it is to ctrl-click your application target in Xcode and select Show in Finder
. Then in Finder, ctrl-click the application bundle and select Show Package Contents
. Locate the folder with the .mlmodelc
extension and copy it out along with all its contents.
Adding Core ML files into the Playground
Drag and drop the .swift
stubs into the top-level Sources
folder in the Playground and similarly the .mlmodelc
folder into the top-level Resources
folder. Initially these are grayed out in the Playground’s navigator tab but should become solid as soon as a file gets placed in there. Having the model placed into the top level folders would make it available to all playground pages.
Alternatively from the Finder, ctrl-click the .playground
bundle and select Show Package Contents
. Then create two folders there, one named Sources
and the other named Resources
. Copy .swift
files into Sources
and the .mlmodelc
folders into Resources
.
Modifying the stub
By default the Swift auto-generated stub sources has internal protection levels. This means that all classes and their members are accessible only from sources compiled for the same module. Unfortunately ancillary classes in Swift Playgrounds — those defined in source files that are not part of a page — needs to have public
accessibility to be usable from a page.
These stubs consists of three classes:
- A façade for the Core ML model object. This holds the
MLModel
object as well as prediction functions to make inferences from the model. - An input object for the ML model, subclass of
MLFeatureProvider
. This describes the input of the ML model in terms of feature names and their respective types. - An output object for the ML model’s results. This is also a subclass of
MLFeatureProvider
but instantiated by the ML model object’s façade class above as a result of prediction calls.
Just go in to the source file and change all three classes’ protection level to public
. Make sure that you also do the same for those class’ members as well.
Note that in the Swift generated classes, the individual features of the input object gets generated as var
with the internal protection level. Hence you probably only need to change init
to public unless you need to mutate the input after making predictions.
However in the output object, feature fields are generated as let
data members without any accessor. Thus you will need to make them public
— don’t worry about setters since they’re let
constants after all.
Sample Playground
I’ve constructed a bare-bones sample Swift playground that brought the pieces together. It brought over the ML model from Apple’s Mars Habitat Price Predictor into a playground environment. Download the playground from my Github account: MarsHabitatModelTest.
That’s all for now, take care!
THANK YOU SO MUCH
Thanks for creating this, it really helped me get started with CoreML in Swift Playgrounds! To generate Swift files use parameters
--language Swift --swift-version 4.0
. In Xcode 9.1 the tool for compiling mlmodel changed name tocoremlc
and can be found at the same path.