Getting Started

Computers can see now, which means that mobile phones can too. The Clarifai Mobile SDK makes it easy to use image and video recognition on device, in real-time and without internet connectivity. This page will provide guidance on creating concepts and models that can be used both locally (on your mobile device), and on the cloud.

If you haven't created a Clarifai account and received your free API key, please do so before proceeding with this guide. After receiving the key, you can begin making API calls for free without inputting your credit card. You will need to create a new Application on your Clarifai account in order to get a Client Id and Client Secret to authenticate usage of the SDK. For more information on creating an application, check our tutorial.

Install iOS

Software Requirements

Before you begin, you will need the most recent version of XCode installed on your computer.

To request access to the iOS Mobile SDK, send an email to mobile-feedback@clarifai.com. You will receive a reply back containing the Mobile SDK framework, and a few example projects to get you started.

Example Projects

To run the example projects:

  1. Unzip the ClarifaiSDK folder you received in your email;
  2. Double click to open CameraExample.xcodeproj or CustomTrainingExample.xcodeproj from their respective project folders;
  3. Navigate to the ViewController.m files and input your AppID and AppSecret (same as Client Id and Client Secret). This can be obtained here.

Custom Projects

To include the Mobile SDK in your own projects:

  1. Open your app’s XCode project or workspace;
  2. Go to the app target’s General configuration page;
  3. Add the framework target to the Embedded Binaries section by clicking the ‘+’ icon. Do not drag in the framework from Finder;
  4. Select the ClarifaiMobileSDK.framework file from the list of binaries that can be embedded.
  5. In the build phases menu for your project, click on the little plus just above "target dependencies" to add a new build phase. On the menu that pops up, select "New Run Script Phase." In the window below "/bin/sh", add the following snippet:
APP_PATH="${TARGET_BUILD_DIR}/${WRAPPER_NAME}"
  
  # This script removed unused architectures from the Clarifai Mobile SDK framework.
  
  find "$APP_PATH" -name 'ClarifaiMobileSDK.framework' -type d | while read -r FRAMEWORK
  do
  FRAMEWORK_EXECUTABLE_NAME=$(defaults read "$FRAMEWORK/Info.plist" CFBundleExecutable)
  FRAMEWORK_EXECUTABLE_PATH="$FRAMEWORK/$FRAMEWORK_EXECUTABLE_NAME"
  echo "Executable is $FRAMEWORK_EXECUTABLE_PATH"
  
  EXTRACTED_ARCHS=()
  
  for ARCH in $ARCHS
  do
  echo "Extracting $ARCH from $FRAMEWORK_EXECUTABLE_NAME"
  lipo -extract "$ARCH" "$FRAMEWORK_EXECUTABLE_PATH" -o "$FRAMEWORK_EXECUTABLE_PATH-$ARCH"
  EXTRACTED_ARCHS+=("$FRAMEWORK_EXECUTABLE_PATH-$ARCH")
  done
  
  echo "Merging extracted architectures: ${ARCHS}"
  lipo -o "$FRAMEWORK_EXECUTABLE_PATH-merged" -create "${EXTRACTED_ARCHS[@]}"
  rm "${EXTRACTED_ARCHS[@]}"
  
  echo "Replacing original executable with thinned version"
  rm "$FRAMEWORK_EXECUTABLE_PATH"
  mv "$FRAMEWORK_EXECUTABLE_PATH-merged" "$FRAMEWORK_EXECUTABLE_PATH"
  
  done

If the framework was successfully added, you will now be able to import the SDK into your project code by adding: #import <ClarifaiMobileSDK/ClarifaiApp.h>.

If you are new to iOS development or want more information about adding frameworks in XCode, check out this tutorial from Apple.

Reference Documentation

You can also view Reference Documentation for the Clarifai Mobile iOS SDK.

Inputs

The SDK is built around a simple idea. You send inputs (images) to the service and it returns predictions (concepts). Further, you can retrieve all inputs (images) by providing a concept.

Add Inputs

You can add inputs by first creating ClarifaiInput objects. These objects will contain the Image, an ID, and an array of Concepts (optional) to tag your input for training. You can add up to 128 inputs at a time. The inputs should be tagged with concepts, as it will later be used to train the model.

The inputs then need to be added to the application that you are building, which will be added to the Clarifai app on cloud once internet connection is available.


  // Create inputs.
  UIImage *basketballImg = [UIImage imageNamed:@"basketball.jpg"];
  ClarifaiInput *input1 = [[ClarifaiInput alloc] initWithImage:basketballImg];
  UIImage *baseballImg = [UIImage imageNamed:@"baseball.jpg"];
  ClarifaiInput *input2 = [[ClarifaiInput alloc] initWithImage:baseballImg];
  
  // Add concepts to each image so they can be used during training.
  // A concept's score has a value of 1 if the input is a positive
  // example of the concept, and 0 if it is a negative example.
  ClarifaiConcept *basketballConcept = [[ClarifaiConcept alloc] initWithConceptName:@"basketball" score:1];
  ClarifaiConcept *baseballConcept = [[ClarifaiConcept alloc] initWithConceptName:@"baseball" score:1];
  input1.concepts = @[basketballConcept];
  input2.concepts = @[baseballConcept];
  
  // Finally, add the inputs to your application. When there is an available internet connection, these inputs will also be added to your remote Clarifai App. 
  [_app addInputs:@[input1, input2] completion:^(NSError *error) {
    if (error) {
      NSLog(@"Inputs not added: %@", error);
    } else {
      NSLog(@"Inputs added successfully");
    }
  }];

Get Inputs

You can retrieve all inputs (images) you have previously added by making a call.


  [_app getInputs:^(NSArray<ClarifaiInput *> *inputs, NSError *error) {
    for (ClarifaiInput *input in inputs) {
      NSLog(@"%@", input);
    }
  }];

Get Inputs for Concept

If you included a list of concepts when adding your inputs and would like to retrieve using that concept, you can do that as well.


  // If you have already added some inputs to your app with attached
  // concepts, you can retrieve all inputs associated with a given concept.
  
  // First, create or get a ClarifaiConcept.
  ClarifaiConcept *concept = [[ClarifaiConcept alloc] initWithConceptName:@"basketball" score:1];
  
  // Then retrieve the inputs in your app associated with this concept.
  [_app getInputsForConcept:concept completion:^(NSArray<ClarifaiInput *> *inputs, NSError *error) {
    for (ClarifaiInput *input in inputs) {
      NSLog(@"%@", input);
    }
  }];

Models

You can create a custom model and train it with your own images and concepts. Once you train the model and add it to your application, you can use it to make predictions on any input.

When you create a model you give it a Name and an ID. All models must have unique IDs, but can be named whatever you want.

Create Model

First, ensure that images are tagged with concepts. Then, create a new model in your application; your model should contain a list of concepts along with an id and name. The list of concepts defines the tags that the model will output for predictions.


  // Create custom concept.
  ClarifaiConcept *concept = [[ClarifaiConcept alloc] initWithConceptName:@"basketball" score:1];
  
  // Create a model with the given concept, locally.
  [_app createModel:@[concept] name:@"basketball" completion:^(ClarifaiModel *model, NSError *error) {
    NSLog(@"Created model: %@", model.name);
  }];

Get Models

The Mobile SDK allows you to retrieve models both locally (from device) and from the cloud. Each returned model will have a 'cloud’ property: ‘YES’ indicates that the model was retrieved from the cloud, and ‘NO’ indicates that the model was retrieved locally.

// [app getModels:] will retrieve models both locally and from the cloud. The model will return a 'cloud' property: 'YES' indicates that the model was retrieved from the cloud, and 'NO' indicates that the model was retrieved locally.
  
  __block ClarifaiModel *modelToTrain;
  [_app getModels:^(NSArray<ClarifaiModel *> *models, NSError *error) {
    if (!error) {  
      modelToTrain = models[0];
    }
  }];

Train Model

Once you’ve created a model with your own images and concepts, the model is ready to be trained. Models created locally will train directly on the device without requiring internet connection. Models that are downloaded from the cloud, from the Clarifai API, will also need to be trained on the cloud - it will require internet connection.


  // Train model. Remote models downloaded from the Clarifai API will not be trained locally, but in the cloud.
  [modelToTrain train:^(ClarifaiModel *trainedModel, NSError *error) {
    if (error) {
      NSLog(@"Error training model: %@", error);
    } else {
      NSLog(@"Model successfully trained!");
    }
  }];

Model Predict

After the model has been trained, it can now be used to make predictions. Give the model one or more new ClarifaiInputs with any images and it will return with predicted concepts for each input. This all happens locally on the device, so no internet connection is required.


  UIImage *basketballImg = [UIImage imageNamed:@"basketball.jpg"];
  ClarifaiInput *input = [[ClarifaiInput alloc] initWithImage:basketballImg];
  
  [trainedModel predictOnInputs:@[input] completion:^(NSArray<ClarifaiOutput *> *outputs, NSError *error) {
    for (ClarifaiOutput *output in outputs) {
      // Print predicted concepts with cooresponding probabilities.
      for (int i = 0; i < [output.tags count]; i++) {
        NSLog(@"%@: %@", output.tags[i], output.probs[i]);
      }
    }
  }];

Add Concepts to Model

After a model has been created, you can always add more concepts at a later point. However, every time you add new concepts, the model must be re-trained to account for the newly added concepts.


  // Create custom concept.
  ClarifaiConcept *baseballConcept = [[ClarifaiConcept alloc] initWithConceptName:@"baseball" score:1];
  
  // Add baseballConcept to model 'modelToTrain'.
  [_app addConcepts:@[baseballConcept] toModel:modelToTrain completion:^(NSError *error) {
    if (!error) {
      NSLog(@"Concept added successfully!");
    } else {
      NSLog(@"Error adding concept to model: %@", error);
    }
  }];
dy>