Clarifai's original public prediction models were based on the simple paradigm of an image or video with concept predictions in a structured JSON response, batched together if necessary.
The Search API aims to extend that functionality by enabling developers to index images and video (coming soon) within a Clarifai application. Not only will users be able to receive predictions for that given content, but a variety of searches can now be run across that library of visual content. You can essentially find visually similar images, all instances of a General Model concept, or even images that appear similar to this image crop.
On an application level, the Search API can be utilized in a variety of powerful ways. These implementations can range from purely internal workflows and subtle background applications to more explicit user focused search and discovery experiences.
- Consumer visual search
- Finding products in an inventory similar to a user's Instagram + Pinterest preferences
- An explicit input into larger recommendation systems
- "You may also like" product carousels
- Recommended items within drip campaign emails
- Flower app identification
Stock photography and content management
- Display visually similar images as the user browses within their library
- Search by image functionality
- De-duplication workflow
Design and content marketing
- Explore large and unstructured internal image libraries
At the moment you can only search across what you've indexed on our Cloud and Clarifai users can index their visual content within a Clarifai Application. Applications can be created within your developer account at http://clarifai.com/developer.
Inputs are images you would like store, train on, and/or search across. If you have an e-commerce catalog containing 10,000 product images that you would like to use visual search with, your Clarifai application would have to contain 10,000 inputs. One input = one image. The addition of an image to an application can be as simple and straightforward as an image url. Details
As you add images to your Application, each image will be predicted on with the General Model. Each image is stored with the top 20 concept predictions from that model. Concepts represent visually similar features (objects, context, patterns, style), and this enables you to you 'Search By Concept' and instantly find all instances of 'dog', 'ocean', 'wildlife', 'abstract', 'portrait', etc across your indexed images. You can perform concept searches using any one of the 11,000 General Model concepts. Details
If you'd like Images can be indexed with just an image url. Alternatively, you can index an image with associated Custom Metadata via the API. Metadata can be any arbitrary JSON, and can include items such as "Brand", "Size" or "Description". With this metadata added, you can perform filtered and more complex searches across your content. Details
Searching for all images with a brand value of 'Adidas' in the UI is as easy as searching with 'metadata:brand:Adidas' in the search bar.
You can view the associated metadata for an image by clicking on it and surfacing the menu in the top right.
Search by Image
Given an external image that isn't in your collection, you can find visually similar images within the application. In its simplest form it can be described as "here's an image, find all that is like this."
By default the API will return the top 20 most similar images each with a respective similarity score. Near duplicates and duplicates will typically result in scores of .99998+
In the UI, the results will be ordered by descending order of visual similarity score.
Search by Image + Metadata
This operation will conduct a search for visually similar lamps to the given image but only amongst inputs that contain the given metadata field of "Modern". Constraining this search can provide you with more specific results within certain product categories.
Search by Image + Concept
Similar to conducting a search with metadata, this search for a visually similar image across your application will be focused within a smaller pool of candidates. Searching by image with a concept will first find all images in your app that the General Model has identified as the respective concept and then find the most visually similar images/products within that subset. This is often helpful if the given image is a visually distinct pattern but not necessarily an image of a whole product.
Cropped search in both the UI and API allows you to pass in a crop of your original query image. Results will reflect that new crop's pixels and visually distinct features.
Sign up and log-in at https://clarifai.com/developer. By default you will have an application automatically created for you: 'My First Application'. You can re-name, delete, and/or create another application.
Uploading, training, and searching are done on https://clarifai.com/explorer, where you will be able to explore each application and its respective inputs.
Upload files from your computer by dragging and dropping into your application.
Copy and paste a list of urls into the upload modal. Click the up arrow in the top right to access the upload modal.
Copy & paste
The UI supports copy and pasting images directly into your application. It also supports pasting in screenshots from your clipboard.
Watch the video below to learn and see possible search workflows
Any questions or suggestions? Please email