Category: Intelligent Search

Contents

• What is Web Search?
• Building a Visual Comparison Engine: Hive’s Similarity Model and Search Index
• How it Works: Web Search API and Visual Search Examples
• Final Thoughts: Future Directions for Web Search

What is Web Search?

Earlier this year, Hive launched a pair of API products built on new deep learning models that analyze visual similarity between images: NFT Search and Custom Search. To help platforms authenticate visual content, these APIs search unstructured datasets for similar images to uncover relationships within a larger content ecosystem.

These launches marked the start of a larger effort by Hive to make broader, more relational content understanding available to enterprise customers. Now, we’re unveiling the most ambitious of our Intelligent Search services yet: Web Search, for visual comparisons to content on the open web. 

Web Search deploys our best-in-class image similarity model across stored media from billions of crawled web pages, retrieves visually similar versions of a query image, and returns these matches via API.  The Web Search API enables automated checks against web images in a variety of use-cases, including: 

• Detecting misuse of image assets in copyright enforcement contexts
• Enforcing paywalls on premium content by identifying unauthorized reposts and shares of protected media
• Verifying originality of user-generated content like profile and marketplace photos

In this announcement, we’ll take a closer look at the two pillars of Hive’s visual search engine – our image similarity model and web index – and preview how the Web Search API works. 

Building a Visual Comparison Engine: Hive’s Similarity Model and Search Index

The backbone of Web Search is Hive’s visual similarity model, a deep vision model that conducts pair-wise visual comparisons between images.  Unlike typical fingerprinting algorithms, our models assess visual similarity based on high-level feature alignment to mimic (and surpass) human perceptual comparison.  To build this, we used contrastive learning on image sets including substantial augmentations and negative examples, training a robust visual similarity model without relying on supervisory labels to teach specific definitions.

The resulting model considers both duplicates of an image and modified versions as similar – including overlay elements, filters and edits, and adversarial modifications. For a pair of images, the model returns a normalized score between 0 and 1 correlated with the output of its contrastive loss function (i.e., based on similarity of feature vectors). A pair-wise similarity score of 1 indicates an exact visual match between images, while lower scores reflect the extent of any visual differences. 

A robust image comparison model is a necessary part of a visual search engine, but not entirely sufficient. For Web Search to be broadly useful, we also needed a comprehensive reference database of images to compare against. To do this, Hive built and deployed a custom web crawler to continuously retrieve image content on public pages. Since we began crawling, we’ve grown this dataset to tens of billion images, which continues to expand as our crawler encounters new web pages and freshly posted content. To enable more detailed search results, we also index URL and domain information, alt text, and other image metadata that can be returned alongside matches in the API response.

Putting it Together: Web Search API and Visual Search Examples

Given a query image, the Web Search API uses the similarity model to compare against all reference images in our web index and returns all matches above a threshold similarity score.  For each match, the API response specifies: 

• A direct link (URL) to the matching image
• A backlink to the domain where the matching image was found
• A similarity score between the query image and the match

Here are responses from a few example searches that show the versatility of the Web Search API:

Web Search is well-suited to help marketplaces automatically identify potential scam listings that use images taken from the open web. For example, we queried the left image from a suspiciously cheap rental ad that looked a little too good to be true. Web Search uncovered photos from a real listing for the unit on the realtor’s website. The two photos are almost identical except for slightly lower resolution in the scam image; our similarity model predicts accordingly with a similarity score of 0.99. 

Let’s look at another example, this time with more visually impactful differences: 

Here, the query image incorporates the original but uses a significant digital overlay. Still, our similarity model identifies the source image as a match with a similarity score of 0.7. The ability to recognize edited photos enables Web Search to help social and dating platforms identify impersonation attempts (“catfishing”) that use web photos on their profile, even if those photos have been noticeably modified. 

Here’s a similar example where the query image is “clean” and the matching image is modified with a text overlay:

In this case, the matching image reuses the original with text stylized as a magazine cover, and our model correctly identifies the edited version. With similar queries, Web Search can help platforms track down misuses of stock photos and copyrighted images, or reposts of premium (paywall) content to other websites. 

In their own searches, platforms can use our model’s similarity scores to target duplicates or close copies at high score thresholds and/or broaden searches to visually related images at lower scores to align with their definitions and intended use-cases.

Final Thoughts: Future Directions for Web Search

Hive’s Visual Search APIs offer enterprise customers new insight into how their visual content is used and where it comes from with on-demand searches on their own content (Custom Search), blockchains (NFT Search), and, now, the open web (Web Search). The capabilities of our image similarity model and other content tagging models raise the bar on what’s possible in the search space.

In building these datasets, we’re also thinking about ways to unlock other actionable insights within our search indexes. As a next step, we’ll be broadening our web index to include videos, GIFs, and audio data. From there, we plan to support using our targeted content tagging models – logo detectors, OCR, scene classification, and more – to enable open web searches across content modalities, content-targeted ad placements, and other use-cases in the future. 

To learn more about Web Search or our other visual search APIs, you can contact us here or reach out to our sales team directly. 

Contents

• Why We Built the NFT Search API
• How our Models Assess Similarity Between NFT Images
• Building a Robust NFT Index for Broad Similarity Searches
• Putting it all together: Example Searches and Model Predictions
• NFT Search API: Response Object and Match Descriptions
• Final Thoughts and Future Directions

Why We Built the NFT Search API

Artists, technologists, and collectors have recently shown growing interest in non-fungible tokens (NFTs) as digital collectibles. With this surge in popularity, however, the red-hot NFT space has also become a prime target for plagiarism, copycats, and other types of fraud. 

While built-in blockchain consensus mechanisms are highly effective at validating the creation, transaction, and ownership of NFTs, these “smart contracts” are typically not large enough to store the files they represent. Instead, the token simply links to a metadata file with a public link to the image asset. So while the token on the blockchain is itself unique, the underlying image may not be.

Additionally, current blockchain technology has no way of understanding image content or the relationships between images. Hashing checks and other conventional methods cannot address the subjective and more complicated problem of human perceptual similarity between images.

Due to these technical limitations, the same decentralization that empowers creators to sell their work independently also enables bad actors to create copycat tokens with unlicensed or modified image assets. At a minimum, this puts less sophisticated NFT buyers at risk as they may be unable to tell the difference between original and stolen arts; beyond this, widespread duplication also undermines the value proposition of original tokens as unique collectibles. 

To help solve this problem, we are excited to offer NFT Search, a new API product built on a searchable index of major blockchain image assets and using Hive’s robust image similarity model.  

NFT Search makes an otherwise opaque dataset easily accessible, allowing marketplaces and other stakeholders to search existing NFT image assets for matches to query images, accurately identifying duplicates and modified copies. NFT Search has the potential to provide much-needed confidence across the NFT ecosystem to help accelerate growth and stability in the market.  

This post explains how our model works and the new API that makes this functionality accessible.

How Our Models Assess Similarity Between NFT Images

Hive’s NFT Search model is a deep vision image similarity model optimized for the types of digital art used in NFTs. To build this model, we used contrastive learning and other self-supervised techniques to expose a range of possible image augmentation methods. We then fine-tuned our notion of image similarity in order to account for a characteristic feature of NFTs: small, algorithmically-generated trait differences between images intended to be unique tokens.

The resulting model is targeted toward exact visual matches, but also resilient to manual manipulations and computer-generated variants that would bypass conventional hashing checks. 

To quantify visual similarity between a query image and existing NFT image assets, the model returns similarity scores normalized between 0 and 1 for each identified match. For a matching NFT image, a similarity score of 1.0 indicates that the query image is an exact duplicate of the matching image. Lower scores indicate that the query image has been modified or is otherwise visually distinct in some way. 

Building a Robust NFT Index for Broad Similarity Searches

Building a robust image comparison model was a necessary first step, but to make a NFT search solution useful we also needed to construct a near-complete set of existing NFT images as a reference set for broad comparisons. To do this, Hive crawls and indexes NFT images referenced on the Ethereum and Polygon blockchains in real-time, with support for additional blockchains in development. We also store identifying metadata from the associated tokens – including token IDs and URLs, contract addresses, and descriptors – to create a searchable “fingerprint” of each blockchain that enables comprehensive visual comparisons. 

Putting it all together: Example NFT Searches and Model Predictions

At a high level: when receiving a query image, our NFT model compares the query image against each existing NFT image in this dataset. The NFT Search API then returns a list of any identified matches, including links to the matching images and token metadata. 

To get a sense of NFT Search’s capabilities and how our scores align with human perceptual similarity, here’s a look at a few copycat tokens the model identified in recent searches: 

This is an example of an exact duplicate (similarity score 1.00): a copy of one of the popular Bored Ape Yacht Club arts minted on the Polygon blockchain. Because NFT Search compares the query image to Hive’s entire NFT dataset, it is able to identify matching images across multiple blockchains and token standards. 

Things get more interesting when we look for manually or programmatically manipulated variants at lower similarity scores. Take a look at the results from the search on another Bored Ape token, number 320: 

This search returned many matches, including several exact matches on both the Ethereum and Polygon blockchains. Here’s a look at other, non-exact matches it found:

• Variant 1: A basic variant where the original Bored Ape 320 image is mirrored horizontally. This simple manipulation has little impact on the model’s similarity prediction. 
  
• Variant 2 – “BAPP 320”: An example of a computer-manipulated copy on the Ethereum blockchain. The token metadata describes the augmented duplicate as an “AI-pixelated NFT” that is “inspired by the original BAYC collection.” Despite visual differences, the resulting image is structurally quite similar to the original, and our NFT model predicted accordingly (score = 0.94). 
  
• Variant 3 – “VAYC 5228”: A slight variant located on the Ethereum blockchain. The matching image has a combination of Bored Ape art traits that does not exist in the original collection, but since many traits match, the NFT model still returns a relatively high similarity score (0.85). 
  
• Variant 4 – These Apes Don’t Exist #274: Another computer-manipulated variant, but this one results in a new combination of Bored Ape traits and visible changes to the background. The token metadata, describes these as “AI-generated apes with hyper color blended visual traits imagined by a neural network.” Due to these clear visual and feature differences, this match yielded a lower similarity score (0.71)

NFT Search API: Response Object and Match Descriptions

Platforms integrate our NFT Search API response into their workflows to automatically submit queries when tokens are minted, listed for sale, or sold, and receive model prediction results in near-real time. 

The NFT Search API will return a full JSON response listing any NFTs that match the query image. For each match, the response object includes:

• A link (URL or IPFS address) to the matching NFT image
  
• A similarity score 
  
• The token URL,
  
• Any descriptive token metadata hosted at the token URL (e.g., traits and other descriptors), and
  
• The unique contract address and token ID pair

To make the details of the API response more concrete, here’s the response object for the “BAPP 320” match shown above: 

"matches": [
    ...    
    {
        "url": "ipfs://QmY6RZ29zJ7Fzis6Mynr4Kyyw6JpvvAPRzoh3TxNxfangt/320.jpg",
        "token_id": "320",
        "contract_address": "0x1846e4EBc170BDe7A189d53606A72d4D004d614D",
        "token_url": "ipfs://Qmc4onW4qT8zRaQzX8eun85seSD8ebTQjWzj4jASR1V9wN/320.json",
        "image_hash": "ce237c121a4bd258fe106f8965f42b1028e951fbffc23bf599eef5d20719da6a",
        "blockchain": "ethereum", //currently, this will be either "ethereum" or "Polygon"
        "metadata":{
             "name": "Pixel Ape #320",
             "description": "**PUBLIC MINTING IS LIVE NOW: https://bapp.club.** *Become a BAPP member for only .09 ETH.* The BAPP is a set of 10,000 Bored Ape NFTs inspired by the original BAYC collection. Each colorful, AI-pixelated NFT is a one-of-a-kind collectible that lives on the Ethereum blockchain. Your Pixel Bored Ape also serves as your Club membership card, granting you access to exclusive benefits for Club members.",
             "image": "ipfs://QmY6RZ29zJ7Fzis6Mynr4Kyyw6JpvvAPRzoh3TxNxfangt/320.jpg",
             "attributes":[
             {
             //list of traits for NFT art if applicable 
             },
"similarity_score": 0.9463750907624477
    },
    ...
]

Aside from identifying metadata, the response object also includes a SHA256 hash of the NFT image currently hosted at the image URL. The hash value (and/or a hash of the query image) can be used to confirm an exact match, or to verify that the NFT image hosted at the URL has not been modified or altered at a later time. 

Final Thoughts

Authenticating NFTs is an important step forward in increasing trust between marketplaces, collectors, and creators who are driving the growth in this new digital ecosystem. We also recognize that identifying duplicates and altered copies within blockchains is just one part of a broader problem, and we’re currently hard at work on complementary authentication solutions that will expand our comparison scope from blockchains to the open web.

If you’d like to learn more about NFT Search and other solutions we’re building in this space, please feel free to reach out to sales@thehive.ai or contact us here.

Contents

• Building a Smarter Way to Search
• Image Similarity Models: A Two-Pronged Approach
• Why Use Similarity Models for Image Comparison?
• Example Image Comparisons and Model Responses
• Hive’s Custom Search: API Overview
• Final Thoughts

Building a Smarter Way to Search

Hive has spent the last two years building powerful AI models served to customers via APIs. At their core, our current models – visual and text classification, logo detection, OCR, speech-to-text, and more – generate metadata that describes unstructured content. Hive customers use these “content tagging” models to unlock value across a variety of use-cases, from brand advertising analytics to automated content moderation.

While these content tagging models are powerful, some content understanding challenges require a more holistic approach. Meeting these challenges requires an AI model that not only understands a piece of content, but also sees how that content relates to a larger set of data.  

Here’s an example: a dating app is looking to moderate their user profile images. Hive’s existing content tagging APIs can solve a number of challenges here, including identifying explicit content (visual moderation), verifying age (demographics), and detecting spam (OCR).  But what if we also needed to detect whether or not a given photo matches (or is very similar to) another user’s profile? That problem would fall outside the scope of the current content tagging models. 

To meet these broader content understanding challenges, we’re excited to launch the first of our intelligent search solutions: Custom Search, an image comparison API built on Hive’s visual similarity models. With the Custom Search APIs, platforms can maintain individualized, searchable databases of images and quickly submit query images for model-based comparisons across those sets. 

This customizability opens up a wide variety of use-cases:

• Detecting spam content: oftentimes, spammers on online platforms will use the same content or variants of the original content. By banning a single piece of content and using our custom search solution, platforms can now more extensively protect their users.
  
• Detecting marketplace scams: identify potentially fraudulent listings based on photos that match or are similar to other listings
  
• Detecting impersonation attempts: on social networks and dating apps, detect whether or not the same or similar profile images are being used across different accounts

This post will preview our visual similarity models and explore how to use Hive’s Custom Search APIs.

Image Similarity Models: A Two-Pronged Approach

More than other classification problems, the question of “image similarity” largely depends on definitions: at what point are two images considered similar or identical? To solve this, we used contrastive learning techniques to build two deep learning models with different but complementary ground-truth concepts of image similarity. 

The first model is optimized to identify exact visual matches between images – in other words: would a human decide that two images are identical upon close inspection? This “exact match” model is sensitive to even subtle augmentations or visual differences, where modifications can have a substantial impact on its similarity predictions.

The second model is optimized towards identifying manipulated images, and is more specifically trained on (manual) modifications such as overlay text, cropping, rotations, filters, and juxtapositions. In other words, is the query image a manipulated copy of the original, or are they actually different images?

Why Use Similarity Models for Image Comparison?

Unlike traditional image duplicate detection approaches, Hive’s deep learning approach to image comparison builds in resilience to image modification techniques, including both manual image manipulations via image editing software and adversarial augmentations (e.g., noise, filters, and other pixel-level alterations). By training on these augmentations specifically, our models can pick up modifications that would defeat conventional image hashing checks, even if those modifications don’t result in visible changes to the image.

Each model quantifies image similarity as a normalized score between 0 and 1. As you might expect, a pair-wise similarity score of 1.0 indicates an exact match between two images, while lower scores correspond to the extent of visual differences or modifications.  

Example Image Comparisons and Model Responses

To illustrate the problem and give a sense of our models’ understanding, here’s how they classify some example image pairs: 

This example is close to an exact match – each image is from the same video frame. Both models predict very high similarity scores (although not an exact visual match). However, the model predictions begin to diverge when we consider manipulated images:

In these examples, the exact match model shows significantly more sensitivity to visual differences, while the broader visual similarity model (correctly) predicts that one image is a manipulated copy of the other. In this way, scores from these models can be used in distinct but complementary ways to identify matching images in your image library. 

Hive’s Custom Search: API Overview

Custom Search includes three API endpoints: two for adding and removing images from individualized image libraries, and a third to submit query images for model-based comparison. 

For comparison tasks, the query endpoint allows images to be submitted for comparison to the library associated with your project. When a query image is submitted, our models will compare the image to each reference image in your custom index to identify visual matches. 

The Custom Search API will return a similarity score from both the exact visual match model and the visual similarity model on – like those shown in the above examples – for any matching images. Each platform can therefore adapt which of these scores to use (and at what threshold) based on their desired use-case. 

Final Thoughts

We’re excited about the ways that our new Custom Search APIs will enable customers to unlock useful insights in their search applications. For Hive, this represents the start of a new generation of enterprise AI that just scratches the surface of what is possible in this space.

If you’d like to learn more about Custom Search APIs or get help designing a solution tailored to your needs, you can reach out to our sales team here or by email at sales@thehive.ai.