{"id":424,"date":"2022-05-04T06:01:00","date_gmt":"2022-05-04T06:01:00","guid":{"rendered":"https:\/\/thehive.ai\/blog\/?p=424"},"modified":"2025-03-04T09:50:33","modified_gmt":"2025-03-04T09:50:33","slug":"searching-custom-image-libraries-with-new-image-similarity-models","status":"publish","type":"post","link":"https:\/\/thehive.ai\/blog\/searching-custom-image-libraries-with-new-image-similarity-models","title":{"rendered":"Search Custom Image Libraries with New Image Similarity Models"},"content":{"rendered":"\n<h5 class=\"at-a-glance-heading\">Contents<\/h5>\n\n\n\n<ul class=\"at-a-glance\"><li><a href=\"#anchor1\">Building a Smarter Way to Search<\/a><\/li><li><a href=\"#anchor2\">Image Similarity Models: A Two-Pronged Approach<\/a><\/li><li><a href=\"#anchor3\">Why Use Similarity Models for Image Comparison?<\/a><\/li><li><a href=\"#anchor4\">Example Image Comparisons and Model Responses<\/a><\/li><li><a href=\"#anchor5\">Hive\u2019s Custom Search: API Overview<\/a><\/li><li><a href=\"#anchor6\">Final Thoughts<\/a><\/li><\/ul>\n\n\n\n<h2 id=\"anchor1\">Building a Smarter Way to Search<\/h2>\n\n\n\n<p>Hive has spent the last two years building powerful AI models served to customers via APIs. At their core, our current models \u2013 visual and text classification, logo detection, OCR, speech-to-text, and more \u2013 generate metadata that describes unstructured content. Hive customers use these \u201ccontent tagging\u201d models to unlock value across a variety of use-cases, from brand advertising analytics to automated content moderation.<\/p>\n\n\n\n<p>While these content tagging models are powerful, some content understanding challenges require a more holistic approach. Meeting these challenges requires an AI model that not only understands a piece of content, but also sees how that content relates to a larger set of data.&nbsp;&nbsp;<\/p>\n\n\n\n<p>Here\u2019s an example: a dating app is looking to moderate their user profile images. Hive\u2019s existing content tagging APIs can solve a number of challenges here, including identifying explicit content (visual moderation), verifying age (demographics), and detecting spam (OCR).&nbsp; But what if we also needed to detect whether or not a given photo matches (or is very similar to) another user\u2019s profile? That problem would fall outside the scope of the current content tagging models.&nbsp;<\/p>\n\n\n\n<p>To meet these broader content understanding challenges, we\u2019re excited to launch the first of our intelligent search solutions: Custom Search, an image comparison API built on Hive\u2019s visual similarity models. With the Custom Search APIs, platforms can maintain individualized, searchable databases of images and quickly submit query images for model-based comparisons across those sets.&nbsp;<\/p>\n\n\n\n<p>This customizability opens up a wide variety of use-cases:<\/p>\n\n\n\n<ul><li><strong>Detecting spam content:&nbsp;<\/strong>oftentimes, spammers on online platforms will use the same content or variants of the original content. By banning a single piece of content and using our custom search solution, platforms can now more extensively protect their users.<br><\/li><li><strong>Detecting marketplace scams:&nbsp;<\/strong>identify potentially fraudulent listings based on photos that match or are similar to other listings<br><\/li><li><strong>Detecting&nbsp;impersonation attempts:&nbsp;<\/strong>on social networks and dating apps, detect whether or not the same or similar profile images are being used across different accounts<\/li><\/ul>\n\n\n\n<p>This post will preview our visual similarity models and explore how to use Hive\u2019s Custom Search APIs.<\/p>\n\n\n\n<h2 id=\"anchor2\">Image Similarity Models: A Two-Pronged Approach<\/h2>\n\n\n\n<p>More than other classification problems, the question of \u201cimage similarity\u201d largely depends on definitions: at what point are two images considered similar or identical? To solve this, we used contrastive learning techniques to build two deep learning models with different but complementary ground-truth concepts of image similarity.&nbsp;<\/p>\n\n\n\n<p>The first model is optimized to identify exact visual matches between images \u2013 in other words: would a human decide that two images are identical upon close inspection? This \u201cexact match\u201d model is sensitive to even subtle augmentations or visual differences, where modifications can have a substantial impact on its similarity predictions.<\/p>\n\n\n\n<p>The second model is optimized towards identifying manipulated images, and is more specifically trained on (manual) modifications such as overlay text, cropping, rotations, filters, and juxtapositions. In other words, is the query image a manipulated copy of the original, or are they actually different images?<\/p>\n\n\n\n<h2 id=\"anchor3\">Why Use Similarity Models for Image Comparison?<\/h2>\n\n\n\n<p>Unlike traditional image duplicate detection approaches, Hive\u2019s deep learning approach to image comparison builds in resilience to image modification techniques, including both manual image manipulations via image editing software and adversarial augmentations (e.g., noise, filters, and other pixel-level alterations). By training on these augmentations specifically, our models can pick up modifications that would defeat conventional image hashing checks, even if those modifications don\u2019t result in visible changes to the image.<\/p>\n\n\n\n<p>Each model quantifies image similarity as a normalized score between 0 and 1. As you might expect, a pair-wise similarity score of 1.0 indicates an exact match between two images, while lower scores correspond to the extent of visual differences or modifications.&nbsp;&nbsp;<\/p>\n\n\n\n<h2 id=\"anchor4\">Example Image Comparisons&nbsp;and Model Responses<\/h2>\n\n\n\n<p>To illustrate the problem and give a sense of our models\u2019 understanding, here\u2019s how they classify some example image pairs:&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" width=\"1024\" height=\"341\" src=\"https:\/\/staticblog.thehive.ai\/uploads\/2024\/07\/1-5-1024x341.jpg\" alt=\"\" class=\"wp-image-448\" srcset=\"https:\/\/staticblog.thehive.ai\/uploads\/2024\/07\/1-5-1024x341.jpg 1024w, https:\/\/staticblog.thehive.ai\/uploads\/2024\/07\/1-5-300x100.jpg 300w, https:\/\/staticblog.thehive.ai\/uploads\/2024\/07\/1-5-768x256.jpg 768w, https:\/\/staticblog.thehive.ai\/uploads\/2024\/07\/1-5-1536x512.jpg 1536w, https:\/\/staticblog.thehive.ai\/uploads\/2024\/07\/1-5.jpg 1920w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p>This example is close to an exact match \u2013 each image is from the same video frame. Both models predict very high similarity scores (although not an exact visual match). However, the model predictions begin to diverge when we consider manipulated images:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" width=\"1024\" height=\"341\" src=\"https:\/\/staticblog.thehive.ai\/uploads\/2024\/07\/2-5-1024x341.jpg\" alt=\"Horizontal flip plus filter adjustments\" class=\"wp-image-449\" srcset=\"https:\/\/staticblog.thehive.ai\/uploads\/2024\/07\/2-5-1024x341.jpg 1024w, https:\/\/staticblog.thehive.ai\/uploads\/2024\/07\/2-5-300x100.jpg 300w, https:\/\/staticblog.thehive.ai\/uploads\/2024\/07\/2-5-768x256.jpg 768w, https:\/\/staticblog.thehive.ai\/uploads\/2024\/07\/2-5-1536x512.jpg 1536w, https:\/\/staticblog.thehive.ai\/uploads\/2024\/07\/2-5.jpg 1920w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><figcaption><em>Horizontal flip plus filter adjustments<\/em><\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" width=\"1024\" height=\"341\" src=\"https:\/\/staticblog.thehive.ai\/uploads\/2024\/07\/3-4-1024x341.jpg\" alt=\"Recoloration plus multiple mask overlay\" class=\"wp-image-450\" srcset=\"https:\/\/staticblog.thehive.ai\/uploads\/2024\/07\/3-4-1024x341.jpg 1024w, https:\/\/staticblog.thehive.ai\/uploads\/2024\/07\/3-4-300x100.jpg 300w, https:\/\/staticblog.thehive.ai\/uploads\/2024\/07\/3-4-768x256.jpg 768w, https:\/\/staticblog.thehive.ai\/uploads\/2024\/07\/3-4-1536x512.jpg 1536w, https:\/\/staticblog.thehive.ai\/uploads\/2024\/07\/3-4.jpg 1920w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><figcaption><em>Recoloration plus multiple mask overlay<\/em><\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" width=\"1024\" height=\"427\" src=\"https:\/\/staticblog.thehive.ai\/uploads\/2024\/07\/4-3-1024x427.jpg\" alt=\"Layered overlay text\" class=\"wp-image-451\" srcset=\"https:\/\/staticblog.thehive.ai\/uploads\/2024\/07\/4-3-1024x427.jpg 1024w, https:\/\/staticblog.thehive.ai\/uploads\/2024\/07\/4-3-300x125.jpg 300w, https:\/\/staticblog.thehive.ai\/uploads\/2024\/07\/4-3-768x320.jpg 768w, https:\/\/staticblog.thehive.ai\/uploads\/2024\/07\/4-3-1536x640.jpg 1536w, https:\/\/staticblog.thehive.ai\/uploads\/2024\/07\/4-3.jpg 1920w\" sizes=\"(max-width: 1024px) 100vw, 1024px\" \/><figcaption><em>Layered overlay text<\/em><\/figcaption><\/figure>\n\n\n\n<p>In these examples, the exact match model shows significantly more sensitivity to visual differences, while the broader visual similarity model (correctly) predicts that one image is a manipulated copy of the other. In this way, scores from these models can be used in distinct but complementary ways to identify matching images in your image library.&nbsp;<\/p>\n\n\n\n<h2 id=\"anchor5\">Hive\u2019s Custom Search: API Overview<\/h2>\n\n\n\n<p>Custom Search includes three API endpoints: two for adding and removing images from individualized image libraries, and a third to submit query images for model-based comparison.&nbsp;<\/p>\n\n\n\n<p>For comparison tasks, the query endpoint allows images to be submitted for comparison to the library associated with your project. When a query image is submitted, our models will compare the image to each reference image in your custom index to identify visual matches.&nbsp;<\/p>\n\n\n\n<p>The Custom Search API will return a similarity score from both the exact visual match model and the visual similarity model on \u2013 like those shown in the above examples \u2013 for any matching images. Each platform can therefore adapt which of these scores to use (and at what threshold) based on their desired use-case.&nbsp;<\/p>\n\n\n\n<h2 id=\"anchor6\">Final Thoughts<\/h2>\n\n\n\n<p>We\u2019re excited about the ways that our new Custom Search APIs will enable customers to unlock useful insights in their search applications. For Hive, this represents the start of a new generation of enterprise AI that just scratches the surface of what is possible in this space.<\/p>\n\n\n\n<p>If you\u2019d like to learn more about Custom Search APIs or get help designing a solution tailored to your needs, you can reach out to our sales team&nbsp;<a href=\"https:\/\/thehive.ai\/contact-us?source=blog\" target=\"_blank\" rel=\"noreferrer noopener\">here<\/a>&nbsp;or by email at&nbsp;<a href=\"mailto:sales@thehive.ai\">sales@thehive.ai<\/a>.&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Hive unveils the first of its visual search products built on new image similarity models: Custom Search.<\/p>\n","protected":false},"author":1,"featured_media":441,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"kia_subtitle":""},"categories":[8,3],"tags":[],"_links":{"self":[{"href":"https:\/\/thehive.ai\/blog\/wp-json\/wp\/v2\/posts\/424"}],"collection":[{"href":"https:\/\/thehive.ai\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/thehive.ai\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/thehive.ai\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/thehive.ai\/blog\/wp-json\/wp\/v2\/comments?post=424"}],"version-history":[{"count":3,"href":"https:\/\/thehive.ai\/blog\/wp-json\/wp\/v2\/posts\/424\/revisions"}],"predecessor-version":[{"id":452,"href":"https:\/\/thehive.ai\/blog\/wp-json\/wp\/v2\/posts\/424\/revisions\/452"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/thehive.ai\/blog\/wp-json\/wp\/v2\/media\/441"}],"wp:attachment":[{"href":"https:\/\/thehive.ai\/blog\/wp-json\/wp\/v2\/media?parent=424"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/thehive.ai\/blog\/wp-json\/wp\/v2\/categories?post=424"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/thehive.ai\/blog\/wp-json\/wp\/v2\/tags?post=424"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}