We’re excited to announce Hive’s new AutoML tool that provides customers with everything they need to train, evaluate, and deploy customized machine learning models.
Our pre-trained models solve a wide range of use cases, but we will always be bounded by the number of models we can build. Now customers who find that their unique needs and moderation guidelines don’t quite match with any of our existing solutions can create their own, custom-built for their platform and easily accessible via API.
AutoML can be used to augment our current offerings or to create new models entirely. Want to flag a particular subject that doesn’t exist as a head in our Text Moderation API, or a certain symbol or action that isn’t part of our Visual Moderation? With AutoML, you can quickly build solutions for these problems that are already integrated with your Hive workflow.
Let’s walk through our AutoML process to illustrate how it works. In this example, we’ll build a text classification model that can determine whether or not a given news headline is satirical.
First, we need to get our data in the proper format. For text classification models, all dataset files must be in CSV format. One column should contain the text data (titled text_data) and all other columns represent model heads (classification categories). The values within each row of any given column represent the classes (possible classifications) within that head. An example of this formatting for our satire model is shown below:
The first page you’ll see on Hive’s AutoML platform is a dashboard with all of your organization’s training projects. In the image below, you’ll see how the training and deployment status of old projects are displayed. To create our satire classifier, we’re going to make a new project by hitting the “Create New Project” button in the top right corner.
We’ll then be prompted to provide a name and description for the project as well as training data in the form of a CSV file. For test data, you can either upload a separate CSV file or choose to randomly split your training data into two files, one to be used for training and the other for testing. If you decide to split your data, you will be able to choose the percentage that you would like to split off.
After all of that is entered, we are ready to train! Beginning model training is as easy as hitting a single button. While your model trains, you can easily view its training status on the Training Projects page.
Once training is completed, your project page will show an analysis of the model’s performance. The boxes at the top allow you to decide if you want to look at this analysis for a particular class or overall. If you’re building a multi-headed model, you can choose which head you’d like to evaluate as well. We provide precision, recall, and balanced accuracy for all confidence thresholds as well as a PR curve. We also display a confusion matrix to show how many predictions were correct and incorrect per class.
Once you’re satisfied with your model’s performance, select the “Create Deployment” to launch the model. Similarly to model training, deployment will take a few moments. After model deployment is complete, you can view the deployment in your Hive customer dashboard, where you can access your API key, view current tasks, as well as access other information as you would with our pre-trained models.
We’re very excited to be adding AutoML to our offerings. The platform currently supports both text and image classification, and we’re working to add support for large language models next. If you’d like to learn more about our AutoML platform and other solutions we’re building, please feel free to reach out to sales@thehive.ai or contact us here.
Hive is excited to announce our new classifier to differentiate between AI-generated and human-written text. This model is hosted on our website as a free demo, and we encourage users to test out its performance.
The recent release of OpenAI’s ChatGPT model has raised questions about how public access to these kinds of large language models will impact the field of education. Certain school districts have already banned access to ChatGPT, and teachers have been adjusting their teaching methods to account for the fact that generative AI has made academic dishonesty a whole lot easier. Since the rise of internet plagiarism, plagiarism detectors have become commonplace at academic institutions. Now a need arises for a new kind of detection: AI-generated text.
Our AI-Generated Text Detector outperforms key competitors, including OpenAI itself. We compared our model to their detector, as well as two other popular AI-generated text detection tools: GPTZero and Writer’s AI Content Detector. Our model was the clear frontrunner, not just in terms of balanced accuracy but also in terms of false positive rate — a critical factor when these tools are deployed in an educational setting.
Our test dataset consisted of 242 text passages, including ChatGPT-generated text as well as human-written text. To ensure that our model behaves correctly on all genres of content, we included everything from casual writing to more technical and academic writing. We took special care to include texts written by those learning English as a second language, so as to be careful that their writing is not incorrectly categorized by our model due to differences in tone or wording. For these test examples, our balanced accuracy stands at an impressive 99% while the closest competitor is GPTZero with 83%. OpenAI got the lowest of the bunch, with only 73%.
Others have tried our model against OpenAI’s in particular, and they have echoed our findings. Following OpenAI’s classifier release, Mark Hachman at PCWorld published an article that suggested that those disappointed with OpenAI’s model should turn to Hive’s instead. In his own informal testing of our model, he praised our results for their accuracy as well as our inclusion of clear confidence scores for every result.
A large fear about using these sorts of detector tools in an educational setting is the potentially catastrophic impact of false positives, or cases in which human-written writing is classified as AI-generated. While building our model, we were mindful of the fact that the risk of such high-cost false positives is one that many educators may not want to take. In response, we prioritized lowering our false positive rate. On the test set above, our false positive rate is incredibly low, at 1%. This is compared to OpenAI’s at 12.5%, Writer’s at 46%, and GPTZeros at 30%.
Even with our low false positive rate, we do encourage that this tool be used as part of a broader process when investigating academic dishonesty and not as the sole decision maker. Just like plagiarism checkers, it is created to be a helpful screening tool and not a final judge. We are continuously working to improve our model, and any feedback is greatly appreciated. Large language models like ChatGPT are here to stay, and it is crucial to provide educators with tools they can use as they decide how to navigate these changes in their classrooms.
When generative AI models first gained popularity in the late 2010s, they brought with them the ability to create deepfakes. Deepfakes are synthetic media, typically video, in which one person’s likeness is replaced by another’s using deep learning. They are powerful tools for fraud and misinformation, allowing for the creation of synthetic videos of political leaders and letting scammers easily take on new identities.
The primary use, though, of deepfake technology is the fabrication of nonconsensual pornography. The term “deepfake” itself was coined in 2017 by a Reddit user of the same name who made fake pornographic videos featuring popular female celebrities. In 2019, the company Sensity AI catalogued deepfakes across the web and reported that a whopping 96% of them were pornographic, all of which were of women. In the years since, more of this sort of deepfake pornography has become readily available online, with countless forums and even entire porn sites dedicated to it. The targets of this are not just celebrities. They are also everyday women superimposed into adult content by request—on-demand revenge porn for anyone with an internet connection.
Many sites have banned deepfakes entirely, since they are far more often used for harm than for good. At Hive, we’re committed to providing API-accessible solutions for challenging moderation problems like this one. We’ve built our new Deepfake Detection API to empower enterprise customers to easily identify and moderate deepfake content hosted on their platforms.
This blog post explains how our model identifies deepfakes and the new API that makes this functionality accessible.
A Look Into Our Model
Hive’s Deepfake Detection model is essentially a version of our Demographic API that is optimized to identify deepfakes as opposed to demographic attributes. When a query is submitted, this visual detection model locates any faces present in the input. It then performs an additional classification step that determines whether or not each detected face is a deepfake. In its response, it provides a bounding-box location and classification (with confidence scores) for each face.
While the face detection aspect of this process is the same as the one used for our industry-leading Demographic API, the classification step was fine-tuned for deepfake identification by training on a vast repository of synthetic and real video data. Many of these examples were pulled from genres commonly associated with deepfakes, such as pornography, celebrity interviews, and movie clips. We also included other types of examples in order to create a classifier that identifies deepfakes across many different content genres.
Putting It All Together: Example Input and Response
With only one head, the response of our Deepfake Detection model is easily interpretable. When an image or video query is submitted, it is first split into frames. Each frame is then analyzed by our visual detection model in order to find any faces present in the image. Every face then receives a deepfake classification — either yes_deepfake or no_deepfake. Confidence scores for these classifications range from 0.0 to 1.0, with a higher score indicating higher confidence in the model’s results.
Example Deepfake Detection input and API response
Here we see the deepfaked image and, to its left, the two original images used to create it. This input image doesn’t appear to be fake at first glance, especially when the image is displayed at a small size. Even with a close examination, a human reviewer could fail to realize that it is actually a deepfake. As the example illustrates, the model correctly identifies this realistic deepfake with a high confidence score of more than 0.99. Since there is only one face present in this image, we see one corresponding “bounding poly” in the response. This “bounding poly” contains all model response information for that face. Vertices and dimensions are also provided, though those fields are truncated here for clarity.
Because deepfakes like this one can be very convincing, they are difficult to moderate with manual flagging alone. Automating this task is not only ideal to accelerate moderation processes, but also to spot realistic deepfakes that human reviewers might miss.
Digital platforms, particularly those that host NSFW media, can integrate this Deepfake Detection API into their workflows by automatically screening all content as it is posted. Video communication platforms and applications that use any kind of visual identity verification can also utilize our model to counter deepfake fraud.
Final Thoughts
Hive’s Deepfake Detection API joins our recently released AI-Generated Media Recognition API in the aim to expand content-moderation to keep up with the fast-growing domain of generative AI. Moving forward, we plan to continually update both models so as to best keep up with new generative techniques, popular content genres, and emerging customer needs.
The recent popularity of diffusion models like Stable Diffusion, Midjourney, and DALL-E 2 has brought deepfakes back into the spotlight and sparked conversation on whether these newer generative techniques can be used to develop brand-new ways of making them. Whether or not this happens, deepfakes aren’t going away any time soon and are only growing in number, popularity, and quality. Identifying and removing them across online platforms is crucial to limit the fraud, misinformation, and digital sexual abuse that they enable.
If you’d like to learn more about our Deepfake Detection API and other solutions we’re building, please feel free to reach out to sales@thehive.ai or contact us here.
In the past few months, AI-generated art has experienced rapid growth in both popularity and accessibility. Engines like DALL-E, Midjourney, and Stable Diffusion have spurred an influx of AI-generated artworks across online platforms, prompting an intense debate around their legality, artistic value, and potential for enabling the propagation of deepfake-like content. As a result, certain digital platforms such as Getty Images, InkBlot Art, Fur Affinity, and Newgrounds have announced bans on AI-generated content entirely, with more to likely follow in the coming weeks and months.
Platforms are enacting these bans for a variety of reasons. Online communities built for artists to share their artwork such as Newgrounds, Fur Affinity, and Purpleport stated they put their AI artwork ban in place in order to keep their sites focused exclusively on human-created art. Other platforms have taken action against AI-generated artwork due to copyright concerns. Image synthesis models often include copyrighted images in their training data, which consist of massive amounts of photos and artwork scraped from across the web, typically without any artists’ consent. It is an open question whether this type of scraping and the resulting AI-generated artwork amount to copyright violations — particularly in the case of commercial use — and platforms like Getty and InkBlot Art don’t want to take that risk.
As part of Hive’s commitment to providing enterprise customers with API-accessible solutions to moderation problems, we have created a classification model made specifically to assist digital platforms in enacting these bans. Our AI-Generated Media Recognition API is built with the same type of robust classification model as our industry-leading visual moderation products, and it enables enterprise customers to moderate AI-generated artwork without relying on users to flag images manually.
This post explains how our model works and the new API that makes this functionality accessible.
Using AI to Identify AI: Building Our Classifier
Hive’s AI-Generated Media Recognition model is optimized for use with the kind of media generated by popular AI generative engines such as DALL-E, Midjourney, and Stable Diffusion. It was trained on a large dataset comprising millions of artificially generated images and human-created images such as photographs, digital and traditional art, and memes sourced from across the web.
The resulting model is able to identify AI-created images among many different types and styles of artwork, even correctly identifying AI artwork that could be misidentified by manual flagging. Our model returns not only whether or not a given image is AI-generated, but also the likely source engine it was generated from. Each classification is accompanied by a confidence score that ranges from 0.0 to 1.0, allowing customers to set a confidence threshold to guide their moderation.
How it Works: An Example Input and Response
When receiving an input image, our AI-Generated Media Recognition model returns classifications under two separate heads. The first provides a binary classification as to whether or not the image is AI-generated. The second, which is only relevant when the image is classified as an AI-made image, identifies the source of that artificial image from among the most popular generation engines that are currently in use.
To get a sense of the capabilities of our AI-Generated Media Recognition model, here’s a look at an example classification:
This input image was created with the AI model Midjourney, though it is so realistic that it may be missed by manual flagging. As shown in the response above, our model correctly classifies this image as AI-generated with a high confidence score of 0.968. The model also correctly identifies the source of the image, with a similarly high confidence score. Other sources like DALL-E are also returned along with their respective confidence scores, and the scores under each of the two model heads sum to 1.
Platforms that host artwork of any kind can integrate this AI-Generated Media Recognition API into their workflows by automatically screening all content as it is being posted. This method of moderating AI artwork works far more quickly than manual flagging and can catch realistic artificial artworks that even human reviewers might miss.
Final Thoughts and Future Directions
Digital platforms are now being flooded with AI-generated content, and that influx will only increase as these generative models continue to grow and spread. On top of this, creating this kind of artwork is fast and easy to access online, which enables large quantities of it to be produced quickly. Moderating artificially created artworks is crucial for many sites to maintain their platform’s mission and protect themselves and their customers from potential legal issues further down the line.
We created our AI-Generated Media Recognition API to solve this problem, but our model will need to continue to evolve along with image generation models as existing ones improve and new ones are released. We plan on adding new generative engines to our sources as well as continually updating our model to keep up with the current capabilities of these models. Since some newer generative models can create video in addition to still images, we are working to add support for video formats within our API in order to best prevent all types of AI-generated artwork from dominating online communities where they are unwelcome.
If you’d like to learn more about this and other solutions we’re building, please feel free to reach out to sales@thehive.ai or contact us here.
Mensio aims to provide users with the most comprehensive and granular data available in the industry to inform better decisions on how to optimize the investment of marketing dollars. In recent months, brands, agencies, and rights holders alike have expressed interest in being able to a) measure the presence and value of verbal mentions within programming alongside visual exposures and b) within sports, understand the relative contributions of different sponsorship assets to total exposure and its associated value.
We are excited to announce a set of major product upgrades to incorporate these requests now live within Mensio’s Sponsorship & Branded Content modules. While we have supported these capabilities “off-platform” using Hive’s Logo Location and Brand Mentions models for multiple years, the inclusion of these capabilities in-platform provide reduced friction, faster access to data, and richer levels of brand- and property-level analysis as well as competitive intelligence.
Below is a brief summary of what’s new; as with all releases, notes will also appear as a pop-up in-platform upon your next log-in. Your Hive point of contact will additionally introduce the new capabilities live in your next scheduled meeting and, if not imminent, will be reaching out to schedule time for an overview of the new features at your earliest convenience.
New Features
Now Available: Reporting by Asset Type For Televised Sports Programming
Sponsorship & Branded Content modules now include reporting of exposures by asset type across most televised sports programming. Reporting includes 25+ standard asset types including jerseys, TVGI / digital overlays, lower level banners (i.e., outfield wall, dasherboards, courtside LED, etc.), basket stanchions, and more.
Within the platform, asset types are integrated as filters into existing reporting of visual exposures by brand, by program, and by occurrence. Additionally, two new pages have been added featuring asset-centric views of exposure by brand and by program.
Data is currently available for all relevant programming since June 1, 2022, and will be available going back to September 1, 2022 shortly. Notifications will appear within the platform as additional historical information becomes available.
Now Available: Reporting of Verbal Mentions Across Television Programming
Sponsorship & Branded Content television modules now include reporting of verbal mentions across all television programming. Verbal mentions are integrated into summary metrics in the Competitive Insights section, and have dedicated pages for deep dives by brand, by program, and by occurrence.
Data is currently available for all relevant programming since June 1, 2021, and will be available going back to October 1, 2018 shortly. Notifications will appear within the platform as additional historical information becomes available.
Now Available: Updated Module Definition and Navigation
To accommodate the expanded data, we have reconfigured module contents and navigation for Sponsorship & Branded Content television modules. Specifically:
“Television – By Brand” merges “National TV (Branded Content)” and “Regional Sports TV (Branded Content)” into a single module, where programming across network types can be viewed in a single chart (and can be separated using the Network Type filter if desired)
“Television – By Team” replaces “TV – Team Sponsorship”, maintaining the ability to additionally filter brand exposures by the associated sports team(s). The programming in this module includes all available NFL, NBA, MLB, and NHL live games and replays across national television and regional sports networks, as well as team-specific studio shows (e.g., Warriors Postgame)
“Television – Teams as Brands” replaces “National TV – Team Exposure”, maintaining the ability to view team-level exposures in sports talk and highlights
The sidebar design across the Television – By Brand and Television – By Team modules has been evolved to accommodate additional metrics and streamline access to individual charts and tables
We are excited by initial feedback to these module updates, and look forward to continuing to provide product innovation on a regular basis. Please reach out to your representative with any questions or needs as you experience the module upgrades. We look forward to your continued feedback and thank you for your trust in Mensio.
Earlier this year, Hive launched a pair of API products built on new deep learning models that analyze visual similarity between images: NFT Search and Custom Search. To help platforms authenticate visual content, these APIs search unstructured datasets for similar images to uncover relationships within a larger content ecosystem.
These launches marked the start of a larger effort by Hive to make broader, more relational content understanding available to enterprise customers. Now, we’re unveiling the most ambitious of our Intelligent Search services yet: Web Search, for visual comparisons to content on the open web.
Web Search deploys our best-in-class image similarity model across stored media from billions of crawled web pages, retrieves visually similar versions of a query image, and returns these matches via API. The Web Search API enables automated checks against web images in a variety of use-cases, including:
Detecting misuse of image assets in copyright enforcement contexts
Enforcing paywalls on premium content by identifying unauthorized reposts and shares of protected media
Verifying originality of user-generated content like profile and marketplace photos
In this announcement, we’ll take a closer look at the two pillars of Hive’s visual search engine – our image similarity model and web index – and preview how the Web Search API works.
Building a Visual Comparison Engine: Hive’s Similarity Model and Search Index
The backbone of Web Search is Hive’s visual similarity model, a deep vision model that conducts pair-wise visual comparisons between images. Unlike typical fingerprinting algorithms, our models assess visual similarity based on high-level feature alignment to mimic (and surpass) human perceptual comparison. To build this, we used contrastive learning on image sets including substantial augmentations and negative examples, training a robust visual similarity model without relying on supervisory labels to teach specific definitions.
The resulting model considers both duplicates of an image and modified versions as similar – including overlay elements, filters and edits, and adversarial modifications. For a pair of images, the model returns a normalized score between 0 and 1 correlated with the output of its contrastive loss function (i.e., based on similarity of feature vectors). A pair-wise similarity score of 1 indicates an exact visual match between images, while lower scores reflect the extent of any visual differences.
A robust image comparison model is a necessary part of a visual search engine, but not entirely sufficient. For Web Search to be broadly useful, we also needed a comprehensive reference database of images to compare against. To do this, Hive built and deployed a custom web crawler to continuously retrieve image content on public pages. Since we began crawling, we’ve grown this dataset to tens of billion images, which continues to expand as our crawler encounters new web pages and freshly posted content. To enable more detailed search results, we also index URL and domain information, alt text, and other image metadata that can be returned alongside matches in the API response.
Putting it Together: Web Search API and Visual Search Examples
Given a query image, the Web Search API uses the similarity model to compare against all reference images in our web index and returns all matches above a threshold similarity score. For each match, the API response specifies:
A direct link (URL) to the matching image
A backlink to the domain where the matching image was found
A similarity score between the query image and the match
Here are responses from a few example searches that show the versatility of the Web Search API:
Web Search is well-suited to help marketplaces automatically identify potential scam listings that use images taken from the open web. For example, we queried the left image from a suspiciously cheap rental ad that looked a little too good to be true. Web Search uncovered photos from a real listing for the unit on the realtor’s website. The two photos are almost identical except for slightly lower resolution in the scam image; our similarity model predicts accordingly with a similarity score of 0.99.
Let’s look at another example, this time with more visually impactful differences:
Here, the query image incorporates the original but uses a significant digital overlay. Still, our similarity model identifies the source image as a match with a similarity score of 0.7. The ability to recognize edited photos enables Web Search to help social and dating platforms identify impersonation attempts (“catfishing”) that use web photos on their profile, even if those photos have been noticeably modified.
Here’s a similar example where the query image is “clean” and the matching image is modified with a text overlay:
In this case, the matching image reuses the original with text stylized as a magazine cover, and our model correctly identifies the edited version. With similar queries, Web Search can help platforms track down misuses of stock photos and copyrighted images, or reposts of premium (paywall) content to other websites.
In their own searches, platforms can use our model’s similarity scores to target duplicates or close copies at high score thresholds and/or broaden searches to visually related images at lower scores to align with their definitions and intended use-cases.
Final Thoughts: Future Directions for Web Search
Hive’s Visual Search APIs offer enterprise customers new insight into how their visual content is used and where it comes from with on-demand searches on their own content (Custom Search), blockchains (NFT Search), and, now, the open web (Web Search). The capabilities of our image similarity model and other content tagging models raise the bar on what’s possible in the search space.
In building these datasets, we’re also thinking about ways to unlock other actionable insights within our search indexes. As a next step, we’ll be broadening our web index to include videos, GIFs, and audio data. From there, we plan to support using our targeted content tagging models – logo detectors, OCR, scene classification, and more – to enable open web searches across content modalities, content-targeted ad placements, and other use-cases in the future.
To learn more about Web Search or our other visual search APIs, you can contact us here or reach out to our sales team directly.
Hive’s visual classifier is a cornerstone of our content moderation suite. Our visual moderation API has consistently been the best solution on the market for moderating key types of image-based content, and some of the world’s largest content platforms continue to trust Hive’s Visual Moderation model for effective automated enforcement on NSFW images, violence, hate, and more.
As content moderation needs have evolved and grown, our visual classifier has also expanded to include 75 moderation classes across 31 different model heads. This is usually an iterative process – as our partners continue to send high volumes of content for analysis, we uncover ways to refine our classification schemes and identify new, useful types of content.
Recently, we’ve worked on broadening our visual model by defining new classes with input from our customers. And today, we’re shipping the general release of our latest visual moderation model, including three new classes to bolster our existing model capabilities:
Undressed to target racier suggestive images that may not be explicit enough to label NSFW
Gambling to capture betting in casinos or on games and sporting events
Confederate to capture imagery of the Confederate flag and graphics based on its design
All Hive customers can now upgrade to our new model to access predictions in these new classes at no additional cost. In this post, we’ll take a closer look at how these classes can be used and our process behind creating them.
New Visual Moderation Classes For Greater Content Understanding
Deep learning classifiers are most effective when given training data that illustrates a clear definition of what does and does not belong in the class. For this release, we used our distributed data labeling workforce – with over 5 million contributors – to efficiently source instructive labels on millions of training images relevant to our class definitions.
Below, we’ll take a closer look at some visual examples to illustrate our ground truth definitions for each new class.
Undressed
In previous versions, Hive’s visual classifier separated adult content into two umbrella classes: “NSFW,” which includes nudity and other explicit sexual classes, and “Suggestive,” which captures milder classes that might still be considered inappropriate.
Our “Suggestive” class is a bit broad by design, and some customers have expressed interest in a simple way to identify the racier cases without also flagging more benign images (e.g., swimwear in beach photos). So, for this release, we trained a new class to refine this distinction: undressed.
We wanted this class to capture images where a subject is clearly nude, even if their privates aren’t visible due to their pose, are temporarily covered by their hands or an object, or are occluded by digital overlays like emojis, scribbles, or shapes. To construct our training set, we added new annotations to existing training images for our NSFW and Suggestive classes and sourced additional targeted examples. Overall, this gave us a labeled set of 2.6M images to teach this ground truth to our new classifier.
Here’s a mild example to help illustrate the difference between our undressed and NSFW definitions (you can find a full definition for undressed and other relevant classes in our documentation):
Confidence scores for unedited version (left): undressed 1.00; general_nsfw 1.00; general_suggestive 0.00. Confidence scores for edited version (right): undressed 0.99; general_nsfw 0.35; general_suggestive 0.61
The first image showing explicit nudity is classified as both undressed and NSFW with maximum confidence. When we add a simple overlay over relevant parts of the image, however, the NSFW score drops far below threshold confidence while the undressed score remains very high.
Platforms can use undressed to flag both nudity and more obviously suggestive images in a single class. For content policies where milder images are allowed but undressed-type images are not, we expect this class to significantly reduce any need for human moderator review to enforce this distinction.
Gambling
Gambling was another type of content that frequently came up in customer feedback. This was a new undertaking for Hive, and building our ground truth and training set for this class was an interesting exercise in definitions and evaluating context in images.
Technically, gambling involves a wager staked on an uncertain outcome with the intent of winning a prize. For practical purposes, though, we decided to consider evidence of betting as the key factor. Certain behavior – like playing a slot machine or buying a lottery ticket – is always gambling since it requires a bet. But cards, dice, and competitive games don’t necessarily involve betting. We found the most accurate approach to be requiring visible money, chips or other tokens in these cases in order to flag an image as gambling. Similarly, we don’t consider photos at races or sporting events to be gambling unless receipts from a betting exchange or website are also shown.
To train our new class on this ground truth definition, we sourced and labeled a custom set of over 1.1M images. The new visual classifier can now distinguish between gambling activity and similar non-gambling behavior, even if the images are visually similar:
For more detailed information, you can see a full description of our gambling class here. Platforms that wish to moderate or identify gambling can access predictions from this model head by default after upgrading to this model release.
Confederate Symbolism
Separately, many of our customers also expressed interest in more complete monitoring of visual hate and white nationalism, especially Confederate symbolism. For this release, we sourced and labeled over 1M images to train a new class for identifying imagery of the commonly used version of the Confederate flag.
In addition to identifying photos of the flag itself, this new model head will also capture the Confederate “stars and bars” shown in graphics, tattoos, clothing, and the like. We also trained the model to ignore visually similar flags and historical variants that are not easily recognizable:
Along with our other hate classes, customers can now use predictions from our Confederate class to keep their online environments safe.
Improvements to Established Visual Moderation Classes
Beyond these new classes, we also focused on improving the model’s understanding around niche edge cases in existing model heads. For example, we leveraged active learning and additional training examples to address biases we occasionally found in our NSFW and Gun classifiers. This corrected some interesting biases where the model sometimes incorrectly identified studio microphones as guns, or mistook acne creams for other, less safe-for-work liquids.
Final Thoughts
This release delivers our most comprehensive and capable Visual Moderation model yet to help platforms develop proactive, cost-effective protection for their online communities. As moderation needs become more sophisticated, we’ll continue to incorporate feedback from our partners and refine our content moderation models to keep up. Stay tuned for our next release with additional classes and improvements later this year.
If you have any questions about this release, please get in touch at support@thehive.ai or api@thehive.ai. You can also find a video tutorial for upgrading to the latest model configuration here. For more information on Visual Moderation more generally, feel free to reach out to sales@thehive.ai or check out our documentation.
Auto manufacturers earned an estimated $1.1 billion in media value from visual logo exposures on national and regional TV over the past year, with Toyota capturing the highest share of voice across programming
TV exposures for auto brands are correlated with sports seasons, with official league sponsors dominating in playoff and championship months; additionally, in the fight for share of voice on TV, auto manufacturers strategically invest in specific categories, with a different brand leader across each sport
The return on investment of different placement types can vary; for example, looking at Major League Baseball in the month of April as a case study, exposures of Toyota and Ford outweighed those of league champion Chevrolet
Measuring amplification of exposures in shoulder programming and highlights – typically overlooked by sponsors today – doubles the number of unique programs with auto logos exposed and increases media value by 17%
As TV viewership continues to fragment across different platforms, the ability of sponsorships to ensure brand exposure within desired content has never been more important. However, sponsorship activations themselves are fragmented across sports and rights holders (e.g., teams, leagues, broadcast partners), resulting in demand for better data to measure the effectiveness of automakers’ own investments and to monitor a dynamic competitive environment.
The sponsorship landscape among automakers was analyzed using data from Mensio, Hive’s AI-powered media intelligence platform. Here’s what we learned.
1. In aggregate, auto manufacturers (OEMs) garnered an estimated $1.1 billion in media value from visual logo exposure on national and regional TV over the past year. Over 80% of this value was owned by the top 10 most exposed brands (out of a total compared dataset of 67 brands), speaking to market concentration on TV. Toyota led the pack by a large margin as the “Let’s go places” manufacturer indeed went everywhere on TV. With half the estimated media value of Toyota, Ford was the second-highest earning OEM, followed by Kia, Honda, and Chevrolet.
2. While sports sponsorships are typically rooted in the objective of aligning automakers’ brands with a given sport and/or team and its fans, sports sponsorship also dictate the time of year when different brands capture outsized share of voice. Each of the official league sponsors of the four largest US sports leagues experienced spikes in their share of voice of in-content brand exposures during playoff and championship periods: Kia (NBA) in April/May, Honda (NHL) in June/July, Chevrolet (MLB) in October, and Toyota (NFL) in February (its most exposed month, despite high visibility throughout the year). Additionally, Mercedes-Benz’ sponsorship of the U.S. Open rockets up its exposures in the month of September, with over $2M in estimated media value from the Men’s Championship match alone.
3. In the fight for share of voice within in-content brand exposure on television, brands placed differing bets across genres. Within sports, a different brand dominates each league, with official league partners leading the way. However, exposure wasn’t limited to league partners; Ford’s sponsorship of NFL pre-game programming is one example among many team- and broadcast-level activations where brands have competed for share of voice within a sport outside of official league sponsorships. Outside of sports, other genres of entertainment saw other investments by auto companies, such as Mercedes-Benz’ top feature in talk shows and awards/special programming.
4. Zooming in on the first month of the 2022 Major League Baseball season across national TV and regional sports networks presents an interesting early season case study. While Chevrolet – the official league sponsor – will likely increase exposures as the season continues, the brand started the year ranked #3 in share of voice for in-game brand exposures. Heavy team and broadcast sponsorship investments made by Toyota and Ford outweighed Chevy, illustrating alternate tactics to reach the same audience at different points during the season.
5. Given the massive investment and competition for the best placements, it is important for brands to fully measure their onscreen exposures. Currently, most brands are limited to “whistle to whistle” measurement focusing on in-game exposures, and sometimes the additional exposure from social media. The fragmentation of shoulder programming and highlights has traditionally been difficult to measure at scale; however, doing so provides a far more comprehensive understanding of performance from a given activation. Using always-on measurement from Mensio, which reports across every second of every program from 100+ national TV networks and regional sports networks, we estimate that amplification from shoulder programming and highlights almost doubles the number of unique programs with auto brand logos exposed, increasing duration of in-content brand exposures by 32% and the associated equivalent media value by 17%.
Credible competitive intelligence data is critical in making decisions on the best sponsorship placements. Mensio, Hive’s AI-powered media intelligence platform, provides always-on measurement of in-content brand exposure for more than 7,000 brands across 24/7 programming from 100+ national TV channels and regional sports networks.
Access to credible competitive intelligence data is critical for branded content and sponsorship decisions. Mensio allows brands to understand how their share of voice compares to that of competitors at the program-level and in aggregate. For more information on Mensio or to schedule a demo and learn how Mensio can support your brand, reach out to Hive at demo@thehive.ai.
Note: This analysis looked at in-program auto manufacturer logo exposures from May 2021 to April 2022 on national and regional TV (excluding commercials) and includes Tier 1, Tier 2, and Tier 3 placements.
Content Moderation Challenges in the Aftermath of Buffalo
The racially-motivated shooting in a Buffalo supermarket – live streamed by the perpetrator and shared across social media – is tragic on many levels. Above all else, lives were lost and families are forever broken as a result of this horrific attack. Making matters worse, copies of the violent recording are spreading on major social platforms, amplifying extremist messages and providing a blueprint for future attacks.
Unfortunately, this is not a new problem: extremist videos and other graphic content have been widely shared for shock value in the past, with little regard for the negative impacts. And bad actors are more sophisticated than ever, uploading altered or manipulated versions to thwart moderation systems.
As the world grapples with broader questions of racism and violence, we’ve been working with our partners behind the scenes to help control the spread of this and other harmful video content in their online communities. This post covers the concerns these partners have raised with legacy moderation approaches, and how newer technology can be more effective in keeping communities safe.
Conventional Moderation and Copy Detection Approaches
Historically, platforms relied on a combination of user reporting and human moderation to identify and react to harmful content. Once the flagged content reaches a human moderator, enforcement is usually quick and highly accurate.
But this approach does not scale for platforms with millions (or billions) of users. It can take hours to identify and act on an issue, especially in the aftermath of a major news event when post activity is highest. And it isn’t always the case that users will catch bad content quickly: when the Christchurch massacre was live streamed in 2019, it was not reported until 12 minutes after the stream ended, allowing the full video to spread widely across the web.
More recently, platforms have found success using cryptographic hashes of the original video to automatically compare against newly posted videos. These filters can quickly and proactively screen high volumes of content, but are generally limited to detecting copies of the same video. Hashing checks often miss content if there are changes to file formats, resolutions, and codecs. And even the most advanced “perceptual” hashing comparisons – which preprocess image data in order to consider more abstract features – can be defeated by adversarial augmentations.
Deep Learning To Advance Video Moderation and Contain Viral Content
Deep learning models can close the moderation capability gap for platforms in multiple ways.
First, visual classifier models can proactively monitor live or prerecorded video for indicators of violence. These model predictions enable platforms to shut down or remove content in real-time, preventing the publishing and distribution of videos that break policies in the first place. The visual classifiers can look for combinations of factors, such as someone holding a gun, bodily injury, blood, and other object or scene information to create automated and nuanced enforcement mechanisms. Specialized training techniques can also accurately teach visual classifiers to identify the difference between real violence and photorealistic violence depicted in video games, so that something like a first-person shooter game walkthrough is not mistaken for an real violent event.
In addition to screening using visual classifiers, platforms can harness new types of similarity models to stop reposts of videos confirmed to be harmful, even if those videos are adversarially altered or manipulated. If modified versions somehow bypass visual classification filters, these models can catch these videos based on visual similarity to the original version.
In these cases, self-supervised training techniques expose the models to a range of image augmentation and manipulation methods, enabling them to accurately assess human perceptual similarity between image-based content. These visual similarity models can detect duplicates and close copies of the original image or video, including more heavily modified versions that would otherwise go undetected by hashing comparisons. Unlike visual classifiers, these models do not look for specific visual subject matter in their analysis. Instead, they quantify visual similarity on a spectrum based on overlap between abstract structural features. This means there’s no need to produce training data to optimize the model for every possible scenario or type of harmful content; detecting copies and modified versions of known content simply requires that the model accurately assess whether images or video come from the same source.
How it works: Deep Learning Models in Automated Content Moderation Systems
Using predictions from these deep learning models as a real-time signal offers a powerful way to proactively screen video content at scale. These model results can inform automated enforcement decisions or triage potentially harmful videos for human review.
Advanced visual classification models can accurately distinguish between real and photorealistic animated weapons. Here are results from video frames containing both animated and real guns.
To flag real graphic violence, automated moderation logic could combine confidence scores in actively held weapons, blood, and/or corpse classes but exclude more benign images like these examples.
As a second line of defense, platforms need to be able to detect reposts or modified versions of known harmful videos from spreading. To do this, platforms use predictions from pre-trained visual similarity models in the same way they use hash comparisons today. With an original version stored as a reference, automated moderation systems can perform a frame-wise comparison with any newly posted videos, flagging or removing new content that scores above a certain similarity threshold.
In these examples, visual similarity models accurately predict that frame(s) in the query video are derived from the original reference, even under heavy augmentation. By screening new uploads against video content known to be graphic, violent, or otherwise harmful, these moderation systems can replace incomplete tools like hashing and audio comparison to more comprehensively solve the harmful content detection problem.
Final Thoughts: How Hive Can Help
No amount of technology can undo the harm caused by violent extremism in Buffalo or elsewhere. We can, however, use new technology to mitigate the immediate and future harms of allowing hate-based violence to be spread in our online communities.
Hive is proud to support the world’s largest and most diverse platforms in fulfilling their obligation to keep online communities safe, vibrant, and hopeful. We will continue to contribute towards state-of-the-art moderation solutions, and can answer questions or offer guidance to Trust & Safety teams who share our mission at support@thehive.ai.
Artists, technologists, and collectors have recently shown growing interest in non-fungible tokens (NFTs) as digital collectibles. With this surge in popularity, however, the red-hot NFT space has also become a prime target for plagiarism, copycats, and other types of fraud.
While built-in blockchain consensus mechanisms are highly effective at validating the creation, transaction, and ownership of NFTs, these “smart contracts” are typically not large enough to store the files they represent. Instead, the token simply links to a metadata file with a public link to the image asset. So while the token on the blockchain is itself unique, the underlying image may not be.
Additionally, current blockchain technology has no way of understanding image content or the relationships between images. Hashing checks and other conventional methods cannot address the subjective and more complicated problem of human perceptual similarity between images.
Due to these technical limitations, the same decentralization that empowers creators to sell their work independently also enables bad actors to create copycat tokens with unlicensed or modified image assets. At a minimum, this puts less sophisticated NFT buyers at risk as they may be unable to tell the difference between original and stolen arts; beyond this, widespread duplication also undermines the value proposition of original tokens as unique collectibles.
To help solve this problem, we are excited to offer NFT Search, a new API product built on a searchable index of major blockchain image assets and using Hive’s robust image similarity model.
NFT Search makes an otherwise opaque dataset easily accessible, allowing marketplaces and other stakeholders to search existing NFT image assets for matches to query images, accurately identifying duplicates and modified copies. NFT Search has the potential to provide much-needed confidence across the NFT ecosystem to help accelerate growth and stability in the market.
This post explains how our model works and the new API that makes this functionality accessible.
How Our Models Assess Similarity Between NFT Images
Hive’s NFT Search model is a deep vision image similarity model optimized for the types of digital art used in NFTs. To build this model, we used contrastive learning and other self-supervised techniques to expose a range of possible image augmentation methods. We then fine-tuned our notion of image similarity in order to account for a characteristic feature of NFTs: small, algorithmically-generated trait differences between images intended to be unique tokens.
The resulting model is targeted toward exact visual matches, but also resilient to manual manipulations and computer-generated variants that would bypass conventional hashing checks.
To quantify visual similarity between a query image and existing NFT image assets, the model returns similarity scores normalized between 0 and 1 for each identified match. For a matching NFT image, a similarity score of 1.0 indicates that the query image is an exact duplicate of the matching image. Lower scores indicate that the query image has been modified or is otherwise visually distinct in some way.
Building a Robust NFT Index for Broad Similarity Searches
Building a robust image comparison model was a necessary first step, but to make a NFT search solution useful we also needed to construct a near-complete set of existing NFT images as a reference set for broad comparisons. To do this, Hive crawls and indexes NFT images referenced on the Ethereum and Polygon blockchains in real-time, with support for additional blockchains in development. We also store identifying metadata from the associated tokens – including token IDs and URLs, contract addresses, and descriptors – to create a searchable “fingerprint” of each blockchain that enables comprehensive visual comparisons.
Putting it all together: Example NFT Searches and Model Predictions
At a high level: when receiving a query image, our NFT model compares the query image against each existing NFT image in this dataset. The NFT Search API then returns a list of any identified matches, including links to the matching images and token metadata.
To get a sense of NFT Search’s capabilities and how our scores align with human perceptual similarity, here’s a look at a few copycat tokens the model identified in recent searches:
This is an example of an exact duplicate (similarity score 1.00): a copy of one of the popular Bored Ape Yacht Club arts minted on the Polygon blockchain. Because NFT Search compares the query image to Hive’s entire NFT dataset, it is able to identify matching images across multiple blockchains and token standards.
Things get more interesting when we look for manually or programmatically manipulated variants at lower similarity scores. Take a look at the results from the search on another Bored Ape token, number 320:
This search returned many matches, including several exact matches on both the Ethereum and Polygon blockchains. Here’s a look at other, non-exact matches it found:
Variant 1: A basic variant where the original Bored Ape 320 image is mirrored horizontally. This simple manipulation has little impact on the model’s similarity prediction.
Variant 2 – “BAPP 320”: An example of a computer-manipulated copy on the Ethereum blockchain. The token metadata describes the augmented duplicate as an “AI-pixelated NFT” that is “inspired by the original BAYC collection.” Despite visual differences, the resulting image is structurally quite similar to the original, and our NFT model predicted accordingly (score = 0.94).
Variant 3 – “VAYC 5228”: A slight variant located on the Ethereum blockchain. The matching image has a combination of Bored Ape art traits that does not exist in the original collection, but since many traits match, the NFT model still returns a relatively high similarity score (0.85).
Variant 4 – These Apes Don’t Exist #274: Another computer-manipulated variant, but this one results in a new combination of Bored Ape traits and visible changes to the background. The token metadata, describes these as “AI-generated apes with hyper color blended visual traits imagined by a neural network.” Due to these clear visual and feature differences, this match yielded a lower similarity score (0.71)
NFT Search API: Response Object and Match Descriptions
Platforms integrate our NFT Search API response into their workflows to automatically submit queries when tokens are minted, listed for sale, or sold, and receive model prediction results in near-real time.
The NFT Search API will return a full JSON response listing any NFTs that match the query image. For each match, the response object includes:
A link (URL or IPFS address) to the matching NFT image
A similarity score
The token URL,
Any descriptive token metadata hosted at the token URL (e.g., traits and other descriptors), and
The unique contract address and token ID pair
To make the details of the API response more concrete, here’s the response object for the “BAPP 320” match shown above:
"matches": [
...
{
"url": "ipfs://QmY6RZ29zJ7Fzis6Mynr4Kyyw6JpvvAPRzoh3TxNxfangt/320.jpg",
"token_id": "320",
"contract_address": "0x1846e4EBc170BDe7A189d53606A72d4D004d614D",
"token_url": "ipfs://Qmc4onW4qT8zRaQzX8eun85seSD8ebTQjWzj4jASR1V9wN/320.json",
"image_hash": "ce237c121a4bd258fe106f8965f42b1028e951fbffc23bf599eef5d20719da6a",
"blockchain": "ethereum", //currently, this will be either "ethereum" or "Polygon"
"metadata":{
"name": "Pixel Ape #320",
"description": "**PUBLIC MINTING IS LIVE NOW: https://bapp.club.** *Become a BAPP member for only .09 ETH.* The BAPP is a set of 10,000 Bored Ape NFTs inspired by the original BAYC collection. Each colorful, AI-pixelated NFT is a one-of-a-kind collectible that lives on the Ethereum blockchain. Your Pixel Bored Ape also serves as your Club membership card, granting you access to exclusive benefits for Club members.",
"image": "ipfs://QmY6RZ29zJ7Fzis6Mynr4Kyyw6JpvvAPRzoh3TxNxfangt/320.jpg",
"attributes":[
{
//list of traits for NFT art if applicable
},
"similarity_score": 0.9463750907624477
},
...
]
Aside from identifying metadata, the response object also includes a SHA256 hash of the NFT image currently hosted at the image URL. The hash value (and/or a hash of the query image) can be used to confirm an exact match, or to verify that the NFT image hosted at the URL has not been modified or altered at a later time.
Final Thoughts
Authenticating NFTs is an important step forward in increasing trust between marketplaces, collectors, and creators who are driving the growth in this new digital ecosystem. We also recognize that identifying duplicates and altered copies within blockchains is just one part of a broader problem, and we’re currently hard at work on complementary authentication solutions that will expand our comparison scope from blockchains to the open web.
If you’d like to learn more about NFT Search and other solutions we’re building in this space, please feel free to reach out to sales@thehive.ai or contact us here.
