Category: AI-Generated Content Detection

Contents

• Navigating an Increasingly Generative World
• Structuring the Study
• Evaluation Methods and Findings
• Final Thoughts Moving Forward

Navigating an Increasingly Generative World

To the untrained eye, distinguishing human-created art from AI-generated content can be difficult. Hive’s commitment to providing customers with API-accessible solutions for challenging problems led to the creation of our AI-Generated Image and Video Detection API, which classifies images as human-created or AI-generated. Our model was evaluated in an independent study conducted by Anna Yoo Jeong Ha and Josephine Passananti from the University of Chicago, which sought to determine who was more effective at classifying images as AI-generated: humans or automated detectors.

Ha and Passananti’s study addresses a growing problem within the generative AI space: As generative AI models become more advanced, the boundary between human-created art and AI-generated images has become increasingly indistinguishable. With such powerful tools being accessible to the general public, various legal and ethical concerns have been raised regarding the misuse of said technology.

Such concerns are pertinent to address because the misuse of generative AI models negatively impacts both society at large and the AI models themselves. Bad actors have used AI-generated images for harmful purposes, such as spreading misinformation, committing fraud, or scamming individuals and organizations. As only human-created art is eligible for copyright, businesses may attempt to bypass the law by passing off AI-generated images as human-created. Moreover, multiple studies (on both generative image and text models) have shown evidence that AI models will deteriorate if their training data solely consists of AI-generated content—which is where Hive’s classifier comes in handy.

The study’s results show that Hive’s model outperforms both its automated peers and highly-trained human experts in differentiating between human-created art versus AI-generated images across most scenarios. This post examines the study’s methodologies and findings, in addition to highlighting our model’s consistent performance across various inputs.

Structuring the Study

In the experiment, researchers evaluated the performance of five automated detectors (three of which are commercially available, including Hive’s model) and humans against a dataset containing both human-created and AI-generated images across various art styles. Humans were categorized into three subgroups: non-artists, professional artists, and expert artists. Expert artists are the only subgroup with prior experience in identifying AI-generated images.

The dataset consists of four different image groups: human-created art, AI-generated images, “hybrid images” which combine generative AI and human effort, and perturbed versions of human-created art. A perturbation is defined as a minor change to the model input aimed at detecting vulnerabilities in the model’s structure. Four perturbation methods are used in the study: JPEG compression, Gaussian noise, CLIP-based Adversarial Perturbation (which performs perturbations at the pixel level), and Glaze (a tool used to protect human artists from mimicry by introducing imperceptible perturbations on the artwork).

After evaluating the model on unperturbed imagery, the researchers proceeded to more advanced scenarios with perturbed imagery.

Evaluation Methods and Findings

The researchers evaluated the automated detectors on four metrics: overall accuracy (ratio of training data classified correctly to the entire dataset), false positive rate (ratio of human-created art misclassified as AI-generated), false negative rate (ratio of AI-generated images misclassified as human-created), and AI detection success rate (ratio of AI-generated images correctly classified as AI-generated to the total amount of AI-generated images).

Among automated detectors, Hive’s model emerged as the “clear winner” (Ha and Passananti 2024, 6). Not only does it boast a near-perfect 98.03% accuracy rate, but it also has a 0% false positive rate (i.e., it never misclassifies human art) and a low 3.17% false negative rate (i.e., it rarely misclassifies AI-generated images). According to the authors, this could be attributed to Hive’s rich collection of generative AI datasets, with high quantities of diverse training data compared to its competitors.

Additionally, Hive’s model proved to be resistant against most perturbation methods, but faced some challenges classifying AI-generated images processed with Glaze. However, it should be noted that Glaze’s primary purpose is as a protection tool for human artwork. Glazing AI-generated images is a non-traditional use case with minimal training data available as a result. Thus, Hive’s model’s performance with Glazed AI-generated images has little bearing on its overall quality.

Final Thoughts Moving Forward

When it comes to automated detectors and humans alike, Hive’s model is unparalleled. Even compared to human expert artists, Hive’s model classifies images with higher levels of confidence and accuracy.

While the study considers the model’s potential areas for improvement, it is important to note that the study was published in February 2024. In the months following the study’s publication, Hive’s model has vastly improved and continues to expand its capabilities, with 12+ model architectures added since.

If you’d like to learn more about Hive’s AI-Generated Image and Video Detection API, a demo of the service can be accessed here, with additional documentation provided here. However, don’t just trust us, test us: reach out to sales@thehive.ai or contact us here, and our team can share API keys and credentials for your new endpoints.

Hive is excited to announce our new classifier to differentiate between AI-generated and human-written text. This model is hosted on our website as a free demo, and we encourage users to test out its performance.

The recent release of OpenAI’s ChatGPT model has raised questions about how public access to these kinds of large language models will impact the field of education. Certain school districts have already banned access to ChatGPT, and teachers have been adjusting their teaching methods to account for the fact that generative AI has made academic dishonesty a whole lot easier. Since the rise of internet plagiarism, plagiarism detectors have become commonplace at academic institutions. Now a need arises for a new kind of detection: AI-generated text.

Our AI-Generated Text Detector outperforms key competitors, including OpenAI itself. We compared our model to their detector, as well as two other popular AI-generated text detection tools: GPTZero and Writer’s AI Content Detector. Our model was the clear frontrunner, not just in terms of balanced accuracy but also in terms of false positive rate — a critical factor when these tools are deployed in an educational setting.

Our test dataset consisted of 242 text passages, including ChatGPT-generated text as well as human-written text. To ensure that our model behaves correctly on all genres of content, we included everything from casual writing to more technical and academic writing. We took special care to include texts written by those learning English as a second language, so as to be careful that their writing is not incorrectly categorized by our model due to differences in tone or wording. For these test examples, our balanced accuracy stands at an impressive 99% while the closest competitor is GPTZero with 83%. OpenAI got the lowest of the bunch, with only 73%.

Others have tried our model against OpenAI’s in particular, and they have echoed our findings. Following OpenAI’s classifier release, Mark Hachman at PCWorld published an article that suggested that those disappointed with OpenAI’s model should turn to Hive’s instead. In his own informal testing of our model, he praised our results for their accuracy as well as our inclusion of clear confidence scores for every result.

A large fear about using these sorts of detector tools in an educational setting is the potentially catastrophic impact of false positives, or cases in which human-written writing is classified as AI-generated. While building our model, we were mindful of the fact that the risk of such high-cost false positives is one that many educators may not want to take. In response, we prioritized lowering our false positive rate. On the test set above, our false positive rate is incredibly low, at 1%. This is compared to OpenAI’s at 12.5%, Writer’s at 46%, and GPTZeros at 30%.

Even with our low false positive rate, we do encourage that this tool be used as part of a broader process when investigating academic dishonesty and not as the sole decision maker. Just like plagiarism checkers, it is created to be a helpful screening tool and not a final judge. We are continuously working to improve our model, and any feedback is greatly appreciated. Large language models like ChatGPT are here to stay, and it is crucial to provide educators with tools they can use as they decide how to navigate these changes in their classrooms.