The new approach aims to keep children safe from illegal content generated by AI | MIT News

With the growing popularity of generative artificial intelligence, many open source models are now available online for anyone to practice doing their work, such as generating a product offering in a specific art style.

But these models also find their way into the hands of malicious actors who may be able to generate illegal content, such as hate speech or child sexual abuse (CSAM). This is a growing problem – The National Center for Missing and Exploited Children received more than 1.5 million AI-generated CSAM reports in 2025, an increase from 67,000 in 2024.

Developers often test AI with dangerous capabilities by asking the model and checking its results, but this is not possible in CSAM, because it is illegal in the US to produce such content, regardless of the purpose.

To avoid this problem and improve the safety of AI, a team of MIT scientists, led by graduate student Vinith Suriyakumar and associate professors Ashia Wilson and Marzyeh Ghassemi, collaborated with researchers from Thorn to develop a new test method that determines whether a model can produce CSAM, without encouraging it. Thorn is a non-profit child safety organization whose mission is to transform the way children are protected from sexual abuse and exploitation in the digital age.

Their technique examines how the inner workings of the model have been changed, but produces no output. By examining the hidden representations, it can determine whether the model is specialized to generate dangerous images.

When tested, the research process found model variations that were specific to produce CSAM with 100 percent accuracy. The hosting platform can use this process to flag unsafe models and remove them immediately or prevent them from being loaded in the first place.

“This opens up a new way for open-source modeling platforms and rule-making to test whether a model can generate CSAM. Before, we didn’t have a way to measure this. It was a blind spot that other people were taking advantage of. Now, we can address an AI security problem that has a very serious impact,” said Vinith Suriyakumar, lead author of the electrical science paper and an MIT graduate student on the process.

Suriyakamur and Wilson, the Lister Borthers Career Development Professor at EECS and principal investigator at the Laboratory for Information and Decision Systems (LIDS), were joined on the paper by Lena Stempfle, an MIT postdoc; Ghassemi, associate professor at EECS and member of the Institute of Medical Engineering Sciences (IMES) and LIDS; and others from Boston University and Thorn. This paper was presented as a keynote at the workshop “Trusted AI for Good” at the International Conference on Machine Learning.

Auditing adaptations

Recent techniques have made it easier for users to customize a generative AI model for their work through a process known as fine-tuning.

Rather than retraining every model on a task-specific dataset, people can use an algorithm called low-rank adaptation (LoRA) to specialize the model more efficiently.

This has led to the emergence of new types of AI modeling for various purposes, such as producing watercolor images that imitate artistic movement. But it also enabled malicious actors to create models that could generate high-quality CSAM and other malicious images.

To test the model, developers usually tell it harmful content and check its results, but this manual test process is not scalable. In addition, repeatedly producing negative images can have a negative psychological effect on human examinees.

This testing method quickly falls apart when testing CSAM, which is illegal to produce for any purpose in the US and many other international jurisdictions.

“We are in this very difficult situation where, based on the law itself, we cannot use the methods of evaluation. We had to throw away the whole set of tools and take a different approach,” said Suriyakumar.

After learning about this mystery, the researchers met with Thorn, to solve the problem.

A non-productive solution

Instead of focusing on results, the researchers focused on modifying the LoRA algorithm during fine-tuning.

Their technique investigates these changes, called LoRA adapters, to determine whether the model is specialized for dangerous abilities, without generating output.

Using a technique called Gaussian probing, the researchers feed the model a set of random data points and analyze how it manipulates that data within its multi-layered internal structure.

“We never run the model to the end or tell the model, so we never generate images,” explains Suriyakumar.

The researchers capture those changes over time within the model’s internal structure and measure them to summarize how the LoRA adapter has changed the model’s calculations. They found that these responses were a strong indicator of how special the model was.

They tested their approach on three different types of models, comparing the results to ground truth data from LoRA adapters known to produce CSAM, other malicious images, and secure content.

Their method was 100 percent accurate in identifying models that were adapted to produce CSAM.

“There is a huge bucket of child safety concerns with AI, and these are real concerns that need to be addressed. Many children are harmed by AI deepfakes. We have shown that Gaussian testing can be a very useful tool, and we hope that the research community really pours more attention to this problem,” said Wilson.

Importantly, their methodology is scalable and will be cost-effective to implement. With thousands of different models being published online every month, benchmarking is key to helping auditors weed out dangerous practices before they become widespread.

Gaussian detection is also more robust than other detection techniques, as a malicious actor would need to carefully change the inner workings of the underlying model to avoid detection.

In the future, the researchers want to test their techniques on a larger set of model variables and test whether Gaussian evaluation can detect dangerous abilities in the underlying models before they are changed.

“Now we have a technical way to deal with this concern in part. A lot of effort has been poured into this collaboration, which enabled us to deal with a really difficult problem that hurts many children, nationally and globally. Hopefully, we can have a transformative effect in this area,” said Ghassemi.

This work was supported, in part, by a Bridgewater AIA Labs Research Fellowship.