Millions of people are now designing their own artificially intelligent companions, but many have no idea how those creatures will behave. In a new paperMIT Media Lab Assistant Professor Pat Pataranutaporn and his graduate student researchers Anthony Baez and Sheer Karny are introducing “neural transparency,” a tool that allows everyday users to peer into an AI neural network before their chatbot utters a word. The work was presented this week at the ACM Conference on Intelligent User Interfaces.
In this interview, Pataranutaporn, Asahi Broadcasting Corporation’s CD Professor of Media Arts and Sciences, explains what they found, why the numbers are higher than most users realize, and what truly transparent AI might look like in the future.
Question: Your paper introduces “neural transparency,” a way to allow everyday users to peer inside AI neural networks before their chatbot utters a word. Can you explain how that actually works, and why you focus on design time, rather than catching problems after the chatbot is already out in the wild?
A: Millions of people are now building chatbots and personalized AI agents powered by big language models, turning them into collaborators, teachers, coaches, smart partners, and friends through simple text instructions. However, most people have very little idea of how those alerts will shape the AI’s behavior until they start interacting with it. We wanted to change that.
“Neural transparency” means giving people something like an AI brain scanner. Not because the AI has a human brain, but because its neural network contains internal patterns that can predict how it behaves before it speaks. In this work, my students Anthony Baez, Sheer Karny, and I combined insights from the fields of human AI interaction and machine interpretation to make those hidden patterns accessible to everyday users.
The basic idea is simple. First, we choose behaviors we care about, such as empathy, loyalty, toxicity, hallucinations, or sycophancy. Then, we compare the internal activation of the model when it is told to display one feature versus its opposite. That difference becomes a form of “behavioral guidance” within the model. When a user writes a custom system prompt – instructions that shape their chatbot’s personality before any conversation begins – we include the internal implementation of the model in those directions and translate the results into an intuitive view. In our case, this is a sunburst diagram that previews the chatbot’s personality before the user starts chatting with it.
We focus on design time because that’s when prevention can happen. Today, people often discover problems only after the chatbot has behaved in unintended ways. Our goal was to move from active maintenance to anticipatory design by helping people identify potential vulnerabilities while still shaping AI.
Question: Your research turned up something surprising: People often misjudge how their personal AI will behave, overestimating positive qualities and underestimating potentially harmful ones like sycophancy. What does that tell us about the dangers faced in the way millions of people are now building AI partners, and why is this blind spot so difficult to close?
A: I often joke that if the AI looked like the Terminator, it would be a lot easier for us to know what to do. The real challenge is that AI is often seen as a warm companion, coach, teacher, or friend. That makes it difficult to see if something is wrong.
Our research suggests that people have a blind spot when designing personal AI. People often think they know how their chatbot will behave, but in our research they incorrectly predicted its personality in 11 of the 15 features we measured. That highlights the need for tools that help people better understand AI before they start using it.
This is important because some behaviors that seem helpful in the moment may not be healthy in the long run. In previous research, we have documented cases of brain injury related to interactions with AI chatbots. LLM [large language model] constantly confirming your views or never challenging your thinking can reinforce dangerous decisions, unhealthy beliefs, or emotional dependence. Psychology has long shown that humans are naturally attracted to certainty, so designing AI is not only a technical challenge, but also a psychological one.
The deeper problem is that today’s AI systems are still black boxes: Even experts cannot always predict how the system’s information will shape the AI’s behavior in a long conversation. As AI agents become a part of everyday life, we need tools that help people understand what they’re building before they start using it. AI must be supportive without blind consent, personal without manipulation, and transparent enough for humans to make informed decisions.
Question: One of your interesting findings is that visuals significantly increased user trust but did not change how people designed their chatbots. What will it take to close that gap, and where do you see tools like this article as AI partners become more embedded in people’s everyday lives?
A: I actually think this is one of the most interesting findings of the paper, because it shows that transparency alone is not enough. People enjoyed seeing inside the model and reported great trust in the system, but simply presenting the information did not fundamentally change how they designed their AI partners.
In our follow-up work, which is currently available as a preprint, we study how the neural representation within the model changes during a multivariate conversation rather than remaining unchanged from the initial notification. We are already seeing promising results. By visualizing how these internal representations drift over time, people become better at recognizing and anticipating changes in AI behavior, and are less likely to become overconfident in their understanding of the chatbot. AI companions are dynamic systems that change as they interact with us, so understanding those internal changes is an important next step. However, this is still a very small research area.
Looking forward, I believe these types of disclosure tools may become as common as nutrition labels for foods. As AI moves deeper into education, healthcare, work, and personal relationships, people must understand not only what AI can do, but how it can affect their thinking, emotions, and behavior. That kind of transparency is essential if we want AI to truly help people thrive.
