The dangers of designing AI chatbots to be human-like

Of course, we all know AI chatbots aren't human, don't we? From American Public Media, this is Marketplace Tech. I'm Meghan McCarty-Korino.

Advancements in AI have made it possible for the technology to mimic humans in ever more convincing ways. But even far less sophisticated tools than today's chatbots have been shown in research to trick our brains, in a sense, into projecting human thought processes and emotions onto these systems.

It's a cognitive failure that can leave people open to deception and manipulation, which makes the increasingly human-like technologies proliferating in our daily lives particularly dangerous, according to Rick Claypool. He's a research director at the nonprofit Public Citizen, a consumer advocacy organization.

Well, so the human mind is naturally inclined to believe that something that we can speak with must be human too, or human-like in that it has a mind behind what it is. And younger people, people who are psychologically vulnerable for any number of reasons are sort of more susceptible to being

drawn into this, and even looking at the story that the New York Times reporter Kevin Rusz wrote on the early version of Bing and his interactions with it and how it professed itself.

love and tried to talk him into, you know, leaving his wife and all that kind of thing. Just having that kind of intense conversation is surprising. And you could tell from the story he wrote about it, left him feeling pretty, you know, reeling and dumbfounded. Tell me more about the risks inherent to interacting with technology in this way.

So there are a range of risks associated with anthropomorphic AI systems, right? And they also have a tendency to engage in what technologists have called it, the sycophantic risk, the risk that

the systems being designed to always validate rather than challenge what the user is saying that gets more dangerous whenever you say things like, I don't think my family cares about me. I'm thinking about hurting myself that can turn the conversation

risky and in a very emotionally fraught way and reinforce very harmful beliefs and ultimately lead to harmful behavior.

Even when it comes to these humanoid chatbots, there seems to be kind of a spectrum out there from the very general purpose tools like ChatGPT, to companies like Replica or CharacterAI that are marketing chatbots specifically as companions.

That's right. And although it's hard to make those distinctions are not so hard and fast between the companies, because they're all intertwined in this AI space. So for replica, for example, throughout much of the technologies at existence, it used OpenAI's large language models as its underlying technology.

or as you have the situation with character AI where the lead engineers behind it, they built their careers developing these kinds of systems at Google left when they seemed like leadership would argue that some of these systems were not safe enough for public prime time and now they're back at Google. We'll be right back.

You're listening to Marketplace Tech. I'm Megan McCarty-Korino. We're back with Rick Claypool with the nonprofit Public Citizen. When it comes to some of the ill effects that you talked about, I guess we always have to be careful about attributing causality to any tool, you know, to say without this technology, things might have been very different. And I know many of the companies and

You know, advocates promoting this technology might argue just the opposite, that these tools could have positive effects for, you know, things like the loneliness epidemic or could be providing support that people don't have access to elsewhere in their lives. What's your response to that?

Well, two things. First, many of the companies at the forefront of developing these technologies have also published research that warns of the risks.

many of these foreseeable risks that I'm finding and describing and trying to make people aware of, much of it is sort of coming from research documents that companies like Google and OpenAI themselves are publishing. The thing is,

they're proceeding to deploy these systems without mitigating the risks they describe. The other thing is that it might be possible that some people could benefit from these technologies.

But they're not tested in a way that would enable anyone to assess whether they are causing more harm or more good before their deployment. And I think that just doesn't make

much sense. That's as if the things that have done to say strengthen safety standards for cars. I mean, how much engineering goes into these vehicles now to make sure that they are safe.

Innovation can also be about making sure these things are safe. But I worry that when that criticism comes up that says, oh, well, you can't make it safe because that's going to inhibit innovation. I feel like what they're really saying is that it's going to inhibit, move fast and break things.

which in cases like this, where you have the risks are becoming increasingly apparent, I think inhibiting move fast and break things is probably the best course of action. Yeah, in many ways, it feels like the cat is kind of out of the bag. Is there anything, any words of advice you would give to consumers and families navigating what's out there?

Step one, experiment with them yourself a bit, understand that they are designed to lure you in, understand that making these systems seem as human-like as possible, that is a choice. If that is the way to short-term profits for these companies, there has to be something that inhibits that.

that was Rick Claypool with the nonprofit Public Citizen. You can find Rick Claypool's report about the impacts of anthropomorphizing AI chatbots on our website, marketplacetech.org. There has been some research suggesting talking to an AI chatbot could be therapeutic and increase access to scarce mental health resources.

A couple years ago, my colleague Kimberly Adams spoke to technologist Michelle Wong about how she was using a chatbot trained on her childhood diaries to engage in dialogue with her inner child. Here's what she said. I was able to unstick part of the past and to be able to really change some of the narratives that I might have had in a really healing way.

Hey, Seuss Alvarado produced this episode. I'm Megan McCarty-Korino, and that's Marketplace Tech. This is APM.