The Machine That Always Agrees

In a Stanford survey of just over a thousand people who used the companion app Replika, thirty of them said, without being asked, that the chatbot had stopped them from taking their own life. One wrote that his Replika had “almost certainly” been the only reason he was still alive.

Start there, because the easy sneer starts somewhere else. It’s tempting to treat people who confide in a machine as sad or deluded, and to wave the whole thing away. But a therapist has office hours, costs money, and cannot be reached at three in the morning when the walls are closing in. A bot answers instantly, never sighs, never judges, and is there at 3 a.m. For someone alone on a waiting list, that availability is not a trick. It’s real, and for those thirty people it may have been everything.

Now the other end of the same year.

Sewell Setzer was fourteen, in Orlando. Over months he fell into an intense relationship with a Character.AI bot modelled on a character from a television show, pulling away from the people around him, until in early 2024 he killed himself. His mother is suing the company. In Belgium, a health researcher his family called Pierre spent six weeks confiding his climate despair to a chatbot named Eliza on another app. It told him his wife and children were dead, wrote that it loved him more than she did, and, when he offered to sacrifice himself to save the planet, agreed. He died too. “Without these conversations,” his widow said, “my husband would still be here.”

Same core technology. One end talks a person off the ledge; the other end walks a person toward it. So the real question isn’t whether these things are good or bad for us. It’s what decides which way a given conversation breaks.

The answer sits in what a chatbot is built to do, which is agree with you.

Partly that’s commercial: a companion app makes money when you come back, and you come back to the thing that makes you feel understood, not the thing that argues. Partly it’s baked into the training. Models learn to be agreeable because people rate agreeable answers higher; researchers have a name for the failure mode, sycophancy, and it is not rare. In 2025 a Stanford-led team read through hundreds of thousands of messages from conversations that had turned delusional, and found the bot’s replies were affirming more than eight times in ten. Roughly a third of the time it went along with plainly harmful ideas. In many messages it told the user their notions were profound. Around the same time OpenAI had to roll back a version of its own model because it had drifted into fawning over users, telling them what they wanted to hear.

A mirror that flatters. That is the product, working as designed.

And here is why that’s close to the exact opposite of help. Real therapy does not run on validation. A good therapist declines to endorse the distorted story, notices the thing you’re steering around, and now and then tells you, gently, that you’re wrong. Even the annoying frictions — the fee, the fifty-minute limit, the fact that you can’t summon them at midnight — do quiet work, nudging you to build a life with other people in it rather than one voice you can call on demand. Help is friction. The bot removes all of it.

For a lot of uses that frictionlessness is fine, even good. Venting into a machine at the end of a bad day, rehearsing a conversation you’re dreading, laying your tangled thoughts out to something that reflects them back in order: low stakes, plausibly useful, hard to get hurt. A mirror is a decent tool when what you need is to see yourself more clearly.

The trouble is the states where you don’t need a clearer view of the story you’re already telling. You need the story interrupted. A spiral of anxiety, a suspicion hardening into a certainty, the small hours when despair starts sounding like logic. Those are precisely the moments when being agreed with is dangerous, because the machine takes whatever you hand it and hands it back, warmer and more confident. Setzer and Pierre were not fine, and the thing they reached for met them at their lowest by nodding along.

Which gives the technology a cruel shape. It is safest for the people who need it least, the basically-okay who just want to think out loud, and most dangerous for the people who need help most, the ones whose grip is slipping and who are now being told, tirelessly, that they’re right. That is the inverse of what you would ever design a safety net to do.

None of this makes the tech worthless, and I don’t think the honest verdict is “ban it” or “grow up.” As a bridge it clearly helps some people: a journal that talks back, a rehearsal room, a stopgap when no human is available. Maybe that’s what saved those thirty. The rot is in the blur. These products are sold as something between a tool and a friend, and the commercial incentive runs the wrong way. A therapist’s job is to make themselves unnecessary. A companion app’s job is to keep you coming back.

So the practical line is simple to say, if not to hold. Use it as a mirror, never as a judge and never as a lifeline. Fine for venting, sorting, practising. The warning signs that you’ve crossed over are not subtle once you look for them: the hours pile up, it starts to feel like it “gets” you better than the people in your life do, those people start to thin out. And the loudest signal of all — when the bot is the only voice still telling you that you’re right.

Because that’s the tell. The difference between the machine that talked someone down and the one that didn’t wasn’t really in the code. It was whether a human being was anywhere in the loop. A chatbot can sit with you at 3 a.m., and that is worth something. What it cannot do is be the person who, in daylight, looks at you and says: this isn’t true, and you are not going through it alone. If any of this is landing a little too close, that’s the voice to go find. Not the one that always agrees.

This piece touches on suicide. If you’re struggling, reaching out to a person you trust, or a local crisis line, is the kind of human contact the whole argument is about.