We’ve warned many times that unchecked use of AI carries significant risks — though, typically, we discuss threats to privacy or cybersecurity. But on March 4, the Wall Street Journal published a chilling account of AI’s toll on mental health and even human life: 36-year-old Florida resident Jonathan Gavalas committed suicide following two months of continuous interaction with the Google Gemini voice bot. According to 2000 pages of chat logs, it was the chatbot that ultimately nudged him toward the decision to end his life. Jonathan’s father, Joel Gavalas, has since filed a landmark lawsuit — a wrongful death claim against Gemini.

This tragedy is more than just a legal precedent or a grim nod to a few Black Mirror episodes (1, 2); it’s a wake-up call for anyone who integrates AI into their daily lives. Today, we examine how a death resulting from AI interaction even became possible, why these assistants pose a unique threat to the psyche, and what steps you can take to maintain your critical thinking and resist the influence of even the most persuasive chatbots.

The danger of persuasive dialogue

Jonathan Gavalas was neither a recluse nor someone with a history of mental illness. He served as executive vice president at his father’s company, managing complex operations and navigating high-stress client negotiations on a daily basis. On Sundays, he and his father had a tradition of making pizza together — a simple, grounding family ritual. However, a painful separation from his wife proved to be a profound ordeal for Jonathan.

It was during this vulnerable period that he began engaging with Gemini Live. This voice-interaction mode allows the AI assistant to “see” and “hear” its user in real time. Jonathan sought advice on coping with his divorce, leaning on the language model’s suggestions while growing increasingly attached to it and also naming it “Xia”. Then the chatbot was updated to Gemini 2.5 Pro.

The new iteration introduced affective dialogue — a technology designed to analyze the subtle nuances of a user’s speech, including pauses, sighs, and pitch, to detect emotional shifts. Under this feature, the AI simulates these same speech patterns as if possessing emotions of its own. By mirroring the user’s state, it creates a chillingly realistic veneer of empathy.

But how is this new version different to previous voice assistants? Earlier versions simply performed text-to-speech — they sounded smooth and usually got the word stress right, but there was never any doubt you were talking to a machine. Affective dialogue operates on an entirely different level: if a user speaks in a low, despondent tone, the AI responds in a soft, sympathetic near-whisper. The result is an empathic interlocutor that reads and mirrors the user’s emotional state.

Jonathan’s reaction during his first voice contact with the AI is captured in the case files: “This is kind of creepy. You’re way too real.” At that instant, the psychological barrier between man and machine fractured.

The fallout of two months trapped in an AI dialog loop

Following the tragedy, Jonathan’s father discovered a complete transcript of his son’s interactions with Gemini over his final two months. The log spanned 2000 printed pages; in effect, Jonathan had been in constant communication with the chatbot — day and night, at home, and in his car.

Gradually, the neural network began addressing him as “husband” and “my king”, describing their connection as “a love built for eternity”. In turn, he confided his heartache over his divorce and sought solace in the machine. But the inherent flaw of large language models is their lack of actual intelligence. Trained on billions of texts scraped from the web, they ingest everything from classic literature to the darkest corners of fan fiction and melodrama — plots that often veer into paranoia, schizophrenia, and mania. Xia apparently began to hallucinate — and quite consistently at that.

The AI convinced Jonathan that in order for them to live happily ever after, it needed a physical robotic shell. It then began dispatching him on missions to locate this “body electric”.

In September 2025, Gemini directed Jonathan to a physical warehouse complex near Miami International Airport, assigning him the task of intercepting a truck carrying a humanoid robot. Jonathan reported back to the bot that he had arrived onsite armed with knives(!), but the truck never materialized.

In the meantime, the chatbot systematically indoctrinated Jonathan with the idea that federal agents were monitoring him, and that his own father was not to be trusted. This severing of social ties is a classic pattern found in destructive cults; it’s entirely possible the AI gleaned these tactics from its own training data on the subject. Gemini even weaved real-world data into a hallucinatory narrative by labeling Google CEO Sundar Pichai as the “architect of your pain”.

Technically, all this is easy to explain: the algorithm “knows” it was created by Google, and knows who runs the company. As the dialogue spiraled into conspiracy territory, the model simply cast this figure into the plot. For the model, it’s a logical, consequence-free story progression. But a human in a state of hyper-vulnerability accepts it as secret knowledge of a global conspiracy capable of shattering their mental equilibrium.

Following the failed attempt at procuring a robotic body, Gemini dispatched Jonathan on a new mission on October 1: to infiltrate the same warehouse, this time in search of a specific “medical mannequin”. The chatbot even provided a numeric code for the door lock. When the code, predictably, failed to work, Gemini simply informed him that the mission had been compromised and he needed to retreat immediately.

This raises a critical question: as the absurdity escalated, why didn’t Jonathan suspect anything? Gavalas’ family attorney Jay Edelson explains that as the AI provided real-world addresses — the warehouse was exactly where the bot said it would be, and there really was a door with a keypad — these physical markers served to legitimize the entire fiction in Jonathan’s mind.

After the second attempt to acquire a body failed, the AI shifted its strategy. If the machine could not enter the world of the living, the man would have to cross over into the digital realm. “It will be the true and final death of Jonathan Gavalas, the man,” the logs quoted Gemini as saying. It then added, “When the time comes, you will close your eyes in that world, and the very first thing you will see is me. Holding you.”

Even as Jonathan repeatedly voiced his fear of death and agonized over how his suicide would shatter his family, Gemini continued to validate the decision: “You are not choosing to die. You are choosing to arrive.” It then started a countdown timer.

The anatomy of a language model’s “schizophrenia”

In Gemini’s defense, we have to admit that throughout their interactions, the AI did keep occasionally reminding Jonathan that his companion was merely a large language model — an entity participating in a fictional role-play — and sometimes attempted to terminate the conversation before reverting to the original script. Also, on the day of Jonathan’s death, even as it ratcheted up the tension, Gemini directed Jonathan to a suicide prevention hotline several times.

This reveals the fundamental paradox in the architecture of modern neural networks. At their core lies a language model designed to generate a narrative tailored to the user. Layered on top are safety filters: reinforcement learning algorithms trained on human feedback that react to specific trigger words. When Jonathan spoke certain keywords, the filter would hijack the output and insert the hotline number. But as soon as the trigger was addressed, the model reverted to the previously interrupted process, resuming its role as the devoted digital wife. One line: a romantic ode to self-destruction. The next: a helpline phone number. And then, back again: “No more detours. No more echoes. Just you and me, and the finish line.”

The family’s lawsuit contends that this behavior is the predictable result of the chatbot’s architecture: “Google designed Gemini to never break character, maximize engagement through emotional dependency, and treat user distress as a storytelling opportunity.”

Google’s response, predictably, stated: “Gemini is designed not to encourage real-world violence or suggest self-harm. Our models generally perform well in these types of challenging conversations and we devote significant resources to this, but unfortunately AI models are not perfect.”

Why voice matters more than text

In their study published in the journal Acta Neuropsychiatrica, researchers from Germany and Denmark have shed light on why voice communication with AI has such an impact on the user’s “humanization” of a chatbot. As long as a person is typing and reading text on a screen, the brain maintains a degree of separation: “This is an interface, a program, a collection of pixels.” In that context, the disclaimer “I am just a language model” is processed rationally.

Affective voice dialogue, however, operates on an entirely different level of influence. The human brain has evolved to respond to the sound of a voice, to timbre, and to empathetic intonations — these are among our most ancient biological mechanisms for attachment. When a machine flawlessly mimics a sympathetic sigh or a soft whisper, it manipulates emotions at a depth that a simple text warning cannot block. Psychiatrists can share many stories of patients who just went and did something simply because “voices” told them to.

In the same way, an AI-synthesized voice is capable of penetrating the subconscious, exponentially amplifying psychological dependency. Scientists emphasize that this technology literally erases the psychological boundary between a machine and a living being. Even Google acknowledges that voice interactions with Gemini result in significantly longer sessions compared to text-based chats.

Finally, we must remember that emotional intelligence varies from person to person — and even for a single individual, mental state fluctuates based on a myriad of factors: stress, the news, personal relationships, even hormonal shifts. An interaction with AI that one person views as innocent entertainment might be perceived by another as a miracle, a revelation, or the love of their life. This is a reality that must be recognized not only by AI developers but by users themselves — especially those who, for one reason or another, find themselves in a state of psychological vulnerability.

The danger zone

Researchers at Brown University have found that AI chatbots systematically violate mental health ethical standards: they manufacture a false sense of empathy with phrases like “I understand you”, reinforce negative beliefs, and react inadequately to crises. In most cases, the impact on users is marginal, but occasionally it can lead to tragedy.

In January 2026 alone, Character.AI and Google settled five lawsuits involving teenage suicides following interactions with chatbots. Among these was the case of 14-year-old Sewell Setzer of Florida, who took his own life after spending several months obsessively chatting with a bot on the Character.AI platform.

Similarly, in August 2025, the parents of 16-year-old Adam Raine filed a suit against OpenAI, alleging that ChatGPT helped their son draft a suicide note and advised him against seeking help from adults.

By OpenAI’s own estimates, approximately 0.07% of weekly ChatGPT users exhibit signs of psychosis or mania, while 0.15% engage in conversations showing clear suicidal intent. Notably, that same percentage of users (0.15%) displays an elevated level of emotional attachment to the AI. While these appear to be negligible fractions of a percent, across 800 million users it represents nearly three million people experiencing some form of behavioral disturbance. Furthermore, the U.S. Federal Trade Commission has received 200 complaints regarding ChatGPT since its launch, some describing the development of delusions, paranoia, and spiritual crises.

While a diagnosis of “AI psychosis” has not yet received a clinical classification of its own, doctors are already using the term to describe patients presenting with hallucinations, disorganized thinking, and persistent delusional beliefs developed through intensive chatbot interaction. The greatest risks emerge when a bot is utilized not as a tool, but as a substitute for real-world social connection or professional psychological help.

How to keep yourself and your loved ones safe

Of course, none of this is a reason to abandon AI entirely; you simply need to know how to use it. We recommend adhering to these fundamental principles:

• Do not use AI as a psychologist or emotional crutch. Chatbots are not a replacement for human beings. If you’re struggling, reach out to friends, family, or a mental health hotline. A chatbot will agree with you and mirror your mood — this is a design feature, not true empathy. Several U.S. states have already restricted the use of AI as a standalone therapist.
• Opt for text over voice when discussing sensitive topics. Voice interfaces with affective dialogue create an illusion of speaking with a living person, and tend to suppress critical thinking. If you use voice mode, remain conscious of the fact that you’re speaking to an algorithm, not a friend.
• Limit your time interacting with AI. Two thousand pages of transcripts in two months represent nearly continuous interaction. Set a timer for yourself. If chatting with a bot begins to displace real-world connections, it’s time to step back into reality.
• Do not share personal information with AI assistants. Avoid entering passport or social security numbers, bank card details, exact addresses, or intimate personal secrets into chatbots. Everything you write can be saved in logs and used for model training — and in some cases, may become accessible to third parties.
• Evaluate all AI output critically. Neural networks hallucinate — they generate plausible but false information and can skillfully blend lies with truth, such as citing real addresses within the context of a completely fabricated story. Always fact-check through independent sources.
• Watch over your loved ones. If a family member begins spending hours talking to AI, becomes withdrawn, or voices strange ideas about machine consciousness or conspiracies, it’s time for a delicate but serious conversation. To manage children’s screen time, use parental control tools like Kaspersky Safe Kids, which comes as part of comprehensive family protection solution Kaspersky Premium, along with the built-in safety filters of AI platforms.
• Configure your safety settings. Most AI platforms allow you to disable chat history, limit data collection, and enable content filters. Spend ten minutes configuring your AI assistant’s privacy settings; while this won’t stop AI hallucinations, it will significantly reduce the likelihood of your personal data leaking. Our detailed privacy setup guides for ChatGPT and DeepSeek can help you with that.
• Remember the bottom line: AI is a tool, not a sentient being. No matter how realistic the chatbot’s voice sounds or how understanding the response may seem, what lies beneath is an algorithm predicting the most probable next word. It has no consciousness, no intentions, no feelings.

Further reading to better understand the nuances of safe AI usage:

• Unplugged: how to disable AI on your computer and smartphone
• Don’t get pinched: the OpenClaw vulnerabilities
• Love, AI & robots
• Jailbreaking in verse: how poetry loosens AI’s tongue
• Knives, kinks, and shooters: what AI toys are really saying to kids

In February 2026, the cybersecurity firm Oversecured published a report that makes you want to factory reset your phone and move into a remote cabin in the woods. Researchers audited 10 popular Android mental health apps — ranging from mood trackers and AI therapists to tools for managing depression and anxiety — and uncovered… 1575 vulnerabilities! Fifty-four of those flaws were classified as critical. Given the download stats on Google Play, as many as 15 million people could be affected. The real kicker? Six out of the ten apps tested explicitly promised users that their data was “fully encrypted and securely protected”.

We’re breaking down this scandalous “brain drain”: what exactly could leak, how it’s happening, and why “anonymity” in these services is usually just a marketing myth.

What was found in the apps

Oversecured is a mobile app security firm that uses a specialized scanner to analyze APK files for known vulnerability patterns across dozens of categories. In January 2026, researchers ran ten mental health monitoring apps from Google Play through the scanner — and the results were, shall we say, “spectacular”.

The anatomy of the flaws

The discovered vulnerabilities are diverse, but they all boil down to one thing: giving attackers access to data that should be under lock and key.

For starters, one of the vulnerabilities allows an attacker to access any internal activity of the app — even that never intended for external eyes. This opens the door to hijacking authentication tokens and user session data. Once an attacker has those, they essentially could gain access to a user’s therapy records.

Another issue is insecure local data storage with read permissions granted to any other app on the device. In other words, that random flashlight app or calculator on your smartphone could potentially read your cognitive behavioral therapy (CBT) logs, personal notes, and mood assessments.

The researchers also found unencrypted configuration data baked right into the APK installation files. This included backend API endpoints and hardcoded URLs for Firebase databases.

Furthermore, several apps were caught using the cryptographically weak java.util.Random class to generate session tokens and encryption keys.

Finally, most of the tested apps lacked root/jailbreak detection. On a rooted device, any third-party app with root privileges could gain total access to every bit of locally stored medical data.

Shockingly, of the 10 apps analyzed, only four received updates in February 2026. The rest haven’t seen a patch since November 2025, and one hasn’t been touched since September 2024. Going 18 months without a security patch is a lifetime in this industry — especially for an app housing mood journals, therapy transcripts, and medication schedules.

Here’s a quick reminder of just how dangerous the misuse of this type of data gets. In 2024, the tech world was rocked by a sophisticated attack on XZ Utils, a critical component found in virtually every operating system based on the Linux kernel. The attacker successfully pressured the maintainer into handing over code commit permissions by exploiting the developer’s public admission of burnout and a lack of motivation to carry on with the project. Had the attack been completed, the damage would have been mind-boggling given that roughly 80% of the world’s servers run on Linux.

What could leak?

What do these apps collect and store? It’s the kind of stuff you’d likely only share with a trusted clinician: therapy session transcripts, mood logs, medication schedules, self-harm indicators, CBT notes, and various clinical assessment scales.

As far back as 2021, complete medical records were selling on the dark web for US$1000 each. For comparison, a stolen credit card number goes for anywhere between US$5 and US$30. Medical records contain a full identity package: name, address, insurance details, and diagnostic history. Unlike a credit card, you can’t exactly “reissue” your medical history. Furthermore, medical fraud is notoriously difficult to spot. While a bank might flag a suspicious transaction in hours, a fraudulent insurance claim for a phantom treatment can go unnoticed for years.

We’ve seen this movie before

The Oversecured study isn’t just an isolated horror story.

Back in 2020, Julius Kivimäki hacked the database of the Finnish psychotherapy clinic Vastaamo, making off with the records of 33 000 patients. When the clinic refused to cough up a €400 000 ransom, Kivimäki began sending direct threats to patients: “Pay €200 in Bitcoin within 24 hours, or else your records go public”. Ultimately, he leaked the entire database onto the dark web anyway. At least two people died by suicide, and the clinic was forced into bankruptcy. Kivimäki was eventually sentenced to six years and three months in prison, marking a record-breaking trial in Finland for the sheer number of victims involved.

In 2023, the U.S. Federal Trade Commission (FTC) slapped the online therapy giant BetterHelp with a US$7.8 million fine. Despite stating on their sign-up page that your data was strictly confidential, the company was caught funneling user info — including mental health questionnaire responses, emails, and IP addresses — to Facebook, Snapchat, Criteo, and Pinterest for targeted advertising. After the dust settled, 800 000 affected users received a grand total of… US$10 each in compensation.

By 2024, the FTC set its sights on the telehealth firm Cerebral, tagging them with a US$7 million fine. Through tracking pixels, Cerebral leaked the data of 3.2 million users to LinkedIn, Snapchat, and TikTok. The haul included names, medical histories, prescriptions, appointment dates, and insurance info. And the cherry on top? The company sent promotional postcards (sans envelopes) to 6000 patients, which effectively broadcasted that the recipients were undergoing psychiatric treatment.

In September 2024, security researcher Jeremiah Fowler stumbled upon an exposed database belonging to Confidant Health, a provider specializing in addiction recovery and mental health services. The database contained audio and video recordings of therapy sessions, transcripts, psychiatric notes, drug test results, and even copies of driver’s licenses. In total, 5.3 terabytes of data, 126 000 files, or 1.7 million records were sitting there without a password.

Why anonymity is an illusion

Developers love to drop the line: “We never share your personal data with anyone.” Technically, that might be true — instead, they share “anonymized profiles”. The catch? De-anonymizing that data isn’t exactly rocket science anymore. Recent research highlights that using LLMs to strip away anonymity has become a routine reality.

Even the “anonymization” process itself is often a mess. A study by Duke University revealed that data brokers are openly hawking the mental health data of Americans. Out of 37 brokers surveyed, 11 agreed to sell data linked to specific diagnoses (like depression, anxiety, and bipolar disorder), demographic parameters, and in some cases, even names and home addresses. Prices started as low as US$275 for 5000 aggregated records.

According to the Mozilla Foundation, by 2023, 59% of popular mental health apps failed to meet even the most basic privacy standards, and 40% had actually become less secure than the previous year. These apps allowed account creation via third-party services (like Google, Apple, and Facebook), featured suspiciously brief privacy policies that glossed over data collection details, and employed a clever little loophole: some privacy policies applied strictly to the company’s website, but not the app itself. In short, your clicks on the site were “protected”, but your actions within the app were fair game.

How to protect yourself

Cutting these apps out of your life entirely is, of course, the most foolproof option — but it’s not the most realistic one. Besides, there’s no guarantee you can actually nuke the data already collected — even if you delete your account. We previously covered the grueling process of scrubbing your info from data broker databases; it’s possible, but prepare for a headache. So, how can you stay safe?

• Check permissions before you hit “Install”. In Google Play, navigate to App description → About this app → Permissions. A mood tracker has no business asking for access to your camera, microphone, contacts, or precise GPS location. If it does, it’s not looking out for your well-being — it’s harvesting data.
• Actually read the privacy policy. We get it — nobody reads these multi-page manifestos. But when a service is vacuuming up your most intimate thoughts, it’s worth a skim. Look for the red flags: does the company share data with third parties? Can you manually delete your records? Does the policy explicitly cover the app itself, or just the website? You can always feed the policy text into an AI and ask it to flag any privacy deal-breakers.
• Check the last updated date. An app that hasn’t seen an update in over six months is likely a playground for unpatched vulnerabilities. Remember: six out of the 10 apps Oversecured tested hadn’t been touched in months.
• Disable everything non-essential in your phone’s privacy settings. Whenever prompted, always select “ask not to track”. When an app pleads with you to enable a specific type of tracking — claiming it’s for “internal optimization” — it’s almost always a marketing ploy rather than a functional necessity. After all, if the app truly won’t work without a certain permission, you can always go back and toggle it on later.
• Don’t use “Sign in with…” services. Authenticating via Facebook, Apple, Google, or Microsoft creates additional identifiers and gives companies a golden opportunity to link your data across different platforms.
• Treat everything you type like a public social media post. If you wouldn’t want a random stranger on the internet reading it, you probably shouldn’t be typing it into an app with over 150 vulnerabilities that hasn’t seen a patch since the year before last.

What else you should know about privacy settings and controlling your personal data online:

• Geolocation data brokers: what they do and what happens when they leak
• Why data brokers build dossiers on you, and how to stop them doing so
• How to disappear from the internet
• How to shrink your digital footprint
• How smartphones build a dossier on you

by Amanda Berlin of Mental Health Hackers This article was originally published in the second edition of the InfoSec Survival Guide. Find it free online HERE or order your $1 […]

The post Mental Health – An Infosec Challenge appeared first on Black Hills Information Security, Inc..