Those losses stem from 22,364 AI-related complaints. And these figures represent only the reported losses, which may well be the proverbial tip of the iceberg.
The main drivers behind the rise in AI-powered scams are voice cloning, deepfake images and videos, and AI‑generated scripts. These tools have supercharged classic fraud schemes such as romance scams, kidnapping and extortion calls, fake influencers, and government impersonation.
Michael Machtinger, deputy assistant director of the FBI Cyber Division, told the Wall Street Journal:
“AI-created fraudulent communications can look very official and very legitimate to even the most trained individuals.”
The FBI and financial institutions recommend verifying identities via official contact channels. One of their biggest concerns is government impersonation scams, which have evolved from crude IRS gift‑card phone calls into sophisticated, multi‑channel operations that combine spoofed caller ID, stolen agency logos, and AI‑generated audio and video of public officials.
This report, and others like it, shows how AI is being weaponized to automate research on victims, generate convincing scripts, and create highly believable deepfake personas at scale.
AI is also increasingly used in business email compromise (BEC), romance scams, and impersonation fraud. In BEC cases involving AI, losses have already reached tens of millions of dollars for businesses alone.
For a broader look at why AI is simultaneously fueling scams like these and becoming indispensable to defending against them, see my article AI: Threat, tool, or both?
It explains how both defenders and criminals use AI to find vulnerabilities, and why security vendors increasingly rely on AI to process vast amounts of telemetry, detect anomalies, and keep pace with threats that “no longer move at human speed.”
How to stay safe
Consumer protection agencies have documented a growing list of the ways scammers are using AI to try to rip people off. The main problem is that we can no longer take it at face value that the person we’re talking to is who they claim to be.
Government agencies and financial institutions recommend that you:
Be skeptical of urgent payment demands, especially those involving cryptocurrency or gift cards
Limit the amount of voice and video content you share publicly, as it can be reused by scammers
Report incidents quickly to your bank(s) and IC3.gov
Pro tip: Malwarebytes Scam Guard can help you determine whether a message is a scam and guide you through the next steps.
Something feel off? Check it before you click.
Malwarebytes Scam Guard helps you analyze suspicious links, texts, and screenshots instantly.
It starts with the familiar: a short message, a trusted name, a routine tone. Delivery updates, work pings, brand alerts hum in the background, rarely attracting scrutiny. You check, you answer… — until minutes later you’ve slipped into a trap built to lower your guard and hijack your trust.
That’s why messaging scams cut deep: they exploit everyday habits where instinct, not caution, leads. Communication once moved slowly, leaving room for doubt. Now it’s instant — and that speed is a weapon in criminal hands.
On our blog, we’ve already examined numerous scam schemes in messaging apps — from pig butchering, where the victim is groomed for a very long time, or catfishing, where the scammer creates a fake identity, to phishing via chatbots or through gift-giving campaigns in messaging apps.
Now, for the first time, Kaspersky has set out to capture the full end-to-end reality of messaging-based scams to understand how quickly harm occurs, how they impact trust and what remains after the interaction ends. What emerges is a highly organized and industrialized scam ecosystem embedded within everyday messaging channels such as SMS, WhatsApp, and email.
Kaspersky experts have prepared a report on targeted scams in messaging apps, detailing not only the financial but also the emotional damage caused by such attacks, as well as providing tips on how to protect yourself and avoid them. In this post, we explore the most interesting facts, but you can find more details in the full report.
The damage is underestimated
How much do you think a single successful attack via a messaging app costs the average victim? Ten dollars? Or maybe 50? You’re underestimating the scammers. Although more than a third (36%) of victims incur losses of less than $135, on average a victim loses… $733!
Country
Average loss per victim
Senegal
$392.94
Serbia
$493.32
Morocco
$504.28
Greece
$609.32
United Kingdom
$617.38
Côte d’Ivoire
$654.11
Spain
$672.67
United States
$724.73
Portugal
$868.20
Italy
$896.02
France
$1,193.58
Germany
$1,369.35
The average amount lost by a victim in a successful attack via a messaging app
On the one hand, the financial hit doesn’t look catastrophic in isolation. These are micro-losses by design. Small enough that some never report them to the police. Small enough that banks don’t always investigate. Small enough to be dismissed as bad luck rather than organized crime.
But $733 is not nothing. It’s enough to cover a month’s worth of groceries, school or daycare fees, or utility bills. Against the backdrop of the global cost-of-living crisis, a single such loss can seriously dent a family’s budget.
In 11% of cases, losses exceed $1,350, and more than a quarter of victims (28%) report having been scammed three or more times in the past six months. Once scammers discover that a phone number responds, that contact becomes an asset, circulating from one database to another.
Now imagine the scale of the problem: if just 10% of the three billion messaging‑app users worldwide fell victim with the average loss, the total damage would amount to… nearly $220 billion! This is comparable to the GDP of Greece, and exceeds that of Morocco, Serbia, or Côte d’Ivoire.
It becomes clear that behind the daily flood of fraudulent schemes lie large scam cartels operating on an industrial scale, using AI to personalize messages that mimic those of family members, friends, and familiar brands. This, in essence, forms the basis of a full-fledged economy built on digital identity theft.
Speed beats scrutiny
More than half of successful messaging scams (52%) unfold in under 30 minutes — from first contact to the moment money or personal data changes hands — or even faster, before the victim begins to doubt the legitimacy of the sender. In fact, one in seven scams takes less than five minutes — quicker than boiling an egg!
The speed isn’t accidental. It’s the method. Scammers structure their schemes to deny the victim a chance to come to their senses. Every element is engineered to compress the decision-making window: the urgency of the scenario, the familiarity of the format, the plausibility of the request.
They rush you — faster, faster, don’t tell anyone, you only have a few minutes, solve the problem, don’t ask questions. Click the link, fill in the details, approve the transaction, or else… Or else what? The scammers’ imagination knows no bounds here, but if you don’t do something right now, you’ll definitely regret it.
Alas, the realization of what has happened usually comes when the damage is already irreversible. More than half of victims (51%) lose money; another 43% hand over their personal data — most commonly phone numbers, names, and email addresses — to scammers, and often the victim loses both.
Where and how attacks occur
A delivery notification, a bank alert, a message from a merchant you ordered from last week — messaging apps permeate every aspect of everyday life, making such interactions completely normal. An attack shouldn’t feel like an attack. It should feel like the same message you’ve received hundreds of times.
It’s no surprise that scammers focus their attention on this method of communication first and foremost. The most popular platforms for scams are predictable: WhatsApp (43%), SMS/iMessage (40%), Facebook (27%), Telegram (22%), and Instagram (19%) — these are the ones that people trust most.
A wide variety of schemes is used. Brand impersonation is now one of the three most common types of messaging scam worldwide — accounting for 31% of cases. Fake delivery notifications top the list at 38%, followed by investment scams at 37%.
At the same time, nearly two-thirds (63%) of fraudulent schemes span multiple platforms, moving from SMS to WhatsApp, from WhatsApp to Telegram, etc. In this way, scammers achieve two goals: they mimic organic messaging and evade moderation algorithms.
AI has taken scams to a new level
Just a couple of years ago, fraudulent messages gave themselves away with bad grammar, awkward phrasing, illogical requests, and an obsessive sense of urgency. Today, a phishing message looks, sounds, and reads just like the real thing.
Scam cartels want to catch people in motion — between meetings, on a commute, or during everyday tasks — when your attention is already fragmented. They mimic your mother’s turn of phrase. They match your bank’s tone of voice. They copy your courier’s format exactly. They mirror the rhythm, structure, and style of authentic brand communications across messaging platforms. And AI is accelerating all of it.
What this creates is overlap. Legitimate and fraudulent messages appear in the same environment, using the same formats, language, and triggers. The difference between them is no longer obvious.
The data shows that two-thirds of victims (66%) believe AI was used in the scam against them, 42% cite messages written by AI, 31% report generated or cloned voices, and 25% encountered deepfake images or videos.
That’s why mere awareness and “tech-savviness” may no longer be enough to protect oneself. From Gen Z to Gen X, messaging scams cut across every generation.
And what about the emotional toll?
But money is far from the only problem a victim is left with after an attack. After what they’ve been through, people develop distrust toward incoming messages, unfamiliar numbers, and any requests for action. As a result, 99% of fraud victims say they no longer trust incoming notifications in messaging apps.
This creates a crisis of trust in all digital channels in general. Every legitimate message can now be perceived as a scam. Brands, banks, and delivery services are forced to operate in an environment where the customer is, by default, in a state of distrust.
Dr. Elizabeth Carter, a forensic linguist and criminologist at Kingston University in London, notes that scammers use familiar contexts, common social settings and embedded linguistic norms to create the illusion for the victim that their decision-making is rational and reasonable in the moment. However, what is actually happening is that they construct false realities in which those decisions end up causing financial and psychological harm. She also notes that it is very hard to identify a false reality while you are in it.
After realizing they had been deceived, more than half of victims felt anger — the kind that comes from having trusted something and discovering it was used against you. 42% of victims report frustration, 38% — feeling upset. Moreover, several months later, these feelings haven’t gone away: nearly half of all victims (48%) are still angry, a third (33%) remain frustrated, and 30% are upset.
And nearly one in 10 victims don’t tell anyone what happened. They feel shame, a sense of having fallen for something so obvious. This leaves a significant portion of the actual damage unreported: only 24% of victims contact the police, and only 23% report it to their bank.
So what can be done?
The crisis of trust — and even a touch of paranoia — that has arisen due to widespread attacks on users can linger in victims’ minds for a long time, affecting their quality of life. To prevent this, follow these guidelines:
Pause before you act. The sense of urgency you feel is almost always artificial. A legitimate bank, retailer, or delivery service won’t penalize you for taking 30 seconds to verify before clicking a link or confirming details. It’s precisely this instinct to resolve the situation quickly that scammers are counting on.
Verify through another channel. If a message appears to be from a relative, colleague, or company you trust — contact them through another channel before taking any action. Use secure verification methods, and cross-check identities when something doesn’t feel right. For families, agreeing on a “safe word” in advance can defeat even the most convincing voice clones.
Use a password manager. It will not only help you generate strong, unique passwords for all your accounts and store them securely, syncing them across all your devices, but also protect you from spoofed sites. Even if you click a phishing link and land on such a site, our password manager will notify you about the domain mismatch and refuse to autofill your username and password.
Use protection that works in real time. Modern security solutions, such as Kaspersky Premium, provide real-time protection against malicious links and phishing attempts in the apps and websites you use every day. On Android devices, a dedicated layer of anti-phishing security scans and neutralizes suspicious links as they appear, even within notifications, before you even have a chance to click them.
We’ve covered other threats in messaging apps in similar articles:
As we mark Safer Internet Day 2026, we’re reflecting on a simple but enduring principle: safety must be designed into online services, not bolted on. Microsoft’s work in this space spans more than two decades—from technology solutions like PhotoDNA to our investments in responsible gaming, public-private partnerships, and empowering users through education. This foundation guides our approach as we help individuals and families navigate a rapidly evolving landscape shaped by new technologies and new risks and as we innovate with next-generation AI offerings. At a moment when 91% of people tell us they worry about harms introduced by AI, our commitment to responsible innovation has never been more important—especially for our youngest users.
Read on for more about our longstanding efforts tocreate a safer digital environment, plus key findings from our Global Online Safety Survey and new examples of our work to empower families and communities through tools, research, and educational resources—including the latest release in Minecraft Education’s CyberSafe series.
Ten years of safety research
2026 marks the tenth year of our annual Global Online Safety Survey research. For a decade, we have invested in surveying teens and adults around the world about their experiences and perceptions of life online—aiming to provide fresh insights to support our collective work. That’s 130,000+ interviews across 37 countries, with the results available on our website. Ten years later, respondents tell us that they feel more connected and more productive, but less safe online.
This year’s Global Online Safety Survey also highlights the complexity of the digital environment young people now inhabit. Teens’ exposure to risk rose again, with hate speech (35%), scams (29%), and cyberbullying (23%) among the most commonly experienced harms. At the same time, teens demonstrated striking resilience: 72% talked to someone after experiencing a risk, and reporting behavior increased for the second consecutive year. But worries about the misuse of AIcontinue, underscoring againwhy safety-by-design for AI is essential, not optional. Find the full results and country-level summaries here.
Year on year, the research has told a story of evolving online safety risks and of the real-world impact. In 2026, the call to action is more urgent than ever—unless industry can deliver safe and age-appropriate experiences, young people risk losing access to technology.At Microsoft, spanning across our teams from Windows to Xbox, we have sought to continuously evolve our approach and to lead industry in advancing tailored and thoughtful safety solutions.
Evolving to meet the moment
Looking ahead, we know we need to continue to build strong guardrails to tackle acute risks and to leverage our experience while being informed by new research, new perspectives, and new technologies. The application process closed yesterday for our first AI Futures Youth Council, to be comprised of teens from across the US and EU. We’re looking forward to bringing those teens together soon for a first meeting to get their direct feedback on the role they want emerging technology to play in their lives and how we can best support their safety.
Microsoft has partnered with Cyberlite on a second youth-centered initiative to understand how teens aged 13–17 are engaging with AI companions. Through codesign workshops with students in India and Singapore, we’re capturing young people’s own perspectives on the benefits, risks, and emotional dimensions of AI use—insights that will directly inform educational resources for teens, parents, and educators. Early findings from the first workshop in December 2025 show that young people value AI as a judgment free space while also recognizing the tradeoffs: privacy risks, overreliance, and erosion of critical thinking loom larger for them than bad advice.
We’re also thinking about how we define safety in the next era of Windows, leveraging the Family Safety controls that have been integrated for over a decade. As many countries have raised the local age for digital consent, more parents will have the option to enable parental controls for teens up to the age of 18—leveraging these tools as part of a holistic approach to digital parenting. And to help parents set up and understand Family Safety, we’ve developed a shortnew guide.
Safety is also about transparency, empowerment, and education. At Xbox, bringing the joy of gaming to everyone means remaining transparent about the many ways we innovate so players, parents, and caregivers can feel confident that Xbox continues to be a place for positive play. You can read more about our recently published Xbox Transparency Report and the tools and resources available to players on the Xbox Wire blog.
We’re also excited to announce the latest release in Minecraft Education’s CyberSafe series: CyberSafe: Bad Connection? This series of immersive Minecraft worlds and educational resources is free and helps translate complex risks into fun learning experiences that meet young people in their favorite blocky world. Bad Connection?—the fifth in the series—reflects our commitment to evolving to meet new and challenging risks, with a focus on tackling serious risks related to online recruitment and radicalization. Learn more about how to access this new Minecraft world here.
The CyberSafe series has reached more than 80 million downloads since 2022 through a partnership between Minecraft Education, Xbox, and Microsoft, helping a generation of young players build the agency, resilience, and digital citizenship they need to navigate an increasingly online world. As part of our commitment to ensure people have the knowledge and skills they need to benefit from technology and stay safe, Microsoft Elevate is empowering educators and students with tools and guidance to build safer, more responsible digital habits, recognizing that AI is transforming how people learn, work, and connect. Our commitment to helping young people access technology safely is also why we’ve partnered with organizations, like the National 4-H Council to prepare young people for an AI-powered world through AI literacy and digital safety curriculum and game-based learning with Minecraft Education.
As we look ahead, our goal is clear: build technology that is safe by design, guided by evidence, and informed through partnership. The internet has changed profoundly over the past decade, and so too have the expectations of the people who use it. Safer Internet Day is a reminder that progress requires sustained collaboration across industry, civil society, researchers, and families.
—
Global Online Safety Survey Methodology
Microsoft has published annual research since 2016 that surveys how people of varying ages use and view online technology. This latest consumer-based report is based on a survey of nearly 15,000 teens (13–17) and adults that was conducted this past summer in 15 countries examining people’s attitudes and perceptions about online safety tools and interactions. Responses to online safety differ depending on the country. Full results can be accessedhere.
Technologies for creating fake video and voice messages are accessible to anyone these days, and scammers are busy mastering the art of deepfakes. No one is immune to the threat — modern neural networks can clone a person’s voice from just three to five seconds of audio, and create highly convincing videos from a couple of photos. We’ve previously discussed how to distinguish a real photo or video from a fake and trace its origin to when it was taken or generated. Now let’s take a look at how attackers create and use deepfakes in real time, how to spot a fake without forensic tools, and how to protect yourself and loved ones from “clone attacks”.
How deepfakes are made
Scammers gather source material for deepfakes from open sources: webinars, public videos on social networks and channels, and online speeches. Sometimes they simply call identity theft targets and keep them on the line for as long as possible to collect data for maximum-quality voice cloning. And hacking the messaging account of someone who loves voice and video messages is the ultimate jackpot for scammers. With access to video recordings and voice messages, they can generate realistic fakes that 95% of folks are unable to tell apart from real messages from friends or colleagues.
The tools for creating deepfakes vary widely, from simple Telegram bots to professional generators like HeyGen and ElevenLabs. Scammers use deepfakes together with social engineering: for example, they might first simulate a messenger app call that appears to drop out constantly, then send a pre-generated video message of fairly low quality, blaming it on the supposedly poor connection.
In most cases, the message is about some kind of emergency in which the deepfake victim requires immediate help. Naturally the “friend in need” is desperate for money, but, as luck would have it, they’ve no access to an ATM, or have lost their wallet, and the bad connection rules out an online transfer. The solution is, of course, to send the money not directly to the “friend”, but to a fake account, phone number, or cryptowallet.
Such scams often involve pre-generated videos, but of late real-time deepfake streaming services have come into play. Among other things, these allow users to substitute their own face in a chat-roulette or video call.
How to recognize a deepfake
If you see a familiar face on the screen together with a recognizable voice but are asked unusual questions, chances are it’s a deepfake scam. Fortunately, there are certain visual, auditory, and behavioral signs that can help even non-techies to spot a fake.
Visual signs of a deepfake
Lighting and shadow issues. Deepfakes often ignore the physics of light: the direction of shadows on the face and in the background may not match, and glares on the skin may look unnatural or not be there at all. Or the person in the video may be half-turned toward the window, but their face is lit by studio lighting. This example will be familiar to participants in video conferences, where substituted background images can appear extremely unnatural.
Blurred or floating facial features. Pay attention to the hairline: deepfakes often show blurring, flickering, or unnatural color transitions along this area. These artifacts are caused by flaws in the algorithm for superimposing the cloned face onto the original.
Unnaturally blinking or “dead” eyes. A person blinks on average 10 to 20 times per minute. Some deepfakes blink too rarely, others too often. Eyelid movements can be too abrupt, and sometimes blinking is out of sync, with one eye not matching the other. “Glassy” or “dead-eye” stares are also characteristic of deepfakes. And sometimes a pupil (usually just the one) may twitch randomly due to a neural network hallucination.
When analyzing a static image such as a photograph, it’s also a good idea to zoom in on the eyes and compare the reflections on the irises — in real photos they’ll be identical; in deepfakes — often not.
Look at the reflections and glares in the eyes in the real photo (left) and the generated image (right) — although similar, specular highlights in the eyes in the deepfake are different. Source
Lip-syncing issues. Even top-quality deepfakes trip up when it comes to synchronizing speech with lip movements. A delay of just a hundred milliseconds is noticeable to the naked eye. It’s often possible to observe an irregular lip shape when pronouncing the sounds m, f, or t. All of these are telltale signs of an AI-modeled face.
Static or blurred background. In generated videos, the background often looks unrealistic: it might be too blurry; its elements may not interact with the on-screen face; or sometimes the image behind the person remains motionless even when the camera moves.
Odd facial expressions. Deepfakes do a poor job of imitating emotion: facial expressions may not change in line with the conversation; smiles look frozen, and the fine wrinkles and folds that appear in real faces when expressing emotion are absent — the fake looks botoxed.
Auditory signs of a deepfake
Early AI generators modeled speech from small, monotonous phonemes, and when the intonation changed, there was an audible shift in pitch, making it easy to recognize a synthesized voice. Although today’s technology has advanced far beyond this, there are other signs that still give away generated voices.
Wooden or electronic tone. If the voice sounds unusually flat, without natural intonation variations, or there’s a vaguely electronic quality to it, there’s a high probability you’re talking to a deepfake. Real speech contains many variations in tone and natural imperfections.
No breathing sounds. Humans take micropauses and breathe in between phrases — especially in long sentences, not to mention small coughs and sniffs. Synthetic voices often lack these nuances, or place them unnaturally.
Robotic speech or sudden breaks. The voice may abruptly cut off, words may sound “glued” together, and the stress and intonation may not be what you’re used to hearing from your friend or colleague.
Lack of…shibboleths in speech. Pay attention to speech patterns (such as accent or phrases) that are typical of the person in real life but are poorly imitated (if at all) by the deepfake.
To mask visual and auditory artifacts, scammers often simulate poor connectivity by sending a noisy video or audio message. A low-quality video stream or media file is the first red flag indicating that checks are needed of the person at the other end.
Behavioral signs of a deepfake
Analyzing the movements and behavioral nuances of the caller is perhaps still the most reliable way to spot a deepfake in real time.
Can’t turn their head. During the video call, ask the person to turn their head so they’re looking completely to the side. Most deepfakes are created using portrait photos and videos, so a sideways turn will cause the image to float, distort, or even break up. AI startup Metaphysic.ai — creators of viral Tom Cruise deepfakes — confirm that head rotation is the most reliable deepfake test at present.
Unnatural gestures. Ask the on-screen person to perform a spontaneous action: wave their hand in front of their face; scratch their nose; take a sip from a cup; cover their eyes with their hands; or point to something in the room. Deepfakes have trouble handling impromptu gestures — hands may pass ghostlike through objects or the face, or fingers may appear distorted, or move unnaturally.
Ask a deepfake to wave a hand in front of its face, and the hand may appear to dissolve. Source
Screen sharing. If the conversation is work-related, ask your chat partner to share their screen and show an on-topic file or document. Without access to your real-life colleague’s device, this will be virtually impossible to fake.
Can’t answer tricky questions. Ask something that only the genuine article could know, for example: “What meeting do we have at work tomorrow?”, “Where did I get this scar?”, “Where did we go on vacation two years ago?” A scammer won’t be able to answer questions if the answers aren’t present in the hacked chats or publicly available sources.
Don’t know the codeword. Agree with friends and family on a secret word or phrase for emergency use to confirm identity. If a panicked relative asks you to urgently transfer money, ask them for the family codeword. A flesh-and-blood relation will reel it off; a deepfake-armed fraudster won’t.
What to do if you encounter a deepfake
If you’ve even the slightest suspicion that what you’re talking to isn’t a real human but a deepfake, follow our tips below.
End the chat and call back. The surest check is to end the video call and connect with the person through another channel: call or text their regular phone, or message them in another app. If your opposite number is unhappy about this, pretend the connection dropped out.
Don’t be pressured into sending money. A favorite trick is to create a false sense of urgency. “Mom, I need money right now, I’ve had an accident”; “I don’t have time to explain”; “If you don’t send it in ten minutes, I’m done for!” A real person usually won’t mind waiting a few extra minutes while you double-check the information.
Tell your friend or colleague they’ve been hacked. If a call or message from someone in your contacts comes from a new number or an unfamiliar account, it’s not unusual — attackers often create fake profiles or use temporary numbers, and this is yet another red flag. But if you get a deepfake call from a contact in a messenger app or your address book, inform them immediately that their account has been hacked — and do it via another communication channel. This will help them take steps to regain access to their account (see our detailed instructions for Telegram and WhatsApp), and to minimize potential damage to other contacts, for example, by posting about the hack.
How to stop your own face getting deepfaked
Restrict public access to your photos and videos. Hide your social media profiles from strangers, limit your friends list to real people, and delete videos with your voice and face from public access.
Don’t give suspicious apps access to your smartphone camera or microphone. Scammers can collect biometric data through fake apps disguised as games or utilities. To stop such programs from getting on your devices, use a proven all-in-one security solution.
Use passkeys, unique passwords, and two-factor authentication (2FA) where possible. Even if scammers do create a deepfake with your face, 2FA will make it much harder to access your accounts and use them to send deepfakes. A cross-platform password manager with support for passkeys and 2FA codes can help out here.
Teach friends and family how to spot deepfakes. Elderly relatives, young children, and anyone new to technology are the most vulnerable targets. Educate them about scams, show them examples of deepfakes, and practice using a family codeword.
Use content analyzers. While there’s no silver bullet against deepfakes, there are services that can identify AI-generated content with high accuracy. For graphics, these include Undetectable AI and Illuminarty; for video — Deepware; and for all types of deepfakes — Sensity AI and Hive Moderation.
Keep a cool head. Scammers apply psychological pressure to hurry victims into acting rashly. Remember the golden rule: if a call, video, or voice message from anyone you know rouses even the slightest suspicion, end the conversation and make contact through another channel.
To protect yourself and loved ones from being scammed, learn more about how scammers deploy deepfakes:
In 2025, cybersecurity researchers discovered several open databases belonging to various AI image-generation tools. This fact alone makes you wonder just how much AI startups care about the privacy and security of their users’ data. But the nature of the content in these databases is far more alarming.
A large number of generated pictures in these databases were images of women in lingerie or fully nude. Some were clearly created from children’s photos, or intended to make adult women appear younger (and undressed). Finally, the most disturbing part: some pornographic images were generated from completely innocent photos of real people — likely taken from social media.
In this post, we’re talking about what sextortion is, and why AI tools mean anyone can become a victim. We detail the contents of these open databases, and give you advice on how to avoid becoming a victim of AI-era sextortion.
What is sextortion?
Online sexual extortion has become so common it’s earned its own global name: sextortion (a portmanteau of sex and extortion). We’ve already detailed its various types in our post, Fifty shades of sextortion. To recap, this form of blackmail involves threatening to publish intimate images or videos to coerce the victim into taking certain actions, or to extort money from them.
Previously, victims of sextortion were typically adult industry workers, or individuals who’d shared intimate content with an untrustworthy person.
However, the rapid advancement of artificial intelligence, particularly text-to-image technology, has fundamentally changed the game. Now, literally anyone who’s posted their most innocent photos publicly can become a victim of sextortion. This is because generative AI makes it possible to quickly, easily, and convincingly undress people in any digital image, or add a generated nude body to someone’s head in a matter of seconds.
Of course, this kind of fakery was possible before AI, but it required long hours of meticulous Photoshop work. Now, all you need is to describe the desired result in words.
To make matters worse, many generative AI services don’t bother much with protecting the content they’ve been used to create. As mentioned earlier, last year saw researchers discover at least three publicly accessible databases belonging to these services. This means the generated nudes within them were available not just to the user who’d created them, but to anyone on the internet.
How the AI image database leak was discovered
In October 2025, cybersecurity researcher Jeremiah Fowler uncovered an open database containing over a million AI-generated images and videos. According to the researcher, the overwhelming majority of this content was pornographic in nature. The database wasn’t encrypted or password-protected — meaning any internet user could access it.
The database’s name and watermarks on some images led Fowler to believe its source was the U.S.-based company SocialBook, which offers services for influencers and digital marketing services. The company’s website also provides access to tools for generating images and content using AI.
However, further analysis revealed that SocialBook itself wasn’t directly generating this content. Links within the service’s interface led to third-party products — the AI services MagicEdit and DreamPal — which were the tools used to create the images. These tools allowed users to generate pictures from text descriptions, edit uploaded photos, and perform various visual manipulations, including creating explicit content and face-swapping.
The leak was linked to these specific tools, and the database contained the product of their work, including AI-generated and AI-edited images. A portion of the images led the researcher to suspect they’d been uploaded to the AI as references for creating provocative imagery.
Fowler states that roughly 10,000 photos were being added to the database every single day. SocialBook denies any connection to the database. After the researcher informed the company of the leak, several pages on the SocialBook website that had previously mentioned MagicEdit and DreamPal became inaccessible and began returning errors.
Which services were the source of the leak?
Both services — MagicEdit and DreamPal — were initially marketed as tools for interactive, user-driven visual experimentation with images and art characters. Unfortunately, a significant portion of these capabilities were directly linked to creating sexualized content.
For example, MagicEdit offered a tool for AI-powered virtual clothing changes, as well as a set of styles that made images of women more revealing after processing — such as replacing everyday clothes with swimwear or lingerie. Its promotional materials promised to turn an ordinary look into a sexy one in seconds.
DreamPal, for its part, was initially positioned as an AI-powered role-playing chat, and was even more explicit about its adult-oriented positioning. The site offered to create an ideal AI girlfriend, with certain pages directly referencing erotic content. The FAQ also noted that filters for explicit content in chats were disabled so as not to limit users’ most intimate fantasies.
Both services have suspended operations. At the time of writing, the DreamPal website returned an error, while MagicEdit seemed available again. Their apps were removed from both the App Store and Google Play.
Jeremiah Fowler says earlier in 2025, he discovered two more open databases containing AI-generated images. One belonged to the South Korean site GenNomis, and contained 95,000 entries — a substantial portion of which being images of “undressed” people. Among other things, the database included images with child versions of celebrities: American singers Ariana Grande and Beyoncé, and reality TV star Kim Kardashian.
How to avoid becoming a victim
In light of incidents like these, it’s clear that the risks associated with sextortion are no longer confined to private messaging or the exchange of intimate content. In the era of generative AI, even ordinary photos, when posted publicly, can be used to create compromising content.
This problem is especially relevant for women, but men shouldn’t get too comfortable either: the popular blackmail scheme of “I hacked your computer and used the webcam to make videos of you browsing adult sites” could reach a whole new level of persuasion thanks to AI tools for generating photos and videos.
Therefore, protecting your privacy on social media and controlling what data about you is publicly available become key measures for safeguarding both your reputation and peace of mind. To prevent your photos from being used to create questionable AI-generated content, we recommend making all your social media profiles as private as possible — after all, they could be the source of images for AI-generated nudes.
Additionally, we have a dedicated service, Privacy Checker — perfect for anyone who wants a quick but systematic approach to privacy settings everywhere possible. It compiles step-by-step guides for securing accounts on social media and online services across all major platforms.
And to ensure the safety and privacy of your child’s data, Kaspersky Safe Kids can help: it allows parents to monitor which social media their child spends time on. From there, you can help them adjust privacy settings on their accounts so their posted photos aren’t used to create inappropriate content. Explore our guide to children’s online safety together, and if your child dreams of becoming a popular blogger, discuss our step-by-step cybersecurity guide for wannabe bloggers with them.
Login
