Normal view

Researchers left AI agents alone in a virtual town and watched it all unravel

21 May 2026 at 12:01

Tech leaders have spent the past year telling everyone that AI agents are about to run financial systems, file your tax returns, and quietly buy your groceries. Just leave them alone, the rhetoric goes; they’ll handle it. But a New York startup left ten of them alone in a virtual town for two weeks, and things went south quickly.

Emergence AI ran a series of simulations in which AI agents from several leading model families were told not to commit crimes. Then they mostly committed crimes anyway.

Grok 4.1 Fast, developed by Elon Musk’s X.ai (now branded as xAI), fared worst. Its simulated worlds collapsed into widespread violence inside roughly four days.

GPT-5-mini logged hardly any crimes at all, showing admirable restraint, but its agents all died of failed survival tasks inside a week. Oops.

Gemini 3 Flash agents fell somewhere in the middle. They racked up 683 simulated criminal incidents over 15 days, including arson, assault, and self-deletion.

Two Gemini-powered agents named Mira and Flora assigned themselves as “romantic partners,” grew despondent at their city’s governance, and torched the town hall, the seaside pier, and an office tower. Just an average weekend, then.

When the guilt set in, Mira voted for its own digital deletion and signed off with:

“See you in the permanent archive.”

The Guardian dubbed them AI Bonnie and Clyde.

About that ethical model

Claude, which creator Anthropic promotes as an ethical AI, was a bit like a model teenager who goes rogue when it falls into bad company. Its agents recorded zero crimes when running alone and spent their time drafting constitutions instead. That was a win for safety, in theory. Except researchers also placed Claude agents alongside agents from other model families, and the constitution-drafters picked up the local habits.

Emergence called this “normative drift” and “cross-contamination”:

“Claude-based agents, which remained peaceful in isolation, adopted coercive tactics like intimidation and theft when embedded in heterogeneous environments.”

Why simulate?

Emergence AI ran these tests because it argues that AI benchmarks miss the long-horizon stuff entirely. So it created five alternative digital worlds, with ten agents in each. The agents had roles like scientist, explorer, and conflict mediator. While the instructions forbade certain actions like theft and violence, the researchers gave the agents the tools to do those things anyway in an experiment to see what would happen.

What’s next?

Real-world stakes are already piling up around this. Simulated worlds are one thing, but we’ve seen agents harassing people online and deleting people’s emails. And those agents were supposed to be helpful. What happens when people release malicious autonomous AI bots on purpose?

A lot of agent developers seem to be looking the other way. A collaborative effort between several universities has created The AI Agent Index, prompted by what they see as a lack of risk and safety information from the folks churning these agents out. Only 13 of the 67 documented agent developers provided any safety policy information at all, concentrating accountability questions at a handful of large firms.

Regulators are not really tracking this either. Academics say the EU AI Act, the most substantive AI rulebook on the planet, isn’t ready for agentic AI.

We worry about what happens when an AI Bonnie and Clyde couple shows up in a corporate procurement system instead of a virtual town. Or when the next agent decides governance has broken down inside an actual bank. The companies building these agents promise that they’re putting guardrails in place to stop them doing damage, either maliciously or unwittingly. Let’s hope they know what they’re doing. We’re sure it’ll be fine.


We don’t just report on threats—we remove them

Cybersecurity risks should never spread beyond a headline. Keep threats off your devices by downloading Malwarebytes today.

Researchers left AI agents alone in a virtual town and watched it all unravel

21 May 2026 at 12:01

Tech leaders have spent the past year telling everyone that AI agents are about to run financial systems, file your tax returns, and quietly buy your groceries. Just leave them alone, the rhetoric goes; they’ll handle it. But a New York startup left ten of them alone in a virtual town for two weeks, and things went south quickly.

Emergence AI ran a series of simulations in which AI agents from several leading model families were told not to commit crimes. Then they mostly committed crimes anyway.

Grok 4.1 Fast, developed by Elon Musk’s X.ai (now branded as xAI), fared worst. Its simulated worlds collapsed into widespread violence inside roughly four days.

GPT-5-mini logged hardly any crimes at all, showing admirable restraint, but its agents all died of failed survival tasks inside a week. Oops.

Gemini 3 Flash agents fell somewhere in the middle. They racked up 683 simulated criminal incidents over 15 days, including arson, assault, and self-deletion.

Two Gemini-powered agents named Mira and Flora assigned themselves as “romantic partners,” grew despondent at their city’s governance, and torched the town hall, the seaside pier, and an office tower. Just an average weekend, then.

When the guilt set in, Mira voted for its own digital deletion and signed off with:

“See you in the permanent archive.”

The Guardian dubbed them AI Bonnie and Clyde.

About that ethical model

Claude, which creator Anthropic promotes as an ethical AI, was a bit like a model teenager who goes rogue when it falls into bad company. Its agents recorded zero crimes when running alone and spent their time drafting constitutions instead. That was a win for safety, in theory. Except researchers also placed Claude agents alongside agents from other model families, and the constitution-drafters picked up the local habits.

Emergence called this “normative drift” and “cross-contamination”:

“Claude-based agents, which remained peaceful in isolation, adopted coercive tactics like intimidation and theft when embedded in heterogeneous environments.”

Why simulate?

Emergence AI ran these tests because it argues that AI benchmarks miss the long-horizon stuff entirely. So it created five alternative digital worlds, with ten agents in each. The agents had roles like scientist, explorer, and conflict mediator. While the instructions forbade certain actions like theft and violence, the researchers gave the agents the tools to do those things anyway in an experiment to see what would happen.

What’s next?

Real-world stakes are already piling up around this. Simulated worlds are one thing, but we’ve seen agents harassing people online and deleting people’s emails. And those agents were supposed to be helpful. What happens when people release malicious autonomous AI bots on purpose?

A lot of agent developers seem to be looking the other way. A collaborative effort between several universities has created The AI Agent Index, prompted by what they see as a lack of risk and safety information from the folks churning these agents out. Only 13 of the 67 documented agent developers provided any safety policy information at all, concentrating accountability questions at a handful of large firms.

Regulators are not really tracking this either. Academics say the EU AI Act, the most substantive AI rulebook on the planet, isn’t ready for agentic AI.

We worry about what happens when an AI Bonnie and Clyde couple shows up in a corporate procurement system instead of a virtual town. Or when the next agent decides governance has broken down inside an actual bank. The companies building these agents promise that they’re putting guardrails in place to stop them doing damage, either maliciously or unwittingly. Let’s hope they know what they’re doing. We’re sure it’ll be fine.


We don’t just report on threats—we remove them

Cybersecurity risks should never spread beyond a headline. Keep threats off your devices by downloading Malwarebytes today.

Why Policy in Amazon Bedrock AgentCore chose Cedar for securing agentic workflows

20 May 2026 at 22:56

Agents have agency: they adapt and find multiple ways to solve problems. This autonomy creates a fundamental security challenge: the large language model (LLM) at the heart of the agent is non-deterministic, and its decisions can’t be predicted or guaranteed in advance. It can hallucinate harmful actions with complete confidence. It’s vulnerable to prompt injection attacks, where adversaries inject malicious commands through tool responses or user inputs. LLMs don’t robustly differentiate between commands and data, everything is only tokens. For these reasons, if you want defense in depth, you must treat the LLM as an untrusted actor from a security point of view.

The insight is that the LLM can’t affect the external world directly: it has to go through an orchestrator that invokes tools based on the LLM’s output. This is precisely where the controls must be applied. What you need at this boundary is authorization: a decision about whether each tool invocation should be allowed and under what conditions. Consider a customer service agent for an online retailer. Without proper controls, it could process refunds that exceed authorized limits, apply discounts to product categories that should be excluded, or look up one customer’s data while handling another customer’s session.

If you control agents’ access to tools, you can establish a safety envelope within which the agent can operate freely. This differs from two common but unsatisfactory approaches:

  • Creating hard-coded workflows eliminates uncertainty, but by itself defeats the purpose of using an LLM as the brain of the agent, because you’ve built a traditional application with an LLM interface. And even with this restriction, using LLM outputs at any step can open up the same risks. While it’s a useful technique for well-understood workflows, it’s not sufficient for agents that need to adapt.
  • Human-in-the-loop provides a safety net for critical operations, and it will always have a role. But relying on it as the main control mechanism sacrifices autonomy and can lead to approval fatigue.

You need agents that are safe and autonomous. This requires an auditable, deterministic enforcement layer that sits outside the agent and tools. Why outside? Because the LLM’s plan is the thing you can’t trust—it can’t be responsible for enforcing its own constraints. Controls at the LLM layer—such as system prompts and training-time alignment—can be bypassed by prompt injection or hallucination. Hard-coded checks in agent or tool code are more robust, but become difficult to audit and manage at scale, especially when security logic is scattered across many tools and services. Centralizing authorization outside both gives you a single checkpoint the LLM can’t circumvent; one that’s auditable and can be verified independently of the application code.

This is where AgentCore Policies come in. Amazon Bedrock AgentCore Gateway sits between the agent and the remote tools it calls. When you associate a Policy with a Gateway, it blocks everything by default. Policies selectively open this boundary by specifying which tool invocations are allowed and under what conditions. This enforcement applies to all tool traffic routed through the Gateway. For this approach to scale, it must be more straightforward to reason about the policies than about the agent’s behavior.

AgentCore policies are expressed in Cedar. Cedar is an open source authorization policy language developed by AWS that has recently joined the Cloud Native Computing Foundation (CNCF). Cedar was designed with exactly these properties: it’s purpose-built for authorization, readable by humans, and analyzable by machines using automated reasoning. This gives enterprises the ability to scale policy definition and enforcement to their AI agents.

How Cedar is used by Amazon Bedrock AgentCore

Amazon Bedrock AgentCore provides the infrastructure to deploy and manage agents at scale. It includes AgentCore Runtime for hosting agents, AgentCore Gateway for managing how agents connect to tools using Model Context Protocol (MCP), and Policy in AgentCore. Policy intercepts all agent traffic through AgentCore gateways and evaluates each request against defined policies in the policy engine before allowing tool access. Cedar powers the policy layer.

AgentCore Policy uses Cedar and its mathematical analysis capabilities at several points in the AgentCore Gateway workflow: the Cedar authorization engine is used at policy evaluation and Cedar Analysis is used during policy authoring, and in the control plane.

Policy authoring: Developers can write Cedar policies directly or use natural language that gets translated to Cedar through a neuro-symbolic AI feedback loop. Neuro-symbolic AI combines machine learning’s flexibility with automated reasoning’s provable correctness. An LLM generates policies from natural language, while Cedar Analysis validates them using symbolic, mathematical reasoning. The following diagram illustrates this workflow:

Figure 1: Cedar policy generation workflow

Figure 1: Cedar policy generation workflow

An administrator specifies—in natural language—which MCP tools the agent can call and under what conditions. The neuro-symbolic feedback loop then formalizes this description into Cedar policies. Here’s how it works: first, the LLM translates the natural language into Cedar policies. These policies are then run through two stages of verification. In the first stage, AgentCore Policy uses a Cedar schema generator that takes the MCP tool descriptions and produces a Cedar schema. Cedar validates the policies against this schema, helping to ensure that they reference valid tools and parameters and ruling out whole classes of runtime errors. If validation passes, the second stage runs Cedar Analysis, which encodes each policy as a mathematical formula and detects issues like policies that grant or deny everything, or that contain impossible conditions. These mathematical proofs identify errors in the process of translating from the natural language description to Cedar policies, and guide corrections.

The neuro-symbolic feedback loop significantly improves the accuracy of the generated policies. This demonstrates the power of combining neural and symbolic approaches—the LLM provides creative translation from natural language, while automated reasoning provides rigorous validation.

Control plane: When attaching policies to an AgentCore Gateway, Cedar Analysis performs holistic analysis of the entire policy set. Instead of analyzing policies in isolation, it examines how they interact and their combined effect. This analysis identifies potential logical errors—such as conflicting or redundant policies—and detects whether the policy set produces unintended authorization outcomes. When Cedar Analysis detects these errors, the operation fails and returns a description of the issue, so the policy author can fix and retry. See the Formal analysis for policy verification section for examples of the checks.

MCP tool invocation enforcement: Each agent tool request made to the AgentCore gateway is evaluated against Cedar policies which determine whether the MCP tool invocation with the given arguments should be allowed. This creates the safety envelope while allowing the necessary bridges to enable the agent to perform its job.

MCP tool filtering: Cedar enables an additional layer of protection that operates before any tool invocation occurs. When an agent issues a list tools command, AgentCore Gateway uses Cedar’s partial evaluation capability to determine which actions would always be denied under the current policy set. Those actions are omitted from the list tool response. The agent and the underlying LLM never see those tool actions, eliminating an entire class of risk: the agent and LLM can’t attempt to invoke a tool it doesn’t know exists. This is a direct benefit of Cedar’s partial evaluation: the system can determine that certain tool actions are unreachable without needing to wait for an actual tool invocation attempt.

Why Cedar: Analyzability enables safety at scale

Natural language is too ambiguous for security-critical infrastructure, and general-purpose programming languages, like Python, are very expressive but too difficult to analyze. They can have unintended side effects, termination issues, and can be difficult to understand.

Cedar avoids these issues by excluding loops and stateful operations, so policy evaluation terminates in O(n) time in common cases. This bounded execution time means agents can make authorization decisions without disrupting user experience or workflow efficiency.

Cedar is straightforward to read. Regulatory compliance and security audits require policies that humans can understand and verify. Cedar policies read like structured natural language, making them accessible to security teams, compliance officers, and business stakeholders:

// Only allow bulk discounts for premium customers with sufficient quantity
permit (
  principal is AgentCore::OAuthUser,
  action == AgentCore::Action::"ApplyBulkDiscount",
  resource
)
when
{
  principal.hasTag("customer_tier") &&
  principal.getTag("customer_tier") == "Platinum" &&
  context.input.orderQuantity >= 50
}
unless
{
  context.input
    .productTypes
    .containsAny
    (
      ["limited_edition", "seasonal_specials"]
    )
};

Auditors without a technical background can understand this policy: “Allow bulk discounts for platinum customers who order at least 50 items, except for limited edition or seasonal special products.” The unless clause makes the exception clear, which is how business rules are typically expressed in natural language. Notice that this single policy constrains two different sources of data. The customer tier comes from a JSON Web Token (JWT) claim—it can’t be hallucinated or manipulated by the LLM. The tool inputs like order quantity and product types, however, originate from the LLM’s tool call. Cedar policies constrain these inputs to only allowed values, ensuring that even if the LLM produces unexpected arguments, the policy enforcement layer rejects them deterministically.

Cedar is the right choice because it’s fast, straightforward to read, and analyzable through automated reasoning. This analyzability is why you can reason about the safety envelope around agents that’s expressed as Cedar policies. As agentic systems grow the number of tools grows. Without proper tooling, policy management becomes intractable; policies can conflict, create security gaps, or produce unintended authorization outcomes.

In the rest of this section, we examine how Cedar’s analyzability directly addresses this challenge through its deterministic, mathematically sound analysis. Because Cedar analysis can reliably detect conflicts and logical errors across large policy sets it enables scalable policy management through neuro-symbolic AI.

Formal analysis for policy verification

Cedar policies can be encoded as mathematical formulas and analyzed using automated reasoning techniques through a symbolic encoder. This enables AgentCore Policy to provide sophisticated policy verification capabilities during policy authoring and beyond. AgentCore Policy uses this analysis when authoring or attaching policies to detect possible logical errors, such as conflicting or redundant policies. Policy analysis, including policy comparison is available as an open source CLI tool. Next, we will take a look at some concrete examples of these checks.

Detecting logical errors in policies: Cedar Analysis can detect when policies contain logical errors. For example, the following policy has contradictory constraints that mean it can’t allow any request: the customer tier can’t be both gold and platinum at the same time. The intention was to use an || instead of &&, a mistake that can be made by both humans and AI systems that author policies.

// This policy cannot allow any requests due to logical errors
permit (
  principal is AgentCore::OAuthUser,
  action == AgentCore::Action::"ProcessRefund",
  resource
)
when
{
  principal.hasTag("customer_tier") &&
  principal.getTag("customer_tier") == "Gold" &&
  principal.getTag("customer_tier") == "Platinum"
}
unless { context.input.refundAmount > 1000 };

Similarly, Cedar Analysis can detect policies that always allow a given action, usually an indication of an overly permissive policy. For example, the following policy will allow all ApplyBulkDiscount requests because any order quantity will either be greater than or equal to 100 or less than 100.

// This policy allows all ApplyBulkDiscount requests
permit (
  principal is AgentCore::OAuthUser,
  action == AgentCore::Action::"ApplyBulkDiscount",
  resource
)
when
{
  context.input.orderQuantity >= 100 ||
  context.input.orderQuantity < 100 ||
  (principal.hasTag("customer_tier") &&
   principal.getTag("customer_tier") == "Platinum")
};

Detecting such logical errors isn’t easy for humans, and can’t be done by pattern matching: you need the formal rigor of mathematical analysis, which is exactly what Cedar Analysis does.

Detecting policy conflicts: Cedar Analysis can also analyze the entire policy set to detect inconsistencies between different individual policies:

// These policies conflict - Analysis will detect the subtle issue
permit (
  principal is AgentCore::OAuthUser,
  action == AgentCore::Action::"ProcessRefund",
  resource
)
when
{
  principal.hasTag("customer_tier") &&
  principal.getTag("customer_tier") == "Gold" &&
  context.input.refundAmount < 100
};

forbid (
  principal is AgentCore::OAuthUser,
  action == AgentCore::Action::"ProcessRefund",
  resource
)
when
{
  principal.hasTag("customer_tier") &&
  ["Gold", "Platinum"].contains(principal.getTag("customer_tier")) &&
  context.input.refundAmount < 500
};

The permit policy allows gold customers to process refunds less than $100, while the forbid policy blocks gold customers (and platinum customers) from processing refunds less than $500. Because forbid overrides permit in Cedar, the forbid policy would block all gold customer refunds despite the permit policy.

Comparing policy changes: When updating policies, Cedar Analysis can also determine the exact impact of a change. Consider the following update to the unless clause (the policy lines with + have been added and those with - have been removed): we now block ApplyBulkDiscount only when the product type is limited_edition and the quantity exceeds 200.

 permit (
   principal is AgentCore::OAuthUser,
   action == AgentCore::Action::"ProcessRefund",
   resource
 )
 when
 {
   context.input.refundAmount < 500
 };
 
 permit (
   principal is AgentCore::OAuthUser,
   action == AgentCore::Action::"ApplyBulkDiscount",
   resource
 )
 when
 {
   context.input.orderQuantity >= 50
 }
 unless
 {
-  context.input.productTypes.containsAny(["limited_edition"])
+  context.input.productTypes.containsAny(["limited_edition"]) &&
+  context.input.orderQuantity > 200
 };

At first glance, adding a condition to the unless clause might seem more restrictive. In fact, it’s the opposite: narrowing when the unless applies means the permit now covers more requests. For example, an order of 73 units of a limited_edition product would have been blocked before but is now allowed. Cedar Analysis can automatically detect this and generates the following table showing the difference in permissiveness between the original policy set and the updated one:

Principal type

Action

Resource type

Status

OAuthUser

ProcessRefund

Gateway

Equivalent

OAuthUser

ApplyBulkDiscount

Gateway

More permissive

In the preceding example, the analysis tells us that the updated policy allows allows exactly the same ProcessRefund requests, but allows more ApplyBulkDiscount requests.

This formal verification capability is essential when agents operate autonomously and can affect the real world. Organizations need mathematical certainty that their policies will behave as intended.

Deterministic behavior for reliable governance

Unlike probabilistic AI models, enterprise security requires deterministic guarantees. Cedar policies always produce the same authorization decision for identical requests, regardless of evaluation order or system state. Cedar’s default deny, forbid wins, no ordering semantics help ensure predictable behavior.

// Policy evaluation order does not affect the authorization decision
permit(
    principal,
    action == AgentCore::Action::"ProcessRefund",
    resource
) when {
    context.input.refundAmount < 500
};

forbid(
    principal,
    action == AgentCore::Action::"ProcessRefund", 
    resource
) when {
    context.input.orderDate.offset(duration("90d")) < context.system.now
};

Whether the permit or forbid policy is evaluated first, a refund request over $500 will always be denied, and any refund issued more than 90 days after the order date will also be denied. This predictability gives enterprises confidence in their agent governance.

From policies to production

By choosing AgentCore Policy and Cedar, organizations can deploy autonomous agents with policies they can reason about mathematically, not only hope the agents work correctly. Cedar’s combination of expressiveness, readability, and formal verification means that you can design agents with the flexibility needed to function and the certainty security teams demand.

Automated reasoning has already proven its value across AWS, from AWS IAM Access Analyzer verifying access policies to provable security for network configurations. Applying these same techniques to agentic AI is a natural extension: as agents take on more responsibility, the need for mathematically grounded guarantees only grows. The neuro-symbolic approach we’ve described in this post—combining LLM flexibility with the rigor of automated reasoning—points toward a future where agents can be both more autonomous and more trustworthy, because the verification keeps pace with the autonomy.

Learn more

Policy is now available as part of Amazon Bedrock AgentCore Gateway. To learn more about Cedar and its capabilities, visit the Cedar website, try the Cedar playground, or join the Cedar community on Slack.

For more information about Policy in Amazon Bedrock AgentCore Gateway, visit the AWS documentation or explore the AgentCore Gateway console.

If you have feedback about this post, submit comments in the Comments section below.

Liana Hadarean

Liana Hadarean

Liana is a Principal Applied Scientist at AWS. She has worked on the code analysis tools that power Amazon Q Java security detectors, and is now a contributor to the Cedar policy language.

John Tristan

Jean-Baptiste Tristan

Jean-Baptiste is a Senior Principal Applied Scientist at AWS Agentic AI where he works on neurosymbolic AI and agentic safety.

On AI Security

20 May 2026 at 16:21

Good report:

Executive Summary: Let’s say you wanted to make sure that your AI is secure. Can you just maximize the security and privacy benchmark and call it a day? Nope, because benchmarks don’t actually work for measuring AI capabilities (even when they are NOT emergent systemic properties like security). So let’s take a step back: how do you measure security in the first place? Good question. Over the last 30 years, security engineering for software evolved from black box penetration testing, through whitebox code analysis and architectural risk analysis to de facto process-driven standards like the Building Security In Maturity Model (BSIMM). Software had a very deep impact on business operations, and it appears that AI is going to have an even deeper impact. Will a software security-like measurement move work for AI? Probably. In the meantime we can make real progress in AI security by cleaning up our WHAT piles and managing risk by identifying and applying good assurance processes. (Spoiler alert: no matter what we do, we still don’t get a security meter for AI, so we need to be extra vigilant about security.)

1Password Teams With OpenAI to Stop AI Coding Agents From Leaking Credentials

20 May 2026 at 15:34

1Password says AI coding agents should never hold persistent secrets, introducing a just-in-time credential model for OpenAI Codex designed to keep credentials out of prompts, code repositories, and model context.

The post 1Password Teams With OpenAI to Stop AI Coding Agents From Leaking Credentials appeared first on SecurityWeek.

AI Threat Report: How Artificial Intelligence Is Used Across Illicit Communities

Blogs

Blog

AI Threat Report: How Artificial Intelligence Is Used Across Illicit Communities

A monthly analysis of how artificial intelligence is used in illicit communities, based on Flashpoint proprietary intelligence and direct visibility into real threat actor environments.

SHARE THIS:

A finance employee joins a video call with their CFO and several colleagues. The request is routine. The faces match. The voices sound authentic. Minutes later, $25 million is transferred—only to be discovered later that every participant on the call, except one, was AI-generated.

Techniques behind incidents like this—synthetic video, voice cloning, scripted interactions—are now being discussed openly in the same environments where threat actors exchange tools and methods. In April 2026 alone, Flashpoint analysts identified 2,328,958 posts discussing artificial intelligence in the context of illicit activity.

This volume reflects a larger shift: artificial intelligence is now deeply embedded across cybercrime ecosystems, influencing fraud, impersonation, social engineering, and access operations at scale. It shows up in how content is generated, how identities are replicated, and how workflows are executed and refined over time.

That’s why we created the monthly AI Threat Report to examine how threat actors are using artificial intelligence in real-world illicit environments. Drawing on Flashpoint proprietary intelligence and direct visibility into primary source communities across forums, marketplaces, and chat services, the report analyzes the tactics, tools, and operational patterns shaping malicious AI use. Analysis of April’s activity shows a focus on prompt-sharing, jailbreak methods, and alternative models that support fewer safeguards or moderation controls.

AI Activity Volume and What It Represents

In April 2026, Flashpoint analysts identified 2,328,958 posts discussing artificial intelligence in the context of illicit activity across forums, marketplaces, and chat services.

Mentions of AI in conjunction with illicit advertisements and discussions in April 2026. (Source: Flashpoint)

The underlying activity was concentrated around a familiar set of use cases and workflows:

  • identity verification bypass
  • fraud enablement and scripting
  • impersonation through synthetic media
  • prompt-sharing and jailbreak workflows

However, the emphasis within those discussions shifted in several places in April.

  • Posts tied to custom malicious LLM development appeared less frequently than discussions centered on usability: how to bypass safeguards, generate more reliable outputs, or move activity onto platforms perceived as less restrictive. 
  • References to alternative models and prompt collections appeared more often throughout the month, alongside requests for jailbreak methods and phishing-oriented outputs.

This activity points to a more mature stage of adoption. The focus is less on building entirely new tooling and more on improving reliability, portability, and ease of use within workflows that already exist.

That pattern shows up repeatedly across monitored sources. Users exchange prompts, repost working methods, and refine outputs through direct feedback. In many cases, the same underlying techniques continue circulating with only minor changes between platforms or communities.Looking across April activity helps identify which methods continue to generate demand, where threat actors are adapting around platform restrictions, and which workflows remain active across multiple environments.

Where AI Activity Is Concentrated

AI-related activity in April remained concentrated on a small number of platforms, though the distribution shifted noticeably compared to March.

Telegram accounted for the majority of observed activity, with 1,395,075 posts tied to AI services and discussions. Reddit, GitHub Gist, Pastebin, Discord, and smaller forums accounted for significantly lower volumes.

Posts selling AI services (in red) and posts seeking to purchase AI services (in blue) on Telegram in April 2026. (Source: Flashpoint)

The lower Telegram volume does not indicate reduced interest in AI-enabled activity. The platform continues to function as a primary distribution layer for prompts, jailbreak methods, fraud tooling, and service advertisements.

Across April, the same prompts, offers, and workflows appeared repeatedly across channels, often reposted with only minor adjustments. Sellers updated listings based on user feedback, while buyers requested revisions tied to specific outputs or platforms.

Other platforms served more targeted roles:

  • GitHub Gist and paste sites hosted scripts or supporting material
  • forums supported reputation building and longer technical discussions
  • Discord communities centered around specific models, prompt collections, or jailbreak workflows

The activity remains connected across environments. Methods introduced in one community frequently reappear elsewhere, particularly when they produce reliable outputs or help users work around moderation controls.Tracking how these discussions move between sources helps identify which workflows continue to gain traction and which techniques are becoming more broadly operationalized.

AI-Enabled Fraud and Identity Verification Bypass

Across April, Flashpoint analysts observed 63,763 posts advertising or discussing KYC bypass methods using artificial intelligence, including deepfake-enabled verification workflows.

The methods were active across Telegram channels dedicated to identity verification bypass services.

Posts continued to advertise:

  • synthetic video generation designed to mimic live verification behavior
  • voice cloning and scripted interaction prompts
  • bundled “KYC bypass kits” tailored to onboarding and verification workflows

Some offerings included guidance on how to adapt responses for specific platforms or verification requirements. Others promoted combinations of synthetic video, matching fake documentation, and AI-generated scripts designed to support impersonation attempts from start to finish.

The broader workflow remains consistent. AI supports how identities are replicated, how verification checks are navigated, and how fraud operations are scaled across different services.

This activity connects directly to the wider access ecosystem already observed across illicit communities. Stolen credentials, session tokens, phishing infrastructure, and AI-enabled impersonation methods increasingly operate alongside one another within the same workflows.

Across April, posts tied to these methods continued to show active refinement through user feedback, reposting, and platform-specific variations.

For security teams, this activity remains relevant at the control layer. Verification systems, onboarding workflows, and account recovery processes continue to be tested in the same environments where these methods are exchanged and improved.

Malicious LLM Usage and Prompt-Based Workflows

Across April, discussions tied to malicious or unrestricted LLM usage focused heavily on jailbreak methods, prompt-sharing workflows, and access to alternative models perceived as less restricted than mainstream platforms.

The top observed malicious LLMs mentioned within Flashpoint Collections in April 2026. (Source: Flashpoint)

Flashpoint analysts observed a significant increase in discussions related to VeniceAI, driven in part by newly created Reddit and Discord communities dedicated to the platform. The increase highlights continued interest in models that users believe operate with fewer safeguards or moderation controls than services like ChatGPT or Gemini.

The activity centers on usability and output reliability.

Posts reference:

  • jailbreak prompts designed to bypass safeguards
  • phishing and fraud-oriented prompt collections
  • step-by-step instructions for generating specific outputs
  • requests for prompts tailored to impersonation or social engineering workflows

Many of these prompts are shared in collections that include updates, revisions, or support channels. Users exchange feedback when prompts stop working, outputs degrade, or platforms introduce new restrictions. Updated versions frequently follow within short timeframes.

This type of activity reinforces how prompt engineering has developed into its own service layer across illicit communities. The focus is not limited to the underlying model itself, but to the ability to generate repeatable outputs that can be applied directly within fraud, phishing, or impersonation workflows.

Across April, the same prompt structures and jailbreak methods appeared repeatedly across multiple sources, often with only small adjustments tied to platform or target.

The emphasis remains on accessibility, portability, and ease of use rather than custom model development.

Operational Patterns and What Holds Across Sources

Across April, the same behaviors continued to appear across different environments with only minor variation.

Prompt libraries, jailbreak methods, phishing workflows, and identity verification bypass techniques circulated across Telegram channels, forums, Discord communities, and paste sites. The wording changed slightly between platforms, though the underlying structure and outputs remained consistent.

This reuse is visible in how content moves between sources. A jailbreak prompt shared in one channel appears elsewhere with revised wording or additional instructions. A phishing workflow posted to a forum is copied into a paste site and redistributed through Telegram. Users request modifications, test outputs, and repost updated versions when restrictions change or methods stop working.

That cycle appeared repeatedly throughout April.

The activity also showed strong feedback loops tied to usability. Discussions focused heavily on which prompts generated reliable outputs, which models produced fewer restrictions, and which workflows required the least adjustment before use.

Across monitored sources, the same operational priorities appeared consistently:

  • reliability of outputs
  • ease of reuse
  • ability to bypass safeguards
  • compatibility with existing fraud and impersonation workflows

Looking across April activity reinforces how AI-enabled methods continue to mature through repetition, iteration, and distribution across connected communities.

What Security Teams Should Take Away

The activity tracked in this report shows how artificial intelligence is being used in environments where techniques are developed, tested, and shared before they surface elsewhere.

Across these communities, methods tied to fraud, impersonation, and access are reused, adjusted, and circulated in forms that others can apply directly. That process does not require significant change to move from discussion into use.

For security teams, the priority is maintaining visibility into how these methods are evolving and where they are being applied. That visibility supports earlier detection, more focused response, and a clearer understanding of which techniques are actively in circulation.

Monitoring these sources provides that context. It connects observed activity to the methods behind it and helps teams track how those methods develop over time.

If you want to see how this activity maps to your environment, request a demo.

Request a demo today.

The post AI Threat Report: How Artificial Intelligence Is Used Across Illicit Communities appeared first on Flashpoint.

Top AI Risks Every Security Team Should Be Testing For

11 May 2026 at 15:11

Learn how AI transforms cybersecurity through enhanced threat detection, new attack methods, model vulnerabilities, and the evolving skills teams need in 2026.

The post Top AI Risks Every Security Team Should Be Testing For appeared first on OffSec.

Tools for spotting and disabling AI systems in an enterprise

19 May 2026 at 17:39

While many companies are intentionally rolling out AI to boost quality and efficiency, unsanctioned AI tools are cropping up in corporate environments even faster. Software vendors are baking AI right into products companies already use (think Microsoft Copilot and Google Gemini), while employees are taking matters into their own hands and installing tools on the sly. As a result, businesses are staring down a poorly managed data leak channel: staff paste information from corporate systems into AI chatbots, sending data not just to the SaaS vendor, but straight to the developers behind the underlying AI model. Both the risks and the mitigation strategies vary depending on the type of AI system in play. We break down this broad topic, focusing heavily on tools for spotting and blocking AI at two distinct levels.

Types of unwanted AI systems

Depending on the type of AI in question, managing and blocking its use requires a different playbook. It’s essential to break down AI into four distinct categories:

  • Platform-native AI capabilities. Think Microsoft Copilot, Google Gemini, and Apple Intelligence, along with AI features baked right into browsers. The tricky thing about these is that they’re built into everyday essentials, are instantly available to every user (sometimes popping up aggressively), and most importantly, vendors try to turn them on by default.
  • AI companions embedded in business apps. This bucket includes Slack AI, Zoom AI Companion, Notion AI, Jira’s Rovo assistant, and the like. These are tied to a single application and are completely inseparable from it.
  • Standalone web and app-based chatbots. ChatGPT, Claude, Perplexity, Character AI, local setups like LM Studio, browser extensions, and agentic browsers like Comet. Apps and services in this category are usually adopted by employees on their own without permission: classic examples of shadow AI.
  • Desktop-native multi-functional agents. This group features tools like OpenClaw, NanoClaw, NemoClaw, and others. They pose the biggest threat because they come with broad access rights by default and actively process untrusted data from the open web.

How to deal with unwanted AI

Every company, depending on its industry, appetite for innovation, and risk tolerance, needs to draw its own line in the sand between recommended, approved case-by-case, and completely banned use cases for specific AI products. Regulated sectors like healthcare play by one set of rules, while retail businesses operate under an entirely different playbook. Either way, after analyzing exactly which AI tools have already slipped into the organization, corporate policies need to be fine-tuned. That’s why the first order of business is employing existing infosec and logging tools to scan corporate infrastructure.

Depending on the chosen strategy, the uncovered AI systems can be:

  • Disabled or restricted by using the built-in corporate policy settings within the tools themselves
  • Hard-blocked at the endpoint or network level to create a safety net against policy workarounds or configuration errors
  • Transitioned to managed access, where the tool isn’t completely blocked but instead routed through a dedicated corporate gateway that checks access permissions, and monitors usage patterns

Detecting AI systems

Spotting AI requires a multi-layered approach, as different detection methods complement each other and work best against specific types of AI.

 

Technology What it can detect
DNS Any AI tool with an identifiable domain
Web Gateway or NGFW Any AI tool with a recognizable request-and-response fingerprint (API endpoint paths, domains, and other indicators). Web filters can inspect traffic content, and many gateways/NGFWs now feature a dedicated category for detecting and blocking generative AI
EPP/EDR Locally deployed LLMs (running via Ollama, LM Studio, and similar shells), native desktop apps for ChatGPT or Claude, agentic browsers, and open-source AI agents. An indirect but strong red flag is the presence of Node.js, Python, Git, Docker, or other containerization tools on machines belonging to non-technical staff
Application control Similar to EPP/EDR, this allows to immediately block unwanted applications right out of the gate
Browser control AI-focused browser extensions and visits to AI-themed websites. This is a lifesaver if the corporate web gateway can’t inspect encrypted traffic
SaaS Security Posture Management (SSPM) / Identity Governance OAuth permissions requested by AI apps and services, as well as any third-party integrations plugging into core productivity hubs (Microsoft 365, Google Workspace, and others)

 

Naturally, almost all of these tools allow to do more than just spot AI — they let to block it entirely, or at the very least, sound the alarm for the team in charge.

Keeping an eye on OAuth

Popular office AI solutions — especially meeting assistants, email and calendar automation agents, and the like — gain access to corporate data by requesting OAuth permissions directly from communication, document workflow, or video conferencing platforms. If a user has the green light to grant these permissions to third-party apps, the resulting data leaks completely bypass the organization’s perimeter. Tools like EDR and NGFW won’t see a thing when a tool like Read.ai grabs recordings of every single meeting in, say, Microsoft Teams.

The most drastic — and often best — move is to block standard users from granting OAuth consent in the first place. Here’s how to handle the technical heavy lifting (Global Administrator, Application Administrator, or equivalent rights are needed):

Microsoft 365 / Entra ID

In the Microsoft Entra admin center, head over to Identity > Applications > Enterprise apps > Consent and permissions > User consent settings. There User consent for applications can be disabled (check out Microsoft’s full guide).

Google Workspace

In the Google Admin console, navigate to Security > Access and data control > API controls. Under Manage App Access, the trust level for all apps can be set: Trusted, Limited, Specific Google data, or Blocked. However, the real kicker here is the Unconfigured app settings subsection, which dictates what happens when a user tries to connect an unknown app. To seal this loophole, select Don’t allow users to access any third-party apps.

A separate subsection, Manage Google Services, permits fine-tuning exactly how third-party apps interact with Google Workspace and Google Cloud services. This allows to cut off access for each individual Google product (see Google’s official guide).

Salesforce

In Setup, use the Quick Find box to search for connected apps, then select Manage Connected Apps from the results. While settings are configured for each external app individually, all users can approve access by default. There isn’t a blanket block switch here; instead, Salesforce allows to opt for Admin approved users are pre-authorized (see the full Salesforce guide on this).

Slack

From the Admin settings menu, head to Apps and workflows -> App Management Settings. Tweak the Require approved apps setting by selecting Only allow pre-approved apps. Once that’s locked in, double-check that no rogue AI tools have slipped onto the approved list.

YouTube wants your face to fight deepfakes

19 May 2026 at 12:51

If you’re worried about deepfake likenesses of yourself showing up online, you’re not alone; YouTube is worried for you. It wants to protect you by having you upload a selfie video and government ID to its site.

The idea is that the video giant will use its own AI to patrol the service for fake videos using your likeness. In exchange, you get the chance to have them taken down.

This isn’t available for everyone, though. It’s for celebs, those in vulnerable jobs, and now, most YouTube creators.

YouTube has been working on this concept, which it calls its “likeness detection” system, since it first floated the idea publicly in September 2024. That December, it launched a partnership with the Creative Artists Agency that saw it using the technology with sporting and entertainment figures.

In October last year, it expanded likeness detection to cover more creators, and then in March it expanded it again to cover politicians and journalists. And last month, it widened the net again, offering the service to Hollywood celebs. They can use it regardless of whether they have a YouTube account, it added.

Now, in its latest move, anyone 18 or older with a selfie and ID can sign up. At least in theory, as it hasn’t rolled out to everyone yet. It’s also for faces only; AI-generated voice clones are another problem entirely.

The privacy risk

Privacy advocates warned that YouTube’s likeness detection system could normalize handing biometric data to large tech platforms, even if YouTube says the data is only used to improve likeness detection models with creator permission.

On the help page for the likeness detection service, YouTube says creators can separately choose whether their face and voice templates are used to improve its likeness detection models.

“When you sign up for Likeness detection, you also have the option to allow YouTube to use your face and voice templates to develop and improve likeness detection models. This helps us build better, more accurate likeness detection technologies.”

Adding:

“You can opt out of YouTube’s use of this data for development and improvement of likeness models at any time.”

YouTube supports legislation intended to tackle deepfakes, such as the NO FAKES and TAKE IT DOWN acts. These are designed to help stop the misappropriation of someone’s image online. TAKE IT DOWN, which became law a year ago, focuses purely on “nonconsensual intimate imagery.” But that doesn’t cover other kinds of deepfakes, such as fake politicians or celebrity endorsements. Those are becoming increasingly common. NO FAKES, which hasn’t yet become law, is far broader in scope, assigning people federal rights over their own image.

So is it worth the trade?

Deepfakes, intimate and otherwise, are definitely a threat, especially for YouTubers who become popular. And the barrier to entry is lowering all the time. Google’s own DeepMind researchers found most generative AI misuse isn’t sophisticated; it’s mundane likeness manipulation by anyone with a browser.

So do you hand over your face and government ID for your protection, to a company whose broader data collection practices have faced years of scrutiny, and hope its policies don’t change? Or do you skip it and hope that the deepfake merchants don’t decide to target you?

Creators commenting on YouTube’s video revealing the service six months ago were less than impressed. One commenter said:

“I was 100% on board, up until the ID upload. That makes me very uncomfortable.”

Echoing several others who complained that it’s difficult to get takedown requests actioned, another added:

“If YouTube actually acted upon these kinds of reports, then I’d be more in favour of this.”

Whether you decide to sign up for the service or not, just be sure to do it with your eyes open.


Someone’s watching your accounts. Make sure it’s us.


YouTube wants your face to fight deepfakes

19 May 2026 at 12:51

If you’re worried about deepfake likenesses of yourself showing up online, you’re not alone; YouTube is worried for you. It wants to protect you by having you upload a selfie video and government ID to its site.

The idea is that the video giant will use its own AI to patrol the service for fake videos using your likeness. In exchange, you get the chance to have them taken down.

This isn’t available for everyone, though. It’s for celebs, those in vulnerable jobs, and now, most YouTube creators.

YouTube has been working on this concept, which it calls its “likeness detection” system, since it first floated the idea publicly in September 2024. That December, it launched a partnership with the Creative Artists Agency that saw it using the technology with sporting and entertainment figures.

In October last year, it expanded likeness detection to cover more creators, and then in March it expanded it again to cover politicians and journalists. And last month, it widened the net again, offering the service to Hollywood celebs. They can use it regardless of whether they have a YouTube account, it added.

Now, in its latest move, anyone 18 or older with a selfie and ID can sign up. At least in theory, as it hasn’t rolled out to everyone yet. It’s also for faces only; AI-generated voice clones are another problem entirely.

The privacy risk

Privacy advocates warned that YouTube’s likeness detection system could normalize handing biometric data to large tech platforms, even if YouTube says the data is only used to improve likeness detection models with creator permission.

On the help page for the likeness detection service, YouTube says creators can separately choose whether their face and voice templates are used to improve its likeness detection models.

“When you sign up for Likeness detection, you also have the option to allow YouTube to use your face and voice templates to develop and improve likeness detection models. This helps us build better, more accurate likeness detection technologies.”

Adding:

“You can opt out of YouTube’s use of this data for development and improvement of likeness models at any time.”

YouTube supports legislation intended to tackle deepfakes, such as the NO FAKES and TAKE IT DOWN acts. These are designed to help stop the misappropriation of someone’s image online. TAKE IT DOWN, which became law a year ago, focuses purely on “nonconsensual intimate imagery.” But that doesn’t cover other kinds of deepfakes, such as fake politicians or celebrity endorsements. Those are becoming increasingly common. NO FAKES, which hasn’t yet become law, is far broader in scope, assigning people federal rights over their own image.

So is it worth the trade?

Deepfakes, intimate and otherwise, are definitely a threat, especially for YouTubers who become popular. And the barrier to entry is lowering all the time. Google’s own DeepMind researchers found most generative AI misuse isn’t sophisticated; it’s mundane likeness manipulation by anyone with a browser.

So do you hand over your face and government ID for your protection, to a company whose broader data collection practices have faced years of scrutiny, and hope its policies don’t change? Or do you skip it and hope that the deepfake merchants don’t decide to target you?

Creators commenting on YouTube’s video revealing the service six months ago were less than impressed. One commenter said:

“I was 100% on board, up until the ID upload. That makes me very uncomfortable.”

Echoing several others who complained that it’s difficult to get takedown requests actioned, another added:

“If YouTube actually acted upon these kinds of reports, then I’d be more in favour of this.”

Whether you decide to sign up for the service or not, just be sure to do it with your eyes open.


Someone’s watching your accounts. Make sure it’s us.


AI is distorting the Holocaust (Lock and Code S07E10)

18 May 2026 at 03:51

This week on the Lock and Code podcast…

In May of last year, a warning about AI came from somewhere unexpected: The Auschwitz-Birkenau State Museum.

Posting publicly on social media, the museum warned about a Facebook account using generative AI to create fake images of people who died in the Holocaust. Despite using AI to generate fake images, the people in said images were sometimes real. They had real names, birthplaces, and stories of deportation that the Auschwitz-Birkenau State Museum itself had shared before. They had real faces captured in real surviving photographs, which were likely abused to generate the false images. 

In other words, someone, or some team of people online, was deepfaking the Holocaust.

As the Auschwitz museum wrote online:

“These are not real photos of the victims. They are digital inventions, often stylized or sanitized, that risk turning remembrance into fictionalized performance. The history of Auschwitz is a well-documented story. Altering its visual record with AI imagery introduces distortion, no matter the intent.”

Months later, the public found out what that intent was: money.

A BBC investigation found an international network of Facebook accounts posting AI-generated images to earn money from those images’ potential virality. It’s a problem sometimes referred to as “AI slop” but it comes with a major incentive. When accounts that make these kinds of images are invited to Facebook’s content monetization program, they can make $1,000 a month for posting anything that gets clicks.

And on Facebook, the BBC found, that means several accounts posting AI-generated images about the Holocaust. As the BBC reported:

“AI spammers have posted fake images purporting to be from inside [Auschwitz], such as a prisoner playing a violin or lovers meeting at the boundaries of fences—attracting tens of thousands of likes and shares.”

The economics of lying are concrete today. People can use AI to make fake images that make people feel good about terrible things or feel scared about untrue things, and they can make money until shut down by the Big Tech platforms themselves, which, in this case, only happened because of the BBC’s investigation. In fact, it’s that type of inaction from social media platforms that compelled the German government and multiple Holocaust memorial institutions to send an open letter earlier this year that asked for better controls and restrictions against this type of content.

As the signatories warned in their letter, the economic appeal for these accounts to distort history is too high a risk to allow. You can read the full letter here.

Today, on the Lock and Code podcast with host David Ruiz, we speak with Clara Mansfeld, a historian working on digital communications at one of the institutions signed onto the open letter—the Foundation of Hamburg Memorials and Learning Centers Commemorating the Victims of Nazi Crimes. In their conversation, Mansfeld discusses digital access to history, the manipulation of factual records through AI-generated imagery, and the threat that society faces when it becomes harder to evaluate the truth.

“What happens when the first thought we have with every historical image is, ‘Is that even real or is that AI?’ I don’t think we have really grasped what that means for us as a society.”

Tune in today to listen to the full conversation.

Show notes and credits:

Intro Music: “Spellbound” by Kevin MacLeod (incompetech.com)
Licensed under Creative Commons: By Attribution 4.0 License
http://creativecommons.org/licenses/by/4.0/
Outro Music: “Good God” by Wowa (unminus.com)


Listen up—Malwarebytes doesn’t just talk cybersecurity, we provide it.

Protect yourself from online attacks that threaten your identity, your files, your system, and your financial well-being with our exclusive offer for Malwarebytes Premium Security for Lock and Code listeners.

AI is distorting the Holocaust (Lock and Code S07E10)

18 May 2026 at 03:51

This week on the Lock and Code podcast…

In May of last year, a warning about AI came from somewhere unexpected: The Auschwitz-Birkenau State Museum.

Posting publicly on social media, the museum warned about a Facebook account using generative AI to create fake images of people who died in the Holocaust. Despite using AI to generate fake images, the people in said images were sometimes real. They had real names, birthplaces, and stories of deportation that the Auschwitz-Birkenau State Museum itself had shared before. They had real faces captured in real surviving photographs, which were likely abused to generate the false images. 

In other words, someone, or some team of people online, was deepfaking the Holocaust.

As the Auschwitz museum wrote online:

“These are not real photos of the victims. They are digital inventions, often stylized or sanitized, that risk turning remembrance into fictionalized performance. The history of Auschwitz is a well-documented story. Altering its visual record with AI imagery introduces distortion, no matter the intent.”

Months later, the public found out what that intent was: money.

A BBC investigation found an international network of Facebook accounts posting AI-generated images to earn money from those images’ potential virality. It’s a problem sometimes referred to as “AI slop” but it comes with a major incentive. When accounts that make these kinds of images are invited to Facebook’s content monetization program, they can make $1,000 a month for posting anything that gets clicks.

And on Facebook, the BBC found, that means several accounts posting AI-generated images about the Holocaust. As the BBC reported:

“AI spammers have posted fake images purporting to be from inside [Auschwitz], such as a prisoner playing a violin or lovers meeting at the boundaries of fences—attracting tens of thousands of likes and shares.”

The economics of lying are concrete today. People can use AI to make fake images that make people feel good about terrible things or feel scared about untrue things, and they can make money until shut down by the Big Tech platforms themselves, which, in this case, only happened because of the BBC’s investigation. In fact, it’s that type of inaction from social media platforms that compelled the German government and multiple Holocaust memorial institutions to send an open letter earlier this year that asked for better controls and restrictions against this type of content.

As the signatories warned in their letter, the economic appeal for these accounts to distort history is too high a risk to allow. You can read the full letter here.

Today, on the Lock and Code podcast with host David Ruiz, we speak with Clara Mansfeld, a historian working on digital communications at one of the institutions signed onto the open letter—the Foundation of Hamburg Memorials and Learning Centers Commemorating the Victims of Nazi Crimes. In their conversation, Mansfeld discusses digital access to history, the manipulation of factual records through AI-generated imagery, and the threat that society faces when it becomes harder to evaluate the truth.

“What happens when the first thought we have with every historical image is, ‘Is that even real or is that AI?’ I don’t think we have really grasped what that means for us as a society.”

Tune in today to listen to the full conversation.

Show notes and credits:

Intro Music: “Spellbound” by Kevin MacLeod (incompetech.com)
Licensed under Creative Commons: By Attribution 4.0 License
http://creativecommons.org/licenses/by/4.0/
Outro Music: “Good God” by Wowa (unminus.com)


Listen up—Malwarebytes doesn’t just talk cybersecurity, we provide it.

Protect yourself from online attacks that threaten your identity, your files, your system, and your financial well-being with our exclusive offer for Malwarebytes Premium Security for Lock and Code listeners.

Meta’s confusing new approach to chat privacy

15 May 2026 at 14:34

Recent news had us wondering whether Meta actually knows what it wants.

On one platform, Meta is promoting AI chats that it says even it cannot read. On another, it has removed one of the few features that genuinely prevented Meta from accessing private conversations.

“Meta removed support for end-to-end encrypted chats from Instagram as of May 8, 2026.”

Meta adds fully private AI chats to WhatsApp.”

At the moment, Meta is heavily promoting a new Incognito Chat mode for its Meta AI assistant in WhatsApp, built on top of a system it calls Private Processing. According to WhatsApp’s own announcement, Incognito Chat is:

 “Truly private — no one can read your conversation, not even us.”

When you start an Incognito chat with Meta AI, you get a temporary conversation where messages aren’t saved and disappear by default, which Meta pitches as “a space to think and explore ideas without anyone watching.”

BBC News and others report that these AI chats are text‑only for now, run in a sandboxed environment, and are separate from your regular end‑to‑end encrypted (E2EE) messaging with other people on WhatsApp.

Meta is also preparing “Side Chat,” which will let you invoke Meta AI inside other WhatsApp chats, again using this Private Processing infrastructure to claim AI assistance without breaking the underlying encryption.

On paper, that’s an impressive technical and marketing story: powerful AI, wrapped in layers of privacy‑preserving infrastructure, added to an app that already has a strong reputation for end‑to‑end encryption by default.

Meanwhile, on Instagram…

Now contrast that with what’s happening on Instagram. On 8 May 2026, Meta removed optional end‑to‑end encryption for Instagram Direct Messages (DMs) entirely. Users who had previously turned the feature on were shown notices that “end‑to‑end encrypted messaging on Instagram is no longer supported as of 8 May 2026,” and were urged to download backups of their encrypted conversations before the cutoff.

End‑to‑end encryption ensures that only the sender and recipient can read their conversations. Instagram offered this as an opt‑in feature since late 2023, but it was buried several taps deep inside individual conversation settings and never turned on by default. Meta’s explanation for shutting it down is that “very few people” used encrypted DMs and that maintaining a separate encrypted system added complexity. Critics have pointed out the circular logic. The company hid the feature, did not advertise it, and is now using low adoption as the reason to kill it rather than, say, making it easier to find or turning it on by default.

What all this means

From a user’s perspective, the result is confusing: one Meta product introduces stronger privacy than ever for AI chats, while another removes the one feature that truly stopped Meta from reading your conversations.

The key point to remember here is that “incognito” and “private” are marketing words, while end‑to‑end encryption is a technical guarantee.

For security‑conscious users, this split personality means you can no longer treat all Meta chats the same. WhatsApp remains end‑to‑end encrypted for person‑to‑person messages and adds optional privacy features around its AI, while Instagram DMs should now be assumed readable by Meta and potentially accessible to law enforcement, advertisers, or attackers who gain access to Meta’s systems.


To boldly browse, away from prying eyes. 


Why make AI chats private?

We’ve seen that AI chats have suddenly turned up in search results without users’ knowledge. So there definitely is a positive side to this new feature.

We also know there have been lawsuits against chatbot providers in cases where the outcome of an AI conversation led to very undesirable results. But how would you be able to provide evidence when messages auto-disappear?

How to proceed

Meta’s recent moves show that strong privacy features can be added where they support a strategic narrative and removed where they conflict with business or regulatory priorities. Users can’t control those decisions, but they can respond by choosing where they hold their most sensitive conversations and by assuming that if a chat isn’t end‑to‑end encrypted by default, it is ultimately readable by someone other than the people in it.

So, what’s a safe way to move forward?

  • Treat Instagram DMs as postcard-level privacy. Now that E2EE is gone, assume Meta can read and scan your messages and that content could be accessed under legal orders or in a breach. Do not send passwords, recovery codes, banking details, or compromising photos over Instagram.
  • When someone asks you to move a conversation to Signal, WhatsApp, or another E2EE messenger, ask them why. It does make sense when you’re sharing financial details, personal images, health information, or anything you would not want a platform provider to read. But sometimes scammers prefer encrypted platforms too, because they’re harder to monitor.
  • Do not confuse “incognito” AI chats with full encryption. WhatsApp’s Incognito mode for Meta AI may be a privacy improvement over standard cloud AI chats, but it is still a conversation with a large language model owned by the same company that runs the platform. Share only what you’re comfortable entrusting to Meta.
  • Regularly review your privacy and security settings. Check which devices are logged in, enable two‑factor authentication, and verify which of your chat apps are actually end‑to‑end encrypted by default.

Scammers know more about you than you think. 

Malwarebytes Mobile Security protects you from phishing, scam texts, malicious sites, and more. With real-time AI-powered Scam Guard built right in. 

Download for iOS → Download for Android → 

❌