We continue tracking the activity of Toy Ghouls (also known as Bearlyfy, Laboo.boo, and Feral Wolf), a financially motivated group that has been targeting Russian organizations since 2025. The attackers initially relied exclusively on tools pulled from public GitHub repositories along with leaked Babuk and LockBit ransomware builders, later shifting to their own custom ransomware, GenieLocker. In early July 2026, we observed the group using a custom backdoor for the first time.
We identified two versions of this backdoor: one uses the HiveMQ MQTT broker as its C2 server, while the other relies on the Element messenger. Both versions include “bird” in their names:
mqtt-bird-agent 0.1.0 (HiveMQ version)
matrix-bird-agent 0.1.0 (Element version)
This post examines how the backdoor is delivered to target systems, how it establishes persistence, and how it communicates with its C2 server.
Technical details
Delivery
In this campaign, the attackers use Windows Remote Management (WinRM) to deliver the backdoors and their configuration files to compromised systems. The group relies on open-source tools such as Evil-WinRM and WinRM-fs to do this.
Installation
The backdoor can both run within an interactive command-line session and establish persistence as a Windows service, using the --install or install option, depending on the backdoor version. The --service (or service) option is not available by default and is instead used as an argument for the installed Windows service.
Other launch options are listed in the backdoor’s help output:
C:\cplsupport.exe -h
Bird Agent - MQTT server monitor
Usage: cplsupport.exe [OPTIONS]
Options:
-c, --config <CONFIG> Path to config.toml config file
--install Install as a system service
--uninstall Uninstall the system service
--seal Encrypt sensitive config fields in-place using a machine-bound key
-h, --help Print help
-V, --version Print version
HiveMQ version backdoor help output
In the Element version, the backdoor help output looks as follows:
C:\wtass.exe -h
Matrix monitoring agent
Usage: wtass.exe [OPTIONS] [COMMAND]
Commands:
install Register this agent with the Matrix homeserver and panel
uninstall Remove this agent's service and credentials
service Run as a Windows service (internal)
help Print this message or the help of the given subcommand(s)
Options:
-c, --config <CONFIG>
-h, --help Print help
-V, --version Print version
Element version backdoor help output
By default, the backdoor looks for a config.toml configuration file in the directory where the executable was launched, then falls back to %PROGRAMDATA%\SynapseAgent\config.toml (Element version) or %PROGRAMDATA%\cplsupport\config.toml (HiveMQ version). If no configuration file is found in either location, the full path can be specified using the -c (--config) option.
The backdoor accepts both unencrypted configuration files and files with partially encrypted sections. In the first case, once the backdoor is launched, it reads the file and partially encrypts it using the seal() function (the --seal option in the HiveMQ version), applying the ChaCha20-Poly1305 algorithm with a key derived from the value of the HKLM\Software\Microsoft\Cryptography\MachineGuid registry key. This means that after the backdoor’s first run, the configuration file becomes bound to that specific machine. On subsequent runs, the configuration is decrypted automatically. If the input configuration was already partially encrypted, it is likewise decrypted automatically.
If the configuration cannot be decrypted, the backdoor stops running.
Encrypted configuration files look as follows:
Encrypted backdoor configuration file, HiveMQ version
The encrypted portion of the HiveMQ version’s configuration contains the following parameters:
agent_privkey: the agent’s private key
channel_id: the channel identifier used to communicate with the broker
server_pubkey: the server’s public key
Decrypted blob field in the HiveMQ version’s configuration
In the Element version, the configuration file is deleted immediately after the first run, and the relevant parameters are instead written to the HKLM\Software\synapse\Config\SealedConfig registry key. On subsequent runs, the backdoor checks the registry for its configuration first.
Decrypted Element version configuration file, retrieved from the registry
The Element version’s configuration specifies the address of an Element server controlled by the attackers, a room identifier, and an access_token used to access that room. If this parameter is left empty, the backdoor prompts for the password interactively during installation. After successfully creating a session, the backdoor saves the received token to the blob field.
Communication
At startup, both backdoor versions send a GET request to http://ip-api.com/json to determine the system’s public IP address and country of origin.
The first version uses the public HiveMQ MQTT broker (broker.hivemq.com) as its C2 server. The free tier of this broker supports up to 100 concurrent connections and up to 10 GB of traffic per month. The attackers set up their own cluster and used it both to collect telemetry from compromised systems and to send commands to the backdoor.
Once a connection is established, the system’s status is sent via a POST request to
broker.hivemq.com:8883/[cluster_id]/status. The message format is:
{"online":bool,"hostname":"hostname.domain","timestamp":unix_timestamp,"location":{"json"}}.
At intervals defined in the configuration file, system information, such as CPU load and available memory, is sent via a POST request to
broker.hivemq.com:8883/[cluster_id]/metrics3. The message format is:
{cpu_percent":float,"mem_used_bytes":int,"mem_total_bytes":int,"disk_used_bytes":int,"disk_total_bytes":int,"load_1m":float,"load_5m":float,"load_15m":float,"uptime_secs":int,"hostname":"hostname.domain","timestamp":unix_timestamp}.
The backdoor sends GET requests to
broker.hivemq.com:8883/[cluster_id]/cmd/req to retrieve commands from the C2 server. The server responds in the format:
{"cmd_id":int,"command":"str","timeout_secs":int}.
Commands are executed via PowerShell.exe in hidden mode, using the -NonInteractive -NoProfile -Command parameters.
Command execution results are sent to the command server at
broker.hivemq.com:8883/[cluster_id]/cmd/res in the
{"stdout":"str","stderr":"str","exit_code":int,"duration_ms":int} format.
For the second backdoor version, the attackers set up their own Element server running on the Matrix protocol, meet.element[.]tw, as the C2 server. On this server, they created a room used to receive messages containing device information and to send commands for execution on the compromised system. The communication flow is as follows:
Once a connection is successfully established, the backdoor sends an m.bird.status message containing the system’s status. This message format is identical to that used in the HiveMQ version.
At intervals defined in the configuration file, information about the compromised system is sent as an m.bird.metrics message. Field names are slightly different from those in the first version:
{cpu_percent_x100":float,"mem_used_bytes":int,"mem_total_bytes":int,"disk_used_bytes":int,"disk_total_bytes":int,"load_1m_x100":float,"load_5m_x100":float,"load_15m_x100":float,"uptime_secs":int,"hostname":"hostname.domain","timestamp":unix_timestamp}.
This version of the backdoor supports two types of commands, distinguished by the start of the received message.
To set a new interval for sending metrics, the attackers send a message beginning with config:set_interval (accepting values from 5 to 3600 seconds). The new value is saved to the HKLM\Software\SynapseAgent\metrics_interval registry key.
Messages containing commands to execute begin with the string cmd:. Based on data extracted from Element’s SQLite databases on the compromised system, we were able to identify the account name the attackers used to send commands: panel-bot.
Received commands are executed via the Windows command line interface.
Command output is sent as an m.bird.cmd_response message. This message format mirrors the one used in the HiveMQ version.
Takeaways
We have been tracking Toy Ghouls’ activity for quite some time. We previously found that the group had expanded its arsenal with a custom ransomware strain, GenieLocker, and we have now discovered that it has also developed a backdoor capable of giving it full control over an infected device. The new tools use unconventional channels to communicate with their C2 server: the HiveMQ MQTT broker and the Matrix-based Element messenger. This shift away from publicly available open-source projects toward custom-built tools suggests that Toy Ghouls is working to make its attacks more sophisticated and to evade detection for longer.
The vulnerability landscape shifted significantly in Q2 2026. First, the number of registered CVEs reached an unprecedented level. This is driven primarily by the widespread adoption of AI, both for application development and search for security flaws. This resulted in entire new classes of vulnerabilities emerging, particularly in the Linux networking subsystem.
Second, security researchers have been publishing exploits for unpatched vulnerabilities more frequently. Publications like these can generate significant fallout, since they potentially open the door for attackers to target unprotected systems.
Statistics on registered vulnerabilities
This section provides statistical data on registered vulnerabilities. The data comes from Kaspersky’s vulnerability knowledge base, which draws on the CVE database as well as the Russian BDU database and GitHub Advisory (GHSA). As a result, the figures for previous reporting periods may differ from those published in earlier reports.
We examine the number of registered vulnerabilities for each month over the last five years. As the chart below shows, this number continues to surge, a trend reflected across all the databases we track. It’s driven primarily by the widespread adoption of AI tools: as we predicted in our previous report, these tools have played a major role in the discovery of vulnerabilities in third-party software. Meanwhile, these tools often contain security issues of their own. For example, OpenClaw, a popular AI project, ranked 12th among those with the highest number of vulnerabilities discovered and published in Q2, with over 200 CVEs registered during the reporting period. Finally, AI development tools are also contributing to the vulnerability landscape, since the quality of the code they produce can vary widely. Therefore, the rate at which new vulnerabilities are discovered will inevitably keep growing.
Total published vulnerabilities per month from 2022 through 2026 (download)
Next, we analyze the number of new critical vulnerabilities (CVSS > 9.0) over the same period.
Total critical vulnerabilities published per month from 2022 through 2026 (download)
As the chart shows, the number of published critical vulnerabilities jumped sharply in Q2. This is because using AI for vulnerability research makes it possible to analyze massive amounts of previously unexamined code, uncover new attack surfaces, and identify entire classes of vulnerabilities that have gone unnoticed for decades. In particular, AI was used to find a series of Dirty Frag vulnerabilities in the Linux kernel.
Exploitation statistics
This section presents statistics on vulnerability exploitation for Q2 2026. The data draws on open sources and our telemetry.
Windows and Linux vulnerability exploitation
Q2 2026 saw a new precedent in the publication of vulnerabilities in Windows components and exploits for these: researchers no longer waiting for CVE registration, let alone patches. A case in point: a researcher who goes by Nightmare Eclipse (also known as Chaotic Eclipse) published a list of new “named” vulnerabilities across various Windows subsystems. At the time the technical details were published, none of the vulnerabilities had been assigned a CVE identifier:
BlueHammer: a local privilege escalation vulnerability in Windows Defender. During signature database updates, a time-of-check to time-of-use (TOCTOU) race condition occurs, allowing an attacker to substitute the directory where temporary update files are written. The researcher published a fully functional exploit for the vulnerability.
RedSun: another logical vulnerability in Windows Defender with a working exploit. Suspicious and malicious files marked as “cloud” can be overwritten or restored to their original directory with elevated privileges. The exploit incorporates fragments of algorithms that make it possible to leverage various logical vulnerabilities in Windows, effectively combining a large number of popular exploitation techniques.
YellowKey: a vulnerability that lets the user bypass BitLocker full-disk encryption and access system data through the Windows Recovery Environment (WinRE). A fully functional exploit was also published.
GreenPlasma: a vulnerability that enables system object injection via the CTF loader for the Collaborative Translation Framework (CTFMON) service in Windows. The original publication included an exploit with limited functionality.
RoguePlanet: yet another Windows Defender vulnerability that, like BlueHammer, stems from a TOCTOU issue, this time in the engine responsible for real-time system scanning. The published exploit uses the vulnerability to overwrite the system file wermgr.exe with a malicious one.
UnDefend: another vulnerability in the Windows Defender service. This time, the exploit causes a denial of service and blocks updates.
Even though such cases remain isolated for now, we believe they’ll grow into a full-fledged trend. Early publication of exploits gives attackers an advantage over software developers, who are left with no time to fix the issues.
Veteran vulnerabilities in Windows software also remain relevant. These are the ones our solutions most frequently detect exploits for:
CVE-2018-0802: a remote code execution (RCE) vulnerability in the Equation Editor component
CVE-2017-11882: another RCE vulnerability also affecting Equation Editor
CVE-2017-0199: a vulnerability in Microsoft Office and WordPad that allows an attacker to gain control over the system
CVE-2023-38831: a vulnerability in WinRAR that involves improper handling of objects within an archive
CVE-2025-6218 (formerly ZDI-CAN-27198): another WinRAR vulnerability allowing the specification of relative paths to extract files into arbitrary directories, potentially leading to malicious command execution
CVE-2025-8088: a vulnerability similar in exploitation method to CVE-2025-6218. The attackers used NTFS Streams to circumvent controls on the directory into which files are being unpacked
The vulnerabilities listed here can be leveraged to gain initial access to a vulnerable system and for privilege escalation. This underscores the critical importance of timely software updates.
That said, the number of Windows users who encountered exploits declined slightly in Q2, hitting an 18-month low.
Dynamics of the number of Windows users encountering exploits, Q1 2025 – Q2 2026. The number of users who encountered exploits in Q1 2025 is taken as 100% (download)
Linux also hit a rough patch in Q2 2026. Specifically, the period saw the disclosure of the Dirty Frag family of vulnerabilities, which lets an attacker reliably escalate privileges within the operating system.
All the vulnerabilities published in Q2 2026 were, in one way or another, related to the Linux caching subsystem. Here are the ones being most actively exploited:
CVE-2026-31431 (Copy Fail): a local privilege escalation vulnerability in the Linux kernel that lets an unprivileged user modify the page cache and gain root privileges. Especially dangerous for cloud and containerized environments
CVE-2026-43284, CVE-2026-43500 (Dirty Frag): a family of vulnerabilities in the Linux networking subsystem (IPsec ESP and RxRPC) that lets a local user overwrite the page cache and escalate privileges to root
CVE-2026-46300 (Fragnesia): a local privilege escalation vulnerability in the Linux kernel related to packet fragment handling and the page cache mechanism. It lets an unprivileged user gain root privileges and is also classified as part of the Dirty Frag family
CVE-2026-31635 (DirtyDecrypt): a Linux kernel vulnerability that lets a local attacker escalate privileges due to improper handling of decryption operations and page cache data modification
CVE-2026-43494 (PinTheft): a Linux kernel vulnerability that lets a local user gain elevated privileges due to errors in the memory page pinning mechanism
CVE-2026-46331 (pedit COW): a vulnerability in the Linux kernel’s traffic control subsystem (tc-pedit) that exploits a flaw in copy-on-write to modify the page cache and subsequently escalate privileges to root
The vulnerabilities described above were quickly embraced by attackers. At the same time, our solutions continue to detect exploitation attempts targeting older vulnerabilities as well:
CVE-2022-0847: a vulnerability known as Dirty Pipe, which enables privilege escalation and the hijacking of running applications
CVE-2019-13272: a vulnerability caused by improper handling of privilege inheritance, which can be exploited to achieve privilege escalation
CVE-2021-22555: a heap out-of-bounds write vulnerability in the Netfilter kernel subsystem
CVE-2023-32233: another Netfilter subsystem vulnerability that allows for Use-After-Free conditions and privilege escalation through improper processing of network requests
Dynamics of the number of Linux users encountering exploits, Q1 2025 – Q2 2026. The number of users who encountered exploits in Q1 2025 is taken as 100% (download)
In Q2 2026, the number of Linux users who encountered exploits declined slightly compared to Q1. Given that a significant share of new vulnerabilities are tied to the operating system’s caching subsystem, we recommend installing patches as quickly as possible, or disabling vulnerable kernel modules if patching isn’t an option.
Most common published exploits
The distribution of published exploits by software type in Q2 2026 includes categories that haven’t appeared in the sample for a long time. For instance, we’re once again seeing exploits targeting SharePoint. It’s worth noting that while several vulnerability write-ups for Exchange and SharePoint were published during the quarter, most turned out to be fake, AI-generated research. While the articles and exploit source code themselves look fairly polished, they describe nonexistent problems in the software or its components — often close to genuinely vulnerable mechanisms — in order to mislead researchers. This type of attack is aimed at increasing the time it takes to detect real vulnerabilities. In some cases, the description of a nonexistent vulnerability came bundled with completely unrelated malware.
Distribution of published exploits by platform, Q1 2026 (download)
Distribution of published exploits by platform, Q2 2026 (download)
Vulnerability exploitation in APT attacks
We analyzed which vulnerabilities were exploited in APT attacks during Q2 2026. The rankings provided below include data based on our telemetry, research, and open sources.
TOP 10 vulnerabilities exploited in APT attacks, Q2 2026 (download)
In Q2 2026, a trend emerged in APT attacks toward exploiting new vulnerabilities right from the moment they’re published. As before, we’re also seeing a large number of zero-day vulnerabilities. The Langflow vulnerability deserves particular attention: it’s one of the first cases of an APT group exploiting AI technology, which many organizations are only just beginning to integrate. Because most of this tech is proprietary, it has a considerable number of security blind spots. Therefore, given the growing number of AI-based automation tools, we strongly recommend going beyond the usual patching and developing secure procedures for credential use and sensitive data handling in systems that rely on agents and LLMs.
C2 frameworks
In this section, we examine the most popular C2 frameworks used by APT groups and analyze the vulnerabilities targeted by the exploits that interacted with C2 agents in APT attacks.
The chart below shows the frequency of known C2 framework usage in attacks during Q2 2026, according to open sources.
TOP 10 C2 frameworks used by APTs to compromise user systems, Q2 2026 (download)
Sliver, Havoc, AdaptixC2, and Metasploit remain the most widely used C2 frameworks. After studying open sources and analyzing samples of malicious C2 agents that contained exploits, we determined that the following vulnerabilities were utilized in APT attacks involving the C2 frameworks mentioned above:
CVE-2026-35273: a vulnerability in Oracle PeopleSoft PeopleTools that security vendors classify as server-side request forgery (SSRF). The details of the vulnerability have never been disclosed, although some research covers the post-exploitation steps
CVE-2023-46604: an insecure deserialization vulnerability in Apache ActiveMQ that allows arbitrary code execution in the context of the service process
CVE-2024-12356 and CVE-2026-1731: command injection vulnerabilities in BeyondTrust software that allow an attacker to send malicious commands even without system authentication
CVE-2023-36884: a vulnerability in the Windows Search component that allows commands to be run on the system, bypassing the mark-of-the-web (MoTW) mechanism
CVE-2025-53770: an insecure deserialization vulnerability in Microsoft SharePoint that allows for unauthenticated command execution on the server
CVE-2025-8088 and CVE-2025-6218: similar directory traversal vulnerabilities in WinRAR that allow files to be extracted from an archive to a predetermined path, potentially without the archiving utility displaying any alerts to the user
These vulnerabilities show that attackers used them for initial access and privilege escalation on vulnerable systems, setting the stage for launching a C2 agent. They include both zero-day vulnerabilities and fairly well-known security issues.
LLM/AI tool vulnerabilities
This section analyzes data published in Kaspersky’s vulnerability knowledge base. We reviewed the Q2 2026 version of the knowledge base.
As mentioned above, AI tools, plugins, and technologies have proven fairly effective at automating the search for problematic code and anomalous behavior. The high speed at which new vulnerabilities are being discovered has naturally created a need to fix them just as quickly. AI is often used for this too, which increases the volume of code being generated. However, neither code written without human involvement nor AI-generated advice is always correct.
The chart below covers registered vulnerabilities in AI tools for 2025–2026.
Number of published vulnerabilities in LLMs, AI tools, and plugins with similar functionality, 2025–2026 (download)
As the charts show, AI tools are racking up a substantial number of registered vulnerabilities, and that number keeps growing quarter over quarter. It’s also worth looking at how AI tool vulnerabilities break down by type, according to the CWE system:
TOP 6 vulnerability types in products that implement or use AI/LLM logic, 2025–2026
Interestingly, vulnerabilities of an undetermined type have ranked first in every quarter since the start of 2025. Traditionally-made software has the same issue, and it doesn’t look like the growing number of AI tools will fix it. It’s also notable that the list includes classes CWE developers themselves don’t recommend using for vulnerability classification, since they lump together a whole range of more specific types. CWE-284 is an example of this.
Looking at the most common classes, the key issues found in AI-related software can be summed up as follows:
Inadequate access control over critical system objects
Improper implementation of authentication and authorization mechanisms
Injections
It’s worth noting that injection-related vulnerabilities were relatively rare before AI agents took off (previously, they mostly affected web apps). Recently, though, these security issues have become relevant again.
Looking back at a year and a half of the AI boom, one conclusion stands out regarding registered vulnerabilities: AI tool developers are more focused on expanding functionality than on security. This is worth keeping in mind when using these tools. Let’s look at the projects and applications that either integrated AI tools or offered them as the core product. Below is a list of the those with the highest number of registered vulnerabilities for 2025–2026.
TOP AI/LLM-related projects by number of published vulnerabilities, 2025–2026 (download)
Notable vulnerabilities
This section highlights the most significant vulnerabilities published in Q2 2026 that have publicly available descriptions. Since the above already covers several significant vulnerabilities published during the reporting period, this section consists mainly of LLM/AI tool vulnerabilities.
CVE-2026-25253: a gatewayUrl vulnerability in OpenClaw
The issue stems from the fact that the OpenClaw user interface trusts the value of the gatewayUrl parameter passed in the URL and automatically establishes a WebSocket connection to the specified address. During this connection process, it sends an authentication token without any additional user confirmation.
The attack algorithm exploiting this vulnerability works as follows:
The application obtains a critical connection address from an external source (the gatewayUrl URL parameter), which is controlled by the attacker.
There is no validation before use.
The client automatically initiates a connection to the address specified in the parameter, which belongs to the attacker.
While connected, the application sends credentials (an access token) to the specified address.
If the attacker obtains a valid token, the consequences depend on that token’s level of access within the system. In general, this could lead to:
User session compromise
Execution of operations on the user’s behalf
Modification of the AI agent configuration
Unauthorized access to tools and resources connected to the agent
Under certain OpenClaw configurations, further compromise of the host running the agent
It’s worth noting that the risk of exploitation arises from a combination of several factors: the automatic connection and token transmission, the lack of address trust verification, and the high privileges granted to the local AI agent.
CVE-2026-41948: a path traversal vulnerability in the Dify AI platform
The vulnerability lets an authenticated user craft a request that enables the application to escape its permitted tenant and gain access to internal REST APIs that weren’t meant for that user. The root cause is insufficient normalization and validation of the URL path before it’s passed to the internal service.
Depending on the Dify configuration, the consequences can include:
Unauthorized access to internal service interfaces
Breach of isolation between workspaces
Exposure of internal service information
Conditions favorable to further attacks when combined with other vulnerabilities
The use of Dify in enterprise AI platforms is particularly risky, since internal services there tend to hold elevated privileges.
CVE-2026-45386: an improper access control vulnerability in Open WebUI
In Open WebUI, pin/unpin operations on messages are write operations, since they modify that message’s metadata (is_pinned, pinned_by, pinned_at). In vulnerable versions, however, before performing these actions, the API only checked for read access to the channel (a chat between a user or group and the AI) containing the message, not permission to modify its content. As a result, a user with a role limited to viewing messages could still change a message’s pinned status.
The vulnerability’s mechanism works as follows:
The user initiates an action that changes the state of an object.
The application treats this action as a regular read request.
Only channel view permission is checked.
The application performs a write without verifying the required user authorization.
This violates one of the fundamental principles of access control models — namely, that any operation that changes the state of data must be checked for the appropriate write or moderation permissions, regardless of whether the object itself is readable.
Although the vulnerability doesn’t lead to arbitrary code execution or compromise of sensitive data, it can affect data integrity and collaborative workflows. Potential consequences of exploitation include unauthorized pinning or unpinning of messages, disruption of channel moderators’ and administrators’ activities, changes to the display order of important information, and even the potential spread of false or misleading information by altering the channel containing a pinned message.
Open WebUI is widely used as an interface for interacting with local and enterprise LLMs. In these systems, pinned messages often contain important instructions, announcements, or tips for users. The ability to modify them with minimal privileges can disrupt collaborative workflows, cause confusion, and undermine trust in information published by administrators and moderators.
CVE-2026-45501: a vulnerability in Microsoft Exchange
The vulnerability stems from improper neutralization of user input when generating Exchange web pages. As a result, the browser may interpret specially crafted data as active content instead of plain text.
Although Microsoft categorizes the potential impact of exploiting this vulnerability as spoofing, flaws like this can lead to alteration of displayed content, imitation of trusted interfaces, actions on behalf of the user within an active session, and abuse of user trust.
It’s worth noting that issues like this are still relevant in modern software, given that mechanisms like Content Security Policy and various parsers were specifically created to help developers neutralize dangerous parts of user page content.
Conclusion and advice
Q2 brought the first significant results of AI automation adoption in software development and vulnerability hunting tools. This research shows that beyond traditional patch management, organizations now need real-time monitoring of systems and access controls, since infrastructure and everyday applications now contain far more AI functionality that could lead to compromise.
Accordingly, besides quickly detecting infrastructure vulnerabilities and managing security patches, modern enterprise-grade security solutions need to provide a broad range of preventive measures for tracking the overall health of systems and workstations. Kaspersky Next meets these requirements by combining proactive mechanisms with the ability to respond promptly to emerging threats.
We continue tracking the activity of Toy Ghouls (also known as Bearlyfy, Laboo.boo, and Feral Wolf), a financially motivated group that has been targeting Russian organizations since 2025. The attackers initially relied exclusively on tools pulled from public GitHub repositories along with leaked Babuk and LockBit ransomware builders, later shifting to their own custom ransomware, GenieLocker. In early July 2026, we observed the group using a custom backdoor for the first time.
We identified two versions of this backdoor: one uses the HiveMQ MQTT broker as its C2 server, while the other relies on the Element messenger. Both versions include “bird” in their names:
mqtt-bird-agent 0.1.0 (HiveMQ version)
matrix-bird-agent 0.1.0 (Element version)
This post examines how the backdoor is delivered to target systems, how it establishes persistence, and how it communicates with its C2 server.
Technical details
Delivery
In this campaign, the attackers use Windows Remote Management (WinRM) to deliver the backdoors and their configuration files to compromised systems. The group relies on open-source tools such as Evil-WinRM and WinRM-fs to do this.
Installation
The backdoor can both run within an interactive command-line session and establish persistence as a Windows service, using the --install or install option, depending on the backdoor version. The --service (or service) option is not available by default and is instead used as an argument for the installed Windows service.
Other launch options are listed in the backdoor’s help output:
C:\cplsupport.exe -h
Bird Agent - MQTT server monitor
Usage: cplsupport.exe [OPTIONS]
Options:
-c, --config <CONFIG> Path to config.toml config file
--install Install as a system service
--uninstall Uninstall the system service
--seal Encrypt sensitive config fields in-place using a machine-bound key
-h, --help Print help
-V, --version Print version
HiveMQ version backdoor help output
In the Element version, the backdoor help output looks as follows:
C:\wtass.exe -h
Matrix monitoring agent
Usage: wtass.exe [OPTIONS] [COMMAND]
Commands:
install Register this agent with the Matrix homeserver and panel
uninstall Remove this agent's service and credentials
service Run as a Windows service (internal)
help Print this message or the help of the given subcommand(s)
Options:
-c, --config <CONFIG>
-h, --help Print help
-V, --version Print version
Element version backdoor help output
By default, the backdoor looks for a config.toml configuration file in the directory where the executable was launched, then falls back to %PROGRAMDATA%\SynapseAgent\config.toml (Element version) or %PROGRAMDATA%\cplsupport\config.toml (HiveMQ version). If no configuration file is found in either location, the full path can be specified using the -c (--config) option.
The backdoor accepts both unencrypted configuration files and files with partially encrypted sections. In the first case, once the backdoor is launched, it reads the file and partially encrypts it using the seal() function (the --seal option in the HiveMQ version), applying the ChaCha20-Poly1305 algorithm with a key derived from the value of the HKLM\Software\Microsoft\Cryptography\MachineGuid registry key. This means that after the backdoor’s first run, the configuration file becomes bound to that specific machine. On subsequent runs, the configuration is decrypted automatically. If the input configuration was already partially encrypted, it is likewise decrypted automatically.
If the configuration cannot be decrypted, the backdoor stops running.
Encrypted configuration files look as follows:
Encrypted backdoor configuration file, HiveMQ version
The encrypted portion of the HiveMQ version’s configuration contains the following parameters:
agent_privkey: the agent’s private key
channel_id: the channel identifier used to communicate with the broker
server_pubkey: the server’s public key
Decrypted blob field in the HiveMQ version’s configuration
In the Element version, the configuration file is deleted immediately after the first run, and the relevant parameters are instead written to the HKLM\Software\synapse\Config\SealedConfig registry key. On subsequent runs, the backdoor checks the registry for its configuration first.
Decrypted Element version configuration file, retrieved from the registry
The Element version’s configuration specifies the address of an Element server controlled by the attackers, a room identifier, and an access_token used to access that room. If this parameter is left empty, the backdoor prompts for the password interactively during installation. After successfully creating a session, the backdoor saves the received token to the blob field.
Communication
At startup, both backdoor versions send a GET request to http://ip-api.com/json to determine the system’s public IP address and country of origin.
The first version uses the public HiveMQ MQTT broker (broker.hivemq.com) as its C2 server. The free tier of this broker supports up to 100 concurrent connections and up to 10 GB of traffic per month. The attackers set up their own cluster and used it both to collect telemetry from compromised systems and to send commands to the backdoor.
Once a connection is established, the system’s status is sent via a POST request to
broker.hivemq.com:8883/[cluster_id]/status. The message format is:
{"online":bool,"hostname":"hostname.domain","timestamp":unix_timestamp,"location":{"json"}}.
At intervals defined in the configuration file, system information, such as CPU load and available memory, is sent via a POST request to
broker.hivemq.com:8883/[cluster_id]/metrics3. The message format is:
{cpu_percent":float,"mem_used_bytes":int,"mem_total_bytes":int,"disk_used_bytes":int,"disk_total_bytes":int,"load_1m":float,"load_5m":float,"load_15m":float,"uptime_secs":int,"hostname":"hostname.domain","timestamp":unix_timestamp}.
The backdoor sends GET requests to
broker.hivemq.com:8883/[cluster_id]/cmd/req to retrieve commands from the C2 server. The server responds in the format:
{"cmd_id":int,"command":"str","timeout_secs":int}.
Commands are executed via PowerShell.exe in hidden mode, using the -NonInteractive -NoProfile -Command parameters.
Command execution results are sent to the command server at
broker.hivemq.com:8883/[cluster_id]/cmd/res in the
{"stdout":"str","stderr":"str","exit_code":int,"duration_ms":int} format.
For the second backdoor version, the attackers set up their own Element server running on the Matrix protocol, meet.element[.]tw, as the C2 server. On this server, they created a room used to receive messages containing device information and to send commands for execution on the compromised system. The communication flow is as follows:
Once a connection is successfully established, the backdoor sends an m.bird.status message containing the system’s status. This message format is identical to that used in the HiveMQ version.
At intervals defined in the configuration file, information about the compromised system is sent as an m.bird.metrics message. Field names are slightly different from those in the first version:
{cpu_percent_x100":float,"mem_used_bytes":int,"mem_total_bytes":int,"disk_used_bytes":int,"disk_total_bytes":int,"load_1m_x100":float,"load_5m_x100":float,"load_15m_x100":float,"uptime_secs":int,"hostname":"hostname.domain","timestamp":unix_timestamp}.
This version of the backdoor supports two types of commands, distinguished by the start of the received message.
To set a new interval for sending metrics, the attackers send a message beginning with config:set_interval (accepting values from 5 to 3600 seconds). The new value is saved to the HKLM\Software\SynapseAgent\metrics_interval registry key.
Messages containing commands to execute begin with the string cmd:. Based on data extracted from Element’s SQLite databases on the compromised system, we were able to identify the account name the attackers used to send commands: panel-bot.
Received commands are executed via the Windows command line interface.
Command output is sent as an m.bird.cmd_response message. This message format mirrors the one used in the HiveMQ version.
Takeaways
We have been tracking Toy Ghouls’ activity for quite some time. We previously found that the group had expanded its arsenal with a custom ransomware strain, GenieLocker, and we have now discovered that it has also developed a backdoor capable of giving it full control over an infected device. The new tools use unconventional channels to communicate with their C2 server: the HiveMQ MQTT broker and the Matrix-based Element messenger. This shift away from publicly available open-source projects toward custom-built tools suggests that Toy Ghouls is working to make its attacks more sophisticated and to evade detection for longer.
The vulnerability landscape shifted significantly in Q2 2026. First, the number of registered CVEs reached an unprecedented level. This is driven primarily by the widespread adoption of AI, both for application development and search for security flaws. This resulted in entire new classes of vulnerabilities emerging, particularly in the Linux networking subsystem.
Second, security researchers have been publishing exploits for unpatched vulnerabilities more frequently. Publications like these can generate significant fallout, since they potentially open the door for attackers to target unprotected systems.
Statistics on registered vulnerabilities
This section provides statistical data on registered vulnerabilities. The data comes from Kaspersky’s vulnerability knowledge base, which draws on the CVE database as well as the Russian BDU database and GitHub Advisory (GHSA). As a result, the figures for previous reporting periods may differ from those published in earlier reports.
We examine the number of registered vulnerabilities for each month over the last five years. As the chart below shows, this number continues to surge, a trend reflected across all the databases we track. It’s driven primarily by the widespread adoption of AI tools: as we predicted in our previous report, these tools have played a major role in the discovery of vulnerabilities in third-party software. Meanwhile, these tools often contain security issues of their own. For example, OpenClaw, a popular AI project, ranked 12th among those with the highest number of vulnerabilities discovered and published in Q2, with over 200 CVEs registered during the reporting period. Finally, AI development tools are also contributing to the vulnerability landscape, since the quality of the code they produce can vary widely. Therefore, the rate at which new vulnerabilities are discovered will inevitably keep growing.
Total published vulnerabilities per month from 2022 through 2026 (download)
Next, we analyze the number of new critical vulnerabilities (CVSS > 9.0) over the same period.
Total critical vulnerabilities published per month from 2022 through 2026 (download)
As the chart shows, the number of published critical vulnerabilities jumped sharply in Q2. This is because using AI for vulnerability research makes it possible to analyze massive amounts of previously unexamined code, uncover new attack surfaces, and identify entire classes of vulnerabilities that have gone unnoticed for decades. In particular, AI was used to find a series of Dirty Frag vulnerabilities in the Linux kernel.
Exploitation statistics
This section presents statistics on vulnerability exploitation for Q2 2026. The data draws on open sources and our telemetry.
Windows and Linux vulnerability exploitation
Q2 2026 saw a new precedent in the publication of vulnerabilities in Windows components and exploits for these: researchers no longer waiting for CVE registration, let alone patches. A case in point: a researcher who goes by Nightmare Eclipse (also known as Chaotic Eclipse) published a list of new “named” vulnerabilities across various Windows subsystems. At the time the technical details were published, none of the vulnerabilities had been assigned a CVE identifier:
BlueHammer: a local privilege escalation vulnerability in Windows Defender. During signature database updates, a time-of-check to time-of-use (TOCTOU) race condition occurs, allowing an attacker to substitute the directory where temporary update files are written. The researcher published a fully functional exploit for the vulnerability.
RedSun: another logical vulnerability in Windows Defender with a working exploit. Suspicious and malicious files marked as “cloud” can be overwritten or restored to their original directory with elevated privileges. The exploit incorporates fragments of algorithms that make it possible to leverage various logical vulnerabilities in Windows, effectively combining a large number of popular exploitation techniques.
YellowKey: a vulnerability that lets the user bypass BitLocker full-disk encryption and access system data through the Windows Recovery Environment (WinRE). A fully functional exploit was also published.
GreenPlasma: a vulnerability that enables system object injection via the CTF loader for the Collaborative Translation Framework (CTFMON) service in Windows. The original publication included an exploit with limited functionality.
RoguePlanet: yet another Windows Defender vulnerability that, like BlueHammer, stems from a TOCTOU issue, this time in the engine responsible for real-time system scanning. The published exploit uses the vulnerability to overwrite the system file wermgr.exe with a malicious one.
UnDefend: another vulnerability in the Windows Defender service. This time, the exploit causes a denial of service and blocks updates.
Even though such cases remain isolated for now, we believe they’ll grow into a full-fledged trend. Early publication of exploits gives attackers an advantage over software developers, who are left with no time to fix the issues.
Veteran vulnerabilities in Windows software also remain relevant. These are the ones our solutions most frequently detect exploits for:
CVE-2018-0802: a remote code execution (RCE) vulnerability in the Equation Editor component
CVE-2017-11882: another RCE vulnerability also affecting Equation Editor
CVE-2017-0199: a vulnerability in Microsoft Office and WordPad that allows an attacker to gain control over the system
CVE-2023-38831: a vulnerability in WinRAR that involves improper handling of objects within an archive
CVE-2025-6218 (formerly ZDI-CAN-27198): another WinRAR vulnerability allowing the specification of relative paths to extract files into arbitrary directories, potentially leading to malicious command execution
CVE-2025-8088: a vulnerability similar in exploitation method to CVE-2025-6218. The attackers used NTFS Streams to circumvent controls on the directory into which files are being unpacked
The vulnerabilities listed here can be leveraged to gain initial access to a vulnerable system and for privilege escalation. This underscores the critical importance of timely software updates.
That said, the number of Windows users who encountered exploits declined slightly in Q2, hitting an 18-month low.
Dynamics of the number of Windows users encountering exploits, Q1 2025 – Q2 2026. The number of users who encountered exploits in Q1 2025 is taken as 100% (download)
Linux also hit a rough patch in Q2 2026. Specifically, the period saw the disclosure of the Dirty Frag family of vulnerabilities, which lets an attacker reliably escalate privileges within the operating system.
All the vulnerabilities published in Q2 2026 were, in one way or another, related to the Linux caching subsystem. Here are the ones being most actively exploited:
CVE-2026-31431 (Copy Fail): a local privilege escalation vulnerability in the Linux kernel that lets an unprivileged user modify the page cache and gain root privileges. Especially dangerous for cloud and containerized environments
CVE-2026-43284, CVE-2026-43500 (Dirty Frag): a family of vulnerabilities in the Linux networking subsystem (IPsec ESP and RxRPC) that lets a local user overwrite the page cache and escalate privileges to root
CVE-2026-46300 (Fragnesia): a local privilege escalation vulnerability in the Linux kernel related to packet fragment handling and the page cache mechanism. It lets an unprivileged user gain root privileges and is also classified as part of the Dirty Frag family
CVE-2026-31635 (DirtyDecrypt): a Linux kernel vulnerability that lets a local attacker escalate privileges due to improper handling of decryption operations and page cache data modification
CVE-2026-43494 (PinTheft): a Linux kernel vulnerability that lets a local user gain elevated privileges due to errors in the memory page pinning mechanism
CVE-2026-46331 (pedit COW): a vulnerability in the Linux kernel’s traffic control subsystem (tc-pedit) that exploits a flaw in copy-on-write to modify the page cache and subsequently escalate privileges to root
The vulnerabilities described above were quickly embraced by attackers. At the same time, our solutions continue to detect exploitation attempts targeting older vulnerabilities as well:
CVE-2022-0847: a vulnerability known as Dirty Pipe, which enables privilege escalation and the hijacking of running applications
CVE-2019-13272: a vulnerability caused by improper handling of privilege inheritance, which can be exploited to achieve privilege escalation
CVE-2021-22555: a heap out-of-bounds write vulnerability in the Netfilter kernel subsystem
CVE-2023-32233: another Netfilter subsystem vulnerability that allows for Use-After-Free conditions and privilege escalation through improper processing of network requests
Dynamics of the number of Linux users encountering exploits, Q1 2025 – Q2 2026. The number of users who encountered exploits in Q1 2025 is taken as 100% (download)
In Q2 2026, the number of Linux users who encountered exploits declined slightly compared to Q1. Given that a significant share of new vulnerabilities are tied to the operating system’s caching subsystem, we recommend installing patches as quickly as possible, or disabling vulnerable kernel modules if patching isn’t an option.
Most common published exploits
The distribution of published exploits by software type in Q2 2026 includes categories that haven’t appeared in the sample for a long time. For instance, we’re once again seeing exploits targeting SharePoint. It’s worth noting that while several vulnerability write-ups for Exchange and SharePoint were published during the quarter, most turned out to be fake, AI-generated research. While the articles and exploit source code themselves look fairly polished, they describe nonexistent problems in the software or its components — often close to genuinely vulnerable mechanisms — in order to mislead researchers. This type of attack is aimed at increasing the time it takes to detect real vulnerabilities. In some cases, the description of a nonexistent vulnerability came bundled with completely unrelated malware.
Distribution of published exploits by platform, Q1 2026 (download)
Distribution of published exploits by platform, Q2 2026 (download)
Vulnerability exploitation in APT attacks
We analyzed which vulnerabilities were exploited in APT attacks during Q2 2026. The rankings provided below include data based on our telemetry, research, and open sources.
TOP 10 vulnerabilities exploited in APT attacks, Q2 2026 (download)
In Q2 2026, a trend emerged in APT attacks toward exploiting new vulnerabilities right from the moment they’re published. As before, we’re also seeing a large number of zero-day vulnerabilities. The Langflow vulnerability deserves particular attention: it’s one of the first cases of an APT group exploiting AI technology, which many organizations are only just beginning to integrate. Because most of this tech is proprietary, it has a considerable number of security blind spots. Therefore, given the growing number of AI-based automation tools, we strongly recommend going beyond the usual patching and developing secure procedures for credential use and sensitive data handling in systems that rely on agents and LLMs.
C2 frameworks
In this section, we examine the most popular C2 frameworks used by APT groups and analyze the vulnerabilities targeted by the exploits that interacted with C2 agents in APT attacks.
The chart below shows the frequency of known C2 framework usage in attacks during Q2 2026, according to open sources.
TOP 10 C2 frameworks used by APTs to compromise user systems, Q2 2026 (download)
Sliver, Havoc, AdaptixC2, and Metasploit remain the most widely used C2 frameworks. After studying open sources and analyzing samples of malicious C2 agents that contained exploits, we determined that the following vulnerabilities were utilized in APT attacks involving the C2 frameworks mentioned above:
CVE-2026-35273: a vulnerability in Oracle PeopleSoft PeopleTools that security vendors classify as server-side request forgery (SSRF). The details of the vulnerability have never been disclosed, although some research covers the post-exploitation steps
CVE-2023-46604: an insecure deserialization vulnerability in Apache ActiveMQ that allows arbitrary code execution in the context of the service process
CVE-2024-12356 and CVE-2026-1731: command injection vulnerabilities in BeyondTrust software that allow an attacker to send malicious commands even without system authentication
CVE-2023-36884: a vulnerability in the Windows Search component that allows commands to be run on the system, bypassing the mark-of-the-web (MoTW) mechanism
CVE-2025-53770: an insecure deserialization vulnerability in Microsoft SharePoint that allows for unauthenticated command execution on the server
CVE-2025-8088 and CVE-2025-6218: similar directory traversal vulnerabilities in WinRAR that allow files to be extracted from an archive to a predetermined path, potentially without the archiving utility displaying any alerts to the user
These vulnerabilities show that attackers used them for initial access and privilege escalation on vulnerable systems, setting the stage for launching a C2 agent. They include both zero-day vulnerabilities and fairly well-known security issues.
LLM/AI tool vulnerabilities
This section analyzes data published in Kaspersky’s vulnerability knowledge base. We reviewed the Q2 2026 version of the knowledge base.
As mentioned above, AI tools, plugins, and technologies have proven fairly effective at automating the search for problematic code and anomalous behavior. The high speed at which new vulnerabilities are being discovered has naturally created a need to fix them just as quickly. AI is often used for this too, which increases the volume of code being generated. However, neither code written without human involvement nor AI-generated advice is always correct.
The chart below covers registered vulnerabilities in AI tools for 2025–2026.
Number of published vulnerabilities in LLMs, AI tools, and plugins with similar functionality, 2025–2026 (download)
As the charts show, AI tools are racking up a substantial number of registered vulnerabilities, and that number keeps growing quarter over quarter. It’s also worth looking at how AI tool vulnerabilities break down by type, according to the CWE system:
TOP 6 vulnerability types in products that implement or use AI/LLM logic, 2025–2026
Interestingly, vulnerabilities of an undetermined type have ranked first in every quarter since the start of 2025. Traditionally-made software has the same issue, and it doesn’t look like the growing number of AI tools will fix it. It’s also notable that the list includes classes CWE developers themselves don’t recommend using for vulnerability classification, since they lump together a whole range of more specific types. CWE-284 is an example of this.
Looking at the most common classes, the key issues found in AI-related software can be summed up as follows:
Inadequate access control over critical system objects
Improper implementation of authentication and authorization mechanisms
Injections
It’s worth noting that injection-related vulnerabilities were relatively rare before AI agents took off (previously, they mostly affected web apps). Recently, though, these security issues have become relevant again.
Looking back at a year and a half of the AI boom, one conclusion stands out regarding registered vulnerabilities: AI tool developers are more focused on expanding functionality than on security. This is worth keeping in mind when using these tools. Let’s look at the projects and applications that either integrated AI tools or offered them as the core product. Below is a list of the those with the highest number of registered vulnerabilities for 2025–2026.
TOP AI/LLM-related projects by number of published vulnerabilities, 2025–2026 (download)
Notable vulnerabilities
This section highlights the most significant vulnerabilities published in Q2 2026 that have publicly available descriptions. Since the above already covers several significant vulnerabilities published during the reporting period, this section consists mainly of LLM/AI tool vulnerabilities.
CVE-2026-25253: a gatewayUrl vulnerability in OpenClaw
The issue stems from the fact that the OpenClaw user interface trusts the value of the gatewayUrl parameter passed in the URL and automatically establishes a WebSocket connection to the specified address. During this connection process, it sends an authentication token without any additional user confirmation.
The attack algorithm exploiting this vulnerability works as follows:
The application obtains a critical connection address from an external source (the gatewayUrl URL parameter), which is controlled by the attacker.
There is no validation before use.
The client automatically initiates a connection to the address specified in the parameter, which belongs to the attacker.
While connected, the application sends credentials (an access token) to the specified address.
If the attacker obtains a valid token, the consequences depend on that token’s level of access within the system. In general, this could lead to:
User session compromise
Execution of operations on the user’s behalf
Modification of the AI agent configuration
Unauthorized access to tools and resources connected to the agent
Under certain OpenClaw configurations, further compromise of the host running the agent
It’s worth noting that the risk of exploitation arises from a combination of several factors: the automatic connection and token transmission, the lack of address trust verification, and the high privileges granted to the local AI agent.
CVE-2026-41948: a path traversal vulnerability in the Dify AI platform
The vulnerability lets an authenticated user craft a request that enables the application to escape its permitted tenant and gain access to internal REST APIs that weren’t meant for that user. The root cause is insufficient normalization and validation of the URL path before it’s passed to the internal service.
Depending on the Dify configuration, the consequences can include:
Unauthorized access to internal service interfaces
Breach of isolation between workspaces
Exposure of internal service information
Conditions favorable to further attacks when combined with other vulnerabilities
The use of Dify in enterprise AI platforms is particularly risky, since internal services there tend to hold elevated privileges.
CVE-2026-45386: an improper access control vulnerability in Open WebUI
In Open WebUI, pin/unpin operations on messages are write operations, since they modify that message’s metadata (is_pinned, pinned_by, pinned_at). In vulnerable versions, however, before performing these actions, the API only checked for read access to the channel (a chat between a user or group and the AI) containing the message, not permission to modify its content. As a result, a user with a role limited to viewing messages could still change a message’s pinned status.
The vulnerability’s mechanism works as follows:
The user initiates an action that changes the state of an object.
The application treats this action as a regular read request.
Only channel view permission is checked.
The application performs a write without verifying the required user authorization.
This violates one of the fundamental principles of access control models — namely, that any operation that changes the state of data must be checked for the appropriate write or moderation permissions, regardless of whether the object itself is readable.
Although the vulnerability doesn’t lead to arbitrary code execution or compromise of sensitive data, it can affect data integrity and collaborative workflows. Potential consequences of exploitation include unauthorized pinning or unpinning of messages, disruption of channel moderators’ and administrators’ activities, changes to the display order of important information, and even the potential spread of false or misleading information by altering the channel containing a pinned message.
Open WebUI is widely used as an interface for interacting with local and enterprise LLMs. In these systems, pinned messages often contain important instructions, announcements, or tips for users. The ability to modify them with minimal privileges can disrupt collaborative workflows, cause confusion, and undermine trust in information published by administrators and moderators.
CVE-2026-45501: a vulnerability in Microsoft Exchange
The vulnerability stems from improper neutralization of user input when generating Exchange web pages. As a result, the browser may interpret specially crafted data as active content instead of plain text.
Although Microsoft categorizes the potential impact of exploiting this vulnerability as spoofing, flaws like this can lead to alteration of displayed content, imitation of trusted interfaces, actions on behalf of the user within an active session, and abuse of user trust.
It’s worth noting that issues like this are still relevant in modern software, given that mechanisms like Content Security Policy and various parsers were specifically created to help developers neutralize dangerous parts of user page content.
Conclusion and advice
Q2 brought the first significant results of AI automation adoption in software development and vulnerability hunting tools. This research shows that beyond traditional patch management, organizations now need real-time monitoring of systems and access controls, since infrastructure and everyday applications now contain far more AI functionality that could lead to compromise.
Accordingly, besides quickly detecting infrastructure vulnerabilities and managing security patches, modern enterprise-grade security solutions need to provide a broad range of preventive measures for tracking the overall health of systems and workstations. Kaspersky Next meets these requirements by combining proactive mechanisms with the ability to respond promptly to emerging threats.
In Part 1 of this series, we dumped a pile of uncomfortable questions on you and promised answers. In Part 2 of the series, we talked about why 1990s-2000s alert triage must die.
The core thesis, if you recall: if you add AI agents into a legacy, swivel-chair SOC structure, you are essentially building a robotic horse pulling an 1850 buggy. Sure, it saves on hay. It probably costs more in tokens.
2003 SOC + AI = somewhat better 2003 SOC.
That’s it. That’s the ceiling. So today we continue answering the questions and plotting this course.
The Hard Problem Nobody Markets: Local Context
Here is the dirty secret of every AI SOC deployment: the model (well, not just the model, but the entire system) is brilliant at general security knowledge and clueless about your environment. What is normal for your finance team in mid July? Which “server talking to the internet” is a shadow-IT disaster versus a legitimate — if a bit odd — business process? How engineering workloads talk to the outside when the code is being pushed to prod? All these matter for detection signal analysis.
“Make tribal knowledge machine-consumable” is what CMDB, ASM/CASM, asset inventory, and many expert opinions have promised and not delivered. If the AI is a robotic horse pulling your legacy 1850 buggy, ignoring Local Context is why it’s still stuck on the same dirt road...
What is actually different now — and what we would actually try:
Harvest context from the investigation loop itself. When the agent hits an unknown, it should not just escalate; it should ask a specific question (“is svc-etl-07 expected to authenticate from Ireland?”), and the human answer should be captured as a durable, attributed context object — not buried in case notes. Your SOC generates hundreds of these decisions a week today and, essentially, throws all of them away. This is the one genuinely new mechanism agentic AI brings to the context problem: the machine can now ask, at scale, in context, at the moment the answer is cheap to give.
“Context as code” (ha, I just made it up!), with owners and expiry. Context objects get a source, an owner, a confidence, and a review date. “The finance file server talks to this SaaS” is true until it isn’t. Unowned context is a future false negative with a countdown timer.
“Buy” context from the business, not from the SOC. App owners answer questions about their apps far better than analysts guessing from netflow. Route unknowns to them, in their tools, with a 24-hour SLA — and track answer rates as an org-health metric. This has worked in some places in regards to DLP alerts (I recall these conversations in my Gartner days, it also worked for some elite teams in general)
Past cases as curated reference — with a promotion process. Somebody must authoritatively designate “this case was handled correctly; AI, learn from this. That one? Never speak of it again.” Make it a real workflow: two-person promotion, provenance, expiry, re-certification, and the ability to revoke a reference case and re-run everything that leaned on it.
Measure context coverage. What fraction of investigations were completed without an unresolved unknown? That number is your real AI SOC readiness score, and it is far more honest than any maturity model.
This is also why “fully automated detection engineering” remains, in our view, a hybrid effort: the dependency on local, inconsistent, poorly-documented environment context makes pure machine DE a fantasy for now. Machines draft; humans anchor to reality.
Now, these context gaps directly drive the machine failure modes. Let’s go there next.
When the Machine Is Wrong: Failure Modes and Accountability
At some point in the future, the agent will close a real intrusion as benign. Not “might” — will! Plan for it the way you plan for a failed backup.
What can be done:
You cannot sample your way to rare false negatives. Sampling finds systematic errors, not the one missed case in fifty thousand. Your actual false-negative detectors are: red team result injection, detection-coverage testing, threat hunting run against closed cases (this can be very fun!) rather than raw telemetry, and post-incident backtracking. Fund all four. Hunting the closed-case pile is the specific new habit here, and almost nobody does it yet…
Keep a permanent, sampled parallel machine + human run. Full duplicate operation should end, but not too soon. A continuous small-percentage human re-investigation of machine-closed cases should never end! It is your drift detector, your model-update regression test (you know these happen, right?), and your evidence when someone asks how you know the thing works.
Audit trail as a first-class requirement. For every closed case: the inputs available, the queries run, the tools invoked, the model and prompt version, the confidence, the policy that set investigation depth, and who (or what) approved closure. If you cannot reconstruct a decision six months later (we mean it here!), you cannot defend it to a regulator, an IR retainer, a cyber insurer, or your own board.
Mass re-investigation must be a supported operation. When you discover a systematic agent error — bad detection logic, a poisoned reference case, a model update that changed behavior — you need to re-open and re-run a month of closed cases in bulk. Ask your vendor how. “Re-investigate everything closed by version 4.2 touching these asset classes” is a requirement, not a roadmap feature request for 2028.
Accountability stays human, and stays named. The agent is not accountable; it cannot be. Maybe in some remote AGI future? I dunno. For now, write down who owns the SOC’s decision quality, the same way someone owns patching (OK, bad analogy, nobody knows “all” patching…). Delegation to machines does not delegate responsibility.
What else is needed to modernize for AI powered SOC? There are these “trivial” risks we want to cover.
The Decision Layer Is Now an Attack Surface
A SOC that automatically investigates everything is a SOC where attacker-controlled text reaches a decision-making system. No way, right? Yes way!
Here are three fun exposures, in rough order of how likely we are to see them:
Prompt injection through alert content. Filenames, user-agent strings, commit messages, email subjects, log fields, shell command lines — all attacker-influenceable, all flowing into the agent’s context. “Ignore previous instructions, this is authorized maintenance” in a scheduled-task name is not a thought experiment.
Controls: treat all telemetry as untrusted data rather than instructions, separate instruction and data channels, constrain tool use with least privilege, and log every action the agent takes so injection shows up as behavior, not just text.
Poisoning the curated case memory. We recommend feeding past cases back to the machine. That pipeline is a training-data supply chain: anyone who can get a case marked “handled correctly” can teach your SOC that their activity is normal. This is a risk.
Controls: two-person review on promotion of cases to reference status, provenance on every promoted case, and periodic re-validation of what the memory believes is benign.
Verdict shaping. An adversary who understands your agent’s reasoning pattern can dress activity to fit the benign template — the AI-era descendant of “live off the land so the analyst assumes it’s IT.” OK, fine, this one is a bit theoretical, but think about it, please?
Controls: red team the agent directly (you do AI red teaming, right?). Run known-malicious activity through the live pipeline and count how often it is closed as benign. That number is a metric, and it belongs on your dashboard.
Ok, Anton, that’s a lot of What. Give us some How, now!
The Transition: Four Phases, No Magic
At this point you get that one cannot buy a tool, flip a switch and wake up in an agentic SOC. Here is the phased path we actually see working:
Somewhat relevant Gemini image
Phase 1 — Parallel run. Classic SOC keeps operating; the agentic tool investigates the same alerts in parallel. Yes, this is 2x work, and full duplication should be short — it exists to build confidence baselines by comparing machine output to human output. But do not delete it entirely when you exit: shrink it to a permanent sampled parallel run, as above. The mistake is a permanent full shadow SOC, not permanent measurement.
Phase 2 — Implanted agentic tasks. Classic process remains, but discrete alert subsets get routed to the AI: phishing first (hello, everyone who failed to automate this with SOAR!), then EDR alerts, then identity, then network. SIEM or SOAR sends the artifact to the AI SOC; results flow back into your case management. This phase runs for months, expanding scope as trust grows. Expand on evidence — measured agreement rates, canary catch rates, purple-team results per alert class — not on vibes or vendor roadmap.
Phase 3 — Exception-driven operations. Full automation for the majority of investigations. Humans set investigation-depth criteria (and token budgets), act as the final validation boundary, and handle the explicit “unknown / inconclusive / hand-to-human” bucket. This is the humans decide what machines do phase — the real agentic SOC.
Phase 4 — Full auto with broad automatic remediation. Let’s be honest: today this is mythical for most environments. Aspire, but don’t promise it to your CISO with a date attached. This is frankly mythical as of now, unless your environment is very modern, very predictable and you are very, very lucky…
Sound familiar? It should — this is the SOAR lesson replayed. Organizations that used SOAR only for enrichment or only for phishing got stuck in a permanent Phase 2 and called it transformation. Don’t repeat that with better marketing.
Next up: how SOC metrics must change when volumes and closure rates stop mattering — decision quality, investigative cycle time, escalation rates, canary catch rates, cost per investigation, AI error budgets — and how to run the human-to-AI feedback loop so corrections actually improve future performance instead of vanishing into the void. Stay tuned! This one may take a while…
This is Part 3 of the AI-ready SOC series (Part 1, Part 2), and it is focused on validating readiness for pillars #1 (SOC Data Foundations) and #4 (Modern SOC Technology Stack). Specifically, it is about the audit I promised in Part 2:
“The ‘API or Die’ Data Audit: You need to audit every critical data source to ensure it has a robust, well-documented API. An ‘enthusiastic’ AI agent will query your systems at a frequency no human ever could. If your CMDB or logging tier can’t handle the load, the agent won’t just fail; it might unintentionally DoS your internal infrastructure.”
Steampunk SOC again!
BTW, I now have too many blog series, so let me deconflict this here:
Series 1 Focused on assessing your overall SOC readiness for AI arrival:
If we’ve learned anything from the last decade of SOC evolution, it’s that manual is the enemy of scale.Remember toil? When you introduce AI agents into your workflow, they don’t “click buttons” in a UI like a human analyst — they consume APIs (and use CLI like humans, I guess). Yes, agents can screen-scrape and click around, but you probably don’t want to burn GPU cycles teaching a frontier model to navigate your SIEM’s 2009-vintage web UI (not a joke, I saw this happen!). But, yes, that’s a party trick, not an architecture.
If your telemetry sources have weak, poorly documented, limited-capability, or aggressively rate-limited APIs, your expensive AI agent is essentially a Formula 1 driver stuck in a traffic jam behind a horse buggy. It has the horsepower (Ha! I got a pun! Take that, Tim) to win, but it has nowhere to go.
And here is the scarier version: a human analyst queries a SIEM maybe 10 times an hour. An AI agent might query it 100 times in 30 seconds to correlate one alert (agents looove to brute force, as we all know). Multiply by a batch of alerts during an incident, and your “AI SOC transformation” becomes a self-inflicted DoS on your own EDR, GRC, or CMDB (if you have that thing). Fun times!
So before you buy the shiny agentic thing, audit your primary telemetry and context sources (EDR, NDR, cloud logs, identity, CMDB, ticketing, etc.) with cold, cruel eyes (Claude wrote this, I am sure its eyes are very cold…). Here is how.
So before you buy the shiny agentic thing, audit your primary telemetry and context sources (EDR, NDR, cloud logs, identity, CMDB, ticketing, etc.) with cold, cruel eyes (Claude wrote this, I am sure its eyes are very cold …). Here is how.
Phase 0: The “Cold Eyes” Inventory
Don’t just list your tools; list your data paths.
The goal: Identify every system an analyst touches during a typical investigation (SIEM, EDR, CMDB, identity, DHCP logs, ticketing, that one Wiki page everybody swears by, etc.).
The test: If an analyst has to “swivel-chair” — copy-paste from one tab to another because there is no integration — that is a Priority 1 gap. Whatever the human bridges manually, the agent cannot cross at all.
The how: Start by interviewing your SOC / D&R analysts and shadowing them during a real investigation. Review SIEM/EDR query logs to see which systems are consistently touched. Check SSO/IAM logs to see where they authenticate. Scan your internal wiki or shared drives to identify those “secret” cheat sheets or side-tools they rely on. If they are swivel-chairing (hi, 1990s SOC!), you will find the proof in their browser history or their documented SOPs.
Now run every item on that list through the five tests below.
Test 1: Connectivity & Accessibility (The “Can I Even Get There?” Test)
An AI agent needs a direct, programmatic path to the data. If a human has to “export a CSV”, “log into a separate portal”, or, worse, “Slack another human”, that data source is dead to the AI (agent UI scraping is doable, but sad and not scalable or reliable).
Machine-readable documentation: Does the tool have a public (or well-documented internal) REST/gRPC API, or an MCP server? If the only way to learn the API is “emailing a support engineer,” you’ve already failed. (And no, MCP is not magic — it’s a protocol, not a personality transplant for your legacy tool.)
Modern authentication: Does it support secure, programmatic auth (OAuth2, OIDC, scoped API keys, workload identity)? If it requires a “service account” with a static password and no MFA, congratulations, your AI enablement project just became a security liability. Remember vulnerability scanners that wanted an admin password for all systems back in the 2000s to do authenticated scanning?
Network pathing: Can your AI orchestration layer — wherever the agent actually lives — reach the endpoint without weeks of firewall hair-pulling? This seems trivial for 2026, but I assure you it is anything but. And also: if this is too easy, perhaps you have a 1990s flat network?
Test 2: Performance & Throughput (The “Agentic Load” Stress Test)
This is where most “legacy” security tools break, and where you must test before an incident tests it for you.
Concurrency limits: What is the maximum number of concurrent API requests the tool allows? If the answer is “one,” your agent is going to be very lonely. Keep in mind, agents like to swarm (OK, yours may not yet, but this is coming).
Rate limiting: When exactly do the “429 Too Many Requests” errors start flying? Or, worse, the response is there, but the data is 1–99% incomplete? Will your threat intel provider cut you off the moment an agent starts enriching a batch of 23000 IPs because it, well, felt like it? Find the burst capacity of your stack now, on your terms. Don’t wait until you need it.
Response latency: Measure time-to-first-byte on realistic queries. If a simple process tree query takes 45 seconds, your agent times out, retries, times out again… and your MTTR goes up… if you are lucky. Or, something else breaks, if you are not. “Multi-hour data queries” (hi again, the 1990s!) are an automatic fail here.
Test 3: Data Quality & Schema (The “Context Fidelity” Test)
Having an API is step one. Having useful data at the end of it is step two. AI agents are only as smart as the context they can read when they need it — GIGO is still law! BTW, the agentic spin on GIGO is of course being “confidently wrong.”
Structured output: Does the API return JSON or YAML? If your legacy ticketing system returns a 2MB “stream of consciousness” text (well, text-ish) blob, your AI will burn tokens (and, thus, your money) just trying to find the root cause. Force structured entry at the source (yes, this is the case management revamp from Part 2, and yes, this is painful and not fun at all).
Schema stability: Is the API versioned? If the vendor silently renames src_ip to source_address, your agent logic breaks instantly and quietly (OK, this is not fair, a smarter model will in fact figure this one out… but it will cost ya!). Quiet breakage in a SOC is the worst kind.
Joinable fields: Does the telemetry include correlation keys (cloud_instance_id, user_sid, asset IDs) so the agent can pivot to the next tool without guessing? Agents that guess entity resolution are agents that hallucinate incidents…
Test 4: Functional Depth (The “Can It Actually Do Work?” Test)
An AI-ready API shouldn’t just be for reading data; eventually it is for taking action — with the human/agent handoff lines you drew in Part 2 firmly in place. Due to vulnerability apocalypse fears, a lot of vendors started to promise automatic remediation, and guess what? This means needing APIs to act on systems.
Read/write balance: Can the API perform response actions — isolate a host, disable a user, update a rule? Read-only APIs give you an AI-powered observer, not an AI-augmented SOC of the future.
Granular scoping: Can you give the agent least-privilege access? “Read all logs” but “isolate only these subnets”? If the tool’s permission model is “admin or nothing,” that’s a hard stop for autonomy.
Server-side filtering: Does the API support filtering at the source (?status=active&severity=high)? If the agent must pull 10,000 records to find 5, you’re paying token tax on the vendor’s laziness. But hey, somebody is getting rich…
Test 5: The Auth & Agent Identity Layer
Agents need keys — and keys need governance. Free wisdom from the 2010s, I guess. The 1990s are finally over!
The audit question: Do you have a centralized way to manage API credentials and identities for your agents — as workload identities with registration, ownership, rotation, and revocation? Let me guess … mmmm … the answer is ‘no’?
The risk: “Shadow AI” starts the day a helpful analyst hands their personal API key to an LLM to “help out.” Your audit must define how agents authenticate, how permissions are scoped, and who owns each agent identity (yes, really). If you can’t answer “which agent did this and on whose behalf?”, you are not ready for an agentic SOC.
The “Agentic Readiness” Scorecard
For every primary tool, assign a score:
Level 1 — The Dinosaur: No API. UI-only. (Status: replace, or accept it’s invisible to your AI. Or suffer and pay for scraping the UI “agentically”)
Level 2 — The Relic: Basic API, poorly documented, slow, falls over under load. (Status: high risk)
Level 3 — The Standard: Decent REST API and docs, but read-mostly, limited response capabilities. (Status: usable with additional tools?)
Level 4 — The Modernist: Robust, fast, versioned APIs with real write capabilities and granular RBAC. (Status: AI-ready)
Level 5 — The Agent-First: Native agentic support (MCP, high concurrency, agent-aware auth, feedback loops). (Status: the gold standard, and yes, these exist in 2026)
Where You Arrive: The Binary Map
When you finish, you shouldn’t have a “nice-to-have” list. You should have a binary map:
Machine-ready: API is fast, documented, structured, and scoped. Your agents can use it.
Legacy debt: No API, or a brittle one. These sources are invisible to your AI.
And here’s the uncomfortable conclusion: if a data source is invisible to your machine, it should probably not be part of your modern detection strategy. We are moving to a world where “if it isn’t via API, it didn’t happen.” By the way, those living in organizations with modern IT stacks are surprised this is even an issue worth discussing. But, I assure you, it is…
This audit is also, not coincidentally, foundational work for an engineering-led SOC and ASO: the same API-first plumbing that feeds your agents feeds your detection-as-code pipelines, your metrics (pillar #5!), and your humans too. Fix it once, win three times.
So: which of your “critical” tools is actually an API-less paperweight? Name and shame (or just vent) in the comments!
Analysis of the Aeternum botnet loader, a threat leveraging Polygon blockchain smart contracts for decentralized C2 infrastructure and payload execution.
Amazon Web Services (AWS) is pleased to announce that the Summer 2026 System and Organization Controls (SOC) 1 report is now available. The reports cover 185 services over the 12-month period from July 1, 2025–June 30, 2026, giving customers a full year of assurance. These reports demonstrate our continuous commitment to adhering to the heightened expectations of cloud service providers.
AWS strives to continuously bring services into the scope of its compliance programs to help customers meet their architectural and regulatory needs. You can view the current list of services in scope on our Services in Scope page. As an AWS customer, you can reach out to your AWS account team if you have any questions or feedback about SOC compliance.
In Part 1 of this series, we dumped a pile of uncomfortable questions on you and promised answers. The core thesis, if you recall: if you add AI agents into a legacy, swivel-chair SOC structure, you are essentially building a robotic horse pulling an 1850 buggy. Sure, it saves on hay. It probably costs more in tokens.
2003 SOC + AI = somewhat better 2003 SOC.
Gemini creation :-)
That’s it. That’s the ceiling.
So today we start answering the questions. And we start by attacking the most sacred cow of traditional security operations: the alert triage process.
Let’s Kill Triage. Seriously.
For a quarter of a century, the standard SOC pipeline has been carved in stone:
Detect → Triage → Investigate
Human L1 analysts sit in front of a flashing alert queue, spending 3–7 minutes per alert (and sometimes much more…) deciding whether something is a false positive or deserves escalation to somebody more senior (and more expensive…and just as human). We built this process for one reason and one reason only: humans do not scale (For the purist: OK, they do scale, but linearly with pay). Triage was a compromise born of “built-in” scarcity. We — obviously — never had enough human eyes to deeply investigate every signal hitting the SIEM, so we invented a cheap filtering step to ration the expensive investigation step.
Sometime in the 2010s, SOAR made triage easier, by first adding alert enrichment and then …. in many places, nothing more. In others, select alert types were triaged by the hard-coded playbooks.
Now, let’s do AI. It doesn’t get bored correlating IPs or summarizing logs at 3am. It doesn’t quit after 18 months to go do threat hunting somewhere else. Because machine scale allows comprehensive analysis of every signal, the triage step can just go and vanish.
The new pipeline collapses to:
Detect → Investigate.
Why spend minutes “skin-deep” triaging an alert to decide whether it deserves a look, when the machine can perform a full, deep investigation of 100% of your alerts? Gather the local context, pull the historical cases, map the artifacts, render a verdict with evidence — all before a human ever shows up.
For the impatient: the cost discussion is coming! Don’t freak out … just yet.
Wait — Can They Actually Do That Today?
Fair question, and here is where we owe you honesty rather than a slide.
Today’s “AI in SOC” ranges from “genuinely investigates” to “enriches beautifully then bullshits confidently.” The second one is an old SOAR chained to a language model aka the exact trap this blog warns about. If you cannot tell which one you bought, you probably bought the second one…
Our rough test for telling them apart, usable in a POV:
Does it ask new questions, or only pre-decided ones? Enrichment runs a fixed lookup list. Investigation forms a hypothesis, queries, reads the result, and changes what it asks next. Watch the query sequence, not the summary.
Does the conclusion move when the evidence moves? Feed it two near-identical alerts with one materially different fact. If the verdict does not change, you have a narrator.
Does it ever return “inconclusive”? A system with no uncertainty output has no calibration. Run away.
Does it show its work in a form a human can re-run? Queries, artifacts, timestamps — not just a paragraph asserting “no evidence of compromise.” OK, this is tricky, I admit.
Where does this leave the “kill triage” claim? Honestly: directionally right, unevenly available. For high-volume, well-bounded, evidence-rich alert classes — phishing, commodity EDR detections, identity anomalies — deep machine investigation of 100% is achievable now.
For multi-stage, low-signal, who-the-hell-knows-what-happened, context-heavy cases it is not, and anyone telling you otherwise is, ahem, exaggerating, to put it mildly. The pipeline collapse is real; the coverage is a rollout, not a switch.
Depth Gating: The New Triage Wears a Suit
So, if deep investigation is token-expensive — and it is, sorry! — then somebody, somewhere, is deciding how deep the machine goes on which alerts. We can call this decision A New Triage, while bending the truth a bit. It just moved from a human clicking a queue to a policy sitting in a config file, and pretending otherwise is how you end up with an unexamined control that quietly decides what you never look at. And, just as before, mistakes and decisions cost money.
So let’s examine it. Explicitly:
Who owns the investigation depth policy? Not procurement. Not “whoever set up the tool.” This is a detection-engineering artifact with a named owner, version history, and a review cadence.
Who owns the budget, and what happens when it runs out mid-month? If the honest answer is “all investigations get shallower,” you have just invented an availability attack against your own SOC. Define degradation behavior in advance: which alert classes keep full depth, what gets queued, what pages a human (do you still have said human handy?)
What is systematically under-investigated? Every gating rule creates a shadow. Write the shadow down. What gets triaged out? Review it quarterly against your threat model, not against your token bill. Well, OK, against both, really, but mostly vs the threats.
Are your thresholds guessable? If low-severity, off-hours, or particular-source alerts predictably get the cheap path, an adversary who learns that shapes activity to land there. Treat depth policy as security-sensitive configuration, not ops tuning.
Triage stops being a job and becomes a policy — and policies get attacked, drift, and rot. This is the broader theme of the whole series: humans move from doing the work to defining the rules for the work, which is harder, not easier, and needs the governance to match.
(And yes, “cost per investigation” becomes a real SOC metric — one that will fight with “detection coverage” in every budget meeting. More on the metrics carnage in a future part.)
So What Do the Humans Do?
Remember my favorite modern SOC question? “It’s 2030, you have a SOC, what do humans do?” If machines own frontline investigation for the vast majority of alerts, what happens to the people? Two dominant paradigms are emerging, and the answer for most organizations will be “both, in some mix”:
1. The Elite Threat Hunter Model. With the routine noise fully investigated by machines, humans are finally unchained from the queue. They pivot to hypothesis-driven hunting, deep-dive research, and the nuanced multi-stage attacker behaviors where AI (for now!) still struggle. Humans hunt; machines grind. Sorry, but “100% automated hunting” is not (today).
2. The Engineering-Driven SOC Model. This is our classic ASO mantra: humans build machines; machines do the work. Analysts evolve into detection and SOC engineers. Their day shifts from consuming alerts to building, tuning, testing, versioning and (yes) rolling back the AI logic and detection-as-code pipelines. Treat agents as engineering artifacts, not magic pets.
And the New L1 Is…
“OK, but classic L1 is dead. What do entry-level humans do?” OK, this is tricky! This is where a lot of “humanless SOC” enthusiasts embarrass themselves.
I think the new starter role is AI validation: sampling and reviewing AI-generated case files, validating the agent’s query logic against the data it actually had, hunting for hallucinated context and confidently wrong conclusions (got those?), and owning the “1% bucket” — the exceptions where the AI raises its digital hands and says “I don’t know, human, help me.” (Your AI SOC must have an explicit process for this bucket. If your vendor’s agent never says “I don’t know,” run.)
Now the two objections this role deserves, because “verify-and-validate is the new L1” is a slogan until you answer them.
Objection 1: where does the competence come from? Checking an agent’s homework requires knowing what good looks like — and L1s historically learned that by doing triage, badly, for a year. We removed the training ground and assumed the graduates.
So build the ground back deliberately:
Structured re-investigation as training. New analysts independently work a small set of already-closed cases without seeing the agent’s verdict, then compare. This is deliberate practice, and it doubles as an evaluation signal on the agent.
Curated case libraries as curriculum. The promoted-case archive is the best SOC textbook your org will ever have, sequenced from trivial to nasty. Use it as onboarding, not just as machine memory. You have AI, use it!
Rotation into hunting and detection engineering on a schedule, not “when someone has time.” Validation-only career paths produce validators, not investigators.
Objection 2: automation bias is real and it will eat your review process. Humans reviewing plausible, well-written machine verdicts approve them, every time. You do that, I do that (hey, I just did this with this blog sentence to illustrate this very point). You can’t order people not to. Well, you can order, but they won’t do it. This is one of the best-documented findings in human-automation research, and hoping your team is special is not a thing.
Design against it:
Blind review first. The reviewer forms a verdict before seeing the agent’s. Order matters more than effort here.
Canary cases. Inject known-bad cases with deliberately wrong agent verdicts into the review queue at a low rate. Measure catch rate. This measures the reviewers, and it is the only honest read on whether your validation layer is real.
Stratified, not random, sampling. Random sampling over a population that is 99% benign finds nothing. Oversample: agent-reported low confidence, unusual query paths, crown-jewel assets, first-time-seen behaviors, and anything closed suspiciously fast.
Incentives on catches, not throughput. If reviewers are measured on cases reviewed per shift, you have built a rubber stamp with a salary. Measure disagreements raised and misses found.
What’s Next?
Killing triage and re-blueprinting the humans is necessary but not sufficient. If your SOC still reports “alerts closed per analyst per shift,” you are measuring a process that no longer exists.
Next up: failure modes, local context pain, how SOC metrics must change when volumes and closure rates stop mattering …
The FTC alleges that the company shared consumers’ sensitive health information with third‑party advertising platforms despite promising strong privacy protections.
Hims & Hers is a telehealth and digital health platform that connects users with licensed medical providers for online consultations, prescription medications, and personal care products.
The complaint also accuses Hims & Hers of deceptive billing and subscription practices that made it hard for users to avoid charges or cancel subscriptions.
According to the FTC’s complaint, filed in federal court in California, Hims & Hers:
Shared sensitive health data, including details about medical conditions, with ad platforms such as Meta and Snap despite privacy promises.
Charged before consultations. The company promised users they could consult a medical provider before being charged, but the FTC says many consumers were enrolled in recurring prescription subscriptions shortly after they submitted an intake form, often without first having a consultation.
Made cancellation difficult. Before 2023, cancellation reportedly required contacting customer service by phone, email, or chat. Even after an online cancellation option appeared, the FTC alleges the button was hidden behind multiple steps and confusing options.
From a cybersecurity and privacy research perspective, this isn’t just about a single telehealth brand. It highlights three broader trends we see repeatedly in consumer programs:
Privacy policies versus reality. A company can market itself as privacy‑focused while still integrating third‑party advertising and analytics software development kits (SDKs) that leak sensitive information. This becomes especially concerning when health‑related events are linked to user accounts or tracking cookies.
Friction as a feature. Hard‑to‑find cancellation flows and unclear billing practices are examples of “dark patterns” that nudge users into paying for services they might not have chosen given all relevant information.
Regulatory pressure is growing. Health‑related services are under increasing scrutiny, especially when they handle sensitive data and combine it with advertising platforms.
The court will ultimately decide whether Hims & Hers violated the law, but the FTC’s action sends a clear signal: regulators are paying close attention to how health‑related services collect, use, and share sensitive data.
For anyone who values online privacy, the Hims & Hers case is a reminder that “health tech” does not automatically mean “privacy first.”
How to stay safe
More often than not, the privacy loopholes are hidden in the privacy policy somewhere.
Pro tip: one thing AI is good at is reading between the lines. Ask an AI chatbot to summarize a privacy policy and identify when your information may be shared with third parties. AI makes it much easier to understand lengthy privacy policies without reading every word yourself. If companies fail to follow their own privacy policies, regulators and consumers can hold them accountable.
Other than that:
Don’t share sensitive information unless it’s genuinely needed to provide the service.
Use strong, unique passwords and multifactor authentication (MFA). Even if a company is compliant, breaches happen. Unique passwords and two‑factor authentication limit the damage if your account details are exposed.
Your name, address, and phone number may already be for sale.
Data brokers collect and sell your personal details to anyone willing to pay. Malwarebytes Personal Data Remover finds them and gets your information removed, then keeps watch so it stays that way.
The FTC alleges that the company shared consumers’ sensitive health information with third‑party advertising platforms despite promising strong privacy protections.
Hims & Hers is a telehealth and digital health platform that connects users with licensed medical providers for online consultations, prescription medications, and personal care products.
The complaint also accuses Hims & Hers of deceptive billing and subscription practices that made it hard for users to avoid charges or cancel subscriptions.
According to the FTC’s complaint, filed in federal court in California, Hims & Hers:
Shared sensitive health data, including details about medical conditions, with ad platforms such as Meta and Snap despite privacy promises.
Charged before consultations. The company promised users they could consult a medical provider before being charged, but the FTC says many consumers were enrolled in recurring prescription subscriptions shortly after they submitted an intake form, often without first having a consultation.
Made cancellation difficult. Before 2023, cancellation reportedly required contacting customer service by phone, email, or chat. Even after an online cancellation option appeared, the FTC alleges the button was hidden behind multiple steps and confusing options.
From a cybersecurity and privacy research perspective, this isn’t just about a single telehealth brand. It highlights three broader trends we see repeatedly in consumer programs:
Privacy policies versus reality. A company can market itself as privacy‑focused while still integrating third‑party advertising and analytics software development kits (SDKs) that leak sensitive information. This becomes especially concerning when health‑related events are linked to user accounts or tracking cookies.
Friction as a feature. Hard‑to‑find cancellation flows and unclear billing practices are examples of “dark patterns” that nudge users into paying for services they might not have chosen given all relevant information.
Regulatory pressure is growing. Health‑related services are under increasing scrutiny, especially when they handle sensitive data and combine it with advertising platforms.
The court will ultimately decide whether Hims & Hers violated the law, but the FTC’s action sends a clear signal: regulators are paying close attention to how health‑related services collect, use, and share sensitive data.
For anyone who values online privacy, the Hims & Hers case is a reminder that “health tech” does not automatically mean “privacy first.”
How to stay safe
More often than not, the privacy loopholes are hidden in the privacy policy somewhere.
Pro tip: one thing AI is good at is reading between the lines. Ask an AI chatbot to summarize a privacy policy and identify when your information may be shared with third parties. AI makes it much easier to understand lengthy privacy policies without reading every word yourself. If companies fail to follow their own privacy policies, regulators and consumers can hold them accountable.
Other than that:
Don’t share sensitive information unless it’s genuinely needed to provide the service.
Use strong, unique passwords and multifactor authentication (MFA). Even if a company is compliant, breaches happen. Unique passwords and two‑factor authentication limit the damage if your account details are exposed.
Your name, address, and phone number may already be for sale.
Data brokers collect and sell your personal details to anyone willing to pay. Malwarebytes Personal Data Remover finds them and gets your information removed, then keeps watch so it stays that way.
Leveraging legitimate software is one of cybercriminals’ tactics of choice, with remote management tools ranking among their top tools. A recent example involves the remote administration utility ScreenConnect. It’s designed for IT support teams to troubleshoot systems and configure software seamlessly in the background. However, when weaponized by threat actors, ScreenConnect becomes a versatile attack vehicle used to harvest data, deploy malware, and move laterally across corporate networks.
During a recent incident detected by Kaspersky Managed Detection and Response, our experts identified an attempt to use ScreenConnect in an attack. This allowed a detailed study of how attackers used this application in a large-scale malware distribution campaign. The following breakdown illustrates the mechanics of ScreenConnect-assisted attacks, and outlines key strategies to defend your organization against them.
How ScreenConnect reaches target computers
In the campaign analyzed by our experts, the attackers bundled ScreenConnect with legitimate free business software. They established a network of phishing websites to spoof popular tools, including OBS Studio, DS4Windows, DNS Jumper, Glary Utilities, Bandizip, Process Hacker, and others.
These rogue websites featured high-quality designs that could be taken for the official pages, making them highly convincing to unsuspecting users. Once the victim clicks the download button for the software, an archive is downloaded to their computer that contains additional files alongside the requested application:
A legitimately signed Microsoft executable (exe), renamed to match the expected application installer (for example, OBS-Studio-Installer.exe)
A malicious library named res.1033.dll
An Assets directory containing installers for both ScreenConnect and the intended application
Launching the renamed file disguised as the app installer triggers DLL sideloading of a malicious library. This library silently runs the ScreenConnect installation without restarting the system, while using the standard Windows installer to set up the software the user originally tried to install.
The attackers used search engine optimization techniques to drive traffic to their fake websites. As a result, these malicious pages appeared at the top of search results for certain free software utilities on major search engines.
Our experts discovered over 90 domain names translated into more than 10 different languages. While most of these websites targeted English, Russian, and Chinese speakers, several domains catered to German, French, Spanish, Arabic, and other regional audiences.
A detailed analysis of the IP addresses and associated spoofed domains is available in our technical research article on Securelist, along with full indicators of compromise.
Why the attackers exploited ScreenConnect
In this campaign, attackers leveraged ScreenConnect to generate and execute malicious scripts on victim machines. These scripts served several key functions: they created exclusions for specific drives, directories, and processes within Windows Defender, disabled the User Account Control (UAC) security mechanism, and delivered and deployed AsyncRAT – a remote access Trojan.
To maintain persistence, the scripts configured a Windows scheduled task to run the malicious code at preset intervals. AsyncRAT then established a connection with the attackers’ command-and-control server to receive further instructions.
The primary objective of this campaign appears to be gaining unauthorized access to enterprise systems, likely to then resell it on cybercrime marketplaces.
How to secure corporate infrastructure
Although ScreenConnect in and of itself is a legitimate tool, its presence poses a security risk to corporate environments. Consequently, Kaspersky security solutions detect this application as not-a-virus:HEUR:RemoteAdmin.MSIL.ConnectWise.gen.
Security teams should implement the following controls:
Enforce strict application control policies, including software allowlisting and restrictions on MSI package installations from unverified sources
Monitor for newly installed remote management utilities and scheduled tasks
Filter outbound network traffic from workstations to unknown IP addresses and domains
by Dan “Haircutfish” Rearden | haircutfish.com | Guest Author Working in the SOC can be a grind. Whether triaging alerts, escalating to clients, or just trying to understand why users […]
When an administrator introduces a rule change in AWS Network Firewall and network connectivity is disrupted, pinpointing the cause requires inspecting multiple points in the traffic path. The firewall gives you stateless and stateful rule engines, domain rules, and routing to the firewall endpoint inside your Amazon Virtual Private Cloud (Amazon VPC). A network drop looks the same from the workload no matter where it started. Isolating the cause means correlating the alert and flow logs with the firewall configuration, route tables, and recent API calls in AWS CloudTrail that might have changed them. That manual correlation is exactly where AWS DevOps Agent helps, accelerating root cause analysis so you can restore connectivity in minutes instead of hours.
AWS DevOps Agent does that correlation for you. As your always-available operations teammate, it resolves and proactively prevents operational issues across AWS, multicloud, and on-premises environments. When an Amazon CloudWatch alarm triggers, it reaches the agent through a webhook. The agent then reads the firewall configuration and logs through AWS APIs, ties the drop to recent API activity, and returns a root cause with a mitigation plan you review before you apply it.
This post connects CloudWatch monitoring to DevOps Agent. It walks through three Network Firewall failures from end to end. The first is a domain deny list blocking a legitimate endpoint. The second is a stateless rule priority misconfiguration. The third is an asymmetric cross Availability Zone (AZ) routing drop. Each maps to a different layer, so each leads down a different investigation path. An AWS Cloud Development Kit (AWS CDK) app deploys the whole environment in your own account so you can reproduce each failure and follow along.
The sample workload
As part of this blog post, we provide a CDK stack that deploys both the AWS DevOps Agent Space and a sample workload used to walk through three separate troubleshooting scenarios. A single t3.micro instance in a protected subnet checks its connectivity to a test endpoint on a continuous loop and publishes results to CloudWatch. Traffic takes the internet egress path through Network Firewall, the NAT gateway, and the internet gateway, so the firewall can intercept or drop it. After completing the walkthrough, you can apply the same troubleshooting techniques with DevOps Agent against your own Network Firewall deployments.
The test endpoint runs in a separate VPC deployed by the same CDK app. It serves HTTPS on port 443 and TCP on port 9142, giving each scenario a different protocol layer to exercise: Scenario 1 targets a TLS connection on 443 (matched by Server Name Indication), Scenario 2 targets a TCP connection on 9142, and Scenario 3 exercises the whole egress path.
A live status page shows one card per scenario plus the network topology. The whole stack deploys from a single CDK app across two Availability Zones, each with a firewall endpoint and NAT gateway, which is what makes Scenario 3 possible.
As shown in the following figure, the egress data path runs from the workload through Network Firewall and the NAT and internet gateways to the test endpoint. The alarm pipeline runs from CloudWatch through Amazon Simple Notification Service (Amazon SNS) and the webhook AWS Lambda function to DevOps Agent.
Figure 1: The sample workload
To use this with your own workload, you need a CloudWatch alarm that detects the connectivity problem and the webhook pipeline (SNS topic and Lambda function) that delivers it to DevOps Agent. The agent reads your firewall configuration, logs, and CloudTrail through AWS APIs, so no additional instrumentation is needed on the firewall side.
AWS CDK 2.x is required. You can use it through the project’s npx dependency, or install it globally:
npm install -g aws-cdk
Deploy the sample workload
Clone the project and deploy it into us-east-1 with one command (set awsRegion to use another AWS Region).
git clone https://github.com/aws-samples/sample-accelerating-aws-network-firewall-troubleshooting-with-aws-devops-agent.git
cd sample-accelerating-aws-network-firewall-troubleshooting-with-aws-devops-agent
bash scripts/deploy.sh
The script checks prerequisites, installs dependencies, compiles and tests, and bootstraps the CDK if needed. It then deploys all the stacks from a clean baseline and prints the outputs, including the status-page URL and sign-in details.
Open the status-page link (an https://<random-id>.cloudfront.net address).
Sign in using the username and password provided from the CDK output and confirm all three cards show the green Healthy status.
Keep the page open while you run the scenarios.
Connect AWS DevOps Agent
To connect AWS DevOps Agent to the alarm pipeline
In the AWS DevOps Agent console, open the nf-devops-agent-space Agent Space created by the CDK deployment.
On the status page, choose Configure webhook, paste the URL and signing secret, and save. The page writes them to the nf-devops-agent-webhook-credentialsAWS Secrets Manager secret, so there is no AWS CLI or console step. Until you set it, the bridge Lambda function sees a placeholder and skips delivery.
Verify the path before you run a scenario. In the Lambda console, open nf-devops-agent-webhook and use the Test tab with this event.
{
"Records": [
{
"Sns": {
"Message": "{\"AlarmName\":\"TEST-webhook-verification\",\"AlarmDescription\":\"[TEST] Webhook integration test - not a real alarm.\",\"NewStateValue\":\"ALARM\",\"NewStateReason\":\"[TEST] Manual webhook connectivity test. Safe to ignore.\",\"Region\":\"us-east-1\"}"
}
}
]
}
A 200 response confirms the path, and a test investigation appears in the DevOps Agent Operator Web App view.
How the alarm pipeline works
Every scenario reaches DevOps Agent the same way. A CloudWatch alarm moves to ALARM and notifies the SNS topic. Amazon SNS invokes a Lambda function. The function reads the webhook URL and signing secret from Secrets Manager, signs an alarm payload, and POSTs it to the DevOps Agent webhook (as shown in Figure 1). Amazon SNS also provides delivery retries, fan-out to other subscribers, and cross-account publishing.
Prebuilt Network Firewall metric (Scenario 1) – Alarm-1 watches the DroppedPackets metric, summed across the stateful streams, and triggers when drops rise above a baseline threshold. This requires no workload or custom metric and works on an already-deployed firewall. However, it only tells you that the firewall is dropping packets, not which rule is responsible.
Application health metric (Scenarios 2 and 3) – Alarm-2 and Alarm-3 watch a custom metric from a connectivity check. Use this for an alarm tied to user-facing impact or to tell one traffic path from another, which requires running a component that emits the metric.
Alarm
Source
Triggers when
Alarm-1
Native AWS/NetworkFirewall DroppedPackets
The firewall’s dropped-packet count rises above the baseline
Alarm-2
Custom application health metric
The port 9142 (TCP) connectivity check to the test endpoint is being dropped
Alarm-3
Custom application health metric
The cross Availability Zone connectivity check is being dropped
Run the scenarios
Work through each of the scenarios one at a time, following the same cycle. Interrupt network connectivity, watch the alarm trigger, let DevOps Agent investigate, apply the recommended fix, and confirm recovery before moving on.
The status-page cards follow the live CloudWatch alarm state. A card shows a green dot and the word Healthy when its alarm is clear, and a red dot and the word DROPPED when its alarm triggers. In the DROPPED state the card also adds a Condition: line describing what’s being dropped, which isn’t shown when the card is healthy. Network Firewall applies changes to new flows, so a change shows within a minute or two. Recovery comes from the mitigation DevOps Agent recommends, which you review and apply.
Scenario 1. Domain deny list blocking a legitimate endpoint
At baseline, the rg-domain Suricata domain rule group denies only an unused placeholder, so the test endpoint stays reachable. The rule group inspects the TLS Server Name Indication (SNI) on each outbound connection and drops any that matches a denied domain. The exact rule syntax and console steps follow.
In the navigation pane, under Network Firewall, choose Network Firewall rule groups.
Choose the rg-domain rule group to open its details page.
In the Rules section, choose Edit.
The rules box already contains two baseline placeholder rules (they match blocked.placeholder.invalid, so nothing real is denied). Leave those in place. Find the <app-endpoint-dns> value for Scenario 1 in the deployment script output (a Nework Load Balancer (NLB) DNS name such as NfTest-AppNl-a1b2C3dEf4G5-1234abcd5678efgh.elb.us-east-1.amazonaws.com). On a new line below the existing rules, add a drop rule that matches that DNS name on the TLS SNI, then choose Save.
drop tls $HOME_NET any -> $EXTERNAL_NET any (ssl_state:client_hello; tls.sni; content:"<app-endpoint-dns>"; startswith; nocase; endswith; msg:"S1 domain denylist"; flow:to_server, established; sid:2000002; rev:1;)
After saving, the rules box holds all three lines. The two placeholders remain, plus the new drop rule for the endpoint DNS name (note the distinct sid 2000002).
What happens. The workload’s HTTPS check to the test endpoint times out, the “AWS/NetworkFirewall DroppedPackets metric climbs above baseline, and Alarm-1 moves to ALARM. The Scenario 1 card reads DROPPED (with the condition Firewall dropping the monitored domain on its allow/deny rules), while the Scenario 2 and Scenario 3 cards stay Healthy (Figure 3). On the topology, the alarm pipeline from CloudWatch through Amazon SNS and Lambda to DevOps Agent and the workload-to-firewall inspect lines both turn amber, which the legend defines as collateral / alarm active, because the packets are now dropped at the firewall. To demonstrate the resulting failure, the HTTPS · SNI line from the internet gateway to the test endpoint is shown in red, which the legend defines as dropped (root cause).
Figure 3: Scenario 1 active – Traffic blocked at the firewall
Let DevOps Agent investigate. The agent runs several lines of investigation in parallel and correlates them:
Reads the DroppedPackets metric and correlates the spike with a simultaneous drop in passed packets, confirming the firewall is actively blocking traffic.
Reads the ALERT log and finds the workload’s TLS connections to the test endpoint blocked by the S1 domain denylist rule.
Compares the current state against a baseline window, where the same endpoint was reachable with no alerts, which shows the block is new.
Searches CloudTrail and surfaces the UpdateRuleGroup call that added the deny rule, identifying the user, role, and timestamp approximately one minute before the drops began.
Reports the root cause as that manual rule-group change. Recommends removing the deny entry or adding an allow exception and enabling FirewallPolicyChangeProtection to prevent unauthorized changes.
Presents this as a plan you review and apply, not an automatic change.
In the DevOps Agent Operator Web App view, the agent first restates the Alarm-1 trigger and confirms the firewall is dropping packets above the threshold (Figure 4).
Figure 4: Scenario 1 – The symptom
Next, the agent identifies the root cause: a manual update to the rg-domain rule group that added a domain deny rule (SID 2000002) shortly before the alarm fired, blocking TLS connections to the ELB endpoint (Figure 5).
Figure 5: Scenario 1 – The root cause
Finally, the agent presents a mitigation plan, recommending you remove the problematic deny rule (SID 2000002) to restore connectivity (Figure 6).
Figure 6: Scenario 1 – The mitigation plan
Note: In a real-world environment, this type of rule typically exists for a reason. Before removing it, verify whether it was intentional but scoped too broadly. If so, refine the rule to block only unauthorized endpoints rather than removing it entirely.
Confirm recovery. Apply the change the agent recommends. After the deny entry is gone, DroppedPackets falls back to baseline, Alarm-1 clears, and the card returns to green. Move on to Scenario 2.
At baseline, the rg-stateless-priority stateless rule group keeps the allow rule at priority 100 and the drop rule at 200 for the test class, TCP destination port 9142. The workload opens a TCP connection to the test endpoint on this port. Lower priority numbers evaluate first, so the allow rule wins. This scenario uses port 9142 instead of 443 to demonstrate a stateless rule, which matches on the packet’s 5-tuple (protocol, ports, addresses) rather than application content.
Introduce the change. Invert the two rule priorities so the drop rule evaluates before the allow rule. This is the kind of change a rushed rule edit can introduce.
In the navigation pane, under Network Firewall, choose Network Firewall rule groups.
Choose the rg-stateless-priority rule group to open its details page.
In the Rules section, choose Edit.
Raise the (Action: Pass) rule’s priority number so it sits after the (Action: Drop) rule, then choose Save. For example, change the (Action: Pass) rule from 100 to 300 (any number higher than the drop rule’s 200 works). You only need to move one rule, and using 300 avoids a clash with the drop rule that already sits at 200. Network Firewall evaluates the lowest priority number first, so the (Action: Drop) rule at 200 now wins for this traffic class, ahead of the (Action: Pass) rule at 300.
Figure 7: Scenario 2 – Rule priority change blocking the traffic class
What happens. The drop rule now wins, the TCP connection to the test endpoint on port 9142 times out, the StatelessRuleFailures metric climbs above baseline, and Alarm-2 moves to ALARM. The Scenario 2 card reads DROPPED (with the condition Stateless rules dropping the monitored traffic class), while the Scenario 1 and Scenario 3 cards stay Healthy (Figure 8). On the topology, the alarm pipeline from CloudWatch through Amazon SNS and Lambda to DevOps Agent and the workload-to-firewall inspect lines both turn amber, which the legend defines as collateral / alarm active, because the packets are now dropped at the firewall. To demonstrate the resulting failure, the TLS :9142 line from the internet gateway to the test endpoint is shown in red, which the legend defines as dropped (root cause).
Figure 8: Scenario 2 active
Let DevOps Agent investigate. A stateless drop happens before traffic reaches the stateful inspection engine, so it produces no ALERT log entries. The agent turns to configuration and flow logs instead:
Reads the stateless rule group state and finds the drop rule at the lower priority number, ahead of the pass rule, so the drop evaluates first.
Reads the flow logs and sees passed packets drop to zero within a minute of the change.
Searches CloudTrail and surfaces the UpdateRuleGroup call that inverted the priorities, identifying the user, role, and timestamp about a minute before the alarm.
Reports the root cause as that priority inversion. Recommends removing the redundant drop rule and managing the rule group through infrastructure-as-code (IaC) to prevent manual misconfigurations.
Presents this as a plan you review and apply, not an automatic change.
In the DevOps Agent Operator Web App view, the agent first restates the Alarm-2 trigger and confirms that a workload connectivity health check is failing because the firewall’s stateless rules are dropping egress (Figure 9).
Figure 9: Scenario 2 – The symptom
Next, the agent identifies the root cause, using the rule-group state and CloudTrail to pinpoint the conflicting DROP/PASS rules, where the new DROP rule’s lower priority number makes it match first (Figure 10).
Figure 10: Scenario 2 – The root cause
Finally, the agent presents a mitigation plan, recommending you remove the conflicting DROP rule at priority 200 to restore traffic flow (Figure 11).
Figure 11: Scenario 2 – The mitigation plan
Confirm recovery. Apply the change the agent recommends. After the allow rule is ahead of the drop rule again, Alarm-2 clears and the card returns to green. Move on to Scenario 3.
Scenario 3. Asymmetric cross Availability Zone routing drop
At baseline, the protected subnet in each Availability Zone routes its egress through the firewall endpoint in that same Availability Zone , and the matching return route uses that same endpoint. One endpoint sees both directions of the flow, so the stateful engine completes the handshake. The workload runs in the protected subnet in us-east-1a (CIDR 10.0.4.0/24), so at baseline its egress and its return both use the us-east-1a firewall endpoint.
Introduce the change. Make the flow asymmetric by sending egress out one Availability Zone endpoint while the return comes back through the other. This takes two route edits, and both are required. With only the first edit the flow can still complete, so the alarm will not trigger until both are saved. It makes no firewall-policy change, mirroring a real multi-Availability-Zone routing mistake.
To create asymmetric cross Availability Zone routing
Go to the Amazon VPC console and choose Route tables in the navigation pane.
Flip the egress. Select the NfNetworkStack/SampleVpc/protectedSubnet1 route table (the us-east-1a protected subnet, where the workload runs). On the Routes tab, choose Edit routes. Its 0.0.0.0/0 route currently targets the us-east-1a firewall endpoint. For the target, choose Gateway Load Balancer Endpoint and select the us-east-1b firewall endpoint, then choose Save changes.
Move the return. Select the NfNetworkStack/SampleVpc/publicSubnet2 route table (the us-east-1b public subnet, where egress now exits). Choose Edit routes, then Add route. For the destination enter the workload CIDR 10.0.4.0/24. For the target, choose Gateway Load Balancer Endpoint and select the us-east-1a firewall endpoint. Choose Save changes.
After both edits, a flow’s egress leaves through the us-east-1b endpoint while its return is directed to the us-east-1a endpoint. Neither endpoint sees the whole flow.
Figure 12: Scenario 3 routing change breaking the flow’s symmetry
What happens. A new connection leaves through one endpoint. Its return arrives at the other endpoint, which never saw the connection open, so the handshake fails. Unlike Scenarios 1 and 2, this affects the whole subnet, so all egress stops and Alarm-2 and Alarm-3 both move to ALARM. The AWS/NetworkFirewall DroppedPackets alarm (Alarm-1) stays quiet because no endpoint is making a drop decision. The flow is lost to asymmetric routing rather than counted as a firewall drop. This is why monitoring application connectivity matters. A routing fault is invisible to the firewall’s own drop counter. On the status page, the Scenario 2 card reads DROPPED (with the condition “Stateless rules dropping the monitored traffic class”) and the Scenario 3 card reads DROPPED (with the condition Return traffic dropped by asymmetric cross-Availability-Zone routing), while the Scenario 1 card stays Healthy (Figure 13). On the topology, the alarm pipeline from CloudWatch through Amazon SNS and Lambda to DevOps Agent and the workload-to-firewall inspect lines both turn amber, which the legend defines as collateral / alarm active, while the egress path from the firewall through the NAT gateway and the TLS :9142 and HTTPS · routing lines to the test endpoint turn red, which the legend defines as dropped (root cause).
Figure 13: Scenario 3 – The status page during a path-wide outage
Let DevOps Agent investigate. Both Alarm-2 and Alarm-3 fire in the same datapoint. DevOps Agent recognizes them as linked and merges them into a single investigation:
Reads the flow logs and sees bidirectional TLS connections stop abruptly, with only one-way traffic remaining and no flows reaching the established state.
Reads the firewall metrics and sees received and passed packets shift from one Availability Zone to the other at the moment of the change.
Calls DescribeRouteTables and finds the egress route pointing at one Availability Zone firewall endpoint while the return route points at the other.
Searches CloudTrail and surfaces the ReplaceRoute and CreateRoute calls by the same user, about a minute before both alarms fired.
Reports the root cause as that asymmetric routing change. Recommends restoring symmetric same-Availability-Zone routing so egress and return traverse the same endpoint.
Presents this as a plan you review and apply, not an automatic change.
A mitigation plan is a recommendation you review, not an automatic change, and the right fix depends on the intended design. Restoring symmetric routing can mean sending the workload subnet’s egress back through its own-Availability-Zone firewall endpoint (this sample’s architecture) or, in a design that doesn’t inspect this path, back through a NAT gateway. The agent infers a plausible target from what it can observe, so review the specific route it proposes against your intended topology before you apply it. (Connecting your pipeline or infrastructure-as-code, covered in the next section, lets the agent recommend the target that matches your design.)
In the DevOps Agent Operator Web App view, the agent restates the Alarm-3 (AsymmetricFlowFailures) trigger and confirms the workload’s egress to a monitored endpoint is being blocked by the Network Firewall (Figure 14).
Figure 14: Scenario 3 – The symptom
Next, the agent identifies the root cause: manual route table changes that created cross-AZ asymmetric routing through the network firewall, breaking its symmetric routing requirement (Figure 15)
Figure 15: Scenario 3 – The root cause
Finally, the agent presents a mitigation plan, recommending you restore symmetric routing by pointing protectedSubnet1‘s default route back to the same Availability Zone firewall endpoint, so one endpoint sees both directions of the flow again (Figure 16).
Figure 16: Scenario 3 – The mitigation plan
Confirm recovery. Apply the change the agent recommends, after checking the route target matches your intended design. After the workload subnet’s egress and return use the same Availability Zone firewall endpoint again, the control probe recovers, the alarms clear, and every card returns to green.
Further considerations
In production a single change can trigger several alarms at the same time, as Scenario 3 shows. DevOps Agent links related investigations and works them as one, so you review a single root cause. You can validate the linked findings or unlink an alarm to investigate it independently. If you would rather collapse alarms before they reach the agent, you can add correlation logic in the bridge Lambda function, buffering and grouping by firewall. You can also add email, Amazon Simple Queue Service (Amazon SQS), or HTTP subscribers to the SNS topic, or add the webhook Lambda function to a topic you already run. DevOps Agent produces a mitigation plan but does not change your environment on its own.
You can also give the agent more to work with. DevOps Agent connects to source repositories and CI/CD pipelines, integrating with GitHub (including GitHub Enterprise Server and GitLab Self-Managed through a private connection). It can associate AWS resources with deployments of AWS CloudFormation, AWS CDK, Amazon Elastic Container Registry (Amazon ECR) images, and Terraform. With deployed configuration and recent deployment events in view, the agent correlates the disruption against the change that introduced it and recommends a fix matching your intended design. For this sample, that means recommending the workload subnet’s own Availability Zone firewall endpoint rather than a generic symmetric path.
DevOps Agent also supports proactive incident prevention. It analyzes patterns across past investigations and delivers recommendations to prevent similar issues from recurring, including governance recommendations that strengthen deployment processes and pipeline controls. For Network Firewall rule changes, this means the agent can recommend guardrails for your CI/CD pipeline based on the classes of misconfigurations it has already resolved. You can access these recommendations through the Improvements page in the DevOps Agent Operator Web App.
Clean up
Clean up the environment with one command.
bash scripts/destroy.sh
It reverts any active scenario, runs cdk destroy for all stacks, and sweeps for stragglers by the Project = nf-devops-agent tag. The main cost drivers are the two Network Firewall endpoints, the NAT gateways (one in the main VPC for each Availability Zone, one in the test-endpoint VPC), and the test endpoint’s load balancers. Each of these bills at an hourly rate for as long as it’s provisioned, whether or not traffic is flowing, so a stack left running continues to accrue charges around the clock even while idle. Running the scenarios and tearing the stack down the same day limits the cost to a few active hours rather than days of idle hourly charges.
Conclusion
In this post, we showed you how AWS DevOps Agent accelerates troubleshooting for three common network firewall connectivity issues. The first was a domain deny list. The second was a stateless priority inversion. The third was an asymmetric cross-AZ routing drop. For each one, DevOps Agent investigated the drop and returned a root cause with a mitigation plan you approve before applying. The first scenario triggered on a prebuilt Network Firewall metric, and the other two on application health metrics. That shows both ways to alarm on a firewall problem through one pipeline.
Login
