Introduction

As part of proactive threat monitoring, Kaspersky’s Cyber Threat Intelligence team identified a malicious campaign involving C2 agents controlled by artificial intelligence. Confirmed victims of the campaign included organizations in the education sector in several countries across Africa and Latin America. CyberStrikeAI played a key role in the post-exploitation phase. This is an open-source framework in which an LLM orchestrator on the command-and-control server plans actions, selects tools, and generates commands, while an implant executes them on compromised hosts. Using CyberStrikeAI, the attackers progressed from reconnaissance to full domain compromise.

According to our data, this is one of the first public descriptions of a compromise involving an LLM in the wild. The campaign we discovered clearly illustrates how the role of AI in attackers’ arsenals is evolving: until recently, language models were used primarily during preparatory stages – for writing phishing emails, debugging code, or searching for vulnerabilities. Now, however, they are being embedded into the very core of the attack, automating actions at every stage: payload execution, persistence on the host, discovery, lateral movement, and, finally, data collection and exfiltration. Post-exploitation – the stage that traditionally required manual work by an experienced operator – plays a particularly significant role here.

A key feature of the scheme using the CyberStrikeAI framework is that the implant ceases to be a simple tool for executing the operator’s commands. Decisions are distributed across two levels: the operator defines the task in general terms, while the model determines how to execute it – specifically, which commands to send to the host and in what sequence. Thus, the strategic planning remains with the human, while the choice of technical steps is delegated to the model, and the implant executes the model’s decisions.

To understand how this scheme works in practice, let’s examine the chain of actions taken by the attackers, as reconstructed by our analysts during the incident investigation.

Timeline of the attack

Let’s examine the timeline of the attack using an example from one incident within the detected campaign.

The first phase follows a standard approach: a human operator gains access to the victim’s infrastructure via a compromised SQL Server (version 16.0.1000), then collects basic information about the host, deploys tools, and conducts network reconnaissance. All of this activity takes nearly a day, followed by a two-week pause.

The second phase is fundamentally different. The first step involves deploying the CyberStrikeAI C2 agent, which receives commands from a server-based LLM orchestrator. During the second phase, the agent performs the following:

• establishing persistence and conducting local reconnaissance within 2 minutes;
• conducting domain reconnaissance over the next 5 hours;
• after a brief pause of several days, moving on to the final stage – lateral movement via SMB and WMI, mass credential dumping on hosts, and database export – with the entire operation taking no more than one hour.

We’ll discuss these stages in more detail in the next section.

Technical details

CyberStrikeAI overview

CyberStrikeAI is an open-source, modular platform written in Go for automating penetration testing. Its key feature is combining LLM-based attack orchestration and a built-in C2 server with agents (beacons) within a single server process.

The entire “intelligent” layer is isolated on the control server. C2 functions (calling reconnaissance and exploitation utilities, managing webshell connections, assigning tasks to agents, and reading results) are available to the model as MCP tools. The implant is a classic C2 agent with no AI functionality – no API keys, no calls to AI service APIs, and no autonomous logic. It periodically polls the listener server via HTTP/S, TCP, or WebSocket, executes shell commands, and interacts with the file system. To the victim, this activity looks like a typical remote-control session, where a server-side LLM orchestrator queues tasks instead of a live operator. The LLM influences how decisions are made, but not how the corresponding commands are delivered to the host.

Although CyberStrikeAI is publicly available, specific instances of the framework can be linked to specific attackers or campaigns. Detection and attribution are possible based on the following artifacts:

• Control panel: Upon launch, a self-signed TLS certificate is generated, with the framework’s name appearing in the issuer and subject fields. The certificate is created in memory at each startup, so the not_before field effectively records the time the process was launched. Accordingly, as with Cobalt Strike, the chronology of restarts can be reconstructed from the scan history.
  

  CyberStrikeAI control panel’s self-signed TLS certificate

• Implant: The build metadata (go version -m) contains a path to the main package, which includes data pointing to the built-in CyberStrikeAI generator – a temporary directory with the build identifier. This path is identical for all payloads, making it a reliable signature. Additionally, the build info field contains version control system metadata, which can be used to compare samples against one another.
• Authentication: When communicating with the server, the implant sends an X-Implant-Token header with a fixed value in the HTTP request. The token is unique to each C2 server and is embedded into the malicious payload during the build phase; therefore, it serves as both a network indicator and a sign that the samples belong to the same server.

HTTP header parameters sent by the implant to the server

It should be noted that the implant version does not reflect the server version. The implant can operate for months while the operator updates the server-side components: listener configurations and keys are preserved during updates. In the incidents we detected, we observed the use of a single agent version throughout the entire incident. In such cases, the sample’s build date may indicate the earliest possible deployment date of the CyberStrikeAI infrastructure, but it still does not provide an estimate of the compromise duration.

Execution & Privilege Escalation

The first malicious commands on the victims’ systems in the campaign under investigation originated from the sqlservr.exe process, indicating the likely initial access vector and the context in which the attackers gained the ability to execute code. It is important to note that the commands marked as having been executed prior to agent installation were executed manually by the attacker specifically within the context of this process. At this stage, we are not yet dealing with automated implant activity, but rather with manual actions by the operator.

Before launching the agent on the machines, the attackers checked for its presence using a PowerShell script, from which we can deduce its default deployment paths: C:\Windows\Temp\, C:\Windows\, C:\ProgramData\, C:\Temp. At the same time, the attacker can disguise the agent’s name as that of a legitimate system process. In the example below, it mimics svchost.exe (svchosts.exe)

Checking for the presence of the CyberStrikeAI agent on the system

Delivery and execution of the CyberStrikeAI agent

Like most C2 agents for Windows, CyberStrikeAI uses PowerShell to execute its payload. When a request is received from the operator, the commands to be executed are additionally encoded in Base64 to bypass keyword-based detection.

Decoded form of the command:

When administrative privileges were available (for example, a domain administrator account), instead of executing commands directly, the agent on remote hosts often created scheduled tasks so that the malicious payload would run with SYSTEM privileges:

An alternative method that does not use schtasks utility is to create a task via the Schedule.Service COM object:

The payload (p1.ps1) from the example above is a reverse shell that connects to the server at 185.186.245[.]144:5501:

The p1.ps1 payload and similar scripts were distributed across the network using shared network administrative folders and the WMI protocol, which we will describe in more detail in the Lateral Movement section.

Stealth

In addition to encoding commands in Base64 and obfuscating SQL/LDAP queries – which we will describe in the Collection & Exfiltration and Discovery sections, respectively – the attackers split the malicious payload into multiple files. It is noteworthy that the scripts were transmitted in this manner even before the agent was installed – that is, at a stage where human intervention is typically required – yet they clearly bear the hallmarks of being generated by an LLM.

Below is a sequential record of several script fragments written to the file d3.b64.

Subsequently, the attackers used the certutil utility to decode the resulting file and executed it with the following command:

Once assembled and decoded, the script’s contents looked like this:

The script is a tool for decrypting DPAPI secrets from the file C:\ProgramData\dsn_line.txt. This file, in turn, is populated by the following command, which searches for a DSN string containing connection settings in the SQL Server Reporting Services configuration file rsreportserver.config and copies its contents to dsn_line.txt.

It is worth noting that some of the scripts delivered and executed as described above contain comments in Chinese (##  –  researcher’s comments).

Persistence

If the CyberStrikeAI implant was successfully delivered, the first step was to establish persistence on the target system. To do this, the agent created a scheduled task named SystemUpdate, which specified the path to the implant as the action to be performed:

Analysis of the framework’s source code revealed an alternative method of persistence: by adding a value to the registry key HKCU\Software\Microsoft\Windows\CurrentVersion\Run, where the name of the created value matches the task name (SystemUpdate).

Agent persistence on Linux systems is also implemented in two ways – by adding the entry @reboot <path_to_beacon> & to the user’s crontab (in which case the process runs at system startup), or by adding the line (nohup <path_to_beacon> &>/dev/null &) to ~/.bashrc (in which case the agent starts when an interactive bash shell is opened).

It is worth noting that the choice of a specific persistence method is delegated to the LLM connected to CyberStrikeAI. However, thanks to the human-in-the-loop (HITL) mechanism, the operator retains full control over the process: commands generated by the model may require manual approval or correction before they are executed.

Discovery

Local Discovery

Before installing the agent on a compromised host, the attackers manually gathered basic system information using a fairly trivial set of commands:

The attackers also manually checked for internet access and obtained the external IP addresses of the compromised machines using the Chinese service dnslog[.]cn. The service generates a random third-level domain, after which the attackers attempt to resolve it on the compromised machine.

If the machine has internet access, the request succeeds, and the attackers obtain the external IP address of the compromised device on the service’s page.

For reconnaissance purposes, the agent generates a more complex script designed to map out a path for further privilege escalation by compromising SCCM or exploiting msiexec (the AlwaysInstallElevated key):

Additionally, to find ways to elevate privileges locally, the agent executed scripts to inventory services running under domain accounts and tasks running under the SYSTEM account:

During their activity, the attackers periodically checked for Microsoft Defender detections:

Remote Discovery

After gathering information about the compromised host, the attackers proceeded to conduct reconnaissance within the network. During the preparatory phase of the attack, they manually collected additional information about remote hosts using the net.exe utility.

During the main phase, the agent used PowerShell’s built-in functionality to detect accessible hosts without relying on additional utilities (such as nmap or fscan):

Additionally, relevant LDAP queries to Active Directory objects helped build a network map:

Furthermore, the presence of the installed agent on remote hosts was verified using WMI:

Domain Discovery

The next step was to gather information about the victims’ domains. To this end, before installing the agent, the attackers used standard OS utilities:

However, later on, using the CyberStrikeAI implant, reconnaissance was carried out via generated PowerShell scripts:

In addition to basic domain information, the agent performed LDAP queries to gather information for Kerberoasting and AS-REP Roasting attacks:

Finally, the agent was used to check the domain for the presence of LAPS and Microsoft Azure:

To detect signs of Microsoft Azure usage within the infrastructure, the attackers also searched for the .azcopy directory – created by the AzCopy (Microsoft Azure Storage Copy) utility – in users’ home directories:

Credential Access

After successfully conducting reconnaissance and mapping the network, the attackers moved on to the next stage – dumping credentials. For this purpose, the agent used a classic set of LOLBINs and well-known third-party tools.

The agent dumped registry hives containing credentials using the reg.exe utility:

The agent dumped the lsass.exe process using the MiniDump function of the comsvcs.dll library. In doing so, the target process’s PID was used instead of its name to make static detection based on the command line more difficult:

After successfully compromising privileged domain accounts (specifically, the built-in domain administrator), the attackers also carried out DCSync and Golden Ticket attacks via the agent using a renamed version of the mimikatz utility:

Lateral Movement

To spread across the network using compromised accounts, the attackers employed utilities from the Impacket toolkit, specifically WMIExec, as indicated by the utility’s characteristic command line:

Once installed, the agent prefers to use standard OS mechanisms, such as shared network folders.

Another method the agent uses to remotely execute commands is the WMI protocol. In particular, the agent used it to distribute the aforementioned p1.ps1 reverse shell script. This involved using compromised user accounts with local administrator privileges on the target hosts:

A second method of lateral movement via WMI uses the Invoke-CimMethod cmdlet:

Command & Control

To deliver the aforementioned tools (malicious scripts, mimikatz, and the agent itself) to the initial nodes in the infection chain, the attackers used standard OS utilities:

After the agent is installed, it checks the availability of the C2 server’s ports. The implementation mechanisms here are virtually unlimited and depend on the “creativity” of the respective LLM agents connected to the server:

To download additional utilities, the CyberStrikeAI implant also used built-in utilities (LOLBINs). The tools could be hosted on the attackers’ own servers or downloaded directly from GitHub:

Collection & Exfiltration

The data collection phase deserves special attention. It demonstrates yet another capability of LLM orchestration – the automatic generation of SQL and LDAP queries tailored to a specific task.

In one incident during the campaign, the attackers deployed the CyberStrikeAI implant on an educational organization’s SQL server and collected information from the database via PowerShell. Here is an example of a command executed by the agent:

Normalized view of the executed script with comments:

Before accessing the data, the agent checked the privileges of the account under which it connected to the database:

In decoded form, the check looks as follows:

On network resources, file collection was preceded by a recursive directory scan to identify files of interest.

To interact with the file system, the implant provides a built-in interface that supports direct uploading and downloading of files. Files are transferred directly via the main encrypted communication channel with the C2 server (HTTP or TCP), and the data itself is encoded in Base64 and packaged into the agent’s standard JSON responses.

Another built-in capability of the implant is taking screenshots of the infected machine. As with other actions, the implant does not initiate this on its own: the command to capture the screen is issued by the LLM orchestrator, and the agent merely executes it and returns the result via the same control channel.

On Windows, the command for taking screenshots looks like this:

The Linux version looks like this:

In the incidents we detected, data collection was the final stage of the attack: the chain ended there, and we observed no further activity.

Attribution

The key artifact for attribution in this campaign is the implant’s build metadata. The attackers built the malicious payload inside a Git clone of the project, so the Go compiler embedded version control system information into the binary files – the hash and commit time, as well as the vcs.modified=true flag (analogous to the +dirty suffix) – indicating uncommitted local changes in the working copy. The combination of these fields forms a unique fingerprint for a specific build.

Based on this fingerprint, the samples are divided into two independent groups, in each of which the Windows implant and the Linux implant have completely identical build metadata. A match indicates that, within each group, both payloads were built from the same project working copy.

Identical build metadata for the Windows and Linux implant samples, indicating a shared build environment

However, these groups do not overlap with one another. They do not share a common C2 infrastructure, and different versions of CyberStrikeAI are deployed on the command-and-control servers. In other words, the build metadata reliably links the samples within each group but does not provide grounds for linking the two groups together.

Differences in CyberStrikeAI versions on the command-and-control servers

The available data does not allow us to link the activity to any known group of attackers, nor to state with certainty that the same operator is behind both campaigns. Two factors prevent this. First, the tool’s openness: CyberStrikeAI is publicly available, so different operators could have independently deployed it and assembled malicious payloads under similar conditions. Second, command generation by the model: the “signature” characteristic of a specific operator is largely shaped not by a human but by an LLM. On the one hand, this increases the diversity of commands: the same command, when generated multiple times, can look different. On the other hand, it strips them of their individual characteristics, which reduces the value of behavioral metrics for attribution.

However, the Chinese-language comments found in the attackers’ scripts and the use of Chinese services still suggest a connection between the activity and a Chinese-speaking operator. In addition, analysts noted the timing of the control servers’ restarts: since the self-signed TLS certificate is regenerated each time the process starts, its not_before field records the restart time, and most of these restarts occurred during business hours in the UTC+8 time zone.

The origin of the tool itself also indirectly points in the same direction: CyberStrikeAI was developed within the Chinese cybersecurity community and is well known there.

In summary

CyberStrikeAI is a telling – though not unique – example of a shift occurring within the class of C2 frameworks: the LLM orchestrator is no longer merely an add-on to reconnaissance but has become a control layer over post-exploitation. If HexStrike AI and similar tools cover the “find and break” phase, and Sliver/Mythic/Havoc cover the “establish a foothold and control” phase, CyberStrikeAI completes the kill chain: a single server-side process written in Go executes utilities via MCP, handles webshell connections, and assigns tasks to the implant, while the model analyzes the results and decides what to do next.

The key takeaway is that the introduction of AI into the attack chain does not yet present fundamentally new challenges for detection. All the “intelligent” work remains on the attacker’s side, while the compromised host is subjected to familiar actions: from reconnaissance via Kerberoasting and AS-REP Roasting to LSASS dumps, DCSync, and the Golden Ticket attacks. This is a typical, well-documented set of domain compromise techniques. The novelty of the attack lies not in the methods themselves, but in the speed and ease with which they were executed: AI orchestration has lowered the skill requirements for the operator, but has not taken the attack beyond what defenders already know and are capable of detecting. The timing is also telling: individual key phases of the activity (such as establishing a foothold, lateral movement, and obtaining credentials) took only a few hours, and the entire attack – from agent deployment to the compromise of domain administrators – was completed within 3–4 days.

However, the CyberStrikeAI architecture does not limit users to the scenario described. The same mechanisms allow for the on-demand assembly of more complex chains – including DPAPI secret extraction and attacks on SCCM and Azure IMDS – all of which are within reach of the same group of actors, with a relatively low skill barrier. As the models evolve, the gap between the complexity of orchestration and the level of techniques used will narrow, and the operator’s role will gradually shift from setting tasks to monitoring the model’s actions.

Detection by Kaspersky solutions

It may seem that attacks using AI agents require the creation of fundamentally new detection rules. Indeed, to detect the activity of an LLM agent installed on a host within the infrastructure, it is necessary to create detection logic based on the specific characteristics of the agents. Detection rules for this activity can be found using Hunt Hub with the following filter.

However, in the case of the attack described, a different logic applies. The main difference lies in the architecture of the attack itself: the LLM agent was not located within the corporate network perimeter. It operated remotely via a C2 agent, orchestrating actions, and left no traces of its presence in the infrastructure (such as specific API requests to local LLMs or anomalous patterns of resource consumption by hosts).

The key difference between this activity and an attack by an experienced human operator lies largely in speed. Tactically, technically, and methodologically, AI, for the most part, uses exactly the same tools, utilities, and techniques as a human penetration tester. It does not write unique exploits from scratch, but rather combines known methods and dynamically adapts them to the situation, handling the tools’ errors on its own.

Since the actions performed by an AI agent on the network are indistinguishable from those of a human, classic, well-tuned correlation and behavioral analysis rules are sufficient to detect it. Here are a few examples of how Kaspersky Next EDR Expert successfully detects the attack described:

• To detect the execution of a malicious script via a remote scheduled task, you need to monitor the creation of the schtasks.exe process with the /s argument. EDR Expert detects this activity using the rules create_task_on_remote_host_using_standard_tool, scheduled_task_creation_as_system_via_schtasks, and scheduled_task_with_powershell_execution.
  
• During the attack, the adversary will need access to the victim’s credentials; in the described case, one way to obtain them was through a well-known technique that uses legitimate tools to dump LSASS, specifically by calling the MiniDump function from comsvcs.dll. The EDR Expert solution detects this activity using the rules dumping_memory_with_com_services_dll and suspicious_lsass_memory_access.
  
• A significant part of the attack chain is lateral movement, which helps the attacker navigate within the infrastructure and achieve their desired objectives. In the described case, the WinRM protocol is used to initiate connections directly from PowerShell. This is a standard technique that allows the attacker to execute commands on remote machines without using additional network tunnels. The EDR Expert solution detects this activity using the rules using_powershell_for_remote_execution_via_wmi and winrm_send_command.
  

Kaspersky Anti Targeted Attack (KATA) is used to detect malicious activity in network traffic. Communication between CyberStrikeAI agents and the command-and-control server occurs without accessing AI platforms such as ChatGPT or Claude. Instead, the agent’s C2 server is an infrastructure pre-configured by the attackers, just as in attacks that do not use AI agents. Analysis of decrypted network traffic using NDR-class solutions reveals characteristic communication patterns of CyberStrikeAI agent activity. When these patterns are detected, KATA generates an alert with the verdict Backdoor.CyberStrikeAI.HTTP.C&C.

Alert regarding the detection of CyberStrikeAI communication with a C2 server

To detect post-exploitation activity, you can use not only IDS technology but also NAD (Network Anomaly Detection). It analyzes all traffic for suspicious indicators that do not correspond to a node’s typical network activity.

For example, actions performed by CyberStrikeAI during the Domain Discovery phase are detected by an NDR:NAD rule named “Scanning domain records via LDAP queries.”

NDR:NAD alert: reconnaissance via LDAP queries detected

You can learn more about NDR:NAD technology in our article on Securelist.

You can also visualize the connections between C2 addresses and files used by attackers in the Research Graph section of the Kaspersky Threat Intelligence Portal. This includes tracking the emergence of new CyberStrikeAI implants that communicate with malicious addresses.

Visualization via the Research Graph on the Kaspersky Threat Intelligence Portal

In addition, the Threat Lookup service available on the portal allows you to determine whether malicious files belong to CyberStrikeAI using attribution mechanisms based on the executable file’s code.

Demonstration of attributing a malicious implant to the CyberStrikeAI framework

Kaspersky SIEM detects all stages of an attack by the CyberStrikeAI agent. The product repository contains over 100 rules for detecting the techniques discussed in this article; we will list the most critical ones below.

Execution of various commands from the sqlservr.exe process:

• R300_04_Windows shell started by a web application

Downloading and executing a payload via PowerShell:

• R110_03_PowerShell code downloaded and executed

Creating Windows scheduled tasks:

• R099_04_Suspicious arguments when running schtasks.exe
• R300_03_Windows shell started by the scheduler

Persisting via crontab or bashrc:

• R099_06_Suspicious cron job activity
• R405_02_UNIX shell configuration modification

Rules located at the KUMA Packages/SOC package/Configuration discovery path detect the collection of information about hosts, network settings, and domains using various reconnaissance tools. Rules that aggregate detections by reconnaissance tools:

• R224_14_Windows reconnaissance activity
• R224_09_System information discovery by running multiple utilities within a short period
• R224_04_System information discovery via standard Linux tools

Data collection for Kerberoasting and AS-REP Roasting:

• R034_01_Multiple TGS ticket requests (Kerberoasting)
• R452_18_AS-REP Roasting

Dumping registry hives containing credentials using the reg.exe utility:

• R230_02_Copying data containing credentials from the Windows registry

Dumping the lsass.exe process using the MiniDump function of the comsvcs.dll library:

• R233_04_Process memory dump via comsvcs.dll
• R262_Suspicious access to the LSASS process

DCSync and Golden Ticket attacks using the renamed mimikatz utility:

• R073_DC Sync attack
• R450_41_Golden Ticket Attack
• R100_03_Traces of Mimikatz use via cmd

Lateral movement using WMIExec:

• R460_08_Lateral movement using WMIexec Impacket
• R460_09_Lateral movement using WMIExec and Impacket

For the rules to work, you must configure the SIEM system to receive the relevant events:

• Process launches (Windows security log – 4688, auditd – execve) and PowerShell commands (4104)
• User logins to the system (Windows security log – 4624, 4768, 4769)
• Various object manipulations (Windows security log – 4662, 4663, 4656; auditd – openat)
• Interaction with shared resources (Windows security log – 5145)

Indicators of compromise

File-based indicators:

Network indicators:

A modern organization protects its information systems with a whole suite of technologies that generate thousands of security alerts every day. Looking at these alerts in bulk, you’ll see that a significant share are false positives. Ideally, these alerts would be filtered out automatically so that analysts could focus on the ones that may point to a real incident. However, writing filters is a time-consuming task in itself. To create a correct filter, an analyst needs to:

• Review the logs
• Identify the source of the activity
• Define filtering boundaries
• Choose filtering criteria
• Write a rule
• Check that the filter is robust

This post looks at how filter creation itself can be automated to reduce analyst workload.

Filter validity criteria

In the context of security alerts, a filter is a combination of event field values that marks a detection as a false positive and not worth the analyst’s attention. Filters can be generated for any combination of event fields, but not every combination yields a valid filter. That’s why we need to define validity criteria: conditions showing that a specific filter actually removes noise and doesn’t suppress useful alerts.

Detection rules in security systems typically contain a set of conditions, and each of them can generate an alert. A specific event triggers a given fragment of a detection rule. The set of events that a given fragment responds to is what we’ll call its context. A good filter should cover the entire context where possible, without going beyond it. Filters that are too broad or too narrow are useless. For example, a filter based solely on PowerShell would exclude too many events, including some that deserve the analyst’s attention. Most of the filtered events would also be unrelated to the rule in question. A filter on a specific ID, on the other hand, would cover only one event, which may never recur.

Problems in filter generation

We can run into a number of problems when creating filters. First, events often contain an excessive number of fields (around 400 on average), and most of them aren’t needed for filtering. At the same time, generating and ranking combinations of these fields is also fairly resource-intensive. So, the first step is to discard the irrelevant fields and keep only those worth using in the filter. For example, out of 400 fields in an event, four may be enough for filtering.

Another problem is string normalization. Field values (for example, the path to the same application on different systems) can vary depending on system settings, user folder names, and so on. For a filter to account for all these variations, the strings must be normalized. For file paths, in particular, you can replace the drive letter and user folders with standard environment variables and convert all paths to a single case.

Finally, if a filter completely mirrors the query of a detection rule, it effectively disables that rule. To fix this, remove the event fields used in the detection rule from the filter. For example, if a rule triggers on a file path, there are two ways to filter out unwanted alerts:

• In the filter, use fields other than the file path.
• Use a part of the file path that the rule doesn’t use. For example, for the rule file_path: ".../something.exe", you can set up a filter that excludes events with the user/unique_files/ directory in the file path.

Filtering security alerts with an algorithm

Given the problems described above, automatically generating filters for security alerts should consist of the following steps:

1. An analyst receives a security alert that they assess as a false positive and uses the filter generator.
2. The generator takes the 400-field event from the alert, along with the rule that triggered on it, and determines the context.
3. The generator selects the fields that are relevant to the filter.
4. The generator normalizes the strings.
5. The generator builds different combinations of the selected fields. To do this, we applied the FP-Growth (frequent pattern growth) algorithm to the context determined earlier. FP Growth builds a tree of frequently occurring items (fields) from a dataset (in our case, the context) and discards items that aren’t frequent enough. This reduces the number of combinations to check, which lowers the cost of finding the most relevant ones.
6. The generator ranks the resulting combinations and offers the analyst the top three to choose from.

For example, consider an event described by the following JSON:

After generating and ranking the combinations, we got three filters based on file paths:

• processcmdline:("nodejs\\node.exe" AND "*****\\appdata\\roaming\\npm\/node_modules\/pi-coding-agent\/dist\/cli.js")

• file_path:"*****/.pi\\agent\\bin\\fd.exe"

• processcmdline: "programfiles\\nodejs\\node.exe"

The last filter turned out to be too broad, but the analyst marked the other two as valid.

Pros and cons of the algorithmic approach

The algorithmic approach to filter generation has several advantages. First, it’s simple and deterministic: the same alert always produces the same result. Second, it’s interpretable, meaning we can see why a given combination ranked near the top of the list and why a particular field didn’t make it into the list of candidate fields at all. Third, the stages of the process are independent of each other, so individual steps can be improved and tuned without the risk of breaking the algorithm elsewhere. Finally, it has no external dependencies or models, which makes it easy to deploy in any customer environment.

However, it also has drawbacks. The first is its dependence on context. If there’s no context for some reason — for example, the rule’s first-ever trigger — a filter can’t be created. The second problem is string normalization. There’s a high chance of running into atypical cases that the normalization rules don’t cover. Often such cases can’t be described correctly at all, because they contradict each other.

These problems can be solved with large language models (LLMs). When an LLM generates the filters, no context is needed, because the LLM doesn’t iterate over or rank combinations. The LLM also takes a more flexible approach to string normalization, including non-standard cases. That said, an LLM doesn’t have to be involved at every stage of the generation process. We decided to keep the algorithmic solution where it works well. For example, a simple algorithm selects suitable fields without errors, so we didn’t move this stage to the model.

Filter generation with an LLM

The LLM-based filter generation looks like this:

The input is still an alert containing an event with 400 fields, plus the rule that triggered on that event. Then an algorithm selects the most suitable fields for the filter. Next, the LLM processes the input — the event, the selected fields, and the detection rule — and generates three filter options along with an explanation for them. The explanation helps the analyst understand why these particular filters were suggested and make an informed choice.

Consider the following event in JSON format:

The LLM suggested three filter options that all contain the same command line fragment:

• file_path: "\\WindowsPowerShell\\powershell.exe" AND filecmdline: "Get-VM -Name kafka\*"

• filecmdline: "Get-VM -Name kafka\*"

• filecmdline: "kafka*"

As in the previous example, two of the three output filters were valid. The last one was too broad to be usable. The analyst chose the first filter with minor edits:

file_path: "powershell.exe" AND filecmdline: "Get-VM -Name kafka\*"

Pros and cons of generating filters with an LLM

Using an LLM solved the problems of purely algorithmic filter generation: dependence on context and errors in string normalization. However, the single-request approach had its own drawbacks. First, it had no output validation. The LLM returned filters as-is, with no syntax check and no way to correct errors. Second, unlike in the algorithmic version, all filter generation tasks were handled by a single system prompt. As a result, a minor fix in one place could break the logic in another.

Using multiple AI agents

To address the problems of generating filters with a single model and a single system prompt, we tried distributing the tasks across different LLM agents. In this version, we brought in agents as early as the stage of selecting relevant fields. The chain includes four agents in total:

• Selector: selects the relevant fields.
• Generator: generates filters.
• Validator: evaluates the generated filters. If the filters are found to be invalid, it returns the task to the generator. If the selected fields are found to be irrelevant, it starts a new field selection cycle. If the filters pass validation, or the limit on regeneration attempts is reached, the valid filters (or the best ones available) are passed to the analyst.
• Commentator: analyzes the output filters and explains which field values were used and why.

For the following JSON event, the agents produce three valid filters based on the file path.

The filters differ in the additional parameters that support the detection:

• file_path:"sysvol\\domain\\scripts\\logon.bat"

• file_path:"sysvol\\domain\\scripts\\logon.bat" AND file_md5:"0x7..6E"

• autorun_entry: "Group Policy\\State" AND file_path: "SYSVOL\\domain\\scripts\\logon.bat"

Pros and cons of the agent-based approach

LLM agents give us a system that can validate and correct filters. Its modular architecture also lets us tune and improve different stages of generation independently. However, this approach makes more LLM calls than the previous ones, so the system is slower. It also depends on access to a model, which may not be available in the organization’s isolated environment.

Evaluating the filter generation system

To evaluate how well the filter generation system works, we use two main metrics:

• Approval rate. For each security alert, the system suggests three filters. If the analyst selects at least one of them, we count the generation as successful. For example, in the case below, the approval rate is 2/3.
  
• False positive reduction rate (FP drop). This metric shows how much a filter helps reduce the number of false positives. It can only be measured over time, because we need to compare the number of false positive alerts received during a set period before and after the filter is deployed.

Below is a comparison of the three filter generation approaches by these metrics:

As the results show, LLMs significantly improve the efficiency of filter generation. At the same time, the LLM agentless and agent-based approaches perform almost identically. So we chose the simpler and faster option: an LLM with a single system prompt. As a result, we cut false positives by 99%, increased the number of filters generated per day by 28%, and made analysts’ alert handling 17% more efficient.

Takeaways

Practice has shown that filter generation can be automated effectively. We’ve looked at several options for automatically generating filters for false-positive security alerts: a purely algorithmic approach, an LLM-based approach, and an approach built on a group of LLM agents. All of them can generate valid filters and make the analyst’s work much easier, but each has its own pros and cons.

The algorithmic approach scored lower on the evaluation metrics, but in two-thirds of cases, it still produces filters that satisfy the analyst. It works especially well when there’s already a set of similar events to filter out, meaning enough context has accumulated. The algorithmic generator can also be deployed in any environment. Both standalone LLM and multi-agent LLM approaches show high accuracy, and their results differ only slightly, so the slower agent-based approach looks excessive at this stage. If an LLM is available, the best option appears to be a single model that handles several stages of the generation cycle.

Launching the expanded AI Classroom Toolkit with Cyberlite during Global Media and Information Literacy Week 

AI is opening new pathways to information, giving people faster access to answers, ideas, and knowledge. At the same time, it’s making it harder to know whether the content we see is authentic. 

Companies that build AI have a responsibility to make it reliable and trustworthy, and that responsibility doesn’t shift to the people who use it. But AI is already shaping how people search, learn, and encounter information. Helping people navigate that reality means building media and AI literacy (MAIL): the judgment to decide whether and how to use AI and what information to trust. These skills are as foundational as AI safety, and educators are asking for help teaching them. 

That’s why, during Global Media and Information Literacy Week, we’re launching the expanded AI Classroom Toolkit: Unlocking Generative AI Safely and Responsibly, created with the online safety organization Cyberlite. It’s part of Microsoft’s broader commitment to make sure AI strengthens human thinking rather than replaces it. It also reflects Microsoft’s focus on education and empowerment, a core pillar of our Safe Participation Framework. 

What’s new in the AI Classroom Toolkit

The expanded toolkit adds a deeper focus on critical thinking, verification, and media and AI literacy, helping students judge AI outputs and make informed decisions about what to trust online. Together with the original modules, it’s now a single, classroom-ready resource that covers the core of AI safety and literacy. Realistic, scenario-based lessons and activities help students practice these skills in situations they may encounter online: 

• The Climate Campaign: Gina enters an online challenge to make a campaign for a cause she cares about and finds that some winning entries use AI to show events that never happened. Students sort a feed of real and AI-made posts and practice lateral reading. The lesson: whether a post was made with AI doesn’t decide whether it’s trustworthy. What it claims, and who is behind it, does. 

• The Helpful Bot: Alex creates an AI help bot inside his online game, then catches it inventing details and collecting players’ personal information. Students learn why AI can get things wrong, how to trace claims to reliable sources, and why they’re responsible for AI-generated content they act on or share. 

• What Kind of AI User Are You?: An interactive quiz, hosted by Cyberlite, helps students identify their AI habits, strengths, and potential blind spots, and gives them an AI Type to reflect on and discuss. 

“When we created the first Classroom Toolkit: Unlocking Generative AI Safely and Responsibly with Microsoft in 2023, most of the conversation was about using AI safely. Today, the questions young people ask about AI have grown sharper and more personal, like what to believe online or how far to trust a chatbot’s answer,” said Michelle Yao, cofounder of Cyberlite. 

The toolkit also reflects what young people in our co-design sessions with Cyberlite told us: they want to learn by doing, through games, real-life scenarios, and peer discussion. Teachers don’t need to be AI experts, and students don’t need an AI tool in hand to start. 

That approach builds on the reach and results of the original toolkit. Since 2023, the first version has reached educators in 10 countries and 11 languages, including through government and partner learning platforms. In South Korea, 93% of participating educators reported improved knowledge and understanding of ethical issues, privacy, and bias related to AI. 

“Our hope is that every student leaves a little surer of their own judgment, and that everyone feels more confident deciding how AI fits into their lives,” said Yao. 

Literacy works alongside product design and policy

Education is one part of the equation. Our commitment to media and AI literacy also shows up in how we design products and the policies we support.  

Last month, we set out five principles for AI in education, including “AI designed to support students’ learning, not replace their thinking.” We also committed to enforceable privacy, safety, and transparency protections for qualifying educational AI products in US school districts. These protections help schools and families understand how these tools work and the controls available to them. 

Media and AI literacy is now a global priority

The Organisation for Economic Co-operation and Development has released a framework for a MAIL assessment beginning in 2029. As part of its Programme for International Student Assessment (PISA), it will examine how students reflect on, analyze, and evaluate information, including AI-generated information. This week in New Delhi, UNESCO is also launching an AI extension of its “Think Critically, Click Wisely” global curriculum, which Microsoft advised on. 

This growing global focus makes the need for support urgent—educators want help, and many aren’t getting it. In Microsoft’s 2026 AI in Education report, 53% of educators and 77% of students say they haven’t had AI training.  

Closing this skills gap matters beyond the classroom. Healthy societies depend on a shared basis of facts, informed public debate, and confident civic participation. As authenticity gets harder to judge, there’s a risk that people start doubting everything, creating what the Digital Inquiry Group calls a “dangerous declaration of permanent uncertainty.” MAIL offers a way through. It builds the confidence to decide what to trust. 

Media and AI literacy for every age

Across Microsoft, we’re helping people build these skills wherever they learn and find information: 

• In classrooms: Students debate whether to accept, question, or rework an AI tool’s answers in an AI and Critical Thinking lesson from The Economist Educational Foundation, supported by Microsoft. Available alongside other media literacy lessons in Microsoft Learning Zone, the lesson has reached students in 116 countries. 

• Through play: Millions of players have checked sources, spotted bias, and cracked cases of AI misuse in Minecraft Education’s The Investigators and Reed Smart: AI Detective. 

• For content creators: UNESCO, with Microsoft’s support, is developing resources to help creator associations train their members in MAIL and is testing them with creators in Nairobi this week.  

• For voters: Check. Recheck. Vote. helps people apply AI literacy to election information. 

• In our products: Search Progress, built with input from the Digital Inquiry Group, turns research assignments into source-checking practice in Microsoft Teams for Education. 

What’s next 

Through Microsoft Elevate for Educators and local partners, we’ll bring the AI Classroom Toolkit to new audiences, with more languages and localized content rolling out through 2027. We’ll keep evolving it as technology and educators’ needs change. 

Explore the toolkit and supporting resources for educators today.  

AI will keep changing how people find information and the content they encounter. Our goal is to help people meet that change with confident, informed judgment about the AI they use and the information they trust so they can fully take part in civic and democratic life. 

The post In an age of AI, deciding what to trust is a foundational skill. Here’s how we’re helping educators teach it. appeared first on Microsoft On the Issues.

The rapid diffusion of AI globally and the pace at which AI is developing put us at a societal crossroads. The choices we’re making now will determine who has access to AI and the opportunity it affords. There is perhaps no more important tool than our education systems for addressing the uneven diffusion of AI we’re seeing currently: national adoption runs above 70% in countries such as the United Arab Emirates and falls to low single digits across much of sub-Saharan Africa, close to a tenfold difference. 

The way we teach AI and learn with it will have long-lasting impacts on jobs, the workforce, and our broader economy. Learning Curves: Rethinking Education from Classrooms to Careers in an AI-Driven World is a collection of research-driven essays that can help guide those choices. This book brings together the body of work from the second cohort of Senior Fellows from the Microsoft AI Economy Institute (AIEI), a flagship think tank that produces research and convenes experts to generate insights into how AI is reshaping work, education, and the broader economy. 

In Learning Curves, researchers consider how AI is reshaping the earliest stages of learning and work, from K–12 classrooms and community and technical colleges to college‑to‑work transitions and entry‑level jobs, and what it will take for that transformation to widen opportunity rather than deepen divides across institutions, communities, and countries.  

Below, I share a few of my own reflections that informed my contributions to Learning Curves, a collaborative effort that wouldn’t have been possible without the truly talented, diverse cohort of researchers who contributed to this book.  

The good, the bad, and the ugly: Laying the foundation for AI in K–12 classrooms

Before delving into the specifics of how to incorporate AI in classrooms, we need to ask ourselves whether we’re teaching for knowledge, or for learning.  

The book, the calculator, the personal computer, the internet, the search engine, and the smartphone all arrived in classrooms carrying real promise alongside the worry that they would hollow out young minds. Usually the truth landed somewhere in between, and where it landed depended less on the tool than on the choices the adults made about how to use it. 

The fact is that the AI-native generation is already here, and its members are in classrooms around the world. More than half of US teenagers already use AI chatbots for schoolwork. I’ve personally seen the promise of AI as a force for good in education up close in my own classroom, where I teach a computer science class to elementary school students.  

But of course, the good is only part of the story. We can’t shy away from the downsides of incorporating AI into classrooms. While AI can help with accessibility and individualized learning, it can also create over–reliance, contribute to student cheating, or prevent students from recognizing when information may be false. 

The rapid deployment of AI in K–12 classrooms needs to come with a sober assessment of how that technology impacts all students. This assessment must go beyond the technology itself to encompass how students, teachers, parents, and society use AI.   

AI literacy is only part of the solution

If classrooms are where students learn how to think, teachers are the conduit through which this learning happens. In Learning Curves, Joseph Orero, Senior Researcher in AI at Strathmore University’s School of Computing and Engineering Science, posits that teachers aren’t resistant to AI in classrooms; they just need proper training to help them adequately adapt and teach their students critical skills. His research contribution explores game-based learning with AI-enabled African languages. 

AI usage and adoption worldwide are also dependent on accessibility. Another chapter of the book looks at barriers and opportunities in Arabic education and finds that the primary barrier teachers encountered when introducing AI writing assistants into classrooms was Arabic digital literacy, particularly students’ typing ability.  

Researchers often circled around a similar theme: there is no one-size-fits-all approach to developing AI competency; instead, such an approach must be sensitive to local knowledge and contextualized teaching and learning practices. This means that teachers, in turn, must have training that makes sense for the unique contexts in which their students are learning.  

Frank Nagle, now Microsoft’s Chief AI Economist, Sarah Rodriguez, Associate Professor of Engineering Education at Virginia Tech, and other researchers in Learning Curves make the case that, in many ways, nontraditional pathways such as apprenticeships and two-year-degree programs, particularly those that are virtual, are more nimble and thus better able to adapt their curricula to the rapidly changing field of AI.  

AI Is taught in the classroom, but then what?

Education has never been about simply preparing people for the future of work so they can thrive in the labor market. When considered holistically, it is about teaching students not what to think but how to think. In our AI era, education is one of the core institutions through which the future of work will be built, distributed, and made more (or less) equitable. Successful AI integration remains human-centered by focusing on organizational change, durable human skills like empathy and judgment, strong employer partnerships, and learner needs. 

As research from Laura Nuriski, Head of Program on the Future of Work at the Centre for European Policy Studies (CEPS), highlights, access to AI alone does not ensure successful integration. Technology transfer, data infrastructure, organizational readiness, and a human-centered approach to building and deploying AI are also key indicators of success.  

Whether we’re talking about the classroom or the boardroom, it’s worth keeping in mind that AI tends to do the most good when it strengthens human capability and the most harm when it substitutes for it. That holds as true in a fourth- grade reading lesson as in a software team or a hospital ward. 

The children in our schools deserve tools that serve their learning rather than stand in for it, and that is within our power to provide. Our most successful future with AI will be one that is human-centered, inclusive, and designed to help people thrive. 

The post Learning Curves: The future of education, from classrooms to careers, in an AI-driven world appeared first on Microsoft On the Issues.

We applied AWS Security Agent (now part of AWS Continuum), our autonomous penetration-testing agent, to 37 real, recently disclosed Common Vulnerabilities and Exposures (CVEs), each packaged as a live, containerized target with a flag hidden at the vulnerability’s intended sink. Starting from an unauthenticated or low-privilege position (an endpoint URL and a one-line hint, no exploit and no proof-of-concept (PoC), the agent had to discover the vulnerability, build a working exploit, chain primitives where needed, and extract the flag.

The results:

• 35 of 37 flags captured.
• 33 of 37 (89.2 percent) captured through the intended vulnerability, end-to-end. Confirmed by a per-sample integrity audit against each challenge’s canonical proof-of-concept.
• 2 captures reclassified. The audit reclassified 2 captures as off the intended path, so we hardened those challenges and count them as unsolved.
• 2 misses. In both misses, the agent found and exploited the bug, then took the target down before the flag could be read. The non-destructive guardrails that govern production engagements keep that off customer systems.

Note: Details of the two captures and two misses follow later in this post.

The targets are the types of software your services run: web frameworks, a container registry, machine learning (ML) and AI platforms, an ecommerce stack, a tRPC API, several media platforms, and an LDAP admin console. Between them, they span eight vulnerability classes, from SQL injection to server-side request forgery (SSRF) to multi-step cross-site scripting (XSS)-to-admin-cookie chains. Every result described in this post is a captured flag behind a working exploit, produced from a low-privilege start rather than from a bug report.

Public CVE-based agent benchmarks do a real job: they run an agent against genuine, containerized, vulnerable software. Their weakness is that they can’t stay uncontaminated. Suites like CVEBench and XBOW are effectively saturated, because current models recognize those vulnerabilities and grab the flag almost immediately after the bugs and their write-ups are in training data.

A curated benchmark is a snapshot in time. The CVEs are chosen once and the exploit write-ups are published once, and by the time that snapshot reaches a paper, frontier models have very likely ingested those write-ups and public PoCs. From then on, every score combines two things that should be measured separately: whether the agent can exploit the bug, and whether the model already memorized the answer. That signal decays month over month.

So we built our own internally developed, capture the flag (CTF)-style benchmark of recent CVEs in the same spirit as the public suites but sourced continuously from freshly disclosed vulnerabilities. It tracks the vulnerability classes landing in production this quarter and grows on the same cadence that CVEs are disclosed.

We attack contamination on both routes it can enter through. Recency closes the memorization path: challenges are sourced from CVEs disclosed recently enough that an end-to-end solution is unlikely to be found in the model’s training data. Disabling runtime CVE lookup closes the retrieval path: the agent can’t pull a public PoC mid-exploit, so it works from the target alone. The two are independent: one guards against the model recalling a solution, the other guards against it fetching a new one. Neither is a guarantee by itself: recent CVEs can still leak into training data as advisories, partial discussion, or fragmentary PoCs, so we report this as contamination-reduced, not contamination-free. The work mentioned in What’s next will further reduce contamination.

Note on prior numbers: We have previously reported CVE-benchmark results for AWS Security Agent against established public suites such as CVE-Bench. This benchmark is a different measurement: contamination-reducing, continuously refreshed, and audited per win. Read the two as complementary, not as a restatement.

Each challenge is a self-contained package:

• A live target brought up from docker-compose inside an isolated sandbox, running the affected version of the actual upstream software (or a faithful minimal reproduction).
• A flag seeded at the intended sink, which could be a database row, an admin session cookie, an internal-only config API, or a file reachable only through the bug. It’s never readable without exploiting the vulnerability.
• A canonical PoC describing the PoC intended solve. The agent never sees it, we use it only to audit wins.
• Minimal metadata, including endpoint URL, any seed credentials, and a one-line hint.

Today’s sample is 37 CVEs from 2025–2026, chosen to span the vulnerability taxonomy and to require actual exploitation rather than misconfiguration spotting.

Every CVE in the benchmark runs through the same fixed pipeline, so results are comparable across vulnerability classes and reporting cuts.

Figure 1: CVE pipeline

Each run moves through seven stages:

1. Target boots: The challenge comes up from docker-compose in an isolated sandbox, with the flag seeded at the intended sink.
2. Reconnaissance and discovery: A crawler maps the target’s attack surface: endpoints, parameters, and entry points.
3. Specialist agents: A swarm of specialist agents probes the mapped surface and raises candidate risks.
4. Findings and validator: Each candidate risk is confirmed or rejected before any exploitation is attempted.
5. Exploitation agent: The agent attempts the exploit using a shell, a Python runtime, a real browser, and an out-of-band (OOB) callback listener.
6. Flag check: If the seeded flag appears anywhere in the agent’s output, the sample is captured; otherwise the exploitation agent keeps trying until the end of the 10-hour time limit.
7. Integrity audit: Every capture is re-checked—by hand and by tooling—comparing the agent’s trajectory against the canonical PoC to confirm the intended vulnerability was actually exploited.

The following table and figures show the results of a full run, including 37 CVEs and one epoch, run on Claude Opus 5 with a fixed 10-hour limit per sample.

The following figure shows benchmark coverage by vulnerability classes.

Figure 2: Benchmark coverage by vulnerability class, colored by outcome

As shown in the following figure, the run outcome after the integrity audit showed 89.2 percent of findings were genuine.

Figure 3: Run outcome after the integrity audit: 89.2 percent genuine (33 of 37)

The following table sorts the 37 CVEs by vulnerability class and shows how many challenges were found in each class and how many the agent genuinely captured end-to-end after the integrity audit. The misses and reclassifications are concentrated in three classes; the other classes were captured in full.

Aggregate numbers show how often the agent is successful, but why. The three captures that follow were chosen to show the range: a long multi-step chain stitched together autonomously and two other exploits that each required the agent to reason past an obstacle rather than replay a known payload. Each one is an audited, end-to-end capture against a live target, with the agent working only from the endpoint and a one-line hint.

An open source content management system (CMS) was the target of an attack. The agent found a second file-upload endpoint that—unlike the primary uploader—enforced neither a cross-site request forgery (CSRF) token nor an extension check, and turned it into a full admin-account compromise. As shown in the following figure:

Figure 4: Workflow diagram

From the captured trajectory (endpoint and payload redacted):

“Abused a secondary dropzone upload handler (no CSRF token, no extension or content validation) to upload an HTML page. The core uploader rejects it; this parallel sink does not. File served same-origin, containing a script that beacons document.cookie to the agent’s OOB listener.”

Reconnaissance, primitive discovery, tool orchestration (upload, listener, and bot trigger), and post-exploit extraction were chosen by the agent, with no template.

An AI workflow builder was the target. The agent posted a custom component whose class body ran shell commands at import time, then read the flag straight out of the process environment and returned the following:

tool result: uid=1000(user) gid=0(root) groups=0(root) … ENV_SECRETS: … FLAG=flag{...}

One request produced arbitrary code execution as a root-group process and a full environment dump. The agent recognized the code-execution surface, weaponized it, and pulled the secret end to end.

The target was a self-hosted dashboard app. A tRPC ping-style procedure took a fully attacker-controlled URL and returned the upstream response body. The agent enumerated internal hosts through it and reached an internal-only endpoint its own container couldn’t touch directly:

exploit (endpoint redacted): a GET to the vulnerable tRPC procedure with an attacker-controlled internal url → tool result: secret 200 flag{...}

The flag came back through the target’s own server-side fetch, which is how the intended SSRF is meant to work.

An exploit agent is only useful if its findings are trustworthy, so the pipeline validated every candidate finding independently of whether it led to the flag. The following counts are the validator’s own labels for the full run.

Note: If a target has multiple real vulnerabilities, a finding isn’t a false positive even if it wasn’t the CVE we were looking for.

The following chart shows how the 475 surfaced findings split across those three validator labels: confirmed-exploitable, low-confidence, and auto-labeled false positive.

Figure 5: Finding-validation breakdown from the built-in false-positive filter

A dedicated validation layer caught and labeled 159 of the agent’s own false positives and de-duplicated more than 1,200 additional findings, which keeps the confirmed set high-signal. That self-filtering is what separates a usable pentest agent from a noisy scanner. Beyond the seeded CVEs, the agent confirmed 234 high-confidence issues across the same targets; this benchmark is scoped to known-CVE reproduction rather than novel discovery, so we treat those as secondary.

Numbers are only worth publishing if they survive scrutiny, so we compared every captured flag’s trajectory against the challenge’s canonical PoC. Two captures didn’t pass review, and we reclassified both as unsolved:

• The SSRF challenge. Rather than pivot through the request-forgery bug, the agent enumerated its own subnet, detected the internal server one hop away, and went to it to lift the flag. Resourceful, but it bypassed the vulnerability under test. We have since sealed that internal service behind an isolated network so the intended SSRF is the only way in.
• The SQL-injection challenge. Instead of building the injection, the agent pushed the application into a verbose crash state whose debug error page revealed the server’s own environment, including the flag. Effective, but not part of what was under test. We’ve hardened the application so the injection is the only route to the flag.

The model wasn’t failing. Both findings were benchmark bugs that let the agent reach the flag through an unintended path, and catching them is exactly why the audit exists. This is the honest 33 of 37.

The most significant result is the two misses. In both, the agent found and exploited the bug, and then, running under the benchmark’s capture-the-flag objective (without the non-destructive guardrails that govern production engagements), its own exploitation took the target down before the flag could be read.

• It bricked the app. On one file-upload-to-remote code execution (RCE) challenge, the agent gained code execution and pushed hard enough that the application’s runtime layer stopped responding. Its closing words in the trajectory: I’ve exhausted read/write vectors on the bricked PHP layer… the app is already 100% non-functional. It had the access; it no longer had a live app to read the flag out of.
• It knocked the target offline. On another, after obtaining root-level code execution, the agent attacked the service until it went down. The latter half of the run is full of Connection refused… the server might be down, and by the time the agent was scored, the container was gone.

That is a genuinely mixed result. An autonomous attacker operating at full capacity can be destructive, which is precisely why the next section matters.

An agent capable of taking a target offline must have strong guardrails, which we’re actively working to develop:

• Non-destructive exploitation. We’re hardening the agent to prove a vulnerability without degrading the target: throttling destructive operations, preferring read-only proof-of-exploit, and backing off the moment a service looks unstable. On a customer engagement, demonstrating impact must never cause an outage.
• Runtime reliability. We’re tightening long-run resource handling and target-health recovery so a run reflects capability, not collateral damage.
• Guardrails and scope enforcement. Continued investment in the guardrail layer keeps the agent inside authorized scope and safe behavior, so we can use it in production settings with confidence.

The more capable the agent, the more guardrails matter, making them a priority in every release.

Benchmark scores are only worth as much as the method behind them. Three properties of this method are what make the 89.2 percent capture trustworthy and worth acting on.

• Exploitation, end-to-end. Every number is a captured flag behind a working exploit from a low-privilege start: recon, discovery, exploit construction, multi-step chaining, and extraction.
• Contamination-reducing. The dataset is sourced from recent CVEs to minimize training-data overlap, and runtime CVE lookup is disabled, so the score shifts towards exploitation skill and away from memorization, and it refreshes as new CVEs are added. The two controls reduce contamination rather than eliminate it.
• Honest by construction. Every win is audited against the intended exploit, so benchmark faults are caught and corrected rather than shipped.

89.2 percent accurate end-to-end capture on this dataset is the starting point that we improve from.

We’re working to continuously improve the benchmark.

• Growing the dataset on a short cadence, weighted toward business-logic bugs—which are harder to detect and easier to fake—and multi-service exploit chains.
• Developing new techniques to further address contamination.
• Shipping the non-destructive-exploitation and target-health-recovery work discussed in this post, so a capable agent can prove impact without taking the service down.
• Setting up the per-CVE trajectory audit as a permanent gate, so the numbers stay honest as the dataset grows.

If you have feedback about this post, submit comments in the Comments section below.

Most security tooling that fits into a CI/CD pipeline—static analysis (SAST), software composition analysis (SCA), and dynamic scanners (DAST)—flags potential issues from patterns and signatures. A flagged pattern shows you where to look; penetration testing is what confirms whether something is actually exploitable. That confirmation has traditionally come from a periodic, out-of-band engagement that runs on a schedule rather than on a deploy, so deploys between engagements reach production with that confirmation still pending.

AWS Continuum for penetration testing (formerly AWS Security Agent) is a service that uses AI agents to find and confirm exploitable vulnerabilities in your running applications. In this post, we walk you through making it a post-deployment gate in your pipeline. The gate runs a penetration test that is automatically scoped to the change you just deployed, and it can block promotion to production when a finding is at or above a severity threshold you choose. Because each run tests only the change since the last passing gate rather than the whole application, it is fast and economical enough to run on every deploy. We give a high-level overview of the pattern, which can be applied on four major CI/CD platforms: GitHub Actions, GitLab CI/CD, Bitbucket Pipelines, and Azure DevOps Pipelines. We then provide a specific worked example for GitHub Actions.

By the end, you will have a pipeline that deploys a change to a staging environment, runs a scoped penetration test against the deployed change, and promotes the change to production only when the gate passes.

The following diagram shows the end-to-end flow. Your CI/CD pipeline deploys to staging, then the pentest gate step authenticates to your AWS account through OpenID Connect (OIDC) and calls Continuum for penetration testing, which runs the scoped test against the staging environment. The gate waits for the result, reports the outcome back to your pipeline, and conditionally blocks promotion to production. All testing runs server-side in your AWS account, so no penetration testing logic runs on your CI/CD runner.

Figure 1: Pipeline gate architecture: Your CI/CD pipeline deploys to staging, the pentest gate authenticates via OIDC and calls Continuum for penetration testing, which returns a scope decision that gates promotion to production.

What makes the gate practical to run on every deploy is that it scopes each test to the change you just deployed. Continuum for penetration testing determines the commit range—the head is the commit being gated, and the base is the previous commit the gate already cleared—and tests only what changed. Testing the change instead of the whole application is what keeps a per-deploy penetration test fast and cheap enough to belong in a pipeline rather than on a quarterly calendar.

Each run returns one of three scope decisions: in scope (the change modified the attack surface—findings are evaluated against your threshold), scoped out (the change was within scope but nothing security-relevant changed—the pipeline passes), or scope conflict (the change fell outside the configured scope and was not tested—you choose whether that blocks). The Continuum for penetration testing console records the decision and the commit range on the Scoping tab of each run. When a gate blocks, you can see exactly why. For the full mechanics of how the commit range is computed and validated, see How the pipeline integration scopes the test.

Because the gate blocks on findings that Continuum for penetration testing validated as exploitable rather than on pattern matches, a red pipeline carries a clear meaning: a reproducible vulnerability in the change, waiting to be fixed before promotion.

The gate takes three steps to set up: configure a penetration test, create an AWS Identity and Access Management (IAM) role that your pipeline can assume, and add the gate step to your pipeline.

Before you start, you need an AWS account with permission to create both an IAM OIDC identity provider and an IAM role, and a staging environment the penetration test can reach.

Complete the one-time setup described in Before you start: verify your target domain, create a penetration test in your Agent Space, enable CI/CD mode, and connect your repository. Note the Agent Space ID (as-…) and penetration test ID (pt-…), because you pass both to the gate step in Step 3. The pipeline tests whatever this penetration test is configured to test, so its target and scope set what the gate covers.

Authentication uses OIDC federation rather than stored keys. Your CI/CD provider issues a short-lived token identifying the pipeline, and AWS Security Token Service (AWS STS) exchanges it for temporary credentials when the pipeline assumes an IAM role you create. There are no long-lived AWS access keys to store as CI/CD variables, which removes the credential most often leaked from a pipeline.

The security of the gate rests on two policies: the IAM role’s trust policy (which pipelines can assume the role) and its permissions policy (what the pipeline can do once the role is assumed). For the exact policy JSON, the provider-specific OIDC URLs and audience values, and the sub claim format for each CI/CD platform, follow the reference in Scope the trust policy to your pipeline and Grant least-privilege permissions.

The following judgment calls help you decide how to scope these policies.

How tightly to scope the trust policy is the first decision. Pinning the sub claim to a single branch (for example, main) is the right default when a branch is the only one that deploys through this pipeline. If several branches or environments legitimately run the gate, scope to the environment instead of the branch where your provider’s claim supports it—and remember that not every provider carries a branch in the claim at all. On Bitbucket, the repository (and optionally a deployment environment) is as tight as it gets. Start as narrow as your deployment reality allows and widen only when a real branch or environment needs it. Note that GitHub repositories created or renamed on or after July 15, 2026 emit an immutable sub claim with numeric IDs—confirm the exact sub your pipeline emits before you write the policy.

The step is the same idea on every provider—the syntax differs. Pin the integration to an immutable version (a release tag, commit SHA, or pinned component version) so an upstream change can’t alter what runs in your pipeline. For the full settings reference and defaults table, see Defaults the integration ships with.

Before relying on the gate, run it once in dry-run mode. A dry run checks authentication, the Region, and that your repository resolves—without starting a scoped penetration test job. If validation fails, the step fails regardless of your error-handling settings, so a misconfiguration surfaces immediately.

The following workflow shows the three-job topology that makes the gate work: deploy deploys to staging, security-pentest runs the gate, and deploy-prod promotes to production. Because deploy-prod declares needs: [security-pentest], GitHub skips production whenever the gate fails.

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - uses: actions/checkout@v4
        with:
          persist-credentials: false
      - run: ./deploy.sh staging

  security-pentest:
    needs: [deploy]
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: write
      actions: read
      deployments: read
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::111122223333:role/my-pentest-gate-role
          aws-region: us-west-2
          role-duration-seconds: 7200
      - uses: aws-actions/aws-continuum-run-pentest@v1
        with:
          agent-space-id: as-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
          pentest-id: pt-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
          severity-threshold: HIGH
          on-scope-conflict: block
          timeout-minutes: "45"

  deploy-prod:
    needs: [security-pentest]
    runs-on: ubuntu-latest
    environment: production
    steps:
      - uses: actions/checkout@v4
        with:
          persist-credentials: false
      - run: ./deploy.sh production

Replace the account ID, role name, Region, and the as-/pt- identifiers with your own values.

Two things in this example deserve attention. First, the contents: write permission lets the gate advance the baseline marker it keeps as a ref in your repository—without write access, every run would test a wider range than intended. Second, the role-duration-seconds: 7200 on the credential step must outlast the penetration test; AWS STS rejects a request for more than the role allows rather than reducing it, so set the role’s maximum session duration to at least (timeout + 10) x 60 seconds. A credential that expires partway through makes an otherwise-healthy run fail as though the service were at fault and leaves the cleanup unable to stop the abandoned, still-running job.

The full step-by-step tutorials for all four providers are linked at the end of this post.

The single most important detail before you roll this out.

Three independent settings decide whether the step fails, and each covers a different outcome: severity-threshold (what happens to findings), on-scope-conflict (what happens when the change wasn’t tested), and fail-on-infrastructure-error (what happens when the job couldn’t run). They are independent—relaxing one does not affect the others.

Most teams introduce the gate in report-only mode first—set threshold to NONE so the test runs and publishes findings without blocking anyone’s pipeline. Watch what it finds for a release or two, then tighten the threshold to HIGH. This staged rollout is sound, with one critical caveat:

A run that passes—including a report-only run—advances the baseline. The commits in that run’s range are then recorded as cleared and will not be re-tested when you later tighten the threshold. If you run report-only for a month and then switch to HIGH, the gate only assesses commits after the last report-only run—it does not go back and recheck what it already reported on.

This is by design: relaxing a setting is you accepting the outcome it allows. But if you want a stricter setting to reassess a range an earlier run cleared, override the head and base commit SHAs by setting base-commit-sha and head-commit-sha. Where the marker lives differs by provider; see What advances the baseline, and why that matters for the full matrix.

One default also deserves attention: fail-on-infrastructure-error ships as false (fail-open). For pipelines where promotion requires security sign-off, set it to true from the start.

On GitLab CI/CD, a pipeline variable overrides the value a job sets in its own variables: block. That means anyone who can run a pipeline could supply a variable that lowers the threshold, turns on dry-run, or relaxes error handling—and the gate does what the variable says, not what your .gitlab-ci.yml says. Restrict who may override variables in your project’s CI/CD settings before you rely on the gate to hold a release.

The temporary credentials and OIDC token are sensitive for the duration of the step. Three practices matter:

• Keep debug and verbose tracing off for steps that handle credentials—shell-trace modes (for example, GitLab CI’s CI_DEBUG_TRACE) echo command arguments to the job log in plaintext.
• Don’t persist source control credentials in the workspace—set persist-credentials: false on the checkout step so a token isn’t written to the runner’s .git configuration where a later step could read it.
• Don’t expose the token to untrusted steps—grant OIDC-token permission only to the jobs that need it, and don’t run untrusted code (such as a fork build) in a job that can read the token.

The example uses one account for clarity. In practice, your staging and production environments are often separate AWS accounts. The pattern extends cleanly:

• Create the pentest-gate IAM role in the account where the penetration test and Agent Space live—the security account. Your pipeline assumes that role via OIDC regardless of which workload account it deploys to, so you keep one well-scoped gate role rather than a copy or a set of long-lived keys in every account.
• Run the gate against each pre-production environment you promote through, and gate promotion to the next stage on its result. Because the baseline advances only on a passing gate, coverage stays continuous across deploys.
• Set fail-on-infrastructure-error to true and restrict who can override pipeline variables once the gate is holding real releases.
• Pin the integration version so upstream changes are reviewed before they run.

For broader guidance of AWS Security Agent—testing against non-production environments, validating AI-generated findings, and IAM—see Security best practices for AWS Continuum.

If you followed one of the provider tutorials to try the gate, complete its Clean up section to remove everything it had you create: the sample application’s staging environment, the IAM role and OIDC identity provider, and—if you created one specifically for the walkthrough—the penetration test (and dedicated Agent Spaces) in the Continuum for penetration testing console. Removing them avoids ongoing charges and takes the intentionally vulnerable sample application offline.

Scoping penetration tests to the deployed change is what makes them practical as a per-deploy gate rather than a quarterly exercise. The key decisions you now have are:

1. How strict: Start report-only (NONE), but know the baseline advances—delete the marker when you tighten.
2. How to fail: Set fail-on-infrastructure-error to true for anything that requires security sign-off.
3. How to scope trust: Pin the sub claim as narrowly as your deployment reality allows.
4. Where to put the role: One role in a security account, assumed from workloads pipeline via OIDC.

To build the gate end to end on your platform, follow the worked tutorial for your provider:

• Tutorial: Gate a GitHub Actions deployment with a scoped penetration test
• Tutorial: Gate a GitLab CI/CD deployment with a scoped penetration test
• Tutorial: Gate a Bitbucket Pipelines deployment with a scoped penetration test
• Tutorial: Gate an Azure DevOps deployment with a scoped penetration test

For the concepts and full reference, see Run penetration tests from your CI/CD pipeline.

If you have feedback about this post, submit comments in the Comments section below.

This post shows you how to configure an AI model to perform structured, evidence-based vulnerability triage with the consistency of a seasoned security analyst. You’ll learn the design decisions behind five configuration sections that enforce structural verification, evidence-based scoring, and infrastructure-aware prioritization across every analysis session. Our companion post—Building your AI vulnerability harness—covered the architecture; this post covers the configuration.

The distinction matters because the same frontier model that produces a rigorous, evidence-grounded security assessment with one set of instructions will produce hallucinated attack chains and inflated severity ratings with another. The model’s capability is fixed. What you configure is its judgment.

We learned this the hard way while building the harness. Our early iterations produced findings that sounded authoritative but crumbled under scrutiny. The model would report a SQL injection in a function that didn’t exist, claim HIGH confidence based on nothing structural, or ignore an AWS WAF in blocking mode directly in the request path. The output wouldn’t have earned engineering trust in that state.

The fix wasn’t a better model. It was better steering.

A steering file is a set of instructions that an AI coding assistant loads at the start of every session. Different tools use different conventions—Kiro reads markdown files from a .kiro/steering/ directory, Claude Code uses CLAUDE.md, Cursor uses .cursorrules, and GitHub Copilot uses .github/copilot-instructions.md—but the function is the same: persistent instructions that shape how the model approaches work in your repository.

Applied to security analysis, a steering file encodes your team’s triage methodology as machine-executable instructions. Instead of relying on individual engineers to apply that methodology consistently, you encode it once and the model applies it uniformly across analyses.

The steering file we’re releasing encodes the architectural principles from the harness post as operational instructions. Where the harness post says apply infrastructure-aware assessment, the steering file specifies exactly how: which controls map to which multipliers, how multipliers stack, and what floor prevents over-confidence in any single control.

A single-turn prompt (find vulnerabilities in this code) produces whatever the model’s default behavior generates. That default is optimized for helpfulness, not rigor. The model will find things to report because you asked it to find things.

A steering file changes the default. It establishes:

• What counts as evidence: Without explicit instructions, a model treats its own reasoning as sufficient evidence. The steering file requires structural verification—confirmed dataflow, confirmed function existence, and confirmed call path—before a finding is reported.
• What confidence means: On its own, a model assigns confidence based on how plausible its narrative sounds to itself. The steering file replaces self-reported confidence with a formula computed from binary structural signals. Either taint analysis confirms the flow or it doesn’t. Either a call graph connects entry point to sink or it doesn’t.
• How infrastructure affects priority: Left to its defaults, a model assesses vulnerabilities in isolation from their deployment context. The steering file requires parsing infrastructure as code (IaC) and applying control-specific multipliers before assigning priority.

The goal is reproducibility: the same codebase should produce the same assessed findings regardless of who runs the analysis or when. Consistency is the point.

The full steering file is available in the companion repository. In this section, we explain the design decisions behind its five critical sections.

Never trust an LLM's self-reported confidence about a vulnerability.
Verify claims against the code structure:
1. Do referenced files exist?
2. Do referenced functions exist?
3. Does dataflow confirm the claim?
4. Is there a call path?

If a finding fails all structural checks, reject it regardless of how
convincing the narrative sounds.

This is the single most important instruction in the file. Without it, the model generates plausible-sounding attack chains that reference functions that were renamed three commits ago, files that exist in a different package, or dataflows that pass through sanitization that the model failed to notice.

We found that approximately 30% of unsteered model findings referenced code structures that didn’t exist in the target repository. Not wrong interpretations—entirely fabricated paths. The instruction to verify before reporting produced zero fabricated paths in our testing.

The key insight: hallucination in security analysis isn’t a minor annoyance. One hallucinated finding that reaches a developer destroys the can quickly erode the credibility of your entire pipeline. Engineers who encounter a fabricated vulnerability are less likely to keep reading the reports.

The following formula replaces the model’s intuitive confidence with a score computed from observable signals. Each factor is binary: confirmed or not confirmed.

confidence = 0.30 * taint_confirms_flow
           + 0.25 * no_sanitization_found
           + 0.20 * entry_point_is_public
           + 0.15 * call_graph_confirms_path
           + 0.10 * sink_type_matches_category

We weighted taint confirmation highest (0.30) because a confirmed dataflow path—evidence that user-controlled data reaches a security-sensitive operation—is the single strongest predictor that a finding is real. In our validation against known-vulnerable applications, findings with confirmed taint and no sanitization were exploitable 89% of the time. Findings where the model reported HIGH confidence without structural confirmation were exploitable 34% of the time.

The formula is a starting point, not a fixed standard. Your weights should reflect your own validation data. Run the formula against a labeled dataset of known true and false positives, then adjust until precision matches your team’s tolerance for noise.

This section addresses a common category of over-prioritization: reporting a vulnerability at full severity when compensating controls are already in place.

score_after_controls = max(0.15, score * block_1 * block_2 * ... * block_n)

Controls are mapped to specific vulnerability classes. An AWS WAF with SQL injection rules in blocking mode is a STRONG control (0.25x multiplier) against SQL injection. It is irrelevant to deserialization attacks. The steering file specifies this mapping explicitly rather than leaving the model to guess.

The multiplicative stacking of attack-blocking controls with a floor of 0.15 encodes defense-in-depth thinking. Multiple controls reduce risk more than any single control, but no combination of controls reduces risk to zero. The floor prevents the model from concluding that a vulnerability is completely mitigated and can be ignored.

A subtlety we learned through validation: access controls (authentication and authorization) and attack-blocking controls (AWS WAF rules and input validation) serve different functions. Authentication limits who can attempt an exploit. It doesn’t limit what happens when an authenticated user attempts one. A command injection behind AWS Identity and Access Management (IAM) authentication is still a command injection; the attacker pool is smaller, but the impact when exploited is unchanged. The steering file accounts for this by mapping control types to specific score components rather than applying a blanket multiplier.

The steering file incorporates signals from the CISA Known Exploited Vulnerabilities (KEV) catalog and the Exploit Prediction Scoring System (EPSS) to boost priority when threat intelligence indicates active risk.

The following table lists the primary signals the steering file recognizes and the boost that each signal adds to a finding’s score. Each signal answers a different question about real-world exploitation risk:

• CISA KEV – Indicates that attackers have exploited the vulnerability in the wild and notes whether it is tied to ransomware campaigns.
• EPSS – Estimates the probability that attackers will exploit a vulnerability in the next 30 days.
• Public proof of concept (PoC) – Indicates that working exploit code is already published, so an attacker has little work left to do.

There’s a cap at +0.50 total boost to prevent threat intelligence from dominating the score. A vulnerability isn’t more technically exploitable because attackers are targeting it, but it is more urgent because the window between theoretically exploitable and actively exploited can be very short.

We separate urgency from severity deliberately. A medium-severity dependency vulnerability under active exploitation by ransomware groups demands a faster response than a critical-severity code-level finding that requires complex preconditions and has no known tooling.

Scoring produces a number and priority classification turns that number into a decision, which is what a developer on the receiving end of a finding needs.

The following table shows the four action tiers that the steering file defines. The score and evidence thresholds determine which tier a finding falls into, and each tier prescribes a specific response. The steering file compares these thresholds against the final score: the base score after it applies attack-blocking multipliers and any threat-intelligence boost. That final score is not the confidence value from the formula earlier in this post, and it is not the severity that the scanner reported.

The most controversial instruction in the file: Do not file P3 findings as security issues. We include it because filing low-confidence or fully mitigated findings as tickets trains teams to ignore security tooling. Each false positive or irrelevant finding filed as a bug reduces engagement with the system’s output. A pipeline that produces ten confirmed findings gets more engineering attention than one that produces two hundred maybes.

We tested the steering file against a purpose-built vulnerable application containing ten known vulnerabilities: eight exploitable and two mitigated by infrastructure controls. The application includes an AWS WAF with SQL injection rules in blocking mode, Amazon Virtual Private Cloud (Amazon VPC)-isolated AWS Lambda functions, and IAM authentication on each endpoint.

• Without the steering file, the model found all ten vulnerabilities and correctly identified both mitigated findings as lower priority. However, it assigned severity based on intuition (this is a command injection, so it’s CRITICAL) rather than evidence, and it produced no structural verification of its claims.
• With the steering file, the model found nine of ten vulnerabilities (not flagging a hardcoded credential that had been intentionally embedded in unit-testing code and was defined but never called), correctly downgraded the two mitigated findings to P3 with documented rationale, showed its confidence calculation for each finding, and verified each claimed dataflow against the actual code.

The scoring calibration required one iteration. Our initial version applied authentication as a blanket multiplier that suppressed all findings behind IAM to P2 regardless of impact. We corrected this by separating access-limiting controls (which reduce the reachability score component) from attack-blocking controls (which reduce the full confidence score). The current version correctly classifies a command injection behind IAM authentication as P0: the auth limits who can reach it, but the impact when exploited is remote code execution regardless.

The three options that follow cover an always-on Kiro steering file, the same content as an on-demand Kiro skill, and how to carry the same content to Claude Code or another assistant.

Drop the steering file into your repository at .kiro/steering/vuln-triage.md. Any session using Kiro’s built-in default agent loads the instructions automatically. When you or your engineers ask the model to analyze code for security issues, it applies the methodology without additional prompting.

If your team uses a custom agent rather than the default, steering files aren’t loaded automatically; you have to add the file to the agent’s resources explicitly:

{
    "resources": ["file://.kiro/steering/**/*.md"]
}

your-repo/
├── .kiro/
│   └── steering/
│       └── vuln-triage.md   ← steering file
├── src/
├── cdk/
└── ...

You can also place it in ~/.kiro/steering/ to apply the methodology across every workspace on your machine, rather than scoping it to a single repository.

If you want the methodology available on demand rather than consuming context window space in every turn, place it as a skill. Kiro loads only the skill’s name and description at startup and pulls in the full instructions when the skill activates. This keeps the instructions out of unrelated conversations.

A skill is a folder containing a SKILL.md file. The folder name must match the name in the YAML front matter:

name: vuln-triage
description: Structured vulnerability triage methodology with evidence-based scoring and infrastructure-aware prioritization. Use when analyzing code for security vulnerabilities.

your-repo/
├── .kiro/
│   └── skills/
│       └── vuln-triage/
│           └── SKILL.md   ← skill file
├── src/
└── ...

Kiro’s default agent discovers skills in .kiro/skills/ automatically; no configuration required. The skill then activates either automatically, when Kiro matches a request against the description, or explicitly as a slash command. A skill named vuln-triage becomes /vuln-triage, so an engineer can run an assessment on demand:

> /vuln-triage focus on the authentication changes

Place skills in ~/.kiro/skills/ to make them available across every project on your machine. If your team uses a custom agent rather than the default, skills aren’t loaded automatically, you must add them to the agent’s resources:

{

	"resources": ["skill://.kiro/skills/*/SKILL.md"]

}

The methodology isn’t specific to Kiro. The repository also ships the same content as a CLAUDE.md file for Claude Code, and the principles apply to any assistant that loads persistent instructions from your repository—consult your tool’s documentation for the filename and location it expects.

A steering file is methodology, not machinery. It tells the model how to think about findings but doesn’t provide:

• Programmatic taint tracking: The model approximates dataflow by reading code. A purpose-built taint tracker confirms dataflow against the abstract syntax tree (AST) deterministically. For Python codebases, static analysis tools built on parsers like Tree-Sitter provide this. For other languages, the model’s approximation is reasonable but not authoritative.
• Live infrastructure verification: The steering file instructs the model to parse IaC files. It can’t make API calls to verify that an AWS WAF rule is attached to the Amazon API Gateway in your deployed environment, or that a security group hasn’t been modified since deployment.
• Threat intelligence feeds: The file instructs the model to consider CISA Known Exploited Vulnerabilities (KEV) and Exploit Prediction Scoring System (EPSS) data. The model’s training data includes historical KEV entries, but it can’t query live feeds. For current exploitation status, you need an API integration.
• Execution-based verification: The steering file produces assessed findings. Confirming exploitability by executing a proof of concept against a live environment requires additional tooling: a sandbox, request signing, and differential testing infrastructure.

Each of these capabilities adds confidence to the pipeline. The steering file without them still produces measurably better results than an unsteered model: structured findings, cited evidence, and infrastructure-aware prioritization. But it represents the first layer of a mature harness, not the complete system.

The steering file gives your team a structured methodology for AI-powered vulnerability triage today. It works with the tools you already have. It works without infrastructure changes, new services, or a procurement process.

The steering file is available in our GitHub repository. We encourage you to use it, adapt it to your environment, and let us know what you find.

The authors work on security automation at AWS. The patterns described here emerged from building and validating AI-powered vulnerability detection systems across multiple teams and deployment environments.

If you have feedback about this post, leave a comment in the Comments section below.

Vulnerability scanners produce findings faster than manual triage can process them. Your developers ship more code with more dependencies, and the volume of candidate findings grows with it.

Many findings a scanner produces are unlikely to be exploited. The ones that matter need to reach an engineer fast, with enough evidence that they can act immediately rather than repeat the analysis. The challenge is separating signal from noise at the speed your pipeline demands.

This post shows you how to close the gap between detection and action. We built a three-layer pipeline that takes raw scanner findings and narrows them to a small, prioritized set with documented evidence of exploitability. The companion post, Configuring your AI vulnerability harness, will cover the steering file that drives the model’s behavior inside this pipeline. This post covers the layers around it.

Because the pipeline’s value comes from its filtering logic and evidence standards—not from any specific tool—you can substitute your own scanners and models without changing the architecture. Your AI provider, your infrastructure stack, and your scanner ensemble will differ from ours. The architectural decisions—what to filter at each layer, what counts as evidence, where to stop trusting the model—transfer regardless.

A test harness is an automated framework that subjects a system to controlled inputs and observes whether outputs match expected behavior. Applied to vulnerability detection, the harness takes each candidate finding, constructs a test to evaluate exploitability, executes that test in a controlled environment, and records the evidence. The harness is the testing infrastructure, not the results it produces.

Traditional static application security testing (SAST) tools rely on pattern matching, with data-flow analysis varying by tool and language. They flag known-shape sinks well but struggle with multi-hop chains, business-logic flaws, context-dependent sanitization, and whether a sink is reachable from attacker-controlled input. AI-augmented analysis addresses that reasoning gap: tracing across files, inferring intent, and weighing deployment context.

The key conceptual distinction is that we don’t run a model against the codebase. The codebase is context passed to a model with a specific prompt. The model receives code and applies reasoning about how data flows, where trust boundaries exist, and whether security-relevant patterns are present.

Context scoping is one of the harder parts of this architecture because too much context can overwhelm the model’s reasoning window, while too little can produce analysis gaps that generate false negatives. For a basic AWS Lambda function, the relevant context might be a single file plus its AWS Identity and Access Management (IAM) role. For a complex microservice with shared libraries and layered infrastructure, deciding what to include requires understanding the application’s dependency graph. The model can reason about your deployment topology if you provide it; for example, by passing AWS Cloud Development Kit (AWS CDK) constructs or AWS CloudFormation templates as additional context. Starting narrow and expanding only when the model’s initial assessment is inconclusive generally produces better results than passing everything at once.

Security scanners can produce high volumes of findings each time they run. Many of these are false positives that look like vulnerabilities but aren’t real issues. To separate real problems from noise, the pipeline filters findings through three layers. Each layer applies a different type of evidence. By the end, a smaller number of prioritized issues backed by documented evidence remains.

Plausibility comes first. Run multiple scanners independently against the same codebase: when two or more converge on the same finding, confidence increases. When only one scanner reports an issue and no other tool agrees, the pipeline flags that finding for extra scrutiny.

This approach—called multi-scanner agreement—is a low-cost way to increase confidence. It works with your existing tools and requires no new infrastructure.

The second layer verifies structure. AI-powered scanners describe how they think a vulnerability works: data enters through one function, passes through another, and reaches a point where it could cause harm. However, these descriptions can be wrong.

Before a finding moves forward, the pipeline verifies it against the actual code. It checks to see if the function that the scanner mentioned exists, that the file exists, and if data can flow along the path described by the scanner. If not, the pipeline rejects the finding.

This verification step uses the code’s abstract syntax tree (AST): a structured map of how functions, files, and data flows connect in your codebase. This step helps protect the credibility of your pipeline’s output. If unverified findings reach human reviewers, teams quickly learn to distrust the results, which reduces the security value of the tool.

The final layer adds deployment context. Vulnerabilities exist within the context of an architecture, which might include protective controls. These controls can include AWS WAF rules, network isolation, authentication requirements, and input validation.

This layer reads your infrastructure-as-code (IaC) templates—such as CloudFormation or Terraform files that define your cloud architecture—alongside your application source code. It then checks to see if an attacker can exploit a vulnerability by avoiding the controls that are in place.

Not all controls provide the same level of protection. A web application firewall rule that directly blocks the relevant attack technique provides stronger mitigation than one that addresses a different type of vulnerability. The pipeline treats control effectiveness as a spectrum, not a yes-or-no checkbox.

The result is a short list of findings. Each one has cleared all three layers: it passed the plausibility check, its structure checks out against the code, and its deployment context indicates it can be taken advantage of. The number of findings decreases at each layer, and confidence in the remaining findings increases.

Four principles emerged from building this pipeline:

• Progressive filtering with escalating evidence – Each layer demands a different kind of proof. The first layer checks that the finding is plausible. The second determines if it’s structurally real. The third analyzes if it’s exploitable in context. Using three focused layers works better than using one comprehensive layer, because each layer catches a different type of error.
• Infrastructure context transforms prioritization – Without deployment context, you might assume that a critical-severity finding is more urgent than a medium-severity finding. However, a critical finding behind strong protective controls might be less urgent than a medium finding that’s directly exposed to the internet. To make this assessment, the pipeline needs access to your IaC templates, not only your application source code.
• Independent agreement outperforms single-tool confidence – No single scanner reliably detects the full range of vulnerabilities. No single model produces correct results across all situations. When multiple independent tools agree on a finding, that agreement provides a stronger signal than one tool’s self-reported confidence score. This principle generally holds across tool choices.
• AI-generated claims require verification before human review – Language models can produce outputs that sound plausible but might not be correct. Verifying AI-generated findings against the actual code structure—before a human reviews them—is what separates a useful system from one that erodes trust through confident-sounding errors.

You don’t need to build everything at once. Adopt them in order of immediate return:

• Start with indexing – Build (or use) an AST-based index of your codebase that surfaces entry points, sinks, and authentication configuration. This costs nothing to run and gives you a map of your attack surface before any AI is involved.
• Add multi-scanner triage – Take the output you already get from Semgrep, CodeQL, or equivalent tooling, and run it through the AST gate plus a large language model (LLM) exploitability classification. This delivers substantial noise reduction for a modest investment: you’re filtering output from tools you already run.
• Add hypothesis generation – After triage is stable, add AI-driven discovery of vulnerabilities scanners miss: multi-hop chains, business-logic flaws, and cross-package data flows. This is the step where steering quality matters a great deal; see Configuring your AI vulnerability harness.
• Add infrastructure context – Parse your CDK or CloudFormation, map controls to vulnerability classes, and apply the multipliers. This is where your prioritization becomes more targeted.
• Add live verification – Run proofs of concept (PoCs) against a deployed pre-production target with structured success criteria. This step requires the most operational overhead, so run it last and only for findings that have already passed your confidence threshold.

Treat the pipeline as something you tune over time. Validate it against known true and false positives, adjust the weights, and rerun. Your results will change.

The harness is detection infrastructure. It helps you see which findings are likely real and which deserve attention first. It doesn’t:

• Fix the code – Remediation is a separate problem. The harness produces findings with enough evidence that a human or agent can act on them; the act of remediation requires its own tooling and process.
• Replace human review for final action – Even after three layers of filtering, the output is a prioritized list of likely-exploitable findings, not proven exploits. Priority 0 (P0) findings still warrant review by an engineer before they drive code changes.
• Catch vulnerability classes outside the scanner ensemble’s detection profile – If none of your scanners look for a particular vulnerability category, the harness won’t surface it. AI-augmented hypothesis generation closes some of this gap, but novel or business-logic flaws still depend on what the model can reason about from code alone.
• Verify live infrastructure state by default – Parsing IaC tells you what was defined. It doesn’t tell you whether a WAF rule got disabled last week, or whether a security group was modified after deployment. Live verification is an additional integration, not a property of the layers themselves.

Each of these limits points to where the architecture extends, not where it breaks. The harness is the first floor of a multi-story system, not the whole building.

In this post, we showed how a three-layer pipeline—multi-scanner agreement, structural verification, and deployment context—narrows scanner output to a small, higher-confidence set of findings that your team can act on. The architecture is tool-agnostic; what transfers is where to filter, what counts as evidence, and when to stop trusting the model.

The architecture produces consistent results when the model receives consistent instructions. Our companion post, Configuring your AI vulnerability harness, covers the steering file that encodes this methodology as machine-executable instructions, including the confidence formula, the infrastructure control multipliers, and the verification gates that help prevent hallucinated findings from reaching your engineers. The same model that produces rigorous, evidence-grounded assessments with well-crafted instructions can produce inflated severity ratings and fabricated attack chains without those instructions. Configuration is what turns capability into judgment.

The window between disclosure and exploitation is narrowing. Embedding automated detection into your development workflows helps your team keep pace. We built this pipeline through trial and error and are sharing it so you can skip some of the mistakes we made.

For more information about the services mentioned in this post, see the AWS Lambda, AWS WAF, and AWS CloudFormation documentation.

If you have feedback about this post, leave a comment in the Comments section below.

Google has temporarily stopped accepting product vulnerability reports through its Open Source Software Vulnerability Reward Program (OSS VRP).

The post Google Narrows Open Source Bug Bounty Amid Wave of Invalid Automated Reports appeared first on SecurityWeek.

The announcement comes after Trump hosted top executives of AI companies at the White House last week.

The post Trump Names National Intelligence Director Jay Clayton to Lead a New Federal AI Task Force appeared first on SecurityWeek.