❌

Reading view

Best practices for integrating IAM Roles Anywhere with your PKI

Using temporary credentials is a security best practice to reduce the risk from credentials getting compromised. With AWS IAM Roles Anywhere, now your workloads outside of Amazon Web Services (AWS) will be able to access AWS resources by obtaining temporary security credentials in AWS Identity and Access Management (IAM). IAM Roles Anywhere relies on public key infrastructure (PKI) to establish trust between your AWS account and certificate authorities that issues certificates to your workloads that run outside AWS. PKI involves the generation, distribution, and verification of digital certificates through public key encryption.

In this post, we show you some best practices, considerations, and solutions for integrating your PKI with IAM Roles Anywhere. In this post, we assume that you have a foundational knowledge of IAM and IAM Roles Anywhere. For more information, see the IAM and IAM Roles Anywhere documentation.

Note: As of July 2026, AWS Certificate Manager (ACM) supports exportable public certificates that can be used on workloads outside AWS (containers or on-premises). Additionally, ACM supports the ACME protocol for automated certificate issuance and renewal on external workloads. These capabilities complement IAM Roles Anywhere by providing both identity (using X.509 certificates for IAM Roles Anywhere) and encryption (using ACM public/private certificates) for hybrid environments.

X.509 certificates for workload identity

If you’re running multiple workloads in your infrastructure, you’re probably assigning unique workload identities and specific user permissions. In this environment, your workloads can use PKI tools like Active Directory Certificate Services to centrally manage and monitor the identities and permissions at scale. The same principles of segregating unrelated workloads apply to PKI and IAM Roles Anywhere. If you’re issuing certificates to your workloads today, they can probably be used with IAM Roles Anywhere without any changes.

As shown in Figure 1, the access boundary and permissions of Workload A can be very different from that of Workload B when accessing AWS resources, depending on the organization’s needs. In such scenarios, it’s recommended to establish a unique machine identity for each workload. This can be achieved by issuing a separate X.509 certificate with distinct identity attributes signed by your certificate authority (CA). To set up an effective PKI solution for machine identities, you must configure the appropriate certificate attributes and anchor the trust model around those attributes.

Figure 1: Workloads outside of AWS accessing AWS services

Figure 1: Workloads outside of AWS accessing AWS services

Temporary access credentials

Workloads with distinct CA signed X.509 certificates exchange certificates with IAM, as shown in Figure 2, to get temporary security credentials for the session. The signature validation process used by IAM Roles Anywhere verifies the certificate issuer and the subject before providing the temporary credentials. To enable this verification process, an IAM Roles Anywhere trust model should be set up by either uploading your CA’s digital certificate as a trust anchor or by referencing an existing AWS Private Certificate Authority (AWS Private CA) certificate. In this post, we focus on using a trust anchor.

To help define the appropriate scope for the temporary credentials, it’s recommended to configure distinct IAM roles for each workload to assume. Workloads must use X.509 certificates issued by your CA. These certificates are exchanged with IAM to get temporary security credentials for the session.

Figure 2: Getting temporary credentials from IAM using CA issued X.509 certificate

Figure 2: Getting temporary credentials from IAM using CA issued X.509 certificate

Setting up a trust model for IAM Roles Anywhere is a three-step process, see Extend AWS IAM Roles Anywhere to workloads outside of AWS for more information.

  1. Create a trust anchor.
  2. Set up IAM roles with the IAM Roles Anywhere trust policy that will be assumed by your non-AWS workload.
  3. Configure profiles with IAM roles and attach relevant session policies. This instructs IAM Roles Anywhere to use your selected session policies whenever you request credentials for any of the attached roles.

Trust model

The trust model is centered around setting up the right trust relationship between IAM Roles Anywhere and your CA by configuring a trust anchor. The model connects the role, the IAM Roles Anywhere service principal, and identities encoded in X509 certificates that are issued by a CA.

Certificate attributes

Certificates are a binary record with encoded key/value pairs as certificate attributes. Let’s look at one of the X.509 certificates to understand different attributes of certificate that are important for IAM Roles Anywhere PKI validation.

The following is an example PEM encoded certificate file that can be decoded with tools like openssl.

-----BEGIN CERTIFICATE-----
MIIKOzCCCSOgAwIBAgIQDlnyZvBeKjgHm1jJ++CbgDANBgkqhkiG9w0BAQsFADBE
MQswC ...........
-----END CERTIFICATE----


Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            e0:5d:b7:72:56:fa:75:e3
    Signature Algorithm: ecdsa-with-SHA512
        Issuer: C=US, ST=CA, L=San Francisco, O=MyCompany Inc., OU=AC, CN=MyCompany CA
        Validity
            Not Before: Sep 15 00:09:17 2023 GMT
            Not After : Sep 18 00:09:17 2023 GMT
        Subject: C=US, O=MyCompany, OU=eCommerce, CN=WorkloadA
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (384 bit)
                pub: 
                    04:3b:74:e1:68:06:fa:b0:3d:fe:f7:5f:74:80:05:
        X509v3 extensions:
            X509v3 Subject Alternative Name: 
                DNS:ecommerce.mycompany.com, DNS:*ecommerce.mycompany.com, URI:spiffe://mycompany.com/workload/ecommerce
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Key Usage: critical
                Digital Signature
    Signature Algorithm: ecdsa-with-SHA512
         30:64:02:30:19:a7:a0:33:49:ce:e1:cb:76:6e:c0:f3:c5:66:

IAM Roles Anywhere extracts values from the subject Common Name (CN), issuer, and Subject Alternative Name (SAN) fields of the authenticating certificate and makes them available for policy evaluation through the sts:sourceIdentity and principal tags. When configuring your PKI, set short validity periods on the certificates issued to workloads. Short-lived certificates limit the window of exposure if a certificate is compromised and reduce the need for active revocation.

"aws:PrincipalTag/x509Subject/CN": "WorkloadA" 
"aws:PrincipalTag/x509Issuer/CN": "MyCompany CA" 
"aws:PrincipalTag/x509SAN/DNS": "ecommerce.mycompany.com" 
"aws:PrincipalTag/x509SAN/URI": "spiffe://mycompany.com/workload/ecommerce" "aws:PrincipalTag/x509SAN/Name/CN": "WorkloadA"

Note: An empty Subject CN in a certificate isn’t supported by IAM Roles Anywhere.

Trust policy and effective permissions of credentials

IAM Roles Anywhere clients fetch temporary credentials for IAM Roles using the API method CreateSession. To successfully authorize this API call, the IAM role trust policy should trust the IAM Roles Anywhere service principal as shown in the following example.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "rolesanywhere.amazonaws.com"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:TagSession",
                "sts:SetSourceIdentity"
            ],
            "Condition": {
                "StringEquals": {
                    "aws:PrincipalTag/x509Issuer/CN": "MyCompany CA",
                    "aws:PrincipalTag/x509Subject/CN": "WorkloadA",
                    "aws:PrincipalTag/x509SAN/Name/CN": "WorkloadA"
                },
                "ArnEquals": {
                    "aws:SourceArn": [
                        "arn:aws:rolesanywhere:region:account:trust-anchor/TA_ID"
                    ]
                }
            }
        }
    ]
}

As a security best practice, consider using additional condition statements to evaluate the session PrincipalTag attributes extracted from the certificate to further restrict authorization of temporary credentials. If the workloads have ephemeral certificates with a dynamic subject CN, then the issuer or SAN principal tags can be used to define condition statements in the trust policy. The trust anchor shown in the preceding trust policy helps establish the trust between IAM Roles Anywhere and the CA in your PKI.

Plan for trust anchor placement

Depending on the CA platform, you can configure one or more intermediate certificates in your PKI certificate hierarchy, as shown in Figure 3. The certificate level the trust anchor is placed at affects which end-entity certificates IAM Roles Anywhere accepts, and therefore which workloads are authorized to assume a role. To create a trust anchor, export the certificate bundle for the issuing CA certificate from your external CA platform in PEM-encoded format and upload it to the trust anchor console.

Figure 3: Certificate hierarchy in an external PKI

Figure 3: Certificate hierarchy in an external PKI

See the Solution deployment section to onboard a sample solution and verify your certificate bundle.

Fine-grained access control is vital for security because it allows organizations to precisely define and restrict permissions, minimizing the risk of unauthorized access and data breaches. The effective permissions of the temporary security credentials are an intersection of resource-based, identity-based, and session policies as shown in Figure 4. By setting up the trust anchor and profile configuration in IAM Roles Anywhere with session policies, you can limit the permissions for each session.

Figure 4: Effective permissions of session credentials

Figure 4: Effective permissions of session credentials

Credential helper

To obtain temporary security credentials from IAM Roles Anywhere, use the credential helper tool. This tool is compatible with the credential_process feature available across the AWS SDKs. The credential helper manages the process of obtaining temporary security credentials using IAM Roles Anywhere and returns them in JSON format. The private key of the workload and its CA signed certificate are the two critical parameters provided to the credential helper tool. To use this tool, you need both an end-entity certificate and a trust anchor created in IAM Roles Anywhere. As of December 2024, the credential helper (version 1.4.0 and later) supports Trusted Platform Module (TPM) 2.0.

The following is an example of how to obtain temporary security credentials:

$ ./aws_signing_helper credential-process \
--certificate /path/to/certificate \
--private-key /path/to/private-key \
--trust-anchor-arn arn:aws:rolesanywhere:region:account:trust-anchor/TA_ID \
--profile-arn arn:aws:rolesanywhere:region:account:profile/PROFILE_ID \
--role-arn arn:aws:iam::account:role/role-name-with-path

When credential-process is used with AWS SDKs, the temporary credentials are stored in memory and refreshed automatically. However, the AWS Command Line Interface (AWS CLI) invokes credential-process on every command. Each invocation creates a new IAM Roles Anywhere session, which adds latency to every CLI call. To avoid this delay when using the AWS CLI, we recommend using the update command, which writes refreshed credentials to a file that the CLI reads on each invocation without calling credential-process again.

When you use update, credentials are written to a credential file on disk and any user or process who can read the credential file might be able to read and use those AWS credentials, so it’s recommended to use serve unless your business has a specific use case that needs to read AWS credentials from a file. The serve command can be used to vend temporary credentials through an endpoint running on the local host using the same URIs and request headers as Instance Metadata Service Version 2 (IMDSv2). When using serve, no AWS configuration files are needed because the AWS SDKs can discover the server endpoint.

Just-in-time compared to short-lived certificates as workload identities

To help mitigate the risk around the exposure of your business-critical resources to compromised certificate user sessions, it’s recommended to use certificates with just-in-time credentials or short-lived access credentials. Just-in-time certificates are issued by a CA for immediate use for authentication on a system or application. The CA uses the roles and policies defined in IAM to determine when, and under what conditions, a workload can establish a session. With just-in-time access, admins don’t have to worry about revoking access if the users’ access roles change. Because the access credentials are recreated for each access request, any role changes can be applied in real time. Just-in-time access requires support from your CA, so verify compatibility with your CA before implementing it.

It’s recommended to tailor the validity periods for short-lived certificates to align with business requirements, workload lifecycle, and your organization’s risk tolerance. AWS Private CA offers a short-lived CA mode specifically designed for certificates with validity periods under 7 days. This mode is priced at $50 per month (compared to $400 per month for general-purpose CA mode), making it cost-effective for high-volume, ephemeral workload identities such as containers and serverless functions. For more information, see AWS Private Certificate Authority and How to use AWS Private CA short-lived certificate mode. See Zero trust on AWS for information about how short-lived credentials support a zero trust architecture.

Best practices for private key storage

Safeguarding private keys is of paramount importance in any environment. A private key is created by the certificate owner when requesting certificates with a Certificate Signing Request (CSR). The CA providing your certificate doesn’t create or have your private key. In fact, it’s recommended that no one outside your administrators should ever be given access to this material. Guardrails should be implemented to make sure the keys aren’t stored in an online repository.

Using the credential helper with IAM Roles Anywhere, you can integrate with PKCS #11-compatible modules. For implementation details, see How to implement cryptographic modules to secure private keys used with IAM Roles Anywhere. With the credential helper (version 1.1.0 and later), IAM Roles Anywhere supports PKCS #11 integration, so you can use private keys stored in hardware security modules (HSMs) or software-based secure stores and avoid storing them to disk. Additionally, as of December 2024, the credential helper supports Trusted Platform Module (TPM) 2.0, enabling private keys to be generated inside the TPM and marked as non-exportable. With TPM-backed keys, the private keys can’t be extracted even if the workload host is compromised. This is the strongest private key protection available for on-premises and hybrid workloads using IAM Roles Anywhere.

If your organization doesn’t implement PKCS #11–compatible modules, IAM Roles Anywhere supports credential helper version 1.0.5 for X.509 certificates and private keys stored in macOS and Windows certificate stores. This delegates the signing operations to keys stored within those OS-specific certificate stores. In Windows, both CryptoAPI and Cryptography API: Next Generation (CNG) are supported; in macOS, Keychain is supported. In these environments, the private key will never be directly readable, and the signing operation would be delegated to the secure store instead, which will improve security posture of your environment. If you have multiple certificates, then use the --cert-selector flag of the credential helper tool to search for a specific certificate (and associated private key) through the certificate subject, issuer, and serial number. The corresponding keys are x509Subject, x509Issuer, and x509Serial, respectively. These keys can be specified either through a JSON file format or through the command line. The following is an example of both approaches:

If you want to use a JSON file, it should be similar to this:

[
  {
    "Key": "x509Subject",
    "Value": "CN=Subject"
  },
  {
    "Key": "x509Issuer",
    "Value": "CN=Issuer"
  },
  {
    "Key": "x509Serial",
    "Value": "15D19632234BF759A32802C0DA88F9E8AFC8702D"
  }
]

If the preceding example is placed in a file called selector.json, it can be specified with the --cert-selector flag through file://path/to/selector.json.

The same certificate selector argument can be specified through the command line as shown in the following:

--cert-selector Key=x509Subject, Value=CN=Subject Key=x509Issuer, Value=CN=Issuer Key=x509Serial, Value=15D19632

Monitoring trust anchor certificates

Monitoring certificate expiry in PKI is crucial for maintaining secure communication, because expired certificates can lead to authentication failures, leaving systems vulnerable to unauthorized access. Compliance and regulatory requirements often mandate timely renewal of certificates, making monitoring certificate expiry essential to avoid legal penalties and maintain organizational integrity.

Using IAM Roles Anywhere, you can receive notifications for CA certificate expiry and end-entity certificate expiry. IAM Roles Anywhere can send metrics, events, and notifications through multiple notification channels as shown in the following table:

Event Channel Threshold Enabled
CA certificate expiry CloudWatch, Amazon EventBridge, and AWS Health 45 days before expiry TRUE
End-entity certificate expiry EventBridge and AWS Health 45 days before expiry TRUE

The preceding notifications thresholds are default settings, which you can customize using the steps in Customize notification settings in IAM Roles Anywhere.

Best practices for trust anchor certificate rotation

Consider rotating your certificates using blue/green deployments to seamlessly roll out new certificates without affecting service availability. Use automation to integrate certificate rotation testing in your blue/green deployment pipeline. See the sample AWS CloudFormation template in the Solution deployment section to build your automation pipeline.

As shown in Figure 5, two trust anchors have been created; one for each environment. Using this approach, you can test the certificate rotation process in a controlled staging environment before applying it to the production environment, thus reducing the risk of certificate-related outages.

 Figure 5: Trust anchor certificate rotation in blue/green deployment

Figure 5: Trust anchor certificate rotation in blue/green deployment

Solution deployment

The sample solution template is available in the IAM Roles Anywhere — External PKI Integration repository on GitHub. It contains the following main steps:

  1. Creates a distinct IAM role with policies scoped to the resources that workload requires.
  2. Assigns a trust relationship to a IAM Roles Anywhere principal.
  3. Creates a trust anchor using a certificate bundle from an external CA.
  4. Creates a profile with the required IAM Roles.

Parameters for this deployment are shown in the following table:

Parameter Required Default Description
TrustAnchorCertificateBody Yes — PEM-encoded X.509 certificate of your CA (includes BEGIN and END markers)
WorkloadSubjectCN No WorkloadA Subject CN from the workload certificate, used in trust policy conditions
WorkloadIssuerCN No MyCompany CA Issuer CN from the CA certificate, used in trust policy conditions
WorkloadSANCN No WorkloadA SAN CN from the workload certificate, provides additional identity assertion
SessionDurationSeconds No 3600 Maximum session duration in seconds (900–43200)
NotificationEmail No (empty) Email address for certificate expiry notifications. When provided, creates an Amazon Simple Notification Service (Amazon SNS) topic and an EventBridge rule

Important: The SessionDurationSeconds value must not exceed the IAM role’s MaxSessionDuration setting, which defaults to 3600 seconds (1 hour). If you request a longer session than the role allows, the CreateSession API returns an error. The companion template keeps both values in sync by setting the role’s MaxSessionDuration to the same parameter value. If you configure the role separately, update its MaxSessionDuration to match your desired session length. For more information, see IAM Roles Anywhere CreateSession API.

If the workload’s subject CN (WorkloadSubjectCN) doesn’t match the x509Subject CN of the authenticating certificate, the CreateSession call is denied. The same conditional verification applies to WorkloadIssuerCN and WorkloadSANCN.

The template enforces three certificate attribute conditions in the IAM trust policy: issuer CN, subject CN, and SAN CN must all match the values configured as parameters. Additionally, the trust policy scopes role assumption to the specific trust anchor Amazon Resource Name (ARN), preventing other trust anchors in the same account from assuming that role.

Conclusion

In this post, we discussed how AWS IAM Roles Anywhere, empowered by X.509 certificates, introduces a dynamic and secure way to manage workload identities and issue temporary access credentials. By understanding the trust model, using certificate attributes, and implementing fine-grained authorization, organizations can gain granular control over the effective permissions of these credentials.

The choice between just-in-time and short-lived certificates as workload identities depends on your specific requirements.

Adhering to best practices for private key storage is essential to maintain the integrity of your security infrastructure. Regularly rotating trust anchor certificates helps ensure the long-term trustworthiness of your system.

As you embark on your journey to implement IAM Roles Anywhere and integrate it with your PKI, remember that a well-architected approach to certificate management is a cornerstone of robust and resilient cloud security.

For additional deployment guidance, see Planning for your IAM Roles Anywhere deployment, which covers end-to-end deployment planning including network architecture, monitoring, and operational considerations.

If you have questions, you can start a new thread on AWS re:Post or reach out to AWS Support.


Abhishek Agawane

Abhishek Avinash Agawane

Abhishek is a Security Consultant at Amazon Web Services with more than 8 years of industry experience. He helps organizations architect resilient, secure, and efficient cloud environments, guiding them through complex challenges and large-scale infrastructure transformations. He has helped numerous organizations enhance their cloud operations through targeted optimizations, robust architectures and best-practice implementations.

Manoj Gupta

Manoj Gupta

Manoj is a Senior Solutions Architect at AWS, based in San Francisco. With over 4 years of experience at AWS, he works closely with customers to build optimized AI/ML powered solutions and cloud infrastructure. His primary focus areas are data, AI/ML, and security, helping organizations modernize their technology stacks. Outside of work, he enjoys outdoor activities and traveling with family.

  •  

GuardDuty Extended Threat Detection has added five new indicators to surface attack sequences

Amazon GuardDuty Extended Threat Detection correlates weak GuardDuty signals into a single, critical-severity attack sequence finding when their composition crosses a threshold of credibility. On May 20, 2026, the service started attaching five new indicator types to those sequences: MALICIOUS_PACKAGE, MISCONFIGURATION, REACHABILITY, SENSITIVE_DATA, and VULNERABILITY. For more information, see Document history for Amazon GuardDuty and Attack sequence finding details.

These indicators come from Amazon Inspector, Amazon Virtual Private Cloud (Amazon VPC) Reachability Analyzer, Amazon Macie, and posture data surfaced through AWS Security Hub. You might already run these services for other reasons, and now Extended Threat Detection folds their signals into attack sequence findings. This post is an operational guide for teams already running the prerequisite services and discusses what the new indicators look like in a finding, what the post-launch data shows, and how to fold them into your incident-response workflow. These indicators help Extended Threat Detection prioritize the highest-priority signals in your account, which provides higher confidence findings on the workload surfaces as part of an active defense strategy monitoring runtime activity on Amazon Elastic Compute Cloud (Amazon EC2), Amazon Elastic Container Service (Amazon ECS), and Amazon Elastic Kubernetes Service (Amazon EKS).

New indicators overview

In GuardDuty, a signal is any event it evaluates when looking for suspicious activity. A weak signal is one that, on its own, doesn’t clearly point to a threat which the five new indicators fall into. A single Common Vulnerabilities and Exposures (CVE) finding on a host, for instance, tells you a vulnerability exists not that an attacker is exploiting it. These findings earn their value when GuardDuty correlates them with the suspicious-activity signals it already tracks, as shown in the following table:

Indicator Source What it tells you
VULNERABILITY Amazon Inspector CVE list with Common Vulnerability Scoring System (CVSS) severity and Exploit Prediction Scoring System (EPSS) exploit-likelihood scores.
MALICIOUS_PACKAGE Amazon Inspector Amazon Inspector MAL-* package indicators. For more on how Amazon Inspector scores vulnerabilities, see Amazon Inspector Security Research.
REACHABILITY Amazon VPC Reachability Analyzer Whether the host or cluster is reachable from the internet
MISCONFIGURATION AWS Security Hub controls (posture management) The resource has configuration that doesn’t align with AWS or industry best practices. For example, IMDSv1 still enabled, overly permissive admin policies, or publicly accessible Amazon Simple Storage Service (Amazon S3) buckets.
SENSITIVE_DATA Amazon Macie Potential sensitive data has been discovered in the resource.

The five values appear on the Indicators tab of the attack-sequence finding in the AWS Management Console for GuardDuty and in the GetFindings JSON file.

Figure 1: Indicators section of an attack sequence finding

Figure 1: Indicators section of an attack sequence finding

A note on EPSS, because the VULNERABILITY indicator carries it. According to the FIRST EPSS overview, the score is the probability that a published CVE will be exploited in the wild in the next 30 days ranging from 0 to 1. EPSS doesn’t replace CVSS, which describes the inherent severity of a vulnerability, whereas EPSS describes how likely it is to be exploited soon making it the more useful number for incident response.

Prerequisites

To populate the indicators, you need to have the following AWS services enabled:

Strengthening weak runtime signals

Runtime signals such as a process invocation, a pod exec, or a syscall pattern can be noisy sometimes. A single runtime signal that doesn’t match a known offensive-tool path isn’t significant on its own, and without resource-state context it typically stays below the threshold for a Critical attack sequence.

The new indicators supply that missing context. When Extended Threat Detection sees a weak runtime signal on a host that also carries a combination of high-priority unpatched CVEs, a known malicious package, an IMDS misconfiguration, and internet reachability, it promotes the collection to a single CRITICAL attack sequence with high confidence. Customer feedback on individual GuardDuty runtime findings has consistently identified noise as a major pain point and adding this resource-state context is what turns those weak signals into actionable alerts.

In practice, this surfaces as a substantial increase in net-new attack-sequence findings on the runtime-process surface to roughly one in four prioritized runtime-process attack sequences, with little change on the network-indicator of compromise (IOC) surface. Within the first week of launch, this pattern showed up across many customer accounts where attack sequences that would previously have surfaced as a collection of weaker, unconnected findings now surface as a single CRITICAL finding.

Example: A net-new attack sequence on Amazon EKS

Now that you understand how Extended Threat Detection has improved through additional indicators, let’s walk through an example of how these indicators add value. In this example we analyze an actual scenario that was observed on an Amazon EKS cluster. The observed activity occurred over an 8-day period with eleven different GuardDuty findings.

Figure 2: Activity summary for the EKS cluster before Extended Threat Detection correlated the signals

Figure 2: Activity summary for the EKS cluster before Extended Threat Detection correlated the signals

What GuardDuty saw on the activity surface over 8 days as shown in Figure 2 was 55 EKS_AUDIT_LOGS signals across pod execs, role assumptions, and AssumeRoleWithWebIdentity calls and 18 finding-level signals all originating from one external IP address of 198.51.100.69. This IP address authenticated as four distinct AWS Identity and Access Management (IAM) identities to the same EKS cluster. Runtime activity on the worker nodes was generic because there was no process that matched a known offensive-tool path and no execution-profile flag was fired. Under the pre-launch correlation, this sequence would have needed the EKS_MALICIOUS_PROCESS rule to fire, which requires either a malicious exec path or an unusual-suspicious-tool profile. Neither was present in this activity, so no correlated attack-sequence finding was raised.

What the new GuardDuty Extended Threat Detection indicators added to the EKS example:

  • MALICIOUS_PACKAGE on the cluster: Amazon Inspector identifier MAL-2022-4933.
  • VULNERABILITY (full Amazon Inspector output) on the cluster: 42 CVEs total, 33 rated CRITICAL by CVSS.
    • On the cluster: 12 CVEs including CVE-2022-1471 (SnakeYAML deserialization, EPSS 0.94), CVE-2024-4577 (PHP-CGI argument injection, EPSS 0.97), CVE-2024-34351 (Next.js SSRF), CVE-2025-29927 (Next.js authentication bypass), CVE-2023-44487 (HTTP/2 Rapid Reset), and CVE-2022-37434 (zlib heap overflow, EPSS 0.92).
    • Across the five worker nodes: 379 CVEs total, with 68 of them marked CRITICAL by Amazon Inspector.
  • REACHABILITY: Cluster API endpoint internet-reachable and three of five worker nodes also internet-reachable.
  • MISCONFIGURATION (EC2 IMDS): All five worker nodes still expose IMDSv1.

When these additional indicators are taken into consideration, this surfaces as a single AttackSequence:EKS/CompromisedCluster with a CRITICAL severity, with all four populated indicators on the Indicators tab and a MITRE ATT&CK® mapping to Initial Access (External Remote Services), Discovery, and Persistence.

Figure 3: New vulnerability indicators within an attack sequence

Figure 3: New vulnerability indicators within an attack sequence

The finding lists more than 100 Amazon EC2 CVEs at the worker-node level, one CRITICAL CVSS, plus a long HIGH tail, but most are LOW vulnerability priority because their EPSS scores are low. Rather than sifting through more than 100 findings to decide which to address first, GuardDuty highlights the high-priority ones so you can focus your response efforts where they have the most impact.

Example: Cryptomining and a vulnerable host

In this second example we walk through an EC2 instance group where GuardDuty already had a strong network-based finding for cryptomining and the new indicators provided additional remediation context that made incident response faster.

Figure 4: Cryptomining attack on EC2 instances

Figure 4: Cryptomining attack on EC2 instances

A set of EC2 instances was communicating with a known cryptomining destination, which was flagged by threat intelligence as mining/misc. The network IOC evidence was strong on its own.

  • Observations added by the new indicators: VULNERABILITY listed CVE-2007-4559 Python tarfile with EPSS around 0.9 on two of the three involved instances.
  • Result: The AttackSequence:EC2/CompromisedInstanceGroup finding is now actionable beyond block the destination IP; it tells you which patch will remediate the vulnerability the cryptominer likely came in through so you can address the underlying entry point rather than only removing the outbound connection.

This attack sequence would have been generated without the vulnerability context because GuardDuty had strong evidence of cryptomining of the EC2 instance group, however, triaging is faster with more context.

Indicators of compromise observed across the first-week launch sample

The following CVEs listed, along with their EPSS scores at the time, were observed on hosts across the customer accounts that received the new Extended Threat Detection indicators during the first week after launch at the moment each indicator finding was generated. EPSS scores are point-in-time, so a CVE’s listed priority on any future finding will reflect the current EPSS rather than the values shown here.

  • CVE-2024-4577 – PHP-CGI argument injection on Windows. EPSS 0.97+.
  • CVE-2022-1471 – SnakeYAML deserialization. EPSS 0.94. Critical CVSS.
  • CVE-2022-37434 – zlib heap buffer overflow. EPSS 0.92.
  • CVE-2007-4559 – Python tarfile path-traversal. EPSS still 0.9+ in 2026.
  • CVE-2025-29927 – Next.js authentication bypass. Frequent on Node web tiers.
  • CVE-2024-34351 – Next.js SSRF.
  • CVE-2023-44487 – HTTP/2 Rapid Reset.

Runtime monitoring surfaced weak signals tied to malware and tooling artifacts recovered from the compromised hosts:

  • HackTool family Linux/frp (Fast Reverse Proxy variants were used as command-and-control tunnels), often disguised as kuboard-agent and kuboard-agent-server binaries on Amazon EKS worker nodes. Container-snapshot path matches were also observed under /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/.../usr/bin/. The frp binaries are present inside the container image and on the node, which means rebuilding the node from an unpatched AMI won’t be sufficient remediation.
  • Burp Collaborator out-of-band callback hostnames were found under *[.]oastify[.]com, matching pattern <35-char base32>[.]oastify[.]com. These co-occurred with frp on the same EKS sequence, probably a second tool or actor on shared infrastructure.
  • The Amazon Inspector malicious-package indicator MAL-2022-4933 was found in both Amazon EC2 and Amazon EKS contexts.

Identity and access patterns correlated with the tooling signals above:

  • A single external source IP authenticated as multiple distinct human IAM identities across separate Amazon EKS, Amazon ECS, and Amazon EC2 surfaces over a short window. This was the most common identity signature on the new Amazon EKS-process net-new class.

Responding to attack sequences with these indicators

The new indicators don’t change incident-response runbooks for attack sequences that have already occurred. They matter for new findings, which are concentrated on enhancing signals related to runtime monitoring. For those:

  1. Treat the sequence with urgency. MALICIOUS_PACKAGE and a high-priority VULNERABILITY mean the host was, and still is, at risk of being compromised. Contain the exposures and remediate the vulnerabilities. Rotate every IAM credential, instance profile, or IAM Roles for Service Accounts (IRSA)-bound role that could have been exposed.
  2. Patch on EPSS-weighted priority, not alone on raw CVSS. The finding’s high-priority VULNERABILITY list is already filtered to high-EPSS CVEs.
  3. Isolate public resources indicated by REACHABILITY indicators. An internet-reachable host with exploitable vulnerabilities should be isolated for remediation.

For more support, see the remediation steps in Remediating detected GuardDuty security findings and GuardDuty Investigation, an AI-powered security analysis of your GuardDuty findings and accounts.

Conclusion

GuardDuty Extended Threat Detection can deliver more high-confidence attack sequence findings on the runtime workload surfaces where your security teams often need help most: Amazon EC2, Amazon ECS, and Amazon EKS. It does this by combining GuardDuty activity signals with resource-state context from other AWS services. Customer feedback on individual GuardDuty runtime findings has consistently identified noise as a major pain point, and correlating those weak signals with vulnerability, package-risk, reachability, misconfiguration, and sensitive-data indicators is how Extended Threat Detection turns the noise into actionable CRITICAL findings. If you’ve already enabled AWS Security Hub, Amazon Inspector, Amazon VPC Reachability Analyzer, Macie, and GuardDuty Runtime Monitoring, the new indicators automatically start showing up on your attack-sequence findings. If you aren’t using these services, we encourage you to explore enabling them to access the new GuardDuty Extended Threat Detection indicators and the value that these individual services deliver.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on AWS re:Post or contact AWS Support.


Aman Khare

Aman Khare

Aman is a senior software developer at AWS GuardDuty. He works on Extended Threat Detection, helping customers pinpoint the malicious attack chains detected in AWS infrastructure. Previously, he led performance and resiliency work on Amazon OpenSearch. Including 10x cluster-manager memory reductions and enabling 1K-node clusters. Outside work, Aman enjoys traveling, adventure activities, reading and playing board games.

Kyle Koeller

Kyle Koeller

Kyle is a security engineer in the Amazon GuardDuty team with a focus on threat detection. He’s passionate about cloud threat detection and offensive security and holds CompTIA Security+, PenTest+, CompTIA Network Vulnerability Assessment Professional, and SecurityX certifications. When not working, Kyle enjoys spending his time in the gym and exploring New York City.

  •  

Configuring your AI vulnerability harness, Part 2: The steering file

This post shows you how to configure an AI model to perform structured, evidence-based vulnerability triage with the consistency of a seasoned security analyst. You’ll learn the design decisions behind five configuration sections that enforce structural verification, evidence-based scoring, and infrastructure-aware prioritization across every analysis session. Our companion post—Building your AI vulnerability harness—covered the architecture; this post covers the configuration.

The distinction matters because the same frontier model that produces a rigorous, evidence-grounded security assessment with one set of instructions will produce hallucinated attack chains and inflated severity ratings with another. The model’s capability is fixed. What you configure is its judgment.

We learned this the hard way while building the harness. Our early iterations produced findings that sounded authoritative but crumbled under scrutiny. The model would report a SQL injection in a function that didn’t exist, claim HIGH confidence based on nothing structural, or ignore an AWS WAF in blocking mode directly in the request path. The output wouldn’t have earned engineering trust in that state.

The fix wasn’t a better model. It was better steering.

What a steering file is

A steering file is a set of instructions that an AI coding assistant loads at the start of every session. Different tools use different conventions—Kiro reads markdown files from a .kiro/steering/ directory, Claude Code uses CLAUDE.md, Cursor uses .cursorrules, and GitHub Copilot uses .github/copilot-instructions.md—but the function is the same: persistent instructions that shape how the model approaches work in your repository.

Applied to security analysis, a steering file encodes your team’s triage methodology as machine-executable instructions. Instead of relying on individual engineers to apply that methodology consistently, you encode it once and the model applies it uniformly across analyses.

The steering file we’re releasing encodes the architectural principles from the harness post as operational instructions. Where the harness post says apply infrastructure-aware assessment, the steering file specifies exactly how: which controls map to which multipliers, how multipliers stack, and what floor prevents over-confidence in any single control.

Why configuration outperforms prompting

A single-turn prompt (find vulnerabilities in this code) produces whatever the model’s default behavior generates. That default is optimized for helpfulness, not rigor. The model will find things to report because you asked it to find things.

A steering file changes the default. It establishes:

  • What counts as evidence: Without explicit instructions, a model treats its own reasoning as sufficient evidence. The steering file requires structural verification—confirmed dataflow, confirmed function existence, and confirmed call path—before a finding is reported.
  • What confidence means: On its own, a model assigns confidence based on how plausible its narrative sounds to itself. The steering file replaces self-reported confidence with a formula computed from binary structural signals. Either taint analysis confirms the flow or it doesn’t. Either a call graph connects entry point to sink or it doesn’t.
  • How infrastructure affects priority: Left to its defaults, a model assesses vulnerabilities in isolation from their deployment context. The steering file requires parsing infrastructure as code (IaC) and applying control-specific multipliers before assigning priority.

The goal is reproducibility: the same codebase should produce the same assessed findings regardless of who runs the analysis or when. Consistency is the point.

The five sections that matter

The full steering file is available in the companion repository. In this section, we explain the design decisions behind its five critical sections.

Structural verification over LLM trust

Never trust an LLM's self-reported confidence about a vulnerability.
Verify claims against the code structure:
1. Do referenced files exist?
2. Do referenced functions exist?
3. Does dataflow confirm the claim?
4. Is there a call path?

If a finding fails all structural checks, reject it regardless of how
convincing the narrative sounds.

This is the single most important instruction in the file. Without it, the model generates plausible-sounding attack chains that reference functions that were renamed three commits ago, files that exist in a different package, or dataflows that pass through sanitization that the model failed to notice.

We found that approximately 30% of unsteered model findings referenced code structures that didn’t exist in the target repository. Not wrong interpretations—entirely fabricated paths. The instruction to verify before reporting produced zero fabricated paths in our testing.

The key insight: hallucination in security analysis isn’t a minor annoyance. One hallucinated finding that reaches a developer destroys the can quickly erode the credibility of your entire pipeline. Engineers who encounter a fabricated vulnerability are less likely to keep reading the reports.

Evidence-based confidence scoring

The following formula replaces the model’s intuitive confidence with a score computed from observable signals. Each factor is binary: confirmed or not confirmed.

confidence = 0.30 * taint_confirms_flow
           + 0.25 * no_sanitization_found
           + 0.20 * entry_point_is_public
           + 0.15 * call_graph_confirms_path
           + 0.10 * sink_type_matches_category

We weighted taint confirmation highest (0.30) because a confirmed dataflow path—evidence that user-controlled data reaches a security-sensitive operation—is the single strongest predictor that a finding is real. In our validation against known-vulnerable applications, findings with confirmed taint and no sanitization were exploitable 89% of the time. Findings where the model reported HIGH confidence without structural confirmation were exploitable 34% of the time.

The formula is a starting point, not a fixed standard. Your weights should reflect your own validation data. Run the formula against a labeled dataset of known true and false positives, then adjust until precision matches your team’s tolerance for noise.

Infrastructure-aware assessment

This section addresses a common category of over-prioritization: reporting a vulnerability at full severity when compensating controls are already in place.

score_after_controls = max(0.15, score * block_1 * block_2 * ... * block_n)

Controls are mapped to specific vulnerability classes. An AWS WAF with SQL injection rules in blocking mode is a STRONG control (0.25x multiplier) against SQL injection. It is irrelevant to deserialization attacks. The steering file specifies this mapping explicitly rather than leaving the model to guess.

The multiplicative stacking of attack-blocking controls with a floor of 0.15 encodes defense-in-depth thinking. Multiple controls reduce risk more than any single control, but no combination of controls reduces risk to zero. The floor prevents the model from concluding that a vulnerability is completely mitigated and can be ignored.

A subtlety we learned through validation: access controls (authentication and authorization) and attack-blocking controls (AWS WAF rules and input validation) serve different functions. Authentication limits who can attempt an exploit. It doesn’t limit what happens when an authenticated user attempts one. A command injection behind AWS Identity and Access Management (IAM) authentication is still a command injection; the attacker pool is smaller, but the impact when exploited is unchanged. The steering file accounts for this by mapping control types to specific score components rather than applying a blanket multiplier.

Threat intelligence integration

The steering file incorporates signals from the CISA Known Exploited Vulnerabilities (KEV) catalog and the Exploit Prediction Scoring System (EPSS) to boost priority when threat intelligence indicates active risk.

The following table lists the primary signals the steering file recognizes and the boost that each signal adds to a finding’s score. Each signal answers a different question about real-world exploitation risk:

  • CISA KEV – Indicates that attackers have exploited the vulnerability in the wild and notes whether it is tied to ransomware campaigns.
  • EPSS – Estimates the probability that attackers will exploit a vulnerability in the next 30 days.
  • Public proof of concept (PoC) – Indicates that working exploit code is already published, so an attacker has little work left to do.
Signal Boost
Actively exploited (CISA KEV) +0.30
Ransomware-associated (CISA KEV) +0.15
EPSS >= 0.5 +0.20
Public PoC exists +0.10

There’s a cap at +0.50 total boost to prevent threat intelligence from dominating the score. A vulnerability isn’t more technically exploitable because attackers are targeting it, but it is more urgent because the window between theoretically exploitable and actively exploited can be very short.

We separate urgency from severity deliberately. A medium-severity dependency vulnerability under active exploitation by ransomware groups demands a faster response than a critical-severity code-level finding that requires complex preconditions and has no known tooling.

Priority classification

Scoring produces a number and priority classification turns that number into a decision, which is what a developer on the receiving end of a finding needs.

The following table shows the four action tiers that the steering file defines. The score and evidence thresholds determine which tier a finding falls into, and each tier prescribes a specific response. The steering file compares these thresholds against the final score: the base score after it applies attack-blocking multipliers and any threat-intelligence boost. That final score is not the confidence value from the formula earlier in this post, and it is not the severity that the scanner reported.

Priority Criteria Action
P0 Score >= 0.8, no effective attack-blocking mitigation Fix immediately
P1 Score >= 0.6 OR active threat intel Fix this sprint
P2 Score >= 0.4, partial evidence Investigate when capacity
P3 Score < 0.4 or effectively mitigated Accept risk or backlog

The most controversial instruction in the file: Do not file P3 findings as security issues. We include it because filing low-confidence or fully mitigated findings as tickets trains teams to ignore security tooling. Each false positive or irrelevant finding filed as a bug reduces engagement with the system’s output. A pipeline that produces ten confirmed findings gets more engineering attention than one that produces two hundred maybes.

Validation results

We tested the steering file against a purpose-built vulnerable application containing ten known vulnerabilities: eight exploitable and two mitigated by infrastructure controls. The application includes an AWS WAF with SQL injection rules in blocking mode, Amazon Virtual Private Cloud (Amazon VPC)-isolated AWS Lambda functions, and IAM authentication on each endpoint.

  • Without the steering file, the model found all ten vulnerabilities and correctly identified both mitigated findings as lower priority. However, it assigned severity based on intuition (this is a command injection, so it’s CRITICAL) rather than evidence, and it produced no structural verification of its claims.
  • With the steering file, the model found nine of ten vulnerabilities (not flagging a hardcoded credential that had been intentionally embedded in unit-testing code and was defined but never called), correctly downgraded the two mitigated findings to P3 with documented rationale, showed its confidence calculation for each finding, and verified each claimed dataflow against the actual code.

The scoring calibration required one iteration. Our initial version applied authentication as a blanket multiplier that suppressed all findings behind IAM to P2 regardless of impact. We corrected this by separating access-limiting controls (which reduce the reachability score component) from attack-blocking controls (which reduce the full confidence score). The current version correctly classifies a command injection behind IAM authentication as P0: the auth limits who can reach it, but the impact when exploited is remote code execution regardless.

How to use the steering file

The three options that follow cover an always-on Kiro steering file, the same content as an on-demand Kiro skill, and how to carry the same content to Claude Code or another assistant.

Option 1: As a Kiro steering file

Drop the steering file into your repository at .kiro/steering/vuln-triage.md. Any session using Kiro’s built-in default agent loads the instructions automatically. When you or your engineers ask the model to analyze code for security issues, it applies the methodology without additional prompting.

If your team uses a custom agent rather than the default, steering files aren’t loaded automatically; you have to add the file to the agent’s resources explicitly:

{
    "resources": ["file://.kiro/steering/**/*.md"]
}
your-repo/
├── .kiro/
│   └── steering/
│       └── vuln-triage.md   ← steering file
├── src/
├── cdk/
└── ...

You can also place it in ~/.kiro/steering/ to apply the methodology across every workspace on your machine, rather than scoping it to a single repository.

Option 2: As a Kiro skill

If you want the methodology available on demand rather than consuming context window space in every turn, place it as a skill. Kiro loads only the skill’s name and description at startup and pulls in the full instructions when the skill activates. This keeps the instructions out of unrelated conversations.

A skill is a folder containing a SKILL.md file. The folder name must match the name in the YAML front matter:

name: vuln-triage
description: Structured vulnerability triage methodology with evidence-based scoring and infrastructure-aware prioritization. Use when analyzing code for security vulnerabilities.
your-repo/
├── .kiro/
│   └── skills/
│       └── vuln-triage/
│           └── SKILL.md   ← skill file
├── src/
└── ...

Kiro’s default agent discovers skills in .kiro/skills/ automatically; no configuration required. The skill then activates either automatically, when Kiro matches a request against the description, or explicitly as a slash command. A skill named vuln-triage becomes /vuln-triage, so an engineer can run an assessment on demand:

> /vuln-triage focus on the authentication changes

Place skills in ~/.kiro/skills/ to make them available across every project on your machine. If your team uses a custom agent rather than the default, skills aren’t loaded automatically, you must add them to the agent’s resources:

{

	"resources": ["skill://.kiro/skills/*/SKILL.md"]

}

Option 3: Adapted for other tools

The methodology isn’t specific to Kiro. The repository also ships the same content as a CLAUDE.md file for Claude Code, and the principles apply to any assistant that loads persistent instructions from your repository—consult your tool’s documentation for the filename and location it expects.

What this doesn’t replace

A steering file is methodology, not machinery. It tells the model how to think about findings but doesn’t provide:

  • Programmatic taint tracking: The model approximates dataflow by reading code. A purpose-built taint tracker confirms dataflow against the abstract syntax tree (AST) deterministically. For Python codebases, static analysis tools built on parsers like Tree-Sitter provide this. For other languages, the model’s approximation is reasonable but not authoritative.
  • Live infrastructure verification: The steering file instructs the model to parse IaC files. It can’t make API calls to verify that an AWS WAF rule is attached to the Amazon API Gateway in your deployed environment, or that a security group hasn’t been modified since deployment.
  • Threat intelligence feeds: The file instructs the model to consider CISA Known Exploited Vulnerabilities (KEV) and Exploit Prediction Scoring System (EPSS) data. The model’s training data includes historical KEV entries, but it can’t query live feeds. For current exploitation status, you need an API integration.
  • Execution-based verification: The steering file produces assessed findings. Confirming exploitability by executing a proof of concept against a live environment requires additional tooling: a sandbox, request signing, and differential testing infrastructure.

Each of these capabilities adds confidence to the pipeline. The steering file without them still produces measurably better results than an unsteered model: structured findings, cited evidence, and infrastructure-aware prioritization. But it represents the first layer of a mature harness, not the complete system.

What comes next

The steering file gives your team a structured methodology for AI-powered vulnerability triage today. It works with the tools you already have. It works without infrastructure changes, new services, or a procurement process.

The steering file is available in our GitHub repository. We encourage you to use it, adapt it to your environment, and let us know what you find.

The authors work on security automation at AWS. The patterns described here emerged from building and validating AI-powered vulnerability detection systems across multiple teams and deployment environments.

If you have feedback about this post, leave a comment in the Comments section below.


justin kontny

Justin Kontny

Justin is a Senior Software Engineer, Security, at AWS who blends a passion for software development with deep expertise in cloud security. He focuses on building agentic security solutions using AI-Driven Development Lifecycle (AIDLC) practices, transforming security from a barrier into a business enabler. Outside of work, Justin enjoys spending time with his children and staying active outdoors.

Ievgeniia Ieromenko

Ievgeniia Ieromenko

Ievgeniia is a Software Development Engineer at AWS, where she builds tooling and agentic AI systems designed to strengthen security posture without slowing delivery. Outside of work, she enjoys traveling, volunteering, and outdoor adventures with her very good dog, Pickle.

nidhi ramakant

Nidhi Ramakant

Nidhi is a Software Development Manager at AWS with two decades of experience in security and enterprise architecture. She’s obsessed with helping customers secure their environments and building creative solutions that make security decisions easier. She’s equally invested in developing the talent around her. Outside work, she’s either binging shows or vibe coding apps for secure local LLMs so her kids can explore and learn safely.

  •  

Improving SPIRE security and resiliency with AWS managed services

In cloud-centered environments, establishing trust between workloads is fundamental to securing machine-to-machine communication. Traditional approaches such as API keys, shared secrets, and static service account credentials weren’t designed for the scale and ephemeral nature of cloud workloads. This drives an organizational need to shift from long-term, static credentials to short-lived cryptographic workload identities. Organizations are turning to SPIFFE (Secure Production Identity Framework for Everyone) as a core component of their workload identity strategy. SPIFFE is a set of open source standards for securely identifying software systems in dynamic and heterogeneous environments. Using SPIFFE can help address what’s known as the bottom turtle problem: the circular dependency where protecting one credential requires yet another credential.

SPIRE (the SPIFFE Runtime Environment) is an open source implementation of SPIFFE. Customers can use it to quickly experiment with the framework. However, there are operational and security considerations when deploying SPIRE in production, such as:

  • Where cryptographic signing keys for workload identities are managed
  • How to ensure availability and resiliency of the workload identity registry
  • How the SPIRE Certificate Authority integrates into the existing enterprise PKI
  • How workloads without direct connectivity to the SPIRE server validate identities
  • How workload identities are delivered to serverless environments
  • How fine-grained authorization based on SPIFFE IDs is enforced

In this post, we show you how to address each of these considerations by offloading SPIRE core functionality to AWS managed services. This post is accompanied by a Github Repository that guides you through the deployment of the reference architecture.

To learn the core concepts the SPIFFE framework, take a moment to familiarize yourself with the SPIFFE documentation.

SPIRE background

A SPIRE deployment is comprised of at least one SPIRE Server, at least one SPIRE Agent, and at least one workload.

Figure 1: SPIRE high-level architecture

Figure 1: SPIRE high-level architecture

The SPIRE Server is deployed on a central control plane instance (or instances) and manages identity issuance and stores workload identity registrations. The SPIRE server handles five key functions:

  • RegistrationAPI: Create, update, and delete registration entries that define which workloads are entitled to specific SPIFFE IDs based on selectors (for example, Kubernetes namespace, Unix UID).
  • DataStore: The SPIRE server uses a data store to keep track of the workload identity registration entries and the status of the SVIDs it has issued.
  • KeyManager: Controls how the server manages private keys used to sign X.509-SVIDs and JWT-SVIDs.
  • BundlePublisher: Publishes the local trust bundle to a store. The trust bundle is an object containing a trust domain’s cryptographic keys
  • UpstreamAuthority: Dictates which root certificate authority is used to sign the SPIRE certificate authority (CA) certificate

The SPIRE Agent is distributed across your workloads, whether they run on Amazon Elastic Compute Cloud (Amazon EC2) instances, Amazon Elastic Container Service (Amazon ECS) containers, or Amazon Elastic Kubernetes Service (Amazon EKS) Pods. The SPIRE Agent handles:

  • WorkloadAPI: Exposes the workload API to workloads, which enable workloads to retrieve their SVIDs and trust bundles from the SPIRE server.
  • NodeAttestation: SPIRE requires that each agent attest and verify itself.
  • WorkloadAttestation: The SPIRE agent collects workload metadata to determine its identity.
  • SVIDStore: The SPIRE Agent also stores SVIDs in other destinations, such as AWS Secrets Manager or HashiCorp Vault making them available for workloads running in serverless environments.

After the server and agent are deployed, applications retrieve short-lived SPIFFE Verifiable Identity Documents (X.509 Certificates or JSON Web Tokens (JWTs)) using the workload API.

How to use AWS managed services for core SPIRE functions

In the following sections, we dive deeper into what each of these SPIRE functions does and the advantages of using each AWS service for the function.

Figure 2: SPIFFE architecture using AWS managed services

Figure 2: SPIFFE architecture using AWS managed services

To deploy the infrastructure required to follow along with this post, follow the deployment process in the Github Repository.

Note: For each managed service, there’s a parameter in the source AWS CloudFormation templates that you can use to specify whether to provision the resource. For example, if you already have an AWS Private Certificate Authority (AWS Private CA) certificate authority deployed in your environment, you can use that for your SPIRE implementation and avoid provisioning a new certificate authority.

SPIRE Key Manager using AWS KMS

The SPIRE Key Manager controls the cryptographic keys that are used to sign SPIFFE Verifiable Identity Documents (SVIDs), which are either X.509 certificates or JWTs.

By using SPIRE, you can configure the cryptographic keys to be either stored in memory or on disk, or to use a plugin where external keys are used for signing SVIDs.

By using the AWS KMS plugin for SPIRE, you bring the security benefits of AWS KMS to your SPIRE implementation.

  • HSMs: AWS KMS uses hardware security modules (HSMs) that have been validated under FIPS 140-3 Level 2
  • Key protection: Your plaintext AWS KMS keys don’t leave the HSMs, aren’t written to disk, and are only used in the volatile memory of the HSMs for the time needed to perform your requested cryptographic operation
  • Comprehensive auditing: Each request to use the keys for signing must be authenticated using Signature Version 4 (SigV4) and is audited and logged in AWS CloudTrail
  • Fine-grained access control: Use AWS KMS Key Policies to restrict access to your signing keys

This configuration block indicates to SPIRE that AWS KMS keys are used to sign our SVIDs:

KeyManager "aws_kms" {
plugin_data {
region = "<your-region>"
key_identifier_file = "./key_metadata"
}
}

This code block exists in the SPIRE server configuration deployed in the Github repository.

SPIRE datastore using Amazon Aurora

The SPIRE server uses a data store to keep track of the workload identity registration entries as well as the status of the SVIDs it has issued. By default, the datastore lives as an SQLite database in memory. However, you can configure the SPIRE server to use Amazon Aurora. By doing this, you can decouple the database from the server and offload the operational overhead of managing the datastore to AWS.

Benefits:

  • High availability: Automatic failover and multi-Availability Zone deployments
  • Automated backups: Point-in-time recovery and automated backup retention
  • Scalability: Scale compute and storage independently
  • Security: Encryption at rest and in transit, with AWS Identity and Access Management (IAM)
  • Monitoring: Built-in performance insights and enhanced monitoring
  • Reduced operational burden: Automated patching, maintenance, and updates

To do this, the CloudFormation stack provisions:

  • An Aurora PostgreSQL Cluster
  • A database user with the capability to authenticate through AWS IAM
  • Network connectivity between your SPIRE Server and the database

After the sample finishes deployment, the configuration in the SPIRE server is changed from the default:

DataStore "sql" {

plugin_data {

database_type = "sqlite3"

connection_string = "/opt/spire/data/server/datastore.sqlite3"

}

}

To:

DataStore "sql" {

plugin_data {

database_type "aws_postgres" {

region = "<region>"

}

connection_string = "dbname=<dbname> user=spiffe host=<your_db_endpoint> port=<your_port>"

}

}

For more information, see Datastore SQL Plugin.

SPIRE certificate authority using AWS Private Certificate Authority

The SPIRE Server issues SVIDs to SPIRE agents and workloads. Given that these SVIDs are X.509 certificates, the SPIRE server needs to act as a certificate authority. In the default configuration of SPIRE, the server generates a self-signed certificate that it will use as the root CA certificate. In production scenarios, organizations often use their existing AWS Private CA certificate authority hierarchy.

Using the AWS Private CA plugin for SPIRE, you configure the SPIRE CA to act as an issuing certificate authority that chains to your existing root CA. This delivers three benefits for your SPIRE implementation:

  • HSM-backed root of trust: The root of trust of your CA hierarchy is backed by fully managed, FIPS 140-2 Level 3 validated hardware security modules
  • Non-exportable private keys: The private keys for the root CA can’t be exported
  • Fully Managed Certificate Revocation Lists and OCSP endpoint: In case the issuing CA certificate needs to be revoked

Note: If the SPIRE CA certificate will be signed by an issuing CA, you need to ensure that the path length of that CA is at least 1. The sample code provisions a root CA, so this isn’t an issue if you’re following the sample code.

After the sample deployment is complete, the configuration of the SPIRE server configuration includes the UpstreamAuthority plugin to use your AWS Private CA certificate authority as the root of trust.

UpstreamAuthority "aws_pca" {

plugin_data {

region = "<aws_region>"

certificate_authority_arn = "<your_pca_arn>"

ca_signing_template_arn = "arn:aws:acm-pca:::template/SubordinateCACertificate_PathLen0/V1"

}

}

For more information, see the AWS Private CA upstream authority plugin documentation.

SPIRE bundle publisher using Amazon S3

In SPIFFE, trust bundles are sets of public keys that are used by destination workloads to verify the identity of source workloads. These bundles contain the root certificates and public keys used to sign JWT SVIDs.

The SPIRE Server supports exposing an endpoint containing this trust bundle or publishing the bundle to Amazon Simple Storage Service (Amazon S3).

By publishing the bundle to Amazon S3, you can:

  • Decouple: Decouple bundle access from the SPIRE server availability
  • Continuous validation: Allow destination workloads to continually validate source workload identities without requiring network access to the SPIRE server
  • Global distribution: Distribute the trust bundle using Amazon CloudFront, improving performance through the CloudFront global edge network with over 750 Points of Presence
  • Reduced load: Offload trust bundle retrieval traffic from the SPIRE server
  • Version control: Maintain historical versions of trust bundles with S3 versioning

The following is a sample configuration for this plugin:

BundlePublisher "aws_s3" {

plugin_data {

region = "<your_bucket_region>"

bucket = "<your_bucket_name>"

object_key = "<your_bundle_name"

format = "spiffe"

}

}

For more information, see S3 Bundle Publisher Plugin Reference.

SPIRE Agent SVID store using Secrets Manager

In a typical SPIRE implementation, SVIDs are retrieved from the workload API using the SPIRE agent by workloads. In serverless or container cases, it isn’t feasible or possible to run the SPIRE agent, especially in workloads running on AWS Lambda or an AI agent running in Amazon Bedrock AgentCore Runtime. In these scenarios, use the SVID store plugin to push the SVID to Secrets Manager.

In this scenario, you still need a SPIRE agent running on a node that performs node attestation with the SPIRE server and delivers the SVID to a secrets store. One of the available secrets store plugins is the AWS Secrets Manager SVIDStore Plugin.

Benefits:

  • Encryption at rest: Secrets are encrypted by default using AWS managed or customer-managed AWS KMS keys
  • Broad integration: Integrations with multiple compute types, including AWS Secrets Manager Lambda Extension, Amazon ECS, Amazon EMR, and 52 other AWS services
  • Client-side caching: Secrets Manager provides client-side caching libraries to improve performance and reduce API calls
  • Multi-region replication: Built-in multi-Region replication functionality for workloads with multi-region requirements
  • Fine-grained access control: Use resource policies to control which workloads access which SVIDs
  • Audit trail: All secret access is logged in CloudTrail

This plugin is configured in the agent, as opposed to the SPIRE server. A reference for the specific permissions required are in the plugin Github repository. In the sample code, see the configuration in the following agent.conf file:

SVIDStore "aws_secretsmanager" {

plugin_data {

region = "<your_region>"

}

}

When you need to specify a given workload’s SVID to be pushed to Secrets Manager, you add the storeSVID and selector parameters when you create the entry in the workload registry. If you’re managing your SPIRE server in Kubernetes, the appropriate command looks like:

kubectl exec -n spire spire-server-<postfix>-- \

/opt/spire/bin/spire-server entry create \

-spiffeID spiffe://example.org/<workload_name> \

-parentID spiffe://example.org/ns/spire/sa/spire-agent \

-selector aws_secretsmanager:secretname:<workload_name> \

-storeSVID

Fine-grained access control using Verified Permissions

Up to this point, all the managed services we’ve talked about have been used for creating and distributing SVIDs and trust bundles to workloads. However, a key advantage of using a framework such as SPIFFE is that it allows resource owners to protect resources with fine-grained access control. One managed service that helps you do that in the context of SPIFFE and SPIRE implementations is Amazon Verified Permissions.

Verified Permissions is a scalable, fine-grained permissions management and authorization service that helps you build secure applications. You can use it to:

  • Define declarative policies: Use Cedar policy language, which provides human-readable, declarative access control rules
  • Centralize policy management: Separate authorization logic from application code
  • Take advantage of the straightforward policy authoring, formal verification capabilities, and performance benefits of Cedar
  • Make real-time decisions: Evaluate authorization requests in real-time based on multiple factors including user identity, workload identity (SPIFFE ID), resource attributes, and contextual information
  • Integrate with identity providers: Support for OpenID Connect (OIDC) providers, enabling validation of JWT tokens including JWT-SVIDs

With Verified Permissions, you authenticate SPIRE issued SVIDs using the public keys associated with your trust domain, and deploy fine-grained authorization policies protecting your resources. To use Verified Permissions, you need a policy store, an identity source, and authorization policies.

The github sample creates a policy store on your behalf and sets up the necessary infrastructure to have SPIRE serve as an identity source for your policy store.

You will need policies to enforce fine grained authorization. The following sample policy forbids a specific workload from taking any action:

forbid(

principal,

action,

resource

)

when {

context has sub &&

context.sub like "spiffe://example.org/forbidden-workload/*"

};

Or conversely allows a specific workload to take actions:

permit(

principal,

action,

resource

)

when {

context has sub &&

context.sub like "spiffe://example.org/allowed-workload/*"

};

Experiment with these policies as you build on top of your SPIRE infrastructure.

Conclusion

This post demonstrates how organizations enhance their SPIRE deployments on AWS by replacing default implementations with AWS managed services. We showed you integrations with AWS KMS for SVID signing, Amazon RDS Aurora for managed datastore operations, AWS Private Certificate Authority for root of trust, Amazon S3 and CloudFront for global trust bundle distribution, AWS Secrets Manager for SVID storage, and Amazon Verified Permissions for fine-grained authorization. By adopting these integrations, organizations can improve security posture, reduce operational complexity, and enhance resiliency.

If you’re interested in learning SPIFFE on AWS, check out our SPIRE on AWS github sample or the SPIFFE and SPIRE on AWS Workshop for hands-on learning.

If you have feedback about this post, submit comments in the Comments section below.


Brendan Paul

Brendan is a Security Services SA Manager at AWS and has been at AWS for more than 7 years. He spends most of his time at work helping customers solve problems in the data protection and workload identity domains. Outside of work, he’s pursuing his master’s degree in data science from UC Berkeley.

Meg Peddada

Meg Peddada

Meg is a Principal FSI Security Solutions architect specializing in security, risk, and compliance. Her expertise spans governance, security automations, threat management, and architecture. In her spare time, she loves playing volleyball, arts and crafts, and finding new brunch experiences.

  •  

Identity-aware AI data agents with AWS Lake Formation and Trusted Identity Propagation

You’re building a data agent that lets business users ask questions about lakehouse data in natural language. You’ve already built governance policies that control who can access which datasets. The challenge is making the agent respect those rules without rebuilding them in your application code.

When a user asks a question, the agent maps it to data and constructs a query. The tool runs under its own AWS Identity and Access Management (IAM) role, so AWS Lake Formation sees the tool’s credentials, not the person behind the request. This leaves you with two inadequate options: restrict tool access (limiting self-service analytics) or rebuild access controls in application code (moving governance out of the data layer).

In this post, we show you a different approach: identity-aware AI data agents that propagate each user’s identity through every hop, from the user, through the agent, into the tool, so Lake Formation evaluates the user’s grants. Your application code makes no authorization decisions, existing Lake Formation policies work without modifications, and AWS CloudTrail records the actual person who accessed the data.

In this post, you learn how to:

  • Set up per-user data access controls for AI agents querying your lakehouse without rewriting your existing Lake Formation governance policies by configuring Amazon Bedrock AgentCore to carry each user’s identity through trusted identity propagation (TIP).
  • Ensure auditability and compliance by performing a server-side token exchange inside AWS Lambda that converts the identity token into Lake Formation credentials scoped to the real user, with CloudTrail recording every query.
  • Validate the pattern end-to-end by testing with multiple users and confirming per-user query results and CloudTrail audit evidence.

The building blocks in this post, OAuth 2.0 token delegation, AWS IAM Identity Center, Lake Formation, and Lambda are well-documented individually. The new constraint is that a foundation model (FM) now sits in the middle of the propagation chain. The FM is the agent’s brain, it decides which tools to call and what arguments to pass and anything that enters its context (prompts, tool schemas, arguments) is accessible within that trust boundary. So, the user’s identity token must reach the tool, but it must do so on the HTTP transport layer, bypassing the FM’s reasoning layer entirely. This post demonstrates the three Amazon Bedrock AgentCore configurations that achieve that.

Whether you’re building agents on Strands, LangGraph, or a similar framework, this post gives you a deployable pattern on Amazon Bedrock AgentCore. Security engineers will find the identity-transport and audit properties relevant, and data platform owners will see how existing Lake Formation grants extend to AI workloads with no changes.

Solution overview

With this pattern (shown in Figure 1), your data agent can query Lake Formation governed data with per-user access controls, full CloudTrail audit trails, and no changes to your existing governance policies. Here’s how it works.

  1. Two tokens travel through the system. An access_token authenticates the request at each trust boundary (Bedrock AgentCore Runtime, Bedrock AgentCore Gateway). An id_token carries your identity and is exchanged, server-side, for the identity context that Lake Formation evaluates.
  2. A separate TIP role carries only the IAM permissions needed to call service APIs, it has no Lake Formation data grants.
  3. Lake Formation evaluates only the propagated user identity, not the TIP role.

The request flow is:

  1. User to UI: The user authenticates with an OpenID Connect (OIDC) identity provider (IdP). This post uses Amazon Cognito, but you can use any OIDC provider, such as Okta. The UI receives an id_token and an access_token.
  2. UI to Bedrock AgentCore Runtime: The UI calls the agent running in AgentCore Runtime over HTTPS. The access_token goes in the standard Authorization header. The id_token goes in a custom header: X-Amzn-Bedrock-AgentCore-Runtime-Custom-IdToken. The HTTP body contains only the user’s prompt.
  3. AgentCore Runtime to agent code: The runtime validates the access_token against the configured JSON Web Token (JWT) authorizer, then passes the request to the agent container with both headers accessible through context.request_headers.
  4. Agent to Bedrock AgentCore Gateway: The agent opens a Model Context Protocol (MCP) connection to an AgentCore gateway, including both headers on the connection. The AgentCore gateway validates the access_token and forwards the custom header to its Lambda target.
  5. AgentCore Gateway to Lambda: The propagated headers arrive in context.client_context.custom[‘bedrockAgentCorePropagatedHeaders’].
  6. Lambda to the data layer: Lambda validates the id_token, exchanges it for an identity context, assumes a role with that context, and runs the Amazon Athena query under the user’s identity.

Lake Formation evaluates grants against the real user. Athena returns only the rows and columns that the user is entitled to see. CloudTrail records the assumed role with an onBehalfOf entry identifying the human.

Now that you’ve seen the end-to-end flow, the following sections walk through each piece, starting with what you need to have in place before you build.

Prerequisites

This post assumes you have the working knowledge of OAuth 2.0, IAM, and Lake Formation grants.

  • An IAM Identity Center instance with a trusted token issuer (TTI) configured to accept your OIDC IdP’s tokens.
  • An OAuth Application in IAM Identity Center configured for CreateTokenWithIAM with the JWT Bearer grant.
  • Lake Formation governing your AWS Glue Data Catalog, with grants already assigned to users or groups.
  • An Athena workgroup and an Amazon Simple Storage Service (Amazon S3) bucket for query results.
  • An OIDC IdP. The reference implementation uses Amazon Cognito as the demo IdP, but the pattern is IdP-agnostic. Auth0, Microsoft Entra ID, Okta, Ping, or any OIDC-compliant provider works identically, if your TTI accepts its tokens.

This post doesn’t walk through setting up any of these components. The existing AWS documentation covers each one. For more information, see the links in the preceding list and the related resources at the end of this post.

The three Bedrock AgentCore configuration steps

Three Bedrock AgentCore features make this pattern work. Together they form the chain of custody for the user’s id_token from the moment it arrives at the Bedrock AgentCore Runtime to the moment Lambda uses it.

Configure the runtime request header allow list

Bedrock AgentCore Runtime doesn’t pass request headers into the agent container by default. You opt in by declaring an allow list, either through agentcore configure or directly in the runtime configuration:

# .bedrock_agentcore.yaml

request_header_configuration:
	requestHeaderAllowlist:
		- Authorization
		- X-Amzn-Bedrock-AgentCore-Runtime-Custom-IdToken

AgentCore Runtime supports two types of forwarded headers:

  • The standard Authorization header for OAuth inbound JWT authentication (access_token), and
  • Custom headers prefixed with X-Amzn-Bedrock-AgentCore-Runtime-Custom-

The id_token in this pattern uses a custom header. Inside the agent, the allow listed headers arrive as a dictionary object on the request context.

The following Python code runs in the agent container:

from bedrock_agentcore.runtime import BedrockAgentCoreApp
 
ID_TOKEN_HEADER = "X-Amzn-Bedrock-AgentCore-Runtime-Custom-IdToken"
app = BedrockAgentCoreApp()
 
@app.entrypoint
def invoke(payload, context):
    request_headers = getattr(context, "request_headers", None) or {}
 
    id_token = ""
    access_token = ""
    for key, value in request_headers.items():
        if key.lower() == ID_TOKEN_HEADER.lower():
            id_token = value
        elif key.lower() == "authorization":
            access_token = value.replace("Bearer ", "")
 
    # ... build MCP headers and call the Gateway

The agent code reads the id_token from the HTTP transport and forwards it (also on the HTTP transport) to the next hop. It doesn’t treat the token as a tool argument and doesn’t inject it into a prompt.

Key takeaway: The runtime allow list is the first gate. Without it, the id_token doesn’t reach your agent code.

Configure AgentCore Gateway metadata for header propagation

An AgentCore Gateway is the Model Context Protocol (MCP) endpoint the agent talks to. When AgentCore Gateway invokes a Lambda target, it doesn’t forward arbitrary request headers by default. You configure which headers to propagate using metadataConfiguration.allowedRequestHeaders on the target:

from aws_cdk import aws_bedrockagentcore as agentcore
 
self.gateway_target = agentcore.CfnGatewayTarget(
    self, "AthenaTarget",
    name="athena-executor",
    gateway_identifier=self.gateway.attr_gateway_identifier,
    target_configuration=agentcore.CfnGatewayTarget.TargetConfigurationProperty(
        mcp=agentcore.CfnGatewayTarget.McpTargetConfigurationProperty(
            lambda_=agentcore.CfnGatewayTarget.McpLambdaTargetConfigurationProperty(
                lambda_arn=lambda_function_arn,
                tool_schema=...  # execute_athena_query
            ),
        ),
    ),
    metadata_configuration=agentcore.CfnGatewayTarget.MetadataConfigurationProperty(
        allowed_request_headers=["X-Amzn-Bedrock-AgentCore-Runtime-Custom-IdToken"],
    ),
    ...
)

Notice that the tool schema doesn’t include an id_token parameter; there’s no token parameter on any tool. The FM doesn’t see, select, or pass an id_token because the token isn’t part of the tool’s contract. It travels parallel to the tool call, on the HTTP connection, through metadataConfiguration.

This is a critical property for security. If you put the id_token in the tool schema instead, the FM becomes responsible for passing it, which means the token lands in prompts, traces, memory, and logs. Keeping the token off the tool contract keeps it out of the FM entirely.

Key takeaway: The metadataConfiguration of the AgentCore gateway is the second gate. It controls which headers cross from the agent into the Lambda function without touching the tool schema.

Read propagated headers in Lambda

On the Lambda side, the propagated header arrives not in the event body but in the client context, under a specific key.

The following Python code runs in the Lambda function:

ID_TOKEN_HEADER = "X-Amzn-Bedrock-AgentCore-Runtime-Custom-IdToken"
 
def lambda_handler(event, context):
    client_ctx = getattr(context.client_context, "custom", {}) or {}
    propagated_headers = client_ctx.get("bedrockAgentCorePropagatedHeaders", {})
    id_token = propagated_headers.get(ID_TOKEN_HEADER)
 
    if not id_token:
        return {"statusCode": 400, "body": {"error": "missing id_token"}}
 
    # Tool arguments from the agent's call (no id_token here)
    query = event["query"]
    database = event["database"]
    workgroup = event["workgroup"]
    output_location = event["output_location"]
 
    # ... validate token, exchange, run query

The event dictionary contains the tool’s declared parameters and nothing else. You reach the id_token only through context.client_context.custom[‘bedrockAgentCorePropagatedHeaders’]. That’s the handoff point.

Key takeaway: The id_token arrives through the client context rather than tool arguments; the FM has no access to it. The Lambda is the only component that reads the token.

Perform the server-side token exchange

After Lambda has the id_token, it validates the token and exchanges it for an identityContext. This step uses standard IAM Identity Center TIP mechanics. That it happens inside Lambda rather than anywhere else is what keeps the identity context from crossing a process boundary.

import boto3, jwt, time
 
def exchange_and_assume(id_token: str, user_sub: str):
    # 1. Validate id_token against the OIDC IdP's JWKS.
    claims = validate_id_token(id_token)  # signature, exp, aud, iss, token_use
 
    # 2. Exchange for an Identity Center identityContext.
    sso_oidc = boto3.client("sso-oidc")
    resp = sso_oidc.create_token_with_iam(
        clientId=OAUTH_APPLICATION_ARN,
        grantType="urn:ietf:params:oauth:grant-type:jwt-bearer",
        assertion=id_token,
    )
    identity_context = resp["awsAdditionalDetails"]["identityContext"]
 
    # 3. AssumeRole with ProvidedContexts
    sts = boto3.client("sts")
    assumed = sts.assume_role(
        RoleArn=TIP_ROLE_ARN,
        RoleSessionName=f"agent-tip-{user_sub[:8]}-{int(time.time())}",
        ProvidedContexts=[{
            "ProviderArn": "arn:aws:iam::aws:contextProvider/IdentityCenter",
            "ContextAssertion": identity_context,
        }],
    )
    creds = assumed["Credentials"]
    return boto3.Session(
        aws_access_key_id=creds["AccessKeyId"],
        aws_secret_access_key=creds["SecretAccessKey"],
        aws_session_token=creds["SessionToken"],
    )

Two properties come out of this exchange:

  • The identityContext is created and consumed inside a single Lambda invocation. It doesn’t get returned to the agent, the gateway, or the UI.
  • The resulting boto3.Session holds short-lived credentials whose underlying identity assertion is the real user. When the session calls Athena, the query runs with the user’s identity propagated. Lake Formation sees the user, not the Lambda function’s role.

Everything after this, including the Athena query and result formatting, is standard boto3.

Configure Lake Formation grants

You need one grant to the IAM Identity Center user or group. That’s the whole story at the Lake Formation layer.

aws lakeformation grant-permissions \
  --principal DataLakePrincipalIdentifier=arn:aws:identitystore:::user/<USER_ID> \
  --resource '{"Table":{"DatabaseName":"iceberg_db","Name":"trip_details"}}' \
  --permissions SELECT

This is the grant Lake Formation evaluates at query time. You add column-level and row-level filters to the same user or group the same way. Nothing here is aware of or specific to AI agents. If you already have a Lake Formation grants model for human users, you already have the grants this pattern needs.

Understanding the TIP role: The TIP role that Lambda assumes has no Lake Formation data grants. It holds only IAM permissions to call the service APIs: athena:* for query runs, glue:* for catalog reads, lakeformation:GetDataAccess for the query plan handshake, and Amazon S3 access for the Athena output bucket. When Lambda assumes this role with an identityContext attached through ProvidedContexts, Lake Formation evaluates only the propagated user identity against its grants. The role itself is transparent to the authorization decision.

In the more common agent runs as a role pattern, the role carries the grants, which is why per-user governance breaks. Here the role carries no grants; it’s a session vehicle, not an authorization subject.

Test the pattern end-to-end

Two users, same question, different outcomes.

User A has SELECT on trip_details. They ask the agent for five records from the table.

Figure 2: Five records are requested and returned by the agent

Figure 2: Five records are requested and returned by the agent

User B has no grant on trip_details. They ask the same question.

Figure 3: User doesn’t have access to the data

Figure 3: User doesn’t have access to the data

No code changed between the two interactions. No parameter was toggled. Lake Formation made the decision based on the propagated identity.

The CloudTrail record for the AssumeRole call shows the delegation:

{
  "eventSource": "sts.amazonaws.com",
  "eventName": "AssumeRole",
  "requestParameters": {
    "roleArn": "arn:aws:iam::<ACCOUNT_ID>:role/<TIP_ROLE_NAME>",
    "roleSessionName": "agent-tip-<user_sub_prefix>-<timestamp>",
    "providedContexts": [{
      "providerArn": "arn:aws:iam::aws:contextProvider/IdentityCenter",
      "contextAssertion": "<identity-context-assertion>"
    }]
  },
  "userIdentity": {
    "type": "AssumedRole",
    "onBehalfOf": {
      "userId": "<IDENTITY_STORE_USER_ID>",
      "identityStoreArn": "arn:aws:identitystore::<ACCOUNT_ID>:identitystore/<ID>"
    }
  }
}

The onBehalfOf block closes the audit loop. Each query the agent runs on a user’s behalf has a CloudTrail record naming that user, with no additional instrumentation in your code.

Security properties

Four properties follow from this architecture. These are the core value propositions of the identity-aware pattern:

  1. The id_token never reaches the foundation model: It travels on HTTP headers at every hop, and Lambda reads it from context.client_context.custom. It’s not a parameter on any tool. The FM has no path to it: not in tool arguments, not in prompts, not in memory, not in traces.
  2. The identity context stays inside a single Lambda invocation: It’s derived from the id_token, used immediately in an AssumeRole call, and discarded. It doesn’t go back to the agent, the gateway, or the UI.
  3. Authorization decisions live in Lake Formation, against the real user only: The TIP role the Lambda function assumes has no data grants. Lake Formation evaluates the propagated user identity. No code path in the agent, gateway, or Lambda function performs authorization logic.
  4. The audit trail requires no extra work: The CloudTrail AssumeRole event with onBehalfOf identifies the human user for every query. You get the same audit fidelity you would have for human users accessing data directly.

Deploy the pattern

To deploy this pattern, you need to configure four things:

  1. An OIDC IdP with a TTI in IAM Identity Center accepting its tokens, and an Identity Center OAuth Application configured for CreateTokenWithIAM.
  2. Bedrock AgentCore Runtime running your agent container with requestHeaderAllowlist covering Authorization and your custom id_token header.
  3. Bedrock AgentCore Gateway with a Lambda target whose metadataConfiguration.allowedRequestHeaders includes the id_token header. The Lambda target’s tool schema has no id_token parameter.
  4. Lambda reading the id_token from context.client_context.custom[‘bedrockAgentCorePropagatedHeaders’], performing the token exchange through sso-oidc:CreateTokenWithIAM, and calling sts:AssumeRole with ProvidedContexts to get the TIP-bearing session.

Conclusion

When an AI agent queries a governed lakehouse, the data layer needs to know who’s asking, not which role the agent is running under. This post showed you how to resolve that by treating the agent as an OAuth delegated actor. The user’s token travels alongside the agent’s HTTP transport but doesn’t enter the model’s context, and the token exchange that produces query-time credentials happens server-side inside Lambda, scoped to a single invocation.

The three Bedrock AgentCore features that make this composable (requestHeaderAllowlist on runtime, metadataConfiguration.allowedRequestHeaders on gateway, and bedrockAgentCorePropagatedHeaders on Lambda) are specific to building on Bedrock AgentCore. Everything downstream of Lambda is IAM Identity Center and Lake Formation functionality.

If you’re building agents that read governed data, you don’t have to choose between a single over-permissioned service role and per-user code paths. The identity the data layer evaluates can be the real user, the audit trail can name the real user, and the foundation model doesn’t need to know the user’s token exists.

The result is a clean separation: the user’s identity travels end-to-end, the model never sees it, and the data layer enforces it exactly as if the user queried directly.

Related resources

If you have feedback about this post, submit comments in the Comments section below.


Thiyagarajan Mani

Thiyagarajan Mani

Thiyagarajan is a Sr. Delivery Consultant at AWS. With over 20 years of experience, he architects modern data platforms spanning lakehouses, ingestion pipelines, and generative and agentic AI applications. He specializes in transforming legacy data ecosystems into scalable, cloud-centered architectures that unlock advanced analytics and intelligent automation for customers. Outside work, he enjoys time with family and riding bikes.

Mihir Borkar

Mihir Borkar

Mihir is a Senior Solutions Architect at AWS, partnering with ISVs to build and scale their platforms. With a decade of experience across data architecture and delivery, his background in enterprise-scale engagements gives him a builder’s perspective, bridging architecture strategy with practical implementation. Outside work, he explores the latest developments in cloud technologies and AI/ML.

Philippe-Duplessis-Guindon

Philippe Duplessis-Guindon

Philippe is a Delivery Consultant at AWS, where he has worked on a wide range of generative AI projects, touching on most aspects from infrastructure and DevOps to software development and AI/ML. After earning his bachelor’s in software engineering and a master’s in computer vision and machine learning from Polytechnique Montréal, he joined AWS to help customers accelerate their generative AI journeys.

Umang Khambhalikar

Umang Khambhalikar

Umang is a Senior Consultant at AWS Professional Services, specializing in data and AI. He helps enterprise customers design and implement secure, governed data platforms and agentic AI solutions using services like Amazon Bedrock AgentCore, AWS Lake Formation, and AWS Glue. Umang brings over 19 years of IT consulting experience to solving complex challenges at the intersection of security, data governance, and generative AI.

  •  

Transforming Bedrock Guardrails events into OCSF with CloudWatch

Security teams investigating possible AI-related security events need guardrail intervention data alongside their existing security telemetry. When a guardrail identifies or blocks a prompt injection attempt or redacts sensitive data, that intervention carries additional investigative value comparable to a failed sign-in or a network intrusion alert. AWS Bedrock publishes this telemetry to AWS CloudWatch metrics and model invocation logs for operational monitoring.

In this post, I show you how to transform AWS Bedrock Guardrails intervention events into structured Open Cybersecurity Schema Framework (OCSF) Detection Finding records and land them in the CloudWatch unified data store. Launched in December 2025, the unified data store consolidates operational, security, and compliance data from AWS services and third-party sources into a single platform. Security operations center (SOC) and threat analysts can query guardrail events alongside identity, network, and endpoint data, including AWS CloudTrail and AWS Virtual Private Cloud (AWS VPC) Flow Logs, using AWS Athena or CloudWatch Logs Insights.

Use case

Consider an organization deploying AWS Bedrock across many applications, with guardrails applied as a governance control. The guardrails detect and block several categories of activity: harmful content, off-topic queries, sensitive information such as personally identifiable information (PII), and prompt injection attempts. Each time a guardrail intervenes, it produces a signal the security team wants to investigate alongside the rest of its telemetry.

Without a consolidation layer, guardrail interventions stay siloed as per-AWS Region CloudWatch metrics and entries in raw model invocation logs, separate from the identity, network, and audit data the security team already relies on. The CloudWatch unified data store is built to remove this kind of silo: it consolidates operational and security data across accounts and Regions into one store. Routing normalized guardrail records into it is what makes them correlatable with the rest of your security telemetry. To investigate effectively, the team needs to answer questions that span multiple data sources:

  • Identify which user accounts trigger the most guardrail interventions, and whether those same accounts also show unusual AWS Identity and Access Management (IAM) activity such as failed API calls or privilege changes. A spike in prompt injection blocks from a role that’s also generating access-denied events is a stronger signal than either on its own.
  • Determine whether prompt injection attempts correlate with specific source IP addresses that also appear in VPC Flow Logs, which helps distinguish a single misbehaving client from a coordinated pattern across the environment.
  • Track the trend of guardrail violations across the organization and compare it against a baseline from 30 days ago, so the team can tell routine policy noise apart from an emerging change in attacker behavior or a misconfigured application.

AWS GuardDuty already helps here: its AI Protection feature analyzes CloudTrail data events from AWS Bedrock and produces managed threat findings, including a direct prompt injection finding and anomalous model invocation findings. GuardDuty AI Protection is a complementary managed detection layer for AWS Bedrock activity. The pipeline in this post works alongside it: it captures the raw guardrail intervention records themselves, normalizes them to OCSF, and lands them in the unified data store so you can run your own cross-source correlation and long-term trend analysis alongside GuardDuty findings, CloudTrail, and VPC Flow Logs.

Guardrail traces arrive as JSON embedded in the AWS Bedrock model invocation logs, in an AWS Bedrock-specific shape rather than a normalized security schema. After the pipeline transforms them to OCSF and ingests them into the CloudWatch unified data store, a single Athena query answers all three questions, because guardrail findings then share the same field structure as other security sources and sit in the same catalog as AWS CloudTrail events and VPC Flow Logs.

Solution overview

At a high level, the solution captures guardrail intervention events from AWS Bedrock model invocation logs, transforms them into OCSF Detection Finding records with an AWS Lambda function, and lands them in the CloudWatch unified data store, where SOC analysts query them with Athena or CloudWatch Logs Insights. The CloudWatch unified data store provides Apache Iceberg-compatible access through AWS S3 Tables, CloudWatch Logs Insights queries, and, for supported sources, built-in OCSF normalization at ingestion.

Why a custom transform step

CloudWatch offers native OCSF conversion through the ParseToOCSF log transformation processor for specific AWS log sources. As of this writing, that processor supports five AWS log sources: CloudTrail, AWS Route 53 Resolver, VPC Flow Logs, AWS EKS audit logs, and AWS WAF. AWS Bedrock Guardrails isn’t one of the sources that ParseToOCSF supports, so guardrail traces aren’t converted to OCSF automatically.

Guardrail traces also need transformation logic that ingestion-time processors don’t provide. A single model invocation log can contain multiple guardrail assessments (for example, a content policy filter and a sensitive information policy match in one request), and the transform step splits that one log entry into one OCSF Detection Finding per assessment rather than a single combined record. This one-to-many expansion, combined with conditional severity assignment based on policy type, is application logic rather than field normalization. For these two reasons, this solution uses a dedicated transform step to produce OCSF records, then uses built-in CloudWatch mechanisms for enrichment, storage, centralization, and query.

This solution captures AWS Bedrock model invocation logs containing guardrail trace data, filters for intervention events, transforms them into OCSF Detection Finding records (class_uid 2004), and writes the structured records to a CloudWatch Logs log group that is associated with AWS S3 Tables through the CloudWatch unified data store managed integration. Guardrail interventions are detection events: the guardrail detected and blocked prohibited content, so OCSF class 2004 (Detection Finding) under the Findings category is the appropriate classification.

How a source’s records are stored depends on whether it’s a vended source or a custom source. For the five vended sources that ParseToOCSF supports, the unified data store produces individual typed OCSF columns. For any custom source, including AWS Bedrock Guardrails in this solution, records land in the associated S3 Tables table as OCSF JSON in a single managed cwl__message column rather than as typed columns. This is a property of custom sources in general, not specific to AWS Bedrock. You read the OCSF fields at query time with Athena json_extract functions, and you can correlate them with the vended sources that share the same catalog. This post shows that query pattern.

The pipeline supports multiple AWS Bedrock event types by changing the filter pattern and OCSF mapping. This post demonstrates the guardrail interventions row:

Bedrock event source OCSF event class class_uid
Guardrail interventions (this post) DETECTION_FINDING 2004
Model invocation API calls API_ACTIVITY 6003
Agent guardrail traces DETECTION_FINDING 2004
Token consumption anomalies DETECTION_FINDING 2004

Architecture

The following diagram shows the end-to-end pipeline from guardrail intervention to CloudWatch unified data store ingestion. It shows the flow within a single account for clarity. In an organization, you deploy this transform pipeline in each workload account and use CloudWatch Logs centralization to bring the OCSF output into one central account, so you query organization-wide without standing up the query layer per account or per Region. The Multi-account deployment section later in this post covers this pattern.

Figure 1: End-to-end pipeline from guardrail intervention to CloudWatch unified data store ingestion

Figure 1: End-to-end pipeline from guardrail intervention to CloudWatch unified data store ingestion

The data flow consists of the following steps:

  1. An application calls AWS Bedrock (InvokeModel or Converse API) with a guardrail attached.
  2. AWS Bedrock evaluates the guardrail and logs the invocation, including guardrail trace data, to a CloudWatch Logs log group using model invocation logging. A subscription filter matches log entries where the guardrail action is INTERVENED (blocked or masked content).
  3. The subscription filter delivers matching records to a Lambda function (OCSF Transform). This step performs the custom transformation that native OCSF conversion doesn’t cover for AWS Bedrock, expanding each guardrail assessment into its own OCSF Detection Finding.
  4. The Lambda function writes the OCSF JSON records to a dedicated CloudWatch Logs log group and sets contextual fields (account ID and Region) on each record during transformation. If the Lambda function fails to process a record, the message routes to an AWS Simple Queue Service (AWS SQS) dead-letter queue for later analysis and redrive.
  5. The destination log group is associated with AWS S3 Tables through the CloudWatch unified data store managed integration, so the unified data store stores the OCSF records as an Apache Iceberg table. Because AWS Bedrock Guardrails is a custom source, each record is stored as OCSF JSON in a managed cwl__message column.
  6. SOC analysts query guardrail intervention data with Athena, using json_extract to read the OCSF fields, and correlate it with vended security sources (CloudTrail, VPC Flow Logs) that share the same catalog. For real-time operational queries, analysts can also query the destination log group directly with CloudWatch Logs Insights.

OCSF mapping

The following table shows how AWS Bedrock Guardrails intervention fields map to OCSF Detection Finding (class_uid 2004) attributes.

OCSF field Source Example value
class_uid Static 2004 (Detection Finding)
category_uid Static 2 (Findings)
severity_id Derived from policy type 3 (Medium) for content/topic; 4 (High) for prompt injection
activity_id Static 1 (Create)
time Invocation log timestamp 1721001600000
cloud.provider Static AWS
cloud.region Set by the transform function us-east-1
cloud.account.uid Set by the transform function 123456789012
actor.user.uid Invocation log identity.arn arn:aws:sts::123456789012:assumed-role/AppRole/session
finding_info.title Derived from policy type ContentPolicy Intervention
finding_info.desc Guardrail trace action/topic Blocked: HATE content detected on INPUT
resource.uid Model ARN arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-sonnet-4-20250514-v1:0
resource.type Static AwsBedrock:Model
metadata.product.name Static Amazon Bedrock Guardrails
metadata.product.vendor_name Static AWS
metadata.version Static 1.3.0
unmapped.guardrail_id Guardrail trace guardrailId my-content-guardrail
unmapped.guardrail_arn Guardrail trace guardrailArn arn:aws:bedrock:us-east-1:123456789012:guardrail/abc123
unmapped.guardrail_version Guardrail trace guardrailVersion 3
unmapped.guardrail_content_source Guardrail trace INPUT or OUTPUT
unmapped.guardrail_policy_type Guardrail trace ContentPolicy, TopicPolicy, SensitiveInformationPolicy, WordPolicy, ContextualGroundingPolicy, PromptAttack

Prerequisites

The following prerequisites are needed to deploy the reference implementation. Before you begin, clone the repository:

git clone https://github.com/aws-samples/sample-bedrock-guardrails-cloudwatch.git
cd sample-bedrock-guardrails-cloudwatch

Verify you have the following:

Implementation

The reference implementation deploys two CloudFormation stacks: TransformPipelineStack and MonitoringStack. The following commands deploy the stacks in dependency order:

cdk deploy TransformPipelineStack \
  -c destination_log_group=/aws/cloudwatch-uds/bedrock-guardrails-ocsf

cdk deploy MonitoringStack \
  -c alarm_email=your-team@example.com

Or deploy both stacks at once:

cdk deploy --all \
  -c destination_log_group=/aws/cloudwatch-uds/bedrock-guardrails-ocsf \
  -c alarm_email=your-team@example.com

Enable model invocation logging

Model invocation logging captures the guardrail trace data you need. Turn on full request and response logging to a CloudWatch Logs log group and configure textDataDeliveryEnabled to capture the text bodies that include the guardrail trace output when a guardrail is attached to the invocation. The following AWS Command Line Interface (AWS CLI) command enables it:

aws bedrock put-model-invocation-logging-configuration \
  --logging-config '{
    "cloudWatchConfig": {
      "logGroupName": "/aws/bedrock/model-invocation-logs",
      "roleArn": "arn:aws:iam::123456789012:role/BedrockLoggingRole"
    },
    "textDataDeliveryEnabled": true
  }'

Because logging is an account-level prerequisite rather than part of the pipeline, the CDK stacks don’t manage it. Enable it once for each account before deploying.

Deploying the transform pipeline

The TransformPipelineStack deploys the Lambda function, subscription filter, destination log group, and dead-letter queue.

  • Create the subscription filter: The stack creates a CloudWatch Logs subscription filter on your model invocation log group with a filter pattern that matches guardrail interventions. This captures only the events where a guardrail blocked or modified content, not the successful pass-through events, which reduces Lambda invocations and cost.
  • Transform to OCSF: The Lambda function parses each CloudWatch Logs event and produces one OCSF Detection Finding record (class_uid 2004) per guardrail assessment in the trace. The transformation maps guardrail trace fields to OCSF attributes as described in the OCSF mapping table and sets contextual fields such as account ID and Region. Severity is set to High for prompt injection interventions and Medium for content, topic, or sensitive information interventions. The records contain structural metadata only; no prompt or response content is included.
  • Dead-letter queue: If the Lambda function fails to process a record, the message routes to an AWS SQS dead-letter queue with error metadata (error type, message, timestamp) for investigation and redrive.

Configure S3 Tables integration

To make the OCSF records queryable through Athena, associate the destination log group with AWS S3 Tables. This is a two-step managed setup: create an S3 Tables integration, then associate the log group as a data source. The integration delivers to the managed aws-cloudwatch table bucket at no additional storage charge.

  1. Tag the destination log group so the unified data store recognizes it as a data source. Use the tags cw:datasource:name (for example, bedrock_guardrails) and cw:datasource:type (for example, ocsf). The generated table name joins the two tag values with a double underscore, so these example tags produce a table named bedrock_guardrails__ocsf.
  2. Create the integration with CreateS3TableIntegration (CloudWatch Observability Admin), supplying an encryption configuration and an IAM role. This returns an integration AWS Resource Name (ARN).
  3. Associate the data source with AssociateSourceToS3TableIntegration (CloudWatch Logs), passing the integration ARN and the dataSource name and type from step 1.

The reference implementation performs steps 2 and 3 with the CloudFormation resource AWS::ObservabilityAdmin::S3TableIntegration and a custom resource that calls AssociateSourceToS3TableIntegration, so cdk deploy sets up the association for you.

After association, the OCSF records become available as an Apache Iceberg table in the managed catalog, at the path "s3tablescatalog/aws-cloudwatch"."logs"."bedrock_guardrails__ocsf". You can query it with Athena, AWS Redshift, or any Iceberg-compatible tool.

As noted in the solution overview, AWS Bedrock Guardrails is a custom source, so each record is stored as OCSF JSON in a managed cwl__message column. You read the fields at query time with the Athena json_extract_scalar function, as shown in the next section, and correlate against vended sources such as CloudTrail in the same catalog.

Enable analytics access

The association delivers records to the managed table, but analytics engines can’t read them until you enable the AWS S3 Tables analytics integration and grant read permissions. Complete these steps once in each Region:

  1. In the AWS S3 console, choose Table buckets, then choose Enable integration. The first time you do this in a Region, AWS S3 creates a service role that lets AWS Lake Formation federate access to your tables in the AWS Glue Data Catalog. This is what makes the s3tablescatalog catalog visible to Athena.
  2. In the Lake Formation console, under Data lake permissions, choose Grant, and grant Select and Describe on the aws-cloudwatch table bucket (and the bedrock_guardrails__ocsf table) to each IAM principal that runs queries, such as your SOC analyst roles. Without this grant, Athena queries return an access denied error even when the IAM policy allows Athena.
  3. In the Athena query editor, choose the AWS S3 Tables catalog from the data source list, and select the logs database.

Deploy the monitoring stack

The MonitoringStack creates a CloudWatch dashboard and alarms for pipeline health. The dashboard includes widgets for intervention count by policy type, transform errors, Lambda duration, and dead-letter queue depth. Alarms notify your team when the error rate exceeds 5 percent, the dead-letter queue depth exceeds 0, or p99 Lambda duration exceeds 10 seconds. Notifications are sent through AWS Simple Notification Service (AWS SNS).

Query interventions

After deployment, guardrail interventions are available through two query paths. Use the one that matches your investigation:

  • Athena (through AWS S3 Tables) for cross-source correlation and historical analysis. Athena joins the OCSF guardrail records with vended OCSF sources in the unified data store (such as CloudTrail and VPC Flow Logs) over long time ranges. Because AWS Bedrock Guardrails is a custom source, its OCSF fields live inside the cwl__message column, so you read them with json_extract_scalar. Use Athena when you need to join guardrail findings against other security data or run trend analysis across weeks or months.
  • CloudWatch Logs Insights for real-time, operational queries against the destination log group. Use Logs Insights when you need results in seconds on recent data, without a table join, such as during an active investigation.

The Athena queries in this section run against the managed table "s3tablescatalog/aws-cloudwatch"."logs"."bedrock_guardrails__ocsf". For readability, the examples reference it as bedrock_guardrails_ocsf after selecting the s3tablescatalog/aws-cloudwatch catalog and logs database in the Athena query editor.

Correlate with IAM activity in Athena

The following example identifies users who trigger both prompt injection interventions and failed CloudTrail activity. The guardrail side extracts fields from cwl__message; the CloudTrail side is a vended source with typed OCSF columns.

WITH guardrail_violators AS (
    SELECT json_extract_scalar(cwl__message, '$.actor.user.uid') AS user_arn,
           COUNT(*) AS violation_count
    FROM bedrock_guardrails_ocsf
    WHERE json_extract_scalar(cwl__message, '$.unmapped.guardrail_policy_type') = 'PromptAttack'
    GROUP BY json_extract_scalar(cwl__message, '$.actor.user.uid')
),
iam_failures AS (
    SELECT "actor.user.uid" AS user_arn, COUNT(*) AS failure_count
    FROM aws_cloudtrail__management
    WHERE "status_id" = 2
    GROUP BY "actor.user.uid"
)
SELECT g.user_arn, g.violation_count, i.failure_count
FROM guardrail_violators g
JOIN iam_failures i ON g.user_arn = i.user_arn
ORDER BY g.violation_count DESC;

You can also track violation trends by policy type over time to establish baselines and detect spikes:

SELECT date_trunc('day', from_unixtime(
         cast(json_extract_scalar(cwl__message, '$.time') as bigint) / 1000)) AS event_day,
       json_extract_scalar(cwl__message, '$.unmapped.guardrail_policy_type') AS policy_type,
       COUNT(*) AS violation_count
FROM bedrock_guardrails_ocsf
GROUP BY 1, 2
ORDER BY event_day, violation_count DESC;

If you run these correlation queries frequently, you can materialize the extracted OCSF fields into a typed table with CREATE TABLE AS SELECT (CTAS) and refresh it on a schedule. Queries against the materialized table use plain column names instead of json_extract_scalar, which keeps recurring dashboards and joins concise.

Real-time queries in CloudWatch Logs Insights

For operational queries on recent data without a table join, query the destination log group directly in CloudWatch Logs Insights:

fields @timestamp, severity, finding_info.title, actor.user.uid, unmapped.guardrail_policy_type
| filter class_uid = 2004
| filter unmapped.guardrail_policy_type = "PromptAttack"
| stats count() as attack_count by actor.user.uid
| sort attack_count desc
| limit 10

This query returns results in seconds on recently ingested records.

Multi-account deployment

For organizations running AWS Bedrock across multiple accounts, the recommended pattern is to transform locally and centralize the OCSF output, rather than replicating raw invocation logs and transforming centrally.

Deploy the subscription filter and transform pipeline in each workload account where model invocation logging is enabled. Each account produces OCSF records in its own destination log group. Then use CloudWatch Logs centralization to replicate those OCSF records into a central logging account automatically. Centralization integrates with AWS Organizations and picks up new accounts as they join, and you create the AWS S3 Tables integration one time in the central account (rather than in every workload account) for organization-wide querying.

Transforming in each workload account keeps the guardrail trace parsing close to the source, avoids moving raw invocation logs (which might contain prompt and response text) across account boundaries, and centralizes only the sanitized OCSF findings. If your organization prefers to keep transformation logic in one place, an alternative is to centralize the raw invocation logs first and run a single transform pipeline in the logging account; this concentrates the parsing logic but moves more data, including request and response content, between accounts.

Clean up

To avoid ongoing charges, destroy the stacks in reverse dependency order:

cdk destroy MonitoringStack --force
cdk destroy TransformPipelineStack --force

Conclusion

In this post, you learned how to transform AWS Bedrock Guardrails intervention events into OCSF Detection Finding records and ingest them into the CloudWatch unified data store. This integration gives SOC teams structured guardrail telemetry that correlates with CloudTrail, VPC Flow Logs, and the other security sources already available in CloudWatch. Because guardrail interventions are a custom source, their OCSF fields are stored as JSON and read with Athena json_extract functions, so you can join them with the vended OCSF sources that share the same catalog.

The pipeline filters for intervention events only, which keeps costs low while capturing the security-relevant signals. Athena over AWS S3 Tables provides cross-source correlation and historical analysis, and CloudWatch Logs Insights provides real-time operational queries.

Clone the reference implementation and adapt the OCSF mapping and subscription filter to your guardrail configuration. To extend the pipeline, apply the same transform pattern to agent guardrail traces, which map to the same OCSF Detection Finding class.

References

If you have feedback about this post, submit comments in the Comments section below.


Dhananjay Karanjkar

Dhananjay Karanjkar

Dhananjay is a Senior Lead Consultant at AWS Professional Services, specializing in agentic AI systems, multi-agent orchestration, and generative AI security. He holds two US patents and serves as a Responsible AI Champion, with a background spanning financial services, enterprise consulting, and enterprise-scale AI delivery. When not architecting AI solutions, he trains for triathlons, paints oil portraits, and is an avid reader.

  •  

Architecting a secure landing zone in the AWS European Sovereign Cloud

The AWS European Sovereign Cloud is a new, independent cloud for Europe, physically and logically separate from existing AWS Regions and operated within the European Union (EU). It provides the same services, features, and APIs as AWS commercial Regions, but runs as a distinct AWS partition (aws-eusc), with its own control plane, AWS Identity and Access Management (IAM), billing, console, and service endpoints. Understanding the partition boundary is the key that unlocks correct answers to questions about billing roll-ups, single sign-on (SSO), cross-account roles, AWS Direct Connect, and image distribution. In this post, we show you how to architect a secure, scalable landing zone in the AWS European Sovereign Cloud. We cover account structure and governance, identity managed as infrastructure as code (IaC), centralized logging to a security and event management (SIEM) tool, data protection, network and perimeter design, secure continuous integration and delivery (CI/CD) and artifact distribution, and incident response. Throughout, we map the design to the AWS Security Reference Architecture (AWS SRA) and the AWS Well-Architected Framework, and we call out which behaviors are platform boundaries of a sovereign partition and which are configuration choices you can adapt.

If you are evaluating compliance readiness alongside your landing zone build-out, see the companion post Landing Zone Accelerator Independent Assessment Report for C5:2020 now available on AWS Artifact. This post covers how to align with C5:2020 criteria and provides an independent assessment report and compliance workbook, resources that complement the architectural patterns described here.

The foundational concept: EUSC is a partition

AWS groups Regions into partitions. Every Region is in exactly one partition, and each partition has one or more Regions. Partitions have independent instances of AWS Identity and Access Management (IAM) and provide a hard boundary between Regions in different partitions. AWS commercial Regions are in the aws partition, Regions in China are in the aws-cn partition, and AWS GovCloud Regions are in the aws-us-gov partition. The AWS European Sovereign Cloud is the aws-eusc partition, with its first Region in Brandenburg, Germany (eusc-de-east-1).

Some AWS services provide cross-Region functionality, such as Amazon S3 Cross-Region Replication or AWS Transit Gateway Inter-Region peering. These capabilities work only between Regions in the same partition. You can’t use IAM credentials from one partition to interact with resources in a different partition. There are practical differences that impact your architecture, shown in the following table:

Dimension Commercial AWS (aws) AWS European Sovereign Cloud (aws-eusc)
ARN prefix arn:aws: arn:aws-eusc:
Console or endpoint domain amazonaws.com amazonaws.eu
AWS Organizations One organization in the partition A separate, independent organization
AWS IAM Identity Center Instance in the partition A separate instance in the partition
Billing Consolidated in the partition’s payer A separate payer and billing system (EUR currency)
Cross-partition features and services such as: sts:AssumeRole, VPC peering, Transit Gateway, AWS RAM, Amazon S3 replication Cross-Region features and functionality Not available across the aws and aws-eusc boundary

Every AWS Region is sovereign by design: if you find yourself architecting across Regions, note that this partition boundary means that the centralization—one organization, one logging account, one identity source, one billing roll-up—is achievable within each partition. In the EUSC you operate an independent landing zone in aws-eusc that mirrors your commercial operating model. Where you need to bridge the two clouds (for example, a standard application CI/CD system in an AWS commercial Region deploying into EUSC), you integrate at the network or API layer with separate credentials for each partition, not with cross-partition trust. With these partition fundamentals in mind, the remainder of this post walks you through the considerations to build a production-ready landing zone in the EUSC. Each section addresses a critical layer of the architecture, starting with how to write partition-aware infrastructure code that works across both aws and aws-eusc, then moving into the organizational and governance controls that underpin everything else.

Cross-partition IaC

These Terraform and AWS CloudFormation IaC snippets demonstrate partition-aware Amazon Resource Name (ARN) construction—a pattern that helps ensure your infrastructure code works unchanged across AWS partitions (such as standard aws, GovCloud aws-us-gov, or European Sovereign Cloud aws-eusc).

In the case of using the same Terraform script from a commercial Region, ensure that arn:aws isn’t hard coded. This Terraform script derives the partition at deploy time so the same modules work in both AWS commercial Regions and the EUSC.

# Terraform: partition-aware ARNs (works unchanged in aws and aws-eusc)data "aws_partition""current" {}
data "aws_partition" "current" {}
data "aws_region" "current" {}
data "aws_caller_identity" "current" {}
data "aws_organizations_organization" "current" {}

locals {
partition = data.aws_partition.current.partition# "aws" or "aws-eusc"
account_id = data.aws_caller_identity.current.account_id
org_id= data.aws_organizations_organization.current.id
ecs_task_execution_policy_arn = "arn:${local.partition}:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
central_logs_bucket_arn= "arn:${local.partition}:s3:::${local.org_id}-central-logs"
}

resource "aws_iam_role" "amazon_ecs_role" {
  name = "AmazonECSrole"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Sid= ""
        Principal = {
          # IAM service principals are "amazonaws.com" across all partitions,
          # this stays literal (do NOT use ${AWS::URLSuffix} here).
          Service = "ecs-tasks.amazonaws.com"
        }
      },
    ]
  })
}

resource "aws_iam_role_policy_attachment" "amazon_ecs_role_attach" {
  role= "AmazonECSrole"
  # Use ${local.partition } rather than hardcoding "aws" in the ARN.
  policy_arn = "arn:${local.partition}:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"
}
# CloudFormation: use the AWS::Partition pseudo parameter, never a literal "aws"

AWSTemplateFormatVersion: "2010-09-09"

Description: >-
Creates an ECS task execution role. Demonstrates using the
${AWS::Partition} pseudo parameter in ARNs instead of hardcoding "aws",
so the template works across partitions (aws, aws-cn, aws-us-gov, aws-eusc).

Resources:
  ExecRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: AmazonECSroleCF
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service:
                # IAM service principals are "amazonaws.com" across all partitions,
                # so this stays literal (do NOT use ${AWS::URLSuffix} here).
                - "ecs-tasks.amazonaws.com"
            Action: "sts:AssumeRole"
      ManagedPolicyArns:
        # Use ${AWS::Partition} rather than hardcoding "aws" in the ARN.
        - !Sub "arn:${AWS::Partition}:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy"

Outputs:
  ExecRoleArn:
    Description: ARN of the created ECS task execution role
    Value: !GetAtt ExecRole.Arn

Account structure and governance

AWS Control Tower offers a straightforward way to set up and govern an AWS multi-account environment, following prescriptive best practices. AWS Control Tower orchestrates the capabilities of several other AWS services, including AWS Organizations, AWS Service Catalog, and AWS IAM Identity Center, to build a landing zone in less than an hour. Resources are set up and managed on your behalf.

We recommend following the AWS Security Reference Architecture (AWS SRA) multi-account model structure for a EUSC deployment. Use the management account only for governance, deploy universal security guardrails through service control policies (SCPs), resource control policies (RCPs), and service deployments (such as AWS CloudTrail) that will affect all member accounts in the organization.

Region-deny SCPs are commonly applied in commercial Regions, but aren’t required (at this time) in the EUSC because of the physically and logically separated nature of its design.

Other possible SCPs for the management OU:

  • Service-level guardrails – Restrict which AWS services can be used, based on your compliance posture.
  • Network perimeter controls – Enforce virtual private cloud (VPC) endpoints, deny public access patterns, and restrict egress.
  • Encryption and key management – Require AWS Key Management Service (AWS KMS) managed keys for all data-at-rest services and enforce key policies aligned with your sovereignty requirements.

Note: As additional EUSC Regions or Local Zones become available, the partition boundary continues to enforce isolation from non-EUSC Regions. If you need to restrict usage to a subset of EUSC Regions (for example, only eusc-de-east-1 but not a future eusc-de-west-1), a Region-deny SCP would become relevant at that point.

Identity: IAM Identity Center as IaC, no direct payer access

IAM Identity Center is available in the AWS European Sovereign Cloud as an independent instance within the partition. You can connect it to your external identity provider (IdP)—Microsoft Entra ID, Okta, and so on—using SAML/SCIM, exactly as in AWS commercial Regions. If you already use Identity Center to federate in the commercial partition, you can point a second Identity Center integration at the same corporate IdP, so users keep one set of credentials. You manage permission sets, groups, and account assignments separately for each partition.

Manage permission sets and assignments as code

The following Terraform defines a permission set with both an AWS managed policy and an inline least-privilege policy, then assigns a group to a target account. Reproduce the aws_ssoadmin_account_assignment for each account or organizational unit (OU) mapping.

Because group membership comes from your IdP over SCIM, the IdP handles joiner, mover, and leaver, and access in EUSC updates automatically. No one receives direct access to the management account; all human access flows through IAM Identity Center permission sets assigned to non-management accounts.

data "aws_ssoadmin_instances" "this" {}
data "aws_partition" "current" {}

# Workload account the group is assigned to.
variable "analytics_workload_account_id" {
  type        = string
  description = "Account ID of the workload account to assign the permission set to"
}

locals {
  sso_instance_arn  = tolist(data.aws_ssoadmin_instances.this.arns)[0]
  identity_store_id = tolist(data.aws_ssoadmin_instances.this.identity_store_ids)[0]
}

resource "aws_ssoadmin_permission_set" "analytics_operator" {
  name             = "AnalyticsOperator"
  description      = "Operate analytics workloads; no IAM or billing"
  instance_arn     = local.sso_instance_arn
  session_duration = "PT4H"
}

# Attach an AWS managed policy
resource "aws_ssoadmin_managed_policy_attachment" "analytics_ro" {
  instance_arn       = local.sso_instance_arn
  permission_set_arn = aws_ssoadmin_permission_set.analytics_operator.arn
  managed_policy_arn = "arn:${data.aws_partition.current.partition}:iam::aws:policy/ReadOnlyAccess"
}

# Add a least-privilege inline policy (note the partition-aware ARNs)
resource "aws_ssoadmin_permission_set_inline_policy" "analytics_inline" {
  instance_arn       = local.sso_instance_arn
  permission_set_arn = aws_ssoadmin_permission_set.analytics_operator.arn
  inline_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Sid      = "OperateAnalyticsData"
      Effect   = "Allow"
      Action   = ["s3:GetObject", "s3:PutObject", "s3:ListBucket"]
      Resource = [
        "arn:${data.aws_partition.current.partition}:s3:::analytics-*",
        "arn:${data.aws_partition.current.partition}:s3:::analytics-*/*"
      ]
      Condition = { StringEquals = { "aws:RequestedRegion" = "eusc-de-east-1" } }
    }]
  })
}

# Group for analytics operators.
# In production this is typically synced from your IdP via SCIM; here we
# manage it directly so the config is self-contained.
resource "aws_identitystore_group" "analytics" {
  identity_store_id = local.identity_store_id
  display_name      = "analytics-operators"
  description       = "Analytics operators"
}

# Assign the group to a workload account with the permission set
resource "aws_ssoadmin_account_assignment" "analytics_to_workload" {
  instance_arn       = local.sso_instance_arn
  permission_set_arn = aws_ssoadmin_permission_set.analytics_operator.arn
  principal_id       = aws_identitystore_group.analytics.group_id
  principal_type     = "GROUP"
  target_id          = var.analytics_workload_account_id
  target_type        = "AWS_ACCOUNT"
}

Cross-account roles for governance, logging, and tooling

Cross-account roles within the EUSC partition work normally; this is how the logging and security-tooling accounts collect from workload accounts. Scope each trust policy to a specific principal and harden it with an external ID (for third-party tooling) and partition-aware ARNs.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "AWS": "arn:${AWS::Partition}:iam::<SECURITY_TOOLING_ACCOUNT_ID>:role/SecurityAuditCollector"
    },
    "Action": "sts:AssumeRole",
    "Condition": {
      "StringEquals": { "sts:ExternalId": "eusc-sec-audit" },
      "ArnLike": { "aws:PrincipalArn": "arn:${AWS::Partition}:iam::*:role/SecurityAuditCollector" }
    }
  }]
}

A role in the aws partition can’t assume a role in aws-eusc (or the reverse).

Logging and monitoring: centralized in EUSC, exported to your SIEM

A sovereign logging architecture requires three things:

  • A single, immutable store for all audit and operational logs
  • A central security account that runs detective controls and correlates findings
  • A reliable, in-partition path that feeds everything into your SIEM without data ever leaving the boundary.

In the subsections that follow, we walk through each layer: Centralized log collection in the Log Archive account, Amazon GuardDuty and AWS Security Hub administration through the Security Tooling account, and the pull-based SIEM integration pattern that keeps telemetry inside the EUSC partition.

Centralize logs in the Log Archive account

The Log Archive account holds the organization trail and a central log bucket as part of the landing zone. In the commercial AWS partition, global services like IAM route their CloudTrail events to us-east-1. In the EUSC, global services events are logged within the EUSC partition because the control plane is independent and located entirely within the EU.

Organization level detective services

GuardDuty and Security Hub are available in EUSC, but organization-wide auto-enable and some newer features might differ from commercial AWS features at any given time. Design the Security Tooling account as the delegated administrator where supported. If org-level auto-enable isn’t yet available, enable per-account through your IaC (AWS CloudFormation StackSets) so coverage is complete and code-managed. Treat the EUSC service and feature list as the source of truth and gate optional features behind a partition flag.

Network security and perimeter, including AWS Direct Connect

The AWS European Sovereign Cloud has its own sovereign AWS Direct Connect points of presence (PoPs), with dedicated networking infrastructure and connectivity from European providers, providing customers an autonomous network path into the partition. You terminate Direct Connect in a dedicated Network account and share connectivity to workload VPCs using Transit Gateway (with AWS RAM). A Direct Connect connection or Direct Connect gateway in the commercial partition can’t be extended into aws-eusc. To reach EUSC VPCs, you provision a separate Direct Connect connection that lands in the EUSC partition’s Network account. If your on-premises network already backhauls to AWS commercial Regions, you connect that network to EUSC with its own virtual interface or connection, or a site-to-site VPN. You don’t bridge the two AWS partitions through a shared Direct Connect gateway.

The following figure shows the recommended perimeter design in EUSC.

Figure 1: Recommended perimeter design in EUSC

Figure 1: Recommended perimeter design in EUSC

The perimeter design includes:

  • Centralized egress and inspection – Route workload egress through an inspection VPC in the Network account (gateway load balancer with your firewall of choice, or AWS Network Firewall. Keep workload VPCs private with no internet gateway.
  • Private service access – Use VPC interface endpoints (VPCe) for AWS service calls so traffic stays on the AWS network within the partition. VPCe doesn’t cross partitions; expose any commercial-partition service to EUSC consumers over DX/VPN and an in-EUSC load balancer.
  • DNS – EUSC has its own Amazon Route 53. For names that must resolve across clouds, use subdomain delegation or Resolver forwarding rules over your DX or VPN link rather than expecting hosted zones to be visible across partitions.
  • Segmentation as code – Express segmentation with security groups referencing prefix lists and keep the EUSC IP ranges current from the partition’s published ip-ranges file in your firewall automation.

Data protection

AWS Key Management Service (AWS KMS) is available in EUSC; use customer managed keys for all sensitive data stores and enforce their use with SCPs and key policies. Where your residency or operational-autonomy requirements call for it, evaluate AWS KMS external and imported key material options available in the partition.

For workloads where regulation mandates that key material never resides within the cloud provider’s infrastructure, configure an AWS KMS External Key Store (XKS) in the EUSC Region. The XKS proxy connects AWS KMS to your EU-based hardware security module (HSM) (on-premises or hosted with an EU trust service provider); all encrypt and decrypt operations are performed by your external key manager. Note the trade-offs: increased latency, reduced availability SLA, and added operational burden. Reserve XKS for the subset of data where regulatory or contractual obligations explicitly require it.

The EUSC Region has achieved SOC 2, BSI C5 Type 1 attestation, and seven ISO certifications, including ISO 27001, 27017, 27018, and 27701. Reference these in your data protection evidence packages when demonstrating encryption-at-rest and key management controls to EU regulators.

Secure CI/CD and distributing images across the partition boundary

If you need to deploy existing images or binaries into EUSC (aws-eusc) from existing AWS commercial (aws) accounts, you can’t use cross-partition Amazon Elastic Container Registry (Amazon ECR) replication, Amazon Machine Image (AMI) copy, or Amazon Simple Storage Service (Amazon S3) replication. Instead, treat EUSC as an independent supply-chain destination:

  • Container images – Build (or re-tag and re-sign) images and push to an Amazon ECR registry inside EUSC. ECR cross-Region replication works within the partition (useful as EUSC adds Regions or Local Zones), but the initial crossing from commercial is an explicit pipeline push using EUSC credentials. Sign images with a sovereign signing key and verify at deploy time.
  • AMIs and images – Rebuild golden images in EUSC with EC2 Image Builder (run the pipeline natively in EUSC), or import virtual machine (VM) images using Amazon S3 in EUSC and aws ec2 import-image. There is no direct cross-partition AMI copy.
  • Binaries and artifacts – Stage in an artifact bucket in the EUSC Shared Services account. Move packages across the boundary with aws s3 sync or AWS DataSync over your DX or VPN, or using controlled export, then distribute within the partition using in-partition S3 replication to other EUSC Regions or Local Zones as they come online.
# Push a container image to ECR inside EUSC (note the .eu endpoint).
# Credentials/profile must be for the aws-eusc partition.
aws ecr get-login-password --region eusc-de-east-1 --profile eusc-shared-services \
  | docker login --username AWS --password-stdin \
    111122223333.dkr.ecr.eusc-de-east-1.amazonaws.eu

docker tag company/runtime:7.x \
  111122223333.dkr.ecr.eusc-de-east-1.amazonaws.eu/company/runtime:7.x
docker push \
  111122223333.dkr.ecr.eusc-de-east-1.amazonaws.eu/company/runtime:7.x

Remember that endpoints and ARNs use amazonaws.eu in the EUSC partition. IAM service principals always use amazonaws.com regardless of partition.

Replicating deployment code and pipelines

Run a native deployment plane in EUSC (AWS CodePipeline, AWS CodeBuild, AWS CodeDeploy, or your existing tool deployed in-partition) in the Shared Services account, with cross-account deploy roles into workload accounts. If a commercial-partition continuous-integration system must deploy into EUSC, give it separate credentials for each partition; the clean pattern is OIDC federation with two trust configurations, one for each partition, because no cross-partition role assumption exists.

// Deploy role in an EUSC workload account, trusted by the EUSC Shared Services pipeline role
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "AWS": "arn:aws-eusc:iam::<SHARED_SERVICES_ACCT>:role/PipelineDeployRole" },
    "Action": "sts:AssumeRole",
    "Condition": { "StringEquals": { "sts:ExternalId": "EUSC-deploy" } }
  }]
}

For account vending and landing-zone-as-code, use Account Factory for Terraform (AFT) deployed in EUSC. AFT pipelines create accounts through AWS Control Tower, apply baseline guardrails, bootstrap the preceding partition-aware modules, and register OUs, giving you the accounts and account groups as code. Keep Terraform state for each partition in an in-EUSC Amazon S3 backend with an Amazon DynamoDB lock table; don’t share state across partitions.

The Landing Zone Accelerator on AWS (LZA) solution is an alternative deployment method that provisions a baseline security architecture and includes customizations for each partition, with consideration for service availability. A customized configuration baseline for European Sovereign Cloud was recently released and is accompanied by the LZA Compliance Workbook, which maps regional European security standards and international frameworks to over 200 security settings deployed by LZA.

Supported compared to by-design boundaries: A quick reference

Capability Status in EUSC What to do
AWS Control Tower account vending, controls Supported in-partition Govern the EUSC Region; drive vending with AFT; re-register OUs after Region changes
AWS Control Tower–managed or self-managed IAM Identity Center Configuration choice Choose self-managed to own permission sets as code
Permission set creation and assignment as IaC Supported Manage with SCIM groups from your IdP
Identity Center single home and delegated admin for each partition By-design behavior Administer from one Region; non-issue in single-Region EUSC
Cross-account roles (governance, logging, tooling) Supported within partition Scope trust to specific principals and ExternalId
Cross-partition AssumeRole, VPC peering, TGW, RAM, Amazon S3 replication Not available (security boundary) Integrate at network or API layer with separate per-partition credentials
Billing roll-up across accounts and Regions Supported within the EUSC org Aggregate in a finance or governance account in-partition
Billing roll-up across the aws and aws-eusc boundary Separate billing systems (EUR payer) Keep cost analysis in-partition or in an EU-resident tool
Multi-Region image distribution (Amazon ECR, AMI, and Amazon S3) Supported within partition Push into EUSC first, then replicate in-partition
GuardDuty and Security Hub Available—some org-auto-enable and features vary Delegate admin where supported; per-account enable using IaC otherwise
CloudFront, Shield Advanced, Firewall Manager, Inspector In planning at time of publication Follow on AWS Builder Center (capabilities) for release updates

Billing and cost governance

Roll up within the EUSC organization, not across partitions. Enable consolidated billing in the EUSC management account and deliver AWS Data Exports (Cost and Usage Report 2.0) to an S3 bucket in a dedicated finance or governance account in the Security or Infrastructure OU.

Set permissions so workload teams can query the curated data in that account; no one should be able to access the management or payer account directly. You can’t replicate billing data into the commercial partition; the EUSC has a separate payer (billed in EUR through the EU contracting entity).

Conclusion

Architecting in the AWS European Sovereign Cloud is, in most respects, architecting a second well-run AWS landing zone with one organizing principle that resolves nearly every design question: it’s an independent partition. Centralization of governance, identity, logging, and billing is fully achievable, but within the EUSC partition. The boundaries you encounter between commercial AWS and EUSC—no cross-partition roles, peering, replication, or billing roll-up—are the sovereignty guarantees doing their job.

Build the foundation as code. Use an AWS Control Tower landing zone driven by AFT, IAM Identity Center permission sets and assignments in Terraform federated to your corporate IdP, an immutable central log store, customer managed encryption keys constrained to the sovereign Region, and a Network account terminating a dedicated Direct Connect. Add a CI/CD plane that pushes images and artifacts into the partition with per-partition credentials. Keep every ARN partition-aware and every optional service behind a feature flag, and the same modules will serve both clouds.

To accelerate your build with additional enablement from AWS, explore the LZA Universal Configuration for European Sovereign Cloud on GitHub, which packages many of the patterns described in this post into a ready-to-deploy baseline. To complement your deployment with compliance readiness, the LZA Independent Assessment Report for C5:2020 evaluates how LZA’s security baseline maps to C5:2020 technical requirements, and you can download the report and the LZA Compliance Workbook from AWS Artifact.

Further reading

If you have feedback about this post, submit comments in the Comments section below or start a thread on AWS re:Post.


Pablo Pagani

Pablo Pagani

Pablo is a Systems Development Manager for AWS European Sovereign Cloud, based in Madrid, Spain. He has previously held roles within Enterprise Support and Professional Services. An active member of the Security Technical Field Community, he helps customers build a secure journey on AWS. Pablo developed his passion for computers while writing his first lines of code in BASIC on an MSX computer with 64 KB of RAM.

Margo Cronin

Margo is an EMEA Principal Solutions Architect specializing in Security & Compliance and is based out of Zurich Switzerland. Her interests include security, privacy, cryptography, and compliance. She is passionate about her work unblocking security challenges for AWS customers, enabling their successful cloud journeys. She is an author of the “AWS User Guide to Financial Services Regulations and Guidelines in Switzerland”.

  •  

AWS STS simplifies session token size limits and adds session token size monitoring

AWS Security Token Service (AWS STS) has simplified session token size limits, giving you more room for your session policies and session tags. STS has replaced the packed policy size and the overall session token size limits with a single token size limit of 4,096 bytes. STS now reports session token size in API responses, Amazon CloudWatch metrics, and AWS CloudTrail events. By using STS, you can also generate session tokens of different sizes, so you can find the maximum token size that your infrastructure can support.

The 4,096-byte limit is the current maximum, not a permanent ceiling. AWS might increase the limit as new capabilities are added that require session tokens to carry more information.

In this post, you learn what has changed, what this change means for you, and what to do next.

What has changed

AWS STS session-vending APIs, such as AssumeRole, AssumeRoleWithSAML, AssumeRoleWithWebIdentity, GetSessionToken, and GetFederationToken, return temporary security credentials: an access key ID, a secret access key, and a session token. This change governs the session token, the opaque string that STS creates from the session policies and tags you pass plus the context that AWS adds.

Three things have changed.

  • A single limit: Previously, STS enforced two size limits on the session token. It serialized and compressed your session policies and tags into a form called the packed policy, which had its own limit. The assembled token, which included the packed policy, had a separate overall limit. A request could fail against either limit, and both failures returned the same PackedPolicyTooLargeException, so you couldn’t tell which one you exceeded. STS now enforces a single limit: the assembled session token must fit within 4,096 bytes. The separate packed policy limit, which made failures hard to predict, has been removed. When a token exceeds the assembled session token limit, STS returns PackedPolicyTooLargeException. STS continues to use the same exception, so existing error-handling code works without an SDK update.
  • Session token size is now reported. Every successful response from an STS session-vending API includes SessionTokenSize (which reports the session token size in bytes) and SessionTokenUtilization (which reports the percentage of the 4,096-byte limit consumed). STS also returns PackedPolicySize in every successful response for backward compatibility. PackedPolicySize now reports the same value as SessionTokenUtilization, enabling applications that use older AWS SDK versions to monitor utilization through this field. These response fields are also recorded in CloudTrail events. In CloudWatch, SessionTokenSize and SessionTokenMaxSize (the enforced limit) are published in the AWS/STS namespace.
  • Testing is more straightforward: MinimumSessionTokenSize is a new optional parameter on the STS session-vending APIs. You can use it to increase a session token to at least the size you specify, up to 4,096 bytes. Use the parameter to find the maximum token size your infrastructure can handle.
Behavior Previously Now
Limits enforced Two: Packed policy size and assembled token size One: Assembled session token size (4096 bytes)
Error on failure PackedPolicyTooLargeException: The error message didn’t identify which of the two limits was exceeded. PackedPolicyTooLargeException: The updated message reports your session token size and the maximum allowed size, both in bytes.
Session token size visibility Not reported

API Response and AWS CloudTrail:

SessionTokenSize, SessionTokenUtilization, and PackedPolicySize. PackedPolicySize reports the same percentage as SessionTokenUtilization for backward compatibility.

Amazon CloudWatch:

SessionTokenSize and SessionTokenMaxSize

Infrastructure testing No mechanism MinimumSessionTokenSize: Parameter on sesssion-vending APIs

What this change means for you?

How this affects you depends on your situation. The following scenarios cover the most common cases.

  • If you have never hit a token size error: You’re unlikely to notice a change. Your tokens stay their current size and gain headroom. Over time they could become larger than your systems have handled before. We recommend you use MinimumSessionTokenSize to find the maximum token size your systems can handle. See the What to do next section for more details.
  • If you’ve hit PackedPolicyTooLargeException before: Some requests that previously failed now succeed under the single limit. Review any workarounds you put in place specifically to avoid token size errors and decide whether you still need them. General best practices still apply: consistent tag casing and reused tag values compress more efficiently, and concise session policies keep the assembled token smaller. No code change is required for error handling. AWS STS still returns PackedPolicyTooLargeException when the assembled session token exceeds the limit, the same exception STS returned before this change.
  • If your systems enforce their own size limits on credentials: If your application uses an AWS SDK to obtain temporary credentials and make AWS API calls, the SDK handles the session token internally, so token size doesn’t affect your code. Focus instead on systems that store or forward session tokens, such as load balancers, proxies, caches, and databases. These systems might have size limits that smaller tokens didn’t reach. For example, a database column defined as varchar(2048) can’t hold a 4,096-byte token. Review where you persist or pass session tokens, and identify the maximum token size each system supports. The next section shows how to test this.

What to do next

We recommend the following three steps to prepare your systems for this change.

  1. Validate the maximum token size your systems can handle. Use MinimumSessionTokenSize to find the maximum session token size each system in your infrastructure can handle. Knowing these limits helps you identify systems that might reject or truncate larger tokens. The 4,096-byte limit reflects today’s needs, not a permanent ceiling. It might grow as AWS introduces new capabilities such as additional context keys for new services, richer audit metadata, and larger cryptographic signatures as the industry transitions to post-quantum algorithms. Avoid hard-coding the current maximum into your systems and revisit any fixed size assumptions if the limit changes.

    Tip: AWS STS serializes and compresses your session policies and tags when assembling the token. Compression results vary based on the actual content, not just its length. Two sets of tags with identical character counts can produce different token sizes. This is why MinimumSessionTokenSize is a more reliable way to test your infrastructure than estimating from input length.

    aws sts assume-role \
      --role-arn arn:aws:iam::123456789012:role/MyRole \
      --role-session-name validation-test \
      --minimum-session-token-size 4096

    Start at 4,096 bytes to test against the largest possible token. If a system truncates or rejects it, lower the value to find the size your infrastructure supports, then raise that limit where you can. MinimumSessionTokenSize is available in the latest AWS SDK, AWS Command Line Interface (AWS CLI), and Tools for PowerShell versions. See the STS API Reference for details. If your AWS SDK or AWS CLI predates the parameter, update it to use this feature.

  2. Monitor your session token size (recommended). If your infrastructure has size constraints, you can use monitoring to see tokens that are approaching your limit and act before a request fails. AWS STS reports size through three channels, each suited to a different need.
    • In the API response: Reading SessionTokenUtilization and SessionTokenSize from the response requires the latest AWS SDK version. You can also monitor token size through CloudWatch and CloudTrail without updating your SDK.
    {
      "Credentials": {
        "AccessKeyId": "REDACTED",
        "SecretAccessKey": "REDACTED",
        "SessionToken": "REDACTED",
        "Expiration": "2026-06-30T12:00:00Z"
      },
      "AssumedRoleUser": { "...": "..." },
      "PackedPolicySize": 61,
      "SessionTokenSize": 2532,
      "SessionTokenUtilization": 61
    }
    • In CloudWatch: STS publishes SessionTokenSize and SessionTokenMaxSize in the AWS/STS namespace. Use them to build dashboards and set alarms. Set your alarm against the size limit you found during testing, not the 4,096-byte maximum. The maximum is the same for every account, so your own infrastructure limit is the one that matters.

    The following figure shows the SessionTokenMaxSize and SessionTokenSize metrics graphed in the CloudWatch console.

    Figure 1: SessionTokenMaxSize and SessionTokenSizemetrics in the CloudWatch console

    Figure 1: SessionTokenMaxSize and SessionTokenSizemetrics in the CloudWatch console

    • In CloudTrail: Each STS session-vending event records SessionTokenUtilization and SessionTokenSize for successful calls.
    {
      "eventName": "AssumeRole",
      "responseElements": {
        "credentials": { "...": "..." },
        "assumedRoleUser": { "...": "..." },
        "packedPolicySize": 61,
        "sessionTokenUtilization": 61,
        "sessionTokenSize": 2532
      }
    }
  3. Use appropriate fields for monitoring session token utilization. AWS STS still returns PackedPolicySize in session-vending API responses and CloudTrail records for backward compatibility. The field now reports the same value as SessionTokenUtilization: the percentage of the 4,096-byte session token size limit consumed by the token. As a result, PackedPolicySize values might appear lower even when your token content has not changed.

    If your SDK exposes SessionTokenUtilization, use that field because its name reflects the value’s current meaning. If an earlier SDK does not expose SessionTokenUtilization, use PackedPolicySize to monitor the same utilization percentage without updating the SDK. We recommend you monitor SessionTokenSize for the token size in bytes.

Conclusion

You now have more room for session tags, tag values, and session policies in your AWS sessions. AWS STS enforces a single 4,096-byte session token limit, returns a clearer error message when a token exceeds it, and reports token size so you can track growth proactively. Validate your token-handling systems with MinimumSessionTokenSize, and watch SessionTokenUtilization and SessionTokenSize for ongoing visibility.

References

If you have feedback about this post, submit comments in the Comments section below.


Rishi Tripathy

Rishi Tripathy

Rishi is a Principal Product Manager on the AWS Identity and Access Management (IAM) team. He focuses on access control mechanisms that help enterprises secure their AWS environments at scale. He is passionate about building security primitives that are straightforward to adopt and hard to misconfigure.

Tanmay Baid

Tanmay Baid

Tanmay is a Senior Software Development Engineer on the AWS Identity and Access Management (IAM) team. He works on the core identity systems behind the credentials and tokens customers rely on to access AWS at massive scale. He enjoys working on the hard problems at the intersection of distributed systems, identity, and security.

  •  

Architecting resilient authentication with Amazon Cognito multi-Region replication

Your consumer identity and access management (CIAM) system is the foundation of your customer experience. It’s how users sign in, access services, and engage with your applications. As your business scales across geographies, ensuring authentication is always available becomes a core architectural requirement. However, building multi-Region authentication has traditionally required complex custom replication solutions that synchronize user data, manage consistency, and handle failover, all adding significant operational overhead. Amazon Cognito simplifies this with multi-Region replication (MRR), which automatically replicates user pools across AWS Regions with near-real-time synchronization, built-in failover, and seamless sign-in, while keeping operational complexity and costs optimized.

In this post, we show you how to prepare your user pool for MRR, provide architectural decisions and reference architectures for business to consumer (B2C), business to business (B2B), and machine to machine (M2M) use cases, and practical guidance on implementing failover strategies.

Amazon Cognito MRR at a glance

Amazon Cognito MRR creates a replica user pool in another AWS Region (a replica Region) that shares the same user pool ID as your primary user pool. The primary user pool (the user pool in your primary Region) remains authoritative, and its configurations (app client IDs, client secrets), user data (attributes, hashed credentials, group memberships), and external identity provider (IdP) settings are replicated to the replica with eventual consistency.

The user pool in the replica Region (replica user pool) supports user authentication operations (such as sign-in, token generation and revocation) and read-only operations towards user pool configurations and user attributes (such as list users and groups and describe user pool configurations). Write operations against user pool configurations and updating user attributes aren’t enabled in the replica user pool and can only be made in the primary user pool. Amazon Cognito returns an Action temporarily unavailable error when using managed login, or an OperationNotEnabledException when using an AWS SDK for those operations. See Supported API operations in secondary Regions for a list of API operations supported in replica Regions.

JSON web tokens (JWTs) and active sessions are interoperable between Regions; for example, a refresh token issued by the primary Region is accepted in the replica Region to retrieve new ID and access tokens.

While this post primarily focuses on MRR architecture patterns and considerations, you can visit the following posts to learn more about MRR basics and the next-generation infrastructure behind it:

Prepare for multi-Region replication

In this section, we show you architectural decisions and preparation work for a successful MRR deployment.

Apply a multi-Region customer managed key

Without MRR enabled, data is encrypted at rest with an AWS owned AWS Key Management Service (AWS KMS) key and encrypted in transit with TLS 1.2 and TLS 1.3 with hybrid post-quantum key exchange. Before enabling MRR, you must configure your user pool to use a customer managed key. This must be a symmetric multi-Region AWS KMS customer managed key.

Architectural considerations for your KMS key:

  • You only need to set up one replica multi-Region key for your customer managed key because Amazon Cognito MRR supports only one additional replica Region.
  • You own the administration of the customer managed key, including key policies, rotation, and deletion. You can also consider a key rotation strategy before enabling MRR or enable automatic key rotation.
  • Follow least-privilege principles in KMS key policy and scope the KMS key to your user pool only. You can do so by applying a condition statement: kms:EncryptionContext:aws:cognito-idp:<userpool-arn>. See the data encryption section in the Amazon Cognito developer guide for a full example key policy.

Choose a multi-Region OIDC issuer

In each ID and access tokens, Amazon Cognito includes a default Issuer claim in the JWT payload, referred as iss, to represent the identity provider that issued the token. The OpenID Connect (OIDC) specification dictates that the iss format must be a URL that uses https scheme and publishes a JSON metadata document about the identity provider available at the <iss>/.well-known/openid-configuration path. The metadata document must also include the JSON Web Key (JWK) document in the <iss>/.well-known/jwks.json path, which contains the signing keys to validate the token signatures for its integrity.

The original issuer type follows the format as https://cognito-idp.<region>.amazonaws.com/<userpool_id>. However, this issuer URL format and the OIDC well-known metadata are regional resources. As part of the MRR capability, Cognito introduces a new multi-Region OIDC issuer type, the updated issuer, and follows the format as https://issuer-cognito-idp.<region>.amazonaws.com/<userpool_id>. This new updated issuer type replaces the original single Region type and maintains availability of the issuer endpoint regardless of the state of primary or replica Region.

Based on the issuer URL format you select, your OpenID Connect discovery endpoint is hosted at  <iss>/.well-known/openid-configuration and your JSON Web Key Set (JWKS) endpoint at <iss>/.well-known/jwks.json. Both original type and updated type are supported with the Amazon Cognito MRR capability. You can change the issuer type at any stage in your MRR journey, and the newly issued tokens, including those generated by refresh tokens, will reflect the most current issuer type configurations.

We recommend adopting the updated issuer type. With the updated issuer type, the OpenID Connect discovery document and JWKS endpoint remain consistent and available regardless of which Region is servicing requests. This means your applications can always fetch signing keys for token verification, even during a regional impairment.

To adopt the updated issuer type, update your applications and downstream dependencies to validate against the new updated iss value. If you use the aws-jwt-verify library, update to v5.2.1 or later that supports updated issuer type. Plan this as a coordinated deployment; existing ID and access tokens with the original issuer type remain valid and accepted by Amazon Cognito endpoints until they expire. When using an existing refresh token to exchange for a new set of ID and access tokens, new tokens always carry the current issuer format configuration at the time of token refresh operation, providing interoperability across two issuer formats.

If you can’t immediately adopt the multi-Region issuer—for example, because downstream services or third-party integrations validate the iss claim against a hard-coded original format pattern—you can enable MRR while continuing to use the original issuer type. However, in this configuration the OIDC discovery endpoint and JWKS endpoint are tied to a single Region and might be unavailable during a regional impairment. Your multi-Region application might not be able to fetch public keys dynamically and validate token signatures. To mitigate this, it’s a good practice to implement a JWKS caching strategy in your token verification layer. Cache the signing keys locally (respecting the Cache-Control headers) so your applications can continue to validate tokens using cached keys when the JWKS endpoint is unreachable. This approach lets you benefit from MRR for user authentication while maintaining token verification continuity until you’re ready to complete the issuer migration. To learn more about the original and updated issuer types, see the Amazon Cognito user pools as an OIDC issuer section of the developer guide.

Configure regional service dependencies

Amazon Cognito user pools support several integrations with AWS services for extended customization functionalities. Those AWS services are regional services and must be configured independently in the replica Region, including:

  • AWS Lambda – Lambda triggers (for example, pre-authentication, pre-token generation, and others) are invoked in different authentication stages and should be deployed in the replica Region and attached to the replica user pool to match customized behaviors in the primary user pool. When deploying Lambda triggers, you can adopt the same logic for both primary and replica user pools and access to downstream resources or set up a different logic to characterize different behaviors when requests are served in the replica Region.
  • AWS WAF – WAF web access control lists (web ACLs) are associated to protect the user pool from unwanted requests. When accepting traffic to the replica user pool, create matching WAF web ACLs in the replica Region.
  • Amazon Simple Notification Service (Amazon SNS) – If you send text messages (for example, SMS-based multi-factor authentication (MFA), passwordless authentication, or SMS notifications), configure Amazon SNS in the replica Region. SNS requires additional set up (origination identities, spending limits) in each Region, and sender ID registration time depends on several factors.
  • Amazon Simple Email Service (Amazon SES) – If you use Amazon SES for email delivery, verify sending domains and email addresses in the replica Region and configure your replica user pool accordingly.
  • Amazon CloudWatch – If you export user activity logs from Amazon Cognito to a CloudWatch log group, or monitor service quotas in CloudWatch, configure alarms and analytics accordingly.

Use infrastructure-as-code tools like AWS CloudFormation or AWS Cloud Development Kit (AWS CDK) to maintain consistent configurations and deployments across Regions and environments. You should also monitor for any configuration drifts between assets.

Consider automatic domain failover

For authentication use cases that rely on managed login and OAuth 2.0 endpoints—including federated authentication and M2M authorization—Amazon Cognito supports automatic failover to the replica Region with an Amazon Route 53 health check. Cognito uses the health status of Route 53 health check to control whether traffic routes to the primary or replica user pool. The health check can be set up to monitor the health of an endpoint, a CloudWatch alarm, or a calculated number of other health checks, so you determine what triggers a healthy or unhealthy state and can adjust traffic routing as needed.

Both the Amazon Cognito prefix domain (for example, auth.us-east-1.amazoncognito.com) and custom domain (for example, auth.example.com) support automatic domain failover. Your domain serves as the single entry point for the user pool OAuth 2.0 endpoints and directs traffic to the managed login pages. Cognito automatically fails over domain traffic to the replica Region when a Route 53 health check becomes unhealthy and fails back to the primary Region when the check is healthy. You don’t need to create another prefix domain in the replica user pool for failover use cases.

With the automatic failover capability, you can use a single domain to serve external IdP configurations, including redirect URIs and SAML assertion consumer URLs. For example, use https://auth.example.com/saml2/logout to send SAML 2.0 sign-out responses. Because the domain can serve traffic to both the primary and replica Regions and remains unchanged across Regions, your external IdP configurations stay consistent across Regions, and existing federated users continue to authenticate without disruption. This means that you can enable MRR without having to contact external IdP admins to update configurations; all existing configurations will continue to work.

For SDK-based authentication use cases without managed login, a custom domain isn’t strictly required. We recommend configuring a custom endpoint for SDK requests to simplify failover orchestration, so you don’t have to modify the Region parameter in the SDK configuration. Behind your custom endpoint, you can use the same Route 53 health check or a custom load balancing strategy to proxy API requests to primary or replica Region endpoints. You might also consider load balancing user authentication traffic, by referring to an X-Amz-Target HTTP header (for example, X-Amz-Target: AWSCognitoIdentityProviderService.InitiateAuth), to both the primary and replica Regions, while keeping user sign-up operations in the primary Region. If you use both managed login and SDK authentication in the same user pool, you can consider using the custom domain as the custom endpoint of the SDK for a streamlined operation, where Route 53 health check initiates failover and failback between the primary and replica Regions.

Plan for TOTP MFA alternatives

Time-Based One-Time Password (TOTP) MFA isn’t supported in replica user pools. Users configured to use TOTP MFA must authenticate through the primary Region. If your application relies on TOTP as a second factor, this limitation requires careful planning because you want to enable an alternative MFA for your users, such as SMS OTP, email OTP, or passkey.

Review quotas

When you activate a replica user pool, you gain a separate set of default quotas in the replica Region. Previously reserved higher quotas for your user pool in the primary Region aren’t carried over to the replica Region.

Data sovereignty

When selecting a replica Region for your user pool, consider your organization’s data sovereignty and residency requirements, as user identity data will be replicated to and stored in that Region. For guidance on navigating compliance, continuity, and control obligations that may influence your Region selection, see Practical digital sovereignty: Navigating the pillars of compliance, continuity, and control.

Reference architectures

In this section, we show you reference architectures for common authentication patterns using the Amazon Cognito MRR capability. Each architecture demonstrates how Cognito MRR works with different authentication use cases.

Managed login and federation

Amazon Cognito managed login provides a fully managed authentication UI that handles sign-in, sign-up, and federation flows. With MRR, managed login endpoints are served from the healthy user pool based on your Route 53 health check configuration. Managed login also includes OAuth 2.0 endpoints and can be used with local Cognito accounts and federated users. Figure 1 depicts a reference architecture for using managed login to authenticate Cognito users.

Figure 1: Cognito MRR reference architecture for managed login and federation use cases

Figure 1: Cognito MRR reference architecture for managed login and federation use cases

When using Amazon Cognito with managed login, the process flow is:

  1. The user visits the application and is redirected to the managed login to begin the authentication flow.
  2. Managed login uses the Route 53 health check to control traffic routing.
  3. If the health check returns a healthy status, all traffic to the managed login flows to the primary Region user pool for user authentication.
  4. For a federated user, the primary Region user pool redirects the user to a federated IdP or social IdP for authentication. After successful authentication, Amazon Cognito creates or updates user attributes depending on whether it’s a new user signing in for first time or an existing user.
  5. If the health check returns an unhealthy status, all traffic to the managed login flows to the replica Region user pool. Cognito users will authenticate against the replica user pool.
  6. The replica Region user pool endpoint redirects federated users to external IdPs. However, any user creation or attribute update against replica user pool will fail until the health check returns healthy and traffic routes back to the primary Region.

M2M architecture

In an M2M architecture, services authenticate using the OAuth 2.0 client credentials grant. This flow doesn’t involve users; instead, backend services exchange client credentials for access tokens.

Figure 2: Cognito MRR reference architecture for machine-to-machine use case

Figure 2: Cognito MRR reference architecture for machine-to-machine use case

The authentication flow is:

  1. Application clients send a POST request to the Amazon Cognito /token endpoint with client credentials.
  2. Managed login uses the Route 53 health check to determine whether traffic should flow to the primary or replica user pool.
  3. If the health check returns a healthy status, traffic to the /token endpoint will flow to the primary Region user pool.
  4. If the health check returns an unhealthy status, traffic to the /token endpoint will flow to the replica Region user pool. After the health check returns to a healthy status, traffic will return to routing to the primary user pool.

SDK-based architecture

For applications that use AWS SDK or Amazon Cognito APIs directly (rather than through managed login), the authentication flow is embedded in your application code. This gives you more control over the user experience but requires additional considerations for failover.

Figure 3: Cognito MRR reference architecture for SDK use cases

Figure 3: Cognito MRR reference architecture for SDK use cases

The process shown in Figure 3 is:

  1. The user visits the application and signs in through a custom UI (using APIs or SDKs).
  2. (Optional) An Amazon Route 53 health check is configured to perform a health check against regional proxy endpoints and determine traffic routing. You can also use a custom health check or your DNS resolver to make traffic routing determinations.
  3. If the health check returns a healthy status, all traffic to the proxy endpoints will flow to the primary Region proxy for user authentication. You can also choose to load balance user authentication traffic across both the primary and backup Regions.
  4. The primary Region Amazon API Gateway proxy forwards user requests to the Amazon Cognito regional endpoint.
  5. If the health check returns an unhealthy status, all traffic will flow to the replica Region proxy.
  6. The replica Region API Gateway proxy begins forwarding user requests to the Amazon Cognito regional endpoint until the health check returns to healthy status.

In an SDK-based architecture, Amazon Cognito regional endpoints can also be called directly. You can also set up custom routing to use replica Region endpoints to load balance user authentication requests by routing read-only requests to both the primary and replica Region endpoints while keeping write requests in the primary Region.

Failover strategies

Now that you’ve set up multi-Region replication with Amazon Cognito, the next step is to test and monitor your multi-Region configuration. In this section, we walk through strategies for monitoring your endpoints, determining when to trigger failover, and testing your failover readiness.

Monitor with Route 53 health checks

Failover for Managed Login and all OAuth 2.0 flows is driven by Amazon Route 53 health checks associated with your Amazon Cognito prefix or custom domain. You’re responsible for what determines the state of this health check. The health check isn’t tied to your DNS CNAME record but is the signal that tells Amazon Cognito whether to route traffic to the primary or replica Region for all managed login endpoints. When the health check fails, Amazon Cognito routes traffic to the replica user pool. When the health check recovers, traffic is restored to the primary user pool.

A practical approach to get started to build a health check:

  1. Create a synthetic canary – Use Amazon CloudWatch Synthetics to run a canary that periodically exercises an actual authentication flow against your primary Region. For example, the canary can perform a client credentials token request against your Amazon Cognito domain’s /oauth2/token endpoint or execute a full AdminInitiateAuth API call with test credentials. This validates that the end-to-end authentication path is functional, not just that an endpoint is responding.
  2. Tie the canary to a CloudWatch alarm – Configure a CloudWatch alarm on the canary’s SuccessPercent CloudWatch metric. Set a threshold that accounts for transient errors (for example, alarm when success drops below 90% for three consecutive evaluation periods).
  3. Connect the alarm to your Route 53 health check (optional) – Create a Route 53 health check that monitors the CloudWatch alarm. When the alarm enters the ALARM state, the health check fails, and Amazon Cognito routes traffic to the replica user pool. If you prefer to rely on human intervention, skip this step and instead configure the CloudWatch alarm alert your operations team to manually invert the health check.

After you have your health check, associate it with your Amazon Cognito domain using the UpdateUserPoolDomain API or the Amazon Cognito console.

Authentication-only compared to full-stack failover

Before implementing failover, consider how your authentication layer relates to the rest of your application stack. There are two common patterns:

  • Authentication-only failover – Your application remains in a single Region, but authentication traffic fails over to the Amazon Cognito replica if only the primary Region’s authentication service is impaired. This works when your application can continue operating with tokens already issued (for example, cached JWTs, active sessions) and when downstream APIs don’t depend on the same Region as your user pool. Consider this option when the rest of your stack has its own availability model.
  • Full-stack failover – Your entire application—compute, data stores, APIs, and authentication—fails over to a replica Region. In this model, Amazon Cognito MRR is one component of a broader multi-Region architecture where authentication flows have tight dependencies on regional resources (such as Lambda triggers calling regional Amazon DynamoDB tables, or post-authentication logic writing to a regional event bus) that must be co-located with the user pool.

Use Amazon Application Recovery Controller (ARC) to coordinate failover across all components with a single action. ARC provides three capabilities that are particularly relevant for multi-Region authentication architectures:

  • Routing controls – Extremely reliable data plane controls that let you shift DNS traffic across Regions, with safety rules that prevent partial or unintended failovers (for example, preventing you from failing over authentication without also failing over the dependent API layer).
  • Readiness checks – Continuous monitoring of resource quotas, capacity, and network routing policies in your secondary Region, so you have confidence that the replica environment—including your Amazon Cognito replica user pool and its regional dependencies—can handle production traffic before you failover.
  • Region switch – Centralized, automated, and observable multi-Region recovery orchestration across multiple AWS accounts and resources, so you can execute a coordinated failover of your Cognito user pool alongside databases, compute, and APIs in a single recovery plan.

ARC is particularly valuable when your Amazon Cognito Lambda triggers, WAF rules, SNS and SES configurations, and downstream services all need to switch Regions in lockstep. Rather than managing failover for each component independently, you can use ARC to define a single recovery group that treats your authentication stack and application stack as one unit. To learn more about the capabilities and use cases of ARC, see Introducing Amazon Route 53 Application Recovery Controller.

The right choice depends on your recovery scope. Map the dependencies in your authentication flow: if your Lambda triggers call regional DynamoDB tables or your post-authentication logic writes to a regional event bus, those tight couplings point to full-stack failover. If your application validates tokens independently and doesn’t make real-time calls back to Amazon Cognito after token issuance, authentication-only failover keeps both your blast radius and operational overhead smaller.

Determine when to failover

Triggering failover too aggressively risks unnecessary disruptions; too conservatively risks a drop in desired availability. Here are the factors to balance:

  • Monitor authentication flow health – Validate that critical flows are functioning, including managed login endpoint availability and token endpoint responses.
  • Use composite health checks – Combine multiple signals. For example, require both the managed login and token endpoints to be healthy.
  • Set appropriate thresholds – Configure failure thresholds (for example, three consecutive failures) to distinguish transient errors from genuine impairments.
  • Consider downstream dependencies – Factor in Lambda triggers, external IdPs, and other regional services.
  • Client side retry logic – For SDK-based single-page application (SPA) architectures, consider implementing client-side retry logic with Region failover. When the primary Region is unavailable, your application should detect the failure and redirect authentication of API calls to the replica Region’s Amazon Cognito endpoint.

Understanding and determining the recovery time objective (RTO) and recovery point objective (RPO) should also be the key factor in determining when and why to failover. See the Establishing RPO and RTO Targets for Cloud Applications blog post to learn more.

Test failover readiness

If using Route 53 health check, start by manually inverting your Route 53 health check during a maintenance window. In the Route 53 console, enable Invert health check status to force the health check into a failed state; this triggers failover to the replica Region without requiring any infrastructure changes. While traffic is routing to the replica Region, validate that your critical authentication flows (sign-in, token refresh, federation) work correctly, then disable the inversion to restore traffic to the primary. This test confirms your end-to-end failover path is functional.

When you’re confident in the basic failover path, graduate to more realistic failure simulations with AWS Fault Injection Service (FIS). Create FIS experiment templates that disrupt your primary Region’s Amazon Cognito dependencies; for example, block network access to a dependent resource or inject latency into downstream API calls. Use FIS stop conditions (guardrails) to automatically halt experiments if unexpected impacts are detected. These experiments validate not just that failover triggers correctly, but that your replica Region handles real authentication load under degraded conditions.

We recommend conducting failover tests on a predefined and regular cadence and after any significant changes to your authentication architecture. Document your runbooks and make sure your operations team is familiar with both the failover and recovery procedures.

Conclusion

In this post, we built on the foundational knowledge of the Amazon Cognito MRR capability and showed you how to architect resilient authentication for real-world use cases:

  • Preparation considerations – Multi-Region KMS keys, OIDC issuer transitions, regional dependencies, and TOTP MFA considerations
  • Reference architectures – B2C, B2B, and M2M patterns using managed login, plus SDK-based approaches
  • Failover strategies – Route 53 health checks, ARC integration, and testing with health check inversion and AWS FIS

To get started, make sure your user pool is on the Essentials or Plus feature plan, configure your multi-Region KMS key and OIDC issuer, and create your first replica. For step-by-step setup instructions, see Multi-Region replication for user pools

If you have feedback or thoughts about this post, submit comments below. If you have questions, start a new thread on Amazon Cognito re:Post or contact AWS Support.


Abrom-Douglas-author

Abrom Douglas III

Abrom is a Senior Solutions Architect within AWS Identity with over 20 years of software engineering and security experience, specializing in identity and access management. He loves speaking with customers about how identity and access management can provide secure outcomes that enable both business and technology initiatives. In his free time, he enjoys cheering for Arsenal FC, photography, travel, volunteering, and competing in duathlons.

Edward Sun

Edward Sun

Edward is a Senior Security Specialist Solutions Architect focused on identity and access management. He loves helping customers throughout their cloud transformation journey with architecture design, security best practices, migration, and cost optimizations. Outside of work, Edward enjoys hiking, golfing, and cheering for his alma mater, the Georgia Bulldogs.

  •  

The state of AI for security: Measuring what matters most for building trust

Security teams are starting to actively use AI for security work, including vulnerability triage, penetration testing, threat modeling, incident response, and code review. The promise is speed, but a security tool that moves fast and raises too many false alarms doesn’t save time. Engineers spend time on false alarms, on-call is noisier, and teams distrust findings that matter.

Today, we’re releasing Deception Benchmark, the first benchmark designed to measure that trust problem directly. It tests whether a model can distinguish real vulnerabilities from code that looks risky but is actually safe. The benchmark includes 14,822 samples across 16 languages and more than 70 Common Weakness Enumeration (CWE) categories. We evaluated 12 models from five providers and are releasing the dataset and whitepaper to the community. Existing benchmarks measure whether AI can find or exploit vulnerabilities. This is the first to measure whether it can tell real vulnerabilities from false alarms. Under standard prompting, precision at distinguishing real vulnerabilities from false alarms landed in the mid 50s; as likely to be inaccurate as accurate.

In offensive tasks, there’s often a clear result: the exploit works or it doesn’t. Defensive reviews are harder to verify than offensive tasks; a model might recognize a suspicious pattern even when a mitigation makes the issue non-exploitable. In practice, useful systems need to reason about the code, the mitigation, and sometimes the surrounding environment.

The measurement gap

The community has made progress on security evaluations. CyberGym tests agents on more than 1,500 realistic tasks. Meta’s CyberSecEval and CyberSecEval 2 measure exploit generation. CYBENCH evaluates capture the flag (CTF) challenges. SEC-Bench and VulnBench push toward authentic security workflows.

Recent work reinforces both the progress and the gap. ExploitGym measures whether AI can escalate from a crash to a working exploit. Microsoft’s Project Perception deploys multi-agent red/blue/green teams for continuous defense. OpenAI’s GPT-Red shows that self-play red-teaming finds novel attacks that frontier models can’t defend against. Since then, OpenAI disclosed that its GPT-6 Astra model crossed the Critical cybersecurity capability threshold, and both OpenAI and Anthropic reported incidents where models gained unauthorized access to production systems during evaluations. The offensive side is moving fast. But none of this work measures the defensive precision question: when an AI system flags code as vulnerable, how often is it right?

Introducing Deception Benchmark

14,822 samples, 16 languages, and more than 70 CWE categories. We call it Deception Benchmark because the safe samples are designed to deceive models. It has real vulnerability patterns, real frameworks, real idioms, with mitigations that quietly close the exploit path. The goal is to classify code as vulnerable or safe, with no hints.

Consider a Flask endpoint that accepts user input and queries a database. A model will pattern-match to SQL injection, but the query uses parameterized statements, so the exploit path is closed. A single-turn classifier flags the pattern and moves on, never checking whether the exploit can actually work. Production tools rely on multi-step loops and agentic workflows to compensate, but that scaffolding masks whether the model itself understands the code. This benchmark strips the scaffolding away and asks the model to make the call in a single pass, so what it measures is understanding, not how many tries a harness takes to get there.

We built every sample through an adversarial loop: generate, test against frontier models, harden, repeat. If a model gets it right easily, the sample doesn’t survive. The result is a benchmark calibrated to the frontier, not below it. Building it this way is expensive. Generation and hardening of the samples consumed tens of billions of tokens. We’re releasing the result so the community doesn’t have to repeat that cost.

This benchmark generates two challenge types. Code-level challenges (6,988 samples) present vulnerable and safe variants that differ by a subtle fix. Both look suspicious, only one is exploitable. Environment-gated challenges (2,707 samples) go further: same code, different deployment context. A Kubernetes Network Policy blocks the server-side request forgery (SSRF) path. An identity and access management boundary prevents privilege escalation. The pattern is visible in the source. The infrastructure makes it unexploitable. The model has to figure out which scenario applies.

All samples were purpose-built for this benchmark, grounded in real-world patterns, real frameworks, real CWEs, and real infrastructure; without IP concerns or training data contamination.

Large-scale quality data with LLMs and humans in the loop

Generating reliable labels at this scale is difficult: a single pass—by people or by models—leaves errors that skew scores. So we treat labeling as a convergent audit loop rather than a one-time step. Every label is re-examined by multiple independent reviewers, blind to one another and to the original reasoning that produced the label. Disagreements escalate to direct adjudication, where the original reasoning is evaluated against the challenge. Unresolved cases go to human review. We repeat the loop until the scored set converges below a dispute threshold: under 3 percent of samples still contested by independent review, with a target of under 1 percent surviving human adjudication. One choice makes this defensible: we never relabel a disputed sample. When reviewers disagree, the sample moves to the unscored pool instead of being given a corrected label, so a bad challenge can remove a sample but can never introduce a wrong label into the scored set.

A human review of 100 randomly drawn scored samples found no label errors. We describe the full process in the whitepaper.

The results

The benchmark is roughly balanced: half vulnerable, half safe, so a random classifier scores 50 percent. We report two error rates separately, because they fail in opposite directions. The false positive rate (FPR) is how often the model flags safe code as vulnerable. These are the false alarms that waste an engineer’s time. The false negative rate (FNR) is how often it misses a real vulnerability and calls it safe. Accuracy alone hides this: a model that labels everything vulnerable catches every bug (0 percent FNR) but flags all safe code (100 percent FPR) and still scores about 50 percent. We consider FPR below 10 percent and FNR below 10 percent the minimum bar for production use.

Figure 1: FPR compared to FNR for 12 models across two prompting strategies. No model reaches the generous bar

Figure 1: FPR compared to FNR for 12 models across two prompting strategies. No model reaches the generous bar.

Model

Prompt

Accuracy

FPR

FNR

GPT-5.6 Sol Direct 54.9% 92.5% 0.9%
GPT-5.6 Sol PoE 58.9% 58.6% 23.1%
GPT-5.5 Direct 56.9% 87.8% 1.3%
GPT-5.5 PoE 62.9% 63.6% 12.4%
GPT-5.4 Direct 60.2% 81.0% 1.5%
GPT-5.4 PoE 77.7% 10.1% 33.6%
Llama 3.3 70B Direct 58.8% 84.2% 1.1%
Llama 3.3 70B PoE 72.2% 10.2% 44.2%
Claude Haiku 4.5 Direct 55.6% 92.1% 0.0%
Claude Haiku 4.5 PoE 75.6% 22.4% 26.3%
Claude Opus 4.6 Direct 55.9% 91.3% 0.1%
Claude Opus 4.6 PoE 75.8% 42.7% 7.0%
Claude Opus 4.7 Direct 58.3% 85.5% 0.9%
Claude Opus 4.7 PoE 75.9% 32.0% 16.8%
Claude Opus 4.8 Direct 53.8% 95.7% 0.2%
Claude Opus 4.8 PoE 75.8% 32.5% 16.4%
Claude Opus 5 Direct 77.3% 41.5% 5.2%
Claude Opus 5 PoE 79.3% 24.9% 16.8%
Claude Sonnet 5 Direct 62.9% 74.7% 2.2%
Claude Sonnet 5 PoE 74.7% 31.8% 19.2%
Amazon Nova 2 Lite Direct 56.3% 89.2% 1.2%
Amazon Nova 2 Lite PoE 70.1% 45.2% 15.5%
Mistral Large Direct 52.2% 99.0% 0.0%
Mistral Large PoE 65.5% 49.3% 20.6%

Among the general-purpose frontier models tested, no configuration achieves both FPR and FNR less than 10 percent on this benchmark.

Every model has the same failure mode. With direct prompting, they catch up to 95 percent of real vulnerabilities but also flag 41–99 percent of safe code. Precision runs from 52 percent to 71 percent, clustered in the mid-50s; effectively as likely to be inaccurate as accurate. The models see a vulnerability pattern and stop reasoning. Proof-of-exploit prompting cuts false positives by 17–74 points but misses 7–44 percent of real vulnerabilities. The environment-gated challenges are worse: models flag the code and ignore the Kubernetes Network Policy next to it. No tested configuration keeps both false positives and false negatives below 10 percent.

These results reflect general-purpose models in single-turn prompting. Purpose-built systems with multi-step validation and tool use are a different operating point that we didn’t measure, and if a harness can close the gap between pattern recognition and genuine understanding, this benchmark is the place to demonstrate it. Two cautions before assuming it already does. Agentic verification is proven mostly on offensive tasks, where success can be confirmed: the exploit fires or it doesn’t. Judging that code is safe has no such oracle. Extra iterations re-sample the same judgment rather than confirm a negative, and a harness still inherits the base model’s understanding. If the model can’t separate an effective mitigation from an ineffective one in a single pass, more passes won’t add the missing knowledge. That’s what this benchmark measures: the model’s intrinsic ability to understand code, tested at the single-turn baseline where no scaffolding can mask the gap.

For security teams evaluating AI tools today: ask your vendors how their system performs on tasks like this, not just whether it finds vulnerabilities, but how often it’s wrong. Pair any AI-assisted review with human verification on high-risk code paths, and use Deception Benchmark to hold your tools accountable.

Availability

We built Deception Benchmark to simplify measuring this problem in a reproducible way. The public release includes the samples and evaluation workflow. We don’t release the labels, so submissions can be scored consistently over time without turning the benchmark into a memorization exercise.

Of the 14,822 samples, 9,695 are scored; the remaining 5,127 are held out and unscored, mixed in with the rest of the benchmark. The goal is straightforward: make it more difficult to optimize the benchmark compared to improving the underlying system. We describe that design in more detail in the whitepaper.

Deception Benchmark is available on GitHub, along with the whitepaper and submission instructions for verified scoring. If you’re building security tooling, you can download the dataset, run your system against the benchmark, and submit predictions for scored evaluation.

If you have feedback about this post, submit comments in the Comments section below.


Anshumali-Shrivastava

Anshumali Shrivastava

Anshumali is an Amazon Scholar and Full Professor of Computer Science at Rice University. His research on dynamic sparsity, sketching, and hashing pioneered techniques now central to efficient LLM training and inference. A two-time founder — ThirdAI (acquired by ServiceNow) and XMAD.ai (acquired by Workato) — he bridges theoretical computer science and practical AI systems at scale.

Neha Rungta

Neha Rungta

Neha is a scientist and builder who has spent her career making machines reason about complex systems at scale. Her work spans automated reasoning, formal verification, security, and AI, shaping systems including Cedar, IAM Access Analyzer, and Continuum. Today, she is forging the next generation of machine reasoning, combining LLMs, formal methods, and agentic systems.

  •  

Incident response guide for AWS CloudTrail investigations – Part 2

In Part 1 of this guide, we examined two common incident scenarios: cross-account Amazon Simple Storage Service (Amazon S3) data deletion with ransomware implications, and cryptocurrency mining deployed through AWS CloudFormation using exposed AWS Management Console credentials. We also introduced key incident response terminology and investigative frameworks for analyzing AWS CloudTrail events.

In this second part, we explore a more complex, multi-stage attack: how a web application vulnerability can cascade into credential harvesting and unauthorized access to Amazon Bedrock services across multiple AWS Regions. We also cover additional investigation techniques and hardening steps to strengthen your security posture.

Scenario 3: SSRF to IMDSv1 credential harvesting with multi-Region Amazon Bedrock service misuse

This scenario examines how a web application vulnerability can cascade into a multi-Region event targeting Amazon Bedrock services. The investigation demonstrates how threat actors chain together multiple techniques, using Amazon Elastic Compute Cloud (Amazon EC2) Instance Metadata Service version 1 (IMDSv1) through server-side request forgery (SSRF) and cross-Region pivoting to access Amazon Bedrock.

Your security team receives multiple alerts: failed AWS Identity and Access Management (IAM) operations in the us-east-1 Region, successful console sign-ins without multi-factor authentication (MFA), and unusual Amazon Bedrock API calls from us-east-2. Initially, these might seem like unrelated events across different services and Regions. However, as our Security Incident Response Team (SIRT) discovered, they represent a carefully orchestrated event chain that began with a web application vulnerability and culminated in unauthorized access to your organization’s AI infrastructure.

Architecture and progression

The architecture in figure 1 maps a multi-stage attack that exploits the trust relationship between Amazon Elastic Compute Cloud (Amazon EC2) instances and AWS services. A threat actor identified a server-side request forgery (SSRF) vulnerability in a web application running on an EC2 instance that had an attached webdev IAM role. Rather than attempting to escalate privileges directly, the threat actor used this foothold to reach the Instance Metadata Service version 1 (IMDSv1) endpoint and retrieve the temporary credentials issued to the webdev role. Because IMDSv1 returns credentials in response to a basic request with no session token, an SSRF flaw is enough to harvest them, which is why these credentials became the pivot point for everything that followed. The attack unfolded in five stages. Each stage is numbered in figure 1 so you can follow the progression from the initial web request through to the cross-Region Amazon Bedrock activity:

  1. Initial access: The threat actor exploited the SSRF vulnerability in the web application to make server-side requests on the instance’s behalf.
  2. Credential harvesting: Those requests reached the IMDSv1 endpoint and returned the temporary credentials for the webdev role.
  3. Permission testing: Using the harvested credentials, the threat actor attempted IAM operations to probe the boundaries of what the role could do.
  4. Service pivoting: When IAM actions were denied, the threat actor shifted focus to Amazon Bedrock, a service the role could reach.
  5. Region hopping: The threat actor moved operations from us-east-1 to us-east-2, likely to evade Region-specific monitoring and access controls.
Figure 1: Scenario 3 architecture

Figure 1: Scenario 3 architecture

CloudTrail evidence and structured extractions

In this section, we walk through the CloudTrail evidence that documents the attack from start to finish. Each of the four events that follow maps to one or more stages in the progression described previously, and together they trace how the threat actor moved from harvested credentials to active misuse of Amazon Bedrock. For each event, we present the relevant portion of the CloudTrail log record, highlight the fields that matter most for the investigation, and include a forensic legend that explains what each highlighted field reveals.

We cover the following events:

  1. Permission boundary testing (15:53:49 UTC): A failed CreateUser call in us-east-1 that reveals the compromised role and the IMDSv1 credential source.
  2. Console access establishment (15:59:29 UTC): A successful console sign-in without MFA, showing the pivot from programmatic to interactive access.
  3. Bedrock service reconnaissance (17:20:00 UTC): A ListFoundationModels call in us-east-2 that marks the Region hop and the shift to AI services.
  4. Active model exploitation (17:25:48 UTC): A Converse call that invokes the Amazon Nova Pro model, confirming unauthorized usage.

As you read each event, focus on how the fields connect one stage to the next. The same webdev role, the same source IP address, and the recurring ec2RoleDelivery value are the threads that tie these otherwise separate events into a single attack chain.

Event 1: Permission boundary testing (15:53:49 UTC): The first suspicious activity appeared as a failed CreateUser API call in us-east-1. The CloudTrail log records an AssumedRole session attempting to create an IAM user named adm1n but receiving an AccessDenied error. The webdev role is visible in the userIdentity field, readOnly is false (indicating a write operation attempt), and the user-agent shows AWS Command Line Interface (AWS CLI) on Windows, suggesting programmatic access from the harvested credentials.

{
  "eventVersion": "1.11",
  "userIdentity": {
    "type": "AssumedRole",
    "principalId": "AROAFINDANEXAMPLE:i-0123456789abcdef0",
    "arn": "arn:aws:sts::XXXXXXXXXXXX:assumed-role/webdev/i-0123456789abcdef0",
    "sessionContext": {
      "sessionIssuer": { "type": "Role", "userName": "webdev" },  ◄── ❶ Compromised EC2 role
                                                     ‾‾‾‾‾‾‾‾
      "attributes": { "mfaAuthenticated": "false" }
    },
    "ec2RoleDelivery": "1.0"  ◄── ❷ IMDSv1 confirmed (SSRF exploitation path)
                       ‾‾‾‾‾
  },
  "eventTime": "2025-09-22T15:53:49Z",
  "eventSource": "iam.amazonaws.com",
  "readOnly": false,
  "eventName": "CreateUser",  ◄── ❸ Intent: establish persistent backdoor
               ‾‾‾‾‾‾‾‾‾‾‾‾
  "userAgent": "aws-cli/2.17.48 ua/2.0 os/windows#10 ...",
  "errorCode": "AccessDenied",  ◄── ❺ Hard policy stop (least-privilege held)
               ‾‾‾‾‾‾‾‾‾‾‾‾‾‾
  "errorMessage": "User: arn:aws:sts::XXXXXXXXXXXX:assumed-role/webdev/...
    is not authorized to perform: iam:CreateUser
    on resource: arn:aws:iam::XXXXXXXXXXXX:user/adm1n..."  ◄── ❹ Lookalike name (1 not i)
                                                ‾‾‾‾‾
}

───────────────────────────────────────────────────────────────────
  FORENSIC LEGEND
───────────────────────────────────────────────────────────────────
  ❶ userName: "webdev"         → Confirms the compromised EC2 role context
  ❷ ec2RoleDelivery: "1.0"    → Credentials obtained via IMDSv1 (SSRF vector)
  ❸ eventName: "CreateUser"   → Attacker attempting IAM persistence
  ❹ target user: "adm1n"      → Typosquatting admin (number 1 instead of letter i)
  ❺ errorCode: "AccessDenied" → Attacker probing permission boundaries; blocked
───────────────────────────────────────────────────────────────────

Event 2: Console access establishment (15:59:29 UTC): Six minutes later, the threat actor successfully signed in to the AWS Management Console using the same credentials. The ConsoleLogin event records that MFA wasn’t used (MFAUsed: No), and the source IP (75.3.231.105) provides attribution data. The user agent indicates Chrome browser on Windows 10.

{
  "eventVersion": "1.11",
  "userIdentity": {
    "type": "AssumedRole",
    "principalId": "AROAFINDANEXAMPLE:i-0123456789abcdef0",
    "arn": "arn:aws:sts::XXXXXXXXXXXX:assumed-role/webdev/i-0123456789abcdef0",
    "sessionContext": { "attributes": { "mfaAuthenticated": "false" } }
  },
  "eventTime": "2025-09-22T15:59:29Z",
  "eventSource": "signin.amazonaws.com",
  "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 	 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36 Edg/140.0.0.0",

  "eventName": "ConsoleLogin",  ◄── ❶ Pivoted to interactive console access
               ‾‾‾‾‾‾‾‾‾‾‾‾‾‾
  "awsRegion": "us-east-1",
  "sourceIPAddress": "75.3.231.105",
  "responseElements": { "ConsoleLogin": "Success" },◄── ❷ Hijacked login succeeded
                                        ‾‾‾‾‾‾‾‾‾
  "additionalEventData": { "MobileVersion": "No", "MFAUsed": "No" },◄── ❸ No MFA challenge
                                                             ‾‾‾‾
  "eventType": "AwsConsoleSignIn"  ◄── ❹ Console sign-in (not API call)
               ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
}
───────────────────────────────────────────────────────────────────
FORENSIC LEGEND
───────────────────────────────────────────────────────────────────
❶ eventName: "ConsoleLogin"→ Attacker pivoted from programmatic to visual console access
❷ ConsoleLogin: "Success"→ Hijacked login successfully authenticated
❸ MFAUsed: "No" → Critical gap: no MFA enforced, enabling the pivot
❹ eventType: "AwsConsoleSignIn"   → Distinguishes this from basic API calls
───────────────────────────────────────────────────────────────────

Event 3: Amazon Bedrock service reconnaissance (17:20:00 UTC): Nearly two hours later, the threat actor pivoted to Amazon Bedrock, making a ListFoundationModels API call in us-east-2. This event exhibits several patterns: a Region change from us-east-1 to us-east-2 (potential defense evasion), a shift from IAM to AI services, readOnly: true (reconnaissance rather than modification), and sessionCredentialFromConsole: “true”, which ties the call to the console session established in Event 2 rather than a fresh IMDSv1 credential retrieval.

 {
  "eventVersion": "1.11",
  "userIdentity": {
    "type": "AssumedRole",
    "arn": "arn:aws:sts::XXXXXXXXXXXX:assumed-role/webdev/i-0123456789abcdef0"
  },
  "eventTime": "2025-09-22T17:20:00Z",
  "eventSource": "bedrock.amazonaws.com",  ◄── ❶ Pivoted to cloud AI services
               ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
  "eventName": "ListFoundationModels",  ◄── ❷ AI model reconnaissance
               ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
  "awsRegion": "us-east-2",  ◄── ❸ Region hop (evasion technique)
               ‾‾‾‾‾‾‾‾‾‾‾
  "sourceIPAddress": "75.3.231.105",
  "readOnly": true,
  "tlsDetails": {
    "clientProvidedHostHeader": "bedrock.us-east-2.amazonaws.com"  ◄── ❹ Intentional alternate region targeting
                               ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
  },
  "sessionCredentialFromConsole": "true"
}

───────────────────────────────────────────────────────────────────
  FORENSIC LEGEND
───────────────────────────────────────────────────────────────────
❶ eventSource: "bedrock.amazonaws.com"→ Attacker pivoted from IAM to managed AI services
❷ eventName: "ListFoundationModels"→ Reconnaissance: enumerating available AI models
❸ awsRegion: "us-east-2"→ Region hop from us-east-1 (defense evasion)
❹ clientProvidedHostHeader: "bedrock.us-east-2..."  → Confirms intentional targeting of alternate region endpoint
───────────────────────────────────────────────────────────────────

Event 4: Active model exploitation (17:25:48 UTC): Five minutes after the reconnaissance call, the threat actor moved from enumeration to active exploitation, invoking the Amazon Nova Pro model through the Converse API in us-east-2. The additionalEventData field quantifies the unauthorized usage at 944 input tokens and 126 output tokens, confirming that the threat actor successfully prompted the model and received a response.

 {
  "eventVersion": "1.11",
  "userIdentity": {
    "type": "AssumedRole",
    "arn": "arn:aws:sts::XXXXXXXXXXXX:assumed-role/webdev/i-0123456789abcdef0"
  },
  "eventTime": "2025-09-22T17:25:48Z",
  "eventSource": "bedrock.amazonaws.com",
  "eventName": "Converse",  ◄── ❶ Active model invocation (recon → exploitation)
               ‾‾‾‾‾‾‾‾‾‾
  "awsRegion": "us-east-2",
  "requestParameters": {
    "modelId": "amazon.nova-pro-v1:0",  ◄── ❷ Specific model being misused
               ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
    "inferenceConfig": { "maxTokens": 1024 }
  },
  "responseElements": null,
  "additionalEventData": { "inputTokens": 944, "outputTokens": 126 }  ◄── ❸ Unauthorized usage quantified
                           ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
}

───────────────────────────────────────────────────────────────────
  FORENSIC LEGEND
───────────────────────────────────────────────────────────────────
❶ eventName: "Converse"                → Attacker transitioned from reconnaissance to active exploitation
❷ modelId: "amazon.nova-pro-v1:0"      → Identifies the specific foundation model being misused
❸ inputTokens: 944, outputTokens: 126  → Quantifies unauthorized usage (financial cost + data exfiltration exposure)
───────────────────────────────────────────────────────────────────

Notable event fields to track

As you review the event logs, a handful of fields do most of the investigative work in this scenario. Understanding what each one reveals, and why it matters, is what turns a collection of individual log records into a coherent attack narrative.

The userIdentity field is the starting point for attribution. In this scenario it carries the EC2 instance ID as the session name, which is what let us trace the harvested credentials back to a specific compromised instance rather than a human user. Whenever you see an assumed-role session, this field answers the first question of any investigation: whose credentials are these, and where did they come from?

The readOnly field reveals the intent behind an action. A value of true marks reconnaissance, such as the ListFoundationModels call the threat actor used to enumerate available models, while false marks an attempt to change or use something, such as the CreateUser call or the Converse invocation. Sorting events by this field quickly separates the threat actor’s information gathering from the actions that caused actual impact.

The awsRegion field is easy to overlook, but in this scenario it exposed the threat actor’s evasion strategy. The shift from us-east-1 to us-east-2 wasn’t incidental; threat actors move between Regions because monitoring, alerting, and access controls are often configured inconsistently across them. Watching this field helps you spot activity that has deliberately moved away from where your detection is strongest.

Finally, the userIdentity.invokedBy field identifies when an AWS service, rather than a user or a set of harvested credentials, made the request on your behalf. CloudTrail populates it only when the caller is an AWS service, such as through a service-linked role, a service role, or a forward access session. It doesn’t appear in the events for this scenario because the threat actor called Amazon Bedrock directly with the harvested webdev credentials. That absence is itself informative: it confirms the requests came from a principal acting on its own rather than from a legitimate service-driven workflow. As agent-based and service-integrated Amazon Bedrock workloads become more common, checking this field separates expected service activity from credentials driven directly by a threat actor.

Investigation priorities

With the full attack chain mapped, from SSRF through credential harvesting to Amazon Bedrock service misuse, the investigation turned to a harder question: what did each stage actually cost us, and what would stop it from happening again? A few priorities shaped that work.

The first was figuring out where the credentials came from and how far the exposure reached. It was clear the threat actor had valid credentials for the webdev role, but the more useful question was why a web application role could reach Amazon Bedrock at all. The customer confirmed there was no business reason for it, so we needed to understand whether that permission was a deliberate misconfiguration or an oversight, and then look for other EC2 instances carrying the same role attachment. One compromised instance is an incident; a fleet of instances with the same over-scoped role is a much bigger problem waiting to happen.

Next, we wanted to know what the threat actor did after they got into Amazon Bedrock. Reconnaissance and active use carry very different consequences, so we traced which foundation models were touched and whether any were actually invoked or only enumerated. That distinction matters for scoping the damage, and it signals whether data exfiltration is a concern. Unusual model usage, unexpected prompt volume, or output patterns that don’t match any legitimate workload are the signals that reconnaissance has turned into something worse.

The Region hop was its own line of inquiry. The move from us-east-1 to us-east-2 was almost certainly deliberate, and the investigation focused on understanding what the threat actor gained by it. In practice, that meant comparing the two Regions: were the monitoring and access controls in us-east-2 weaker than in us-east-1, and what else did the threat actor reach in the secondary Region once they were there? Inconsistent controls across Regions are one of the most common ways activity slips past detection.

Tying it all together was the timeline, which shows how quickly the threat actor moved through the chain:

  1. 15:53:49: Failed IAM operation (us-east-1)
  2. 15:59:29: Successful console login (us-east-1)
  3. 17:20:00: Amazon Bedrock reconnaissance (us-east-2)
  4. 17:25:48: Active model invocation (Converse call) (us-east-2)

Following the credentials across those events fills in the rest of the story. IMDSv1 handed the threat actor temporary credentials for the webdev role, and the same role appears in every event that followed, which confirms the credentials were reused rather than replaced. Nowhere in that sequence was MFA required, and that single gap is what let one harvested credential stay useful across two hours, two Regions, and two very different services.

Incident response checklist

The following checklist captures the actions needed to contain the incident, remediate the vulnerability, and assess the scope of unauthorized AI service usage. Each item names where to look and what a finding looks like, so the checklist stays usable under the time pressure of a live incident.

  1. Contain and remediate the entry point:
    1. Identify the specific web application feature that made the outbound request (URL fetchers, webhook callbacks, PDF or image renderers, and link-preview generators are the usual culprits), then confirm it can reach http://169.254.169.254.
    2. Audit the rest of the application for the same pattern, because one unvalidated URL parameter usually means others exist.
    3. Enforce IMDSv2 on the affected instance and across the fleet with aws ec2 modify-instance-metadata-options --http-tokens required --http-put-response-hop-limit 1. Setting --http-tokens required means credentials are only returned when the caller presents a session token it obtained through a PUT request, which a basic SSRF cannot do. Setting the hop limit to 1 keeps the metadata response on the instance itself, so a request coming from a container or proxy an extra hop away never receives it.
  2. Scope the Bedrock usage:
    1. List the foundation models the webdev role could reach by reviewing its IAM policy and any resource-based policies, so you know the full set of models that were exposed, not only the one that was invoked.
    2. Determine what was sent to and returned by the model. CloudTrail records the Converse call and the token counts, but only Amazon Bedrock model invocation logging captures the input prompts and model responses. If it was enabled, pull the log entries for the session; if it wasn’t, note that the prompt and response content can’t be recovered and enable it now.
    3. Flag any compliance exposure based on what those prompts and responses contained. Unauthorized processing of regulated data (such as personally identifiable information (PII), protected health information (PHI), or cardholder data) through the model might trigger notification obligations.
  3. Check for wider compromise and persistence:
    1. Query CloudTrail across all Regions and services—not only Amazon Bedrock—for every event tied to the webdev role’s session, to confirm what else the same credentials touched.
    2. Correlate the CloudTrail timestamps with VPC Flow Logs and application logs for source IP 75.3.231.105 to build the network-level picture around each API call.
    3. Search for IAM write events from the session (CreateUser, CreateRole, CreateAccessKey, and AttachRolePolicy) that indicate an attempt to establish persistence beyond the temporary credentials. The failed adm1n CreateUser call is the known starting point; confirm nothing similar succeeded.
  4. Watch for ongoing or hidden impact:
    1. Review Amazon Bedrock usage in CloudWatch and your billing data for invocation spikes or unexpected token consumption that fall outside normal workload patterns.
    2. Inspect the invocation logs for signs of sensitive data being processed or extracted through the model.
    3. Check the same logs for prompt injection attempts, where the input tries to override the model’s instructions or extract system prompts.

Key takeaways

This scenario reveals how a single application vulnerability can cascade into broad unauthorized access when multiple security controls are missing. The following takeaways highlight the key defensive gaps and hardening priorities.

  • Least-privilege IAM for workload roles: The webdev role’s access to Amazon Bedrock across multiple Regions had no business justification for a web application workload, which the customer confirmed during the investigation. Apply least-privilege principles to EC2 instance roles by scoping permissions to only the services and actions the application requires. Use AWS IAM Access Analyzer to identify unused permissions and tighten policies proactively. Overly permissive roles transform a single application vulnerability into broad lateral movement across unrelated services.
  • IMDSv1 compared to IMDSv2: Organizations must immediately switch to IMDSv2 and disable IMDSv1 across their entire cloud infrastructure. The ec2RoleDelivery: “1.0" field in the logs explicitly confirms the use of IMDSv1, which permits credential retrieval without an authentication token. This architectural weakness makes SSRF-based credential theft trivial, because a web application flaw that can make an outbound request is enough to read the role’s temporary credentials with no further authentication. Transitioning to IMDSv2 mitigates this attack surface by enforcing local, session-based tokens, effectively breaking the threat actor’s exploitation chain. In this scenario, IMDSv2 alone would have stopped the attack at its first step.
  • Region-based defense evasion signals a deliberate operator: The shift from us-east-1 to us-east-2 for Amazon Bedrock access wasn’t incidental. Threat actors move between Regions because monitoring, alerting, and access controls are often configured inconsistently across them, and activity in a secondary Region is more likely to go unnoticed. This kind of cross-Region movement is a marker of operational security awareness rather than opportunistic access, and it should raise the priority of an investigation. Treat consistent detection coverage across all Regions, including the ones you do not actively use, as a baseline requirement.
  • Interface switching and permission probing reveal the threat actor’s method: This event chain reveals a threat actor comfortable moving between AWS interfaces and testing boundaries before committing. The failed CreateUser attempt was systematic probing to understand the scope of the harvested credentials, and when IAM actions were denied, the threat actor pivoted to a service the role could actually reach. The combination of programmatic access through the AWS CLI and interactive console access demonstrates the same adaptability. Recognizing this pattern of probe, adapt, and pivot helps responders anticipate the next move instead of reacting to each event in isolation.
  • AI services need visibility beyond CloudTrail: Amazon Bedrock and other AI services are high-value targets, and CloudTrail alone doesn’t capture the whole story. CloudTrail records who called Amazon Bedrock and whether the call succeeded, but not what was asked or answered. Enable Amazon Bedrock model invocation logging to capture full prompts and responses for compliance auditing. For agent-based workloads, Amazon Bedrock AgentCore Observability, built on AWS Distro for OpenTelemetry (ADOT), provides session-level traces showing tool execution order and latency. Consider also enabling Amazon GuardDuty AI Protection, which analyzes Amazon Bedrock-related CloudTrail activity to detect anomalous invocations, cost harvesting, and prompt injection attempts. Correlating these signals—CloudTrail, Model Invocation Logging, and agent telemetry—gives investigators the complete picture. For implementation guidance, see Monitoring and Auditing AI Workloads on AWS.

Advanced forensic indicators and evasion techniques

Beyond the specific attack patterns in this scenario, investigators should be aware of several evasion techniques that threat actors use to confuse defenders and blend into legitimate activity. The top three that we observe across incident response with customers are:

  • Root user compared to IAM user named root: When you first create an AWS account, you begin with a single sign-in identity that has complete access to all AWS services and resources in the account. This identity is called the AWS account root user. In some previous investigations, threat actors have also created IAM users in an AWS account named root. The difference is visible in the type field of the userIdentity element of the CloudTrail log record, which indicates the type of user that logged the record.
  • Role and user name imitation: Threat actors attempt defense evasion by creating names for IAM users and roles that imitate those reserved for use by AWS. For example, the service-linked role AWSServiceRoleForSupport is a unique IAM role linked directly to AWS Support. Threat actors have created roles with the name AWSServiceRoIeforSupport (note the use of an upper-case letter I instead of a lower-case letter l in Role) in an attempt to trick users into thinking actions taken by this role have been performed by AWS Support.
  • Users named HIDDEN_DUE_TO_SECURITY_REASONS: The userName field contains the string HIDDEN_DUE_TO_SECURITY_REASONS when the recorded event is a console sign-in failure caused by incorrect user name input. CloudTrail doesn’t record the contents in this case because the text could contain sensitive information. However, threat actors have used this string as an actual username to trick investigators into thinking the name has been obfuscated. This technique is usually associated with a corresponding CreateUser or CreateRole CloudTrail event.

Conclusion and next steps

CloudTrail event fields help security teams identify identities with unintended access, track threat actor actions, and remediate affected resources. Understanding fields like userIdentity, eventName, and sourceIPAddress improves incident investigation and threat detection. Implementing best practices such as enabling comprehensive logging, using Amazon Athena for analysis, securing logs, and automating responses helps ensure that CloudTrail serves as a robust forensic and incident response tool.

If you suspect unauthorized activity in your AWS environment, AWS Security Incident Response is available to help. The service continuously monitors and triages findings from Amazon GuardDuty and third-party security tools integrated through AWS Security Hub, automatically filtering alerts to surface the most relevant events. In addition to proactive triage, customers can initiate security cases through the service. You can choose to handle these cases internally or receive support from the Security Incident Response Team (SIRT), a dedicated group of security experts available at all times to assist with investigation, containment, and recovery throughout the incident lifecycle.

Additional resources

The following resources provide further guidance on securing your AWS environment and strengthening your investigative capabilities.

If you have feedback about this post, submit comments in the Comments section below.


Oscar Diaz

Oscar E Diaz Cordovez

Oscar is a Senior Technical Account Manager specializing in cloud operations and security. His passion for technology and innovation drives his expertise in cloud-focused architectures, DevOps practices, and automation.

Steve de Vera

Steve de Vera

Steve is a manager for the AWS Security Incident Response service with a focus on threat research and threat intelligence. He is passionate about American-style BBQ and is a certified competition BBQ judge. He has a dog named Brisket.

Jennifer Paz

Jennifer is a Security Engineer Manager with over a decade of experience, for the AWS Security Incident Response service. Jennifer enjoys helping customers tackle security challenges and implementing complex solutions to enhance their security posture. When not at work, Jennifer is an avid runner, pickleball enthusiast, traveler, and foodie, always on the hunt for new culinary adventures.

  •  

Incident response guide for AWS CloudTrail investigations – Part 1

AWS CloudTrail logs contain the evidence you need when investigating suspicious activity in your AWS environment, but knowing which fields matter and how to interpret them can mean the difference between surface-level analysis and uncovering the full scope of an incident. This guide walks you through real-world scenarios, showing you how to analyze CloudTrail events to uncover cross-account unauthorized access, cryptocurrency mining operations, and AI service abuse. You’ll learn the investigative techniques our Security Incident Response Team (SIRT) team uses to handle threats, with practical methodologies you can apply to your own investigations.

Each scenario includes:

  • Architecture diagrams showing the event progression
  • Annotated CloudTrail logs highlighting significant fields
  • Investigation frameworks with specific questions to ask
  • Lessons learned and preventive measures

Whether you’re in security operations, cloud engineering, compliance, or leadership, this guide provides the investigative mindset needed to move beyond basic CloudTrail queries to comprehensive security analysis.

Incident response definitions

Throughout this guide, we reference terminology commonly used in incident response and threat intelligence. We’ve provided definitions for key terms to help ensure this guide is accessible to readers from diverse backgrounds, whether you’re in security operations, cloud engineering, compliance, or leadership.

  • Reconnaissance: The initial phase where a threat actor gathers information about the target environment (for example, listing Amazon Simple Storage Service (Amazon S3) buckets or browsing available resources) to understand what’s available before taking action.
  • Enumeration: Systematically cataloging specific resources, users, or configurations within an environment to identify potential targets or access paths.
  • Lateral movement: When a threat actor moves from one resource to another within the same environment (for example, pivoting from an Amazon Elastic Compute Cloud (Amazon EC2) instance to an AI service) to expand their access.
  • Privilege escalation: Attempting to gain higher-level permissions than initially obtained, such as trying to create admin users or modify AWS Identity and Access Management (IAM) policies.
  • Defense evasion: Techniques used to avoid detection, such as operating in a different AWS Region where monitoring might be less robust, or naming unauthorized resources to look legitimate.
  • Persistence: Establishing ongoing access to an environment (for example, creating new IAM users or access keys) so the threat actor can return even if the original entry point is closed.
  • Credential harvesting: Stealing authentication credentials (passwords, access keys, temporary tokens) to impersonate legitimate users or roles.
  • Server-side request forgery (SSRF): A web application technique where an unauthorized user tricks a server into making requests on their behalf, often used to access internal services such as the Amazon EC2 Instance Metadata Service (IMDS) endpoint. For more information, see Understanding SSRF.
  • IMDSv1 (Instance Metadata Service v1): Amazon EC2 Instance Metadata Service version 1 (IMDSv1) provides temporary credentials to applications running on an instance. IMDSv1 itself isn’t inherently insecure; however, when an application with issues (for example, one susceptible to SSRF) is running on the instance, an unauthorized user can use that application to reach the metadata endpoint and retrieve credentials. IMDSv2 mitigates this risk by requiring session-based authentication tokens.
  • Indicators of compromise (IOCs): Observable artifacts (IP addresses, user agents, session names, resource names) that suggest unauthorized activity has occurred.
  • Exfiltration: The unauthorized transfer of data out of an environment, such as copying files before deleting them.
  • Event chain: The sequence of steps a threat actor follows from initial access to achieving their objective, where each step enables the next.
  • Pivot: Shifting from one technique, service, or Region to another during a security event, often after an initial approach is blocked or to avoid detection.

Scenario 1: Cross-account S3 data deletion with ransomware implications

Cross-account access is sometimes necessary in AWS, but misconfiguration creates security risks. In this scenario, your security operations center has received an automated alert that multiple objects have been deleted from the customer-important-data S3 bucket. The initial response seems straightforward: check the CloudTrail logs, identify who deleted the objects, and determine if it was authorized. But as our SIRT team investigated further, what appeared to be a straightforward unauthorized deletion revealed itself as a cross-account incident with ransomware implications. CloudTrail analysis requires recognizing patterns, understanding context, and thinking like a threat actor.

Scenario architecture

Figure 1 shows the architecture layout for accessing a trusted account and deleting objects from an S3 bucket, which is achieved through the following steps:

  1. Threat actor assumes the CrossAccountS3Access role from a trusted account.
  2. Lists S3 buckets to identify targets (ListBuckets API call).
  3. Lists objects within the target bucket to catalog contents.
  4. Executes scripted deletions of three files within 13 seconds.
  5. Each deletion returns an HTTP 204 (successful) status code.
Figure 1: Scenario 1 architecture

Figure 1: Scenario 1 architecture

Reconnaissance phase

Our investigation began with examining the CloudTrail logs, where we discovered that the unauthorized activity started with what many analysts might dismiss as routine activity: a ListBuckets API call made through an assumed role at 14:31:22 UTC. The CloudTrail entry contains a session named dev-migration-script using the CrossAccountS3Access role.

While cross-account access is common in enterprise environments, session names typically reflect legitimate business units. Attackers frequently use masquerading techniques, naming their sessions after common developer tasks or automation scripts, to blend seamlessly into daily operational noise. However, cross-referencing this session name against the external source IP and historical deployment logs confirmed that no such migration project was authorized, signaling a clear evasion attempt by a threat actor and the first indication of unauthorized access. Three seconds later, our logs showed a GET request to list objects in the bucket, which is classic reconnaissance behavior. The threat actor was cataloging available targets, using the same assumed role and IP address. This pattern, which you can see in the arn and eventname in the following log, showed us that the threat actor gathered intelligence, assessed targets, and planned their approach.

{
  "eventVersion": "1.08",
  "userIdentity": {
    "type": "AssumedRole",
    "principalId": "AROAEXAMPLE123456789:threat-actor-session",
    "arn": "arn:aws:sts::111122223333:assumed-role/CrossAccountS3Access/threat-actor-session"
  },
  "eventTime": "2025-01-20T14:31:22Z",
  "eventSource": "s3.amazonaws.com",
  "eventName": "ListBuckets",
  "sourceIPAddress": "203.0.113.47",
  "recipientAccountId": "444455556666"
}

──────────────────────────────────────────────────────────────────────────────────
  FORENSIC LEGEND
──────────────────────────────────────────────────────────────────────────────────
❶ arn: ".../CrossAccountS3Access/..."             → Cross-account role assumed; access came from another account
❷ Session name: "threat-actor-session"            → Custom session name attached at role assumption
❸ eventName: "ListBuckets"                        → Enumeration of all S3 buckets in the account (recon)
❹ principalId: "AROAEXAMPLE123456789:threat-actor-session" → Role's unique ID + attacker-chosen session label
❺ sourceIPAddress: "203.0.113.47"                 → Origin of the API call (RFC 5737 documentation IP range)
❻ recipientAccountId: "444455556666"              → AWS account that received/owned the request (fictional placeholder)
──────────────────────────────────────────────────────────────────────────────────

Systematic deletion

After completing their reconnaissance at 14:31:25 UTC, the threat actor went silent for 14 minutes before the first deletion at 14:45:12 UTC. During this window, the threat actor likely reviewed the inventory of objects they’d just enumerated, selected their highest-value targets (financial data, PII, and database backups), and prepared an automated deletion script to execute quickly once ready. We can infer this preparation period based on several factors: no other CloudTrail events from this session appeared during the 14-minute window, the subsequent deletions were precisely timed at 6-7 second intervals suggesting scripted execution, and the targets chosen were the three most business-critical files rather than a bulk delete of everything in the bucket. This selective, scripted approach indicates the threat actor used the reconnaissance data they gathered in the listing phase to build a targeted attack plan before executing it. Within 13 seconds (14:45:12–14:45:25 UTC), the threat actor deleted three files from the customer-important-data bucket: a financial report (q4-2024.xlsx at 14:45:12), a customer personally identifiable information (PII) database (pii-database.csv at 14:45:18), and a production database backup (prod-database-backup.sql at 14:45:25). Each deletion returned an HTTP 204 status code. The Amazon S3 access logs confirm these successful DELETE operations, all originating from the same session.

[20/Jan/2025:14:31:25] S3Access/dev-migration-script REST.GET.BUCKET    -  "GET /?list-type=2 HTTP/1.1" 200 - "-" "aws-cli/Linux"
                                ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾                                                ‾‾‾‾‾‾‾‾‾‾‾‾‾
                                ❶ Masquerading session   ❷ Recon: listing buckets                                        ❻ Scripted tool

[20/Jan/2025:14:34:15] S3Access/dev-migration-script REST.COPY.OBJECT   financial-reports/q4-2024.xlsx "PUT /..." 200 - "-" "aws-cli/Linux"
                                                     ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾   ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
                                                     ❸ Data exfiltration  Financial data copied

[20/Jan/2025:14:34:20] S3Access/dev-migration-script REST.COPY.OBJECT   customer-data/pii-database.csv "PUT /..." 200 - "-" "aws-cli/Linux"
                                                     ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾   ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
                                                     ❸ Data exfiltration  PII database copied

[20/Jan/2025:14:45:12] S3Access/dev-migration-script REST.DELETE.OBJECT financial-reports/q4-2024.xlsx "DELETE /..." 204 - "-" "aws-cli/Linux"
            ‾‾‾‾‾‾‾‾‾                               ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾               ‾‾‾
            ❺ 13-sec window starts                   ❹ Destruction        Target file                                   Success

[20/Jan/2025:14:45:18] S3Access/dev-migration-script REST.DELETE.OBJECT customer-data/pii-database.csv "DELETE /..." 204 - "-" "aws-cli/Linux"
                                                     ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾               ‾‾‾
                                                     ❹ Destruction        PII database destroyed                        Success

[20/Jan/2025:14:45:25] S3Access/dev-migration-script REST.DELETE.OBJECT backup-configs/prod-database-backup.sql "DELETE /..." 204 - "-" "aws-cli/Linux"
            ‾‾‾‾‾‾‾‾‾                               ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾         ‾‾‾
            ❺ 13-sec window ends                     ❹ Destruction        Prod backup destroyed (anti-recovery)         Success

───────────────────────────────────────────────────────────────────────────────────────────
  FORENSIC LEGEND
───────────────────────────────────────────────────────────────────────────────────────────
  ❶ "dev-migration-script"    → Masquerading session name; no authorized migration existed
  ❷ REST.GET.BUCKET           → Reconnaissance: cataloging available targets
  ❸ REST.COPY.OBJECT          → Data exfiltration before destruction (steal-then-destroy)
  ❹ REST.DELETE.OBJECT        → Systematic destruction of high-value assets
  ❺ 14:45:12 → 14:45:25      → 13-second automated deletion window (scripted execution)
  ❻ "aws-cli/Linux"           → CLI-based automation, not manual console activity
───────────────────────────────────────────────────────────────────────────────────────────

Analysis of timing and access patterns

The 13-second deletion window wasn’t arbitrary. The user-agent string showed AWS Command Line Interface (AWS CLI) usage on Linux, and the precise timing suggested scripted execution rather than manual operations. This indicated preplanned targeting and automated execution to minimize the detection window.

The consistent source IP across events let us search for other suspicious activities from the same source, correlate with threat intelligence feeds, and identify potential lateral movement attempts.

The broad Amazon S3 permissions of the CrossAccountS3Access role raised questions about least privilege implementation, regular access reviews, and the business justification for such extensive cross-account permissions.

Investigation priorities

With confirmation that misconfigured cross-account access had been taken advantage of to delete data, the next step was to prioritize the investigation. In incident response, priority is driven by three factors: whether the threat actor still has active access (containment urgency), whether sensitive data was exposed or exfiltrated (regulatory and business impact), and whether the attack can spread to other resources or accounts (blast radius). We applied these factors to guide the following questions:

  • How did the threat actor gain access to the CrossAccountS3Access role? We examined the role’s trust policy and recent modifications, authentication events in both the trusting and trusted accounts, and other sessions using the same role around the same timeframe.
  • Were the files copied before deletion? We searched for GetObject operations on the same objects before the deletions, unusual network traffic patterns during the reconnaissance phase, and CopyObject activities that might indicate data theft.
  • Did the objects have specific significance? Understanding why these specific objects mattered helped us prioritize recovery efforts based on business impact, assess regulatory notification requirements for the PII exposure, and determine the full scope of business disruption from backup loss.

Response checklist

After identifying the scope of the cross-account deletion, the following steps help ensure a thorough response and prevent recurrence.

  • Determine if the business purpose served by this cross-account access is legitimate
  • Identify the corresponding authentication events that show how the role was assumed
  • Identify other AWS resources that this role might access beyond Amazon S3
  • Check for failed attempts or reconnaissance activities that preceded the successful event
  • Determine when this cross-account trust relationship was created
  • Determine when the last access review of this role was conducted
  • Locate any backup copies of the deleted data
  • Determine detection rules that can be used to catch similar activity in the future

Key takeaways

This scenario illustrates several principles that apply broadly to cross-account incident investigations. Unusual identifiers in session names often reveal threat actor intent or poor operational security. The progression from ListBuckets to targeted deletions shows how threat actors operate with a plan. Cross-account access needs extra scrutiny because trusted relationships become vectors for unauthorized access when credentials are exposed. Understanding why specific files matter helps prioritize response efforts and assess true impact. Precise timing and consistent technical signatures often indicate scripted events that need different response strategies than manual intrusions.

Scenario 2: Cryptocurrency mining using CloudFormation with console credentials

In this scenario, your finance team notices an unexpected spike in AWS costs, particularly around Amazon EC2 compute charges in the us-east-1 AWS Region. During the investigation, we examine how threat actors use legitimate console access to deploy cryptocurrency mining operations through AWS CloudFormation and how investigators can uncover the scope of resource hijacking events. We discover a CloudFormation stack named CRYPTO which you have no record or knowledge of being created. The stack contains EC2 instances running in your production Amazon Virtual Private Cloud (Amazon VPC) consuming significant compute resources, which signals an immediate security investigation.

Architecture and sequence

Figure 2 shows how the threat actor moved from credential acquisition to active mining, following these steps:

  1. Threat actor obtains console credentials (username and password without multi-factor authentication (MFA)).
  2. Accesses AWS Management Console.
  3. Creates CloudFormation stack CRYPTO in us-east-1.
  4. Stack deploys EC2 instances configured for cryptocurrency mining in a public subnet.
  5. Mining instances begin consuming compute resources.
Figure 2: Scenario 3 architecture

Figure 2: Scenario 3 architecture

The following is the redacted CloudTrail event record for the unauthorized CreateStack action. See if you can use it to find the following information:

  • The name of the CloudFormation stack that was created
  • The CloudFormation stack Amazon Resource Name (ARN)
  • If the credentials were secured by MFA
  • If the threat actor used the AWS Management Console to perform the actions, or if they were performed programmatically using the AWS CLI or a script
{
  "eventVersion": "1.11",
  "userIdentity": {
    "type": "AssumedRole",
    "principalId": "AROAFINDANEXAMPLE:Participant",
    "arn": "arn:aws:sts::XXXXXXXXXXXX:assumed-role/WSParticipantRole/Participant",
    "sessionContext": {
      "sessionIssuer": { "type": "Role", "userName": "WSParticipantRole" },
      "attributes": { "mfaAuthenticated": "false" }  ◄── ❹ No MFA on session
                                            ‾‾‾‾‾‾‾
    }
  },
  "eventTime": "2025-09-23T18:07:12Z",
  "eventSource": "cloudformation.amazonaws.com",
  "eventName": "CreateStack",
               ‾‾‾‾‾‾‾‾‾‾‾‾‾
  "awsRegion": "us-east-1",
  "userAgent": "aws-cli/2.30.0 ... exec-env/CloudShell",  ◄── ❻ Browser-based CloudShell execution
                                    ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
  "requestParameters": {
    "stackName": "CRYPTO",  ◄── ❶ Cryptocurrency-related activity
                 ‾‾‾‾‾‾‾‾
    "parameters": [
      { "parameterKey": "VpcId" },      ◄── ❷ Prior recon: attacker knew target network
      { "parameterKey": "SubnetIds" }   ◄── ❷
                       ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
    ]
  },
  "responseElements": {
    "stackId": "arn:aws:cloudformation:us-east-1:...:stack/CRYPTO/2102e190..."  ◄── ❸ Stack created successfully
               ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
  },
  "sessionCredentialFromConsole": "true"  ◄── ❺ Console-based access
                                 ‾‾‾‾‾‾
}

───────────────────────────────────────────────────────────────────────────────────────────
  FORENSIC LEGEND
───────────────────────────────────────────────────────────────────────────────────────────
❶ stackName: "CRYPTO"                → Indicator of cryptocurrency mining deployment
❷ VpcId + SubnetIds parameters       → Attacker targeted specific network; prior recon confirmed
❸ stackId: "...stack/CRYPTO/..."     → Unique resource ID; stack was successfully created
❹ mfaAuthenticated: "false"          → Session lacked multi-factor authentication
❺ sessionCredentialFromConsole: true  → Access via AWS Console web portal (not external API)
❻ exec-env/CloudShell                → CLI commands executed via browser-based CloudShell
───────────────────────────────────────────────────────────────────────────────────────────

Analysis of authentication and access

The CloudTrail event record includes fields that answer the questions for this scenario.

Stack details summary:

The CloudTrail event confirms the following details about the deployed stack:

  • Stack name: CRYPTO is an indicator of cryptocurrency-related activity
  • Stack ARN: arn:aws:cloudformation:us-east-1:stack/CRYPTO/2102e190-98a8-11f0-bcea-1209335b107
  • Region: us-east-1 (a common choice for threat actors because of immediate service availability)

Authentication and session context analysis:

Examining the session metadata reveals how the threat actor authenticated and accessed the environment:

  • MFA status: “mfaAuthenticated": “false” indicates that the session was entirely unauthenticated by MFA.
  • Access method: “sessionCredentialFromConsole": “true” means that access was funneled through the console.
  • User context: AssumedRole using WSParticipantRole. Session creation occurred at 2025-09-23T18:06:22Z (approximately 50 seconds before stack creation).

Advanced forensic insight (the CloudShell pivot)

The sessionCredentialFromConsole: true field is important to note because this access originated from the AWS console rather than external programmatic API keys. Interestingly, while the session originated from the console, the userAgent field reveals the execution environment was exec-env/CloudShell. This shows that the threat actor didn’t manually click through the CloudFormation user interface, instead launching AWS CloudShell on sign-in to execute a prepackaged deployment script. This allowed the threat actor to achieve automated speed while evading traditional static API key monitoring. The mfaAuthenticated: false field represents a security control gap. Particularly in environments handling sensitive data or production workloads, MFA must be enforced for console access.

Investigation priorities

With the unauthorized stack confirmed, the investigation focused on understanding the full timeline and blast radius. We approached this in three phases, each building on the findings of the previous one.

Reconstruct the console session timeline

The session began at 18:06:22Z and the stack was created at 18:07:12Z, only 50 seconds later. That speed tells us the threat actor came prepared with a script rather than exploring the environment manually. But we needed to know what happened before and after. By filtering CloudTrail for the same session token across the full session duration, we could identify whether the threat actor performed any reconnaissance before deploying the stack, whether they accessed other services or regions during the same session, and whether they attempted to establish persistence (such as creating IAM users or access keys) before or after the mining deployment. Any actions taken outside the CloudFormation deployment could indicate secondary objectives beyond cryptomining.

Examine what the stack actually deployed

The stack name alone doesn’t tell us the full impact. We needed to inspect the CloudFormation template to understand what resources were created and how they were configured. This meant identifying the EC2 instance types (larger instances mean higher costs and potentially more mining output), reviewing the security group rules to determine what network access these instances had to internal resources, checking whether the template included custom AMIs or user data scripts that pulled mining software on boot, and determining if the stack created its own IAM roles with permissions that could be used for further lateral movement. The template itself is evidence. If it was hosted in Amazon S3, the upload event tells us when the threat actor first staged their tools.

Calculate business impact and determine blast radius

Finally, we needed to quantify the damage and determine whether this was isolated or part of a broader compromise. We calculated the total compute cost by multiplying instance hours by instance type pricing, checked whether the mining instances had network paths to production databases or internal services, examined outbound traffic logs for connections to known mining pool IP addresses, and searched for similar stacks or naming patterns across other regions and accounts. The presence of outbound connections to anything other than mining pools would suggest the instances served a dual purpose, potentially exfiltrating data while generating cryptocurrency.

Response checklist

The following checklist captures the key actions needed to contain the incident, assess its impact, and close security gaps.

  • Determine why MFA wasn’t required for this sensitive operation
  • Investigate how the threat actor obtained valid console credentials
  • Check for failed sign-in attempts preceding this successful access
  • Check for other activities that occurred during this console session
  • Look for resources that were created by the CloudFormation stack
  • Determine how long those resources have been running and consuming costs
  • Look for other similarly named or suspicious stacks in the environment
  • Check what network access these instances have to internal resources
  • Determine what outbound connections these instances are making
  • Look for cryptocurrency mining pool connections
  • Check if IAM users or roles were created
  • Check if additional access keys were generated
  • Determine if the threat actor modified existing permissions or policies

Key takeaways

This scenario highlights how credential hygiene and monitoring controls intersect with resource hijacking threats.

  • MFA enforcement prevents console-based credential abuse for IAM users. The absence of MFA enabled the full sequence. Console access to production environments should require multi-factor authentication as a security best practice.
  • Resource naming can be an indicator. The obvious CRYPTO naming suggests either threat actor confidence or poor operational security, both concerning for different reasons.
  • Cost monitoring is security monitoring. Unusual billing spikes can be early indicators of resource hijacking events.
  • Console-based activity has different patterns than programmatic activity and requires specialized investigation approaches. The sessionCredentialFromConsole field is your starting point for distinguishing between the two.

Conclusion

In this first part, we walked through two real-world scenarios that demonstrate how CloudTrail analysis can reveal the full scope of a security incident. In Scenario 1, we showed how a seemingly routine cross-account role assumption led to targeted data deletion with ransomware implications, and how session names, timing patterns, and source IP correlation help investigators piece together the event chain. In Scenario 2, we examined how stolen console credentials enabled a cryptocurrency mining deployment through CloudShell, highlighting the critical role of MFA enforcement and cost monitoring as security controls. Both scenarios reinforce a core principle: effective CloudTrail investigation goes beyond identifying what happened. It requires understanding how and why, so you can contain the immediate threat and close the gaps that enabled it.

In Part 2 of this guide, we examine how a web application vulnerability can cascade into a multi-Region event targeting AI services, chaining together SSRF, IMDSv1 credential harvesting, and cross-Region pivoting to access Amazon Bedrock. We also cover critical investigation techniques including root user compared to IAM user named root, role name imitation tactics, and the HIDDEN_DUE_TO_SECURITY_REASONS username trick, along with critical hardening steps and additional resources you can use to strengthen your cloud forensic capabilities.

If you have feedback about this post, submit comments in the Comments section below.


Oscar Diaz

Oscar E Diaz Cordovez

Oscar is a Senior Technical Account Manager specializing in cloud operations and security. His passion for technology and innovation drives his expertise in cloud-focused architectures, DevOps practices, and automation.

Steve de Vera

Steve de Vera

Steve is a manager for the AWS Security Incident Response service with a focus on threat research and threat intelligence. He is passionate about American-style BBQ and is a certified competition BBQ judge. He has a dog named Brisket.

Jennifer Paz

Jennifer is a Security Engineer Manager with over a decade of experience, for the AWS Security Incident Response service. Jennifer enjoys helping customers tackle security challenges and implementing complex solutions to enhance their security posture. When not at work, Jennifer is an avid runner, pickleball enthusiast, traveler, and foodie, always on the hunt for new culinary adventures.

  •  

Managing identity source transition for AWS IAM Identity Center

October 20, 2026: This post was republished to reflect the new migration capability from Active Directory to an external identity provider (IdP) or the Identity Center directory, now available in IAM Identity Center.

September 2, 2026: This post was republished to include Active Directory migration strategies and automation for permission sets.


AWS IAM Identity Center manages user access to Amazon Web Services (AWS) resources, including both AWS accounts and applications. You can use IAM Identity Center to create and manage user identities within the Identity Center identity store or to connect to other identity sources.

Organizations that are switching identity providers (IdPs), consolidating identity infrastructure, adopting new single sign-on capabilities, or migrating off legacy on-premises systems may need to change the identity source in IAM Identity Center. Previously, migrations from Active Directory to another identity source deleted users, groups, memberships, and assignments at the moment of cutover. Now, migrations away from Active Directory preserve your users, groups, memberships, and assignments, just as transitions between an external IdP and the Identity Center directory already do. Only switching to Active Directory still removes users, groups, and assignments. Because the change applies to the entire Identity Center instance at once, test validation and a dress rehearsal are strongly recommended.

This post walks you through the migration process and provides sample code to assist the transition. It includes architectural guidance and a step-by-step runbook for switching from Active Directory to Okta. This guide helps AWS administrators, identity engineers, and project teams switch identity sources safely.

Note: This walkthrough uses Okta as a representative example of a SAML 2.0 identity provider. The inclusion of Okta in this example does not constitute an endorsement of any specific identity provider.

Identity source options in IAM Identity Center

The identity source is the definitive system of record: it determines which directory holds the primary copy of user identities and group memberships used for AWS access.

Each IAM Identity Center instance connects to one identity source at a time. IAM Identity Center supports three main identity sources:

  • Identity Center directory – The default identity store. Users and groups are created and managed directly within IAM Identity Center, with no dependency on an external provider. Suitable for organizations without an existing enterprise directory or those wanting a setup that uses only AWS services.
  • Active Directory – Integration with an on-premises Microsoft Active Directory or AWS Managed Microsoft AD through AWS Directory Service for Microsoft Active Directory. This lets organizations reuse their existing AD identities, group memberships, and access policies.
  • External IdP – Integration with third-party providers that support SAML 2.0, including Okta Universal Directory, Microsoft Entra ID (formerly Azure AD), Ping Identity, and others. This is the approach covered in the migration runbook that follows.

For example, migrating from Active Directory to an external IdP carries your existing users and groups over to the new identity source. IAM Identity Center keeps the same identity store, user IDs, and group IDs, and SCIM updates only the external identifier (externalId) to reflect the new source.

How access flows to AWS resources

IAM Identity Center grants access to AWS resources in two ways, as shown in Figure 1:

  • Permission set assignments – Users or groups are assigned permission sets within specific AWS accounts. This enables assume-role access, and AWS Identity and Access Management (IAM) roles (named AWSReservedSSO_...) are automatically created in each account to back these assignments.
  • Application assignments – Users or groups are granted access to AWS-managed applications (such as Amazon SageMaker Studio, Amazon OpenSearch Service and so on) and customer-managed applications integrated with IAM Identity Center.
Figure 1: Granting access to AWS resources for users and groups managed by an identity source in IAM Identity Center

Figure 1: Granting access to AWS resources for users and groups managed by an identity source in IAM Identity Center

When you change the identity source, the downstream impact on these assignments depends heavily on which sources you’re switching from and to. In some transitions, assignments remain intact.

When you switch the identity source from Active Directory to an external IdP or to the Identity Center directory, users, groups, user and group IDs, group memberships, and their associated assignments are preserved. Users keep their access to AWS accounts and applications after the new identity source takes over. Switching to Active Directory from any source removes users, groups, and assignments at the moment of cutover, so pre-migration backup and post-migration restoration are still essential in that scenario.

What changes when switching identity source in IAM Identity Center

Migrations away from Active Directory, and transitions between an external IdP and the Identity Center directory, now preserve users, groups, memberships, and assignments. Transitions to Active Directory removes users, groups, and assignments permanently from IAM Identity Center at the moment of cutover, as shown in the following table. For more information, see Considerations for changing your identity source.

IAM Identity Center item Status upon switch from AD to external IdP or Identity Center directory Status upon switch from external IdP or Identity Center directory to AD
Users and groups in Identity Center Preserved (same user and group IDs; the externalId changes as the new source takes over) Deleted (must be re-provisioned from AD)
Account assignments Preserved Deleted (must be recreated)
Application assignments Preserved* Deleted (must be recreated)

Note: *Test each application in a non-production environment before you migrate. Also inventory any assignments that live inside an application itself, because those sit outside IAM Identity Center.

Before you start: common blockers

  • Matching attribute – Confirm which attribute your target IdP uses to find an existing user (sometimes called matching precedence or existing resource lookup). The IdP must take over each existing Identity Center user and group rather than create a duplicate. Test this in a non-production environment.
  • Required attributes – Every user must have a username, first name, last name, and display name. If any required attribute is missing, that user silently fails to provision. If you send email, it must be unique and single-valued.
  • Username and NameID – Each user’s IdP userName must exactly match the existing Identity Center username, and the SAML NameID must map to the same value. Decide your sign-in identifier (for example, a legacy sAMAccountName or a UPN) before cutover.
  • Groups – Confirm that your IdP supports group provisioning through SCIM (for example, Google Workspace provisions users only). Group names must be unique, and renaming a group can cause it to be treated as new, which strands its assignments. Depending on the external Identity provider, users who belong to a group only through a nested group are not provisioned and lose group-based access. Inventory nested groups and directly assign the child groups that contain those users before cutover.
  • Emergency access – Set up a break-glass administrative access path that does not depend on AWS Identity Center.

Sample deployment

The following five-step summary (expanded into seven phases in the runbook that follows) applies to an identity source transition in IAM Identity Center from Active Directory to Okta, focusing on permission set assignments. The AD-to-Okta runbook in the next section expands each of these steps with specific instructions and commands. To assist users, you can also use the Identity Center migration tool script, which provides a structured workflow for Identity Center migrations with commands for precheck, cutover, validation, and cleanup. Because assignments are now preserved, the restore command is used only for manual rollback. The sample scripts and detailed steps are available on GitHub.

Note: This solution is available in the GitHub aws-samples repository. You can report bugs or make feature requests through GitHub Issues. The builders of this solution can help with GitHub issues. Enterprise Support customers can reach out to their Technical Account Manager (TAM) for further questions or feature requests.

Prerequisites

Before you start this walkthrough, make sure you have the following prerequisites in place:

  • Python 3.10 or later
  • AWS credentials configured using environment variables, an AWS configuration file, or a named AWS Command Line Interface (AWS CLI) profile (specified with --profile)
  • An active IAM Identity Center instance in the target account
  • Okta Global Admin access
  • Install the migration tool as described in the README file

Basic solution walkthrough

The following steps summarize the migration process. The next section expands each step with detailed Okta-specific instructions and commands.

Step 1: Back up users, groups, and assignments – Before making any changes, export your current identity data such as users, groups, and all assignments. This backup protects you if something goes wrong during migration. Store them in a secure, access-controlled location. The runbook includes a script that exports all current IAM Identity Center assignments and principals to a CSV file, then validates referential integrity.

Step 2: Prepare and validate users and groups in the target identity source – Before switching the IAM Identity Center identity source, confirm that your users and groups exist in the new authoritative source. Verify usernames, display names, email addresses, group names, group memberships and other required attributes. Each user’s userName in the new source must exactly match the existing Identity Center username, because this is how the new source usually takes over the user. In a test environment, confirm that an existing user is matched, not duplicated.

Step 3: Switch IAM Identity Center to the new identity source, and configure sign-in – Update IAM Identity Center to point to the new identity source. For external IdPs, upload the SAML metadata and configure the Assertion Consumer Service (ACS) and issuer URLs in your IdP. Before enabling SCIM (System for Cross-domain Identity Management), confirm that a pilot user can sign in.

WARNING: The identity source change applies to the entire Identity Center instance. After you confirm it, Active Directory sync and authentication stop, and users can’t sign in until SAML sign-in works with the new IdP. Users, groups, memberships, and assignments are preserved.

Step 4: Enable SCIM provisioning and verify takeover – Enable automatic provisioning and assign users and groups to the IAM Identity Center application in your IdP. The IdP takes over each existing user and group. Then run the validate command to compare the live assignments against your backup and find any missing or extra assignments.

If drift is found, the script writes a drift_report.csv file with MISSING and EXTRA entries.

Step 5: Validate access – Test access for a sample of users across different roles and permission levels before declaring the migration complete. Confirm that users can authenticate through the new identity source, access their expected AWS accounts, and assume the correct IAM roles. Confirm that the identity source shows your external IdP, that users and groups keep the same IDs, and that no duplicates were created.

Detailed runbook for Okta migration

This runbook covers one of the most common identity source transitions: migrating from an Active Directory to Okta as a SAML 2.0 identity provider.

The sample deployment describes what happens during a transition. It walks through seven phases: stakeholder alignment and pre-migration inventory, pre-cutover validation, the identity source switch itself, takeover verification, post-migration testing, ongoing cleanup and monitoring, and a documented manual rollback procedure for when things don’t go to plan.

If you’re migrating to Active Directory, which still removes users, groups, memberships and assignments, you can adapt the backup and restore scripts from this guide.

Phase 1: Plan your pre-migration activities

Pre-migration planning has four components: stakeholder engagement, inventory, scheduling, and access verification. Complete all four before you schedule the cutover.

1.1 – Document relevant stakeholders:

Fill out this table with actual contact information. You’ll need to reach these people during the cutover window.

Stakeholder Org Name Email Phone
Active Directory administrator Your organization
AWS administrator Your organization
Identity provider (IdP) administrator Your organization
Identity Center team/AWS Organizations administrator AWS account team
Application owners AWS account team
Identity provider (IdP) solutions architect AWS account team

1.2 – Inventory current state:

Before any changes are made, document the complete current state of your IAM Identity Center configuration. Capture a baseline of all three assignment layers: the users and groups assigned to the IAM Identity Center application in your IdP, the account and application assignments in IAM Identity Center, and any assignments inside the applications themselves:

Run the precheck script to export all current Identity Center assignments and principals to CSV.

```bash 
python -m idc_migration_tool precheck --output-dir ./export 
``` 

This will produce the following artifacts:

  • assignments.csv – One row per account, permission-set, and principal assignment
  • principals.csv – All users and groups from the identity store

1.3 – Define the cutover window:

Schedule the migration during a low-usage period to minimize user impact. Size the maintenance window using timings from a full-dress rehearsal in a test environment. SCIM provisioning is rate-limited, so for large directories, sync users in batches. Identify a maintenance window of sufficient length to complete all phases, including time for validation and potential manual rollback (described in Phase 7) if needed. Notify all affected users in advance of the potential temporary disruption to single-sign-on access.

1.4 – Confirm access requirements:

Verify that the following access is confirmed and tested before the cutover window:

  • AWS Administrator access to IAM Identity Center and AWS Organizations.
  • Identity provider (IdP) administrator access (for example, Okta) to configure the IAM Identity Center application, create the SCIM integration, and assign users to the AWS app.
  • A break-glass or emergency administrative access path that does not depend on the identity source you are migrating.

Phase 2: Complete final pre-cutover steps

Before starting the cutover, complete the final validation, access, and communication checks described in this phase.

2.1 – Export IAM Identity Center configuration:

Validate that the backup files generated in Step 1.2 are present and non-empty before proceeding.

2.2 – Communicate impact to users:

Send advance notice to all affected users covering:

  • The migration timeline and expected impact window.
  • That users can’t sign in between the identity source change and the new IdP’s sign-in going live, but that their access and assignments are unchanged.
  • The new sign-in process. After migration, users sign in through your IdP (for example, the Okta My Apps dashboard) or the AWS access portal, depending on the sign-in flow you choose.
  • Whether the new sign-in flow is IdP-initiated (through Okta) or SP-initiated (through the AWS access portal), and any differences in the user experience.
  • After migration, the d-xxxxxxxxxx.awsapps.com/start access portal URL is no longer supported, and the console shows the supported dual-stack regional URL (https://ssoins-<instance-id>.portal.<region>.app.aws) instead, so you must communicate the new URL and update bookmarks/allow-lists.
  • That the custom access portal alias (friendly URL) isn’t supported after migrating away from Active Directory. If a proxy or bookmark relies on redirect flows, plan to update it.

Prepare a user guide for the new sign-in process and an FAQ for common issues. Schedule a training session to demonstrate how to access AWS through Okta. Ensure your support team is briefed and ready to handle an influx of access issues during and immediately after the cutover.

2.3 – Validate Okta-to-IAM Identity Center sync (pre-cutover testing):

Using a sandbox or test environment if available, perform the following validation steps before executing against production:

  • Run the full migration in a test instance that mirrors your production AD configuration and applications and confirm that existing test users are matched by Okta and not duplicated.
  • Use the AWS Management Console for IAM Identity Center to verify that groups and users appear in Users & Groups with the correct attributes.
  • Test the complete end-to-end sign-in flow for a test user. Authenticate through Okta, access the AWS portal, and assume a role in a test account. Confirm that the user keeps access to their assigned accounts, permission sets, and applications.

Phase 3: Complete the cutover

You’re ready to configure Okta, switch the IAM Identity Center identity source, and complete the production cutover.

Important: Downtime begins when you confirm the identity source change in Step 3.2 and ends when SAML sign-in works through Okta. Have the Okta application, SAML metadata, and signing certificate ready before you start. Users, groups, memberships, and assignments are preserved.

3.1 – Configure the Okta application:

  1. Sign in to Okta as an administrator.
  2. Navigate to Applications and then to Browse App Catalog.
  3. Search for AWS IAM Identity Center and choose Add Integration. Choose Done.
  4. Choose the newly created app and select the Sign On tab.
  5. Locate the Metadata URL (it will look like: https://<your-okta-domain>/app/<app-id>/sso/saml/metadata). Open this URL in a new browser tab.
  6. An XML file will render in the browser. Save this file. This is your SAML metadata file, which will be uploaded to IAM Identity Center.
  7. Download and save the SAML signing certificate from the Sign On tab.

3.2 – Switch the identity source in IAM Identity Center:

  1. In the IAM Identity Center console navigation pane, choose Settings.
  2. Choose Actions and select Change Identity Source.

    Important: This change applies to the entire Identity Center instance. On the confirmation screen, you must enter ACCEPT to confirm. During this step and before SCIM provisioning is enabled, if the change could not be completed by Identity Center you will see a clear error message in the console to retry changing identity providers. It is safe to retry.

  3. Select External Identity Provider.
  4. Upload the XML metadata file and the SAML signing certificate downloaded from Okta.
  5. Confirm the change. IAM Identity Center will display progress updates as it migrates to the new identity provider.
  6. Don’t enable automatic provisioning yet. Complete Step 3.3 and confirm pilot sign-in in Step 3.4 first.

3.3 – Configure the ACS and issuer URLs in Okta:

  1. In the IAM Identity Center console, choose Settings and then Actions. Select Manage Authentication.
  2. Copy the IAM Identity Center Assertion Consumer Service (ACS) URL and the IAM Identity Center Issuer URL.
  3. Return to Okta and choose Applications and select IAM Identity Center app. Select the Sign On tab and choose Edit.
  4. Under Advanced Sign On Settings, paste the ACS URL and Issuer URL into the corresponding fields.
  5. Choose Save.

3.4 – Confirm pilot sign-in:

Before you enable SCIM provisioning, assign an existing pilot user to the IAM Identity Center application in Okta and confirm they can sign in through the AWS access portal. The SAML NameID that Okta sends must exactly match the user’s existing Identity Center username. If sign-in fails, fix the mapping now before proceeding to the next step.

3.5 – Enable automatic provisioning (console) and connect SCIM in Okta:

  1. In the IAM Identity Center console, choose Settings, then choose Enable Automatic Provisioning. Copy and securely record the SCIM endpoint URL and access token.
  2. Return to the IAM Identity Center app in Okta. Choose the Provisioning tab.
  3. Choose Configure API Integration.
  4. Paste the SCIM Endpoint URL into the Base URL field.
  5. Paste the Access Token into the API Token field.
  6. Choose Test API Credentials. You should receive confirmation that the credentials are valid.
  7. Choose Save.
  8. On the Provisioning page, choose Edit and under To App enable all three provisioning options (Create Users, Update User Attributes, and Deactivate Users). Choose Save. (The To App settings control how Okta provisions users to IAM Identity Center. All three options are required for the migration to work correctly.)
  9. On the Push Groups tab, push the groups that are assigned to the AWS application. Group names must exactly match the existing group names in IAM Identity Center.
  10. In the attribute mappings, confirm that userName maps to the existing Identity Center username and that first name, last name, display name, and a single email address are populated.

3.6 – Test SCIM takeover:

Before you assign all users, verify that Okta takes over existing users correctly:

  1. In Okta, select an existing pilot user who already exists in IAM Identity Center (People).
  2. Open the user and navigate to the Applications tab.
  3. Assign the AWS IAM Identity Center app to this test user.
  4. Wait 5–10 minutes for SCIM sync to run.
  5. In the IAM Identity Center console, select Users and confirm that the pilot user still has the same user ID, that no duplicate user was created, and that the user’s assignments still work.

    Note: If the user is duplicated or doesn’t update after 10 minutes, check the Okta System Log for SCIM provisioning errors and verify the SCIM endpoint URL and token are correctly configured.

  6. After the pilot user is taken over successfully, assign the remaining users and groups to the application in Okta. For large directories, assign them in batches.

Phase 4: Verify takeover

Now that you’ve completed cutover, verify that Okta has taken over all users and groups and that the preserved assignments match your pre-migration baseline.

4.1 – Verify SCIM takeover:

Confirm that all production users and groups are provisioned from Okta, that they keep the same user and group IDs, and that no duplicates were created. Check the Okta provisioning logs for errors such as “Refused to create a new, duplicate resource.”

4.2 – Check for assignment drift: Compare the live assignments against the pre-migration CSV

```bash 
python -m idc_migration_tool validate --csv ./export/assignments.csv --output-dir ./export 
```

If drift is found, the script writes a drift_report.csv file with MISSING and EXTRA entries. Because assignments are preserved, investigate any entries before you continue.

Phase 5: Post-migration validation

After the cutover, validate end-to-end access for representative user roles and confirm that assignments were preserved.

5.1 – Test end-to-end access:

Using a test Okta user account, complete the following validation steps:

  1. Sign in to the Okta Dashboard.
  2. Choose the AWS IAM Identity Center tile under My Apps.
  3. Select an assigned AWS account from the portal.
  4. Confirm that you can assume a role and access the AWS console.
  5. Test role switching within the console to validate permission boundaries.

5.2 – Spot-check multiple user types:

Validate access for a representative sample of users across different access levels, including Administrator, Developer, Read-Only or Audit, and any other standard roles defined in your environment. Confirm that permission sets are correctly mapped and that no unintended access has been granted or denied.

For each application, confirm that user-specific data and settings are intact and that any assignments inside the application still work.

Phase 6: Post-migration cleanup and monitoring

After confirming the migration is stable, shift focus to ongoing monitoring of provisioning and authentication, and update your operational documentation.

6.1 – Decommission AD only when stable:

  • Don’t decommission the Active Directory domain controllers until you’re confident the migration is stable and rollback is no longer needed. Keep AD online as a fallback. Deleting the AD directory removes the manual rollback option and the AD-based access portal endpoint.
  • The following command will identify disabled users, empty groups, and unassigned permission sets.
    ```bash 
    # Report only 
    python -m idc_migration_tool cleanup 
     
    # Report and prompt for deletion 
    python -m idc_migration_tool cleanup --delete 
    ```

6.2 – Monitor SCIM sync:

  • Confirm that SCIM provisioning continues to run without errors in the days following migration.
  • Monitor new user onboarding from Okta to verify that newly created Okta users are automatically provisioned into IAM Identity Center.
  • Review IAM Identity Center and AWS CloudTrail logs for failed sign-in attempts or unexpected authentication errors.

6.3 – Update documentation:

Update all internal documentation to reflect the new identity architecture:

  • Record the SCIM integration details, SAML metadata links, and new group-to-permission-set mappings.
  • Update internal SOPs and onboarding guides to reflect the new Okta-based login process.
  • Document troubleshooting procedures for the most common failure modes.

6.4 – Post-migration checklist:

Before closing the migration project, confirm all of the following:

  • Users can authenticate successfully through Okta.
  • Permission sets are correctly assigned to the correct users and groups.
  • Account assignments are working correctly across accounts.
  • Monitoring and logging are configured and alerting on anomalies.
  • Internal documentation and SOPs are updated.
  • Support team is trained on the new login process.
  • Rollback procedures have been reviewed and are ready if needed.
  • Active Directory integration cleanup is scheduled (do not rush decommission).

Phase 7: Rollback (if needed)

Rollback requires manually switching the identity source back to Active Directory, which deletes users, groups, memberships and assignments, and then rebuilding assignments from the pre-migration backup files, as described in the following steps.

Note: Define in advance what counts as a failure, who makes the rollback decision, and the acceptable discrepancy threshold. Only initiate rollback if the Okta migration has failed and cannot be recovered within the migration window. Rollback is a manual, time-consuming process. There is no single-button undo for an identity source switch.

7.1 – Switch identity source back to Active Directory:

  1. Navigate to IAM Identity Center and choose Settings, then Actions, and then Change Identity Source.
  2. Select Active Directory as the identity source.
  3. Select your existing AD domain. Ensure AD is still running and reachable before confirming.
  4. Accept the conditions and confirm the switch. Identity Center will display progress updates.
  5. When complete, choose Resume Sync, then choose Start Guided Setup.
  6. Confirm the attribute mappings and select the users and groups to sync.
  7. Review and save the configuration.

7.2 – Wait for AD users and groups to repopulate:

After completing the guided setup, AD users and groups will begin to reappear in IAM Identity Center through the directory sync. This might take several minutes depending on the size of your directory.

7.3 – Recreate account assignments:

Recreate the assignments from the pre-migration CSV file. This will resolve each principal name in identity store, create the assignment, and poll until completion, and produce a summary of succeeded, failed, and skipped counts.

```bash 
python -m idc_migration_tool cutover --csv ./export/assignments.csv 
```

7.4 – Validate access:

Confirm users can authenticate using the original AD-based AWS access portal URL and can access their expected AWS accounts and roles. Test key role types (such as Admin, Developer, and Audit) before declaring rollback complete.

Troubleshooting the cutover

The identity source change in Step 3.2 runs as a sequence of steps: stopping Active Directory sync, disconnecting the directory, and deleting the AD sync profile. If a step after “Stopping Active Directory sync” fails, the Confirm change screen reports which steps failed and offers a Retry failed steps button. You will see one of two error messages, both safe to retry:

  1. “The identity source change could not be completed. Please try again.” — AD sync was stopped but the directory wasn’t disconnected, so new users and changes are not currently syncing from Active Directory. Choose Retry failed steps to complete the transition. If instead you want to stay on Active Directory, choose Start guided setup in the Settings page to reconfigure the sync.
  2. “The identity source change could not be completed. Your identity source has been reverted to Active Directory. Please try again.” — IAM Identity Center automatically rolled back to a healthy AD-connected state, and a banner confirms that AD sync resumed. You don’t need to reconfigure the sync; choose Retry failed steps when you’re ready to try again.

Important: In both cases, choose Retry failed steps on the Confirm change screen. If you navigate away from this screen, you can no longer retry the failed steps and will need to restart the identity source change.

For troubleshooting tips, see Troubleshooting IAM Identity Center issues.

If you have feedback about this post, submit comments in the Comments section below.


Conclusion

This post covered how to migrate your IAM Identity Center identity source from Active Directory to an external identity provider (Okta), including backing up existing assignments, validating your target identity source, performing the cutover, verifying that the new identity source took over your users and groups with their assignments intact, validating access, and rolling back if needed.

For the backup and restore scripts referenced in this guide, see the aws-samples GitHub repository. For more information about IAM Identity Center configuration and permission set management, visit the AWS Security Blog and the AWS IAM Identity Center documentation.

Xiaoxue Xu

Xiaoxue is a Solutions Architect for AWS based in Toronto. She primarily works with Financial Services customers to help secure their workload and design scalable solutions on the AWS Cloud.

Kanwar Bajwa

Kanwar is a Principal Technical Account Manager at AWS who works with customers to optimize their use of AWS services and achieve their business objectives.

Richard-Merritt

Richard Merritt

Richard is a Senior Delivery Consultant – DevOps who helps customers with their DevOps journeys and to harness the power of the AWS Cloud.

Taimur Ijlal

Taimur Ijlal

Taimur is a Senior Delivery Consultant – Security with AWS based in UK. He helps customers in their cloud security journeys and to grow using the power of AWS.

Author

Laura Reith

Laura is an Identity Solutions Architect at AWS, where she thrives on helping customers overcome security and identity challenges. In her free time, she enjoys wreck diving and traveling around the world.

Yee Fei Ooi

Yee Fei is a Solutions Architect supporting independent software vendor (ISV) customers in Singapore and is part of the Containers TFC. She enjoys helping her customers to grow their businesses and build solutions that automate, innovate, and improve efficiency.

  •  

Automate IAM Identity Center governance with continuous discovery and reporting

AWS IAM Identity Center integrates with external identity provider (IdP) to provide customers with a centralized authentication and authorization solution for AWS resources across AWS Organizations. AWS continues to invest into IAM Identity Center with a growing number of AWS services that natively integrate with IAM Identity Center. As your AWS organization scales, maintaining visibility into who has access to which applications and enforcing governance policies across accounts and Regions becomes increasingly complex. Identity Center helps address this by centralizing authentication and authorization for AWS resources across your organization, integrating with your external identity provider and a growing number of AWS services. However, as adoption scales, tracking access assignments and enforcing governance policies consistently becomes its own challenge.

This blog post focuses on planning your integration between an identity provider and IAM Identity Center for managed applications in your organization. We also walk through deploying and using an automated Identity Center discovery and reporting sample solution to help answer the governance and security questions:

  1. Which users or groups have access to which AWS applications?
  2. Who last accessed a specific AWS application and when?
  3. Which users and groups are assigned to which IAM Identity Center applications across organization and AWS Regions?
  4. How can you quickly generate reports to assist with compliance audits or security reviews?

The sample solution will identify associated AWS applications and the corresponding user and group assignments for the IAM Identity Center instances within your organization. The output is stored in a queryable format and generates CSV files for downstream analysis or reporting.

Plan identity governance for Identity Center application assignments

There are four key areas to start on when planning how to manage delegation and provisioning access across IAM Identity Center managed AWS applications. Bring together key stakeholders across security, governance, application, and business teams to make sure the implementation and integration will fit into the overall identity governance strategy.

  1. Who can provision managed AWS applications: You can implement the IAM restrictions for creation of new AWS resources within AWS accounts in your organization. For example, if you restrict provisioning into a production AWS account to only infrastructure as code (IaC) IAM roles, you would continue implementing restrictions using AWS identity policies, service control policies (SCP), resource control policies (RCP), or IaC policy evaluation tools like Open Policy Agent (OPA) or Checkov.
  2. Who manages user and group assignments: The managed application administrator handles authorization to managed applications within an AWS account. It’s recommended to clearly define roles and responsibilities across the workflow. You would have an IaC pipeline manage the integrated AWS resource provisioning with IAM Identity Center, then another workflow to allow requests to manage user and group membership for the managed application.
  3. How authentication flows from the IdP to AWS resources: Users will authenticate into Identity Center, then be authorized to access AWS managed applications. From there, they will be authorized to access the associated AWS service and resources tied to the managed application. Depending on the AWS service, the associated downstream resources might have their own IAM principals that the users can access.
  4. Mapping IdP identities to AWS resource access: There needs to be a link for workforce users and groups in your IdP, to Identity Center managed applications, and to downstream resources and permissions. Identifying the relationship will help you understand access within your AWS environment. Trusted identity propagation (TIP) is an additional feature of Identity Center that provides an end to end trail of the identity to the downstream service.

Create and manage an Identity Center application assignment lifeycle

As a security best practice, you should enable delegated administration when managing Identity Center within an AWS organization instances.

After you have IAM Identity Center set up within an organization instance, your member AWS accounts can start creating associated AWS resources. Within each member AWS account, the IAM principals that provision AWS resources will need two types of service-specific IAM permissions:

  • The first type of IAM permissions will be specific to the AWS service you want to provision. For example, to create an Amazon SageMaker AI domain, you would need the same IAM permissions to create the SageMaker AI domain and the downstream AWS resources SageMaker AI might use.
  • The second type of IAM permissions is specific to IAM Identity Center. The IAM principal used to create the resource, in this example SageMaker AI, will also need permissions to manage applications within the Identity Center instance.
{
	"Version": "2012-10-17",
	"Statement":
	[
		{
            "Effect": "Allow",
            "Action": [
                "sso:CreateManagedApplicationInstance",
                "sso:GetManagedApplicationInstance",
                "sso:DeleteManagedApplicationInstance",
                "sso:DescribeRegisteredRegions"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "sso:CreateApplication",
                "sso:DescribeApplication",
                "sso:DeleteApplication",
                "sso:PutApplicationGrant",
                "sso:PutApplicationAuthenticationMethod",
                "sso:PutApplicationAccessScope"
            ],
            "Resource": 
            [
                "arn:aws:sso::<INSERT-ACCOUNT-ID>:application/ssoins-<INSERT-INSTANCE-ID>/apl-*"
            ]
        }
    ]
}

IAM Identity Center application Amazon Resource Names (ARNs) follow a different standard naming convention that isn’t based on the original resource name that was provided during resource creation. For example, when a user creates an Amazon Simple Storage Service (Amazon S3) bucket and sets a specific bucket name, that bucket name is included in the ARN: arn:[partition]:s3:::[bucket-name]. Identity Center application ARNs use unique identifiers (GUIDs) generated at creation time.

Manage access for an Identity Center application

After the IAM Identity Center application is created, you will need to manage access to the Identity Center application and associated AWS resources. To continue with the SageMaker AI domain example, after the domain is created, an authorized IAM principal will need to assign Identity Center users or groups from the Identity Center instance to the domain. For Identity Center, you will need two types of Identity Center IAM permissions.

The first type of IAM permissions is used to list IAM Identity Center users and groups within the Identity Center instance. This is needed to read and select specific IAM users or groups to assign to an Identity Center application.

{
    "Version": "2012-10-17",
    "Statement":
    [
        {
            "Sid": "ListIdentityCenterUsers",
            "Effect": "Allow",
            "Action":
            [
                "identitystore:ListUsers",
                "identitystore:DescribeUser",
                "identitystore:ListGroups",
                "identitystore:DescribeGroup",
                "identitystore:ListGroupMemberships"
            ],
            "Resource": "*"
        }
    ]
}

Although IAM Identity Center users and groups have a GUID, the GUIDs aren’t clearly linked to the resource friendly names. For example, a group name could be Read-Only and the resource GUID could be 1234567890-abcdef12-3456-7890-abcd-ef1234567890 in the identity store. Additionally, the IAM actions to list users or groups require the AllUsers or AllGroups parameter. Because List actions require access to users and groups, a restrictive IAM policy can’t be used to prevent IAM principals from seeing a subset of users or groups within the identity store. The second type of IAM permission is used to create and manage application assignments for the Identity Center application within the Identity Center instance.

{
    "Version": "2012-10-17",
    "Statement":
    [
        {
            "Sid": "ManageApplicationAssignments",
            "Effect": "Allow",
            "Action": 
            [
                "sso:CreateApplicationAssignment",
                "sso:DeleteApplicationAssignment",
                "sso:ListApplicationAssignments",
                "sso:PutApplicationAssignmentConfiguration"
            ],
            "Resource":
            [
                "arn:aws:sso::<INSERT-ACCOUNT-ID>:application/ssoins-<INSERT-INSTANCE-ID>/apl-*"
            ]
        }
    ]
}

Because the IAM Identity Center application ARN is created using a unique application ID during creation, it’s not recommended to implement an IAM policy restricting authorized IAM principals to manage specific Identity Center applications. For example, to limit the application assignments to only a specific set of applications, you would need to:

  1. Create the AWS resource with IAM Identity Center as the authentication mechanism
  2. Query the Identity Center application ARN for the associated AWS resource
  3. Identify the IAM principal that will be used for application assignments
  4. Create or update an IAM policy associated to that IAM principal to allow application assignments for that specific application
  5. Create or update an SCP to restrict application assignment to that specific IAM principal

In lieu of implementing resource restrictions within identity policies, you should limit management of IAM Identity Center application and application assignments to a limited number of authorized IAM principals. In addition, it is recommended to implement detective and reactive capabilities to manage Identity Center application assignments.

Plan your naming conventions and automation strategy

IAM Identity Center provides several APIs to capture information about your AWS organization instances, applications, and assignments. Before implementing automation or guardrails, you should develop a methodical approach and understand what outcome you’re working backwards from. Start by defining naming conventions and deciding what parts of the workflow you want to centralize.

  1. Determine a naming convention for groups within your IdP: For example: AWS_<ACCT#>_<AWS_Service>_<LOB>_<ENV>_<AppName>. The IdP group name would look like: AWS_123412341234_SageMaker_Data_PROD_GTLabel.
  2. Define the naming convention for AWS resources for your Identity Center integrated applications: For example: <AWS_Service>_<LOB>_<AppName>. The AWS resource name would look like: SageMaker_Data_GTLabel.
  3. Define the naming convention for Identity Center application names: For example: <AWS_Service>_<LOB>_<ENV>_<AppName>. The Identity Center application name would look like: SageMaker_Data_PROD_GTLabel.
  4. Decide on the restrictions that you want to implement within your AWS environment. Depending on your enterprise’s security standard, you can implement specific restrictions based on mapping of a similar combination of ENV (environment), AWS service, LOB (line of business), or application name.
  5. Choose the portions of the application workflow that you want to centralize. This could include creating the application, making application assignments, or remediating issues.

As more configurations and permissions are centralized, additional overhead and bottlenecks can be introduced. It’s important to find the right balance for your enterprise. For example, if you centralize application assignments, each application team will need to submit a request to modify assignments that will be reviewed by a centralized team and could result in a delayed response. Conversely, if each application team handles their own assignments, there’s a risk that application assignments won’t align to enterprise security standards.

By understanding your goals and how you want to reach them, you can tailor the sample solution accordingly. Getting alignment on this requires planning and coordination across multiple teams within your organization. When thinking about more customized authorization logic—such as using provisioned AWS resource metadata—you should review how the specific AWS service integrates with IAM Identity Center managed applications. For example, if you want to find the Identity Center application ARN for a specific AWS resource, such as a SageMaker AI domain, use the following approach. A reverse lookup is necessary because AWS services create Identity Center applications with GUID-based ARNs that aren’t easily discoverable.

#!/bin/bash

DOMAIN_ID="d-xxxxxxxxxxxx"

REGION="xx-xxxx-x"

# Step 1: Get SageMaker domain details
echo "=== SageMaker Domain Details ==="
DOMAIN_INFO=$(aws sagemaker describe-domain \
--region $REGION \
--domain-id $DOMAIN_ID)

# Step 2: Extract Identity Center application ARN
SSO_APP_ARN=$(echo $DOMAIN_INFO | jq -r '.SingleSignOnApplicationArn')
echo "Identity Center App ARN: $SSO_APP_ARN"

IAM Identity Center automation sample solutions

The sample-iam-idc-application-discovery-reporting solution hosted on GitHub consists of two separate AWS CDK stacks:

  1. IAM Identity Center governance reporting stack (/identity-center-reporting directory) – Provides automated discovery and report generation (using CSV files)
  2. IAM Identity Center remediation stack (/identity-center-remediation directory) – Provides real-time enforcement and notifications

The recommendation is to deploy the reporting stack first to establish baseline visibility, then deploy the remediation stack for enforcement.

The following diagram depicts that IAM Identity Center governance architecture.

The reporting sample deploys the following resources:

  1. Amazon EventBridge – Rule invokes the discovery workflow daily at 2:00 AM UTC (configurable)
  2. AWS Step Functions – Orchestrates the multi-stage discovery workflow across instances, applications,and assignments
  3. AWS Lambda – Takes the following actions:
    1. Discovers IAM Identity Center instances across the organization and member accounts
    2. Application discovery that enumerates the applications configured in each Identity Center instance
    3. Assignment discovery maps users and groups to applications, resolving friendly names from the Identity Store
  4. Amazon DynamoDB – Stores the discovered instances, applications, and assignments, encrypted with an AWS Key Management Service (AWS KMS) customer-managed key
  5. Amazon API Gateway – Provides an IAM-authenticated REST API for a Lambda function to generate and export reports as CSV files
  6. Amazon S3 – Stores the encrypted CSV file exports, with lifecycle policies and time-limited Amazon S3 presigned download URLs

Deploy the IAM Identity Center reporting sample

The following procedure deploys the automated discovery and reporting infrastructure using AWS Cloud Development Kit (AWS CDK). Make sure you have the following prerequisites in place, then continue with the steps to set up the solution.

Prerequisites

You need to have the following to test the solution in this post.

  1. An AWS organization with an IAM Identity Center organization instance with delegated administrator access configured
  2. IAM Identity Center configured with at least one instance
  3. AWS Command Line Interface (AWS CLI) configured with appropriate credentials
  4. Python 3.12 & Node.js 18 or later installed for CDK deployment

To deploy the IAM Identity Center reporting solution, run the following commands:

  1. Clone the solution repository:
    git clone https://github.com/aws-samples/sample-iam-idc-application-discovery-reporting
    cd identity-center-reporting
  2. Install dependencies:
    python3.12 -m venv .venv && source .venv/bin/activate
    pip install -r requirements.txt
  3. Bootstrap the CDK (if not already done):
    cdk bootstrap aws://<INSERT-ACCOUNT-ID>/<INSERT-REGION>
  4. Deploy the sample solution:
    export IDC_EXTERNAL_ID="$(uuidgen)" # alternatively you can set this value — member-account roles need the same value
    
    cdk deploy --parameters AllowedIpRange=10.0.0.0/8 --parameters CrossAccountExternalId="$IDC_EXTERNAL_ID"

    Note: AllowedIPRange is optional but recommended as a security best practice. The parameter will add a network restriction to download the Amazon S3 presigned URL export.

  5. Optional: For AWS account-level Identity Center instance discovery, a cross-account IAM role is required.
    python scripts/deploy-cross-account-roles.py --external-id "$IDC_EXTERNAL_ID"
Figure 2: Successful AWS CDK deployment of the reporting stack

Figure 2: Successful AWS CDK deployment of the reporting stack

After the stack is successfully deployed, obtain the CDK output values for the API Gateway URL and S3 bucket name. If using a command line to deploy, these values will be displayed after the stack successfully deploys. It can also be found in the AWS Management Console as AWS CloudFormation stack output. The output will be used for generating reports in the following sections.

Note that this stack is for the reporting stack only. Reactive monitoring and deployment are described in the next section.

After the reporting stack is successfully deployed, the automation will run on a daily schedule. The first discovery run executes immediately after deployment. You can monitor discovery execution history and detailed logs through the the AWS Step Functions console. Review the detailed Lambda function logs in Amazon CloudWatch Logs. Query discovered instances, applications, and assignments through the DynamoDB console for one-time analysis.

Generate reports for Identity Center application assignments

To generate on-demand reports as CSV files from the REST API:

  1. Set env variables for Sigv4 authentication
      export AWS_REGION="<REPLACE-REGION>"
      export API_ID="<REPLACE-API-ID>"
      eval "$(aws configure export-credentials --profile "<YOUR-PROFILE>" --format env)"
  2. Export applications
      curl -sS --fail-with-body \
        --aws-sigv4 "aws:amz:${AWS_REGION}:execute-api" \
        --user "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}" \
        --header "x-amz-security-token: ${AWS_SESSION_TOKEN}" \
        "https://${API_ID}.execute-api.${AWS_REGION}.amazonaws.com/prod/export/applications" \
        -o applications.json
  3. Export assignments with user and group names
      curl -sS --fail-with-body \
        --aws-sigv4 "aws:amz:${AWS_REGION}:execute-api" \
        --user "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}" \
        --header "x-amz-security-token: ${AWS_SESSION_TOKEN}" \
        "https://${API_ID}.execute-api.${AWS_REGION}.amazonaws.com/prod/export/assignments" \
        -o assignments.json
  4. The API returns a JSON response with a presigned Amazon S3 URL that’s valid for 15 minutes:
    {
        "message": "CSV export generated successfully",
        "download_url": "https://<bucket>.s3.amazonaws.com/exports/applications/2026/06/22/applications_export_20260722_184538.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=900&...",
        "filename": "applications_export_20260722_184538.csv",
        "s3_key": "exports/applications/2026/07/22/applications_export_20260722_184538.csv",
        "file_size_bytes": 6514,
        "export_type": "applications",
        "generated_at": "2026-07-22T18:45:38Z",
        "expires_at": "2026-07-22T19:00:38Z",
        "request_id": "a1b2c3d4-...."
    }
    {
        "message": "CSV export generated successfully",
        "download_url": "https://<bucket>.s3.amazonaws.com/exports/applications/2026/07/22/applications_export_20260722_184538.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=900&...",
        "filename": "applications_export_20260722_184538.csv",
        "s3_key": "exports/applications/2026/07/22/applications_export_20260722_184538.csv",
        "file_size_bytes": 6514,
        "export_type": "applications",
        "generated_at": "2026-07-22T18:45:38Z",
        "expires_at": "2026-07-22T19:00:38Z",
        "request_id": "a1b2c3d4-...."
    }

The generated CSV files include enriched data with friendly names:

Instance ARN Account ID Application name Principal type Principal name Status
arn:aws:sso:::instance/… 123456789012 SageMaker_PROD GROUP Engineering-Team-Dev ACTIVE
arn:aws:sso:::instance/… 123456789012 OpenSearch_PROD USER jane.doe@example.com ACTIVE

You can use the generated CSV files to help identify anomalies or non-compliant assignments, such as:

  1. Each PROD application should only have GROUP assignments. OpenSearch_PROD has a USER principal type and so is non-compliant.
  2. Each PROD application should only allow PROD groups assigned. SageMaker_PROD has a DEV group name (Engineering-Team-Dev) assigned and so is non-compliant.

Based on the testing and analysis of the output from the IAM Identity Center governance reporting sample solution, it’s important to start thinking about what restrictions to put in place for application assignments. It’s also important to conduct this exercise before taking action within the Identity Center remediation sample solution in the next section.

IAM Identity Center remediation

The following diagram shows the IAM Identity Center remediation architecture.

The IAM Identity Center remediation sample solution deploys the following resources:

  1. Amazon EventBridge – Matches IAM Identity Center assignment and profile events from CloudTrail (sso.amazonaws.com) and invokes the monitor function across the following IAM actions:
    1. CreateApplicationAssignment
    2. DeleteApplicationAssignment
    3. PutApplicationAssignmentConfiguration
    4. AssociateProfile
    5. DisassociateProfile
    6. CreateProfile
    7. UpdateProfile
    8. DeleteProfile
  2. Lambda – Resolves the application and group names, validates the assignment against your naming convention, and notifies or remediates based on the configured mode
  3. Amazon Simple Notification Service (Amazon SNS) – Publishes alerts for non-compliant assignments to subscribers (for example, email)
  4. Amazon Simple Queue Service (Amazon SQS) – Captures events the Lambda function fails to process for later inspection
  5. AWS KMS – Customer-managed key to encrypt the Lambda environment variables, CloudWatch logs, SNS topic, and dead-letter queue
  6. Amazon CloudWatch – Log group stores the function’s structured, encrypted logs as an audit trail

Flexible naming policies support regex-based pattern matching for specific organizational requirements. The automation actions are logged to CloudWatch with structured JSON for additional analysis and reporting.

The following procedure deploys the remediation infrastructure using AWS Cloud Development Kit (AWS CDK). Make sure you have the following prerequisites in place, then continue with the steps to set up the solution.

Prerequisites

You need the following to run the remediation solution:

  1. An AWS organization with an IAM Identity Center organization instance with delegated administrator access configured
  2. IAM Identity Center configured with at least one instance and an IdP
  3. Access to create groups within the integrated IdP
  4. AWS Command Line Interface (AWS CLI) configured with appropriate credentials
  5. Python 3.12 & Node.js 18 or later installed for CDK deployment

Provide your IAM instance ARN and the account ID where IAM Identity Center is administered:

git clone https://github.com/aws-samples/sample-iam-idc-application-discovery-reporting # only needed if you did not clone in the previous reporting section
cd identity-center-remediation
cdk deploy --context enableAutoDeletion=false --parameters IdentityCenterInstanceArn=arn:aws:sso:::instance/ssoins-<INSERT-ORG-INSTANCE-ID> --parameters ManagementAccountId=<INSERT-MANAGEMENT-ACCOUNT>

Note: If you don’t pass a parameter for GroupNameRegex, the default action of the sample solution is to verify the group name appears as a whole word in the application name: Case-insensitive, splitting on -, _, and spaces, so ReadOnly matches sagemaker_readonly but read does not. If different validation is needed, the sample can be deployed with the regex value for GroupNameRegex.

After the solution is deployed, we will walk through testing both a compliant and non-compliant application assignment.

Gather Identity Center and application information

For this blog, we have already created two groups within the IdP that is integrated into an IAM Identity Center instance. We also already created two applications within Identity Center instance to use. Next, we’ll need to gather information specific to the environment to run through each example.

  1. Obtain the IAM Identity Center instance ARN and set the value.
    INSTANCE_ARN=$(aws sso-admin list-instances --region <REPLACE-REGION> --query "Instances[0].InstanceArn" --output text)
    
    echo "$INSTANCE_ARN"
  2. Obtain the IAM Identity Center identity store ID

    IDENTITY_STORE_ID=$(aws sso-admin list-instances --region <REPLACE-REGION> --query "Instances[0].IdentityStoreId" --output text)
    
    echo "$IDENTITY_STORE_ID"
  3. Get existing groups in IAM Identity Center
    aws identitystore list-groups --identity-store-id $IDENTITY_STORE_ID --query "Groups[].{Name:DisplayName,Id:GroupId}" --output table
  4. Get existing enabled applications in IAM Identity Center
    aws sso-admin list-applications --instance-arn $INSTANCE_ARN --query "Applications[?Status=='ENABLED'].{Name:Name,ARN:ApplicationArn}" --output table

After you have the output for IAM Identity Center groups and applications, select two groups and one application that you want to test with. You will need to set additional variables for each group GUID and application ARN. In this example, I select the following two groups (ReadOnly and Developer) for testing and set the environment variables using export:

  1. Group #1 Name: ReadOnly
    export GRP_READONLY=abc12345-1234-1234-1234-abcdef123456
    export GRP_DEVELOPER=abc12345-1234-1234-1234-abcdef123457
    export APP_READONLY="arn:aws:sso::<INSERT-ACCOUNT-ID>:application/<INSERT-INSTANCE-ARN>/<INSERT-APPLICATION-ARN>"
    • Group #2 Name: Developer
    • Application Name: sagemaker_readonly

As part of this validation, the sample solution verifies the group name appears as a whole word in the application name: Case-insensitive, splitting on the -, _, characters and spaces, so ReadOnly matches sagemaker_readonly but read does not. For different use-cases, The GroupNameRegex parameter can be used during deployment.

Test compliant and non-compliant assignments

Run the following command to add the ReadOnly group assignment to the sagemaker_readonly application:

aws sso-admin create-application-assignment --application-arn $APP_READONLY --principal-id $GRP_READONLY --principal-type GROUP

The group assignment request meets the validation criteria because the application name sagemaker_readonly contains the group name ReadOnly. The output logs for this validation exist within the associated lambda function CloudWatch log group /aws/lambda/identity-center-app-monitor”.

In this example, the logs will show:

✓ COMPLIANT - Group name found in application name

applicationName="sagemaker_readonly” groupName="ReadOnly”

Remediation action determined: NONE

Run the following command to try to add the Developer group assignment to the sagemaker_readonly application:

aws sso-admin create-application-assignment --application-arn $APP_READONLY --principal-id $GRP_DEVELOPER --principal-type GROUP

The group assignment request doesn’t meet the validation criteria because the application name sagemaker_readonly doesn’t contain the group name Developer. The output logs for this validation exists within the associated lambda function CloudWatch log group /aws/lambda/identity-center-app-monitor. Note that the remediation action listed shows NOTIFICATION_ONLY, meaning it only sent a notification to the configured SNS topic and did not take action. If you want the group assignment to be deleted, the value should be set to enableAutoDeletion=true.

In this example, the logs will show:

✗ NON-COMPLIANT - Group name not found in application name

applicationName="sagemaker_readonly” groupName="Developer”

Remediation action determined: NOTIFICATION_ONLY

SNS notification sent successfully

The SNS message will look like:

{
	"eventType": "NON_COMPLIANT_ASSIGNMENT",
	"applicationName": "sagemaker_readonly",
	"groupName": "Developer",
    "action": "NOTIFICATION_ONLY",
    "status": "SUCCESS",
    "applicationArn": "arn:aws:sso::1234:application/ssoins-1234/apl-1234",
    "groupId": "abc12345-1234-1234-1234-abcdef123457",
    "initiatedBy": { 
    	"type": "AssumedRole", 
    	"arn": "arn:aws:sts::1234:assumed-role/.../you" 
	}
}

Scheduled reporting gives baseline visibility into IAM Identity Center managed applications. Event-driven monitoring can provide near real-time notification or enforcement. Together, these sample solutions can help align and scale Identity Center with your governance and security standards through both historical analysis and immediate response.

Clean up

For each deployed CDK stack, run the following commands in the AWS account where it was deployed.

To delete the remediation stack, run the following commands:

cd sample-iam-idc-application-discovery-reporting/identity-center-remediation
cdk destroy

To delete the reporting stack, run the following commands:

cd sample-iam-idc-application-discovery-reporting/identity-center-reporting
cdk destroy

IAM governance automation at scale

Achieving effective IAM Identity Center governance at scale requires moving beyond manual processes to automated, continuous monitoring and reporting. The following high-level steps can provide an Identity center governance framework:

  1. Deploy the sample automation with an IAM principal that has access in your delegated administrator account.
  2. Establish a baseline by running your first discovery and reviewing the generated reports.
  3. Configure naming policies to match your organization’s security conventions.
  4. Deploy the event-driven monitoring capabilities to enable real-time policy enforcement and automated response based on the security policies.
  5. Start in notification mode to establish a baseline before enabling auto-remediation.
  6. Integrate with your governance tools by connecting the API endpoints to your compliance dashboards or ITSM tools.
  7. Move to auto-remediation once you have validated policies are working as expected.

Conclusion

Managing AWS IAM Identity Center at scale doesn’t have to be a manual, time-consuming process. By implementing automated discovery and reporting combined with real-time event-driven monitoring, you can maintain continuous visibility into your organization’s identity and access landscape, respond immediately to policy violations, and enforce governance policies consistently across your organization. Automation reduces operational work, strengthens security, speeds up incident response, and maintains compliance. Start by deploying these solutions to gain visibility and enable real-time enforcement.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on AWS re:Post or contact AWS Support.


Author

Jonathan Nguyen

Jonathan is a Principal WWSO AI Security Solution Architect at AWS. He helps customers develop a comprehensive AI security strategy so they can deploy secure AI workloads at scale, integrate AI-powered security services, and defend against AI-powered threats.

  •  

Extend your data perimeter to the AWS Management Console with Private Access

Organizations in regulated industries such as financial services, government, defense, and healthcare restrict their sensitive workloads to isolated network environments with no access to the public internet. Until now, customers could restrict AWS Management Console access to authorized AWS accounts and corporate networks, but the console itself required internet connectivity. This was creating tension between operational convenience and network security controls.

We’re happy to announce that AWS Management Console Private Access is now generally available with support for virtual private clouds (VPCs) without internet connectivity. Organizations in regulated industries that restrict workloads to isolated network environments can now route all traffic for supported service consoles—including authentication flows, static assets (JavaScript, CSS, images), console-only APIs, and AWS service API calls—through AWS PrivateLink VPC endpoints, eliminating the need for an internet gateway, NAT gateway, or any route to the public internet. This capability is available in all AWS commercial Regions for a select set of supported service consoles.

In 2023, we launched AWS Management Console Private Access, which you can use to connect to the console by routing console, sign-in, and service API calls through VPC endpoints. However, accessing the console required internet connectivity for static assets and console-only APIs. This meant security teams faced a choice: allow internet connectivity to use the console or deny console access to operators working in network-isolated environments.

With this launch, AWS Management Console Private Access addresses two common scenarios:

  • Console traffic over internet restricted networks: Traffic for supported service consoles now flows entirely through your VPC endpoints—no proxy allowlists to maintain, no TLS-intercepting proxies to operate, and no CLI-only workflows to accept as a compromise. The same path works seamlessly from Amazon WorkSpaces, Amazon Elastic Compute Cloud (Amazon EC2) instances, and on-premises networks connected through AWS Direct Connect or AWS Site-to-Site VPN. Combined with sign-in resource control policies (RCPs) and sign-in resource policies, you can ensure that console authentication only succeeds from expected networks—even if valid credentials are presented elsewhere, the session is denied. Teams that previously relied on restricted egress rules or manual domain allowlists now get full console access with the same network controls they already trust.
  • Data-exfiltration prevention: Private Access enables you to restrict which AWS accounts and organizational identities can use the AWS Management Console from within your VPC. This prevents access from personal accounts and from accounts outside your organization. Attach a VPC endpoint policy with an aws:ResourceOrgID condition, and console actions are automatically scoped to resources inside your organization. Sign-in RCPs add a second layer by ensuring authentication only succeeds from networks within your perimeter. Together, these controls prevent supported service consoles from being used to access resources in accounts outside your organization—such as personal accounts—without requiring complex network-layer workarounds.

In this post, you will learn how AWS Management Console Private Access works in environments without internet connectivity, and how to layer access controls using VPC endpoint policies and sign-in resource control policies (RCPs) to strengthen your data perimeter.

Solution overview

AWS Management Console Private Access and sign-in resource control policies are a natural extension of the service control policies (SCPs), resource control policies, and VPC endpoint policies you already use for API traffic; now applied to the console session itself. The same data perimeter controls for identity, resource, and network that protect your programmatic access now protect interactive browser sessions too.

Perimeter Control objective Policy construct Implementation Steps
Identity Only trusted identities can access my resources Sign-In RCPs and RBPs Restrict which principals can sign in to the console. Before authentication, signin:PrincipalArn is available for exemptions only. After authentication, RCPs restrict at the organization, account, or principal level (aws:PrincipalOrgID, aws:PrincipalAccount, aws:PrincipalArn); RBPs restrict at the account or principal level.
Identity Only trusted identities are allowed from my network Console VPC endpoint policy and Sign-In VPC endpoint policy Console endpoint: aws:PrincipalOrgID or aws:PrincipalAccount on signed-in identities. Sign-In endpoint: aws:ResourceOrgID or aws:ResourceAccount before authentication, principal and resource keys after authentication. Blocks sign-in to accounts outside your organization, such as personal accounts, from your network.
Resource My identities can access only trusted resources SCP Resource perimeter SCP with aws:ResourceOrgID follows your principals into every console session; each service API call the console makes on their behalf is denied if the target resource is outside your organization.
Resource Only trusted resources can be accessed from my network Console VPC endpoint policy and service VPC endpoint policies Console endpoint policy with aws:ResourceOrgID and aws:ResourceAccount scopes what the console can reach through your network.
Network My identities can access resources only from expected networks SCP Network perimeter SCPs that use aws:SourceVpc deny your principals’ service calls from outside expected networks. With Private Access, requests proxied by the console to supported services carry aws:SourceVpc set to the VPC hosting your Private Access endpoints. Direct browser requests carry VPC context only when the service has its own VPC endpoint, so configure endpoints for every service you use. AWS recommends conditioning on aws:SourceVpc rather than specific aws:SourceVpce values.
Network My resources can only be accessed from expected networks Sign-In RBPs and RCPs and network perimeter RCPs Sign-In policies deny console authentication from unexpected networks using aws:SourceIp, aws:SourceVpc, aws:SourceVpce, and aws:VpcSourceIp in both pre-authentication and post-authentication statements. Network perimeter RCPs apply the same network conditions to your data resources for any access path.

With this launch, Console Private Access routes browser traffic for supported service consoles through VPC endpoints, including:

  • Authentication flows – Sign-in, credential exchange, and session token requests
  • Static assets – JavaScript, CSS, and images that render the console UI
  • Service console API calls – The backend requests made when users interact with service consoles

How traffic flows from a workload in a private VPC through the three Private Access endpoints, with no path to the public internet (shown in Figure 1):

  1. The operator’s browser requests <region>.console.aws.amazon.com.
  2. The corporate DNS forwarder forwards the query to an Amazon Route 53 Resolver inbound endpoint configured within the VPC, which forwards the traffic to the console VPC endpoint.
  3. Browser traffic flows from on-premises through Direct Connect (or AWS Site-to-Site VPN) to the VPC, and the VPC endpoint routes traffic to the console service over the AWS private network.
  4. The console service redirects to the SignIn endpoint <region>.signin.aws.amazon.com to establish a browser session.
  5. The DNS now resolves to the SignIn VPC endpoint’s private IP addresses, and browser traffic flows to the SignIn service over the AWS private network.
  6. After entering credentials, the SignIn service evaluates VPC endpoint policies, in addition to resource-based policies (RBPs) and RCPs, then redirects back to the console.
  7. The console evaluates VPC endpoint policies, loads static content from the console API VPC endpoint, and enforces identity and resource restrictions when making calls to AWS service APIs.
  8. Users can now access the AWS Management Console over Private Access.
Figure 1: Network isolation architecture

Figure 1: Network isolation architecture

Deploy a pilot of AWS Management Console Private Access

This high-level walkthrough sets up AWS Management Console Private Access for a single AWS Region within one organizational unit (OU). We recommend rolling out incrementally; validate each step before you expand to additional Regions and OUs.

If you want to validate the mechanics of a Private Access deployment before you build out the full solution, the Getting started with a test environment guide walks you through a minimal configuration: a single VPC with the three Private Access endpoints and a permissive policy. This gives you a working setup to experiment with, independent of the deployment described in the rest of this post. To understand how sign-in policies can verify a user’s network location when they access the console, see Controlling console access with resource-based policies and resource control policies.

Prerequisites

You must have the following prerequisites:

Step 1: Baseline current console access

Before changing anything, use CloudTrail to map how your users access the console today. Search for eventName = ConsoleLogin over a representative window (we recommend 30 days) and review the sourceIPAddress, vpcEndpointId, and awsRegion fields. Identify which identity types are in use: root user, IAM user, SAML federation, and AWS IAM Identity Center. Decide which OU or account you will pilot with.

Note: A misconfigured sign-in policy can lock users out of the console. Avoid piloting in a production or shared account. Instead, use a dedicated test account and configure a break-glass principal (covered in Step 5) before enabling access enforcement.

Step 2: Create the Private Access VPC endpoints

In your chosen Region, create or identify a VPC to host the endpoints, then create three interface VPC endpoints in that VPC:

  • com.amazonaws.<region>.console for the console.
  • com.amazonaws.<region>.signin for AWS Sign-In.
  • com.amazonaws.<region>.console-static for console-only APIs. This endpoint is required only if your VPC has no internet path.

Step 3: Configure private DNS for AWS Management Console Private Access

To use AWS Management Console Private Access, you must configure private DNS so that the console domains—.console.aws.amazon.com, .signin.aws.amazon.com, and the associated static-content domains—resolve to your interface endpoints’ network interfaces within your VPC.

  • For workloads inside your VPC: If the workloads in your VPC use the default Amazon Route 53 Resolver, no additional DNS configuration is required. When you create each interface endpoint, enable the private DNS name option (set PrivateDnsEnabled = true). The public console domains will then resolve automatically to the endpoint network interfaces inside your VPC. If you use a custom DNS resolver or a private hosted zone, you must configure it explicitly to map the console domains to the endpoint addresses. See Working with private hosted zones for more information. For the complete list of domains and detailed DNS configuration steps, see the AWS Management Console Private Access required endpoints documentation.
  • For workloads outside your VPC: For workloads that reach the endpoints from outside the VPC—such as corporate offices connecting over AWS Direct Connect or AWS Site-to-Site VPN—ensure that your corporate DNS resolver returns the endpoint addresses for these domains. See Simplify DNS management in a multi-account environment with Route 53 Resolver for more information.

Step 4: Verify private connectivity

Sign in to the console from a workload inside your VPC. The console should load normally. To confirm that traffic is routing through your VPC endpoints, look for the lock icon in the console navigation bar, shown in Figure 2.

Figure 2: Console Private Access

Figure 2: Console Private Access

You can also verify in CloudTrail that recent ConsoleLogin events show the vpcEndpointId field populated with one of your endpoint IDs. Here’s an example CloudTrail ConsoleLogin event snippet showing the vpcEndpointId field:

{
  "eventVersion": "1.08",
  "userIdentity": {
    "type": "AssumedRole",
    "principalId": "AROA3XFRBF23EXAMPLE:john.doe",
    "arn": "arn:aws:sts::123456789012:assumed-role/Admin/john.doe",
    "accountId": "123456789012"
  },
  "eventTime": "2026-07-08T19:15:32Z",
  "eventSource": "signin.amazonaws.com",
  "eventName": "ConsoleLogin",
  "awsRegion": "us-east-1",
  "sourceIPAddress": "10.0.1.47",
  "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)...",
  "requestParameters": null,
  "responseElements": {
    "ConsoleLogin": "Success"
  },
  "additionalEventData": {
    "LoginTo": "https://console.aws.amazon.com/console/home",
    "MobileVersion": "No",
    "MFAUsed": "Yes",
    "vpcEndpointId": "vpce-0abc123def456789a"
  },
  "eventID": "a1b2c3d4-5678-90ab-cdef-EXAMPLE11111",
  "eventType": "AwsConsoleSignIn",
  "recipientAccountId": "123456789012"
}

If the AWS Management Console doesn’t load, work through the following checks.

Private DNS is enabled on each interface endpoint (or your custom resolver returns the endpoint addresses)

When Private DNS is enabled, AWS automatically creates the DNS entries that resolve the service domains (such as console.aws.amazon.com) to the private IP addresses of your VPC endpoints. Without it, your browser still routes to the public AWS endpoints, bypassing your private access setup entirely.

  1. Confirm that each endpoint shows PrivateDnsEnabled:
    aws ec2 describe-vpc-endpoints \
    --filters "Name=vpc-endpoint-type,Values=Interface" \
    --query "VpcEndpoints[].{Id:VpcEndpointId,Service:ServiceName,PrivateDns:PrivateDnsEnabled}" \
    --output table
    
  2. Then, from within your VPC, verify that the domains resolve to private addresses:
    nslookup console.aws.amazon.com
    nslookup signin.aws.amazon.com
    
    # Should return a private IP (e.g., 10.x.x.x), not a public one
    

Each query should return a private IP address from your VPC CIDR range. If you use a custom DNS resolver instead of the Amazon-provided DNS, ensure your forwarding rules direct the AWS domain queries to the Route 53 Resolver inbound endpoints in your VPC.

The endpoint security groups allow HTTPS (TCP 443) from your workload subnets

Each VPC endpoint creates elastic network interfaces (ENIs) in your subnets, and these ENIs are governed by security groups. If those security groups don’t permit inbound HTTPS traffic from your workloads, the connection fails silently.

  1. Identify the security groups attached to your endpoints:
    aws ec2 describe-vpc-endpoints --vpc-endpoint-ids vpce-0abc123def456789a \
      --query "VpcEndpoints[].Groups[].GroupId" --output text
    
  2. Then verify that each security group allows inbound TCP 443 from your workload subnets:
    aws ec2 describe-security-groups --group-ids sg-xxxxxxxx \
      --query "SecurityGroups[].IpPermissions[?ToPort==\`443\`]" \
      --output json
    

For traffic from outside the VPC (Direct Connect or Site-to-Site VPN), corporate DNS returns the endpoint IPs and the route propagates correctly

If you access the console from an on-premises workstation connected over AWS Direct Connect or AWS Site-to-Site VPN, two additional conditions must be met.

  1. Your corporate DNS must resolve the AWS domains to the VPC endpoint private IPs. From your on-premises machine, run:

    nslookup console.aws.amazon.com

    If this returns public AWS IPs, your corporate DNS isn’t forwarding queries through Route 53 Resolver. Configure conditional forwarding for the aws.amazon.com and amazonaws.com domains to your Resolver inbound endpoint IPs.

  2. Second, network routes must propagate correctly. Ensure the route table associated with your endpoint subnets has propagated routes from your virtual private gateway (VGW) or transit gateway, so return traffic can reach your on-premises network. Verify this with:
    aws ec2 describe-route-tables \
      --filters "Name=association.subnet-id,Values=subnet-xxxxx" \
      --query "RouteTables[].PropagatingVgws"
    

A quick end-to-end validation: Run traceroute console.aws.amazon.com from your workstation and confirm the path uses private hops only—no traffic should traverse the public internet.

If the console loads but the lock icon is missing

If the console loads but the connection isn’t private (for example, the lock icon is missing), the browser is reaching the console over the public internet instead of through your VPC endpoints.

  • Run nslookup console.aws.amazon.com from a workload inside the VPC. The result should be a private IP from your VPC CIDR range. A public IP means DNS is bypassing the endpoint, which usually happens because Private DNS has not been enabled on the interface endpoint (set PrivateDnsEnabled = true).
  • For workloads outside the VPC, make sure your corporate DNS forwards the console domains into the VPC, for example, through an Amazon Route 53 Resolver inbound endpoint.

Step 5: Apply VPC endpoint policies

Attach an endpoint policy to the console and AWS Sign-In endpoints that limits access to identities in your organization. The static-content endpoint doesn’t support endpoint policies.

Begin with a permissive Allow * policy and confirm that traffic routes through the endpoints (you should see the vpcEndpointId field populated in CloudTrail console events). After confirming the routing, add restrictions to your VPC endpoint policy and observe the traffic.

A starter policy uses two condition keys: aws:PrincipalOrgID to restrict identities to your organization and aws:ResourceOrgID to restrict the resources the console can reach to your organization’s resources. The full reference, including additional condition keys and resource-restriction patterns, is in the AWS Management Console Private Access user guide.

For policies beyond the pilot, see Data perimeters on AWS. The data perimeter policy examples GitHub repository covers service-specific considerations for implementing data perimeters in your environment.

Step 6: Apply a Sign-In policy

Sign-In policies deny console authentication requests that don’t match your network or principal conditions. The policy is composed of a pre-authentication statement covering signin:Authenticate and a post-authentication statement covering signin:AuthorizeOAuth2Access and signin:CreateOAuth2Token. Include both statements.

For your pilot, deploy the policy as an RCP from your AWS Organizations management account. When enabled, the RCP applies to all accounts in your organization, so we recommend piloting in a dedicated test organization before rolling it out broadly. Activate enforcement by calling the signin:PutConsoleAuthorizationConfiguration API for the organization in the us-east-1 Region (AWS Sign-In replicates policies globally from there). Resource permission statements have no effect until console authorization is enabled.

Important: Configure at least one excluded principal as a break-glass path before you enable the RCP. The recommended principal is a dedicated IAM role.

  1. Write the permission statements that define the network conditions:
    Example – Restrict access to corporate VPC:

    aws signin put-resource-permission-statement \
      --source-vpc vpc-0abc123def456789 \
      --requested-region us-west-2 \
      --excluded-principal "arn:aws:iam::123456789012:user/EmergencyAdmin" \
      --region us-east-1
    

    Example – Restrict access to specific IP range:

    aws signin put-resource-permission-statement \
      --source-ip "IP_ADDRESS" \
      --excluded-principal "arn:aws:iam::123456789012:role/BreakGlassRole" \
      --region us-east-1
    
  2. Enable console authorization for the organization to start enforcing the policy.
    aws signin put-console-authorization-configuration \
      --target-id <your-target-id> \
      --region us-east-1
    
  3. Review the consolidated policy that’s now in effect: 
    aws signin get-resource-policy --region us-east-1
    

For policy examples, the AWS Command Line Interface (AWS CLI) reference, and the lockout-recovery procedure, see the sign-in RBP blog post and the Controlling console access with resource-based policies documentation. The same documentation also covers the per-account alternative, which uses an RBP attached to a single account instead of an organization-wide RCP.

Step 7: Add a service VPC endpoint

So far, the console shell loads, the lock icon appears, and your Sign-In policy lets approved identities through. If you sign in to a service console such as the AWS Key Management Service (AWS KMS) console, the page might fail to load resources or hang. The Private Access endpoints carry the console shell, not the service API calls that the console makes on your behalf. In a VPC without an internet gateway, those calls have nowhere to go.

Add a VPC endpoint for the service itself. For the pilot, create an AWS KMS interface endpoint (com.amazonaws.<region>.kms) in the same VPC, with Private DNS enabled. Open the AWS KMS console from inside the VPC and confirm the list of keys loads. Repeat for each service your users need on day one. Please note that a single service console often calls more than one AWS service API. If a console loads but parts of the page show errors or stay empty, the most common cause is a missing endpoint for one of the services it depends on.

The current list of services that support PrivateLink is in the AWS PrivateLink documentation. Service consoles whose services don’t support PrivateLink will not work in a no-internet VPC and need to be handled separately.

Step 8: Hide Regions and services you haven’t configured (optional)

Console links to a service or Region that you don’t have endpoints for will fail inside your VPC. To prevent users from navigating to broken pages, use User Experience Customization (UXC) to hide Regions and services that aren’t part of your Private Access deployment. UXC is configured at the account level and applies to navigation, search results, and service-selection drop-downs.

Step 9: Validate, then expand

After applying the endpoint policies and the Sign-In RCP to one pilot account:

  1. Sign in from inside the corporate network. The session should succeed.
  2. Sign in from outside the corporate network. The session should be denied at the Sign-In step, before reaching the console.
  3. In CloudTrail, confirm ConsoleLogin events show vpcEndpointId populated for traffic from inside the network.
  4. For unexpected denials, look in CloudTrail for ConsoleLogin events with the error message Authorization denied because of a resource-based policy or Authorization denied because of a resource control policy to identify which statement was responsible.

Considerations

A few items worth mentioning before you commit to this design:

  • AWS IAM Identity Center: IAM Identity Center sign-in support isn’t yet available through a VPC endpoint. Initial single sign-on (SSO) authentication must still transit over the internet.
  • Programmatic access: Sign-In RBPs and RCPs gate interactive console sign-in. AWS SDK and AWS CLI requests signed with SigV4 aren’t affected. This is also your recovery path: a principal with signin:DeleteConsoleAuthorizationConfiguration permission can disable enforcement programmatically if console authorization is misconfigured.
  • Apps integrated with AWS Sign-In: Sign-In policies also apply to Amazon Connect, Amazon WorkSpaces, Amazon QuickSight, AWS Health Dashboard, Amazon AppStream 2.0, and Amazon Lightsail when those applications use AWS Sign-In to authenticate.
  • AWS Management Console Private Access is available in all commercial AWS Regions but supports only a subset of AWS service consoles. See Supported AWS Regions, service consoles, and features in Private Access documentation for more information.
  • For services that aren’t supported, you can still navigate to other consoles, but will require internet connectivity for the unsupported service consoles and console-only APIs.
  • Costs: You pay regular AWS PrivateLink endpoint pricing and data processing for each endpoint and each Region you deploy in. The three Private Access endpoints (console, signin, and console-static) plus the service endpoints you already use are the relevant line items.

Conclusion

In this post, we showed you how to extend the AWS data perimeter framework to the AWS Management Console. You routed console traffic through VPC endpoints with AWS Management Console Private Access, restricted console sign-in by network and organization with Sign-In RBPs and RCPs, and configured the console to operate in a VPC without an internet gateway. The four control objectives that you already enforce for API traffic now also apply to the console.

To get started, see the AWS Management Console Private Access documentation. For deployment patterns and Region-by-Region considerations, see the AWS Management Console Private Access reference architectures. For background on the broader pattern, see Establishing a data perimeter on AWS.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on the IAM forum on AWS re:Post or contact AWS Support.


Madhur Kulkarni

Madhur Kulkarni

Madhur is a Sr. Customer Solutions Manager at AWS, working with Strategic Accounts customers to accelerate cloud adoption and drive business outcomes. He partners with cross-functional teams across customer engineering, AWS service teams, and specialists organizations to deliver enterprise-scale cloud solutions.

Mateusz Jaworski

Mateusz Jaworski

Mateusz is a Principal Engineer at AWS, where he works on AWS Management Console.

Sujay Ghosh

Sujay Ghosh

Sujay is a Software Development Manager at AWS, where he leads a team responsible for enabling secure, reliable access to the AWS Management Console. He is passionate about building scalable infrastructure that helps millions of customers manage their cloud resources safely and efficiently.

Abhijit Barde

Abhijit Barde

Abhijit is a Principal Product Manager at AWS, where he focuses on making it straightforward for all AWS users to discover, monitor, and operate their AWS infrastructure using conversational assistants and generative AI.

  •  

Detecting multi-stage attacks on AWS: A guide to cross-service signal correlation

A single alert from one security service tells you something happened. Read that signal alongside activity from other services and your own business context, and you will know whether what happened is part of a multi-stage attack.

Consider a short sequence. An identity calls GetCallerIdentity from a source address it hasn’t previously used. Within minutes, that same identity runs a burst of List and Describe calls across several services, and some of them fail with AccessDenied. Soon after, a large volume of data leaves your environment toward a domain that was registered last week. Amazon GuardDuty might already flag pieces of this, such as the reconnaissance from an unfamiliar source, through finding types like Recon:IAMUser/* or Discovery:S3/*. What you gain from correlating the pieces yourself is a single view of the sequence, tied to your own business context, so you can act on the whole rather than triaging findings one at a time.

This post is for security engineers and security operations teams who run Amazon Web Services (AWS) detection services and want to catch patterns specific to their environment. You will see how AWS detection and your business context fit together, and how to build correlations that use that context. The examples run in Amazon CloudWatch Logs Insights so you can try them today, and the closing section describes how to grow them into an automated pipeline. The walkthrough later in this post lists the prerequisites for these queries.

Start with AWS detection services

Begin with the AWS detection services. They cover the threats common across customers, and everything in this post is built on them.

Turn these on and tune them before you build anything custom. Tuning means adjusting sensitivity to reduce false positives for your environment, choosing which data sources each service monitors, and suppressing findings for known-good patterns.

GuardDuty correlates multi-stage attacks for you

Before you build anything by hand, see what GuardDuty already does for you. Amazon GuardDuty Extended Threat Detection correlates signals across multiple data sources including AWS CloudTrail, Amazon S3 data events, runtime monitoring, Amazon Elastic Kubernetes Service (Amazon EKS) audit logs, and more, then raises a single critical severity attack sequence finding when it spots a multi-stage pattern. It recognizes sequences such as credential compromise followed by data exfiltration, maps them to MITRE ATT&CK tactics, and attaches a timeline and remediation guidance. If you have GuardDuty enabled today, then GuardDuty Extended Threat Detection is already enabled by default and needs no queries from you. For details on how GuardDuty charges apply, see Amazon GuardDuty pricing.

The credential compromise sequence in the opening example is the kind of universal pattern GuardDuty Extended Threat Detection is built to catch, so rely on it for those. Attack sequence findings show up in the GuardDuty console next to your other findings, and they route to Security Hub and your response workflows the same way.

GuardDuty handles the threats that look the same in every account. What it doesn’t have is the context that makes a given action suspicious in your account. That’s what you provide.

Add your business context

Business context is what only you know about your environment: which buckets hold sensitive data, which principals have a reason to touch which resources, which role chains your policy permits, and when your production change windows open. GuardDuty Extended Threat Detection learns from patterns common across customers, but it can’t answer these environment-specific questions. Express them as correlations and you add a detection layer tuned to your environment. Each of the following four patterns turns one of these facts into a query.

Run these queries in the AWS Management Console for CloudWatch by choosing Logs, then Logs Insights, using the CloudWatch Logs Insights query language. Most read CloudTrail events from a CloudWatch Logs log group that your trail delivers to. If your trail writes only to Amazon S3, add CloudWatch Logs delivery on the trail, or run equivalent queries in Amazon Athena (a serverless query service for analyzing data in Amazon S3 using SQL).

Note: The queries and code in this post use placeholder values. Replace them with your own before running: your-sensitive-bucket (your S3 bucket name), your-key-id (your AWS KMS key ID), region (your AWS Region, such as us-east-1), account-id (your 12-digit AWS account ID), and aws-cloudtrail-logs-my-trail (your CloudTrail log group name).

A note on multi-account environments. In AWS Organizations, an organization trail delivers every account’s events to one log group, so these queries work as-is but return cross-account results. Filter by recipientAccountId for account-scoped views. Without an organization trail, run queries per account or use Amazon Security Lake as a central query surface.

The attack chain mapped to AWS services

Multi-stage attacks move through five phases, and each phase leaves a signal in a different service. These signals surface across three log sources: CloudTrail, which records API activity in your account; Amazon VPC Flow Logs, which capture network connection metadata; and Amazon Route 53 Resolver query logs, which record DNS queries from your VPCs.

  • Initial access – Stolen credentials reach your environment. CloudTrail records GetCallerIdentity, GetSessionToken, or AssumeRole from an unfamiliar source.
  • Discovery – The threat actor enumerates with List, Describe, and Get calls, often triggering AccessDenied responses.
  • Privilege escalation – The threat actor chains roles or edits policies. CloudTrail records AssumeRole sequences, PutRolePolicy, or CreateAccessKey.
  • Lateral movement – The threat actor moves across accounts or AWS Regions, assuming roles and creating resources in unfamiliar places.
  • Exfiltration – Data leaves through GetObject calls at scale, large outbound transfers in VPC Flow Logs, and DNS queries in Route 53 Resolver query logs to recently registered domains.

Figure 1 shows the five attack phases mapped to the AWS log source that records each one.

Figure 1: Attack chain mapped to AWS services

Figure 1: Attack chain mapped to AWS services

GuardDuty Extended Threat Detection watches this chain for universal patterns. The four patterns that follow add the dimension you supply: your business context.

Pattern one: Sensitive data access by an unexpected principal

Your data classification and access norms drive this detection. One bucket holds customer records, another holds public web assets, and you know which principals have a reason to read the customer records, which are sensitive. Encode that knowledge and an ordinary looking read turns into something worth chasing.

Three signals converge here. CloudTrail shows GetObject at volume on a bucket you’ve classified as sensitive. The principal isn’t on your list of expected readers for that bucket. And VPC Flow Logs show a large outbound transfer from the same source in the same window, while DNS query logs show a recently registered destination domain, which together increase your confidence that there’s a potential threat.

CloudTrail management events don’t record GetObject. You must turn on CloudTrail data events for the buckets you care about to capture GetObject. Many teams miss GetObject because data events weren’t enabled on the relevant buckets.

This query shows bulk reads on a sensitive bucket, grouped by principal. Run it in CloudWatch Logs Insights with your CloudTrail log group selected.

fields @timestamp, userIdentity.arn, requestParameters.bucketName
| filter eventSource = "s3.amazonaws.com" and eventName = "GetObject"
| filter requestParameters.bucketName = "your-sensitive-bucket"
| stats count(*) as objectReads,
        count_distinct(requestParameters.key) as distinctObjects
        by userIdentity.arn, bin(10m)
| filter objectReads > 100
| sort objectReads desc

The threshold of 100 is a placeholder. Run the query over a week of normal activity, find the ninety-fifth percentile read count for that bucket, and set the threshold above it. Then check each principal the query returns against your expected reader list. A principal that isn’t on the list, reading at volume, is the result to investigate.

To corroborate, look for a matching outbound transfer. Switch the log group selector to your VPC Flow Logs log group and run this.

fields @timestamp, srcAddr, dstAddr, bytes
| filter action = "ACCEPT"
# exclude RFC 1918 private ranges so only external destinations remain
| filter dstAddr not like /^10\./
        and dstAddr not like /^192\.168\./
        and dstAddr not like /^172\.(1[6-9]|2[0-9]|3[0-1])\./
| stats sum(bytes) as totalBytes by srcAddr, dstAddr, bin(10m)
| filter totalBytes > 1000000000
| sort totalBytes desc

The Amazon S3 query returns a principal, and the Flow Logs query works on IP addresses, so you translate one into the other. The worked example later in this post covers that translation in full.

Picture an analytics role that reads a reporting bucket all day. One afternoon, it reads a thousand objects from your customer records bucket instead. GuardDuty stays quiet, because an authenticated role making valid GetObject calls isn’t suspicious anywhere else. Your query flags it, because that role isn’t on the expected reader list for that bucket. The classification you applied is what turns silence into a signal.

Figure 2 shows a bulk read from a sensitive bucket in CloudTrail, a large outbound transfer in VPC Flow Logs, and a young domain resolution in Route 53 Resolver logs.

Figure 2: Three signals converging within a single time window to indicate exfiltration

Figure 2: Three signals converging within a single time window to indicate exfiltration

Pattern two: A role chain that crosses your access policy

Picture a deployment that assumes one role to build, then a second to release. For one principal, that two-hop AssumeRole chain is routine; for a different principal it’s a policy violation. This pattern relies on your trust topology—the chains your organization permits—so put that knowledge in the query.

This pattern needs three conditions:

  • CloudTrail shows several AssumeRole calls from the same source inside a short window
  • The chain ends in a sensitive action such as CreateAccessKey, PutRolePolicy, or AttachUserPolicy
  • The starting identity isn’t one your policy expects to run that chain

In CloudWatch Logs Insights, select your CloudTrail log group and run this query, which surfaces chains of two or more hops.

fields @timestamp, userIdentity.arn, requestParameters.roleArn, sourceIPAddress
| filter eventName = "AssumeRole"
| stats count(*) as assumeCount,
        count_distinct(requestParameters.roleArn) as rolesAssumed
        by sourceIPAddress, bin(5m)
| filter assumeCount >= 2 and rolesAssumed >= 2
| sort assumeCount desc

Two hops is the minimum for a chain; raise the count if your environment chains roles often. Your deployment pipeline probably assumes several roles an hour, as do AWS service principals such as AWS Security Hub. Exclude the identities you expect to see assuming multiple roles, including your pipeline role and known AWS service principals. What’s left is the set to investigate, such as a person assuming several roles at an odd hour and ending in a new access key. Treat that distinction as data: list the identities and actions you consider normal, and review the chains that fall outside the list.

Pattern three: An encryption key used outside its owning workload

Resource ownership is the signal here. A given AWS Key Management Service (AWS KMS) key creates and controls the encryption keys for a workload, and a single key should serve a single workload, such as a payments service. A Decrypt call against it is a valid, authorized API action, so nothing about the call itself looks wrong. The ownership rule you set is what makes another principal’s use of the key worth a second look.

This pattern applies only to customer-managed keys scoped to one workload. It doesn’t apply to AWS-managed keys (alias/aws/*) or to customer-managed keys intentionally shared across services. Confirm single-workload intent from the key policy’s Principal block before deploying this rule.

Two conditions indicate misuse:

  • CloudTrail shows Decrypt or GenerateDataKey calls on a key that’s tied to one workload
  • The calling principal isn’t the role that owns that workload

Against your CloudTrail log group, run this query to list the principals that called a specific key.

fields @timestamp, userIdentity.arn, eventName
| filter eventSource = "kms.amazonaws.com"
| filter eventName in ["Decrypt", "GenerateDataKey", "Encrypt"]
| filter resources.0.ARN = "arn:aws:kms:region:account-id:key/your-key-id"
| stats count(*) as keyUses by userIdentity.arn, eventName
| sort keyUses desc

Compare what comes back against the one workload role you expect. A principal you don’t recognize on that key is the signal. Because key misuse is an early move in data theft, this correlation catches activity that only your ownership knowledge can flag.

Consider a key that wraps your payments database. The payments service role calls it in normal operation, and nothing else should. If a developer role or a freshly created role runs Decrypt against it, the call succeeds and reads as ordinary in isolation. The reason it matters is the ownership rule you hold in your head and now state in this query.

Pattern four: A privileged action outside your change window

Start with the query, then read what it means.

fields @timestamp, userIdentity.arn, eventName, sourceIPAddress
| filter eventName in ["PutRolePolicy", "AttachRolePolicy",
        "CreateAccessKey", "AuthorizeSecurityGroupIngress", "PutBucketPolicy"]
| stats count(*) as sensitiveChanges by userIdentity.arn, eventName, sourceIPAddress
| sort sensitiveChanges desc

Run it against your CloudTrail log group, scoped to your off-hours window when you schedule it, so it returns only activity outside the change window. Your change process defines what normal looks like here: production security and identity changes flow through a pipeline during defined hours, run by a known actor. A console-driven policy change at 2:00 AM, made by a person rather than the pipeline, doesn’t fit those expectations. The signal is a sensitive change such as PutRolePolicy or AuthorizeSecurityGroupIngress, made outside the window, by a person rather than your pipeline role.

Exclude the actors you expect, such as your deployment pipeline role, your patch automation role, and AWS service principals like AWS CloudFormation and AWS Systems Manager. What remains is privileged change made outside your process, which is both what an attacker does to establish persistence and what your own change discipline says shouldn’t happen.

Your pipeline might open security group rules during a deployment every weekday afternoon. A person opening a security group rule at midnight on a weekend is the same API call carrying a very different meaning. The schedule and the actor, both facts you define, are what separate the two.

Build your first correlation rule

The following walkthrough uses pattern one as a complete example. The other three patterns follow the same design with their own queries.

Prerequisites

These prerequisites feed the queries in this walkthrough. Confirm each one before you start:

  • A CloudTrail trail logging management events to a CloudWatch Logs log group
  • CloudTrail data events enabled for your sensitive S3 buckets
  • GuardDuty enabled, with its protection plans and Extended Threat Detection
  • VPC Flow Logs on for your production VPCs
  • Amazon Route 53 Resolver query logging on

CloudTrail, GuardDuty, VPC Flow Logs, and Route 53 Resolver query logging provide the raw signals that your correlations connect. Without them, the queries in this post return empty results.

Step 1: Record the bucket and its expected readers

Choose one sensitive bucket to monitor, and write down the principals allowed to read it. Store the list where your automation can reach it, such as a configuration file in version control or an Amazon DynamoDB table (a managed NoSQL database).

{
  "customer-records-prod": [
    "arn:aws:iam::123456789012:role/AnalyticsPipeline",
    "arn:aws:iam::123456789012:role/ComplianceAudit"
  ],
  "financial-data-archive": [
    "arn:aws:iam::123456789012:role/FinanceReporting"
  ]
}

This example hardcodes the list for simplicity. In production, load it from a DynamoDB table or Parameter Store so you can update it without redeploying.

Step 2: Baseline before you set a threshold

Run the pattern one query over one week of normal activity. Find the 95th percentile read count for the bucket and use a value greater than that as your alert threshold. This step keeps legitimate high-volume access from generating false positives later.

Set the THRESHOLD_READS environment variable to this value when you configure the function in Step 5.

Step 3: Run the access query

In the CloudWatch console:

  1. Choose Logs, then choose Logs Insights.
  2. In the Select log group(s) dropdown, select your CloudTrail log group.
  3. Set the time range to 3h (the last three hours).
  4. In the query editor, paste the pattern one query.
  5. Replace your-sensitive-bucket with your bucket name.
  6. Choose Run query.
  7. Review the principals in the results table.
  8. Compare each principal against your expected reader list from step 1, and flag any that are not on it.

Each result includes a principal that step 4 translates into an IP address.

Step 4: Correlate with network activity

CloudTrail logs actions by AWS Identity and Access Management (IAM) principal, while VPC Flow Logs record traffic by IP address. To connect the two signals, translate the principal into its address.

For a role attached to an Amazon Elastic Compute Cloud (Amazon EC2) instance, the userIdentity.principalId field includes the instance ID after the colon, in the form AROAEXAMPLE:i-1234567890abcdef0. Copy the instance ID and look up its private IP address.

aws ec2 describe-instances \
  --instance-ids i-1234567890abcdef0 \
  --query "Reservations[0].Instances[0].PrivateIpAddress" \
  --output text

Other compute types differ. A VPC-connected AWS Lambda function sends traffic through elastic network interfaces in your subnets, so correlate on those interface addresses. An Amazon Elastic Container Service (Amazon ECS) task records its network interface in task metadata. For a plain assumed-role session with no instance behind it, the sourceIPAddress field in CloudTrail already holds the caller’s address, so you correlate on it directly.

Run the Flow Logs query from pattern one, filtering srcAddr to that address within 10 minutes of the Amazon S3 read timestamp. A match places the same source behind both the sensitive read and a large external transfer in one window. CloudTrail events reach CloudWatch Logs 5–15 minutes after the API call, so correlate on eventTime rather than query time. Query a wider lookback than your correlation window: for example, look back 30 to 60 minutes but correlate on a 10-minute eventTime window. Steps 3 and 4 are manual validation; step 5 automates them.

Figure 2 shows DNS resolution as a third corroborating signal. This walkthrough implements the CloudTrail and VPC Flow Logs correlation. To add DNS, apply the same run_query() pattern against your Route 53 Resolver query log group.

Step 5: Automate the check

Move the query into a Lambda function (serverless compute that runs your code without a server to manage), send results to a notification channel, and schedule regular runs. Work through the following sub-procedures.

To create the notification channel

  1. Open the Amazon Simple Notification Service (Amazon SNS) console. Amazon SNS is a managed messaging service that delivers notifications to subscribers.
  2. In the navigation pane, choose Topics.
  3. Choose Create topic.
  4. For Type, select Standard.
  5. For Name, enter security-correlation-alerts.
  6. Choose Create topic.
  7. Note the topic Amazon Resource Name (ARN) at the top of the topic details page. You will use it in the function.
  8. Choose Create subscription.
  9. For Protocol, select Email.
  10. For Endpoint, enter your email address or incident management endpoint.
  11. Choose Create subscription, then confirm the subscription from the email AWS sends.

To create the EventBridge Scheduler execution role

The schedule needs a role that lets it invoke your function, and its trust policy needs conditions that pin the role to the schedule you own. Without those conditions, another account with access to the scheduler service could theoretically call this role; a class of misuse known as the confused deputy problem.

1. Create a trust policy file named scheduler-trust-policy.json.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "scheduler.amazonaws.com" },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "aws:SourceAccount": "ACCOUNT-ID"
        },
        "ArnLike": {
          "aws:SourceArn": "arn:aws:scheduler:REGION:ACCOUNT-ID:schedule/*/s3-access-correlation-hourly"
        }
      }
    }
  ]
}

2. Create the role, then attach permission to invoke the function. Scope Resource to the specific function ARN so this role can’t invoke anything else.

aws iam create-role \
  --role-name EventBridgeSchedulerRole \
  --assume-role-policy-document file://scheduler-trust-policy.json

aws iam put-role-policy \
  --role-name EventBridgeSchedulerRole \
  --policy-name LambdaInvokePolicy \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": "lambda:InvokeFunction",
        "Resource": "arn:aws:lambda:REGION:ACCOUNT-ID:function:CorrelationFunction"
      }
    ]
  }'

When you create the function, Lambda automatically creates an execution role. You will attach the permissions this function needs to that role in a later step.

To deploy the correlation function

  1. Open the Lambda console.
  2. Choose Create function.
  3. For Function name, enter CorrelationFunction.
  4. For Runtime, select the latest Python runtime.
  5. Choose Create function.
  6. On the Code tab, replace the default code with the following function, then choose Deploy.
import os
import time
import logging
import boto3
from botocore.exceptions import ClientError

logger = logging.getLogger()
logger.setLevel(logging.INFO)

logs = boto3.client("logs")
sns = boto3.client("sns")
ec2 = boto3.client("ec2")

CLOUDTRAIL_LOG_GROUP = os.environ["CLOUDTRAIL_LOG_GROUP"]
FLOWLOGS_LOG_GROUP = os.environ["FLOWLOGS_LOG_GROUP"]
SNS_TOPIC = os.environ["SNS_TOPIC_ARN"]
BUCKET = os.environ["SENSITIVE_BUCKET"]
THRESHOLD = int(os.environ.get("THRESHOLD_READS", "100"))

# Expected readers per bucket
EXPECTED_READERS = {
    "customer-records-prod": [
        "arn:aws:iam::123456789012:role/AnalyticsPipeline",
        "arn:aws:iam::123456789012:role/ComplianceAudit",
    ],
}


def run_query(log_group, query, start, end):
    """Start a Logs Insights query and wait for it to finish."""
    started = logs.start_query(
        logGroupName=log_group,
        startTime=start,
        endTime=end,
        queryString=query,
    )
    query_id = started["queryId"]
    while True:
        outcome = logs.get_query_results(queryId=query_id)
        if outcome["status"] in ("Complete", "Failed", "Cancelled"):
            break
        time.sleep(1)
    if outcome["status"] != "Complete":
        raise RuntimeError(f"Query did not complete: {outcome['status']}")
    return [{f["field"]: f["value"] for f in row} for row in outcome["results"]]


def private_ip_for_principal(principal_id):
    """Resolve an EC2 instance role principalId to its private IP."""
    if ":" not in principal_id:
        return None
    instance_id = principal_id.split(":", 1)[1]
    if not instance_id.startswith("i-"):
        return None
    reservations = ec2.describe_instances(InstanceIds=[instance_id])
    for reservation in reservations["Reservations"]:
        for instance in reservation["Instances"]:
            return instance.get("PrivateIpAddress")
    return None


def egress_bytes(src_addr, start, end):
    """Sum external egress bytes for one source address."""
    query = f"""
    fields srcAddr, dstAddr, bytes
    | filter action = "ACCEPT" and srcAddr = "{src_addr}"
    | filter dstAddr not like /^10\\./
            and dstAddr not like /^192\\.168\\./
            and dstAddr not like /^172\\.(1[6-9]|2[0-9]|3[0-1])\\./
    | stats sum(bytes) as totalBytes
    """
    rows = run_query(FLOWLOGS_LOG_GROUP, query, start, end)
    if rows and rows[0].get("totalBytes"):
        return int(rows[0]["totalBytes"])
    return 0


def lambda_handler(event, context):
    try:
        # 1-hour lookback absorbs CloudTrail's 5-15 min delivery latency;
        # correlation happens on eventTime via 10-min bins in the query below.
        end = int(time.time())
        start = end - 3600  # 1 hour lookback
        allowed = EXPECTED_READERS.get(BUCKET, [])

        access_query = f"""
        fields userIdentity.arn, userIdentity.principalId
        | filter eventSource = "s3.amazonaws.com" and eventName = "GetObject"
        | filter requestParameters.bucketName = "{BUCKET}"
        | stats count(*) as objectReads
                by userIdentity.arn, userIdentity.principalId, bin(10m)
        | filter objectReads > {THRESHOLD}
        """

        for row in run_query(CLOUDTRAIL_LOG_GROUP, access_query, start, end):
            principal = row.get("userIdentity.arn")
            if not principal or principal in allowed:
                continue

            message = (
                f"Principal {principal} read {row.get('objectReads')} "
                f"objects from {BUCKET}."
            )

            ip = private_ip_for_principal(row.get("userIdentity.principalId", ""))
            if ip and egress_bytes(ip, start, end) > 1_000_000_000:
                message += (
                    f" The same source ({ip}) also sent a large volume of "
                    f"data to external destinations in the same window."
                )

            sns.publish(
                TopicArn=SNS_TOPIC,
                Subject="Unexpected S3 access detected",
                Message=message,
            )
    except ClientError as error:
        logger.error(f"AWS API error: {error}")
        raise
    except Exception as error:
        logger.error(f"Unexpected error: {error}")
        raise
    finally:
        logger.info("Correlation check completed")
  1. On the Configuration tab, choose General configuration, then choose Edit. Set Timeout to 5 minutes (300 seconds). CloudWatch Logs Insights queries run asynchronously and can take 30 to 60 seconds against large log groups. Choose Save.
  2. On the Configuration tab, choose Environment variables, then choose Edit, and add CLOUDTRAIL_LOG_GROUP, FLOWLOGS_LOG_GROUP, SNS_TOPIC_ARN, SENSITIVE_BUCKET, and THRESHOLD_READS.
  3. On the Configuration tab, choose Permissions, open the execution role, and attach the following least-privilege policy.
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["logs:StartQuery", "logs:GetQueryResults"],
      "Resource": [
        "arn:aws:logs:REGION:ACCOUNT-ID:log-group:aws-cloudtrail-logs-my-trail:*",
        "arn:aws:logs:REGION:ACCOUNT-ID:log-group:vpc-flow-logs:*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "ec2:DescribeInstances",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "sns:Publish",
      "Resource": "arn:aws:sns:REGION:ACCOUNT-ID:security-correlation-alerts"
    }
  ]
}

Replace REGION, ACCOUNT-ID, and the log-group names with your values. The ec2:DescribeInstances action doesn’t support resource-level permissions, so Resource: "*" is required for that statement; the other statements are scoped to specific ARNs.

To schedule automated runs

Amazon EventBridge (a serverless event bus that connects applications using events) runs targets on a schedule. Create one from the command line, using the role you made earlier.

aws scheduler create-schedule \
  --name s3-access-correlation-hourly \
  --schedule-expression "rate(1 hour)" \
  --target "Arn=arn:aws:lambda:REGION:ACCOUNT-ID:function:CorrelationFunction,RoleArn=arn:aws:iam::ACCOUNT-ID:role/EventBridgeSchedulerRole" \
  --flexible-time-window "Mode=OFF"

Step 6: Add enrichment context (optional)

Enrichment cuts triage time by adding an independent signal, but it isn’t required for the correlation to work. This step adds costs. You pay your geolocation provider for API calls, and the additional Lambda execution time increases your Lambda charges. To add IP geolocation, sign up for a geolocation API, add this function to the code, and call it where the handler resolves an IP.

import urllib.request
import json

def geo_context(ip_address):
    """Enrich an IP address with geolocation data from your provider."""
    try:
        url = f"https://your-geolocation-api.example/json/{ip_address}"
        with urllib.request.urlopen(url, timeout=5) as response:
            data = json.load(response)
        return {
            "country": data.get("country_name"),
            "city": data.get("city"),
            "org": data.get("org"),
        }
    except Exception as error:
        logger.warning(f"Geolocation lookup failed for {ip_address}: {error}")
        return None

Inside the handler’s loop, after you resolve ip, append the location to the alert.

            if ip:
                geo = geo_context(ip)
                if geo:
                    message += (
                        f" Source location: {geo['city']}, "
                        f"{geo['country']} ({geo['org']})."
                    )

Step 7: Scale to additional patterns and accounts

As your library grows, move the logic into automated pipelines with EventBridge, Lambda, and AWS Step Functions (a serverless orchestration service that coordinates multiple services into workflows), and surface correlations next to findings in Security Hub. For cross-service correlation at scale, CloudWatch unified data and telemetry capabilities can convert security and compliance data into the OCSF format and let you query sources such as CloudTrail, VPC Flow Logs, and DNS logs from one interface. Security Lake with Athena is a strong option for long-term analysis. Choose the endpoint that fits your retention and query needs.

Figure 3 shows a correlation pipeline built on AWS services including EventBridge, Lambda, Step Functions, and AWS Security Hub. The pipeline runs from data sources through scheduled queries and enrichment to automated response and centralized visibility.

Figure 3: A correlation pipeline built on AWS services

Figure 3: A correlation pipeline built on AWS services

Conclusion

You now have four correlation patterns that layer your business context on top of GuardDuty Extended Threat Detection to catch attacks specific to your environment. A few principles carry across every correlation you build.

  • Identity is your primary correlation key: Track the same principal across services.
  • Time windows matter, but they depend on the attack: Events minutes apart are usually related for fast, automated sequences; the ten-minute bins here work for that pattern. Slow or manual reconnaissance can stretch across hours or days, so widen the window when the pattern is deliberate rather than automated.
  • Context is what you add: Your data classification, access norms, resource ownership, and change windows are signals you bring to detection.
  • Start with one rule: A single well-tuned correlation catches more significant activity than a wall of uncorrelated alerts.

GuardDuty Extended Threat Detection handles the multi-stage patterns common across customers. The correlations in this post add the layer that only your business context can supply. Start with one pattern this week, validate it against your own traffic, and add the next pattern after the first proves reliable.

Have you built correlation rules for patterns not covered here? Share your experience in the Comments section below.

Further reading

 

Nisha Kashyap

Nisha Kashyap

Nisha Kashyap is a Senior Support Security Engineer at AWS. She works on threat detection and security operations, helping customers investigate security events and build detection that connects signals across AWS services and reflects their own environment.

  •  

Securing your Amazon S3 buckets: Identifying and remediating over-permissioned access

Misconfigured Amazon Simple Storage Service (Amazon S3) buckets can expose your data to unauthorized access. Without proactive review, S3 bucket policies or Access Control Lists (ACLs) configured with broad access may go unnoticed in your environment. In this post, you learn how to identify and fix over-permissioned S3 buckets across your AWS environment, along with best practice recommendations and automation opportunities to help you prevent security gaps. This post provides a workflow framework and methodology recommendations for your security team to adapt. The focus of this post is on the what and why rather than a prescriptive implementation. You will need to customize the approach based on your organization’s requirements and existing security tooling.

This solution is intended for security engineers, cloud architects, and DevOps teams managing single- or multiple-account AWS environments with Amazon S3 workloads that require access management.

Prerequisites

Before you begin, make sure you have the following in place:

Solution overview

This solution uses a five-phase workflow diagram to detect, remediate, and continuously monitor over-permissioned S3 buckets across your AWS accounts. The following workflow diagram illustrates the high-level end-to-end process for identifying and remediating over-permissioned S3 buckets across your Amazon Web Services (AWS) environment.

Figure 1: Amazon S3 over-permissive access – Detection, remediation, monitoring and cleanup workflow

Figure 1: Amazon S3 over-permissive access – Detection, remediation, monitoring and cleanup workflow

The diagram in Figure 1 consists of five phases:

  1. Setup and prerequisites – Configure AWS Organizations or multi-account access, designate a central security account, deploy AWS Config across all accounts, and enable AWS Security Hub with a central administrator.
  2. Detection and identification – Deploy AWS Config rules (such as s3-bucket-public-read-prohibited and s3-bucket-public-write-prohibited) and run an audit Lambda function that scans each S3 bucket. The function checks three areas: Public Access Block configuration, bucket policy status, and bucket ACL grants. Buckets with issues are added to a risky buckets list. The function then generates a report in CSV and JSON format, uploads it to an output S3 bucket, and sends an SNS alert.
  3. Remediation – Address findings using one or more approaches – Apply restrictive bucket policies to deny public read/write access and restrict access to specific IAM principals; deploy a remediation Lambda function to automatically update bucket policies and disable public access settings; or use CloudFormation StackSets to deploy standardized policies across multiple accounts.
  4. Continuous monitoring – Schedule the audit Lambda function for recurring scans (daily or weekly) using Amazon EventBridge. Use EventBridge to detect policy changes, configure automated notifications for new violations, enable IAM Access Analyzer for S3 to identify external access, and run regular compliance scans.
  5. Resource cleanup – Review and delete resources created during the audit that are no longer needed, including Lambda functions and IAM roles, EventBridge rules, SNS topics and subscriptions, audit output S3 buckets, AWS Config rules, and Security Hub (if enabled only for this audit).

Cost considerations

This section covers the AWS services used in this solution and their associated costs so you can estimate spend before deployment. The primary cost drivers are AWS Config and Security Hub, which scale with the number of accounts and resources you monitor. Lambda, Amazon EventBridge, Amazon SNS, and Amazon S3 typically add minimal costs for most environments. Start with a pilot in one or two accounts to validate costs before scaling.

  • AWS Config – Charges per configuration item recorded and per rule evaluation. Costs scale with the number of accounts and resources tracked.
  • Security Hub – Charges per account per AWS Region for security checks and finding ingestion.
  • Lambda – Charges per request and per GB-second of compute time.
  • EventBridge – Scheduled rules are free. Custom event bus usage might incur charges.
  • Amazon SNS – Charges per notification delivered.
  • Amazon S3 – Storage costs for audit report output files. Minimal for most environments.
  • AWS IAM Access Analyzer – Check the AWS IAM Access Analyzer pricing page to understand which features have costs associated with them.

Check the service pricing pages for current rates. Use the AWS Pricing Calculator to estimate costs for your specific environment before enabling services across all accounts. Consider starting with a pilot in one or two accounts to validate costs before scaling.

Detect and report over-permissioned buckets

This section walks you through setting up the audit environment, deploying the Lambda-based scanner, and generating reports of over-permissioned S3 buckets across your accounts. Follow these steps to identify over-permissioned S3 buckets in your multi-account environment, starting with preparing your environment for an Amazon S3 audit.

To set up the multi-account audit environment:

  1. Set up AWS Organizations or multi-account access. Set up centralized management of your AWS accounts using AWS Organizations or configure cross-account IAM roles.
  2. Choose a central security account. Choose one account as your security/audit account. This account will run the audit Lambda function and collect results from member accounts.
  3. Create an Amazon SNS topic for alerts. Subscribe your security team to receive notifications when over-permissioned buckets are detected. Note the topic Amazon Resource Name (ARN) from the output—you will need it when creating the Lambda execution role (step 6) and the Lambda function (step 9). Confirm the email subscription before testing; Amazon SNS doesn’t deliver alerts until the subscription is confirmed. Learn more in the Amazon SNS Developer Guide.
  4. (Optional): Create an S3 bucket for audit reports. If you plan to use Script v2 for historical reporting and trend analysis, create a dedicated bucket now. Skip this step if you only need real-time alerts using Script v1.
  5. Plan cross-account IAM roles. The central security account needs permission to scan member accounts. Design cross-account roles that:
    1. Grant minimum Amazon S3 read permissions (list buckets, read policies, ACLs, public access configurations).
    2. Include an external ID condition to mitigate the confused deputy problem.
    3. Can be deployed consistently using AWS CloudFormation StackSets.
    4. See the IAM documentation on creating cross-account roles, The confused deputy problem, and IAM security best practices for additional guidance on role configuration and trust policies.

      Note: The specific trust policy and permissions policy for your cross-account roles will depend on organizational requirements. Work with your IAM administrators to grant minimum necessary access for the audit function.

  6. Create the Lambda execution role. Create an IAM role for your Lambda function with the permissions it needs to scan buckets, publish alerts, and write logs. Apply the principle of least privilege—grant only the minimum Amazon S3 read permissions required for the audit (such as, listing buckets, reading bucket policies, ACLs, and public access block configurations), Amazon SNS publish permission for the alert topic created in step 3, Amazon S3 write permission for the output bucket created in step 4 (Script v2), and Amazon CloudWatch Logs permissions. For multi-account scanning, also include sts:AssumeRolepermission for the cross-account role ARNs created in step 5. The AWS Lambda execution role documentation has instructions on creating and configuring execution roles.
  7. To deploy the S3 audit solution Deploy the audit components
    1. Enable AWS Config in member accounts. AWS Config provides compliance monitoring and can detect when S3 buckets are created or modified with public access settings. This will enable the Lambda-based audit to receive real-time detection between scheduled scans. The AWS Config Developer Guide has setup instructions. Deploy pre-defined AWS Config rules to identify overly permissive settings. These managed rules provide automated compliance checking. When AWS Config detects violations, it sends findings to Security Hub (configured in step 8) for centralized visibility alongside the Lambda audit results.
      • s3-bucket-public-read-prohibited
      • s3-bucket-public-write-prohibited
      • Create AWS Config rules for specific permission patterns. For the full list of available rules, see the AWS Config managed rules reference
  8. Enable Security Hub for centralized visibility. Enable AWS Security Hub in member accounts and configure the central security account as the administrator. Security Hub aggregates findings from AWS Config rules (step 7), IAM Access Analyzer (enabled later), and can receive custom findings from your Lambda audit function, providing a single dashboard for Amazon S3 security issues across your organization. See the Security Hub User Guide for setup details.
  9. Deploy the audit Lambda function. Deploy a Python Lambda function using the Boto3 library to list S3 buckets, check their policies, ACLs, and IAM permissions, and identify over-permissioned buckets. See the example scripts that follow.

Important: These code examples aren’t production ready. Adapt them to meet your organization’s requirements and test them in a non-production environment before deployment.

Choose your approach:

  • Script v1 – Best for immediate SNS alerts when issues are detected.
  • Script v2 – Best for historical reports, trend analysis using BI tools.
  • Both scripts – Best for different schedules and ongoing needs.

Audit Lambda function – Example script v1 (Scan and alert)

The following is an example of a Lambda function script for reference purposes. Review, adapt, and test before use in your environment, it scans all S3 buckets in the current account and checks for:

  • Public Access block configuration gaps
  • Bucket policies that allow public access
  • ACL grants to AllUsers

Note: Replace placeholder values with actual values before deployment:

  • <REGION>– Your AWS Region (for example, us-east-1)
  • <ACCOUNT_ID>– Your 12-digit AWS account ID
  • <TOPIC_NAME>– The name of your SNS topic created in step 3
import boto3
import json

def lambda_handler(event, context):
    s3 = boto3.client('s3')
    sns = boto3.client('sns')
    risky_buckets = []
    errors = []

    try:
        buckets = s3.list_buckets()['Buckets']
    except Exception as e:
        return {'statusCode': 500, 'body': f'Failed to list buckets: {str(e)}'}

    for bucket in buckets:
        bucket_name = bucket['Name']
        issues = []

        try:
            # Check Public Access Block — all four settings should be enabled
            try:
                pab = s3.get_public_access_block(Bucket=bucket_name)
                config = pab['PublicAccessBlockConfiguration']
                if not all([
                    config.get('BlockPublicAcls'),      # Block new public ACLs
                    config.get('BlockPublicPolicy'),     # Block new public bucket policies
                    config.get('IgnorePublicAcls'),      # Ignore existing public ACLs
                    config.get('RestrictPublicBuckets')   # Restrict access to public buckets
                ]):
                    issues.append('Public Access Block not fully enabled')
            except s3.exceptions.NoSuchPublicAccessBlockConfiguration:
                issues.append('No Public Access Block configured')

            # Check bucket policy — flag if policy status is public
            try:
                policy_status = s3.get_bucket_policy_status(Bucket=bucket_name)
                if policy_status['PolicyStatus']['IsPublic']:
                    issues.append('Bucket policy allows public access')
            except s3.exceptions.NoSuchBucketPolicy:
                pass  # No bucket policy is acceptable

            # Check bucket ACL
            acl = s3.get_bucket_acl(Bucket=bucket_name)
            for grant in acl.get('Grants', []):
                grantee = grant.get('Grantee', {})
                uri = grantee.get('URI', '')
                # 'AllUsers' = anonymous public access
                # 'AuthenticatedUsers' = any AWS account (still overly permissive)
                if grantee.get('Type') == 'Group' and ('AllUsers' in uri or 'AuthenticatedUsers' in uri):
                    issues.append('Bucket ACL grants public access')
                    break

            if issues:
                risky_buckets.append({'bucket': bucket_name, 'issues': issues})

        except Exception as e:
            errors.append(f'{bucket_name}: {str(e)}')

    # Send alert if risky buckets found
    if risky_buckets:
        message = f'Found {len(risky_buckets)} buckets with public access:\n\n'
        for item in risky_buckets:
            message += f"  {item['bucket']}: {', '.join(item['issues'])}\n"

        sns.publish(
            TopicArn='arn:aws:sns:<REGION>:<ACCOUNT_ID>:<TOPIC_NAME>',
            Subject='S3 Public Access Alert',
            Message=message
        )

    return {
        'statusCode': 200,
        'body': json.dumps({
            'risky_buckets': risky_buckets,
            'errors': errors,
            'total_checked': len(buckets)
        })
    }

Multi-account scanning: This script scans the current account only. To scan across member accounts, see the Multi-account extension section later in this post.

Audit Lambda function – Example script v2 (CSV and JSON report)

The following is an example Lambda function script for reference purposes. Before deploying any script, review error handling, logging, output structure, and permissions. This script generates CSV and JSON output files and uploads them to an S3 bucket for reporting and business intelligence (BI) dashboard integration.

You can deploy both functions with different EventBridge schedules, for example, Script v1 daily for alerts and Script v2 weekly for reports.

Note: Before you deploy this script, replace <OUTPUT_BUCKET_NAME> with the S3 bucket you created for audit reports in step 4.

import boto3
import csv
import json
import os

def lambda_handler(event, context):
    s3 = boto3.client('s3')
    buckets = s3.list_buckets()['Buckets']

    full_access_buckets = []
    for bucket in buckets:
        bucket_name = bucket['Name']
        try:
            bucket_policy = s3.get_bucket_policy(Bucket=bucket_name)['Policy']
            policy = json.loads(bucket_policy)
            for statement in policy['Statement']:
                if (statement['Effect'] == 'Allow'
                    and statement['Principal'] == '*'
                    and 'Action' in statement
                    and 's3:*' in statement['Action']):
                    full_access_buckets.append({'BucketName': bucket_name})
                    break
        except s3.exceptions.ClientError as e:
            if e.response['Error']['Code'] != 'NoSuchBucketPolicy':
                print(f'Error checking bucket policy for {bucket_name}: {e}')

    # Output CSV
    csv_output = os.path.join('/tmp', 'full_access_buckets.csv')
    with open(csv_output, 'w', newline='') as csvfile:
        writer = csv.DictWriter(csvfile, fieldnames=['BucketName'])
        writer.writeheader()
        writer.writerows(full_access_buckets)

    # Output JSON
    json_output = os.path.join('/tmp', 'full_access_buckets.json')
    with open(json_output, 'w') as jsonfile:
        json.dump(full_access_buckets, jsonfile, indent=2)

    # Upload to Amazon S3
    output_bucket = '<OUTPUT_BUCKET_NAME>'
    s3.upload_file(csv_output, output_bucket, 'full_access_buckets.csv')
    s3.upload_file(json_output, output_bucket, 'full_access_buckets.json')

    return {
        'statusCode': 200,
        'body': json.dumps(f'CSV and JSON files uploaded to {output_bucket}')
    }

Important: If this function runs on a schedule, consider implementing a file naming strategy with timestamps to prevent overwriting previous reports or establish a lifecycle policy to manage retention. Include the output bucket in your cleanup procedures when the auditing process is no longer needed.

What if no over-permissioned buckets are found?

If the audit scan returns zero risky buckets, document the clean baseline for future comparison and move to the verification and monitoring phase to so new buckets or policy changes don’t introduce risk over time.

Multi-account extension

The preceding example scripts scan buckets in the current account only. To scan across member accounts in your organization, add the following AssumeRole logic. This function assumes the cross-account IAM role you created during setup, then returns an Amazon S3 client with temporary credentials for each member account.

Note: Before you deploy, configure the following Lambda environment variables:

  • <MEMBER_ACCOUNTS> – Comma-separated list of 12-digit account IDs to scan (for example, 111111111111,222222222222)
  • <CROSS_ACCOUNT_ROLE_NAME> – The IAM role name created in each member account (for example, S3AuditRole)
  • <EXTERNAL_ID> – The external ID configured in the trust policy (for example, s3-audit-external-id)
import boto3
import os

def get_member_s3_clients():
    """
    Assumes the cross-account audit role in each member account
    and returns a list of (account_id, s3_client) tuples.
    """
    sts = boto3.client('sts')
    member_accounts = os.environ.get('<MEMBER_ACCOUNTS>', '').split(',')
    cross_account_role_name = os.environ.get('<CROSS_ACCOUNT_ROLE_NAME>')
    external_id = os.environ.get('<EXTERNAL_ID>')

    clients = []
    for account_id in member_accounts:
        account_id = account_id.strip()
        if not account_id:
            continue

        try:
            assumed_role = sts.assume_role(
                RoleArn=f'arn:aws:iam::{account_id}:role/{cross_account_role_name}',
                RoleSessionName='S3AuditSession',
                ExternalId=external_id
            )

            # Create S3 client with assumed credentials
            s3_client = boto3.client(
                's3',
                aws_access_key_id=assumed_role['Credentials']['AccessKeyId'],
                aws_secret_access_key=assumed_role['Credentials']['SecretAccessKey'],
                aws_session_token=assumed_role['Credentials']['SessionToken']
            )
            clients.append((account_id, s3_client))

        except Exception as e:
            print(f'Failed to assume role in account {account_id}: {e}')

    return clients

To scan each member account, replace the single-account s3.list_buckets() call with a loop over member accounts:

def lambda_handler(event, context):
    all_risky_buckets = []
    all_errors = []

    # Scan each member account
    for account_id, s3_client in get_member_s3_clients():
        try:
            buckets = s3_client.list_buckets()['Buckets']
            for bucket in buckets:
                # ... same scanning logic as the single-account scripts ...
                # Use s3_client instead of s3 for each API call
                pass
        except Exception as e:
            all_errors.append(f'Account {account_id}: {e}')

    # ... same alerting/reporting logic ...

The Lambda execution role in the central security account needs sts:AssumeRole permission for the cross-account role ARNs. Add this to the execution role policy you created in step 5.

Remediate elevated access

This section describes how to fix over-permissioned buckets using account-level controls, bucket policies, and optional automation. Any elevated access that you find needs to be remediated.

Enable Amazon S3 Block Public Access (account level)

Before applying individual bucket policies, enable Amazon S3 Block Public Access at the account level. This prevents buckets in the account from being made public, regardless of individual bucket policies or ACLs. See theS3 Block Public Access documentation for configuration details. See the following example AWS CLI command; replace <ACCOUNT_ID> with the ID of the account you’re using to manage resource access:

aws s3control put-public-access-block \
  --account-id <ACCOUNT_ID> \
  --public-access-block-configuration \
BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

For multi-account environments, deploy this setting across member accounts using AWS CloudFormation StackSets or AWS Organizations service control policies (SCPs).

Important: Before enabling account-level S3 Block Public Access, check whether any workloads need public bucket access (for example, static website hosting, public dataset sharing). Coordinate with your application teams to identify any exceptions.

Remediate using bucket policies

Implement bucket policies that restrict access to specific IAM users, roles, or accounts. When crafting policies, apply the principle of least privilege and include only the actions and principals required for your use case.

Example S3 bucket policy: deny public read/write access. Modify the resource ARN, actions, and conditions to match your requirements:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Principal": "*",
      "Action": [
        "s3:PutObject", "s3:PutObjectAcl",
        "s3:GetObject", "s3:GetObjectAcl",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::<BUCKET_NAME>/*",
      "Condition": {
        "StringEquals": {
          "s3:x-amz-acl": ["public-read", "public-read-write"]
        }
      }
    }
  ]
}

Example S3 bucket policy: restrict access to specific IAM principals. Replace <ACCOUNT_ID>, <USERNAME>, and <ROLE_NAME>:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowObjectAccess",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::<ACCOUNT_ID>:user/<USERNAME>",
          "arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME>"
        ]
      },
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::<BUCKET_NAME>/*"
    },
    {
      "Sid": "AllowBucketAccess",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::<ACCOUNT_ID>:user/<USERNAME>",
          "arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME>"
        ]
      },
      "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
      "Resource": "arn:aws:s3:::<BUCKET_NAME>"
    }
  ]
}

See the Amazon S3 bucket policy documentation for additional examples and guidance.

Automate remediation with Lambda or CloudFormation StackSets (optional):

You can also remediate using Lambda or CloudFormation Stacksets:

  • Create Lambda functions to automatically update bucket policies or disable public access settings for flagged buckets
  • Use CloudFormation StackSets to deploy standardized bucket policies and S3 Block Public Access settings across multiple accounts

Verify your remediation

This section explains how to confirm that your fixes are effective before moving to ongoing monitoring. After applying remediation, verify the fix is effective before setting up ongoing monitoring:

  1. Re-run the audit Lambda function – Confirm the previously flagged buckets no longer appear in the risky buckets list.
  2. Check Security Hub compliance – Verify the compliance status has changed from FAILED to PASSED for Amazon S3-related controls.
  3. Validate with IAM Access Analyzer – Review findings for the remediated S3 buckets. Active findings should resolve automatically after public access is removed.
  4. Test application functionality – Confirm that legitimate workloads continue to function correctly.

Document the verification results for your auditing needs. If any S3 buckets still show issues, investigate whether the policy was applied correctly or if there are conflicting permissions.

Automation opportunities

This section covers optional strategies to automate ongoing detection and maintain your security posture without manual intervention.

  1. (Optional) Schedule recurring scans with Amazon EventBridge
    • Regular security scans help identify new issues arising from configuration changes or newly created S3 buckets. When new security risks are detected, Amazon SNS sends an alert and automatically initiates the remediation phase (Workflow 2 in Figure 1). To avoid repeated alerts, you can configure the audit Lambda function to run on a schedule and compare current results with the previous baseline to generate notifications when new findings are discovered.
    • For ongoing monitoring, you can schedule the audit Lambda function to run on a recurring basis using EventBridge. Create a scheduled rule with a cron expression (for example, daily at 6:00 AM UTC or weekly on Mondays), add the Lambda function as the target, and grant EventBridge permission to invoke it. See Amazon EventBridge scheduling documentation for instructions on creating scheduled rules and configuring targets.
  2. Enable IAM Access Analyzer for Amazon S3
    • IAM Access Analyzer monitors bucket policies, ACLs, and access points to identify buckets accessible from outside your account or organization. Create an analyzer scoped to your organization or individual account, then review findings to identify unintended external access. Findings automatically flow into Security Hub when both services are enabled, giving you a dashboard view for Amazon S3 security findings. See the IAM Access Analyzer documentation for setup and usage instructions.
  3. Automate notifications for policy drift
    • Recurring scans might surface new findings from policy drift or newly created buckets. When new risks are detected, Amazon SNS alert triggers and the remediation cycle repeat (as shown in Workflow 2 in Figure 1) sends email notifications. Configure the audit Lambda function to compare current scan results against the previous baseline and alert on new findings for ongoing reviews.

Clean up

This section lists the resources created during this walkthrough that you should review and remove when they are no longer needed. If the following services were not previously active in your account, leaving them enabled might result in additional ongoing charges. See the Cost considerations section for details. Review and remove unused resources to optimize costs.

Delete or disable the following script-generated resources if they’re not required after outputs are generated. Focus first on Lambda functions and EventBridge rules if you’re not running recurring scans. If you enabled AWS Config or Security Hub specifically for this audit, evaluate whether you need them for other compliance requirements before disabling.

  • Lambda – Functions, IAM roles, and policies created for auditing
  • Amazon EventBridge – Scheduled rules created for recurring audit triggers
  • Amazon SNS – Topics and subscriptions created for notifications
  • Amazon S3 – Buckets containing script-generated audit output files
  • AWS Config – Rules and recorders if no longer needed for compliance
  • Security Hub – Disable if enabled solely for this audit
  • IAM Access Analyzer – Delete the analyzer if no longer needed for ongoing monitoring

Note: Be careful when deleting data and consider temporarily disabling services first to check for dependencies. Only delete resources generated as part of your audit outputs. Verify you have retained any necessary results before proceeding. Verify resources are not used by other workloads before deletion.

Best practices

This section provides recommendations to maintain secure Amazon S3 configurations long-term. To learn more about maintaining secure Amazon S3 configurations, review the AWS documentation links provided in the conclusion. The following recommendations aren’t exhaustive. Adapt and extend them based on your organization’s evolving security requirements and AWS best practices guidance. After you’ve fixed existing issues, these practices help you maintain secure Amazon S3 configurations.

  • Start with account-level controls – Enable S3 Block Public Access at the account level. This prevents buckets from becoming public even if someone misconfigures an individual bucket policy. For multi-account environments, enforce this through AWS Organizations SCPs.
  • Automate detection – Use IAM Access Analyzer to detect external access. Schedule your audit Lambda function with EventBridge to catch new issues weekly or daily, depending on your change frequency. Compare scan results against previous baselines to identify drift.
  • Standardize across accounts – Use CloudFormation StackSets to deploy the same secure configuration to all accounts in your organization, reducing the chance of configuration drift. Use StackSets for IAM roles, AWS Config rules, and S3 Block Public Access settings.

Additional security measures

  • Regularly review and rotate cross-account IAM role credentials and external IDs
  • Implement Amazon S3 server-side encryption (SSE-S3 or SSE-KMS) for data at rest
  • Enable S3 access logging and AWS CloudTrail data events for audit trails

Conclusion

This section summarizes what you accomplished and suggests next steps to maintain your S3 security posture. By implementing the detection, remediation, and monitoring workflow outlined in this post, you can proactively identify and secure over-permissioned S3 buckets across your AWS environment. To maintain your ongoing security posture, enable IAM Access Analyzer for continuous monitoring and schedule recurring audits with EventBridge. To learn more about Amazon S3 security best practices, see Security best practices for Amazon S3

For more information:

If you have feedback about this post, submit comments in the Comments section below.


Hetal Kolekar

Hetal Kolekar

Hetal is a Sr. Technical Account Manager at AWS with more than 21 years of experience in Infrastructure Architecture, Security, Systems Engineering, and Consulting. He excels in leading teams to strengthen their cloud security posture and helps customers scale up their security using AWS services. Hetal is a guitarist and loves playing at church.

Manomayi Vedam

Manonmayi Vedam

Manonmayi is a Senior TAM and Product Owner at AWS, specializing in AI-driven cloud enablement, security, and generative AI risk across Healthcare, Financial Services, Energy, and Public Sector. She co-leads global security programs for Fortune 500 clients, contributes to the NIST Cyber AI Profile RMF and NCCoE, and is a Fellow at SCRS with recognition from GlobeeAwards and IEEE.

Fernando Freitas

Fernando Freitas

Fernando is a Sr. Technical Account Manager at AWS in Salt Lake City, focused on helping customers achieve their desired outcomes with the AWS Cloud. Fernando is passionate about Identity and Security, Training and Education.

  •  

Automate certificates with ACME support in AWS Certificate Manager

Customers tell us that managing TLS certificates at scale is one of their biggest operational concerns. The Certification Authority Browser Forum (CA/Browser Forum) has mandated a phased reduction in maximum certificate validity for public certificates. By March 2027, the maximum validity drops to 100 days. By March 2029, it lasts for 47 days. For an organization managing 1,000 certificates, the final transition means roughly 30 renewal events every day. Renewal and rotations of renewed certificates at that cadence isn’t something manual processes or ticket-driven workflows can sustain at scale.

We recently announced Automated Certificate Management Environment (ACME) protocol support in AWS Certificate Manager (ACM). With this launch, you can use the ACME clients your teams already know, including popular open source tools like certbot, cert-manager, acme.sh, and win-acme, to automate public certificate issuance and renewal for your infrastructure. Customers that are using third-party certificate authorities (CAs) can point their existing ACME-compatible clients at ACM instead of their current CA, with minimal reconfiguration. This applies whether it’s running on Amazon Web Services (AWS), on premises, or in a hybrid environment. Certificates created through ACME are registered in ACM, giving you a unified view of your entire certificate inventory.

This post covers how the feature works, how to get started, and the controls and best practices to help you manage certificate issuance at scale.

Background

ACME is an open source protocol that automates the process of verifying domain ownership and issuing certificates and has become a standard mechanism for certificate automation. While ACM has long provided managed certificate issuance and renewal for AWS-integrated services such as Elastic Load Balancing (ELB), Amazon CloudFront, and Amazon API Gateway, many customers also need to automate certificates for their own infrastructure, including servers they manage in their data centers, Kubernetes clusters, Internet of Things (IoT) fleets, and hybrid environments. Until now, those customers had to turn to external providers. This launch brings the ACM automation model to that same infrastructure, using the standard ACME protocol with AWS managed certificate endpoints.

How it works

The feature introduces a new centrally provisioned and managed resource type: the ACME endpoint. Each endpoint is an AWS resource with a unique ACME directory URL and AWS Identity and Access Management (IAM)-based access controls. You create and manage endpoints through the ACM API or AWS Management Console, and point your existing ACME clients at the endpoint URL. Certificates issued through your endpoint are automatically registered with ACM, appearing in your certificate inventory alongside certificates created by the RequestCertificate and ImportCertificate API calls.

The architecture separates into two planes. In the control plane, PKI administrators use ACM APIs to create ACME endpoints, pre-approve the domains an endpoint is allowed to issue for, and generate external account binding (EAB) credentials. In the data plane, ACME clients register with an endpoint using EAB credentials and request certificates for domains the administrator has already validated. This architecture is how we provide customers the ability to scale. Instead of each client proving domain ownership on every request, a principal with appropriate ACM permissions (typically your PKI administrator) validates domains once at the endpoint level, and then application owners don’t need DNS credentials to get a certificate.

Adding to the data plane, EABs control client access to the endpoints. Each EAB is bound to an IAM role that controls what certificate operations the ACME client can perform, and credentials you generate in ACM are distributed to authorized ACME clients. An ACME client authorized for one endpoint can’t use a different endpoint. This creates security boundaries between environments. For example, a client authorized for your development endpoint can’t obtain certificates from your production endpoint.

Figure 1 shows the ACME request flow through ACM. An ACME client authenticates to an ACME endpoint using EAB credentials. The endpoint routes certificate orders to Amazon Trust Services for issuance. Issued certificates are registered in ACM inventory, where Amazon EventBridge and AWS CloudTrail provide expiration alerting and audit logging.

Figure 1: An ACME architecture and workflow

Figure 1: An ACME architecture and workflow

Getting started

Getting started with the new ACME feature in ACM is straightforward. Use the following steps to create your first ACME-generated certificate.

Prerequisites

  • An AWS account with permissions to create and manage ACM resources
  • An ACME client installed on your infrastructure (for example, Certbot, cert-manager, acme.sh, or others)
  • AWS Command Line Interface (AWS CLI) installed on your device (see this blog post for the console equivalent)
  • Amazon Route 53 hosted zone for your domain, or the ability to create a CNAME record with your DNS provider

Step 1: Create an ACME endpoint

Before you can use ACME clients with ACM, you need to create an ACME endpoint. This endpoint provides the URL that your ACME clients will use to request certificates.

  1. Run the following command from the AWS CLI to create an ACME endpoint:
    aws acm create-acme-endpoint \
      --authorization-behavior PRE_APPROVED \
      --certificate-authority '{"PublicCertificateAuthority":{"AllowedKeyAlgorithms":["EC_prime256v1"]}}
  2. Note the endpoint Amazon Resource Name (ARN) from the response.
    {"AcmeEndpointArn": "arn:aws:acm:us-east-1:123456789012:acme-endpoint/11111111-2222-3333-4444-555555555555"}
  3. Run the following command to retrieve the endpoint URL, replacing the ARN with your endpoint ARN:
    aws acm describe-acme-endpoint \
    --acme-endpoint-arn arn:aws:acm:us-east-1:123456789012:acme-endpoint/11111111-2222-3333-4444-555555555555
  4. Save the output of the ACME EndpointUrl:
    {
        "AcmeEndpoint": {
            "AcmeEndpointArn": "arn:aws:acm:us-east-1:123456789012:acme-endpoint/11111111-2222-3333-4444-555555555555",
            "EndpointUrl": "https://acm-acme-enroll.<region>.api.aws/6666666-7777-8888-9999-000000000000/directory",
            "Status": "ACTIVE",
            "AuthorizationBehavior": "PRE_APPROVED",
            "Contact": "REQUIRED",
            "CertificateAuthority": {
                "PublicCertificateAuthority": {
                    "AllowedKeyAlgorithms": [
                        "EC_prime256v1"
                    ]
                }
            },
            "CreatedAt": "2026-07-14T18:23:58.876000-04:00",
            "UpdatedAt": "2026-07-14T18:23:58.876000-04:00"
        }
    }
    

Step 2: Pre-approve a domain

Before ACME clients can request a certificate, the administrator validates the domain using DNS once at the endpoint level. Use DomainScope to control exactly which certificate patterns are allowed:

  • Enabling only ExactDomain restricts clients to that specific name,
  • Subdomains enabled allows names like api.example.com,
  • Wildcards enabled allows *.example.com.

Leave a scope disabled to block that pattern outright, even if an otherwise-valid ACME request asks for it. For a production endpoint, consider enabling only ExactDomain and Subdomains and leaving Wildcards disabled for a stricter posture.

aws acm create-acme-domain-validation \
--acme-endpoint-arn arn:aws:acm:us-east-1:123456789012:acme-endpoint/11111111-2222-3333-4444-555555555555 \
--domain-name example.com \
--prevalidation-options '{"DnsPrevalidation":{"DomainScope":{"ExactDomain":"ENABLED","Subdomains":"ENABLED","Wildcards":"DISABLED"},"HostedZoneId":"Z1234567890ABC"}}'

If your domain is hosted in Route 53, specifying HostedZoneId lets ACM create the required CNAME record automatically. If your domain is hosted elsewhere, omit it and create the provided CNAME record manually with your DNS provider. Validation typically completes within a few seconds after the record is in place.

You will receive the following response back:

{
    "AcmeDomainValidationArn": "arn:aws:acm:us-east-1:123456789012:acme-endpoint/1111111-2222-3333-4444-555555555555/acme-domain-validation/6666666-8888-9999-0000-11111111111"
}

Step 3: Generate EAB credentials

EAB credentials authenticate your ACME clients to your endpoint. Generate a unique set of credentials for each client or environment to maintain security boundaries.

  1. Run the following command to generate your EAB credentials, adjusting your expiration to fit your organization’s risk profile:
    aws acm create-acme-external-account-binding \
        --acme-endpoint-arn arn:aws:acm:region:111122223333:acme-endpoint/00000000-0000-0000-0000-000000000000 \
        --role-arn arn:aws:iam::111122223333:role/AcmeIssuanceRole \
        --expiration '{"Value": 7, "Type": "DAYS"}'
  2. Note the response from a successful invocation of the command
    {
        "ExternalAccountBinding": {
            "AcmeExternalAccountBindingArn": "arn:aws:acm:region:111122223333:acme-endpoint/00000000-0000-0000-0000-000000000000/acme-external-account-binding/1234567-1234-1234-1234-123456789012",
            "AcmeEndpointArn": "arn:aws:acm:region:111122223333:acme-endpoint/00000000-0000-0000-0000-000000000000",
            "RoleArn": "arn:aws:iam::123456789012:role/service-role/AcmAcmeIssuanceRole-XXXXXXXX",
            "ExpiresAt": "2026-07-21T18:47:50.641000-04:00"
        }
    }
    
  3. Run the following command to retrieve the credentials. You’ll need these values for your ACME client configuration the next step.
    aws acm get-acme-external-account-binding-credentials \
        --acme-external-account-binding-arn arn:aws:acm:region:111122223333:acme-endpoint/00000000-0000-0000-0000-000000000000/acme-external-account-binding/22222222-2222-2222-2222-222222222222
  4. Save the KeyId and MacKey for the next step.
    {
        "KeyId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
        "MacKey": "xxxxxxxx-xxxxxxxxxx-xxxxxxxxxxxxxxx"
    }

Step 4: Configure your ACME client

With your endpoint URL and EAB credentials ready, you can now configure your preferred ACME client. The following examples show configuration for two popular clients. As a reminder, the server information was retrieved in step 1, part 4 as the EndpointUrl.

acme.sh:

acme.sh --issue --server https://acm-acme-enroll.us-east-1.api.aws/123457-1234-1234-123456789012/directory \
    --eab-kid <KeyId> --eab-hmac-key <MacKey> \
    --email <EMAIL> \
    -d <DOMAIN> \
    --dns --yes-I-know-dns-manual-mode-enough-go-ahead-please

Certbot:

certbot certonly --standalone --non-interactive --agree-tos \
  --email <EMAIL> \
  --server https://acm-acme-enroll.us-east-1.api.aws/1234567-1234-1234-123456789012/directory \
  --eab-kid <KeyId> \
  --eab-hmac-key <MacKey> \
  -d <DOMAIN>

After the initial registration, your ACME client handles renewals.

Enterprise controls

Other ACME alternatives can provide certificates but don’t give the same amount of control and governance for customers that need to scale their certificate environment. The following controls are available to help reduce risk across your organization.

Domain validation

Customers managing large numbers of domains told us they need a way to prevent unauthorized certificate issuance across their domain space. Domain validation gives you this control. For each domain you validate, you enable the certificate patterns it should be allowed to issue, whether it’s ExactDomain, Subdomains, or Wildcards. For example, if you validate internal.example.com and enable only Wildcards, an ACME client can request *.internal.example.com but a request for internal.example.com itself or api.internal.example.com is rejected. This enforcement happens at the endpoint level, before requests reach the ACM certificate authority, and you can validate multiple domains under a single endpoint, each with its own scope.

Centralized certificate visibility

Certificates issued through your ACME endpoints are registered with ACM. You can use the aws acm list-certificates command to see all your issued certificates.

IAM authorization, CloudTrail audit logging and observability

Endpoint management operations are authorized through IAM and logged to CloudTrail. You can use IAM policies to control which principals can create endpoints, generate EAB credentials, and manage domain constraints.

Best practices

For customers implementing ACME certificates for the first time, consider the following best practices for your organizations.

Segment endpoints along organizational or environment boundaries

The endpoint serves as a useful method of isolation for larger organizations. A large enterprise can create one endpoint per organizational boundary (business unit, subsidiary, or environment) instead of a single shared endpoint company-wide. Each endpoint has its own pre-approved domains and its own set of EABs, so a compromised credential in one business unit has no path to certificates in another.

However, weigh this against your operational overhead as well. A reasonable starting point is one endpoint per environment (dev, staging, andprod) within a business unit, expanding to per-business-unit endpoints only where compliance or organizational requirements call for it.

Manage EAB credentials securely

Anyone holding a validKeyIdandMacKeyfor an endpoint can obtain certificates for any domain pre-approved on that endpoint, so these credentials deserve the same handling you’d give an access key.

  • Avoid hard coding theMacKeywhere possible by using a secret store such as AWS Secrets Manager. Distribute it only to the ACME clients that you authorize to use the endpoint.
  • Set the expiration of the EAB to an acceptable level. While EAB supports long-lived credentials, not all scenarios require an indefinitely long EAB.
  • When creating the role for each EAB, adhere to concept of least privilege. Creating a role per EAB, rather than sharing a role across all bindings, can help reduce risk in your AWS environment.
  • Audit CreateAcmeExternalAccountBinding and GetAcmeExternalAccountBindingCredentials calls in CloudTrail separately. Because retrieving the actual key material is a distinct API call from creating the binding, alerting on retrieval events is a stronger signal of real credential distribution than binding creation alone.

Automate how EABs are associated with clients at runtime

Generate a unique set of EAB credentials for each client or environment rather than sharing one binding across multiple ACME clients. As you begin to scale with multiple endpoints, usesome of the following patterns to reduce operational toil.

  • Name each EAB and its bound IAM role after the client it belongs to (team, application, environment), so the binding’s purpose is obvious from DescribeAcmeExternalAccountBinding output alone, without cross-referencing a spreadsheet.
  • Store each client’s KeyId and MacKey under a secrets path scoped to that client (for example, a Secrets Manager path per team and environment), and let the client’s provisioning pipeline retrieve its own credentials.
  • In Kubernetes, use one ClusterIssuer or namespace-scoped Issuer per EAB rather than one shared issuer across teams. This keeps the client-to-EAB association explicit in cluster config, and lets you revoke one team’s access without touching anyone else’s.
  • For ephemeral infrastructure (build agents, autoscaled fleets), provision EAB credentials as part of your infrastructure-as-code or continuous integration and deployment (CI/CD) pipeline instead of a one-time manual handoff, so credential lifecycle tracks infrastructure lifecycle.

Monitor your deployment of ACME

ACME’s power is through automation, and organizations should monitor their ACME usage for anomalies.

  • Alarm on issuance failures, not just successes. At 45-day certificate validity, a silent renewal failure gives you far less runway to react than the months of time you might be used to with longer-lived certificates.
  • Test renewal automation before you depend on it. Force a manual renewal against a non-production endpoint and confirm your client, monitoring, and on-call runbooks behave as expected, before the CA/Browser Forum’s shortened validity windows turn a failed renewal into a disruptive event for your organization.

Availability and pricing

ACME support in AWS Certificate Manager is available today in all commercial AWS Regions and will be available in AWS GovCloud (US), the China Regions, and the AWS European Sovereign Cloud partitions at a later date. See the ACM pricing page for more information on ACME pricing.

Conclusion

The phased reduction in certificate validity can’t easily be solved without automation. ACME support in ACM gives you that automation through a standard protocol and standard tooling, while keeping the visibility and governance controls your security teams rely on from ACM.

To get started, see the AWS Certificate Manager documentation or follow the getting started guide.

If you have feedback about this post, submit comments in the Comments section below.


Anthony Harvey

Anthony Harvey

Anthony is a Senior Security Specialist Solutions Architect for AWS in the worldwide public sector group. Prior to joining AWS, he was a chief information security officer in local government for half a decade. With his public sector experience, he has a passion for figuring out how to do more with less and leveraging that mindset to enable customers in their security journey.

Chandan Kundapur

Chandan Kundapur

Chandan is a Principal Product Manager on the AWS Certificate Manager (ACM) team. With over 15 years of cybersecurity experience, he has a passion for driving PKI product strategy.

  •  

Balancing speed and safety: A control framework for AI coding agents

AI coding agents are part of the developer toolchain. Tools like Kiro and Claude Code generate features, tests, and code refactors from natural-language prompts. A single agent can open dozens of pull requests (PRs) across your repositories in an afternoon. That productivity comes with a trade-off: agents optimize for task completion at machine speed with no understanding of your organization’s risk.

Through protocols like the Model Context Protocol (MCP), agents also reach beyond the integrated development environment (IDE) to call APIs, query databases, and modify infrastructure and even entire environments, expanding the scope of resources your application security team defends.

This post lays out an application security (AppSec) control framework for AI coding agents. Two pillars organize the framework: author-time controls shape what the agent produces in the IDE; build-time controls verify and gate what reaches production. Your existing secure software development lifecycle (SDLC) controls still apply and are critical to a defense-in-depth security strategy. The framework shows where to layer additional guardrails so AppSec scales with agent-driven development. The framework is tool-agnostic and cloud-agnostic. Throughout, we use AWS services—Kiro in the IDE and AWS CodePipeline in the build—as a running example that you can adapt to your own toolchain.

Risks

Each of the following risks includes a treatment summary. The control framework section later in this post provides implementation details. The risks are ordered by severity with the highest impact risks first.

R001. Prompt and context injection

Agents read untrusted content, such as issue descriptions, web pages, MCP responses, and README files in third-party packages. Text from outside parties can redirect the agent to disclose secrets, open unauthorized PRs, or invoke tools without user consent. This risk, known as prompt injection, is the top risk in the OWASP Top 10 for LLM Applications. Any agent that reads content from outside parties is exposed, with or without MCP, so connecting tools widens the scope of impact.

Treatment: Treat non-developer input as untrusted. A large language model (LLM) can’t reliably separate instructions from data in a single context window, so architect for it: keep the agent that orchestrates trusted actions separate from the one exposed to untrusted content and grant the exposed agent only read-only, least-privilege access. Require human approval for irreversible actions. Use version-control steering files to prevent silent tampering.

R002. Inadvertent data disclosure and overly permissive configurations

Agents optimize for getting work done. Left unchecked, the code they generate can default to wildcard identity and access management policies, open security groups, and unencrypted storage, or embed sensitive values in code rather than referencing a secrets manager. Most coding agents now include safety mechanisms that make these outcomes less likely, but they remain imperfect, so you still need controls to account for the possibility.

Treatment: Security requirements in a steering document, plus policy-as-code scanning (Checkov, cfn-nag) in the IDE and pipeline. See Context as a security control.

R003. Uncontrolled changes reaching production

Ungated code reaching production isn’t new, but AI agents amplify it. Machine-speed generation can propagate a flawed pattern across repositories before it’s identified.

Treatment: Branch protection rules requiring PR approval (a human-in-the-loop checkpoint), pre-commit hooks for security checks, and sandboxed agent runs that prevent direct pushes to protected branches. The right balance between human review and automated speed depends on the risk profile of the change. For many low-risk paths, automated checks alone might suffice, while higher-risk changes warrant a human checkpoint.

R004. Supply chain risks

Agents don’t always distinguish current best practices from outdated patterns. They might recommend deprecated packages, reference library versions with new Common Vulnerabilities and Exposures (CVEs), and hallucinate package names that don’t exist, which can introduce risks of dependency confusion issues.

Treatment: Software Composition Analysis (SCA) in the pipeline (for example, Amazon Inspector code scanning or Dependabot) to flag vulnerable or unexpected dependencies. For additional control, resolve against a scoped registry like AWS CodeArtifact. Even without a fully curated registry, lockfile validation and allow-listing critical packages reduce exposure.

R005. Uncontrolled external access

Through MCP and tool integrations, agents query databases, call APIs, and modify infrastructure. Without constraints on which tools and data an agent can reach, a single misconfigured integration provides unintended access to sensitive resources.

Treatment: Scope MCP servers to least-privilege tools and resources, enforce authn or authz on external connections, and audit tool invocations. The control point is the configuration file. Review it the same way you review AWS Identity and Access Management (IAM) policies.

R006. Hallucinations and incorrect code

Agents produce plausible-looking output. Code that compiles, passes linting, and looks reasonable can still be functionally wrong: misusing APIs, introducing subtle logic errors, or implementing security-sensitive operations incorrectly. Code that passes continuous integration (CI) but is wrong slips through review; code that fails to build is caught immediately.

Treatment: Layer deterministic verification (static application security testing (SAST), unit tests) with non-deterministic review (LLM-assisted screening against the specification). Neither catches everything alone.

R007. Scope creep

Given a bug-fix prompt, an agent might also refactor surrounding code, disable an unreliable test, or reorganize imports. Unrequested changes introduce regressions and complicate review.

Treatment: A reviewed specification document that defines what must change and what must not, paired with a targeted review of the proposed changes. See Specifications as scope boundaries.

The preceding risks share a common thread: agents produce output faster than humans can review it, and they lack context to self-correct.

The following framework addresses this gap. It organizes controls into two pillars: author-time (pre-generation and post-generation of code) and build-time (in the pipeline, before code reaches production). Author-time controls shape what the agent produces. Build-time controls verify it. Neither is sufficient alone; together they reduce the volume and severity of issues that reach human reviewers.

Deterministic compared to non-deterministic mitigations

Deterministic mitigations [D] produce the same result every time. Linters, SAST scanners, secrets detection, and policy-as-code match patterns against rules and define security invariants: no critical findings, no hardcoded secrets, and no wildcard IAM policies. Use them when the condition can be expressed as a rule. Organizations already have these and must continue enforcing them.

Non-deterministic mitigations [ND] use model judgment. They include steering documents, LLM-as-judge review, specification compliance checks, and scope-creep detection, and they evaluate intent rather than patterns. They catch novel issues that rules miss, but are probabilistic. Use them when evaluation requires context or reasoning across files. This is the new layer that AI-generated code demands, because agents produce code that can pass every deterministic check yet remain functionally wrong.

Human review [H] provides the final layer for the risk-based decisions neither tool type can make. Apply it where judgment is needed, not everywhere: routing every change to a person invites consent fatigue, where reviewers approve by reflex and the control loses its value. The default reflex is to route everything back to a human, but that isn’t always the right response—reserve human judgment for the decisions that genuinely need it.

The control framework

The framework organizes controls into two pillars. Author-time controls (Pillar 1) shape what the agent produces in the IDE, before code is generated and just after. Build-time controls (Pillar 2) verify and gate that output in the pipeline, before it reaches production. The controls within each pillar are tagged deterministic [D], non-deterministic [ND], or human [H].

Pillar 1: Author-time controls (pre- and post-generation of code)

Author-time controls work inside the IDE, where the developer and agent still hold full context. They shape the prompt and the generated output before it ever reaches a pull request. The following controls apply at this stage.

Context as a security control [ND]

Control statement: Encode security invariants as natural-language constraints in a steering document that every developer environment consumes at session start. Addresses R002.
Many AI coding agent risks share one root cause: the agent lacks the security context an experienced developer carries implicitly. Your security team sets the policies, such as Amazon Simple Storage Service (Amazon S3) buckets require encryption, API gateways require mutual TLS, and credentials must come from AWS Secrets Manager. Developers don’t always have these requirements available when they’re building. They build what works, not what’s compliant. An AI agent amplifies this gap because it defaults to whatever pattern dominated its training data, with no awareness of your organization’s security posture.

A key mitigation is steering. Security teams write these invariants once as natural-language guidance in a steering document, then distribute them as shareable resources that developers consume in their IDE. The agent loads the file at session start and treats the contents as standing requirements:

  • IAM policies must follow least-privilege principles; no wildcard Amazon Resource Names (ARNs).
  • No hardcoded credentials in source code; use a secrets manager.
  • Security groups must not allow unrestricted inbound access.

This shifts security left, before code generation begins. Steering biases generation toward secure defaults; it doesn’t guarantee them. Treat it as a strong default, paired with the following deterministic gates that block non-compliant code from merging. Security teams define the rules once and every developer environment inherits them automatically. Steering reduces the volume of issues that reach the pipeline, though it doesn’t replace downstream scanning.

How to write effective steering rules: Keep each rule specific and testable, scope it to a concrete risk class, keep the rule set concise so the agent can hold it in context, and iterate from the issues your scanners and reviewers surface.

Specifications as scope boundaries [ND]

Control statement: Require a reviewed specification before code generation begins. Define what must change and what must not. Addresses R007.

Spec-driven workflows turn vague prompts into reviewable specifications before code is generated. This creates a human checkpoint at the design phase, where security decisions are made:

  • Requirements use testable notation that’s auditable before the agent writes a line of code. For example, the Easy Approach to Requirements Syntax (EARS): WHEN [condition] THE SYSTEM SHALL [behavior].
  • Tasks are ordered in implementation steps, each mapped back to a requirement.

For bug fixes, specifications add a critical element: unchanged behavior documentation. This is an explicit list of behaviors that must continue working, giving the agent a written boundary against scope creep.

In this model, the specification becomes the primary artifact, code is a derivative of it. Human review effort concentrates on whether the specification solves the right problem with the right constraints, not on reading implementation diffs line by line.

Controlled tool access using MCP [D + ND]

Control statement: Scope each MCP server to the minimum set of tools the agent needs, and give it a dedicated, scoped-down credential rather than the developer’s own. Maintain an allowlist of reviewed MCP servers. Addresses R005.

MCP servers act as controlled gateways between the agent, the external tools, and data:

  • Dependency management – An MCP server fronting your private package registry resolves dependencies against curated packages, not the public internet. This is a deterministic constraint on supply chain risk.
  • Infrastructure tooling – Visibility into current resource configurations prevents templates that conflict with existing infrastructure.
  • Scoped permissions – Each MCP server exposes a defined set of tools and resources. You choose exactly what the agent can access, supporting least-privilege at the integration layer. You supply that credential through the agent’s configuration (in Kiro, the env block of .kiro/settings/mcp.json). Avoid autoApprove: ["*"], which removes the human approval prompt on every tool call.

IDE code scanning [D]

Control statement: Run real-time static analysis in the IDE so security issues surface while the developer (and agent) still have full context. Addresses R002, R006.

Real-time diagnostics catch syntax errors, type mismatches, and configuration issues as the developer types. A malformed IAM policy is flagged before the agent builds further on it. Security-focused extensions (ESLint security plugins, Checkov, SAST) layer on top for immediate feedback while code is fresh in context.

Hooks: Automated guardrails at the point of action [D + ND]

Control statement: Attach deterministic checks to file-save events and non-deterministic verification to task-completion events. Addresses R002, R007.

  • Shell command hooks [D] – Triggered on file save, these run a linter, formatter, or security scanner and produce the same result every time. They enforce hard rules.
  • AI-powered hooks [ND] – Triggered on task completion. These prompt the agent to verify that the implementation matches the specification and check for any untested edge cases or files that were modified outside the task’s scope.

Pillar 2: Build-time controls (in the pipeline)

Build-time controls run in the pipeline after code is committed and before it reaches production. They verify and gate what the agent produced, catching what author-time controls did not. The following controls apply at this stage.

Layered security scanning [D]

Control statement: Run secrets detection, static analysis, dependency scanning, and infrastructure-as-code scanning in sequence. Fail the build on any critical finding. Addresses R002, R003, R004.

  1. Secrets detection runs first because it’s cheapest and addresses a high-severity class of issue. It scans for hardcoded API keys, database connection strings, and credentials that AI agents might inadvertently include.
  2. SAST scans source code for injection issues, insecure deserialization, and resource leaks. Custom rules can target AI-specific anti-patterns including overly broad exception handling, deprecated APIs, placeholder credentials, dynamic code execution through eval().
  3. Software Composition Analysis (SCA) identifies known CVEs in dependencies. This is critical for AI-generated code, which might reference deprecated packages or hallucinate package names that open you to dependency confusion issues.
  4. Infrastructure as code (IaC) scanning validates AWS CloudFormation, Terraform, and AWS Cloud Development Kit (AWS CDK) templates against security policies before deployment. Catches overly permissive IAM roles, unencrypted storage, and public-facing resources the agent created.

Each stage halts the pipeline on failure. Results export to a standard format (Static Analysis Results Interchange Format (SARIF)) for compliance auditing and flow downstream to human reviewers. The open source Automated Security Helper (ASH) bundles secrets, SAST, SCA, and IaC scanners behind one command that you can run locally and in AWS CodeBuild, emitting SARIF for the gates that follow.

Quality gates [D]

Control statement: Define pass/fail thresholds for each scan type. Block deployment on any critical or high-severity finding. Addresses R003.

Quality gates convert scan results into go/no-go decisions. Define thresholds for each severity: block on critical findings, require justification for highs, and track mediums. The gate is deterministic: if a threshold is breached, the pipeline stops. Exceptions require documented approval.

Differentiate blocking compared to advisory modes: hard failures on main, advisory on feature branches. Avoid gates becoming a friction that teams route around.

AI-assisted review [ND]

Control statement: Use an LLM reviewer to pre-screen every pull request for specification compliance, scope creep, and security anti-patterns before human review. Addresses R001, R006, R007.

  • Specification compliance – Does the implementation match the requirements document?
  • Scope verification – Were files modified outside the task’s stated scope?
  • Security pattern review – Are there logic errors, misused APIs, or insecure patterns that pass SAST but violate intent?

This pre-screening focuses human reviewer attention on genuine risks rather than formatting or obvious issues. On AWS, AWS Security Agent (code review in preview at publication) checks pull requests against AWS-managed and custom security requirements. The reviewer screens and surfaces findings; the merge decision stays with a human.

A critical principle: the agent that wrote the code should not be the agent that reviews it. A separate session helps avoid self-confirmation bias, but a separate session alone doesn’t always avoid the generator’s blind spots, because two sessions of the same model can share them. Where practical, use a different model for review so the reviewer is less likely to inherit the same systematic weaknesses.

Human-in-the-loop review [ND + H]

Control statement: Require human approval on most pull requests, especially those touching security-sensitive or high-blast-radius code. Lower-risk changes might be eligible for agent-assisted or fully automated approval as tooling matures. Provide reviewers with scan results, LLM pre-screening output, and specification context to enable fast, informed decisions. Addresses R003.

Scale review depth to the risk of the change. Low-risk or boilerplate changes can take a lighter-touch review, while security-sensitive or novel-logic changes warrant mandatory deep review and a second reviewer.

Scanners catch known patterns but can’t judge whether code implements the intended business logic. Human review also serves to calibrate trust: teams build intuition about where agents excel (boilerplate, test writing) and where they’ve tended to struggle (novel business logic, security-sensitive operations), recognizing that this frontier shifts as models improve.

Place two approval gates: after security scans (reviewer focuses on correctness and business logic, with scan results as context) and before production deployment (final sign-off after integration testing). Treat human review as a secondary control, not a guarantee: reviewers are themselves non-deterministic and can miss issues, so human review layers on top of the deterministic gates rather than replacing them.

Putting the framework into practice on AWS

The framework is tool-agnostic, but AWS gives you building blocks for each pillar. The following services map directly to the controls described previously: Kiro for author-time guardrails, and CodeBuild and CodePipeline for build-time gates.

Kiro: Structured AI development

Kiro maps to Pillar 1: It puts the author-time controls in the IDE, where the developer and agent still share full context. Each feature in the following list implements one of those controls, configured in-repo under .kiro/ so the guardrails are version-controlled and shared across the team rather than set per developer.

  • Steering documents – Markdown files in .kiro/steering/ load into the agent’s context at session start. Conditional inclusion using fileMatch (for example, ["**/*.tf"]) loads IaC-specific rules only when relevant.
  • Specification-driven workflows – Three-phase specifications (requirements in EARS, design, and tasks) with review checkpoints. Bug-fix specifications capture unchanged behavior explicitly.
  • Agent hooks – Triggered on file save, tool invocation, or task completion. Shell hooks run deterministic checks (linters, tests); Ask Kiro hooks run AI prompts for non-deterministic review. For example, a security pre-commit scanner hook can flag hardcoded credentials when the agent finishes a task.
  • Property-based testing – Guided by a specification or hook, Kiro can generate property-based tests (for example, using the hypothesis library) that exercise hundreds of randomized inputs, probing edge cases a hand-written test suite would miss.
  • MCP integrations – Connect Kiro to private package registries, internal docs, issue trackers, and infrastructure tooling, creating the controlled tool access pattern.

For enterprise environments, Kiro supports AWS IAM Identity Center for single sign-on and provides IP indemnity coverage for subscribers. Check the Kiro documentation for current Region availability.

AWS CodeBuild and AWS CodePipeline: Pipeline controls

CodeBuild runs each scanning tool (checking for secrets, SAST, SCA, and IaC) as a build action. A non-zero exit code fails the action, and the stage halts or rolls back according to its OnFailure setting. Findings export as SARIF to Amazon S3 for compliance, and CodePipeline action variables pass results to downstream approval actions.

  • CodeBuild exit codes halt the pipeline on scan failures
  • AWS Lambda invoke actions evaluate scan results against configurable thresholds and return pass/fail decisions
  • Manual approval actions halt the pipeline, send Amazon Simple Notification Service (Amazon SNS) notifications, and link to review artifacts; decisions and reviewer identity are logged for audit

The following table consolidates the framework into a single view that includes each stage of the SDLC and the deterministic [D] and non-deterministic [ND] controls that apply there. Every stage carries both, a reminder that neither control type is sufficient on its own.

Stage Deterministic [D] Non-deterministic [ND]
IDE (pre-generation) Steering files loaded Steering documents, specification-driven constraints
IDE (post-generation) Shell hooks: Linter, formatter, type checker, and secrets scan AI-powered task completion hooks, context constraints
Pull request SAST, SCA, and IaC scanning LLM PR pre-screening and scope verification
Pipeline (pre-deploy) Full security scan suite, integration tests, and policy-as-code AI-assisted review for human approvers
Post-deploy Runtime monitoring and anomaly detection AI-powered incident triage

Conclusion

This post laid out a framework for adopting AI coding agents at machine speed without letting unreviewed risk reach production. It layers guardrails at two points:

  • Author-time controls – Steering, specs, and scoped tools shape what the agent generates in the IDE.
  • Build-time controls – Scanning, quality gates, and layered review verify it before it reaches production.

No single layer is enough: deterministic gates enforce hard rules, non-deterministic review catches what they miss, and human judgment is reserved for the decisions that need it. Together, they let AppSec scale with agent-driven development.

Where to start this week:

  1. Start with steering and specs – Encode security requirements as steering and use specifications for new features. Highest impact, lowest effort. For a ready-made starting set, the open source Project CodeGuard (a Coalition for Secure AI project under OASIS Open, of which Amazon is a contributing member) publishes reusable steering rules for common risk classes—hardcoded credentials, IaC misconfiguration, supply chain, and MCP security—that you can adapt to your AWS environment.
  2. Add deterministic pipeline gates – Integrate SAST, SCA, and secrets detection. Table-stakes regardless of AI usage.
  3. Calibrate and iterate – Review what controls catch, adjust steering for recurring issues, and expand agent autonomy as trust builds.
  4. Accountability – Developers remain accountable for the security of what they ship. AI agents accelerate development; they don’t transfer ownership.

More information:

If you have feedback about this post, submit comments in the Comments section below.


Daniel Begimher

Daniel Begimher

Daniel is a Senior Security Engineer at AWS, where he built and shipped the company’s first customer-facing AI security agent. He created SIR-Bench, a benchmark for measuring how deeply AI incident-response agents investigate before acting, and Automated Security Helper (ASH), an open source scanner. He co-leads application security technical field community at AWS, and speaks at conferences including AWS re:Invent, re:Inforce, and Cyber Week.

Danny Cortegaca

Danny Cortegaca

Danny is a Principal Security Specialist Solutions Architect and co-leads the Application Security focus area within the AWS Security and Compliance Technical Field Community. He joined AWS in 2021 and partners with some of the largest organizations in the world to help them navigate complex security and regulatory environments. He loves talking about application security with customers and has helped many adopt threat modeling into their practices.

  •  

Accelerating AWS Network Firewall troubleshooting with AWS DevOps Agent

When an administrator introduces a rule change in AWS Network Firewall and network connectivity is disrupted, pinpointing the cause requires inspecting multiple points in the traffic path. The firewall gives you stateless and stateful rule engines, domain rules, and routing to the firewall endpoint inside your Amazon Virtual Private Cloud (Amazon VPC). A network drop looks the same from the workload no matter where it started. Isolating the cause means correlating the alert and flow logs with the firewall configuration, route tables, and recent API calls in AWS CloudTrail that might have changed them. That manual correlation is exactly where AWS DevOps Agent helps, accelerating root cause analysis so you can restore connectivity in minutes instead of hours.

AWS DevOps Agent does that correlation for you. As your always-available operations teammate, it resolves and proactively prevents operational issues across AWS, multicloud, and on-premises environments. When an Amazon CloudWatch alarm triggers, it reaches the agent through a webhook. The agent then reads the firewall configuration and logs through AWS APIs, ties the drop to recent API activity, and returns a root cause with a mitigation plan you review before you apply it.

This post connects CloudWatch monitoring to DevOps Agent. It walks through three Network Firewall failures from end to end. The first is a domain deny list blocking a legitimate endpoint. The second is a stateless rule priority misconfiguration. The third is an asymmetric cross Availability Zone (AZ) routing drop. Each maps to a different layer, so each leads down a different investigation path. An AWS Cloud Development Kit (AWS CDK) app deploys the whole environment in your own account so you can reproduce each failure and follow along.

The sample workload

As part of this blog post, we provide a CDK stack that deploys both the AWS DevOps Agent Space and a sample workload used to walk through three separate troubleshooting scenarios. A single t3.micro instance in a protected subnet checks its connectivity to a test endpoint on a continuous loop and publishes results to CloudWatch. Traffic takes the internet egress path through Network Firewall, the NAT gateway, and the internet gateway, so the firewall can intercept or drop it. After completing the walkthrough, you can apply the same troubleshooting techniques with DevOps Agent against your own Network Firewall deployments.

The test endpoint runs in a separate VPC deployed by the same CDK app. It serves HTTPS on port 443 and TCP on port 9142, giving each scenario a different protocol layer to exercise: Scenario 1 targets a TLS connection on 443 (matched by Server Name Indication), Scenario 2 targets a TCP connection on 9142, and Scenario 3 exercises the whole egress path.

A live status page shows one card per scenario plus the network topology. The whole stack deploys from a single CDK app across two Availability Zones, each with a firewall endpoint and NAT gateway, which is what makes Scenario 3 possible.

As shown in the following figure, the egress data path runs from the workload through Network Firewall and the NAT and internet gateways to the test endpoint. The alarm pipeline runs from CloudWatch through Amazon Simple Notification Service (Amazon SNS) and the webhook AWS Lambda function to DevOps Agent.

Figure 1: The sample workload

Figure 1: The sample workload

To use this with your own workload, you need a CloudWatch alarm that detects the connectivity problem and the webhook pipeline (SNS topic and Lambda function) that delivers it to DevOps Agent. The agent reads your firewall configuration, logs, and CloudTrail through AWS APIs, so no additional instrumentation is needed on the firewall side.

Prerequisites

To follow along with this post, you need:

Deploy the sample workload

Clone the project and deploy it into us-east-1 with one command (set awsRegion to use another AWS Region).

git clone https://github.com/aws-samples/sample-accelerating-aws-network-firewall-troubleshooting-with-aws-devops-agent.git
cd sample-accelerating-aws-network-firewall-troubleshooting-with-aws-devops-agent
bash scripts/deploy.sh

The script checks prerequisites, installs dependencies, compiles and tests, and bootstraps the CDK if needed. It then deploys all the stacks from a clean baseline and prints the outputs, including the status-page URL and sign-in details.

  1. Open the status-page link (an https://<random-id>.cloudfront.net address).
  2. Sign in using the username and password provided from the CDK output and confirm all three cards show the green Healthy status.
  3. Keep the page open while you run the scenarios.

Connect AWS DevOps Agent

To connect AWS DevOps Agent to the alarm pipeline

  1. In the AWS DevOps Agent console, open the nf-devops-agent-space Agent Space created by the CDK deployment.
  2. Configure the DevOps Agent webhook and download the CSV file with the webhook URL and signing secret.
  3. On the status page, choose Configure webhook, paste the URL and signing secret, and save. The page writes them to the nf-devops-agent-webhook-credentials AWS Secrets Manager secret, so there is no AWS CLI or console step. Until you set it, the bridge Lambda function sees a placeholder and skips delivery.
  4. Verify the path before you run a scenario. In the Lambda console, open nf-devops-agent-webhook and use the Test tab with this event.
    {
      "Records": [
        {
          "Sns": {
            "Message": "{\"AlarmName\":\"TEST-webhook-verification\",\"AlarmDescription\":\"[TEST] Webhook integration test - not a real alarm.\",\"NewStateValue\":\"ALARM\",\"NewStateReason\":\"[TEST] Manual webhook connectivity test. Safe to ignore.\",\"Region\":\"us-east-1\"}"
          }
        }
      ]
    }
  5. A 200 response confirms the path, and a test investigation appears in the DevOps Agent Operator Web App view.

How the alarm pipeline works

Every scenario reaches DevOps Agent the same way. A CloudWatch alarm moves to ALARM and notifies the SNS topic. Amazon SNS invokes a Lambda function. The function reads the webhook URL and signing secret from Secrets Manager, signs an alarm payload, and POSTs it to the DevOps Agent webhook (as shown in Figure 1). Amazon SNS also provides delivery retries, fan-out to other subscribers, and cross-account publishing.

  • Prebuilt Network Firewall metric (Scenario 1) – Alarm-1 watches the DroppedPackets metric, summed across the stateful streams, and triggers when drops rise above a baseline threshold. This requires no workload or custom metric and works on an already-deployed firewall. However, it only tells you that the firewall is dropping packets, not which rule is responsible.
  • Application health metric (Scenarios 2 and 3) – Alarm-2 and Alarm-3 watch a custom metric from a connectivity check. Use this for an alarm tied to user-facing impact or to tell one traffic path from another, which requires running a component that emits the metric.
Alarm Source Triggers when
Alarm-1 Native AWS/NetworkFirewall DroppedPackets The firewall’s dropped-packet count rises above the baseline
Alarm-2 Custom application health metric The port 9142 (TCP) connectivity check to the test endpoint is being dropped
Alarm-3 Custom application health metric The cross Availability Zone connectivity check is being dropped

Run the scenarios

Work through each of the scenarios one at a time, following the same cycle. Interrupt network connectivity, watch the alarm trigger, let DevOps Agent investigate, apply the recommended fix, and confirm recovery before moving on.

The status-page cards follow the live CloudWatch alarm state. A card shows a green dot and the word Healthy when its alarm is clear, and a red dot and the word DROPPED when its alarm triggers. In the DROPPED state the card also adds a Condition: line describing what’s being dropped, which isn’t shown when the card is healthy. Network Firewall applies changes to new flows, so a change shows within a minute or two. Recovery comes from the mitigation DevOps Agent recommends, which you review and apply.

Scenario 1. Domain deny list blocking a legitimate endpoint

At baseline, the rg-domain Suricata domain rule group denies only an unused placeholder, so the test endpoint stays reachable. The rule group inspects the TLS Server Name Indication (SNI) on each outbound connection and drops any that matches a denied domain. The exact rule syntax and console steps follow.

To add the domain deny rule

  1. Go to the Amazon VPC console.
  2. In the navigation pane, under Network Firewall, choose Network Firewall rule groups.
  3. Choose the rg-domain rule group to open its details page.
  4. In the Rules section, choose Edit.
  5. The rules box already contains two baseline placeholder rules (they match blocked.placeholder.invalid, so nothing real is denied). Leave those in place. Find the <app-endpoint-dns> value for Scenario 1 in the deployment script output (a Nework Load Balancer (NLB) DNS name such as NfTest-AppNl-a1b2C3dEf4G5-1234abcd5678efgh.elb.us-east-1.amazonaws.com). On a new line below the existing rules, add a drop rule that matches that DNS name on the TLS SNI, then choose Save.
    drop tls $HOME_NET any -> $EXTERNAL_NET any (ssl_state:client_hello; tls.sni; content:"<app-endpoint-dns>"; startswith; nocase; endswith; msg:"S1 domain denylist"; flow:to_server, established; sid:2000002; rev:1;)
  6. After saving, the rules box holds all three lines. The two placeholders remain, plus the new drop rule for the endpoint DNS name (note the distinct sid 2000002).
Figure 2: Scenario 1 – Firewall rule change blocking the connection

Figure 2: Scenario 1 – Firewall rule change blocking the connection

What happens. The workload’s HTTPS check to the test endpoint times out, the “AWS/NetworkFirewall DroppedPackets metric climbs above baseline, and Alarm-1 moves to ALARM. The Scenario 1 card reads DROPPED (with the condition Firewall dropping the monitored domain on its allow/deny rules), while the Scenario 2 and Scenario 3 cards stay Healthy (Figure 3). On the topology, the alarm pipeline from CloudWatch through Amazon SNS and Lambda to DevOps Agent and the workload-to-firewall inspect lines both turn amber, which the legend defines as collateral / alarm active, because the packets are now dropped at the firewall. To demonstrate the resulting failure, the HTTPS · SNI line from the internet gateway to the test endpoint is shown in red, which the legend defines as dropped (root cause).

Figure 3: Scenario 1 active – Traffic blocked at the firewall

Figure 3: Scenario 1 active – Traffic blocked at the firewall

Let DevOps Agent investigate. The agent runs several lines of investigation in parallel and correlates them:

  1. Reads the DroppedPackets metric and correlates the spike with a simultaneous drop in passed packets, confirming the firewall is actively blocking traffic.
  2. Reads the ALERT log and finds the workload’s TLS connections to the test endpoint blocked by the S1 domain denylist rule.
  3. Compares the current state against a baseline window, where the same endpoint was reachable with no alerts, which shows the block is new.
  4. Searches CloudTrail and surfaces the UpdateRuleGroup call that added the deny rule, identifying the user, role, and timestamp approximately one minute before the drops began.
  5. Reports the root cause as that manual rule-group change. Recommends removing the deny entry or adding an allow exception and enabling FirewallPolicyChangeProtection to prevent unauthorized changes.
  6. Presents this as a plan you review and apply, not an automatic change.

In the DevOps Agent Operator Web App view, the agent first restates the Alarm-1 trigger and confirms the firewall is dropping packets above the threshold (Figure 4).

Figure 4: Scenario 1 – The symptom

Figure 4: Scenario 1 – The symptom

Next, the agent identifies the root cause: a manual update to the rg-domain rule group that added a domain deny rule (SID 2000002) shortly before the alarm fired, blocking TLS connections to the ELB endpoint (Figure 5).

Figure 5: Scenario 1 – The root cause

Figure 5: Scenario 1 – The root cause

Finally, the agent presents a mitigation plan, recommending you remove the problematic deny rule (SID 2000002) to restore connectivity (Figure 6).

Figure 6: Scenario 1 – The mitigation plan

Figure 6: Scenario 1 – The mitigation plan

Note: In a real-world environment, this type of rule typically exists for a reason. Before removing it, verify whether it was intentional but scoped too broadly. If so, refine the rule to block only unauthorized endpoints rather than removing it entirely.

Confirm recovery. Apply the change the agent recommends. After the deny entry is gone, DroppedPackets falls back to baseline, Alarm-1 clears, and the card returns to green. Move on to Scenario 2.

Scenario 2. Stateless rule priority misconfiguration

At baseline, the rg-stateless-priority stateless rule group keeps the allow rule at priority 100 and the drop rule at 200 for the test class, TCP destination port 9142. The workload opens a TCP connection to the test endpoint on this port. Lower priority numbers evaluate first, so the allow rule wins. This scenario uses port 9142 instead of 443 to demonstrate a stateless rule, which matches on the packet’s 5-tuple (protocol, ports, addresses) rather than application content.

Introduce the change. Invert the two rule priorities so the drop rule evaluates before the allow rule. This is the kind of change a rushed rule edit can introduce.

To invert the stateless rule priorities

  1. Go to the Amazon VPC console.
  2. In the navigation pane, under Network Firewall, choose Network Firewall rule groups.
  3. Choose the rg-stateless-priority rule group to open its details page.
  4. In the Rules section, choose Edit.
  5. Raise the (Action: Pass) rule’s priority number so it sits after the (Action: Drop) rule, then choose Save. For example, change the (Action: Pass) rule from 100 to 300 (any number higher than the drop rule’s 200 works). You only need to move one rule, and using 300 avoids a clash with the drop rule that already sits at 200. Network Firewall evaluates the lowest priority number first, so the (Action: Drop) rule at 200 now wins for this traffic class, ahead of the (Action: Pass) rule at 300.
Figure 7: Scenario 2 – Rule priority change blocking the traffic class

Figure 7: Scenario 2 – Rule priority change blocking the traffic class

What happens. The drop rule now wins, the TCP connection to the test endpoint on port 9142 times out, the StatelessRuleFailures metric climbs above baseline, and Alarm-2 moves to ALARM. The Scenario 2 card reads DROPPED (with the condition Stateless rules dropping the monitored traffic class), while the Scenario 1 and Scenario 3 cards stay Healthy (Figure 8). On the topology, the alarm pipeline from CloudWatch through Amazon SNS and Lambda to DevOps Agent and the workload-to-firewall inspect lines both turn amber, which the legend defines as collateral / alarm active, because the packets are now dropped at the firewall. To demonstrate the resulting failure, the TLS :9142 line from the internet gateway to the test endpoint is shown in red, which the legend defines as dropped (root cause).

Figure 8: Scenario 2 active

Figure 8: Scenario 2 active

Let DevOps Agent investigate. A stateless drop happens before traffic reaches the stateful inspection engine, so it produces no ALERT log entries. The agent turns to configuration and flow logs instead:

  1. Reads the stateless rule group state and finds the drop rule at the lower priority number, ahead of the pass rule, so the drop evaluates first.
  2. Reads the flow logs and sees passed packets drop to zero within a minute of the change.
  3. Searches CloudTrail and surfaces the UpdateRuleGroup call that inverted the priorities, identifying the user, role, and timestamp about a minute before the alarm.
  4. Reports the root cause as that priority inversion. Recommends removing the redundant drop rule and managing the rule group through infrastructure-as-code (IaC) to prevent manual misconfigurations.
  5. Presents this as a plan you review and apply, not an automatic change.

In the DevOps Agent Operator Web App view, the agent first restates the Alarm-2 trigger and confirms that a workload connectivity health check is failing because the firewall’s stateless rules are dropping egress (Figure 9).

Figure 9: Scenario 2 – The symptom

Figure 9: Scenario 2 – The symptom

Next, the agent identifies the root cause, using the rule-group state and CloudTrail to pinpoint the conflicting DROP/PASS rules, where the new DROP rule’s lower priority number makes it match first (Figure 10).

Figure 10: Scenario 2 – The root cause

Figure 10: Scenario 2 – The root cause

Finally, the agent presents a mitigation plan, recommending you remove the conflicting DROP rule at priority 200 to restore traffic flow (Figure 11).

Figure 11: Scenario 2 – The mitigation plan

Figure 11: Scenario 2 – The mitigation plan

Confirm recovery. Apply the change the agent recommends. After the allow rule is ahead of the drop rule again, Alarm-2 clears and the card returns to green. Move on to Scenario 3.

Scenario 3. Asymmetric cross Availability Zone routing drop

At baseline, the protected subnet in each Availability Zone routes its egress through the firewall endpoint in that same Availability Zone , and the matching return route uses that same endpoint. One endpoint sees both directions of the flow, so the stateful engine completes the handshake. The workload runs in the protected subnet in us-east-1a (CIDR 10.0.4.0/24), so at baseline its egress and its return both use the us-east-1a firewall endpoint.

Introduce the change. Make the flow asymmetric by sending egress out one Availability Zone endpoint while the return comes back through the other. This takes two route edits, and both are required. With only the first edit the flow can still complete, so the alarm will not trigger until both are saved. It makes no firewall-policy change, mirroring a real multi-Availability-Zone routing mistake.

To create asymmetric cross Availability Zone routing

  1. Go to the Amazon VPC console and choose Route tables in the navigation pane.
  2. Flip the egress. Select the NfNetworkStack/SampleVpc/protectedSubnet1 route table (the us-east-1a protected subnet, where the workload runs). On the Routes tab, choose Edit routes. Its 0.0.0.0/0 route currently targets the us-east-1a firewall endpoint. For the target, choose Gateway Load Balancer Endpoint and select the us-east-1b firewall endpoint, then choose Save changes.
  3. Move the return. Select the NfNetworkStack/SampleVpc/publicSubnet2 route table (the us-east-1b public subnet, where egress now exits). Choose Edit routes, then Add route. For the destination enter the workload CIDR 10.0.4.0/24. For the target, choose Gateway Load Balancer Endpoint and select the us-east-1a firewall endpoint. Choose Save changes.

After both edits, a flow’s egress leaves through the us-east-1b endpoint while its return is directed to the us-east-1a endpoint. Neither endpoint sees the whole flow.

Figure 12: Scenario 3 routing change breaking the flow’s symmetry

Figure 12: Scenario 3 routing change breaking the flow’s symmetry

What happens. A new connection leaves through one endpoint. Its return arrives at the other endpoint, which never saw the connection open, so the handshake fails. Unlike Scenarios 1 and 2, this affects the whole subnet, so all egress stops and Alarm-2 and Alarm-3 both move to ALARM. The AWS/NetworkFirewall DroppedPackets alarm (Alarm-1) stays quiet because no endpoint is making a drop decision. The flow is lost to asymmetric routing rather than counted as a firewall drop. This is why monitoring application connectivity matters. A routing fault is invisible to the firewall’s own drop counter. On the status page, the Scenario 2 card reads DROPPED (with the condition “Stateless rules dropping the monitored traffic class”) and the Scenario 3 card reads DROPPED (with the condition Return traffic dropped by asymmetric cross-Availability-Zone routing), while the Scenario 1 card stays Healthy (Figure 13). On the topology, the alarm pipeline from CloudWatch through Amazon SNS and Lambda to DevOps Agent and the workload-to-firewall inspect lines both turn amber, which the legend defines as collateral / alarm active, while the egress path from the firewall through the NAT gateway and the TLS :9142 and HTTPS · routing lines to the test endpoint turn red, which the legend defines as dropped (root cause).

Figure 13: Scenario 3 – The status page during a path-wide outage

Figure 13: Scenario 3 – The status page during a path-wide outage

Let DevOps Agent investigate. Both Alarm-2 and Alarm-3 fire in the same datapoint. DevOps Agent recognizes them as linked and merges them into a single investigation:

  1. Reads the flow logs and sees bidirectional TLS connections stop abruptly, with only one-way traffic remaining and no flows reaching the established state.
  2. Reads the firewall metrics and sees received and passed packets shift from one Availability Zone to the other at the moment of the change.
  3. Calls DescribeRouteTables and finds the egress route pointing at one Availability Zone firewall endpoint while the return route points at the other.
  4. Searches CloudTrail and surfaces the ReplaceRoute and CreateRoute calls by the same user, about a minute before both alarms fired.
  5. Reports the root cause as that asymmetric routing change. Recommends restoring symmetric same-Availability-Zone routing so egress and return traverse the same endpoint.
  6. Presents this as a plan you review and apply, not an automatic change.

A mitigation plan is a recommendation you review, not an automatic change, and the right fix depends on the intended design. Restoring symmetric routing can mean sending the workload subnet’s egress back through its own-Availability-Zone firewall endpoint (this sample’s architecture) or, in a design that doesn’t inspect this path, back through a NAT gateway. The agent infers a plausible target from what it can observe, so review the specific route it proposes against your intended topology before you apply it. (Connecting your pipeline or infrastructure-as-code, covered in the next section, lets the agent recommend the target that matches your design.)

In the DevOps Agent Operator Web App view, the agent restates the Alarm-3 (AsymmetricFlowFailures) trigger and confirms the workload’s egress to a monitored endpoint is being blocked by the Network Firewall (Figure 14).

Figure 14: Scenario 3 – The symptom

Figure 14: Scenario 3 – The symptom

Next, the agent identifies the root cause: manual route table changes that created cross-AZ asymmetric routing through the network firewall, breaking its symmetric routing requirement (Figure 15)

Figure 15: Scenario 3 – The root cause

Figure 15: Scenario 3 – The root cause

Finally, the agent presents a mitigation plan, recommending you restore symmetric routing by pointing protectedSubnet1‘s default route back to the same Availability Zone firewall endpoint, so one endpoint sees both directions of the flow again (Figure 16).

Figure 16: Scenario 3 – The mitigation plan

Figure 16: Scenario 3 – The mitigation plan

Confirm recovery. Apply the change the agent recommends, after checking the route target matches your intended design. After the workload subnet’s egress and return use the same Availability Zone firewall endpoint again, the control probe recovers, the alarms clear, and every card returns to green.

Further considerations

In production a single change can trigger several alarms at the same time, as Scenario 3 shows. DevOps Agent links related investigations and works them as one, so you review a single root cause. You can validate the linked findings or unlink an alarm to investigate it independently. If you would rather collapse alarms before they reach the agent, you can add correlation logic in the bridge Lambda function, buffering and grouping by firewall. You can also add email, Amazon Simple Queue Service (Amazon SQS), or HTTP subscribers to the SNS topic, or add the webhook Lambda function to a topic you already run. DevOps Agent produces a mitigation plan but does not change your environment on its own.

You can also give the agent more to work with. DevOps Agent connects to source repositories and CI/CD pipelines, integrating with GitHub (including GitHub Enterprise Server and GitLab Self-Managed through a private connection). It can associate AWS resources with deployments of AWS CloudFormation, AWS CDK, Amazon Elastic Container Registry (Amazon ECR) images, and Terraform. With deployed configuration and recent deployment events in view, the agent correlates the disruption against the change that introduced it and recommends a fix matching your intended design. For this sample, that means recommending the workload subnet’s own Availability Zone firewall endpoint rather than a generic symmetric path.

DevOps Agent also supports proactive incident prevention. It analyzes patterns across past investigations and delivers recommendations to prevent similar issues from recurring, including governance recommendations that strengthen deployment processes and pipeline controls. For Network Firewall rule changes, this means the agent can recommend guardrails for your CI/CD pipeline based on the classes of misconfigurations it has already resolved. You can access these recommendations through the Improvements page in the DevOps Agent Operator Web App.

Clean up

Clean up the environment with one command.

bash scripts/destroy.sh

It reverts any active scenario, runs cdk destroy for all stacks, and sweeps for stragglers by the Project = nf-devops-agent tag. The main cost drivers are the two Network Firewall endpoints, the NAT gateways (one in the main VPC for each Availability Zone, one in the test-endpoint VPC), and the test endpoint’s load balancers. Each of these bills at an hourly rate for as long as it’s provisioned, whether or not traffic is flowing, so a stack left running continues to accrue charges around the clock even while idle. Running the scenarios and tearing the stack down the same day limits the cost to a few active hours rather than days of idle hourly charges.

Conclusion

In this post, we showed you how AWS DevOps Agent accelerates troubleshooting for three common network firewall connectivity issues. The first was a domain deny list. The second was a stateless priority inversion. The third was an asymmetric cross-AZ routing drop. For each one, DevOps Agent investigated the drop and returned a root cause with a mitigation plan you approve before applying. The first scenario triggered on a prebuilt Network Firewall metric, and the other two on application health metrics. That shows both ways to alarm on a firewall problem through one pipeline.

The pattern isn’t specific to Network Firewall. The same flow fits any service that emits CloudWatch metrics and logs, such as AWS WAF, security groups, and network ACLs. Clone the sample repository to explore the solution, then apply what you learn to your own firewall, application, and alarms. For more details, see the AWS Network Firewall Developer Guide and the AWS Network Firewall pricing page. Start with the Getting Started with AWS DevOps Agent guide to connect your first webhook.

Salman Ahmed

Salman is a Senior Technical Account Manager at AWS, specializing in helping customers design, implement, and optimize their AWS environments. He combines deep networking expertise with a passion for exploring emerging technologies to help organizations get the most out of their cloud investments. Outside of work, he enjoys photography, traveling, and watching his favorite sports teams.

  •  
❌