You want to set a cryptographic algorithm policy and stop non-compliant choices from being made in your Amazon Web Services (AWS) environment. This post shows you how. As you adopt post-quantum cryptography, one of the first decisions you face is which algorithms and parameter sets fit your business needs, balancing security against signature size, key size, and performance. Your cloud environment’s job is to let you express that decision and enforce it consistently, so an approved choice stays fully usable while an excluded one cannot be introduced. This post works through one concrete example on AWS: taking an organization’s cryptographic algorithm policy and turning it into preventive and detective controls. The same patterns apply to whatever your approved set is today and whatever it becomes as standards evolve.

This post is Part 2 of a two-part series on quantum-resistant signing on AWS. Part 1, Build a software supply chain with ML-DSA, is for builders and constructs an end-to-end ML-DSA signing pipeline for binaries, container images, and software bills of materials. This part is for the security, governance, and compliance teams who decide which algorithms that pipeline, and everything else in the organization, is permitted to use, and who need that decision enforced.

AWS Key Management Service (AWS KMS) supports the ML-DSA digital signature scheme (FIPS 204) through three parameter sets, ML-DSA-44, ML-DSA-65, and ML-DSA-87, targeting NIST security categories 2, 3, and 5. All three are strong, standardized choices. Each one trades signature size, key size, and speed against security margin, the amount of cryptographic strength a parameter set holds in reserve above its target security category. ML-DSA-44 is the most compact and fastest. ML-DSA-65 and ML-DSA-87 carry a larger security margin, which is what you want for signatures that must stay trustworthy over a long lifetime.

Throughout this post, the examples use the same example policy: an organization that has standardized on ML-DSA-65 and ML-DSA-87 and wants ML-DSA-44 kept out of its approved set. That is a standardization decision, not a judgment that ML-DSA-44 is insecure (see the following note). Wherever you see ML-DSA-44 in the controls that follow, read it as the variant your policy doesn’t include; the mechanics are identical for any allow/deny split you choose. The approved set here is also the one the Part 1 architecture deploys: ML-DSA-87 for the root certificate authority (CA) and ML-DSA-65 for the subordinate CA and the code-signing key. Part 1 makes the choice; this post makes it stick.

Note: ML-DSA-44 is a valid choice. ML-DSA-44 is a fully NIST-standardized parameter set (FIPS 204) at NIST security category 2, which targets roughly 128-bit-class classical security (category 2 is benchmarked against the collision resistance of SHA-256). It produces the smallest signatures and public keys of the three variants (a 1312-byte public key and 2420-byte signature) and is the fastest to sign and verify, which makes it a strong fit for constrained devices, high-volume signing, and bandwidth- or storage-sensitive workloads. You would choose ML-DSA-65 or ML-DSA-87 instead to gain extra security margin for long-lived signatures or to align with an internal standard, not because ML-DSA-44 is weak.

The approved variants (ML-DSA-65 and ML-DSA-87 in the example) remain fully usable throughout. What you want to rule out is the introduction of a variant outside the policy (here, ML-DSA-44). Concretely, within the AWS service surfaces in scope, principals shouldn’t be able to:

• Create ML-DSA-44 key material
• Use existing ML-DSA-44 keys for signing or verification
• Stand up private certificate authorities (CAs) that would introduce ML-DSA-44.

And any drift from those settings should be detected.

These controls operate on AWS service APIs, principally AWS KMS, AWS Private Certificate Authority (AWS Private CA), and AWS Certificate Manager (ACM). They don’t and can’t govern cryptography that runs on compute you control. For example, OpenSSL or an application might perform ML-DSA-44 signing on an Amazon Elastic Compute Cloud (Amazon EC2) instance, container, or AWS Lambda function. Protocol-level use, such as OpenSSH’s hybrid ssh-mldsa44-ed25519 public key authentication, is also outside the reach of Service control policies (SCPs) and AWS Identity and Access Management (IAM) and isn’t addressed in this post.

Certificate material brought in through acm:ImportCertificate or iam:UploadServerCertificate is also not algorithm-aware (see the Private CA section). Other TLS-terminating or signing services (for example Amazon CloudFront and Amazon API Gateway) and AWS IAM Roles Anywhere aren’t covered in this post and should be assessed separately. (Part 1 of this series uses IAM Roles Anywhere with ML-DSA certificates; controls for that service are out of scope here.)

These controls are intended to be applied as defense in depth. You can apply all of them or only a subset, depending on how prescriptive you want to be:

• A blanket SCP at the organization root prevents member accounts from creating or using ML-DSA-44 keys. (SCPs don’t apply to the management account, so workloads there aren’t restricted.)
• An SCP at a production organizational unit (OU) restricts keys only in production accounts.
• If you need flexibility inside an account, use IAM permission boundaries instead of account-wide settings enforced through an SCP.

Choose the combination that matches your operating model. The policy examples here are written to apply across all AWS Regions. If you need to scope a control to specific Regions, add an aws:RequestedRegion condition.

The policies in this post are illustrative starting points. Authoring, validating, and maintaining the specific SCPs, IAM policies, and permission boundaries for your environment is your responsibility. Test all policies in a non-production environment before deployment and adjust example values (such as policy names and key Amazon Resource Names (ARNs)) to match your organization’s approved standards.

Your algorithm policy isn’t fixed forever, and these controls shouldn’t be either. The point of expressing the policy as AWS controls is that you can revise it as your needs and the relevant standards change, tightening, loosening, or swapping the approved set without re-architecting. Treat the examples here as a template you adjust to whatever your current policy is.

A deny-based control that names specific algorithms is, in effect, an allow-list of everything it does not name. The risk is default-allow: any parameter set you have not explicitly denied is permitted, so a newly introduced variant, including one you have not yet evaluated, becomes usable until you notice it and add it to the deny list. Deprecation of an already-approved variant is easy to handle in this model, because you simply add the deprecated variant to the deny list. What the model does not catch on its own is the arrival of something new. Review these controls periodically and pair the deny-list with detection so a variant outside your policy surfaces quickly rather than waiting to be discovered.

The alternative is to write the condition against the approved set rather than the excluded one. A Deny that uses StringNotEquals with a list of approved key specs fires when the request’s key spec matches none of them, so a new or not-yet-evaluated variant is blocked by default until you add it. That closes the default-allow gap, at the cost of a policy update before anyone can use a newly approved variant. The family-level SCP in Part 1 of this series takes this shape: it denies creation of any signing key whose key spec isn’t ML_DSA_*. The examples in this post take the other shape and name the excluded variant, so that approved variants stay usable without a policy change. Both are valid. Choose based on whether you would rather block something new first and approve it later, or notice it first and decide then.

{
  "Sid": "OnlyApprovedMlDsaVariants",
  "Effect": "Deny",
  "Action": "kms:CreateKey",
  "Resource": "*",
  "Condition": {
    "StringLike": {
      "kms:KeySpec": "ML_DSA_*"
    },
    "StringNotEquals": {
      "kms:KeySpec": ["ML_DSA_65", "ML_DSA_87"]
    }
  }
}

The StringLike clause scopes the statement to ML-DSA key specs so it doesn’t affect other key types; the StringNotEquals clause denies any ML-DSA variant outside the approved list.

Just as algorithms and key sizes will be introduced over time, or move from approved to unapproved, so will AWS service operations. The set of operations through which an ML-DSA-44 key can be used today (AWS KMS Sign, Verify, and GetPublicKey) reflects current service capabilities and might expand as AWS releases new features. If your policy is to block use in addition to creation, you should periodically review the list of operations you restrict, because services evolve over time.

AWS KMS support for ML-DSA introduces three key specs—ML_DSA_44, ML_DSA_65, and ML_DSA_87—that all use a single post-quantum signing algorithm: ML_DSA_SHAKE_256. This distinction drives which controls are effective.

1. kms:KeySpec: Reflects the KeySpec property of the key. It’s evaluated both on CreateKey (the requested key spec) and on AWS KMS key resource operations that act on a specific key, including Sign, Verify, and GetPublicKey. This makes kms:KeySpec a reliable lever for both preventing creation of ML-DSA-44 keys and restricting use of existing ones, because it can single out ML_DSA_44 from ML_DSA_65 and ML_DSA_87.
2. kms:SigningAlgorithm: Reflects the signing algorithm supplied on Sign and Verify, which is ML_DSA_SHAKE_256 for all three variants. It therefore can’t distinguish ML-DSA-44 from the approved variants and shouldn’t be used for this purpose.

The practical consequence: for keys held in AWS KMS, preventing ML-DSA-44 key material from being created is the most robust single control. If no ML-DSA-44 KMS keys exist, none can be used for signing through AWS KMS and none can become the subject key of a certificate. CA keys and TLS server keys are separate material: AWS Private CA generates and holds a CA’s key pair in its own HSMs, and ACM-issued, imported, or uploaded TLS certificates carry their own keys, so those surfaces are governed by the AWS Private CA and certificate controls later in this post. Where ML-DSA-44 KMS keys already exist, their use can additionally be restricted with kms:KeySpec.

SCPs can be applied at the organization root or to a specific OU to restrict ML-DSA-44. Placement determines the effective scope: it applies organization-wide when attached at the root; attached to an OU, it applies to the accounts within that OU. An SCP is the broadest-scope preventive control available.

Note that SCPs apply to member accounts and OUs but not to the organization’s management account, so avoid running ML-DSA-44-capable workloads in the management account, where an SCP will not restrict them.

The following example SCP denies both the creation of ML-DSA-44 keys (kms:CreateKey) and their use (kms:Sign, kms:Verify, kms:GetPublicKey), matching on the ML_DSA_44 key spec in each case. The creation deny stops new keys from being created; the usage deny covers keys that already exist, whether created before the SCP was attached or living in an account that later joins the OU, because kms:KeySpec is also evaluated at request time on those usage operations.SCPs are guardrails: a Deny in an SCP can’t be overridden by any IAM policy within member accounts.

Attach the following SCP at the organization root (or relevant OU):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyMLDSA44KeyCreation",
      "Effect": "Deny",
      "Action": "kms:CreateKey",
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "kms:KeySpec": "ML_DSA_44"
        }
      }
    },
    {
      "Sid": "DenyMLDSA44KeyUsage",
      "Effect": "Deny",
      "Action": [
        "kms:Sign",
        "kms:Verify",
        "kms:GetPublicKey"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "kms:KeySpec": "ML_DSA_44"
        }
      }
    }
  ]
}

Scope the usage deny to the three key-spec-aware actions. Limit the use of Deny to kms:Sign, kms:Verify, and kms:GetPublicKey (testing confirmed kms:KeySpec is evaluated on all three). Avoid a blanket kms:*: a wildcard that would also catch management operations such as DescribeKey, DisableKey, and ScheduleKeyDeletion, which you need to inspect or clean up those same keys, effectively locking you out of managing the keys you’re trying to remove. If you use multi-Region keys, also add kms:ReplicateKey to the creation deny: replicating an ML-DSA-44 primary key creates a new ML-DSA-44 key in another Region, and kms:KeySpec is evaluated on ReplicateKey as well.

Combining with a family-level guardrail. This SCP governs which ML-DSA parameter set is allowed; it says nothing about RSA or ECDSA. Part 1 of this series includes a complementary SCP that denies creation of any signing key whose key spec isn’t ML-DSA. Attached together, they require every new signing key to be ML-DSA and to be an approved variant.

Operational considerations. A blanket Deny at the organization root is the strongest control but has trade-offs: it removes any ability to create ML-DSA-44 keys anywhere in scope, with no exception path, which can block legitimate testing and might cause infrastructure as code (IaC) or pipeline runs that attempt ML-DSA-44 creation to fail. To preserve flexibility, consider scoping the Deny to a production OU and using IAM permission boundaries (see the following section) in build and non-production accounts.

What the caller sees. When an SCP denies a request, the caller receives an AccessDenied error indicating an explicit deny in a service control policy. For same-account and same-organization requests, the error includes the type of the denying policy and, as the enhanced error messages roll out across services, its ARN. The error doesn’t expose the SCP policy contents, and the user being blocked often can’t see the SCPs applied to their environment. Document the SCP ARN and its purpose internally so support teams can map an AccessDenied quickly.

The following IAM policy blocks ML-DSA-44 at the principal level. It is the identity-based counterpart to the SCP shown earlier, useful when you want per-account or per-role control rather than an organization-wide guardrail. Attach it to the IAM role or user you want to constrain.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyMLDSA44KeyCreation",
      "Effect": "Deny",
      "Action": "kms:CreateKey",
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "kms:KeySpec": "ML_DSA_44"
        }
      }
    }
  ]
}

1. The condition value is ML_DSA_44 (underscores, not hyphens).
2. This policy blocks CreateKey only. To also block use of existing ML-DSA-44 keys at the principal level, add the Sign, Verify, and GetPublicKey deny shown in the Restricting use of existing ML-DSA-44 keys section.
3. For comprehensive, non-overridable coverage, deploy as an SCP rather than relying solely on principal-level policies.

IAM permission boundaries set the maximum permissions an identity-based policy can grant to an IAM role or user and can be used as an alternative or complement to using SCPs. The effective permissions are the intersection of the identity-based policy and the boundary, and a Deny in a boundary can’t be overridden by an attached policy. Boundaries are useful where you want per-account or per-role flexibility (for example, in build or non-production accounts) rather than a blanket, multi-account deny.

Operational consideration. A permission boundary only takes effect on principals that have it attached. To rely on boundaries as a control, you need a mechanism to enforce the boundary on all IAM principals programmatically. One example enforcement mechanism would be an SCP that denies iam:CreateRole, iam:CreateUser, iam:PutRolePermissionsBoundary, and iam:PutUserPermissionsBoundary unless the approved boundary is set (using the iam:PermissionsBoundary condition key), and denies iam:DeleteRolePermissionsBoundary and iam:DeleteUserPermissionsBoundary outright so the boundary can’t be removed after the fact. You would also want to probe for non-conforming principals periodically and respond, perhaps by automatically attaching the PermissionsBoundary. IAM users shouldn’t be used, but the enforcement should account for the possibility that they exist.

Because a boundary only sets a ceiling and does not itself grant anything, it opens with Allow on all actions, then denies ML-DSA-44 key creation specifically. The following permission boundary denies creation of ML-DSA-44 keys:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowAll",
      "Effect": "Allow",
      "Action": "*",
      "Resource": "*"
    },
    {
      "Sid": "DenyMLDSA44KeyCreation",
      "Effect": "Deny",
      "Action": "kms:CreateKey",
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "kms:KeySpec": "ML_DSA_44"
        }
      }
    }
  ]
}

To also block use of existing ML-DSA-44 keys for the principals carrying this boundary, add the Sign, Verify, and GetPublicKey deny statement shown in the following section (which also uses kms:KeySpec).

As described in the Background section, kms:KeySpec is also evaluated on Sign, Verify, and GetPublicKey, reflecting the KeySpec of the key being used. You can therefore deny those operations when kms:KeySpec equals ML_DSA_44 while continuing to permit ML_DSA_65 and ML_DSA_87. The IAM policy below, attached to an IAM principal, does exactly that.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyMLDSA44KeyUsage",
      "Effect": "Deny",
      "Action": [
        "kms:Sign",
        "kms:Verify",
        "kms:GetPublicKey"
      ],
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "kms:KeySpec": "ML_DSA_44"
        }
      }
    }
  ]
}

This deny can also live in an AWS KMS key policy. The same kms:KeySpec deny can be placed directly in the key policy of an AWS KMS key. A key policy is a resource policy, so the statement must include a Principal element. Identity policies and SCPs do not name principals to which they apply. A deny in the key policy is a resource-side backstop that protects a specific key, while an SCP gives non-overridable organization-wide coverage. To turn the previous example into an AWS KMS key policy, add a Principal element to the statement (for example "Principal": {"AWS": "*"}, or a more specific principal), and keep the rest of it the same.

Depending on your goals you might also want to remove the key material itself rather than only deny its use. Remember that it is difficult to know what data has been signed by a specific key, unless you track that somehow. Deleting a key that was used will make signature verification impossible. Take great care to make sure keys are not used before deleting them.

1. Disable or schedule deletion of the keys. Disabling stops key use immediately and is reversible. Key deletion is definitive but irreversible after the waiting period, so confirm before scheduling.
2. Deny Sign, Verify, and GetPublicKey by specific key ARN. You can target named keys rather than the whole key spec. To do this, you would list the exact key ARNs in the Resource element of the IAM policy (no condition key required). This is effective for targeting a single key specifically, but it is difficult to maintain lists of key ARNs. Use this strategy sparingly. If you’re considering using an IAM policy naming a key ARN, remember that a Deny in the key policy of the target key accomplishes the same goal and is attached to the key itself.
3. Finding existing keys (only needed if you disable or delete). A Deny in an IAM policy or SCP, based on kms:KeySpec, doesn’t require you to know which keys exist – it applies by key spec at request time. If you want to disable or delete existing ML-DSA-44 keys, you need to locate them. AWS KMS has no server-side filter for key spec, so you must enumerate keys with ListKeys and call DescribeKey and inspect each KeySpec. This client-side filtering pattern is common across AWS APIs and is well supported by pagination in the AWS SDKs. You can also use AWS Config to record and query KMS key resources.

Reference: Choosing a KMS key spec – kms:KeySpec condition key

AWS Private CA sets a certificate authority’s (CA’s) signing algorithm at creation time, in the CertificateAuthorityConfiguration parameter of CreateCertificateAuthority (valid values include ML_DSA_44, ML_DSA_65, and ML_DSA_87). There is currently no condition key for the algorithm, so an SCP cannot enforce signing algorithm. The approach in this section controls who can create CAs and pins the algorithm in that trusted path, backed by AWS CloudTrail-based detection.

1. Restrict who can create CAs. Deny acm-pca:CreateCertificateAuthority broadly via SCP, and allow it only for a small set of trusted roles or an approved provisioning pipeline. This removes the ability for arbitrary principals to stand up a CA with an unapproved algorithm.
2. Pin the algorithm in the trusted path. Have that approved pipeline or infrastructure-as-code template hard-code SigningAlgorithm to ML_DSA_65 or ML_DSA_87 in CertificateAuthorityConfiguration. Because AWS Private CA exposes no algorithm condition key, this provisioning path is the enforcement point. Note that the CA’s algorithm pins the signature on the certificates it issues; the subject key in each certificate comes from the CSR, which AWS Private CA copies without validating its algorithm, so pin the CSR key algorithm in the same issuance pipeline.
3. Control the trust chain. On the PKI side, only add approved CAs to your trust stores and issuance hierarchies, and do not trust or subordinate CAs that could issue ML-DSA-44. This prevents an unapproved CA from being introduced into a trusted path.
4. Detect drift. Use CloudTrail to capture CreateCertificateAuthority calls and AWS Config to flag any CA whose configuration uses ML_DSA_44, so an unapproved CA is caught quickly.

Part 1 of this series shows what this looks like in practice: it creates an ML-DSA-87 root CA and an ML-DSA-65 subordinate CA with CreateCertificateAuthority, activates them, and distributes the resulting chain to verifiers. In the model described here, those are the calls that only your approved provisioning pipeline is allowed to make.

Setting the ML-DSA value in CertificateAuthorityConfiguration. The same variant goes in both the KeyAlgorithm and SigningAlgorithm fields (for example, KeyAlgorithm=ML_DSA_65 with SigningAlgorithm=ML_DSA_65). Note a difference in signing algorithms between AWS KMS and AWS Private CA. With AWS Private CA, the SigningAlgorithm takes the variant value directly (ML_DSA_44, ML_DSA_65, and ML_DSA_87), and doesn’t accept ML_DSA_SHAKE_256. In contrast, AWS KMS uses ML_DSA_SHAKE_256 as its signing algorithm. Be aware that some older AWS Command Line Interface (AWS CLI) builds omitted descriptions of the ML-DSA variants from the built-in CLI help, so don’t rely on the help in the AWS CLI to determine valid values. For more information, see CertificateAuthorityConfiguration API Reference.

Known limitation. Some surfaces have no algorithm condition key, so you enforce them through provisioning and detection rather than policy. acm-pca:CreateCertificateAuthority has no condition key for the algorithm, and acm:ImportCertificate and iam:UploadServerCertificate are not algorithm-aware, so a disallowed signature setting on a created, imported, or uploaded certificate cannot be blocked by an IAM policy or SCP condition. Govern these through your provisioning process, restrict who can call them, and deny the import and upload actions entirely where they are not required. The controls above remain the available mitigation.

In particular, iam:UploadServerCertificate doesn’t support any method to prevent use of particular algorithms, and AWS doesn’t recommend using this API to generate and upload certificates. This API predates ACM and exists mainly to cover AWS Regions or partitions where ACM is not yet available. While a new Region or partition is being built out, workloads can still need a TLS server certificate before ACM support has shipped there, and uploading a certificate to IAM gives you a place to store and reference it in the meantime. Once ACM is available in that Region, you should use ACM instead. Outside that narrow case, deny iam:UploadServerCertificate unless it is genuinely required.

ACM-issued leaf certificates. A CA’s own key and signing algorithm are separate from the leaf certificates that ACM issues from it. If you use ACM to request leaf certificates, you can constrain the leaf’s key algorithm with the acm:KeyAlgorithm condition key on acm:RequestCertificate. This lets you allow-list approved algorithms for leaf certificates using SCPs or IAM policies, similar to the AWS KMS examples earlier in this post. Be aware that, at the time of writing, ACM doesn’t offer ML-DSA leaf certificates. So a policy like this is a forward-looking defense-in-depth control rather than an immediate ML-DSA-44 risk. The IAM example below is illustrative (using example approved algorithms). Since this is an evolving part of the service, be sure to validate this in your environment before relying on it.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RestrictACMLeafKeyAlgorithm",
      "Effect": "Deny",
      "Action": "acm:RequestCertificate",
      "Resource": "*",
      "Condition": {
        "StringNotEqualsIfExists": {
          "acm:KeyAlgorithm": [
            "EC_secp384r1",
            "RSA_2048"
          ]
        }
      }
    }
  ]
}

This example uses StringNotEqualsIfExists so it denies only requests that specify a disallowed algorithm. With plain StringNotEquals, a request that omits acm:KeyAlgorithm would also be denied, because the missing key makes the condition evaluate true. Choose the operator that matches whether you want to block requests that leave the algorithm unspecified.

Elastic Load Balancing, for both network load balancers (NLB) and application load balancers (ALB), terminates TLS on behalf of your applications. The certificates used by these listeners are requested through or imported into ACM, or uploaded to IAM. A listener’s configured security policy and its certificate determine what’s offered during negotiation. At the time of writing, ACM requests and imports accept RSA and ECDSA keys only, and NLB and ALB listeners accept RSA and ECDSA certificates only, so no ML-DSA certificate can be bound to a listener today. As with the ACM leaf-certificate policy earlier, the controls in this section are forward-looking defense in depth rather than a mitigation of an immediate ML-DSA-44 risk.

Because the listener API calls reference a certificate only by ARN, you can’t write an SCP on CreateListener, ModifyListener, or AddListenerCertificates that inspects the certificate’s algorithm. IAM conditions can’t reference attributes of a separately-referenced resource. Keeping ML-DSA-44 out of TLS therefore relies on there being no ML-DSA-44 certificate material to associate:

1. Configure your private CA to issue only ML_DSA_65 or ML_DSA_87 certificates, and control who can create CAs.
2. Prevent ML-DSA-44 key creation in AWS KMS, so that no KMS-held ML-DSA-44 key can become the subject key of a certificate. TLS server keys for ACM-issued, imported, or uploaded certificates are not KMS keys; those are governed by items 1 and 3.
3. Control imported and uploaded certificates through your provisioning process or deny the import and upload actions where not required, as described in the AWS Private CA section.

CloudTrail records the certificate and listener API calls in this section: certificate creation and import (acm-pca:CreateCertificateAuthority, acm:ImportCertificate, iam:UploadServerCertificate) and listener configuration (CreateListener, ModifyListener, AddListenerCertificates). Whether the algorithm shows up in the event depends on what the call itself carries. Creation calls that take the algorithm as an input record it: kms:CreateKey logs the KeySpec, and acm-pca:CreateCertificateAuthority logs the signing algorithm. That makes CloudTrail effective for catching the creation of ML-DSA-44 key material or a CA. A listener binding references its certificate only by ARN, so the certificate’s algorithm does not appear in that event. To evaluate the algorithm of a bound or imported certificate, use AWS Config resource evaluation as the detective backstop.

The primary preventive control is the certificate material itself: with no ML-DSA-44 CA, no ML-DSA-44 KMS key, and certificate import and upload controlled, no ML-DSA-44 certificate can be issued or bound. Where you also operate an allow-list of permitted TLS security policies, the elasticloadbalancing:SecurityPolicy condition key (on CreateListener and ModifyListener) can additionally constrain which TLS policies listeners can use.

The following table maps the example policy (allow ML-DSA-65 or ML-DSA-87, exclude ML-DSA-44) to the available controls. Modify the controls to limit use to the approved set of algorithms your policies require.

In this post, you learned how to use ML-DSA as an example of turning a cryptographic algorithm policy into enforceable AWS controls. The key lever is the kms:KeySpec condition key: it keeps your approved parameter sets (ML-DSA-65 and ML-DSA-87) fully usable while preventing the creation and use of a parameter set you’ve excluded (ML-DSA-44) on the CreateKey, in addition to the Sign, Verify, and GetPublicKey operations, where kms:SigningAlgorithm can’t tell the variants apart. Applied as defense in depth across SCPs, IAM policies, permission boundaries, and KMS key policies, and extended to the surfaces that have no algorithm condition key (AWS Private CA, certificate import, and Elastic Load Balancing TLS) through provisioning controls plus AWS CloudTrail and AWS Config drift detection, these controls give you a consistent way to keep an unapproved variant out of your accounts.

The pattern isn’t specific to ML-DSA-44. Because your algorithm policy will change as standards and your own requirements evolve, treat these controls as a reusable template and re-point the same condition keys and guardrails at whatever your approved set becomes. This adaptability is the practical meaning of cryptographic agility: changing what you allow without re-engineering how you enforce it.

To get started, review ML-DSA keys in AWS KMS and Choosing a KMS key spec, and test the example policies in a non-production environment before you deploy them.

To see the workloads these controls govern, read Part 1 of this series, which builds a complete ML-DSA signing pipeline, from key creation and PKI through artifact signing and verification, using the approved set enforced here.

• Quantum-resistant signing on AWS, Part 1: Build a software supply chain with ML-DSA
• AWS KMS adds support for post-quantum ML-DSA digital signatures
• ML-DSA keys in AWS KMS
• AWS KMS condition keys (kms:KeySpec evaluated on CreateKey and key resource operations)
• Choosing a KMS key spec (kms:KeySpec)
• Use condition keys with ACM (acm:KeyAlgorithm)
• Actions, resources, and condition keys for AWS Private Certificate Authority
• Private CA CertificateAuthorityConfiguration API
• Actions, resources, and condition keys for Elastic Load Balancing v2 (elasticloadbalancing:SecurityPolicy)
• Enhanced access denied error messages with policy ARNs

If you have feedback about this post, submit comments in the Comments section below.

Software supply chain attacks have grown in both frequency and sophistication. Recent high-profile compromises of build systems and open source dependencies demonstrate that attackers increasingly target the trust mechanisms that verify software integrity. Those trust mechanisms rely on digital signature algorithms, and today many use traditional algorithms such as Rivest-Shamir-Adleman (RSA) and the Elliptic Curve Digital Signature Algorithm (ECDSA). If a sufficiently powerful quantum computer could forge these signatures, an attacker could potentially inject unauthorized code into signed artifacts, fabricate build attestations, or impersonate legitimate publishers. Using quantum-resistant signatures today defends against threats that might emerge in the future.

Since 2025, the Commercial National Security Algorithm Suite 2.0 (CNSA 2.0), published by the US National Security Agency (NSA), has required US National Security Systems (NSS) to “support and prefer” quantum-resistant signatures for software and firmware, with exclusive use mandated by 2030. The EU Cyber Resilience Act (CRA) takes full effect in December 2027. It requires manufacturers to maintain verifiable software integrity throughout a product’s lifecycle. Organizations that ship software to US federal agencies or sell products in the European market face converging deadlines that demand action now.

Amazon Web Services (AWS) provides the building blocks to construct a quantum-resistant software supply chain today using production-ready, generally available services. In this post, I show you how to combine Module-Lattice-Based Digital Signature Algorithm (ML-DSA) keys in AWS Key Management Service (AWS KMS), ML-DSA certificate hierarchies in AWS Private Certificate Authority, and ML-DSA trust anchors in AWS Identity and Access Management (IAM) Roles Anywhere into an end-to-end signing architecture for binaries, container images, and software bills of materials (SBOMs).

This post is Part 1 of a two-part series on adopting ML-DSA signatures, standardized in NIST FIPS 204 (ML-DSA) signatures, on AWS. As of Oct 2026, this series is the first end-to-end guide from any cloud provider for post-quantum signing across the software supply chain, from establishing an ML-DSA root of trust to enforcing algorithm policy across an organization. This part is for builders and walks through constructing and operating the signing pipeline. Part 2, Enforce your ML-DSA algorithm policy with SCPs and IAM, is for security and governance teams and shows how to turn the algorithm choices made here into preventive and detective controls across an organization.

After a cryptographically relevant quantum computer (CRQC) exists, the mathematical problems protecting today’s public-key cryptography can be solved efficiently. Attacks that are impractical today because of the computational complexity suddenly become practical using a CRQC. Most discussions of post-quantum cryptography (PQC) focus on the harvest now, decrypt later threat to data confidentiality. The asymmetric keys protecting data in transit can be efficiently derived using a CRQC, exposing the data that was meant to be protected. Digital signatures rely on the same asymmetric mathematical problems that a CRQC would break. This means that software supply chains face a parallel threat that targets integrity rather than confidentiality. An attacker who records signed artifacts today can forge those signatures in the future on a totally different artifact, using a CRQC.

Consider firmware signed with ECDSA P-256 and deployed to a fleet of embedded devices in 2026. These devices verify the firmware signature on every boot for their 15-year operational lifetime and can’t be recalled to re-sign after deployment. If a CRQC becomes operational in 2035, an attacker could produce a valid-looking signature over tampered firmware, and every device in the fleet would accept it as authentic on next restart. For organizations that rely on classical digital signatures as the primary mechanism for verifying software integrity, this weakens a critical link in the verification chain. This is why CNSA 2.0 specifies aggressive timelines for digital signatures: the artifacts you sign today must remain trustworthy for their full operational lifetime.

ML-DSA is designed to resist attacks from both classical and quantum computers. AWS KMS, AWS Private CA, and AWS IAM Roles Anywhere all support ML-DSA, making it possible to address the forge-later threat across your supply chain today without additional tooling.

The architecture comprises five trust pillars that together protect your software delivery pipeline:

1. ML-DSA key management in AWS KMS – All signing keys are generated and protected in FIPS 140-3 Security Level 3 hardware security modules (HSMs). Private keys never leave the HSM boundary.
2. ML-DSA public key infrastructure (PKI) through AWS Private CA – A certificate hierarchy (root CA, subordinate CA, end-entity certificates) binds ML-DSA public keys to identities, establishing who signed what. CA private keys are generated and stored in hardware-protected HSMs managed by the service and never leave the HSM boundary.
3. Quantum-resistant workload identity (for external build agents) – For continuous integration and delivery (CI/CD) agents running outside AWS that authenticate through X.509 certificates, IAM Roles Anywhere with ML-DSA certificates ties the signer’s identity back to the same quantum-resistant root of trust.
4. ML-DSA artifact signing – Build outputs (binaries, container image manifests, packages) are signed using AWS KMS ML-DSA keys. Signatures are stored alongside artifacts in Amazon Simple Storage Service (Amazon S3) or Amazon Elastic Container Registry (Amazon ECR).
5. ML-DSA SBOM integrity attestation – The SBOM for each release is wrapped in an in-toto Statement that names the artifact it describes by digest, and the Statement is signed with the same ML-DSA code-signing key. This provides quantum-resistant proof that the dependency manifest hasn’t been tampered with and that it belongs to the artifact it accompanies.

These pillars create a chain of trust: AWS Private CA protects the root CA key in its own HSMs, issues certificates that bind identities to AWS KMS code-signing keys, IAM Roles Anywhere uses those certificates to authenticate build agents, and authenticated agents sign every artifact and SBOM they produce.

Figure 1: End-to-end quantum-resistant software supply chain on AWS

Figure 1 illustrates this architecture in four layers, numbered in the order you build them:

1. Root of trust (Pillars 1 and 2) – Establish your AWS Private CA hierarchy and create your code-signing key in AWS KMS.
2. Workload identity (Pillar 3) – Register the CA as an IAM Roles Anywhere trust anchor and issue ML-DSA certificates to build agents, ensuring the signer’s identity chains back to the same quantum-resistant root of trust.
3. Artifact signing (Pillars 4 and 5) – Sign binaries, container images, and SBOMs using the authenticated build agent.
4. Verification and enforcement – Validate signatures at deployment and enforce ML-DSA-only policies.

To follow along, you need:

• An AWS account with permissions to create resources in AWS KMS, AWS Private CA, IAM Roles Anywhere, and Amazon ECR.
• AWS Command Line Interface (AWS CLI) v2 configured with appropriate credentials.
• IAM Roles Anywhere credential helper (aws_signing_helper) v1.8.0 or later, the first release that supports ML-DSA keys and certificates, for workload authentication. Download from the IAM Roles Anywhere documentation.
• OpenSSL 3.5 or later (required for ML-DSA signature verification and mu computation).
• ORAS CLI v1.2 or later (for attaching Open Container Initiative (OCI) referrer artifacts to container images). Alternatively, you can use the Notation CLI with a custom plugin or any OCI-compliant client that supports the OCI Referrers API.
• jq, for constructing and parsing the in-toto Statements used for SBOM attestation and for parsing ORAS output.
• Amazon Inspector SBOM Generator (Sbomgen), or another tool that produces a CycloneDX JSON SBOM, for the SBOM example. Sbomgen runs only on Linux (x86-64 and ARM64), so run that step on a Linux build agent or container. It uses the registry credentials that oras login stores in the Docker config file, and it also accepts them through --username or the INSPECTOR_SBOMGEN_USERNAME and INSPECTOR_SBOMGEN_PASSWORD environment variables.
• Java 11 or later and Apache Maven, to run the diy-code-signing-kms-private-ca sample that produces the code-signing CSR in Step 4 of Building the ML-DSA PKI foundation.
• A container image pushed to Amazon ECR (for the container signing example).

Start by creating the certificate hierarchy that will anchor your supply chain trust. AWS Private CA generates and stores its own CA signing keys internally in hardware-protected HSMs when you create a certificate authority. Use ML-DSA-87 (NIST security category 5, defined as at least as hard to break as exhaustive key search on AES-256) for the root CA because it protects the longest-lived trust anchor. Use ML-DSA-65 (NIST security category 3) for the subordinate CA to balance security with smaller signature sizes for frequently issued certificates.

The parameter sets used here, ML-DSA-87 and ML-DSA-65, are also the approved set in the example algorithm policy that Part 2 of this series turns into enforceable controls. If your organization has standardized on a different set, substitute it in the commands that follow; the build steps don’t change.

Step 1: Create the root CA in AWS Private CA

When you specify KeyAlgorithm: ML_DSA_87, Private CA generates an ML-DSA-87 key pair internally. You don’t need to create a separate AWS KMS key for the CA.

aws acm-pca create-certificate-authority \
  --certificate-authority-configuration '{
    "KeyAlgorithm": "ML_DSA_87",
    "SigningAlgorithm": "ML_DSA_87",
    "Subject": {
      "CommonName": "PQC Supply Chain Root CA",
      "Organization": "Example Corp",
      "Country": "US"
    }
  }' \
  --certificate-authority-type ROOT

After creation, the root CA is in PENDING_CERTIFICATE state. Activate it by self-signing its certificate. This is where you set the root CA’s validity period (10 years is typical for a root):

# Retrieve the root CA's CSR
aws acm-pca get-certificate-authority-csr \
  --certificate-authority-arn <root-ca-arn> \
  --output text > root-ca.csr

# Self-sign the root CA certificate (valid for 10 years)
ROOT_CERT_ARN=$(aws acm-pca issue-certificate \
  --certificate-authority-arn <root-ca-arn> \
  --csr fileb://root-ca.csr \
  --signing-algorithm ML_DSA_87 \
  --validity Value=3650,Type=DAYS \
  --template-arn arn:aws:acm-pca:::template/RootCACertificate/V1 \
  --query CertificateArn --output text)

# Retrieve and install the self-signed root certificate
aws acm-pca get-certificate \
  --certificate-authority-arn <root-ca-arn> \
  --certificate-arn $ROOT_CERT_ARN \
  --query Certificate --output text > root-ca-cert.pem

aws acm-pca import-certificate-authority-certificate \
  --certificate-authority-arn <root-ca-arn> \
  --certificate fileb://root-ca-cert.pem

Step 2: Create a subordinate CA for code signing

aws acm-pca create-certificate-authority \
  --certificate-authority-configuration '{
    "KeyAlgorithm": "ML_DSA_65",
    "SigningAlgorithm": "ML_DSA_65",
    "Subject": {
      "CommonName": "PQC Code Signing CA",
      "Organization": "Example Corp",
      "Country": "US"
    }
  }' \
  --certificate-authority-type SUBORDINATE

After creation, the subordinate CA enters PENDING_CERTIFICATE state with no link to the root. Activate it by signing its certificate signing request (CSR) with the root CA:

# Retrieve the subordinate CA's certificate signing request
aws acm-pca get-certificate-authority-csr \
  --certificate-authority-arn <sub-ca-arn> \
  --output text > sub-ca.csr

# Sign the CSR using the root CA (this establishes the trust hierarchy)
SUB_CERT_ARN=$(aws acm-pca issue-certificate \
  --certificate-authority-arn <root-ca-arn> \
  --csr fileb://sub-ca.csr \
  --signing-algorithm ML_DSA_87 \
  --validity Value=1825,Type=DAYS \
  --template-arn arn:aws:acm-pca:::template/SubordinateCACertificate_PathLen0/V1 \
  --query CertificateArn --output text)

# Retrieve the signed certificate and import it to activate the subordinate CA
aws acm-pca get-certificate \
  --certificate-authority-arn <root-ca-arn> \
  --certificate-arn $SUB_CERT_ARN \
  --query Certificate --output text > sub-ca-cert.pem

aws acm-pca import-certificate-authority-certificate \
  --certificate-authority-arn <sub-ca-arn> \
  --certificate fileb://sub-ca-cert.pem \
  --certificate-chain fileb://root-ca-cert.pem

Step 3: Create the code-signing key in AWS KMS

Your build agents use this key to sign artifacts, so their IAM roles need the kms:Sign permission on it. The end-entity certificate issued by your subordinate CA binds an identity to this key’s public key.

aws kms create-key \
  --key-spec ML_DSA_65 \
  --key-usage SIGN_VERIFY \
  --description "PQC code-signing key for supply chain" \
  --tags TagKey=Purpose,TagValue=CodeSigning

Create an alias for convenient reference in your CI/CD scripts:

aws kms create-alias \
  --alias-name alias/pqc-code-signing \
  --target-key-id <key-id-from-above>

Step 4: Issue the end-entity code signing certificate

To issue a certificate, you need a CSR. Because the ML-DSA private key never leaves the AWS KMS HSM, you can’t generate the CSR locally with OpenSSL. Instead, you construct the CSR’s TBS (to-be-signed) structure containing your subject name and public key, sign it remotely by calling aws kms sign, and assemble the final CSR from those components.

For a working implementation that assembles an ML-DSA CSR around a KMS-managed private key and signs it with AWS KMS, see Post-quantum (ML-DSA) code signing with AWS Private CA and AWS KMS and its diy-code-signing-kms-private-ca sample on GitHub (Java, built with Maven). There is no AWS CLI-only path for this step: the CSR must be constructed and signed in code, and the sample is the reference implementation.

After you have the code-signing.csr file, issue the certificate from your subordinate CA:

# Issue a code-signing certificate valid for one year, signed by the subordinate CA using ML-DSA-65.
# Replace <sub-ca-arn> with the ARN that create-certificate-authority returned for the subordinate CA in Step 2.
CODE_SIGNING_CERT_ARN=$(aws acm-pca issue-certificate \
  --certificate-authority-arn <sub-ca-arn> \
  --csr fileb://code-signing.csr \
  --signing-algorithm ML_DSA_65 \
  --validity Value=365,Type=DAYS \
  --template-arn arn:aws:acm-pca:::template/CodeSigningCertificate/V1 \
  --query CertificateArn --output text)

# Retrieve the issued certificate
aws acm-pca get-certificate \
  --certificate-authority-arn <sub-ca-arn> \
  --certificate-arn $CODE_SIGNING_CERT_ARN \
  --query Certificate --output text > code-signing-cert.pem

This certificate binds the signer identity to the AWS KMS code-signing key material. Note that the certificate validity period (365 days in this example) determines how long verifiers can validate signatures by checking the certificate alone. AWS Private CA lets you choose the validity period of an issued certificate, as long as it ends no later than the issuing CA’s own certificate; in this example the subordinate CA certificate from Step 2 is valid for 1,825 days, so the three-year option below fits inside it. For artifacts that must remain verifiable beyond the certificate’s expiry, consider these approaches:

• Longer certificate validity – For private PKI, code-signing certificates are commonly issued with multi-year validity (for example, Value=1095,Type=DAYS for 3 years) because you control the issuance policy.
• Trusted timestamping (RFC 3161) – Countersign each signature with an RFC 3161 timestamp from a Time Stamping Authority (TSA) at signing time. The timestamp proves the signature was created while the certificate was valid, allowing verification indefinitely. AWS doesn’t provide a managed TSA; use a third-party TSA or operate your own.
• Re-signing – Before a certificate expires, re-sign active artifacts with a newly issued certificate.

Distribute the certificate chain (root CA certificate, subordinate CA certificate, and end-entity certificate) to systems that need to verify your signatures.

With the PKI hierarchy in place, integrate ML-DSA signing into your build process. The approach depends on artifact type. ML-DSA-65 signatures are 3,309 bytes (compared to 64 bytes for ECDSA P-256), and the latency of the AWS KMS Sign operation for ML-DSA is comparable to existing asymmetric operations. The larger signature size has negligible impact when stored alongside multi-megabyte container images or packages, but is worth accounting for in size-constrained environments like embedded device images transmitted over low-bandwidth links.

Most build outputs, including compiled binaries and packages, are far larger than the 4,096-byte limit that AWS KMS places on the message passed to aws kms sign. Make the EXTERNAL_MU method your pipeline’s default: compute the 64-byte message representative (mu) externally with OpenSSL 3.5, then pass only mu to AWS KMS. This handles an artifact of any size and never needs to change as your artifacts grow. AWS KMS also accepts content directly with --message-type RAW, up to 4,096 bytes. Use RAW only for inputs whose size is fixed by construction, such as the 71-character manifest digest in the container image example that follows. An artifact that fits today is likely to outgrow the limit later, and a pipeline that depends on RAW fails when it does. Both methods produce cryptographically identical signatures. The verifier doesn’t need to know which method was used; standard ML-DSA verification applies in either case.

# Retrieve the public key
aws kms get-public-key --key-id alias/pqc-code-signing \
  --query PublicKey --output text | base64 --decode > pk.der

# Extract raw public key bytes from the DER-encoded SubjectPublicKeyInfo.
# The BIT STRING starts at offset 17 for all ML-DSA key sizes. 
# Expected output: 1,312 bytes (ML-DSA-44), 1,952 bytes (ML-DSA-65), or 2,592 bytes (ML-DSA-87).

openssl asn1parse -inform DER -in pk.der -strparse 17 -noout -out pk_raw.bin

# Compute mu per FIPS 204 Section 6.2
printf '\x00\x00' | cat - artifact.bin > m_prime.bin
openssl dgst -shake256 -xoflen 64 -binary pk_raw.bin > pk_hash.bin
cat pk_hash.bin m_prime.bin | openssl dgst -shake256 -xoflen 64 -binary > mu.bin

# Sign the mu value
aws kms sign \
  --key-id alias/pqc-code-signing \
  --message fileb://mu.bin \
  --message-type EXTERNAL_MU \
  --signing-algorithm ML_DSA_SHAKE_256 \
  --query Signature --output text | base64 --decode > artifact.bin.sig

Store the .sig file alongside the artifact in Amazon S3 or your package repository. Protect the integrity of both the artifact and its signature by enabling S3 Object Lock (WORM protection) or at minimum S3 Versioning with restricted write access. If an attacker can modify or delete either file, verification can’t be trusted. For comprehensive guidance, see Security best practices for Amazon S3.

For container image signing, use the OCI referrer pattern, defined in the OCI Image and Distribution Specifications v1.1. A referrer is an artifact, such as a signature or an SBOM, whose manifest names another image in its subject field. Registries that support the pattern, including Amazon ECR, return these artifacts when a client asks for an image’s referrers. This approach uses AWS KMS ML-DSA keys to generate the signature and stores it as a referrer of the image:

# Repository URI used by ORAS here and in the SBOM steps; replace the account ID and Region
ECR_URI=111122223333.dkr.ecr.us-east-1.amazonaws.com/myapp

# Authenticate ORAS to Amazon ECR
aws ecr get-login-password --region us-east-1 | \
  oras login --username AWS --password-stdin 111122223333.dkr.ecr.us-east-1.amazonaws.com

# Get the image manifest digest
IMAGE_DIGEST=$(aws ecr batch-get-image \
  --repository-name myapp \
  --image-ids imageTag=v1.2.0 \
  --query 'images[0].imageId.imageDigest' --output text)

# Sign the digest. It is a fixed 71 characters, so RAW is safe here (see Binary and package signing).
# AWS CLI v2 expects blob parameters to be base64, so write the digest to a file and pass it with fileb://
echo -n "$IMAGE_DIGEST" > digest.txt
aws kms sign \
  --key-id alias/pqc-code-signing \
  --message fileb://digest.txt \
  --message-type RAW \
  --signing-algorithm ML_DSA_SHAKE_256 \
  --query Signature --output text | base64 --decode > image.mldsa.sig

# Attach signature as OCI referrer using ORAS
oras attach $ECR_URI@$IMAGE_DIGEST \
  --artifact-type "application/vnd.example.mldsa-signature.v1" \
  image.mldsa.sig:application/octet-stream

At deployment time, your admission controller retrieves the referrer artifact and verifies the ML-DSA signature before allowing the image to run. See Verifying a container image signature before deployment later in this post for the full verification workflow and admission control implementation.

A software bill of materials (SBOM) is a structured inventory of every component, library, and dependency included in a software release. It lists package names, versions, suppliers, and dependency relationships. Organizations use SBOMs to track vulnerabilities, verify license compliance, and respond quickly when a compromised component is disclosed.

Signing the SBOM file on its own proves that the inventory hasn’t been tampered with since it was signed, but not which artifact it describes. If the image signature and the SBOM signature are independent of each other, an attacker who holds a legitimately signed SBOM for a clean image can present it alongside a different signed image and claim that the two correspond. Attaching the SBOM to the image as an OCI referrer doesn’t close this gap by itself, because the subject field of a referrer manifest is set by whoever pushes the referrer; it’s a discovery mechanism, not a cryptographic binding. The binding must be inside the bytes that the signature covers.

To create that binding, wrap the SBOM in an in-toto Statement before signing it. A Statement names the artifacts it describes in its subject field, by digest, and carries the SBOM as its predicate. Because the image manifest digest is part of the signed content, a verifier can confirm that the SBOM in hand was issued for exactly the image it is about to deploy, and a swapped SBOM or a swapped digest fails verification. Embedding the SBOM in the image filesystem is another way to bind the two, because the image digest then covers the SBOM bytes. This post uses the Statement approach because it also works for SBOMs generated by scanning a finished image, for artifacts that have no filesystem such as binaries and firmware, and for images that are already in your registry, and because the SBOM stays readable from the registry without pulling the image.

Generate the SBOM with your preferred tool (such as Amazon Inspector SBOM Generator (Sbomgen) for container images), wrap it in a Statement, and sign the Statement with the same EXTERNAL_MU step you use for binaries:

# Scan the pushed image by digest so the inventory describes exactly the manifest you signed.
# Sbomgen reuses the registry login from the oras login step in the previous section.
inspector-sbomgen container --image $ECR_URI@$IMAGE_DIGEST -o sbom.json

# Wrap the SBOM in an in-toto Statement whose subject is the image manifest digest.
# This puts the digest inside the signed bytes, which binds this SBOM to this image.
jq -n \
  --arg name "$ECR_URI" \
  --arg digest "${IMAGE_DIGEST#sha256:}" \
  --slurpfile sbom sbom.json \
  '{
    "_type": "https://in-toto.io/Statement/v1",
    "subject": [{"name": $name, "digest": {"sha256": $digest}}],
    "predicateType": "https://cyclonedx.org/bom",
    "predicate": $sbom[0]
  }' > sbom.statement.json

# Sign using EXTERNAL_MU (same mu computation as binary signing above,
# with sbom.statement.json as the input)
# ... compute mu from sbom.statement.json ...
aws kms sign \
  --key-id alias/pqc-code-signing \
  --message fileb://mu.bin \
  --message-type EXTERNAL_MU \
  --signing-algorithm ML_DSA_SHAKE_256 \
  --query Signature --output text | base64 --decode > sbom.statement.json.sig

# Attach the Statement and its signature to the image as an OCI referrer so
# verifiers discover them alongside the image signature
oras attach $ECR_URI@$IMAGE_DIGEST \
  --artifact-type "application/vnd.example.mldsa-sbom-attestation.v1" \
  sbom.statement.json:application/vnd.in-toto+json \
  sbom.statement.json.sig:application/octet-stream

Ship the Statement and its .sig file together; the SBOM itself is the Statement’s predicate field, and verifiers extract it after verification. The same pattern binds an SBOM to a binary or package: set the subject digest to the SHA-256 of the file instead of an image manifest digest. The Verifying a binary or SBOM signature section later in this post walks through the full verification workflow, including the subject check that completes the binding.

Signing artifacts with ML-DSA keys addresses the integrity of the artifacts themselves. Separately, build agents running outside AWS (on-premises Jenkins, self-hosted GitHub Actions runners, or third-party CI/CD systems) authenticate using X.509 certificates or OIDC tokens typically signed with RSA or ECDSA. These authentication credentials represent a classical cryptographic dependency in the pipeline. IAM Roles Anywhere with ML-DSA certificates replaces that dependency: the credential that the external build agent presents, and the CA that vouches for it, no longer rely on algorithms that are vulnerable to quantum attack. (The TLS connection that carries the request still authenticates the AWS endpoint with a traditional certificate, as public TLS does today; that is a transport concern separate from the signing identity this post is about.) Build agents running on AWS with an attached IAM role don’t have this dependency: their API requests are authenticated with AWS Signature Version 4 (SigV4), which uses a symmetric hash-based message authentication code (HMAC-SHA256). Symmetric schemes like this are already considered quantum-resistant, so no additional action is needed.

Step 1: Register a workload identity CA as a trust anchor

For this walkthrough, reuse the subordinate CA that you created earlier; that avoids the monthly charge for a second CA, and the commands that follow assume it. In production, use a dedicated subordinate CA for workload identity certificates, separate from your code-signing CA. This limits the blast radius if either CA is compromised and has to be revoked, and it lets you apply different issuance policies to the two certificate types.

# Use the ARN that create-certificate-authority returned when you created the
# subordinate CA earlier. If you no longer have it, list your CAs and copy the
# ARN of the SUBORDINATE one.
aws acm-pca list-certificate-authorities \
  --query "CertificateAuthorities[?Type=='SUBORDINATE'].[Arn,Status,CertificateAuthorityConfiguration.Subject.CommonName]" \
  --output table
SUB_CA_ARN=arn:aws:acm-pca:us-east-1:111122223333:certificate-authority/abc-123   # replace with yours

# Register the subordinate CA as an IAM Roles Anywhere trust anchor
aws rolesanywhere create-trust-anchor \
  --name "PQC-CI-CD-Trust-Anchor" \
  --source "sourceType=AWS_ACM_PCA,sourceData={acmPcaArn=$SUB_CA_ARN}" \
  --enabled

Step 2: Create the build agent role and a profile that references it

An IAM Roles Anywhere profile specifies which IAM roles a workload can assume after it authenticates with a certificate from your trust anchor. Create the role first. Its trust policy lets IAM Roles Anywhere assume it only through your trust anchor and only for a certificate whose subject common name identifies the build agent; without those conditions, any certificate that the trust anchor CA issues could assume the role. Its permissions policy grants only what signing requires, on only the code-signing key. Save the trust policy as build-agent-trust.json, replacing the trust anchor ARN with the one that create-trust-anchor returned in Step 1:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {"Service": "rolesanywhere.amazonaws.com"},
      "Action": ["sts:AssumeRole", "sts:TagSession", "sts:SetSourceIdentity"],
      "Condition": {
        "ArnEquals": {
          "aws:SourceArn": "arn:aws:rolesanywhere:us-east-1:111122223333:trust-anchor/ta-id"
        },
        "StringEquals": {
          "aws:PrincipalTag/x509Subject/CN": "pqc-build-agent-01"
        }
      }
    }
  ]
}

Save the permissions policy as build-agent-permissions.json, replacing the resource with your key’s ARN (aws kms describe-key --key-id alias/pqc-code-signing --query KeyMetadata.Arn --output text):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["kms:Sign", "kms:GetPublicKey", "kms:DescribeKey"],
      "Resource": "<code-signing-key-arn>"
    }
  ]
}

Create the role, attach the policy, and then create a profile that references the role:

aws iam create-role \
  --role-name PQCBuildAgentRole \
  --assume-role-policy-document file://build-agent-trust.json

aws iam put-role-policy \
  --role-name PQCBuildAgentRole \
  --policy-name pqc-code-signing \
  --policy-document file://build-agent-permissions.json

aws rolesanywhere create-profile \
  --name "PQC-Build-Agent-Profile" \
  --role-arns "arn:aws:iam::111122223333:role/PQCBuildAgentRole" \
  --enabled

Step 3: Issue the build agent certificate and configure the credential helper

Generate the build agent’s ML-DSA-65 key pair and CSR locally with OpenSSL, have the subordinate CA issue a client authentication certificate for it, and then configure the IAM Roles Anywhere credential helper to present that certificate:

# Generate the build agent's key pair and CSR. The CN must match the
# aws:PrincipalTag/x509Subject/CN condition in the role's trust policy.
openssl genpkey -algorithm ML-DSA-65 -out build-agent-mldsa-key.pem
chmod 400 build-agent-mldsa-key.pem
openssl req -new -key build-agent-mldsa-key.pem \
  -subj "/C=US/O=Example Corp/CN=pqc-build-agent-01" \
  -out build-agent.csr

# Issue a 30-day client authentication certificate from the subordinate CA
# registered as the trust anchor in Step 1.
BUILD_AGENT_CERT_ARN=$(aws acm-pca issue-certificate \
  --certificate-authority-arn $SUB_CA_ARN \
  --csr fileb://build-agent.csr \
  --signing-algorithm ML_DSA_65 \
  --validity Value=30,Type=DAYS \
  --template-arn arn:aws:acm-pca:::template/EndEntityClientAuthCertificate/V1 \
  --query CertificateArn --output text)

aws acm-pca get-certificate \
  --certificate-authority-arn $SUB_CA_ARN \
  --certificate-arn $BUILD_AGENT_CERT_ARN \
  --query Certificate --output text > build-agent-mldsa.pem

# In production, store the private key in a TPM or HSM rather than as a plaintext file.
aws_signing_helper credential-process \
  --certificate build-agent-mldsa.pem \
  --private-key build-agent-mldsa-key.pem \
  --trust-anchor-arn arn:aws:rolesanywhere:us-east-1:111122223333:trust-anchor/ta-id \
  --profile-arn arn:aws:rolesanywhere:us-east-1:111122223333:profile/profile-id \
  --role-arn arn:aws:iam::111122223333:role/PQCBuildAgentRole

The helper produces short-lived AWS Security Token Service (AWS STS) credentials scoped to the build role, replacing long-lived AWS access keys. In production, protect the ML-DSA private key with hardware-backed storage such as a Trusted Platform Module (TPM), an HSM, or an encrypted credential store with restricted file permissions. The plaintext file path shown here is for demonstration purposes. The private key authorizes access to your code-signing infrastructure and should be treated with the same protection level as any signing key. On Linux build agents, at minimum restrict the file to chmod 400 owned by the service account and consider using a PKCS#11 provider to avoid storing the raw key on disk entirely.

A signing architecture is only as strong as its verification. Producing signatures is half the story. The other half is giving the systems and people that deploy your artifacts, such as deployment pipelines, Kubernetes admission controllers, and engineers installing a package, a concrete way to verify those signatures and make trust decisions before an artifact is deployed or run.

Before any verifier can check a signature, it needs to trust the CA that issued the code-signing certificate. This trust is established through a trust store: a local collection of root CA certificates that a system treats as authoritative. Trust stores exist in operating systems (such as /etc/pki/ca-trust/ on Linux, Keychain on macOS), language run-times (Java cacerts), container base images, and custom application configurations.

For private PKI hierarchies like the one in this post, your root CA certificate isn’t in any default public trust store. You must explicitly add it to the trust stores of every system that will verify your signatures. Common approaches include:

• Baking the root CA certificate into your container base images during the build process
• Distributing it through configuration management (AWS Systems Manager Parameter Store, AWS Secrets Manager, or your existing configuration tooling)
• Publishing it to an internal S3 bucket with versioning and access logging
• Storing it as an OCI artifact in Amazon ECR alongside the images it protects

Treat trust store management with the same rigor as credential management. If an attacker can inject a rogue root certificate into your trust stores, they can make forged signatures appear valid. Restrict write access to trust stores and audit changes through AWS CloudTrail or your system-level audit logs.

Exporting your certificate chain requires exporting the root CA, and then any subordinate CAs that issue certificates that will be involved in verifying signatures:

# Export the root CA certificate
aws acm-pca get-certificate-authority-certificate \
  --certificate-authority-arn <root-ca-arn> \
  --query Certificate --output text > root-ca.pem

# Export the subordinate CA certificate. Select only the Certificate field: for a
# subordinate CA this call also returns CertificateChain, and --output text would
# print both on one tab-separated line, producing a file that OpenSSL can't parse.
aws acm-pca get-certificate-authority-certificate \
  --certificate-authority-arn <sub-ca-arn> \
  --query Certificate --output text > sub-ca.pem
  
# Bundle the full chain for verifiers
cat root-ca.pem sub-ca.pem > trust-chain.pem

Verifiers use this chain to confirm that the code-signing public key was issued by a CA they trust.

The CAs in this post are created without a revocation configuration, and the verification commands that follow don’t check revocation status, which keeps the walkthrough short. In production, configure a certificate revocation list (CRL) or an Online Certificate Status Protocol (OCSP) responder on the subordinate CA (see Plan your AWS Private CA certificate revocation method in the AWS Private CA User Guide; both work with ML-DSA CAs). If you use a CRL, distribute it alongside the trust chain and add -crl_check -CRLfile <crl-file> to the openssl verify command below, where <crl-file> is the CRL that AWS Private CA publishes to your S3 bucket, so that a revoked code-signing certificate fails verification.

Verify the code-signing certificate against the trust chain, confirm that it is a code-signing certificate issued to the signer you expect, extract its public key, and then verify the signature with OpenSSL 3.5. The second check matters: a chain check alone accepts every certificate that the CA hierarchy has issued, including the build agent client certificates from the previous section, so pin the expected subject in your verifier configuration:

# Step 1: Verify that the code-signing certificate chains to the trusted root CA and carries the
# Code Signing extended key usage. -purpose codesign (OpenSSL 3.2 and later) rejects a certificate
# that chains correctly but was issued for another purpose, such as a build agent's client
# authentication certificate from the same subordinate CA.
# trust-chain.pem contains the root CA and subordinate CA certificates distributed in the previous section.
openssl verify -CAfile trust-chain.pem -purpose codesign code-signing-cert.pem
# Confirm that the certificate was issued to the signer you expect. Replace <signer-cn> with the
# common name in your code-signing certificate's subject.
openssl x509 -in code-signing-cert.pem -noout -subject -nameopt RFC2253 | grep -q "CN=<signer-cn>" || { echo "Unexpected signer"; exit 1; }

# Step 2: Extract the public key from the verified certificate.
openssl x509 -in code-signing-cert.pem -pubkey -noout > signing-key.pem

# Step 3: Verify the artifact signature using the trusted public key.
openssl pkeyutl -verify \
  -pubin -inkey signing-key.pem \
  -in artifact.bin \
  -sigfile artifact.bin.sig \
  -rawin

A successful verification prints Signature Verified Successfully and returns an exit status of 0. A failed verification prints Signature Verification Failure and returns a nonzero exit status. In shell scripts, check the exit status of each command, or run the block with set -e, so that a failed certificate chain check stops the script before signature verification and a failed signature check blocks downstream actions.

For an SBOM, run the same three steps with sbom.statement.json as the -in file and sbom.statement.json.sig as the -sigfile, then add a fourth: confirm that the subject digest inside the verified Statement is the digest of the artifact you are about to deploy. Signature verification proves the Statement is authentic; the subject check proves it describes this artifact and not another one. The digest you compare against must come from the artifact itself, resolved from the registry for an image or computed from the file for a binary, never from the Statement. For an image, this is the same digest you verify the image signature against, so the two checks together establish that this SBOM describes this signed image.

# Step 4 (SBOM only): the signed Statement must name the image being deployed.
# Resolve the digest from Amazon ECR, not from the Statement.
IMAGE_DIGEST=$(aws ecr batch-get-image \
  --repository-name myapp \
  --image-ids imageTag=v1.2.0 \
  --query 'images[0].imageId.imageDigest' --output text)
SUBJECT_DIGEST=$(jq -r '.subject[0].digest.sha256' sbom.statement.json)
[ "sha256:$SUBJECT_DIGEST" = "$IMAGE_DIGEST" ] || { echo "SBOM does not describe this image"; exit 1; }

# Only now extract the SBOM for downstream tooling
jq '.predicate' sbom.statement.json > sbom.json

For a binary, compare the subject digest to the output of sha256sum artifact.bin instead. Reject the SBOM if either the signature or the subject check fails.

Pull the ML-DSA signature from the OCI referrer artifact in Amazon ECR, then verify it against the image manifest digest:

# Pull the signature referrer. ORAS 1.2 lists referrers under "manifests";
# ORAS 1.3 and later list them under "referrers". The jq filter accepts either.
oras discover $ECR_URI@$IMAGE_DIGEST \
  --artifact-type "application/vnd.example.mldsa-signature.v1" --format json | \
  jq -r '(.referrers // .manifests)[0].digest' | \
  xargs -I{} oras pull $ECR_URI@{} -o /tmp/sig/

# Re-derive the image digest and verify
echo -n "$IMAGE_DIGEST" > /tmp/digest.txt
openssl pkeyutl -verify \
  -pubin -inkey signing-key.pem \
  -in /tmp/digest.txt \
  -sigfile /tmp/sig/image.mldsa.sig \
  -rawin

# Exit code 0 = image is authentic; non-zero = reject deployment

The same oras discover command with --artifact-type "application/vnd.example.mldsa-sbom-attestation.v1" retrieves the SBOM attestation and its signature; verify them as described in the previous section, including the subject check. The jq filter shown takes the first referrer of the requested type. Don’t rely on the order in which a registry lists referrers, and expect more than one after an image has been re-signed: a production verifier should check every referrer of that type and accept the image if at least one signature verifies against a certificate that chains to your trust store. Taking only the first can reject a correctly signed image, but it can never accept an unsigned one.

To enforce these checks at deployment time, run them in a Kubernetes admission controller, a component that the API server calls before it accepts a pod and that can reject the request. You can write one as a validating admission webhook (see Dynamic Admission Control in the Kubernetes documentation), or use Open Policy Agent (OPA) Gatekeeper, an admission webhook that enforces policies evaluated by the OPA policy engine, and package the verification steps as a Gatekeeper external data provider that a constraint calls. For a worked example of gating Amazon EKS deployments on image signatures, see Announcing Container Image Signing with AWS Signer and Amazon EKS, which verifies AWS Signer signatures at admission with Kyverno and discusses the Gatekeeper and custom webhook alternatives. The ML-DSA signature referrer in this post is a custom artifact type that those tools don’t verify out of the box, so the verification steps in this section run in code you supply. In either design, the controller runs before any pod is scheduled, rejecting images that lack a valid ML-DSA signature referrer in Amazon ECR or whose SBOM attestation doesn’t name the image being deployed.

Signature verification stops the use of an untrusted artifact at deployment. Organization-level controls apply earlier and more broadly: preventive controls stop teams from creating new traditional signing keys, and detective controls record every signing operation so you can audit which principal signed each artifact and when.

Preventive guardrails. Use an AWS Organizations Service control policy (SCP) to block creation of traditional signing keys in security-critical accounts, forcing all new signing workloads to use ML-DSA:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RequirePQCSigningKeys",
      "Effect": "Deny",
      "Action": "kms:CreateKey",
      "Resource": "*",
      "Condition": {
        "StringNotLike": {
          "kms:KeySpec": "ML_DSA_*"
        },
        "StringEquals": {
          "kms:KeyUsage": "SIGN_VERIFY"
        }
      }
    }
  ]
}

This SCP prevents teams from creating new RSA or ECDSA signing keys, enforcing quantum-resistant signatures as the only option for new workloads. Note that this SCP constrains the algorithm family, not the parameter set: it would still permit an ML-DSA-44 key, which this architecture doesn’t use. Part 2 of this series picks up at exactly this point. It shows how to express a complete algorithm policy (in its example, approve ML-DSA-65 and ML-DSA-87 and exclude ML-DSA-44) as layered controls using the kms:KeySpec condition key in SCPs, IAM permission boundaries, and AWS KMS key policies; how to restrict the use of keys that already exist; and how to govern surfaces such as AWS Private CA that have no algorithm condition key. This SCP and the parameter-set SCP in Part 2 work together: attach both, and every new signing key must use an approved parameter set.

Audit trail. AWS CloudTrail logs every kms:Sign and kms:Verify API call, providing an immutable record of when each artifact was signed and by which principal. Use an organization trail and query its logs with Amazon Athena to analyze signing activity across accounts and correlate it with your deployment events.

The following table maps regulatory milestones to specific actions you can take today using the architecture described in this post. This information is provided for general awareness only and does not constitute legal or compliance advice. Consult your legal and compliance teams to determine your organization’s specific obligations. For a broader overview of global PQC regulatory timelines, see Migrating to Post-Quantum Cryptography.

Note for NSS customers and other CNSA 2.0 adopters: CNSA 2.0 is the quantum-resistant algorithm suite that the NSA requires for NSS, the federal systems that carry classified information or support intelligence, cryptologic, military command and control, and weapons functions. When ML-DSA is used for any signature purpose, including software and firmware signing, CNSA 2.0 permits only the ML-DSA-87 parameter set. (For software and firmware signing it also approves the stateful hash-based schemes LMS and XMSS, which this post doesn’t cover.) The ML-DSA-65 parameter set used in this post’s code-signing examples is appropriate for commercial workloads but isn’t permitted on NSS. CNSA 2.0 is binding only for NSS, but it’s followed voluntarily well beyond that boundary: by suppliers to the US defense and intelligence market, in the United States and allied countries, whose products must meet NSS requirements, and by commercial organizations in and outside the United States that adopt it as a conservative baseline because it specifies the highest NIST security category and removes the need to revisit parameter choices later. If your systems fall under CNSA 2.0 scope, or you’ve chosen to align with it, substitute ML-DSA-87 for all signing keys and certificates. Part 2 of this series shows how to make that substitution enforceable: the same kms:KeySpec controls that exclude ML-DSA-44 in its example can be pointed at ML_DSA_87 as the only approved key spec.

For organizations serving both US government and EU commercial markets, the January 2027 and December 2027 deadlines converge into a single migration window. Starting now provides margin for testing, key rotation planning, and certificate chain validation.

You created a number of cryptographic resources in the prior steps and examples. AWS Private CA charges a monthly fee per CA and AWS KMS charges per key per month. To avoid ongoing charges, delete them if you aren’t actively using them. Start by listing what exists, so that you catch every resource you created, including duplicates from repeated runs. These commands list one Region; if you worked in more than one, repeat them with --region for each.

# List every private CA with its type, status, key algorithm, and subject.
# A CA you already deleted shows DELETED until its restoration window ends.
aws acm-pca list-certificate-authorities \
  --query 'CertificateAuthorities[].[Arn,Type,Status,CertificateAuthorityConfiguration.KeyAlgorithm,CertificateAuthorityConfiguration.Subject.CommonName]' \
  --output table

# List every ML-DSA key in AWS KMS with its state and creation date.
# A key already scheduled for deletion shows PendingDeletion.
for KEY_ID in $(aws kms list-keys --query 'Keys[].KeyId' --output text); do
  aws kms describe-key --key-id "$KEY_ID" \
    --query 'KeyMetadata.[KeyId,KeySpec,KeyState,CreationDate]' --output text
done | grep ML_DSA

# Show which alias points at which key. A key that no alias points at
# is usually a leftover from an earlier run.
aws kms list-aliases --query 'Aliases[].[AliasName,TargetKeyId]' --output text | grep -v '^alias/aws/'

AWS Private CA has no command that lists the certificates a CA has issued. If you need serial numbers for the revocation step that follows, generate an audit report with aws acm-pca create-certificate-authority-audit-report, which writes a list of every certificate the CA has issued or revoked to an S3 bucket that grants the service write access.

Delete each resource that you don’t need, including any extra copies that the listing revealed, then run the listing again to confirm that nothing is left:

# Revoke any issued end-entity certificates while the subordinate CA is still active
aws acm-pca revoke-certificate \
  --certificate-authority-arn <sub-ca-arn> \
  --certificate-serial <serial-number> \
  --revocation-reason CESSATION_OF_OPERATION

# Delete the IAM Roles Anywhere resources that reference the CA
aws rolesanywhere delete-profile --profile-id <profile-id>
aws rolesanywhere delete-trust-anchor --trust-anchor-id <trust-anchor-id>

# Delete the build agent role that the profile referenced, and the local copy of its private key
aws iam delete-role-policy --role-name PQCBuildAgentRole --policy-name pqc-code-signing
aws iam delete-role --role-name PQCBuildAgentRole
rm -f build-agent-mldsa-key.pem

# Delete the AWS KMS code-signing key (schedule 7-day waiting period).
# ScheduleKeyDeletion doesn't accept an alias, so resolve the key ID first.
KEY_ID=$(aws kms describe-key --key-id alias/pqc-code-signing \
  --query KeyMetadata.KeyId --output text)
aws kms schedule-key-deletion --key-id $KEY_ID --pending-window-in-days 7

# Remove the alias. It survives key deletion and would otherwise show up in the
# listing above on your next run.
aws kms delete-alias --alias-name alias/pqc-code-signing

# Disable and delete the subordinate CA
aws acm-pca update-certificate-authority \
  --certificate-authority-arn <sub-ca-arn> --status DISABLED
aws acm-pca delete-certificate-authority \
  --certificate-authority-arn <sub-ca-arn> --permanent-deletion-time-in-days 7

# Disable and delete the root CA
aws acm-pca update-certificate-authority \
  --certificate-authority-arn <root-ca-arn> --status DISABLED
aws acm-pca delete-certificate-authority \
  --certificate-authority-arn <root-ca-arn> --permanent-deletion-time-in-days 7

If you are adapting this for long-term use, plan for log retention. CloudTrail Event History keeps only the last 90 days of management events and is not a substitute for a trail. For the audit trail described earlier, create a trail that delivers to an S3 bucket with the lifecycle policy and access controls your retention requirements call for.

In this post, I showed you how to build a quantum-resistant software supply chain on AWS by combining ML-DSA signing in AWS KMS, ML-DSA certificate hierarchies in AWS Private CA, and ML-DSA workload authentication through IAM Roles Anywhere. The architecture covers the full pipeline; from authenticated build agents through signed binaries, container images, and SBOMs, to verification and policy enforcement at deployment.

Every ML-DSA capability used in this post is generally available. AWS Private CA and IAM Roles Anywhere support ML-DSA in all commercial AWS Regions, the AWS GovCloud (US) Regions, and the China Regions, and AWS KMS supports ML-DSA in commercial AWS Regions. You don’t need to wait for a standards body or third-party tool to finalize PQC support. The primitives are ready.

To get started, create your first ML-DSA key in AWS KMS and sign a test artifact. From there, extend to your PKI hierarchy, integrate IAM Roles Anywhere for your build agents, and add ML-DSA signatures to your container and SBOM workflows.

Building the pipeline is half the work. The other half is making sure the algorithm choices you made stay in force as accounts, teams, and standards change. That’s the subject of Part 2 of this series, Enforce your ML-DSA algorithm policy with SCPs and IAM, written for the security and governance teams who own that problem.

For more information, see:

• Quantum-resistant signing on AWS, Part 2: Enforce your ML-DSA algorithm policy with SCPs and IAM
• Post-quantum cryptography on AWS
• ML-DSA keys in AWS KMS
• Post-quantum ML-DSA code signing with AWS Private CA and AWS KMS
• IAM Roles Anywhere post-quantum digital certificates
• AWS post-quantum cryptography migration plan

If you have feedback about this post, submit comments in the Comments section below.

We explore how AWS neutralizes exposed IAM credentials using managed policies, detailing GitHub secret scanning and CloudTrail monitoring strategies.

The post From Exposure to Lockdown: How AWS Neutralizes Compromised IAM Credentials through Managed Policies appeared first on Unit 42.

We designed a behavioral clustering model to map cloud identity roles from audit logs, enabling continuous threat detection using standard SQL queries.

The post Unmasking Cloud Identities: From Behavioral Clustering to Automated Detection appeared first on Unit 42.

The principle of least privilege is straightforward to articulate but challenging to maintain at scale. When teams first deploy applications to AWS, they often grant broader permissions than strictly necessary; it’s faster to get things working, and the plan is always to tighten permissions later. But later rarely comes. Permissions accumulate, AWS Identity and Access Management (IAM) principals that once needed broad access for initial deployment retain those permissions long after they’re necessary, and some principals stop being used entirely. Even small teams face this challenge—permission reviews aren’t a one-time task but an ongoing operational burden that demands automation.

AWS IAM Access Analyzer addresses detection and recommendation. It identifies unused permissions across IAM roles and users: actions that haven’t been exercised, services that haven’t been accessed, and principals that aren’t being assumed at all. For each finding, it generates a recommended policy with the excess permissions removed. Security teams can see exactly what to fix, but manual remediation doesn’t persist. A security engineer can right-size a role today, but if that role is defined in an AWS CloudFormation template or AWS Cloud Development Kit (AWS CDK) stack, the next deployment restores the original permissions. The fix must live where the role is defined, and not every role starts in the same place. Some are managed through infrastructure-as-code (IaC), where remediation means updating source code and deploying through a pipeline. Others were created manually through the AWS Management Console and have no code representation. And some principals aren’t being used at all and need a controlled decommission path. Each scenario requires a different remediation strategy.

This post walks through an automated remediation workflow that bridges the gap between detection and action. Instead of findings accumulating in a dashboard waiting for someone to investigate, the automation classifies each role by how it was created and produces a ready-to-review remediation artifact: a pull request with production-ready CDK code and a plain-English explanation for IaC-managed roles, an issue with the recommended policy and step-by-step IaC migration guidance for manually created roles, or a soft-disable issue with a monitored decommission plan for unused principals. Each output flows through your existing code review and issue tracking processes—the same workflows your teams already follow. By the end of this post, you’ll have a pattern that converts IAM Access Analyzer findings into tested, deployable code changes rather than a growing backlog of security tickets.

Unused IAM permissions increase the attack surface. Removing unused permissions limits the actions available to any compromised credentials, reducing potential impact. Roles that aren’t being assumed represent unused resources; removing them simplifies your IAM inventory and reduces potential access paths that aren’t actively monitored.

The challenge isn’t knowing what to fix. As we said earlier, Access Analyzer provides both the findings and the recommended policies. The challenge is acting on that knowledge consistently across your environment. Each finding requires context:

• What the role does
• Who created the role
• Determining if the permission is unused or used infrequently
• If the role is managed in a CloudFormation stack, or was created through the console

Multiply this by hundreds of roles and security teams face a backlog that grows faster than they can address it.

Manual remediation compounds the problem. A security engineer can right-size a role directly in the console, but that fix is fragile. If the role is defined in an IaC template, the next deployment restores the original permissions. If it was created manually, there’s no record of what changed or why, and no easy way to revert if the change causes issues.

This is where IaC changes the equation. When roles are defined in code, remediation means updating that code. Changes flow through pull requests, are reviewed by the team that owns the role, and deploy consistently across environments. The fix becomes permanent, not a point-in-time correction that drifts back on the next deployment. And because every change is tracked in version control, teams can confidently remove permissions knowing they can revert if something breaks. That safety net matters; it’s often the difference between a team acting on a finding and leaving it in the backlog.

The solution automates remediation by connecting four capabilities: IAM Access Analyzer for detection and policy recommendations, CloudTrail for role attribution, Amazon Bedrock for CDK code generation and plain-English explanations, and your existing continuous integration and delivery (CI/CD) pipeline for remediation execution. The workflow operates on a core principle: every IAM role has an origin, and that origin determines the remediation path.

Figure 1 shows the solution architecture: Amazon EventBridge triggers an AWS Lambda orchestrator on a daily schedule. The Lambda orchestrator integrates with IAM Access Analyzer, CloudTrail, Amazon Bedrock, and Amazon CloudWatch. Each finding is routed to one of three remediation paths: a pull request for IaC-managed roles, an issue for manually created roles, and a soft-disable issue for unused roles.

Figure 1: The daily remediation workflow; from scheduled trigger to the three role-based remediation paths

On each scheduled run, the automation retrieves active findings from IAM Access Analyzer and queries CloudTrail to determine how each role was created. Roles created through CloudFormation or AWS CDK have a traceable origin: the service principal, stack name, and originating repository. Roles created manually through the console have a different origin: the IAM user who created them and the timestamp. This distinction drives the remediation strategy.

For IaC-managed roles, the automation retrieves the IAM Access Analyzer-recommended policy and uses Amazon Bedrock to wrap it in production-ready CDK code that includes the role definition and policy statements and imports what your CI/CD pipeline needs to deploy the update. It then creates a pull request in the originating repository. The pull request (PR) includes the updated CDK code, a policy diff showing exactly which permissions are being removed, and a plain-English explanation of the changes, for example, “This change removes write access to S3, keeping only read and list permissions.” Your existing code review process evaluates the change, and after being merged, the fix deploys consistently across environments.

For manually created roles, the automation creates an issue that includes the IAM Access Analyzer-recommended policy with unused permissions removed, a diff highlighting the changes, and an Amazon Bedrock-generated explanation of what the permission changes accomplish. The issue also provides guidance on importing the role into your IaC codebase. This gives teams an immediate remediation path while encouraging long-term governance through IaC adoption.

For roles that aren’t being assumed at all, the automation takes a more cautious approach. Instead of taking direct action, it creates an issue recommending a soft-disable workflow: attach a deny-all policy to the role, monitor for 30 days to confirm no workload depends on it, then delete. The issue provides the steps and context, the team executes the decommission through their preferred process, whether that’s a console change, an AWS Command Line Interface (AWS CLI) script, or a PR removing the role from the IaC. This controlled decommission path reduces the risk of removing a role that’s used infrequently or seasonally.

The solution supports both single-account and organization-wide deployment. In single-account mode, it uses an ACCOUNT_UNUSED_ACCESS analyzer to process findings for one account. In organization mode, it uses an ORGANIZATION_UNUSED_ACCESS analyzer deployed in a delegated administrator account, which generates findings across all member accounts from a single vantage point. The Lambda function automatically detects which analyzer type is available and extracts the account ID from each finding’s resource Amazon Resource Name (ARN), so role attribution and remediation routing work the same way regardless of scope.

This three-path strategy acknowledges operational reality. Not all roles start in IaC, not all unused roles are safe to delete immediately, and forcing immediate migration isn’t always practical. The solution provides a clear path forward for each scenario: remediate IaC roles through code, give teams actionable recommendations for manually created roles, and safely decommission what’s no longer needed. Over time, your infrastructure becomes increasingly code-driven, and remediation becomes a routine part of your CI/CD process rather than a manual security task.

Consider a company—call them AnyCompany—running 200 IAM roles across three AWS accounts. Some roles were created through AWS CDK stacks during initial deployment. Others were created manually through the console by engineers who needed quick access during incident response or prototyping. A handful haven’t been assumed in over 6 months. AnyCompany’s security team wants to act on their IAM Access Analyzer findings, but each role requires different handling. The solution’s architecture addresses this by routing each finding through a classification and remediation pipeline.

Figure 2 shows how each IAM Access Analyzer finding is processed:

1. The finding is first checked against exclusions and excluded findings are skipped.
2. Remaining findings are split by type: UnusedPermission findings retrieve a recommended policy from IAM Access Analyzer and then query CloudTrail for role origin, while UnusedIAMRole findings follow the unused role path.
3. By origin, IaC-managed roles generate AWS CDK code using Amazon Bedrock and create a pull request.
4. Manually created or unknown-origin roles create an issue with the recommended policy and IaC migration guidance.
5. Unused roles create a soft-disable issue to deny-all, monitor for 30 days, then delete.
6. All paths publish CloudWatch metrics.

Figure 2: Detailed component interactions—the orchestrator’s five steps, its four service integrations, and the three remediation paths

The rest of this section walks through each component using AnyCompany’s roles as examples.

Before processing any finding, the Lambda function loads an exclusion configuration and checks whether the role should be skipped. This prevents the automation from creating remediation items for roles that legitimately need broad permissions.

{
  "excluded_roles": [
    "arn:aws:iam::123456789012:role/BreakGlassRole",
    "arn:aws:iam::123456789012:role/ServiceLinkedRole"
  ],
  "excluded_permissions": [
    "iam:*",
    "sts:AssumeRole"
  ],
  "excluded_by_tag": {
    "NoRemediation": ["true"],
    "CriticalService": ["true"]
  },
  "min_unused_days": 30
}

AnyCompany excludes their break-glass role (used only during incidents), any service-linked roles, and roles tagged CriticalService. The min_unused_days threshold prevents false positives from seasonal workloads; a role that ran a quarterly batch job 25 days ago won’t generate a finding.

IAM Access Analyzer generates two types of findings relevant to this solution. UnusedPermission findings identify roles with permissions that haven’t been exercised within the analysis period. UnusedIAMRole findings identify roles that haven’t been assumed at all. The Lambda function queries both finding types separately because they follow different remediation paths.

The Lambda function auto-detects the analyzer type at startup. When ANALYZER_SCOPE is set to organization, it checks for an ORGANIZATION_UNUSED_ACCESS analyzer first and falls back to ACCOUNT_UNUSED_ACCESS if none exists. If multiple analyzers of the same type exist in the account, the Lambda function selects the first active analyzer returned by the API. To target a specific analyzer, set the ANALYZER_ARN environment variable explicitly. With an organization-level analyzer, findings include roles from all member accounts. The Lambda function extracts the account ID from each finding’s resource ARN (for example, account 111122223333 from arn:aws:iam::111122223333:role/MyRole) and carries that context through the entire pipeline: attribution, remediation, and issue or PR creation all include the originating account.

For UnusedPermission findings, the Lambda function calls GenerateFindingRecommendation to initiate policy generation, then retrieves the IAM Access Analyzer-recommended policy through the GetFindingRecommendation API. This is a key integration point: IAM Access Analyzer provides the right-sized policy with unused permissions removed, so the automation doesn’t need to generate policies itself.

Here’s what a typical finding looks like for one of AnyCompany’s application roles:

{
  "id": "a1b2c3d4-5678-90ab-cdef-example11111",
  "resource": "arn:aws:iam::123456789012:role/AnyCompanyOrderProcessorRole",
  "findingType": "UnusedPermission",
  "analyzedAt": "2026-03-01T00:00:00Z",
  "unusedPermissions": [
    { "action": "s3:PutObject", "lastAccessed": null },
    { "action": "s3:DeleteObject", "lastAccessed": null },
    { "action": "s3:PutBucketPolicy", "lastAccessed": null },
    { "action": "dynamodb:DeleteItem", "lastAccessed": null }
  ],
  "activePermissions": [
    { "action": "s3:GetObject", "lastAccessed": "2026-02-28T14:30:00Z" },
    { "action": "s3:ListBucket", "lastAccessed": "2026-02-28T14:30:00Z" },
    { "action": "dynamodb:Query", "lastAccessed": "2026-02-28T12:00:00Z" }
  ]
}

The OrderProcessorRole has write and delete permissions for Amazon Simple Storage Service (Amazon S3) and Amazon DynamoDB, but only uses read operations. The IAM Access Analyzer recommendation removes the four unused actions while preserving the three active ones.

For UnusedIAMRole findings, no recommendation is needed: the role isn’t being assumed at all, so the remediation is to disable or delete it. The Lambda function caps the number of unused role issues per run (configurable using MAX_UNUSED_ROLE_ISSUES, default 10) to avoid overwhelming teams with a flood of issues on the first execution.

For each finding, the Lambda function queries CloudTrail to determine how the role was created. The CreateRole event contains the information needed to classify the role’s origin.

An IaC-created role looks like this in CloudTrail:

{
  "eventName": "CreateRole",
  "userIdentity": {
    "type": "AWSService",
    "invokedBy": "cloudformation.amazonaws.com"
  },
  "requestParameters": {
    "roleName": "AnyCompanyOrderProcessorRole"
  },
  "userAgent": "cloudformation.amazonaws.com"
}

The cloudformation.amazonaws.com service principal and user agent tell the automation this role was created through a CloudFormation or AWS CDK deployment. The Lambda function then looks up the role’s tags to find the originating repository (stored in a Repository tag set during deployment).

A manually-created role looks different:

{
  "eventName": "CreateRole",
  "userIdentity": {
    "type": "IAMUser",
    "userName": "jstiles"
  },
  "requestParameters": {
    "roleName": "AnyCompanyIncidentResponseRole"
  },
  "userAgent": "console.amazonaws.com"
}

Here, the IAMUser type and console.amazonaws.com user agent indicate someone created this role through the console. Roles created through the AWS CLI show a similar pattern: the IAMUser type with a user agent like aws-cli/2.x.x. The automation classifies both console and AWS CLI-created roles as manually created, because neither has an IaC origin that can be updated programmatically. The automation captures the username and timestamp for the remediation issue.

When the Lambda function processes findings from an organization-level analyzer, the role might live in a different account than the one running the function. The automation handles this by assuming a cross-account role (configurable using CROSS_ACCOUNT_ROLE_NAME, defaulting to OrganizationAccountAccessRole) in the member account, then querying that account’s CloudTrail and IAM APIs for the CreateRole event. If the cross-account assume fails—because the role doesn’t exist in that account or permissions aren’t configured—the automation falls back gracefully, classifying the role as unknown origin and creating an issue with the account ID and available context. This approach helps the automation produce an actionable output for findings even when attribution is incomplete.

For IaC-managed roles with UnusedPermission findings, the Lambda function retrieves the IAM Access Analyzer-recommended policy and sends it to Amazon Bedrock to generate production-ready AWS CDK code. This is an important distinction: IAM Access Analyzer decides what the policy should be, and Amazon Bedrock wraps that policy in the AWS CDK constructs, imports, and resource definitions that the CI/CD pipeline needs to deploy the update.

The prompt instructs Amazon Bedrock to convert the recommended policy to AWS CDK code exactly as provided, with no modifications:

Generate Python CDK code that creates/updates the role with the
RECOMMENDED policy exactly as provided. Include proper imports
(aws_cdk, aws_iam), use CDK best practices (PolicyStatement,
proper resource ARNs), and add tags: ManagedBy=CDK,
RemediatedBy=AccessAnalyzer.

IAM Access Analyzer generates recommendations for both inline policies and customer managed policies. When a managed policy has partially unused permissions, the recommendation contains the full right-sized policy. The automation wraps this in AWS CDK code as an iam.ManagedPolicy construct. Note that if a managed policy is shared across multiple roles, the recommendation applies to the specific role’s usage pattern. In this case, the automation generates an issue for manual review rather than a PR, because modifying a shared policy could affect other roles.

The generated code goes through a validation step before inclusion in any PR. The Lambda function compiles the Python code to check for syntax errors and verifies that required AWS CDK patterns (iam, PolicyStatement) are present. If validation fails, the finding is logged as an error rather than creating a broken PR.

The solution doesn’t currently invoke the IAM Access Analyzer ValidatePolicy API to check the generated policy for errors or overly permissive statements. However, this is a natural extension point. Teams can add a validation step that calls ValidatePolicy on the Amazon Bedrock-generated policy before including it in a PR, detecting issues like missing resource constraints or invalid action names.

Amazon Bedrock also generates a plain-English explanation of the policy changes. For AnyCompany’s OrderProcessorRole, the explanation might read:

“The role currently has full S3 write access and DynamoDB delete permissions, but only uses read operations. Removing s3:PutObject, s3:DeleteObject, s3:PutBucketPolicy, and dynamodb:DeleteItem reduces the scope of impact if credentials are compromised, while preserving the s3:GetObject, s3:ListBucket, and dynamodb:Query permissions the application needs.”

The solution uses the Anthropic Claude Sonnet model on Amazon Bedrock for CDK code generation (where accuracy matters) and Claude Haiku on Amazon Bedrock for explanations (where speed and cost efficiency matter more).

The Lambda function evaluates each finding’s origin and routes it to one of three remediation paths.

Path 1: IaC-managed roles (pull request) – For AnyCompany’s OrderProcessorRole, the automation creates a PR in the originating repository. The PR includes:

• The Amazon Bedrock-generated AWS CDK code implementing the IAM Access Analyzer-recommended policy
• A policy diff showing exactly which permissions are being removed
• The plain-English explanation of what the changes accomplish
• Labels (security, iam-remediation, automated) for filtering and tracking

The team that owns the role reviews the PR through their normal code review process. Once merged, the fix deploys consistently across environments through the existing CI/CD pipeline.

Path 2: Manually-created roles (issue) – For AnyCompany’s IncidentResponseRole, the automation creates an issue that includes the Access Analyzer-recommended policy with unused permissions removed, a diff highlighting the changes, an Amazon Bedrock-generated explanation, and step-by-step guidance on importing the role into IaC. This gives the team an immediate remediation path (apply the recommended policy) while encouraging long-term governance through IaC adoption.

Path 3: Unused roles (soft-disable issue) – For roles that haven’t been assumed at all, the automation creates an issue recommending a three-stage decommission workflow: attach a deny-all policy to the role, monitor for 30 days to confirm no workload depends on it, then delete. This controlled approach reduces the risk of removing a role that’s used infrequently or seasonally – if something breaks during the monitoring period, removing the deny-all policy restores access immediately.

Dry-run mode

Before creating real PRs and issues, you can run the automation in dry-run mode by setting “dry_run": true in the CI/CD configuration or setting the CI_CD_PLATFORM environment variable to dryrun. In this mode, the Lambda function processes findings, classifies roles, and generates remediation data, but logs what it would create instead of making actual API calls to your repository platform. You can use the log to validate the automation’s behavior, review the classification accuracy, and tune exclusions before going live.

Operational metrics

The Lambda function publishes CloudWatch metrics after each run:

These metrics feed into dashboards and alarms. AnyCompany sets an alarm on errors > 5 to catch API throttling or configuration issues, and tracks prs_created + issues_created over time to measure remediation velocity.

The solution ships as two AWS CDK stacks and deploys in minutes. The accompanying GitHub repository contains the complete source code, AWS CDK stacks, configuration templates, and step-by-step deployment instructions.

At a high level, deployment involves:

1. Prerequisites: An AWS account with an ACCOUNT_UNUSED_ACCESS or ORGANIZATION_UNUSED_ACCESS analyzer enabled, Python 3.11 or later, AWS CDK v2, a CI/CD platform API token stored in AWS Secrets Manager, and Amazon Bedrock model access for the Anthropic Claude models you plan to use. The model IDs are configurable environment variables (BEDROCK_CODEGEN_MODEL and BEDROCK_EXPLANATION_MODEL); Amazon Bedrock retires older foundation models over time, so if the shipped defaults stop working, set these variables to current models you have enabled and redeploy. The repository README documents this.
2. Configuration: Two files in the config/ directory control behavior. exclusions.json defines which roles and permissions to skip (break-glass roles, service-linked roles, tagged exceptions), and ci_cd_config.json configures your repository platform integration (GitLab or GitHub), labels, and throttling limits.
3. Deploy: Run cdk deploy --all to create the Lambda function, EventBridge schedule, IAM roles, and CloudWatch alarms.
4. Validate in dry-run mode: Start with “dry_run": true to see how the automation classifies your roles without creating real PRs or issues. Review the CloudWatch logs to confirm attribution accuracy and tune exclusions.
5. Go live: Set “dry_run": false and redeploy. The Lambda function runs on schedule (daily by default) and begins creating PRs and issues.

The repository README covers each step in detail, including organization-wide deployment, cross-account configuration, and platform-specific setup for GitLab and GitHub.

Deploying the automation is only the starting point. Running it in production means making decisions about how roles are retired, how the volume of findings is managed at scale, which roles warrant human review before any change is proposed, and how you measure the automation’s impact over time. The following practices keep remediation sustainable as your IAM footprint grows, so the automation reduces operational burden rather than adding to it.

Unused roles follow a three-stage decommission workflow. When the automation identifies a role that hasn’t been assumed within the analysis period, it creates an issue with the recommended decommission steps; the automation doesn’t modify the role directly. The team then follows the soft-disable approach:

1. Attach a deny-all inline policy to the role. This blocks all actions without deleting the role or its existing policies.
2. Monitor for 30 days. If a workload depends on the role (seasonal jobs, infrequent batch processes), the deny-all policy surfaces the dependency quickly. Removing the deny-all policy restores full access immediately; no need to recreate the role or reattach policies.
3. Delete the role after the monitoring period confirms no impact.

This approach is deliberately conservative. Deleting a role is irreversible; you lose the trust policy, attached policies, and any resource-based policies that reference it. The soft-disable step gives teams a safety net while still making progress on reducing their unused role inventory.

On AnyCompany’s first run, the automation found 47 unused permission findings and 4 unused roles. That’s manageable. But organizations with hundreds of accounts and thousands of roles might see significantly more findings on initial deployment.

This is especially true with an organization-level analyzer. A single-account deployment might surface dozens of findings; an organization-level analyzer across multiple accounts could surface hundreds or thousands on the first run. The throttling controls become critical at this scale.

Two throttling controls prevent the automation from overwhelming teams:

• max_findings_per_run (default 50): Caps the total UnusedPermission findings processed per Lambda function execution. Remaining findings are picked up on the next scheduled run.
• MAX_UNUSED_ROLE_ISSUES (default 10): Caps unused role issues per run. This is especially important during initial deployment when you might have a large backlog of roles that haven’t been assumed in months.

Start with conservative limits and increase them as your team builds confidence in the review process. A team that can review 10 PRs per week shouldn’t receive 50 on Monday morning.

Not every role should receive automated PRs. Roles with administrative permissions or access to sensitive data might warrant manual review before any remediation is created. The exclusion configuration supports this through the approval_required_for_tags field:

{
  "approval_required_for_tags": {
    "Sensitive": ["true"],
    "Admin": ["true"]
  }
}

Roles matching these tags generate issues for manual review instead of automated PRs, regardless of whether they’re IaC-managed. This gives security teams a checkpoint for high-risk roles while still automating remediation for standard application roles.

The metrics published after each Lambda function run (covered in the Technical details section) feed into CloudWatch dashboards and alarms. A few patterns worth setting up:

• Alert on errors > 5 per run to catch API throttling, expired CI/CD tokens, or Amazon Bedrock availability issues.
• Track prs_created + issues_created over time. A healthy trend shows this number decreasing as your environment converges toward least privilege.
• Monitor unused_roles_found as a leading indicator. A sudden increase might signal a team spinning up roles for a project and not cleaning up afterward.
• Compare iac_roles_found to manual_roles_found over time. As teams adopt IaC, the ratio should shift toward IaC-managed roles, which means more automated remediation and less manual work.

The solution uses Lambda (minimal cost at daily execution), CloudTrail (typically already enabled), IAM Access Analyzer (charges per IAM role or user analyzed per month for the unused access analyzer), and Amazon Bedrock (pay-per-token for AWS CDK code generation and explanations). For most organizations the ongoing cost is low, and Amazon Bedrock token usage is the largest variable, scaling with the number of findings processed per day and the complexity of each policy. Review the pricing pages for each service for current rates.

For organization-level deployments, the IAM Access Analyzer cost scales with the number of IAM roles analyzed across all member accounts. The ORGANIZATION_UNUSED_ACCESS analyzer charges per role per month across the organization, so an organization with 500 roles across 20 accounts will see higher analyzer costs than a single account with 50 roles. Review the IAM Access Analyzer pricing page for current rates.

To remove the solution, run cdk destroy --all from the infrastructure/ directory. This removes the Lambda function, EventBridge rule, CloudWatch alarms, and IAM roles created by the stacks.

If you stored a CI/CD platform API token in Secrets Manager as part of deployment, delete it with aws secretsmanager delete-secret --secret-id <your-secret-name> --recovery-window-in-days 7. The 7-day recovery window lets you restore the secret if the deletion was accidental. After 7 days, the secret is permanently deleted and can’t be recovered. To delete immediately without a recovery window, add --force-delete-without-recovery.

Lambda automatically creates a CloudWatch Logs log group at /aws/lambda/<function-name> that persists after cdk destroy --all and continues to incur log storage charges. To remove it, run aws logs delete-log-group --log-group-name /aws/lambda/<function-name>. WARNING: This permanently deletes all execution logs.

The IAM Access Analyzer isn’t created by the AWS CDK stacks. WARNING: Deleting the analyzer permanently removes all findings, analysis history, and unused permission data. Export any findings you need to retain before deletion. After exporting, run aws accessanalyzer delete-analyzer --analyzer-name <your-analyzer-name> to delete it. The ACCOUNT_UNUSED_ACCESS and ORGANIZATION_UNUSED_ACCESS analyzer types incur charges based on the number of IAM roles and users analyzed per month.

If you deployed in organization mode and created cross-account roles (default name: OrganizationAccountAccessRole) in member accounts solely for this solution, remove them from those accounts.

Any PRs or issues already created in your CI/CD platform remain after stack deletion; they’re artifacts in your repository, not AWS resources. See the repository README for detailed cleanup instructions.,

Automating IAM permission remediation turns least privilege from a periodic compliance exercise into an operational practice. By connecting IAM Access Analyzer findings and recommendations to your CI/CD pipeline, remediation shifts from manual security tasks to code review processes that your teams already follow.

The three-path strategy acknowledges how infrastructure evolves. IaC-managed roles receive pull requests with production-ready AWS CDK code and plain-English explanations. Manually created roles receive actionable issues with recommended policies and IaC migration guidance. Unused roles are put on a controlled decommission path that protects against accidental disruption. Over time, the manual role count decreases as teams adopt IaC, and remediation becomes a routine part of your deployment pipeline.

Start with a pilot. Choose 10–20 non-production roles, deploy in dry-run mode, and review the classification results. Tune your exclusions, confirm the CloudTrail attribution is accurate for your environment, and then enable live remediation. Expand to production roles after your team is comfortable with the review cadence.

When you’re ready to scale beyond a single account, switch to an organization-level analyzer and the same Lambda function will process findings across all member accounts with no architectural changes required, only a configuration toggle.

The complete source code, AWS CDK stacks, and configuration templates are available in the accompanying GitHub repository.

If you have feedback about this post, submit comments in the Comments section below.

In Part 1 of this guide, we examined two common incident scenarios: cross-account Amazon Simple Storage Service (Amazon S3) data deletion with ransomware implications, and cryptocurrency mining deployed through AWS CloudFormation using exposed AWS Management Console credentials. We also introduced key incident response terminology and investigative frameworks for analyzing AWS CloudTrail events.

In this second part, we explore a more complex, multi-stage attack: how a web application vulnerability can cascade into credential harvesting and unauthorized access to Amazon Bedrock services across multiple AWS Regions. We also cover additional investigation techniques and hardening steps to strengthen your security posture.

This scenario examines how a web application vulnerability can cascade into a multi-Region event targeting Amazon Bedrock services. The investigation demonstrates how threat actors chain together multiple techniques, using Amazon Elastic Compute Cloud (Amazon EC2) Instance Metadata Service version 1 (IMDSv1) through server-side request forgery (SSRF) and cross-Region pivoting to access Amazon Bedrock.

Your security team receives multiple alerts: failed AWS Identity and Access Management (IAM) operations in the us-east-1 Region, successful console sign-ins without multi-factor authentication (MFA), and unusual Amazon Bedrock API calls from us-east-2. Initially, these might seem like unrelated events across different services and Regions. However, as our Security Incident Response Team (SIRT) discovered, they represent a carefully orchestrated event chain that began with a web application vulnerability and culminated in unauthorized access to your organization’s AI infrastructure.

The architecture in figure 1 maps a multi-stage attack that exploits the trust relationship between Amazon Elastic Compute Cloud (Amazon EC2) instances and AWS services. A threat actor identified a server-side request forgery (SSRF) vulnerability in a web application running on an EC2 instance that had an attached webdev IAM role. Rather than attempting to escalate privileges directly, the threat actor used this foothold to reach the Instance Metadata Service version 1 (IMDSv1) endpoint and retrieve the temporary credentials issued to the webdev role. Because IMDSv1 returns credentials in response to a basic request with no session token, an SSRF flaw is enough to harvest them, which is why these credentials became the pivot point for everything that followed. The attack unfolded in five stages. Each stage is numbered in figure 1 so you can follow the progression from the initial web request through to the cross-Region Amazon Bedrock activity:

1. Initial access: The threat actor exploited the SSRF vulnerability in the web application to make server-side requests on the instance’s behalf.
2. Credential harvesting: Those requests reached the IMDSv1 endpoint and returned the temporary credentials for the webdev role.
3. Permission testing: Using the harvested credentials, the threat actor attempted IAM operations to probe the boundaries of what the role could do.
4. Service pivoting: When IAM actions were denied, the threat actor shifted focus to Amazon Bedrock, a service the role could reach.
5. Region hopping: The threat actor moved operations from us-east-1 to us-east-2, likely to evade Region-specific monitoring and access controls.

Figure 1: Scenario 3 architecture

In this section, we walk through the CloudTrail evidence that documents the attack from start to finish. Each of the four events that follow maps to one or more stages in the progression described previously, and together they trace how the threat actor moved from harvested credentials to active misuse of Amazon Bedrock. For each event, we present the relevant portion of the CloudTrail log record, highlight the fields that matter most for the investigation, and include a forensic legend that explains what each highlighted field reveals.

We cover the following events:

1. Permission boundary testing (15:53:49 UTC): A failed CreateUser call in us-east-1 that reveals the compromised role and the IMDSv1 credential source.
2. Console access establishment (15:59:29 UTC): A successful console sign-in without MFA, showing the pivot from programmatic to interactive access.
3. Bedrock service reconnaissance (17:20:00 UTC): A ListFoundationModels call in us-east-2 that marks the Region hop and the shift to AI services.
4. Active model exploitation (17:25:48 UTC): A Converse call that invokes the Amazon Nova Pro model, confirming unauthorized usage.

As you read each event, focus on how the fields connect one stage to the next. The same webdev role, the same source IP address, and the recurring ec2RoleDelivery value are the threads that tie these otherwise separate events into a single attack chain.

Event 1: Permission boundary testing (15:53:49 UTC): The first suspicious activity appeared as a failed CreateUser API call in us-east-1. The CloudTrail log records an AssumedRole session attempting to create an IAM user named adm1n but receiving an AccessDenied error. The webdev role is visible in the userIdentity field, readOnly is false (indicating a write operation attempt), and the user-agent shows AWS Command Line Interface (AWS CLI) on Windows, suggesting programmatic access from the harvested credentials.

{
  "eventVersion": "1.11",
  "userIdentity": {
    "type": "AssumedRole",
    "principalId": "AROAFINDANEXAMPLE:i-0123456789abcdef0",
    "arn": "arn:aws:sts::XXXXXXXXXXXX:assumed-role/webdev/i-0123456789abcdef0",
    "sessionContext": {
      "sessionIssuer": { "type": "Role", "userName": "webdev" },  ◄── ❶ Compromised EC2 role
                                                     ‾‾‾‾‾‾‾‾
      "attributes": { "mfaAuthenticated": "false" }
    },
    "ec2RoleDelivery": "1.0"  ◄── ❷ IMDSv1 confirmed (SSRF exploitation path)
                       ‾‾‾‾‾
  },
  "eventTime": "2025-09-22T15:53:49Z",
  "eventSource": "iam.amazonaws.com",
  "readOnly": false,
  "eventName": "CreateUser",  ◄── ❸ Intent: establish persistent backdoor
               ‾‾‾‾‾‾‾‾‾‾‾‾
  "userAgent": "aws-cli/2.17.48 ua/2.0 os/windows#10 ...",
  "errorCode": "AccessDenied",  ◄── ❺ Hard policy stop (least-privilege held)
               ‾‾‾‾‾‾‾‾‾‾‾‾‾‾
  "errorMessage": "User: arn:aws:sts::XXXXXXXXXXXX:assumed-role/webdev/...
    is not authorized to perform: iam:CreateUser
    on resource: arn:aws:iam::XXXXXXXXXXXX:user/adm1n..."  ◄── ❹ Lookalike name (1 not i)
                                                ‾‾‾‾‾
}

───────────────────────────────────────────────────────────────────
  FORENSIC LEGEND
───────────────────────────────────────────────────────────────────
  ❶ userName: "webdev"         → Confirms the compromised EC2 role context
  ❷ ec2RoleDelivery: "1.0"    → Credentials obtained via IMDSv1 (SSRF vector)
  ❸ eventName: "CreateUser"   → Attacker attempting IAM persistence
  ❹ target user: "adm1n"      → Typosquatting admin (number 1 instead of letter i)
  ❺ errorCode: "AccessDenied" → Attacker probing permission boundaries; blocked
───────────────────────────────────────────────────────────────────

Event 2: Console access establishment (15:59:29 UTC): Six minutes later, the threat actor successfully signed in to the AWS Management Console using the same credentials. The ConsoleLogin event records that MFA wasn’t used (MFAUsed: No), and the source IP (75.3.231.105) provides attribution data. The user agent indicates Chrome browser on Windows 10.

{
  "eventVersion": "1.11",
  "userIdentity": {
    "type": "AssumedRole",
    "principalId": "AROAFINDANEXAMPLE:i-0123456789abcdef0",
    "arn": "arn:aws:sts::XXXXXXXXXXXX:assumed-role/webdev/i-0123456789abcdef0",
    "sessionContext": { "attributes": { "mfaAuthenticated": "false" } }
  },
  "eventTime": "2025-09-22T15:59:29Z",
  "eventSource": "signin.amazonaws.com",
  "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 	 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36 Edg/140.0.0.0",

  "eventName": "ConsoleLogin",  ◄── ❶ Pivoted to interactive console access
               ‾‾‾‾‾‾‾‾‾‾‾‾‾‾
  "awsRegion": "us-east-1",
  "sourceIPAddress": "75.3.231.105",
  "responseElements": { "ConsoleLogin": "Success" },◄── ❷ Hijacked login succeeded
                                        ‾‾‾‾‾‾‾‾‾
  "additionalEventData": { "MobileVersion": "No", "MFAUsed": "No" },◄── ❸ No MFA challenge
                                                             ‾‾‾‾
  "eventType": "AwsConsoleSignIn"  ◄── ❹ Console sign-in (not API call)
               ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
}
───────────────────────────────────────────────────────────────────
FORENSIC LEGEND
───────────────────────────────────────────────────────────────────
❶ eventName: "ConsoleLogin"→ Attacker pivoted from programmatic to visual console access
❷ ConsoleLogin: "Success"→ Hijacked login successfully authenticated
❸ MFAUsed: "No" → Critical gap: no MFA enforced, enabling the pivot
❹ eventType: "AwsConsoleSignIn"   → Distinguishes this from basic API calls
───────────────────────────────────────────────────────────────────

Event 3: Amazon Bedrock service reconnaissance (17:20:00 UTC): Nearly two hours later, the threat actor pivoted to Amazon Bedrock, making a ListFoundationModels API call in us-east-2. This event exhibits several patterns: a Region change from us-east-1 to us-east-2 (potential defense evasion), a shift from IAM to AI services, readOnly: true (reconnaissance rather than modification), and sessionCredentialFromConsole: “true”, which ties the call to the console session established in Event 2 rather than a fresh IMDSv1 credential retrieval.

 {
  "eventVersion": "1.11",
  "userIdentity": {
    "type": "AssumedRole",
    "arn": "arn:aws:sts::XXXXXXXXXXXX:assumed-role/webdev/i-0123456789abcdef0"
  },
  "eventTime": "2025-09-22T17:20:00Z",
  "eventSource": "bedrock.amazonaws.com",  ◄── ❶ Pivoted to cloud AI services
               ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
  "eventName": "ListFoundationModels",  ◄── ❷ AI model reconnaissance
               ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
  "awsRegion": "us-east-2",  ◄── ❸ Region hop (evasion technique)
               ‾‾‾‾‾‾‾‾‾‾‾
  "sourceIPAddress": "75.3.231.105",
  "readOnly": true,
  "tlsDetails": {
    "clientProvidedHostHeader": "bedrock.us-east-2.amazonaws.com"  ◄── ❹ Intentional alternate region targeting
                               ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
  },
  "sessionCredentialFromConsole": "true"
}

───────────────────────────────────────────────────────────────────
  FORENSIC LEGEND
───────────────────────────────────────────────────────────────────
❶ eventSource: "bedrock.amazonaws.com"→ Attacker pivoted from IAM to managed AI services
❷ eventName: "ListFoundationModels"→ Reconnaissance: enumerating available AI models
❸ awsRegion: "us-east-2"→ Region hop from us-east-1 (defense evasion)
❹ clientProvidedHostHeader: "bedrock.us-east-2..."  → Confirms intentional targeting of alternate region endpoint
───────────────────────────────────────────────────────────────────

Event 4: Active model exploitation (17:25:48 UTC): Five minutes after the reconnaissance call, the threat actor moved from enumeration to active exploitation, invoking the Amazon Nova Pro model through the Converse API in us-east-2. The additionalEventData field quantifies the unauthorized usage at 944 input tokens and 126 output tokens, confirming that the threat actor successfully prompted the model and received a response.

 {
  "eventVersion": "1.11",
  "userIdentity": {
    "type": "AssumedRole",
    "arn": "arn:aws:sts::XXXXXXXXXXXX:assumed-role/webdev/i-0123456789abcdef0"
  },
  "eventTime": "2025-09-22T17:25:48Z",
  "eventSource": "bedrock.amazonaws.com",
  "eventName": "Converse",  ◄── ❶ Active model invocation (recon → exploitation)
               ‾‾‾‾‾‾‾‾‾‾
  "awsRegion": "us-east-2",
  "requestParameters": {
    "modelId": "amazon.nova-pro-v1:0",  ◄── ❷ Specific model being misused
               ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
    "inferenceConfig": { "maxTokens": 1024 }
  },
  "responseElements": null,
  "additionalEventData": { "inputTokens": 944, "outputTokens": 126 }  ◄── ❸ Unauthorized usage quantified
                           ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
}

───────────────────────────────────────────────────────────────────
  FORENSIC LEGEND
───────────────────────────────────────────────────────────────────
❶ eventName: "Converse"                → Attacker transitioned from reconnaissance to active exploitation
❷ modelId: "amazon.nova-pro-v1:0"      → Identifies the specific foundation model being misused
❸ inputTokens: 944, outputTokens: 126  → Quantifies unauthorized usage (financial cost + data exfiltration exposure)
───────────────────────────────────────────────────────────────────

As you review the event logs, a handful of fields do most of the investigative work in this scenario. Understanding what each one reveals, and why it matters, is what turns a collection of individual log records into a coherent attack narrative.

The userIdentity field is the starting point for attribution. In this scenario it carries the EC2 instance ID as the session name, which is what let us trace the harvested credentials back to a specific compromised instance rather than a human user. Whenever you see an assumed-role session, this field answers the first question of any investigation: whose credentials are these, and where did they come from?

The readOnly field reveals the intent behind an action. A value of true marks reconnaissance, such as the ListFoundationModels call the threat actor used to enumerate available models, while false marks an attempt to change or use something, such as the CreateUser call or the Converse invocation. Sorting events by this field quickly separates the threat actor’s information gathering from the actions that caused actual impact.

The awsRegion field is easy to overlook, but in this scenario it exposed the threat actor’s evasion strategy. The shift from us-east-1 to us-east-2 wasn’t incidental; threat actors move between Regions because monitoring, alerting, and access controls are often configured inconsistently across them. Watching this field helps you spot activity that has deliberately moved away from where your detection is strongest.

Finally, the userIdentity.invokedBy field identifies when an AWS service, rather than a user or a set of harvested credentials, made the request on your behalf. CloudTrail populates it only when the caller is an AWS service, such as through a service-linked role, a service role, or a forward access session. It doesn’t appear in the events for this scenario because the threat actor called Amazon Bedrock directly with the harvested webdev credentials. That absence is itself informative: it confirms the requests came from a principal acting on its own rather than from a legitimate service-driven workflow. As agent-based and service-integrated Amazon Bedrock workloads become more common, checking this field separates expected service activity from credentials driven directly by a threat actor.

With the full attack chain mapped, from SSRF through credential harvesting to Amazon Bedrock service misuse, the investigation turned to a harder question: what did each stage actually cost us, and what would stop it from happening again? A few priorities shaped that work.

The first was figuring out where the credentials came from and how far the exposure reached. It was clear the threat actor had valid credentials for the webdev role, but the more useful question was why a web application role could reach Amazon Bedrock at all. The customer confirmed there was no business reason for it, so we needed to understand whether that permission was a deliberate misconfiguration or an oversight, and then look for other EC2 instances carrying the same role attachment. One compromised instance is an incident; a fleet of instances with the same over-scoped role is a much bigger problem waiting to happen.

Next, we wanted to know what the threat actor did after they got into Amazon Bedrock. Reconnaissance and active use carry very different consequences, so we traced which foundation models were touched and whether any were actually invoked or only enumerated. That distinction matters for scoping the damage, and it signals whether data exfiltration is a concern. Unusual model usage, unexpected prompt volume, or output patterns that don’t match any legitimate workload are the signals that reconnaissance has turned into something worse.

The Region hop was its own line of inquiry. The move from us-east-1 to us-east-2 was almost certainly deliberate, and the investigation focused on understanding what the threat actor gained by it. In practice, that meant comparing the two Regions: were the monitoring and access controls in us-east-2 weaker than in us-east-1, and what else did the threat actor reach in the secondary Region once they were there? Inconsistent controls across Regions are one of the most common ways activity slips past detection.

Tying it all together was the timeline, which shows how quickly the threat actor moved through the chain:

1. 15:53:49: Failed IAM operation (us-east-1)
2. 15:59:29: Successful console login (us-east-1)
3. 17:20:00: Amazon Bedrock reconnaissance (us-east-2)
4. 17:25:48: Active model invocation (Converse call) (us-east-2)

Following the credentials across those events fills in the rest of the story. IMDSv1 handed the threat actor temporary credentials for the webdev role, and the same role appears in every event that followed, which confirms the credentials were reused rather than replaced. Nowhere in that sequence was MFA required, and that single gap is what let one harvested credential stay useful across two hours, two Regions, and two very different services.

The following checklist captures the actions needed to contain the incident, remediate the vulnerability, and assess the scope of unauthorized AI service usage. Each item names where to look and what a finding looks like, so the checklist stays usable under the time pressure of a live incident.

1. Contain and remediate the entry point:
   1. Identify the specific web application feature that made the outbound request (URL fetchers, webhook callbacks, PDF or image renderers, and link-preview generators are the usual culprits), then confirm it can reach http://169.254.169.254.
   2. Audit the rest of the application for the same pattern, because one unvalidated URL parameter usually means others exist.
   3. Enforce IMDSv2 on the affected instance and across the fleet with aws ec2 modify-instance-metadata-options --http-tokens required --http-put-response-hop-limit 1. Setting --http-tokens required means credentials are only returned when the caller presents a session token it obtained through a PUT request, which a basic SSRF cannot do. Setting the hop limit to 1 keeps the metadata response on the instance itself, so a request coming from a container or proxy an extra hop away never receives it.
2. Scope the Bedrock usage:
   1. List the foundation models the webdev role could reach by reviewing its IAM policy and any resource-based policies, so you know the full set of models that were exposed, not only the one that was invoked.
   2. Determine what was sent to and returned by the model. CloudTrail records the Converse call and the token counts, but only Amazon Bedrock model invocation logging captures the input prompts and model responses. If it was enabled, pull the log entries for the session; if it wasn’t, note that the prompt and response content can’t be recovered and enable it now.
   3. Flag any compliance exposure based on what those prompts and responses contained. Unauthorized processing of regulated data (such as personally identifiable information (PII), protected health information (PHI), or cardholder data) through the model might trigger notification obligations.
3. Check for wider compromise and persistence:
   1. Query CloudTrail across all Regions and services—not only Amazon Bedrock—for every event tied to the webdev role’s session, to confirm what else the same credentials touched.
   2. Correlate the CloudTrail timestamps with VPC Flow Logs and application logs for source IP 75.3.231.105 to build the network-level picture around each API call.
   3. Search for IAM write events from the session (CreateUser, CreateRole, CreateAccessKey, and AttachRolePolicy) that indicate an attempt to establish persistence beyond the temporary credentials. The failed adm1n CreateUser call is the known starting point; confirm nothing similar succeeded.
4. Watch for ongoing or hidden impact:
   1. Review Amazon Bedrock usage in CloudWatch and your billing data for invocation spikes or unexpected token consumption that fall outside normal workload patterns.
   2. Inspect the invocation logs for signs of sensitive data being processed or extracted through the model.
   3. Check the same logs for prompt injection attempts, where the input tries to override the model’s instructions or extract system prompts.

This scenario reveals how a single application vulnerability can cascade into broad unauthorized access when multiple security controls are missing. The following takeaways highlight the key defensive gaps and hardening priorities.

• Least-privilege IAM for workload roles: The webdev role’s access to Amazon Bedrock across multiple Regions had no business justification for a web application workload, which the customer confirmed during the investigation. Apply least-privilege principles to EC2 instance roles by scoping permissions to only the services and actions the application requires. Use AWS IAM Access Analyzer to identify unused permissions and tighten policies proactively. Overly permissive roles transform a single application vulnerability into broad lateral movement across unrelated services.
• IMDSv1 compared to IMDSv2: Organizations must immediately switch to IMDSv2 and disable IMDSv1 across their entire cloud infrastructure. The ec2RoleDelivery: “1.0" field in the logs explicitly confirms the use of IMDSv1, which permits credential retrieval without an authentication token. This architectural weakness makes SSRF-based credential theft trivial, because a web application flaw that can make an outbound request is enough to read the role’s temporary credentials with no further authentication. Transitioning to IMDSv2 mitigates this attack surface by enforcing local, session-based tokens, effectively breaking the threat actor’s exploitation chain. In this scenario, IMDSv2 alone would have stopped the attack at its first step.
• Region-based defense evasion signals a deliberate operator: The shift from us-east-1 to us-east-2 for Amazon Bedrock access wasn’t incidental. Threat actors move between Regions because monitoring, alerting, and access controls are often configured inconsistently across them, and activity in a secondary Region is more likely to go unnoticed. This kind of cross-Region movement is a marker of operational security awareness rather than opportunistic access, and it should raise the priority of an investigation. Treat consistent detection coverage across all Regions, including the ones you do not actively use, as a baseline requirement.
• Interface switching and permission probing reveal the threat actor’s method: This event chain reveals a threat actor comfortable moving between AWS interfaces and testing boundaries before committing. The failed CreateUser attempt was systematic probing to understand the scope of the harvested credentials, and when IAM actions were denied, the threat actor pivoted to a service the role could actually reach. The combination of programmatic access through the AWS CLI and interactive console access demonstrates the same adaptability. Recognizing this pattern of probe, adapt, and pivot helps responders anticipate the next move instead of reacting to each event in isolation.
• AI services need visibility beyond CloudTrail: Amazon Bedrock and other AI services are high-value targets, and CloudTrail alone doesn’t capture the whole story. CloudTrail records who called Amazon Bedrock and whether the call succeeded, but not what was asked or answered. Enable Amazon Bedrock model invocation logging to capture full prompts and responses for compliance auditing. For agent-based workloads, Amazon Bedrock AgentCore Observability, built on AWS Distro for OpenTelemetry (ADOT), provides session-level traces showing tool execution order and latency. Consider also enabling Amazon GuardDuty AI Protection, which analyzes Amazon Bedrock-related CloudTrail activity to detect anomalous invocations, cost harvesting, and prompt injection attempts. Correlating these signals—CloudTrail, Model Invocation Logging, and agent telemetry—gives investigators the complete picture. For implementation guidance, see Monitoring and Auditing AI Workloads on AWS.

Beyond the specific attack patterns in this scenario, investigators should be aware of several evasion techniques that threat actors use to confuse defenders and blend into legitimate activity. The top three that we observe across incident response with customers are:

• Root user compared to IAM user named root: When you first create an AWS account, you begin with a single sign-in identity that has complete access to all AWS services and resources in the account. This identity is called the AWS account root user. In some previous investigations, threat actors have also created IAM users in an AWS account named root. The difference is visible in the type field of the userIdentity element of the CloudTrail log record, which indicates the type of user that logged the record.
• Role and user name imitation: Threat actors attempt defense evasion by creating names for IAM users and roles that imitate those reserved for use by AWS. For example, the service-linked role AWSServiceRoleForSupport is a unique IAM role linked directly to AWS Support. Threat actors have created roles with the name AWSServiceRoIeforSupport (note the use of an upper-case letter I instead of a lower-case letter l in Role) in an attempt to trick users into thinking actions taken by this role have been performed by AWS Support.
• Users named HIDDEN_DUE_TO_SECURITY_REASONS: The userName field contains the string HIDDEN_DUE_TO_SECURITY_REASONS when the recorded event is a console sign-in failure caused by incorrect user name input. CloudTrail doesn’t record the contents in this case because the text could contain sensitive information. However, threat actors have used this string as an actual username to trick investigators into thinking the name has been obfuscated. This technique is usually associated with a corresponding CreateUser or CreateRole CloudTrail event.

CloudTrail event fields help security teams identify identities with unintended access, track threat actor actions, and remediate affected resources. Understanding fields like userIdentity, eventName, and sourceIPAddress improves incident investigation and threat detection. Implementing best practices such as enabling comprehensive logging, using Amazon Athena for analysis, securing logs, and automating responses helps ensure that CloudTrail serves as a robust forensic and incident response tool.

If you suspect unauthorized activity in your AWS environment, AWS Security Incident Response is available to help. The service continuously monitors and triages findings from Amazon GuardDuty and third-party security tools integrated through AWS Security Hub, automatically filtering alerts to surface the most relevant events. In addition to proactive triage, customers can initiate security cases through the service. You can choose to handle these cases internally or receive support from the Security Incident Response Team (SIRT), a dedicated group of security experts available at all times to assist with investigation, containment, and recovery throughout the incident lifecycle.

The following resources provide further guidance on securing your AWS environment and strengthening your investigative capabilities.

• AWS Security Best Practices
• The Agentic AI Security Scoping Matrix: A framework for securing autonomous AI systems
• CloudTrail Log Event Reference
• AWS Security Incident Response Guide

If you have feedback about this post, submit comments in the Comments section below.

AWS CloudTrail logs contain the evidence you need when investigating suspicious activity in your AWS environment, but knowing which fields matter and how to interpret them can mean the difference between surface-level analysis and uncovering the full scope of an incident. This guide walks you through real-world scenarios, showing you how to analyze CloudTrail events to uncover cross-account unauthorized access, cryptocurrency mining operations, and AI service abuse. You’ll learn the investigative techniques our Security Incident Response Team (SIRT) team uses to handle threats, with practical methodologies you can apply to your own investigations.

Each scenario includes:

• Architecture diagrams showing the event progression
• Annotated CloudTrail logs highlighting significant fields
• Investigation frameworks with specific questions to ask
• Lessons learned and preventive measures

Whether you’re in security operations, cloud engineering, compliance, or leadership, this guide provides the investigative mindset needed to move beyond basic CloudTrail queries to comprehensive security analysis.

Throughout this guide, we reference terminology commonly used in incident response and threat intelligence. We’ve provided definitions for key terms to help ensure this guide is accessible to readers from diverse backgrounds, whether you’re in security operations, cloud engineering, compliance, or leadership.

• Reconnaissance: The initial phase where a threat actor gathers information about the target environment (for example, listing Amazon Simple Storage Service (Amazon S3) buckets or browsing available resources) to understand what’s available before taking action.
• Enumeration: Systematically cataloging specific resources, users, or configurations within an environment to identify potential targets or access paths.
• Lateral movement: When a threat actor moves from one resource to another within the same environment (for example, pivoting from an Amazon Elastic Compute Cloud (Amazon EC2) instance to an AI service) to expand their access.
• Privilege escalation: Attempting to gain higher-level permissions than initially obtained, such as trying to create admin users or modify AWS Identity and Access Management (IAM) policies.
• Defense evasion: Techniques used to avoid detection, such as operating in a different AWS Region where monitoring might be less robust, or naming unauthorized resources to look legitimate.
• Persistence: Establishing ongoing access to an environment (for example, creating new IAM users or access keys) so the threat actor can return even if the original entry point is closed.
• Credential harvesting: Stealing authentication credentials (passwords, access keys, temporary tokens) to impersonate legitimate users or roles.
• Server-side request forgery (SSRF): A web application technique where an unauthorized user tricks a server into making requests on their behalf, often used to access internal services such as the Amazon EC2 Instance Metadata Service (IMDS) endpoint. For more information, see Understanding SSRF.
• IMDSv1 (Instance Metadata Service v1): Amazon EC2 Instance Metadata Service version 1 (IMDSv1) provides temporary credentials to applications running on an instance. IMDSv1 itself isn’t inherently insecure; however, when an application with issues (for example, one susceptible to SSRF) is running on the instance, an unauthorized user can use that application to reach the metadata endpoint and retrieve credentials. IMDSv2 mitigates this risk by requiring session-based authentication tokens.
• Indicators of compromise (IOCs): Observable artifacts (IP addresses, user agents, session names, resource names) that suggest unauthorized activity has occurred.
• Exfiltration: The unauthorized transfer of data out of an environment, such as copying files before deleting them.
• Event chain: The sequence of steps a threat actor follows from initial access to achieving their objective, where each step enables the next.
• Pivot: Shifting from one technique, service, or Region to another during a security event, often after an initial approach is blocked or to avoid detection.

Cross-account access is sometimes necessary in AWS, but misconfiguration creates security risks. In this scenario, your security operations center has received an automated alert that multiple objects have been deleted from the customer-important-data S3 bucket. The initial response seems straightforward: check the CloudTrail logs, identify who deleted the objects, and determine if it was authorized. But as our SIRT team investigated further, what appeared to be a straightforward unauthorized deletion revealed itself as a cross-account incident with ransomware implications. CloudTrail analysis requires recognizing patterns, understanding context, and thinking like a threat actor.

Figure 1 shows the architecture layout for accessing a trusted account and deleting objects from an S3 bucket, which is achieved through the following steps:

1. Threat actor assumes the CrossAccountS3Access role from a trusted account.
2. Lists S3 buckets to identify targets (ListBuckets API call).
3. Lists objects within the target bucket to catalog contents.
4. Executes scripted deletions of three files within 13 seconds.
5. Each deletion returns an HTTP 204 (successful) status code.

Figure 1: Scenario 1 architecture

Our investigation began with examining the CloudTrail logs, where we discovered that the unauthorized activity started with what many analysts might dismiss as routine activity: a ListBuckets API call made through an assumed role at 14:31:22 UTC. The CloudTrail entry contains a session named dev-migration-script using the CrossAccountS3Access role.

While cross-account access is common in enterprise environments, session names typically reflect legitimate business units. Attackers frequently use masquerading techniques, naming their sessions after common developer tasks or automation scripts, to blend seamlessly into daily operational noise. However, cross-referencing this session name against the external source IP and historical deployment logs confirmed that no such migration project was authorized, signaling a clear evasion attempt by a threat actor and the first indication of unauthorized access. Three seconds later, our logs showed a GET request to list objects in the bucket, which is classic reconnaissance behavior. The threat actor was cataloging available targets, using the same assumed role and IP address. This pattern, which you can see in the arn and eventname in the following log, showed us that the threat actor gathered intelligence, assessed targets, and planned their approach.

{
  "eventVersion": "1.08",
  "userIdentity": {
    "type": "AssumedRole",
    "principalId": "AROAEXAMPLE123456789:threat-actor-session",
    "arn": "arn:aws:sts::111122223333:assumed-role/CrossAccountS3Access/threat-actor-session"
  },
  "eventTime": "2025-01-20T14:31:22Z",
  "eventSource": "s3.amazonaws.com",
  "eventName": "ListBuckets",
  "sourceIPAddress": "203.0.113.47",
  "recipientAccountId": "444455556666"
}

──────────────────────────────────────────────────────────────────────────────────
  FORENSIC LEGEND
──────────────────────────────────────────────────────────────────────────────────
❶ arn: ".../CrossAccountS3Access/..."             → Cross-account role assumed; access came from another account
❷ Session name: "threat-actor-session"            → Custom session name attached at role assumption
❸ eventName: "ListBuckets"                        → Enumeration of all S3 buckets in the account (recon)
❹ principalId: "AROAEXAMPLE123456789:threat-actor-session" → Role's unique ID + attacker-chosen session label
❺ sourceIPAddress: "203.0.113.47"                 → Origin of the API call (RFC 5737 documentation IP range)
❻ recipientAccountId: "444455556666"              → AWS account that received/owned the request (fictional placeholder)
──────────────────────────────────────────────────────────────────────────────────

After completing their reconnaissance at 14:31:25 UTC, the threat actor went silent for 14 minutes before the first deletion at 14:45:12 UTC. During this window, the threat actor likely reviewed the inventory of objects they’d just enumerated, selected their highest-value targets (financial data, PII, and database backups), and prepared an automated deletion script to execute quickly once ready. We can infer this preparation period based on several factors: no other CloudTrail events from this session appeared during the 14-minute window, the subsequent deletions were precisely timed at 6-7 second intervals suggesting scripted execution, and the targets chosen were the three most business-critical files rather than a bulk delete of everything in the bucket. This selective, scripted approach indicates the threat actor used the reconnaissance data they gathered in the listing phase to build a targeted attack plan before executing it. Within 13 seconds (14:45:12–14:45:25 UTC), the threat actor deleted three files from the customer-important-data bucket: a financial report (q4-2024.xlsx at 14:45:12), a customer personally identifiable information (PII) database (pii-database.csv at 14:45:18), and a production database backup (prod-database-backup.sql at 14:45:25). Each deletion returned an HTTP 204 status code. The Amazon S3 access logs confirm these successful DELETE operations, all originating from the same session.

[20/Jan/2025:14:31:25] S3Access/dev-migration-script REST.GET.BUCKET    -  "GET /?list-type=2 HTTP/1.1" 200 - "-" "aws-cli/Linux"
                                ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾                                                ‾‾‾‾‾‾‾‾‾‾‾‾‾
                                ❶ Masquerading session   ❷ Recon: listing buckets                                        ❻ Scripted tool

[20/Jan/2025:14:34:15] S3Access/dev-migration-script REST.COPY.OBJECT   financial-reports/q4-2024.xlsx "PUT /..." 200 - "-" "aws-cli/Linux"
                                                     ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾   ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
                                                     ❸ Data exfiltration  Financial data copied

[20/Jan/2025:14:34:20] S3Access/dev-migration-script REST.COPY.OBJECT   customer-data/pii-database.csv "PUT /..." 200 - "-" "aws-cli/Linux"
                                                     ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾   ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
                                                     ❸ Data exfiltration  PII database copied

[20/Jan/2025:14:45:12] S3Access/dev-migration-script REST.DELETE.OBJECT financial-reports/q4-2024.xlsx "DELETE /..." 204 - "-" "aws-cli/Linux"
            ‾‾‾‾‾‾‾‾‾                               ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾               ‾‾‾
            ❺ 13-sec window starts                   ❹ Destruction        Target file                                   Success

[20/Jan/2025:14:45:18] S3Access/dev-migration-script REST.DELETE.OBJECT customer-data/pii-database.csv "DELETE /..." 204 - "-" "aws-cli/Linux"
                                                     ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾               ‾‾‾
                                                     ❹ Destruction        PII database destroyed                        Success

[20/Jan/2025:14:45:25] S3Access/dev-migration-script REST.DELETE.OBJECT backup-configs/prod-database-backup.sql "DELETE /..." 204 - "-" "aws-cli/Linux"
            ‾‾‾‾‾‾‾‾‾                               ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾         ‾‾‾
            ❺ 13-sec window ends                     ❹ Destruction        Prod backup destroyed (anti-recovery)         Success

───────────────────────────────────────────────────────────────────────────────────────────
  FORENSIC LEGEND
───────────────────────────────────────────────────────────────────────────────────────────
  ❶ "dev-migration-script"    → Masquerading session name; no authorized migration existed
  ❷ REST.GET.BUCKET           → Reconnaissance: cataloging available targets
  ❸ REST.COPY.OBJECT          → Data exfiltration before destruction (steal-then-destroy)
  ❹ REST.DELETE.OBJECT        → Systematic destruction of high-value assets
  ❺ 14:45:12 → 14:45:25      → 13-second automated deletion window (scripted execution)
  ❻ "aws-cli/Linux"           → CLI-based automation, not manual console activity
───────────────────────────────────────────────────────────────────────────────────────────

The 13-second deletion window wasn’t arbitrary. The user-agent string showed AWS Command Line Interface (AWS CLI) usage on Linux, and the precise timing suggested scripted execution rather than manual operations. This indicated preplanned targeting and automated execution to minimize the detection window.

The consistent source IP across events let us search for other suspicious activities from the same source, correlate with threat intelligence feeds, and identify potential lateral movement attempts.

The broad Amazon S3 permissions of the CrossAccountS3Access role raised questions about least privilege implementation, regular access reviews, and the business justification for such extensive cross-account permissions.

With confirmation that misconfigured cross-account access had been taken advantage of to delete data, the next step was to prioritize the investigation. In incident response, priority is driven by three factors: whether the threat actor still has active access (containment urgency), whether sensitive data was exposed or exfiltrated (regulatory and business impact), and whether the attack can spread to other resources or accounts (blast radius). We applied these factors to guide the following questions:

• How did the threat actor gain access to the CrossAccountS3Access role? We examined the role’s trust policy and recent modifications, authentication events in both the trusting and trusted accounts, and other sessions using the same role around the same timeframe.
• Were the files copied before deletion? We searched for GetObject operations on the same objects before the deletions, unusual network traffic patterns during the reconnaissance phase, and CopyObject activities that might indicate data theft.
• Did the objects have specific significance? Understanding why these specific objects mattered helped us prioritize recovery efforts based on business impact, assess regulatory notification requirements for the PII exposure, and determine the full scope of business disruption from backup loss.

After identifying the scope of the cross-account deletion, the following steps help ensure a thorough response and prevent recurrence.

• Determine if the business purpose served by this cross-account access is legitimate
• Identify the corresponding authentication events that show how the role was assumed
• Identify other AWS resources that this role might access beyond Amazon S3
• Check for failed attempts or reconnaissance activities that preceded the successful event
• Determine when this cross-account trust relationship was created
• Determine when the last access review of this role was conducted
• Locate any backup copies of the deleted data
• Determine detection rules that can be used to catch similar activity in the future

This scenario illustrates several principles that apply broadly to cross-account incident investigations. Unusual identifiers in session names often reveal threat actor intent or poor operational security. The progression from ListBuckets to targeted deletions shows how threat actors operate with a plan. Cross-account access needs extra scrutiny because trusted relationships become vectors for unauthorized access when credentials are exposed. Understanding why specific files matter helps prioritize response efforts and assess true impact. Precise timing and consistent technical signatures often indicate scripted events that need different response strategies than manual intrusions.

In this scenario, your finance team notices an unexpected spike in AWS costs, particularly around Amazon EC2 compute charges in the us-east-1 AWS Region. During the investigation, we examine how threat actors use legitimate console access to deploy cryptocurrency mining operations through AWS CloudFormation and how investigators can uncover the scope of resource hijacking events. We discover a CloudFormation stack named CRYPTO which you have no record or knowledge of being created. The stack contains EC2 instances running in your production Amazon Virtual Private Cloud (Amazon VPC) consuming significant compute resources, which signals an immediate security investigation.

Figure 2 shows how the threat actor moved from credential acquisition to active mining, following these steps:

1. Threat actor obtains console credentials (username and password without multi-factor authentication (MFA)).
2. Accesses AWS Management Console.
3. Creates CloudFormation stack CRYPTO in us-east-1.
4. Stack deploys EC2 instances configured for cryptocurrency mining in a public subnet.
5. Mining instances begin consuming compute resources.

Figure 2: Scenario 3 architecture

The following is the redacted CloudTrail event record for the unauthorized CreateStack action. See if you can use it to find the following information:

• The name of the CloudFormation stack that was created
• The CloudFormation stack Amazon Resource Name (ARN)
• If the credentials were secured by MFA
• If the threat actor used the AWS Management Console to perform the actions, or if they were performed programmatically using the AWS CLI or a script

{
  "eventVersion": "1.11",
  "userIdentity": {
    "type": "AssumedRole",
    "principalId": "AROAFINDANEXAMPLE:Participant",
    "arn": "arn:aws:sts::XXXXXXXXXXXX:assumed-role/WSParticipantRole/Participant",
    "sessionContext": {
      "sessionIssuer": { "type": "Role", "userName": "WSParticipantRole" },
      "attributes": { "mfaAuthenticated": "false" }  ◄── ❹ No MFA on session
                                            ‾‾‾‾‾‾‾
    }
  },
  "eventTime": "2025-09-23T18:07:12Z",
  "eventSource": "cloudformation.amazonaws.com",
  "eventName": "CreateStack",
               ‾‾‾‾‾‾‾‾‾‾‾‾‾
  "awsRegion": "us-east-1",
  "userAgent": "aws-cli/2.30.0 ... exec-env/CloudShell",  ◄── ❻ Browser-based CloudShell execution
                                    ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
  "requestParameters": {
    "stackName": "CRYPTO",  ◄── ❶ Cryptocurrency-related activity
                 ‾‾‾‾‾‾‾‾
    "parameters": [
      { "parameterKey": "VpcId" },      ◄── ❷ Prior recon: attacker knew target network
      { "parameterKey": "SubnetIds" }   ◄── ❷
                       ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
    ]
  },
  "responseElements": {
    "stackId": "arn:aws:cloudformation:us-east-1:...:stack/CRYPTO/2102e190..."  ◄── ❸ Stack created successfully
               ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
  },
  "sessionCredentialFromConsole": "true"  ◄── ❺ Console-based access
                                 ‾‾‾‾‾‾
}

───────────────────────────────────────────────────────────────────────────────────────────
  FORENSIC LEGEND
───────────────────────────────────────────────────────────────────────────────────────────
❶ stackName: "CRYPTO"                → Indicator of cryptocurrency mining deployment
❷ VpcId + SubnetIds parameters       → Attacker targeted specific network; prior recon confirmed
❸ stackId: "...stack/CRYPTO/..."     → Unique resource ID; stack was successfully created
❹ mfaAuthenticated: "false"          → Session lacked multi-factor authentication
❺ sessionCredentialFromConsole: true  → Access via AWS Console web portal (not external API)
❻ exec-env/CloudShell                → CLI commands executed via browser-based CloudShell
───────────────────────────────────────────────────────────────────────────────────────────

The CloudTrail event record includes fields that answer the questions for this scenario.

Stack details summary:

The CloudTrail event confirms the following details about the deployed stack:

• Stack name: CRYPTO is an indicator of cryptocurrency-related activity
• Stack ARN: arn:aws:cloudformation:us-east-1:stack/CRYPTO/2102e190-98a8-11f0-bcea-1209335b107
• Region: us-east-1 (a common choice for threat actors because of immediate service availability)

Authentication and session context analysis:

Examining the session metadata reveals how the threat actor authenticated and accessed the environment:

• MFA status: “mfaAuthenticated": “false” indicates that the session was entirely unauthenticated by MFA.
• Access method: “sessionCredentialFromConsole": “true” means that access was funneled through the console.
• User context: AssumedRole using WSParticipantRole. Session creation occurred at 2025-09-23T18:06:22Z (approximately 50 seconds before stack creation).

Advanced forensic insight (the CloudShell pivot)

The sessionCredentialFromConsole: true field is important to note because this access originated from the AWS console rather than external programmatic API keys. Interestingly, while the session originated from the console, the userAgent field reveals the execution environment was exec-env/CloudShell. This shows that the threat actor didn’t manually click through the CloudFormation user interface, instead launching AWS CloudShell on sign-in to execute a prepackaged deployment script. This allowed the threat actor to achieve automated speed while evading traditional static API key monitoring. The mfaAuthenticated: false field represents a security control gap. Particularly in environments handling sensitive data or production workloads, MFA must be enforced for console access.

With the unauthorized stack confirmed, the investigation focused on understanding the full timeline and blast radius. We approached this in three phases, each building on the findings of the previous one.

Reconstruct the console session timeline

The session began at 18:06:22Z and the stack was created at 18:07:12Z, only 50 seconds later. That speed tells us the threat actor came prepared with a script rather than exploring the environment manually. But we needed to know what happened before and after. By filtering CloudTrail for the same session token across the full session duration, we could identify whether the threat actor performed any reconnaissance before deploying the stack, whether they accessed other services or regions during the same session, and whether they attempted to establish persistence (such as creating IAM users or access keys) before or after the mining deployment. Any actions taken outside the CloudFormation deployment could indicate secondary objectives beyond cryptomining.

Examine what the stack actually deployed

The stack name alone doesn’t tell us the full impact. We needed to inspect the CloudFormation template to understand what resources were created and how they were configured. This meant identifying the EC2 instance types (larger instances mean higher costs and potentially more mining output), reviewing the security group rules to determine what network access these instances had to internal resources, checking whether the template included custom AMIs or user data scripts that pulled mining software on boot, and determining if the stack created its own IAM roles with permissions that could be used for further lateral movement. The template itself is evidence. If it was hosted in Amazon S3, the upload event tells us when the threat actor first staged their tools.

Calculate business impact and determine blast radius

Finally, we needed to quantify the damage and determine whether this was isolated or part of a broader compromise. We calculated the total compute cost by multiplying instance hours by instance type pricing, checked whether the mining instances had network paths to production databases or internal services, examined outbound traffic logs for connections to known mining pool IP addresses, and searched for similar stacks or naming patterns across other regions and accounts. The presence of outbound connections to anything other than mining pools would suggest the instances served a dual purpose, potentially exfiltrating data while generating cryptocurrency.

Response checklist

The following checklist captures the key actions needed to contain the incident, assess its impact, and close security gaps.

• Determine why MFA wasn’t required for this sensitive operation
• Investigate how the threat actor obtained valid console credentials
• Check for failed sign-in attempts preceding this successful access
• Check for other activities that occurred during this console session
• Look for resources that were created by the CloudFormation stack
• Determine how long those resources have been running and consuming costs
• Look for other similarly named or suspicious stacks in the environment
• Check what network access these instances have to internal resources
• Determine what outbound connections these instances are making
• Look for cryptocurrency mining pool connections
• Check if IAM users or roles were created
• Check if additional access keys were generated
• Determine if the threat actor modified existing permissions or policies

This scenario highlights how credential hygiene and monitoring controls intersect with resource hijacking threats.

• MFA enforcement prevents console-based credential abuse for IAM users. The absence of MFA enabled the full sequence. Console access to production environments should require multi-factor authentication as a security best practice.
• Resource naming can be an indicator. The obvious CRYPTO naming suggests either threat actor confidence or poor operational security, both concerning for different reasons.
• Cost monitoring is security monitoring. Unusual billing spikes can be early indicators of resource hijacking events.
• Console-based activity has different patterns than programmatic activity and requires specialized investigation approaches. The sessionCredentialFromConsole field is your starting point for distinguishing between the two.

In this first part, we walked through two real-world scenarios that demonstrate how CloudTrail analysis can reveal the full scope of a security incident. In Scenario 1, we showed how a seemingly routine cross-account role assumption led to targeted data deletion with ransomware implications, and how session names, timing patterns, and source IP correlation help investigators piece together the event chain. In Scenario 2, we examined how stolen console credentials enabled a cryptocurrency mining deployment through CloudShell, highlighting the critical role of MFA enforcement and cost monitoring as security controls. Both scenarios reinforce a core principle: effective CloudTrail investigation goes beyond identifying what happened. It requires understanding how and why, so you can contain the immediate threat and close the gaps that enabled it.

In Part 2 of this guide, we examine how a web application vulnerability can cascade into a multi-Region event targeting AI services, chaining together SSRF, IMDSv1 credential harvesting, and cross-Region pivoting to access Amazon Bedrock. We also cover critical investigation techniques including root user compared to IAM user named root, role name imitation tactics, and the HIDDEN_DUE_TO_SECURITY_REASONS username trick, along with critical hardening steps and additional resources you can use to strengthen your cloud forensic capabilities.

If you have feedback about this post, submit comments in the Comments section below.

A single alert from one security service tells you something happened. Read that signal alongside activity from other services and your own business context, and you will know whether what happened is part of a multi-stage attack.

Consider a short sequence. An identity calls GetCallerIdentity from a source address it hasn’t previously used. Within minutes, that same identity runs a burst of List and Describe calls across several services, and some of them fail with AccessDenied. Soon after, a large volume of data leaves your environment toward a domain that was registered last week. Amazon GuardDuty might already flag pieces of this, such as the reconnaissance from an unfamiliar source, through finding types like Recon:IAMUser/* or Discovery:S3/*. What you gain from correlating the pieces yourself is a single view of the sequence, tied to your own business context, so you can act on the whole rather than triaging findings one at a time.

This post is for security engineers and security operations teams who run Amazon Web Services (AWS) detection services and want to catch patterns specific to their environment. You will see how AWS detection and your business context fit together, and how to build correlations that use that context. The examples run in Amazon CloudWatch Logs Insights so you can try them today, and the closing section describes how to grow them into an automated pipeline. The walkthrough later in this post lists the prerequisites for these queries.

Begin with the AWS detection services. They cover the threats common across customers, and everything in this post is built on them.

• Amazon GuardDuty analyzes activity across AWS CloudTrail, Amazon Virtual Private Cloud (Amazon VPC) Flow Logs, DNS query logs, Amazon Simple Storage Service (Amazon S3) data events, and more and it produces high confidence findings out of the box.
• Amazon Detective (a security investigation service that visualizes relationships between resources and findings) lets you investigate findings with graph-based context across services and accounts.
• AWS Security Hub aggregates security findings into a single prioritized dashboard.
• Amazon Security Lake centralizes security data into a purpose-built data lake, stored in the Open Cybersecurity Schema Framework (OCSF) format for long-term analysis.
• Amazon Inspector (a vulnerability management service that scans workloads for software vulnerabilities) correlates vulnerability data with workload context.

Turn these on and tune them before you build anything custom. Tuning means adjusting sensitivity to reduce false positives for your environment, choosing which data sources each service monitors, and suppressing findings for known-good patterns.

Before you build anything by hand, see what GuardDuty already does for you. Amazon GuardDuty Extended Threat Detection correlates signals across multiple data sources including AWS CloudTrail, Amazon S3 data events, runtime monitoring, Amazon Elastic Kubernetes Service (Amazon EKS) audit logs, and more, then raises a single critical severity attack sequence finding when it spots a multi-stage pattern. It recognizes sequences such as credential compromise followed by data exfiltration, maps them to MITRE ATT&CK tactics, and attaches a timeline and remediation guidance. If you have GuardDuty enabled today, then GuardDuty Extended Threat Detection is already enabled by default and needs no queries from you. For details on how GuardDuty charges apply, see Amazon GuardDuty pricing.

The credential compromise sequence in the opening example is the kind of universal pattern GuardDuty Extended Threat Detection is built to catch, so rely on it for those. Attack sequence findings show up in the GuardDuty console next to your other findings, and they route to Security Hub and your response workflows the same way.

GuardDuty handles the threats that look the same in every account. What it doesn’t have is the context that makes a given action suspicious in your account. That’s what you provide.

Business context is what only you know about your environment: which buckets hold sensitive data, which principals have a reason to touch which resources, which role chains your policy permits, and when your production change windows open. GuardDuty Extended Threat Detection learns from patterns common across customers, but it can’t answer these environment-specific questions. Express them as correlations and you add a detection layer tuned to your environment. Each of the following four patterns turns one of these facts into a query.

Run these queries in the AWS Management Console for CloudWatch by choosing Logs, then Logs Insights, using the CloudWatch Logs Insights query language. Most read CloudTrail events from a CloudWatch Logs log group that your trail delivers to. If your trail writes only to Amazon S3, add CloudWatch Logs delivery on the trail, or run equivalent queries in Amazon Athena (a serverless query service for analyzing data in Amazon S3 using SQL).

Note: The queries and code in this post use placeholder values. Replace them with your own before running: your-sensitive-bucket (your S3 bucket name), your-key-id (your AWS KMS key ID), region (your AWS Region, such as us-east-1), account-id (your 12-digit AWS account ID), and aws-cloudtrail-logs-my-trail (your CloudTrail log group name).

A note on multi-account environments. In AWS Organizations, an organization trail delivers every account’s events to one log group, so these queries work as-is but return cross-account results. Filter by recipientAccountId for account-scoped views. Without an organization trail, run queries per account or use Amazon Security Lake as a central query surface.

Multi-stage attacks move through five phases, and each phase leaves a signal in a different service. These signals surface across three log sources: CloudTrail, which records API activity in your account; Amazon VPC Flow Logs, which capture network connection metadata; and Amazon Route 53 Resolver query logs, which record DNS queries from your VPCs.

• Initial access – Stolen credentials reach your environment. CloudTrail records GetCallerIdentity, GetSessionToken, or AssumeRole from an unfamiliar source.
• Discovery – The threat actor enumerates with List, Describe, and Get calls, often triggering AccessDenied responses.
• Privilege escalation – The threat actor chains roles or edits policies. CloudTrail records AssumeRole sequences, PutRolePolicy, or CreateAccessKey.
• Lateral movement – The threat actor moves across accounts or AWS Regions, assuming roles and creating resources in unfamiliar places.
• Exfiltration – Data leaves through GetObject calls at scale, large outbound transfers in VPC Flow Logs, and DNS queries in Route 53 Resolver query logs to recently registered domains.

Figure 1 shows the five attack phases mapped to the AWS log source that records each one.

Figure 1: Attack chain mapped to AWS services

GuardDuty Extended Threat Detection watches this chain for universal patterns. The four patterns that follow add the dimension you supply: your business context.

Your data classification and access norms drive this detection. One bucket holds customer records, another holds public web assets, and you know which principals have a reason to read the customer records, which are sensitive. Encode that knowledge and an ordinary looking read turns into something worth chasing.

Three signals converge here. CloudTrail shows GetObject at volume on a bucket you’ve classified as sensitive. The principal isn’t on your list of expected readers for that bucket. And VPC Flow Logs show a large outbound transfer from the same source in the same window, while DNS query logs show a recently registered destination domain, which together increase your confidence that there’s a potential threat.

CloudTrail management events don’t record GetObject. You must turn on CloudTrail data events for the buckets you care about to capture GetObject. Many teams miss GetObject because data events weren’t enabled on the relevant buckets.

This query shows bulk reads on a sensitive bucket, grouped by principal. Run it in CloudWatch Logs Insights with your CloudTrail log group selected.

fields @timestamp, userIdentity.arn, requestParameters.bucketName
| filter eventSource = "s3.amazonaws.com" and eventName = "GetObject"
| filter requestParameters.bucketName = "your-sensitive-bucket"
| stats count(*) as objectReads,
        count_distinct(requestParameters.key) as distinctObjects
        by userIdentity.arn, bin(10m)
| filter objectReads > 100
| sort objectReads desc

The threshold of 100 is a placeholder. Run the query over a week of normal activity, find the ninety-fifth percentile read count for that bucket, and set the threshold above it. Then check each principal the query returns against your expected reader list. A principal that isn’t on the list, reading at volume, is the result to investigate.

To corroborate, look for a matching outbound transfer. Switch the log group selector to your VPC Flow Logs log group and run this.

fields @timestamp, srcAddr, dstAddr, bytes
| filter action = "ACCEPT"
# exclude RFC 1918 private ranges so only external destinations remain
| filter dstAddr not like /^10\./
        and dstAddr not like /^192\.168\./
        and dstAddr not like /^172\.(1[6-9]|2[0-9]|3[0-1])\./
| stats sum(bytes) as totalBytes by srcAddr, dstAddr, bin(10m)
| filter totalBytes > 1000000000
| sort totalBytes desc

The Amazon S3 query returns a principal, and the Flow Logs query works on IP addresses, so you translate one into the other. The worked example later in this post covers that translation in full.

Picture an analytics role that reads a reporting bucket all day. One afternoon, it reads a thousand objects from your customer records bucket instead. GuardDuty stays quiet, because an authenticated role making valid GetObject calls isn’t suspicious anywhere else. Your query flags it, because that role isn’t on the expected reader list for that bucket. The classification you applied is what turns silence into a signal.

Figure 2 shows a bulk read from a sensitive bucket in CloudTrail, a large outbound transfer in VPC Flow Logs, and a young domain resolution in Route 53 Resolver logs.

Figure 2: Three signals converging within a single time window to indicate exfiltration

Picture a deployment that assumes one role to build, then a second to release. For one principal, that two-hop AssumeRole chain is routine; for a different principal it’s a policy violation. This pattern relies on your trust topology—the chains your organization permits—so put that knowledge in the query.

This pattern needs three conditions:

• CloudTrail shows several AssumeRole calls from the same source inside a short window
• The chain ends in a sensitive action such as CreateAccessKey, PutRolePolicy, or AttachUserPolicy
• The starting identity isn’t one your policy expects to run that chain

In CloudWatch Logs Insights, select your CloudTrail log group and run this query, which surfaces chains of two or more hops.

fields @timestamp, userIdentity.arn, requestParameters.roleArn, sourceIPAddress
| filter eventName = "AssumeRole"
| stats count(*) as assumeCount,
        count_distinct(requestParameters.roleArn) as rolesAssumed
        by sourceIPAddress, bin(5m)
| filter assumeCount >= 2 and rolesAssumed >= 2
| sort assumeCount desc

Two hops is the minimum for a chain; raise the count if your environment chains roles often. Your deployment pipeline probably assumes several roles an hour, as do AWS service principals such as AWS Security Hub. Exclude the identities you expect to see assuming multiple roles, including your pipeline role and known AWS service principals. What’s left is the set to investigate, such as a person assuming several roles at an odd hour and ending in a new access key. Treat that distinction as data: list the identities and actions you consider normal, and review the chains that fall outside the list.

Resource ownership is the signal here. A given AWS Key Management Service (AWS KMS) key creates and controls the encryption keys for a workload, and a single key should serve a single workload, such as a payments service. A Decrypt call against it is a valid, authorized API action, so nothing about the call itself looks wrong. The ownership rule you set is what makes another principal’s use of the key worth a second look.

This pattern applies only to customer-managed keys scoped to one workload. It doesn’t apply to AWS-managed keys (alias/aws/*) or to customer-managed keys intentionally shared across services. Confirm single-workload intent from the key policy’s Principal block before deploying this rule.

Two conditions indicate misuse:

• CloudTrail shows Decrypt or GenerateDataKey calls on a key that’s tied to one workload
• The calling principal isn’t the role that owns that workload

Against your CloudTrail log group, run this query to list the principals that called a specific key.

fields @timestamp, userIdentity.arn, eventName
| filter eventSource = "kms.amazonaws.com"
| filter eventName in ["Decrypt", "GenerateDataKey", "Encrypt"]
| filter resources.0.ARN = "arn:aws:kms:region:account-id:key/your-key-id"
| stats count(*) as keyUses by userIdentity.arn, eventName
| sort keyUses desc

Compare what comes back against the one workload role you expect. A principal you don’t recognize on that key is the signal. Because key misuse is an early move in data theft, this correlation catches activity that only your ownership knowledge can flag.

Consider a key that wraps your payments database. The payments service role calls it in normal operation, and nothing else should. If a developer role or a freshly created role runs Decrypt against it, the call succeeds and reads as ordinary in isolation. The reason it matters is the ownership rule you hold in your head and now state in this query.

Start with the query, then read what it means.

fields @timestamp, userIdentity.arn, eventName, sourceIPAddress
| filter eventName in ["PutRolePolicy", "AttachRolePolicy",
        "CreateAccessKey", "AuthorizeSecurityGroupIngress", "PutBucketPolicy"]
| stats count(*) as sensitiveChanges by userIdentity.arn, eventName, sourceIPAddress
| sort sensitiveChanges desc

Run it against your CloudTrail log group, scoped to your off-hours window when you schedule it, so it returns only activity outside the change window. Your change process defines what normal looks like here: production security and identity changes flow through a pipeline during defined hours, run by a known actor. A console-driven policy change at 2:00 AM, made by a person rather than the pipeline, doesn’t fit those expectations. The signal is a sensitive change such as PutRolePolicy or AuthorizeSecurityGroupIngress, made outside the window, by a person rather than your pipeline role.

Exclude the actors you expect, such as your deployment pipeline role, your patch automation role, and AWS service principals like AWS CloudFormation and AWS Systems Manager. What remains is privileged change made outside your process, which is both what an attacker does to establish persistence and what your own change discipline says shouldn’t happen.

Your pipeline might open security group rules during a deployment every weekday afternoon. A person opening a security group rule at midnight on a weekend is the same API call carrying a very different meaning. The schedule and the actor, both facts you define, are what separate the two.

The following walkthrough uses pattern one as a complete example. The other three patterns follow the same design with their own queries.

These prerequisites feed the queries in this walkthrough. Confirm each one before you start:

• A CloudTrail trail logging management events to a CloudWatch Logs log group
• CloudTrail data events enabled for your sensitive S3 buckets
• GuardDuty enabled, with its protection plans and Extended Threat Detection
• VPC Flow Logs on for your production VPCs
• Amazon Route 53 Resolver query logging on

CloudTrail, GuardDuty, VPC Flow Logs, and Route 53 Resolver query logging provide the raw signals that your correlations connect. Without them, the queries in this post return empty results.

Choose one sensitive bucket to monitor, and write down the principals allowed to read it. Store the list where your automation can reach it, such as a configuration file in version control or an Amazon DynamoDB table (a managed NoSQL database).

{
  "customer-records-prod": [
    "arn:aws:iam::123456789012:role/AnalyticsPipeline",
    "arn:aws:iam::123456789012:role/ComplianceAudit"
  ],
  "financial-data-archive": [
    "arn:aws:iam::123456789012:role/FinanceReporting"
  ]
}

This example hardcodes the list for simplicity. In production, load it from a DynamoDB table or Parameter Store so you can update it without redeploying.

Run the pattern one query over one week of normal activity. Find the 95th percentile read count for the bucket and use a value greater than that as your alert threshold. This step keeps legitimate high-volume access from generating false positives later.

Set the THRESHOLD_READS environment variable to this value when you configure the function in Step 5.

In the CloudWatch console:

1. Choose Logs, then choose Logs Insights.
2. In the Select log group(s) dropdown, select your CloudTrail log group.
3. Set the time range to 3h (the last three hours).
4. In the query editor, paste the pattern one query.
5. Replace your-sensitive-bucket with your bucket name.
6. Choose Run query.
7. Review the principals in the results table.
8. Compare each principal against your expected reader list from step 1, and flag any that are not on it.

Each result includes a principal that step 4 translates into an IP address.

CloudTrail logs actions by AWS Identity and Access Management (IAM) principal, while VPC Flow Logs record traffic by IP address. To connect the two signals, translate the principal into its address.

For a role attached to an Amazon Elastic Compute Cloud (Amazon EC2) instance, the userIdentity.principalId field includes the instance ID after the colon, in the form AROAEXAMPLE:i-1234567890abcdef0. Copy the instance ID and look up its private IP address.

aws ec2 describe-instances \
  --instance-ids i-1234567890abcdef0 \
  --query "Reservations[0].Instances[0].PrivateIpAddress" \
  --output text

Other compute types differ. A VPC-connected AWS Lambda function sends traffic through elastic network interfaces in your subnets, so correlate on those interface addresses. An Amazon Elastic Container Service (Amazon ECS) task records its network interface in task metadata. For a plain assumed-role session with no instance behind it, the sourceIPAddress field in CloudTrail already holds the caller’s address, so you correlate on it directly.

Run the Flow Logs query from pattern one, filtering srcAddr to that address within 10 minutes of the Amazon S3 read timestamp. A match places the same source behind both the sensitive read and a large external transfer in one window. CloudTrail events reach CloudWatch Logs 5–15 minutes after the API call, so correlate on eventTime rather than query time. Query a wider lookback than your correlation window: for example, look back 30 to 60 minutes but correlate on a 10-minute eventTime window. Steps 3 and 4 are manual validation; step 5 automates them.

Figure 2 shows DNS resolution as a third corroborating signal. This walkthrough implements the CloudTrail and VPC Flow Logs correlation. To add DNS, apply the same run_query() pattern against your Route 53 Resolver query log group.

Move the query into a Lambda function (serverless compute that runs your code without a server to manage), send results to a notification channel, and schedule regular runs. Work through the following sub-procedures.

To create the notification channel

1. Open the Amazon Simple Notification Service (Amazon SNS) console. Amazon SNS is a managed messaging service that delivers notifications to subscribers.
2. In the navigation pane, choose Topics.
3. Choose Create topic.
4. For Type, select Standard.
5. For Name, enter security-correlation-alerts.
6. Choose Create topic.
7. Note the topic Amazon Resource Name (ARN) at the top of the topic details page. You will use it in the function.
8. Choose Create subscription.
9. For Protocol, select Email.
10. For Endpoint, enter your email address or incident management endpoint.
11. Choose Create subscription, then confirm the subscription from the email AWS sends.

To create the EventBridge Scheduler execution role

The schedule needs a role that lets it invoke your function, and its trust policy needs conditions that pin the role to the schedule you own. Without those conditions, another account with access to the scheduler service could theoretically call this role; a class of misuse known as the confused deputy problem.

1. Create a trust policy file named scheduler-trust-policy.json.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "scheduler.amazonaws.com" },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "aws:SourceAccount": "ACCOUNT-ID"
        },
        "ArnLike": {
          "aws:SourceArn": "arn:aws:scheduler:REGION:ACCOUNT-ID:schedule/*/s3-access-correlation-hourly"
        }
      }
    }
  ]
}

2. Create the role, then attach permission to invoke the function. Scope Resource to the specific function ARN so this role can’t invoke anything else.

aws iam create-role \
  --role-name EventBridgeSchedulerRole \
  --assume-role-policy-document file://scheduler-trust-policy.json

aws iam put-role-policy \
  --role-name EventBridgeSchedulerRole \
  --policy-name LambdaInvokePolicy \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": "lambda:InvokeFunction",
        "Resource": "arn:aws:lambda:REGION:ACCOUNT-ID:function:CorrelationFunction"
      }
    ]
  }'

When you create the function, Lambda automatically creates an execution role. You will attach the permissions this function needs to that role in a later step.

To deploy the correlation function

1. Open the Lambda console.
2. Choose Create function.
3. For Function name, enter CorrelationFunction.
4. For Runtime, select the latest Python runtime.
5. Choose Create function.
6. On the Code tab, replace the default code with the following function, then choose Deploy.

import os
import time
import logging
import boto3
from botocore.exceptions import ClientError

logger = logging.getLogger()
logger.setLevel(logging.INFO)

logs = boto3.client("logs")
sns = boto3.client("sns")
ec2 = boto3.client("ec2")

CLOUDTRAIL_LOG_GROUP = os.environ["CLOUDTRAIL_LOG_GROUP"]
FLOWLOGS_LOG_GROUP = os.environ["FLOWLOGS_LOG_GROUP"]
SNS_TOPIC = os.environ["SNS_TOPIC_ARN"]
BUCKET = os.environ["SENSITIVE_BUCKET"]
THRESHOLD = int(os.environ.get("THRESHOLD_READS", "100"))

# Expected readers per bucket
EXPECTED_READERS = {
    "customer-records-prod": [
        "arn:aws:iam::123456789012:role/AnalyticsPipeline",
        "arn:aws:iam::123456789012:role/ComplianceAudit",
    ],
}


def run_query(log_group, query, start, end):
    """Start a Logs Insights query and wait for it to finish."""
    started = logs.start_query(
        logGroupName=log_group,
        startTime=start,
        endTime=end,
        queryString=query,
    )
    query_id = started["queryId"]
    while True:
        outcome = logs.get_query_results(queryId=query_id)
        if outcome["status"] in ("Complete", "Failed", "Cancelled"):
            break
        time.sleep(1)
    if outcome["status"] != "Complete":
        raise RuntimeError(f"Query did not complete: {outcome['status']}")
    return [{f["field"]: f["value"] for f in row} for row in outcome["results"]]


def private_ip_for_principal(principal_id):
    """Resolve an EC2 instance role principalId to its private IP."""
    if ":" not in principal_id:
        return None
    instance_id = principal_id.split(":", 1)[1]
    if not instance_id.startswith("i-"):
        return None
    reservations = ec2.describe_instances(InstanceIds=[instance_id])
    for reservation in reservations["Reservations"]:
        for instance in reservation["Instances"]:
            return instance.get("PrivateIpAddress")
    return None


def egress_bytes(src_addr, start, end):
    """Sum external egress bytes for one source address."""
    query = f"""
    fields srcAddr, dstAddr, bytes
    | filter action = "ACCEPT" and srcAddr = "{src_addr}"
    | filter dstAddr not like /^10\\./
            and dstAddr not like /^192\\.168\\./
            and dstAddr not like /^172\\.(1[6-9]|2[0-9]|3[0-1])\\./
    | stats sum(bytes) as totalBytes
    """
    rows = run_query(FLOWLOGS_LOG_GROUP, query, start, end)
    if rows and rows[0].get("totalBytes"):
        return int(rows[0]["totalBytes"])
    return 0


def lambda_handler(event, context):
    try:
        # 1-hour lookback absorbs CloudTrail's 5-15 min delivery latency;
        # correlation happens on eventTime via 10-min bins in the query below.
        end = int(time.time())
        start = end - 3600  # 1 hour lookback
        allowed = EXPECTED_READERS.get(BUCKET, [])

        access_query = f"""
        fields userIdentity.arn, userIdentity.principalId
        | filter eventSource = "s3.amazonaws.com" and eventName = "GetObject"
        | filter requestParameters.bucketName = "{BUCKET}"
        | stats count(*) as objectReads
                by userIdentity.arn, userIdentity.principalId, bin(10m)
        | filter objectReads > {THRESHOLD}
        """

        for row in run_query(CLOUDTRAIL_LOG_GROUP, access_query, start, end):
            principal = row.get("userIdentity.arn")
            if not principal or principal in allowed:
                continue

            message = (
                f"Principal {principal} read {row.get('objectReads')} "
                f"objects from {BUCKET}."
            )

            ip = private_ip_for_principal(row.get("userIdentity.principalId", ""))
            if ip and egress_bytes(ip, start, end) > 1_000_000_000:
                message += (
                    f" The same source ({ip}) also sent a large volume of "
                    f"data to external destinations in the same window."
                )

            sns.publish(
                TopicArn=SNS_TOPIC,
                Subject="Unexpected S3 access detected",
                Message=message,
            )
    except ClientError as error:
        logger.error(f"AWS API error: {error}")
        raise
    except Exception as error:
        logger.error(f"Unexpected error: {error}")
        raise
    finally:
        logger.info("Correlation check completed")

7. On the Configuration tab, choose General configuration, then choose Edit. Set Timeout to 5 minutes (300 seconds). CloudWatch Logs Insights queries run asynchronously and can take 30 to 60 seconds against large log groups. Choose Save.
8. On the Configuration tab, choose Environment variables, then choose Edit, and add CLOUDTRAIL_LOG_GROUP, FLOWLOGS_LOG_GROUP, SNS_TOPIC_ARN, SENSITIVE_BUCKET, and THRESHOLD_READS.
9. On the Configuration tab, choose Permissions, open the execution role, and attach the following least-privilege policy.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["logs:StartQuery", "logs:GetQueryResults"],
      "Resource": [
        "arn:aws:logs:REGION:ACCOUNT-ID:log-group:aws-cloudtrail-logs-my-trail:*",
        "arn:aws:logs:REGION:ACCOUNT-ID:log-group:vpc-flow-logs:*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "ec2:DescribeInstances",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "sns:Publish",
      "Resource": "arn:aws:sns:REGION:ACCOUNT-ID:security-correlation-alerts"
    }
  ]
}

Replace REGION, ACCOUNT-ID, and the log-group names with your values. The ec2:DescribeInstances action doesn’t support resource-level permissions, so Resource: "*" is required for that statement; the other statements are scoped to specific ARNs.

To schedule automated runs

Amazon EventBridge (a serverless event bus that connects applications using events) runs targets on a schedule. Create one from the command line, using the role you made earlier.

aws scheduler create-schedule \
  --name s3-access-correlation-hourly \
  --schedule-expression "rate(1 hour)" \
  --target "Arn=arn:aws:lambda:REGION:ACCOUNT-ID:function:CorrelationFunction,RoleArn=arn:aws:iam::ACCOUNT-ID:role/EventBridgeSchedulerRole" \
  --flexible-time-window "Mode=OFF"

Enrichment cuts triage time by adding an independent signal, but it isn’t required for the correlation to work. This step adds costs. You pay your geolocation provider for API calls, and the additional Lambda execution time increases your Lambda charges. To add IP geolocation, sign up for a geolocation API, add this function to the code, and call it where the handler resolves an IP.

import urllib.request
import json

def geo_context(ip_address):
    """Enrich an IP address with geolocation data from your provider."""
    try:
        url = f"https://your-geolocation-api.example/json/{ip_address}"
        with urllib.request.urlopen(url, timeout=5) as response:
            data = json.load(response)
        return {
            "country": data.get("country_name"),
            "city": data.get("city"),
            "org": data.get("org"),
        }
    except Exception as error:
        logger.warning(f"Geolocation lookup failed for {ip_address}: {error}")
        return None

Inside the handler’s loop, after you resolve ip, append the location to the alert.

            if ip:
                geo = geo_context(ip)
                if geo:
                    message += (
                        f" Source location: {geo['city']}, "
                        f"{geo['country']} ({geo['org']})."
                    )

As your library grows, move the logic into automated pipelines with EventBridge, Lambda, and AWS Step Functions (a serverless orchestration service that coordinates multiple services into workflows), and surface correlations next to findings in Security Hub. For cross-service correlation at scale, CloudWatch unified data and telemetry capabilities can convert security and compliance data into the OCSF format and let you query sources such as CloudTrail, VPC Flow Logs, and DNS logs from one interface. Security Lake with Athena is a strong option for long-term analysis. Choose the endpoint that fits your retention and query needs.

Figure 3 shows a correlation pipeline built on AWS services including EventBridge, Lambda, Step Functions, and AWS Security Hub. The pipeline runs from data sources through scheduled queries and enrichment to automated response and centralized visibility.

Figure 3: A correlation pipeline built on AWS services

You now have four correlation patterns that layer your business context on top of GuardDuty Extended Threat Detection to catch attacks specific to your environment. A few principles carry across every correlation you build.

• Identity is your primary correlation key: Track the same principal across services.
• Time windows matter, but they depend on the attack: Events minutes apart are usually related for fast, automated sequences; the ten-minute bins here work for that pattern. Slow or manual reconnaissance can stretch across hours or days, so widen the window when the pattern is deliberate rather than automated.
• Context is what you add: Your data classification, access norms, resource ownership, and change windows are signals you bring to detection.
• Start with one rule: A single well-tuned correlation catches more significant activity than a wall of uncorrelated alerts.

GuardDuty Extended Threat Detection handles the multi-stage patterns common across customers. The correlations in this post add the layer that only your business context can supply. Start with one pattern this week, validate it against your own traffic, and add the next pattern after the first proves reliable.

Have you built correlation rules for patterns not covered here? Share your experience in the Comments section below.

• Amazon GuardDuty Extended Threat Detection – How GuardDuty automatically correlates multi-stage attack patterns
• CloudWatch Logs Insights query syntax – Reference for the query language used in this post
• Logging data events with AWS CloudTrail – Turn on the Amazon S3 data events pattern you depend on
• Sending CloudTrail events to CloudWatch Logs – Deliver your trail to a log group you can query
• Creating a trail for an organization – Deliver events from every account in AWS Organizations to a single log group
• VPC Flow Logs – Field reference for the network traffic signal
• Amazon Route 53 Resolver query logging – Capture the DNS signal used for corroboration
• MITRE ATT&CK – Industry Framework for attack tactics and techniques
• Open Cybersecurity Schema Framework (OCSF) – The schema Security Lake uses

When an administrator introduces a rule change in AWS Network Firewall and network connectivity is disrupted, pinpointing the cause requires inspecting multiple points in the traffic path. The firewall gives you stateless and stateful rule engines, domain rules, and routing to the firewall endpoint inside your Amazon Virtual Private Cloud (Amazon VPC). A network drop looks the same from the workload no matter where it started. Isolating the cause means correlating the alert and flow logs with the firewall configuration, route tables, and recent API calls in AWS CloudTrail that might have changed them. That manual correlation is exactly where AWS DevOps Agent helps, accelerating root cause analysis so you can restore connectivity in minutes instead of hours.

AWS DevOps Agent does that correlation for you. As your always-available operations teammate, it resolves and proactively prevents operational issues across AWS, multicloud, and on-premises environments. When an Amazon CloudWatch alarm triggers, it reaches the agent through a webhook. The agent then reads the firewall configuration and logs through AWS APIs, ties the drop to recent API activity, and returns a root cause with a mitigation plan you review before you apply it.

This post connects CloudWatch monitoring to DevOps Agent. It walks through three Network Firewall failures from end to end. The first is a domain deny list blocking a legitimate endpoint. The second is a stateless rule priority misconfiguration. The third is an asymmetric cross Availability Zone (AZ) routing drop. Each maps to a different layer, so each leads down a different investigation path. An AWS Cloud Development Kit (AWS CDK) app deploys the whole environment in your own account so you can reproduce each failure and follow along.

As part of this blog post, we provide a CDK stack that deploys both the AWS DevOps Agent Space and a sample workload used to walk through three separate troubleshooting scenarios. A single t3.micro instance in a protected subnet checks its connectivity to a test endpoint on a continuous loop and publishes results to CloudWatch. Traffic takes the internet egress path through Network Firewall, the NAT gateway, and the internet gateway, so the firewall can intercept or drop it. After completing the walkthrough, you can apply the same troubleshooting techniques with DevOps Agent against your own Network Firewall deployments.

The test endpoint runs in a separate VPC deployed by the same CDK app. It serves HTTPS on port 443 and TCP on port 9142, giving each scenario a different protocol layer to exercise: Scenario 1 targets a TLS connection on 443 (matched by Server Name Indication), Scenario 2 targets a TCP connection on 9142, and Scenario 3 exercises the whole egress path.

A live status page shows one card per scenario plus the network topology. The whole stack deploys from a single CDK app across two Availability Zones, each with a firewall endpoint and NAT gateway, which is what makes Scenario 3 possible.

As shown in the following figure, the egress data path runs from the workload through Network Firewall and the NAT and internet gateways to the test endpoint. The alarm pipeline runs from CloudWatch through Amazon Simple Notification Service (Amazon SNS) and the webhook AWS Lambda function to DevOps Agent.

Figure 1: The sample workload

To use this with your own workload, you need a CloudWatch alarm that detects the connectivity problem and the webhook pipeline (SNS topic and Lambda function) that delivers it to DevOps Agent. The agent reads your firewall configuration, logs, and CloudTrail through AWS APIs, so no additional instrumentation is needed on the firewall side.

To follow along with this post, you need:

• An AWS account with permissions to deploy Amazon VPC, Network Firewall, Amazon Elastic Compute Cloud (Amazon EC2), CloudWatch, Amazon SNS, Lambda, Amazon CloudFront, and AWS Identity and Access Management (IAM) resources
• Access to AWS DevOps Agent, with permission to configure its webhook
• Node.js 18 or later and npm installed
• AWS Command Line Interface (AWS CLI) 2.x configured with credentials
• AWS CDK 2.x is required. You can use it through the project’s npx dependency, or install it globally:
  • npm install -g aws-cdk

Clone the project and deploy it into us-east-1 with one command (set awsRegion to use another AWS Region).

git clone https://github.com/aws-samples/sample-accelerating-aws-network-firewall-troubleshooting-with-aws-devops-agent.git
cd sample-accelerating-aws-network-firewall-troubleshooting-with-aws-devops-agent
bash scripts/deploy.sh

The script checks prerequisites, installs dependencies, compiles and tests, and bootstraps the CDK if needed. It then deploys all the stacks from a clean baseline and prints the outputs, including the status-page URL and sign-in details.

1. Open the status-page link (an https://<random-id>.cloudfront.net address).
2. Sign in using the username and password provided from the CDK output and confirm all three cards show the green Healthy status.
3. Keep the page open while you run the scenarios.

To connect AWS DevOps Agent to the alarm pipeline

1. In the AWS DevOps Agent console, open the nf-devops-agent-space Agent Space created by the CDK deployment.
2. Configure the DevOps Agent webhook and download the CSV file with the webhook URL and signing secret.
3. On the status page, choose Configure webhook, paste the URL and signing secret, and save. The page writes them to the nf-devops-agent-webhook-credentials AWS Secrets Manager secret, so there is no AWS CLI or console step. Until you set it, the bridge Lambda function sees a placeholder and skips delivery.
4. Verify the path before you run a scenario. In the Lambda console, open nf-devops-agent-webhook and use the Test tab with this event.

   {
     "Records": [
       {
         "Sns": {
           "Message": "{\"AlarmName\":\"TEST-webhook-verification\",\"AlarmDescription\":\"[TEST] Webhook integration test - not a real alarm.\",\"NewStateValue\":\"ALARM\",\"NewStateReason\":\"[TEST] Manual webhook connectivity test. Safe to ignore.\",\"Region\":\"us-east-1\"}"
         }
       }
     ]
   }

5. A 200 response confirms the path, and a test investigation appears in the DevOps Agent Operator Web App view.

Every scenario reaches DevOps Agent the same way. A CloudWatch alarm moves to ALARM and notifies the SNS topic. Amazon SNS invokes a Lambda function. The function reads the webhook URL and signing secret from Secrets Manager, signs an alarm payload, and POSTs it to the DevOps Agent webhook (as shown in Figure 1). Amazon SNS also provides delivery retries, fan-out to other subscribers, and cross-account publishing.

• Prebuilt Network Firewall metric (Scenario 1) – Alarm-1 watches the DroppedPackets metric, summed across the stateful streams, and triggers when drops rise above a baseline threshold. This requires no workload or custom metric and works on an already-deployed firewall. However, it only tells you that the firewall is dropping packets, not which rule is responsible.
• Application health metric (Scenarios 2 and 3) – Alarm-2 and Alarm-3 watch a custom metric from a connectivity check. Use this for an alarm tied to user-facing impact or to tell one traffic path from another, which requires running a component that emits the metric.

Work through each of the scenarios one at a time, following the same cycle. Interrupt network connectivity, watch the alarm trigger, let DevOps Agent investigate, apply the recommended fix, and confirm recovery before moving on.

The status-page cards follow the live CloudWatch alarm state. A card shows a green dot and the word Healthy when its alarm is clear, and a red dot and the word DROPPED when its alarm triggers. In the DROPPED state the card also adds a Condition: line describing what’s being dropped, which isn’t shown when the card is healthy. Network Firewall applies changes to new flows, so a change shows within a minute or two. Recovery comes from the mitigation DevOps Agent recommends, which you review and apply.

At baseline, the rg-domain Suricata domain rule group denies only an unused placeholder, so the test endpoint stays reachable. The rule group inspects the TLS Server Name Indication (SNI) on each outbound connection and drops any that matches a denied domain. The exact rule syntax and console steps follow.

To add the domain deny rule

1. Go to the Amazon VPC console.
2. In the navigation pane, under Network Firewall, choose Network Firewall rule groups.
3. Choose the rg-domain rule group to open its details page.
4. In the Rules section, choose Edit.
5. The rules box already contains two baseline placeholder rules (they match blocked.placeholder.invalid, so nothing real is denied). Leave those in place. Find the <app-endpoint-dns> value for Scenario 1 in the deployment script output (a Nework Load Balancer (NLB) DNS name such as NfTest-AppNl-a1b2C3dEf4G5-1234abcd5678efgh.elb.us-east-1.amazonaws.com). On a new line below the existing rules, add a drop rule that matches that DNS name on the TLS SNI, then choose Save.

   drop tls $HOME_NET any -> $EXTERNAL_NET any (ssl_state:client_hello; tls.sni; content:"<app-endpoint-dns>"; startswith; nocase; endswith; msg:"S1 domain denylist"; flow:to_server, established; sid:2000002; rev:1;)

6. After saving, the rules box holds all three lines. The two placeholders remain, plus the new drop rule for the endpoint DNS name (note the distinct sid 2000002).

Figure 2: Scenario 1 – Firewall rule change blocking the connection

What happens. The workload’s HTTPS check to the test endpoint times out, the “AWS/NetworkFirewall DroppedPackets metric climbs above baseline, and Alarm-1 moves to ALARM. The Scenario 1 card reads DROPPED (with the condition Firewall dropping the monitored domain on its allow/deny rules), while the Scenario 2 and Scenario 3 cards stay Healthy (Figure 3). On the topology, the alarm pipeline from CloudWatch through Amazon SNS and Lambda to DevOps Agent and the workload-to-firewall inspect lines both turn amber, which the legend defines as collateral / alarm active, because the packets are now dropped at the firewall. To demonstrate the resulting failure, the HTTPS · SNI line from the internet gateway to the test endpoint is shown in red, which the legend defines as dropped (root cause).

Figure 3: Scenario 1 active – Traffic blocked at the firewall

Let DevOps Agent investigate. The agent runs several lines of investigation in parallel and correlates them:

1. Reads the DroppedPackets metric and correlates the spike with a simultaneous drop in passed packets, confirming the firewall is actively blocking traffic.
2. Reads the ALERT log and finds the workload’s TLS connections to the test endpoint blocked by the S1 domain denylist rule.
3. Compares the current state against a baseline window, where the same endpoint was reachable with no alerts, which shows the block is new.
4. Searches CloudTrail and surfaces the UpdateRuleGroup call that added the deny rule, identifying the user, role, and timestamp approximately one minute before the drops began.
5. Reports the root cause as that manual rule-group change. Recommends removing the deny entry or adding an allow exception and enabling FirewallPolicyChangeProtection to prevent unauthorized changes.
6. Presents this as a plan you review and apply, not an automatic change.

In the DevOps Agent Operator Web App view, the agent first restates the Alarm-1 trigger and confirms the firewall is dropping packets above the threshold (Figure 4).

Figure 4: Scenario 1 – The symptom

Next, the agent identifies the root cause: a manual update to the rg-domain rule group that added a domain deny rule (SID 2000002) shortly before the alarm fired, blocking TLS connections to the ELB endpoint (Figure 5).

Figure 5: Scenario 1 – The root cause

Finally, the agent presents a mitigation plan, recommending you remove the problematic deny rule (SID 2000002) to restore connectivity (Figure 6).

Figure 6: Scenario 1 – The mitigation plan

Note: In a real-world environment, this type of rule typically exists for a reason. Before removing it, verify whether it was intentional but scoped too broadly. If so, refine the rule to block only unauthorized endpoints rather than removing it entirely.

Confirm recovery. Apply the change the agent recommends. After the deny entry is gone, DroppedPackets falls back to baseline, Alarm-1 clears, and the card returns to green. Move on to Scenario 2.

At baseline, the rg-stateless-priority stateless rule group keeps the allow rule at priority 100 and the drop rule at 200 for the test class, TCP destination port 9142. The workload opens a TCP connection to the test endpoint on this port. Lower priority numbers evaluate first, so the allow rule wins. This scenario uses port 9142 instead of 443 to demonstrate a stateless rule, which matches on the packet’s 5-tuple (protocol, ports, addresses) rather than application content.

Introduce the change. Invert the two rule priorities so the drop rule evaluates before the allow rule. This is the kind of change a rushed rule edit can introduce.

To invert the stateless rule priorities

1. Go to the Amazon VPC console.
2. In the navigation pane, under Network Firewall, choose Network Firewall rule groups.
3. Choose the rg-stateless-priority rule group to open its details page.
4. In the Rules section, choose Edit.
5. Raise the (Action: Pass) rule’s priority number so it sits after the (Action: Drop) rule, then choose Save. For example, change the (Action: Pass) rule from 100 to 300 (any number higher than the drop rule’s 200 works). You only need to move one rule, and using 300 avoids a clash with the drop rule that already sits at 200. Network Firewall evaluates the lowest priority number first, so the (Action: Drop) rule at 200 now wins for this traffic class, ahead of the (Action: Pass) rule at 300.

Figure 7: Scenario 2 – Rule priority change blocking the traffic class

What happens. The drop rule now wins, the TCP connection to the test endpoint on port 9142 times out, the StatelessRuleFailures metric climbs above baseline, and Alarm-2 moves to ALARM. The Scenario 2 card reads DROPPED (with the condition Stateless rules dropping the monitored traffic class), while the Scenario 1 and Scenario 3 cards stay Healthy (Figure 8). On the topology, the alarm pipeline from CloudWatch through Amazon SNS and Lambda to DevOps Agent and the workload-to-firewall inspect lines both turn amber, which the legend defines as collateral / alarm active, because the packets are now dropped at the firewall. To demonstrate the resulting failure, the TLS :9142 line from the internet gateway to the test endpoint is shown in red, which the legend defines as dropped (root cause).

Figure 8: Scenario 2 active

Let DevOps Agent investigate. A stateless drop happens before traffic reaches the stateful inspection engine, so it produces no ALERT log entries. The agent turns to configuration and flow logs instead:

1. Reads the stateless rule group state and finds the drop rule at the lower priority number, ahead of the pass rule, so the drop evaluates first.
2. Reads the flow logs and sees passed packets drop to zero within a minute of the change.
3. Searches CloudTrail and surfaces the UpdateRuleGroup call that inverted the priorities, identifying the user, role, and timestamp about a minute before the alarm.
4. Reports the root cause as that priority inversion. Recommends removing the redundant drop rule and managing the rule group through infrastructure-as-code (IaC) to prevent manual misconfigurations.
5. Presents this as a plan you review and apply, not an automatic change.

In the DevOps Agent Operator Web App view, the agent first restates the Alarm-2 trigger and confirms that a workload connectivity health check is failing because the firewall’s stateless rules are dropping egress (Figure 9).

Figure 9: Scenario 2 – The symptom

Next, the agent identifies the root cause, using the rule-group state and CloudTrail to pinpoint the conflicting DROP/PASS rules, where the new DROP rule’s lower priority number makes it match first (Figure 10).

Figure 10: Scenario 2 – The root cause

Finally, the agent presents a mitigation plan, recommending you remove the conflicting DROP rule at priority 200 to restore traffic flow (Figure 11).

Figure 11: Scenario 2 – The mitigation plan

Confirm recovery. Apply the change the agent recommends. After the allow rule is ahead of the drop rule again, Alarm-2 clears and the card returns to green. Move on to Scenario 3.

At baseline, the protected subnet in each Availability Zone routes its egress through the firewall endpoint in that same Availability Zone , and the matching return route uses that same endpoint. One endpoint sees both directions of the flow, so the stateful engine completes the handshake. The workload runs in the protected subnet in us-east-1a (CIDR 10.0.4.0/24), so at baseline its egress and its return both use the us-east-1a firewall endpoint.

Introduce the change. Make the flow asymmetric by sending egress out one Availability Zone endpoint while the return comes back through the other. This takes two route edits, and both are required. With only the first edit the flow can still complete, so the alarm will not trigger until both are saved. It makes no firewall-policy change, mirroring a real multi-Availability-Zone routing mistake.

To create asymmetric cross Availability Zone routing

1. Go to the Amazon VPC console and choose Route tables in the navigation pane.
2. Flip the egress. Select the NfNetworkStack/SampleVpc/protectedSubnet1 route table (the us-east-1a protected subnet, where the workload runs). On the Routes tab, choose Edit routes. Its 0.0.0.0/0 route currently targets the us-east-1a firewall endpoint. For the target, choose Gateway Load Balancer Endpoint and select the us-east-1b firewall endpoint, then choose Save changes.
3. Move the return. Select the NfNetworkStack/SampleVpc/publicSubnet2 route table (the us-east-1b public subnet, where egress now exits). Choose Edit routes, then Add route. For the destination enter the workload CIDR 10.0.4.0/24. For the target, choose Gateway Load Balancer Endpoint and select the us-east-1a firewall endpoint. Choose Save changes.

After both edits, a flow’s egress leaves through the us-east-1b endpoint while its return is directed to the us-east-1a endpoint. Neither endpoint sees the whole flow.

Figure 12: Scenario 3 routing change breaking the flow’s symmetry

What happens. A new connection leaves through one endpoint. Its return arrives at the other endpoint, which never saw the connection open, so the handshake fails. Unlike Scenarios 1 and 2, this affects the whole subnet, so all egress stops and Alarm-2 and Alarm-3 both move to ALARM. The AWS/NetworkFirewall DroppedPackets alarm (Alarm-1) stays quiet because no endpoint is making a drop decision. The flow is lost to asymmetric routing rather than counted as a firewall drop. This is why monitoring application connectivity matters. A routing fault is invisible to the firewall’s own drop counter. On the status page, the Scenario 2 card reads DROPPED (with the condition “Stateless rules dropping the monitored traffic class”) and the Scenario 3 card reads DROPPED (with the condition Return traffic dropped by asymmetric cross-Availability-Zone routing), while the Scenario 1 card stays Healthy (Figure 13). On the topology, the alarm pipeline from CloudWatch through Amazon SNS and Lambda to DevOps Agent and the workload-to-firewall inspect lines both turn amber, which the legend defines as collateral / alarm active, while the egress path from the firewall through the NAT gateway and the TLS :9142 and HTTPS · routing lines to the test endpoint turn red, which the legend defines as dropped (root cause).

Figure 13: Scenario 3 – The status page during a path-wide outage

Let DevOps Agent investigate. Both Alarm-2 and Alarm-3 fire in the same datapoint. DevOps Agent recognizes them as linked and merges them into a single investigation:

1. Reads the flow logs and sees bidirectional TLS connections stop abruptly, with only one-way traffic remaining and no flows reaching the established state.
2. Reads the firewall metrics and sees received and passed packets shift from one Availability Zone to the other at the moment of the change.
3. Calls DescribeRouteTables and finds the egress route pointing at one Availability Zone firewall endpoint while the return route points at the other.
4. Searches CloudTrail and surfaces the ReplaceRoute and CreateRoute calls by the same user, about a minute before both alarms fired.
5. Reports the root cause as that asymmetric routing change. Recommends restoring symmetric same-Availability-Zone routing so egress and return traverse the same endpoint.
6. Presents this as a plan you review and apply, not an automatic change.

A mitigation plan is a recommendation you review, not an automatic change, and the right fix depends on the intended design. Restoring symmetric routing can mean sending the workload subnet’s egress back through its own-Availability-Zone firewall endpoint (this sample’s architecture) or, in a design that doesn’t inspect this path, back through a NAT gateway. The agent infers a plausible target from what it can observe, so review the specific route it proposes against your intended topology before you apply it. (Connecting your pipeline or infrastructure-as-code, covered in the next section, lets the agent recommend the target that matches your design.)

In the DevOps Agent Operator Web App view, the agent restates the Alarm-3 (AsymmetricFlowFailures) trigger and confirms the workload’s egress to a monitored endpoint is being blocked by the Network Firewall (Figure 14).

Figure 14: Scenario 3 – The symptom

Next, the agent identifies the root cause: manual route table changes that created cross-AZ asymmetric routing through the network firewall, breaking its symmetric routing requirement (Figure 15)

Figure 15: Scenario 3 – The root cause

Finally, the agent presents a mitigation plan, recommending you restore symmetric routing by pointing protectedSubnet1‘s default route back to the same Availability Zone firewall endpoint, so one endpoint sees both directions of the flow again (Figure 16).

Figure 16: Scenario 3 – The mitigation plan

Confirm recovery. Apply the change the agent recommends, after checking the route target matches your intended design. After the workload subnet’s egress and return use the same Availability Zone firewall endpoint again, the control probe recovers, the alarms clear, and every card returns to green.

In production a single change can trigger several alarms at the same time, as Scenario 3 shows. DevOps Agent links related investigations and works them as one, so you review a single root cause. You can validate the linked findings or unlink an alarm to investigate it independently. If you would rather collapse alarms before they reach the agent, you can add correlation logic in the bridge Lambda function, buffering and grouping by firewall. You can also add email, Amazon Simple Queue Service (Amazon SQS), or HTTP subscribers to the SNS topic, or add the webhook Lambda function to a topic you already run. DevOps Agent produces a mitigation plan but does not change your environment on its own.

You can also give the agent more to work with. DevOps Agent connects to source repositories and CI/CD pipelines, integrating with GitHub (including GitHub Enterprise Server and GitLab Self-Managed through a private connection). It can associate AWS resources with deployments of AWS CloudFormation, AWS CDK, Amazon Elastic Container Registry (Amazon ECR) images, and Terraform. With deployed configuration and recent deployment events in view, the agent correlates the disruption against the change that introduced it and recommends a fix matching your intended design. For this sample, that means recommending the workload subnet’s own Availability Zone firewall endpoint rather than a generic symmetric path.

DevOps Agent also supports proactive incident prevention. It analyzes patterns across past investigations and delivers recommendations to prevent similar issues from recurring, including governance recommendations that strengthen deployment processes and pipeline controls. For Network Firewall rule changes, this means the agent can recommend guardrails for your CI/CD pipeline based on the classes of misconfigurations it has already resolved. You can access these recommendations through the Improvements page in the DevOps Agent Operator Web App.

Clean up the environment with one command.

bash scripts/destroy.sh

It reverts any active scenario, runs cdk destroy for all stacks, and sweeps for stragglers by the Project = nf-devops-agent tag. The main cost drivers are the two Network Firewall endpoints, the NAT gateways (one in the main VPC for each Availability Zone, one in the test-endpoint VPC), and the test endpoint’s load balancers. Each of these bills at an hourly rate for as long as it’s provisioned, whether or not traffic is flowing, so a stack left running continues to accrue charges around the clock even while idle. Running the scenarios and tearing the stack down the same day limits the cost to a few active hours rather than days of idle hourly charges.

In this post, we showed you how AWS DevOps Agent accelerates troubleshooting for three common network firewall connectivity issues. The first was a domain deny list. The second was a stateless priority inversion. The third was an asymmetric cross-AZ routing drop. For each one, DevOps Agent investigated the drop and returned a root cause with a mitigation plan you approve before applying. The first scenario triggered on a prebuilt Network Firewall metric, and the other two on application health metrics. That shows both ways to alarm on a firewall problem through one pipeline.

The pattern isn’t specific to Network Firewall. The same flow fits any service that emits CloudWatch metrics and logs, such as AWS WAF, security groups, and network ACLs. Clone the sample repository to explore the solution, then apply what you learn to your own firewall, application, and alarms. For more details, see the AWS Network Firewall Developer Guide and the AWS Network Firewall pricing page. Start with the Getting Started with AWS DevOps Agent guide to connect your first webhook.

Unit 42 research examines attack scenarios targeting cloud logging services. Learn how to defend against log manipulation and defense evasion.

The post Blinding the Watchmen: Abusing Cloud Logging Services for Defense Evasion and Visibility appeared first on Unit 42.