This post assumes familiarity with envelope encryption and the AWS Encryption SDK.

When your encryption system generates millions of duplicate API calls per hour, costs spiral and performance degrades. That’s exactly the challenge NICE Actimize faced while operating their global-scale, event-driven financial crime detection platform on Amazon Web Services (AWS).

NICE Actimize, a leading provider of financial crime, risk, and compliance solutions, processes millions of encrypted messages daily across hundreds of tenants. By rethinking how they cache encryption keys, they reduced their AWS Key Management Service (AWS KMS) costs by 77% while maintaining strict security guarantees and per-tenant encryption isolation.

In this post, we explore the cache stampede problem that emerges when envelope encryption meets high-concurrency, multi-tenant architectures. We walk through two solutions: the AWS-recommended hierarchical keyring pattern and a custom caching approach that NICE Actimize built for their regulated environment. These patterns apply to multi-tenant software as a service (SaaS) environments and high-throughput systems where per-tenant encryption generates significant KMS API volume.

Financial services systems operate under strict regulatory requirements. You must encrypt data at rest and in transit. For multi-tenant SaaS providers, this requirement might go further: each tenant’s data must be encrypted with separate keys to provide complete cryptographic isolation. If one tenant’s key is compromised, no other tenant’s data is at risk.

Consider an enterprise SaaS environment built on an event-driven architecture using Amazon Managed Streaming for Apache Kafka (Amazon MSK), with many different databases for storing data and Amazon Simple Queue Service (Amazon SQS) for messaging. Messages flow continuously between producers and consumers, and each message must be encrypted with the correct tenant-specific key. At scale with millions of messages daily across hundreds of tenants, this creates a massive volume of encryption and decryption operations.

To handle this volume efficiently, the standard approach is envelope encryption: a two-tier model where an AWS KMS key encrypts short-lived data keys, and those data keys encrypt the actual data. Your application can encrypt large volumes of data locally without calling AWS KMS for every operation, reducing latency and costs.

Envelope encryption reduces AWS KMS calls, but it doesn’t eliminate them. Each encrypt operation still requires a data key, either generated fresh using GenerateDataKey or retrieved from a cache, and each decrypt operation must unwrap an encrypted data key (EDK) by calling Decrypt. In high-throughput systems processing millions of messages, these calls add up quickly.

The AWS Encryption SDK provides a built-in solution for this: the CachingCryptoMaterialsManager. This component caches data encryption materials (data keys) locally, so your application can reuse them across multiple operations without calling AWS KMS each time. You configure a time-to-live (TTL), a maximum message-use limit, and a local cache, and the SDK handles the rest.

This approach works well under moderate load when you partition the cache by tenant AWS KMS key Amazon Resource Name (ARN) so that each tenant’s encryption materials remain cryptographically isolated. However, a critical problem emerges as concurrency scales to hundreds of threads processing millions of encrypted messages in parallel: the cache stampede, also known as the thundering herd problem.

The CachingCryptoMaterialsManager caches the result of the SDK’s internal getMaterialsForEncrypt and decryptMaterials calls at the materials level. The cache stampede, however, happens at the KMS API call level. When a cached data key expires or a new, previously-unseen EDK arrives, the following sequence unfolds:

1. On encrypt – data key explosion: Multiple threads simultaneously call encrypt() for the same tenant. Each thread finds the cache entry expired and independently calls GenerateDataKey against AWS KMS. Instead of one thread generating a data key while others wait, N threads create N distinct data keys. Each new data key produces a unique EDK, which inflates the EDK cardinality across the system.
2. On decrypt – redundant unwrap calls: Those extra unique EDKs propagate downstream. When consumers later read encrypted records, each distinct EDK is a separate cache key. Multiple threads encountering the same EDK simultaneously each trigger an independent Decrypt call to AWS KMS because the cache has no coordination mechanism to make competing threads wait for a single in-flight request.
3. Compounding effect: The encrypt-side stampede creates excess EDK cardinality, which degrades the decrypt-side cache hit ratio, which triggers more KMS calls, which drives up costs further. In the NICE Actimize case, this produced a ratio of 30% unique data keys to data records in DynamoDB tables, meaning nearly one in three records was encrypted with a different data key.

At enterprise SaaS scale, this compounding effect can generate millions of redundant AWS KMS GenerateDataKey and Decrypt calls per hour, even with the SDK’s built-in caching enabled. The following figure shows the pattern leading to a stampede.

Figure 1: Cache stampede – multiple threads independently calling AWS KMS for the same encrypted data key, creating duplicate requests

The stampede follows this sequence on the encrypt side:

1. Multiple threads call encrypt() for the same tenant concurrently.
2. Each thread checks the CachingCryptoMaterialsManager and finds the cache entry expired.
3. With no coordination mechanism, each thread independently calls GenerateDataKey.
4. AWS KMS returns N distinct data keys (one per thread).
5. Each data key produces a unique EDK, inflating cardinality across the system.

On the decrypt side, the inflated EDK cardinality compounds the problem:

6. Consumer threads encounter unique EDKs that were never cached.
7. Multiple threads hitting the same EDK simultaneously each trigger a separate Decrypt call. AWS KMS returns the same plaintext data key N times, doing redundant work.

We evaluated two approaches to solve the cache stampede problem. Each fits different architectural requirements and regulatory constraints.

AWS addresses the cache stampede challenge through the hierarchical keyring pattern, which introduces an additional level of key hierarchy that significantly reduces how often cache stampedes occur.

In this architecture, branch keys serve as intermediate wrapping keys stored in a DynamoDB table. This DynamoDB table acts as a shared cache layer that coordinates across all instances in your distributed fleet.

Figure 2: Hierarchical keyring architecture – branch keys in DynamoDB coordinating across distributed instances

The architecture (shown in Figure 2) works as follows:

1. The application requests encryption through the hierarchical keyring.
2. The keyring checks the local cache for the tenant’s branch key.
3. On a cache miss, it queries the DynamoDB Key Store table for the active branch key.
4. AWS KMS decrypts the branch key (this is the only KMS call in the flow).
5. The decrypted branch key is returned to the keyring.
6. The keyring stores the branch key in the local cache for subsequent requests.
7. The keyring derives a unique wrapping key from the branch key and generates the data key locally.

The key insight is that the cache is thread-aware. When the cache expires, threads coordinate to make a single request to refresh the cache. Only a single thread is used to make a call to the branch key, rather than all the threads acting independently. Additionally, by adding an additional key into the key hierarchy, branch keys don’t live within AWS KMS. This means cache misses and the stampedes they trigger interact with the branch key, and don’t make as many calls to the AWS KMS service at the top of the hierarchy:

• Without hierarchical keyrings: Your local cache needs to store all the data encryption keys, and has constant misses as new, unique data keys arrive with each encrypted message. A miss can trigger a stampede.
• With hierarchical keyrings: The same branch key wraps thousands or millions of data keys. A cache miss only occurs when a branch key expires or is first requested, which happens orders of magnitude less frequently than without hierarchical keyrings.

The DynamoDB table acts as a coordination point. The first thread to request a missing branch key retrieves it from AWS KMS and stores it in DynamoDB (the Key Store table). Subsequent requests from instances in the fleet retrieve the cached branch key from DynamoDB instead of making duplicate AWS KMS calls.

Beyond reducing cache miss frequency, the hierarchical keyring provides built-in stampede protection within its local cache implementation. The SDK offers multiple cache types, and the Default cache, designed for heavily multi-threaded environments, prevents multiple threads from calling AWS KMS on cache expiry by notifying a single thread that the branch key materials entry is about to expire 10 seconds in advance. That one thread refreshes the cache while all other threads continue serving requests using the still-valid entry.

This solution integrates with the AWS Encryption SDK and requires minimal code changes to existing applications. For event-driven architectures processing encrypted Kafka streams, this approach reduces KMS call volume by orders of magnitude while preserving per-tenant cryptographic isolation.

While the hierarchical keyring (Option A) addresses the stampede by reducing how often cache misses occur, there’s a complementary approach: eliminating the stampede at its source by caching KMS API responses directly, using atomic, single-flight cache loading that prevents concurrent threads from issuing duplicate calls. This is the path NICE Actimize took.

The IClientSupplier extension point in AWS Encryption SDK v3

In the AWS Encryption SDK v2, decorating the AWS KMS client on a per-request basis was possible through the RegionalClientSupplier interface, but it was an advanced and undocumented use case. Without explicit guidance or a supported pattern, caching strategies typically operated above the SDK layer, making it difficult to prevent duplicate KMS calls at their source. The AWS Encryption SDK v3 introduced the IClientSupplier interface, which the AwsKmsMrkMultiKeyring accepts at construction time. This interface is called by the SDK whenever it needs a KMS client for a given AWS Region, and you control what it returns, making it possible to insert a caching layer between the SDK and AWS KMS.

Architecture: A decorated KMS client with two Caffeine caches
The solution is a CachedKmsClient—a decorator that wraps the standard AWS SDK KmsClient and interposes two Caffeine LoadingCache instances between the application and AWS KMS:

Both caches are configured with refreshAfterWrite (default: 1 hour, configurable), which means:

• During the refresh window, concurrent threads receive the cached response instantly resulting in zero KMS calls.
• When a cache entry expires, Caffeine’s LoadingCache.get() guarantees that exactly one thread executes the loader function (the actual KMS API call), while all other concurrent threads block and wait for that single result. This is the atomic, single-flight property that eliminates the stampede.

Security consideration: Caching plaintext data keys in memory means the keys exist in process memory for the duration of the cache TTL. The TTL acts as a security control: shorter TTLs reduce the window of exposure in the event of a memory dump, while longer TTLs reduce KMS call volume. Choose a TTL that balances your security requirements with your cost and performance goals. Key rotation at the KMS key level remains unaffected by the cache, because rotated keys produce new data keys on the next cache refresh.

Integration with the AWS Encryption SDK v3

The integration is minimal. The IClientSupplier AWS Lambda function returns a CachedKmsClient singleton for each AWS Region, this singleton is passed into the AwsKmsMrkMultiKeyring at keyring construction time. From that point forward, each GenerateDataKey and Decrypt call the SDK makes flows through the caching decorator transparently, with no changes to the encrypt or decrypt call sites.

The CachedKmsClient is a singleton per Region (managed using a ConcurrentHashMap), so all tenants on the same node share the same caching layer but their data keys remain fully isolated because the cache keys include the tenant-specific AWS KMS key ARN.

Why Caffeine?

Caffeine is a high-performance, near-optimal Java caching library well-suited for this pattern for several reasons:

• Atomic loading: LoadingCache.get() guarantees that on a cache miss, only one thread executes the loader while others wait. This is the core property that eliminates the stampede.
• refreshAfterWrite semantics: Unlike expireAfterWrite (which blocks all threads during refresh), refreshAfterWrite allows one thread to asynchronously reload the entry while other threads continue to serve the stale-but-valid cached value. This eliminates latency spikes during key rotation.
• Observability: Cache eviction listeners and Micrometer metric counters can be wired in to track actual KMS call volume per tenant KMS key, enabling real-time cost monitoring.

The hierarchical keyring with DynamoDB (Option A) is a production-ready, AWS-recommended solution that reduces stampede frequency by introducing longer-lived branch keys. It’s the best choice for most organizations. Particularly when starting fresh or when the operational overhead of an additional data store is acceptable.

NICE Actimize chose the custom caching approach (Option B) for a pragmatic reason: it avoided introducing a new infrastructure dependency into the encryption critical path. Their platform already operated at scale across hundreds of tenants, and adding a DynamoDB table as a key coordination layer would have meant taking on additional operational responsibility: provisioning, monitoring, backup, access control, and ensuring high availability for a component that sits directly in the encrypt/decrypt hot path. In a regulated financial services environment, each new stateful component in the security chain requires its own resilience planning, failure-mode analysis, and compliance review. The Caffeine cache used in Option B, by contrast, is an in-process library (a JAR on the classpath). It is stateless, requires no network calls, no provisioning and no operational overhead. It makes a lighter dependency than a managed cloud resource in the critical path. There is no shared state to lose, no additional infrastructure to protect, and no new failure mode beyond what already exists with AWS KMS itself. If a node restarts, the cache rebuilds on the next KMS call.

By implementing a rotation policy with the optimized caching approach, NICE Actimize achieved the following results:

• 77% reduction in AWS KMS costs – Eliminating millions of redundant API calls translated directly into significant cost savings.
• Maintained strict per-tenant isolation – Per-tenant encryption isolation remained fully intact, with no compromise to their security posture.
• Improved system performance – Removing the stampede of duplicate AWS KMS calls reduced latency and freed up system resources for core processing.
• Simplified operations – A coordinated caching layer replaced fragmented, per-thread caching, reducing operational complexity.

The cache stampede problem compounds in multi-tenant encryption systems: excess data key generation on the encrypt side degrades cache hit ratios on the decrypt side, creating a feedback loop of redundant KMS calls. The AWS-recommended hierarchical keyring pattern with DynamoDB provides a production-ready solution that integrates with the AWS Encryption SDK with minimal code changes. For regulated environments requiring additional control, a custom caching approach can deliver similar results.

If you operate a multi-tenant SaaS platform or a high-throughput system with per-tenant encryption requirements, consider these patterns to optimize your encryption costs and performance.

To get started, explore the following resources:

• AWS Encryption SDK Developer Guide
• AWS KMS Developer Guide
• Amazon DynamoDB Developer Guide
• Amazon MSK Developer Guide

If you have questions or feedback about this post, leave a comment in the Comments section.

Software as a service (SaaS) providers building AI-powered applications on Amazon Bedrock AgentCore often need to serve multiple tenants with distinct security requirements from a shared infrastructure. Some tenants require cross-account access from their own Amazon Web Services (AWS) accounts, while others mandate that traffic stay within a private virtual private cloud (VPC) for regulatory compliance. Without centralized resource-level control, managing these diverse requirements can be complex.

AgentCore supports resource-based policies, giving you centralized, resource-level control over who can access your AgentCore Runtime and AgentCore Runtime endpoint resources and under what conditions.

In this post, you walk through a multi-tenant AI customer service platform where two tenants need different levels of access to the same agent. You learn how to use resource-based policies on AgentCore to grant cross-account access for one tenant while restricting another to VPC-only traffic—all while sharing the same underlying AgentCore Runtime and AgentCore Runtime endpoint.

Imagine you’re an SaaS provider who builds and operates an AI-powered customer service platform. You use AgentCore to deploy intelligent agents that handle customer inquiries, answering product questions, processing returns, and escalating complex issues to human agents.

You serve multiple enterprise clients (tenants), each with their own AWS account and unique security requirements:

• Tenant A: Example Corp is a large retailer operating in AWS account 111122223333. Their development team is building a customer-facing chat agent that calls your AI agent to answer product questions in real time, and their admin team needs access to test agent behavior and monitor responses. Both roles must invoke the agent directly from Example Corp’s own AWS account without you having to share credentials or create AWS Identity and Access Management (IAM) users on their behalf. Example Corp has no network restriction requirements—their teams can invoke the agent from any network path as long as they have valid AWS credentials.
• Tenant B: AnyCompany is a healthcare company operating in AWS account 444455556666. Because of regulatory (HIPAA) requirements, AI agent traffic must originate only from their private VPC (vpc-health1234). Their internal support staff uses the AI agent to assist with patient billing inquiries, which might involve protected health information (PHI). Their compliance team mandates that no API call to the agent can be made from developer laptops, public endpoints, or any network outside the controlled VPC boundary.
• Your platform (SaaS provider) runs in account 555555555555 in the us-west-2 AWS Region. You operate an AgentCore Runtime (support-agent-runtime) that handles the core customer service logic, and an AgentCore Runtime endpoint (DEFAULT) that routes requests to the latest version of the support agent. Both tenants share this same agent infrastructure.

You can use resource-based policies to define who can access your AgentCore Runtime and AgentCore Runtime endpoint directly on the resources themselves—centralizing access control on the resource side. For cross-account scenarios like Example Corp, both a resource-based policy on your resources and an identity-based policy in the tenant’s account are required. For VPC-restricted scenarios like AnyCompany, you can use specific IAM conditions to enforce that requests originate only from an approved VPC, adding a network-level security boundary on top of identity-based controls.

The following diagram shows the architecture for the multi-tenant AI customer service platform with both access patterns.

Figure 1: Architecture for the multi-tenant AI customer service platform with both access patterns

• Your account (555555555555) with AgentCore Runtime and AgentCore Runtime endpoint
• Example Corp’s account (111122223333) with DeveloperRole and AdminRole
• AnyCompany’s account (444455556666) with VPC boundary and ApplicationRole
• Policy enforcement points on both resources
• VPC endpoint in AnyCompany’s VPC connecting to AgentCore

The SaaS provider account (555555555555) hosts the AgentCore Runtime and AgentCore Runtime endpoint that both tenants share. Example Corp (111122223333) accesses the agent cross-account using IAM roles—DeveloperRole and AdminRole—authenticated with Signature Version 4 (SigV4), the standard AWS request signing protocol. AWS evaluates both the resource-based policy on your resources and the identity-based policy in Example Corp’s account before granting access.

AnyCompany (444455556666) also accesses the agent cross-account, but with an additional constraint: all requests must originate from within their private VPC (vpc-health1234) through a VPC endpoint for AgentCore. The resource-based policy on your resources includes an explicit Deny statement that blocks any request from AnyCompany’s ApplicationRole when it doesn’t originate from the approved VPC.

In both cases, resource-based policies must be applied to both the AgentCore Runtime and AgentCore Runtime endpoint. AWS evaluates policies on both resources for InvokeAgentRuntime operations—if either resource denies access or lacks an explicit Allow, the request is denied.

Before you begin, ensure you have the following:

• An AWS account with AgentCore access and permissions to call PutResourcePolicy, GetResourcePolicy, and DeleteResourcePolicy on AgentCore resources
• AWS Command Line Interface (AWS CLI) v2 installed and configured with the bedrock-agentcore-control API available
• An AgentCore Runtime with SigV4 authentication and a DEFAULT AgentCore Runtime endpoint pointing to the latest runtime version

For the VPC-restricted scenario, the tenant must have a VPC endpoint for AgentCore configured in their VPC. An interface VPC endpoint creates a private connection between the tenant’s VPC and the AgentCore service without requiring traffic to traverse the public internet. For more information, see Interface VPC endpoints for Amazon Bedrock AgentCore.

Both Example Corp and AnyCompanyoperate in separate AWS accounts from your platform. For cross-account access to AgentCore Runtime, AWS requires that both of the following allow the action:

• A resource-based policy in your platform account applied to both the AgentCore Runtime and its AgentCore Runtime endpoint. InvokeAgentRuntime operations require an explicit Allow on both resources—if either lacks one, the request is denied.
• An identity-based policy attached to the caller’s IAM role in the tenant’s account.

If either side is missing or denies the action, the request is denied.

Example Corp’s DeveloperRole and AdminRole in account 111122223333 need to invoke your AI customer service agent. Without resource-based policies, enabling this cross-account access would typically require Example Corp’s roles to assume a role in your platform account through IAM role chaining—adding operational complexity, introducing temporary credential management, and creating additional IAM roles that must be maintained in your account for each tenant. With resource-based policies, you grant Example Corp’s roles direct access to your AgentCore Runtime and AgentCore Runtime endpoint without role chaining. Example Corp’s roles can invoke the agent directly from their own account using their own credentials, while you maintain centralized control over access on the resource side.

AgentCore Runtime resource-based policy

The following policy grants Example Corp’s DeveloperRole and AdminRole permission to invoke the agent runtime. This is the first of two resource-based policies required—it controls access to the runtime resource itself. Save this as runtime-policy.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowExampleCorpCrossAccountAccess",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::111122223333:role/DeveloperRole",
          "arn:aws:iam::111122223333:role/AdminRole"
        ]
      },
      "Action": "bedrock-agentcore:InvokeAgentRuntime",
      "Resource": "arn:aws:bedrock-agentcore:us-west-2:555555555555:runtime/support-agent-runtime"
    }
  ]
}

AgentCore Runtime endpoint resource-based policy

The following policy grants the same roles permission to invoke the AgentCore Runtime endpoint. Without this second policy, requests are allowed at the runtime level but denied at the endpoint level, and the invocation fails. Save this as endpoint-policy.json:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowExampleCorpCrossAccountAccess",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::111122223333:role/DeveloperRole",
                    "arn:aws:iam::111122223333:role/AdminRole"
                ]
            },
            "Action": "bedrock-agentcore:InvokeAgentRuntime",
            "Resource": "arn:aws:bedrock-agentcore:us-west-2:999999999999:runtime/support-agent-runtime/runtime-endpoint/DEFAULT"
        }
    ] 
}

To apply the resource-based policies

aws bedrock-agentcore-control put-resource-policy \--resource-arn arn:aws:bedrock-agentcore:us-west-2:555555555555:runtime/support-agent-runtime \--policy file://runtime-policy.json \--region us-west-2
aws bedrock-agentcore-control put-resource-policy \--resource-arn arn:aws:bedrock-agentcore:us-west-2:555555555555:runtime/support-agent-runtime/runtime-endpoint/DEFAULT \--policy file://endpoint-policy.json \--region us-west-2

To verify the resource-based policies

aws bedrock-agentcore-control get-resource-policy \--resource-arn arn:aws:bedrock-agentcore:us-west-2:555555555555:runtime/support-agent-runtime \--region us-west-2
aws bedrock-agentcore-control get-resource-policy \--resource-arn arn:aws:bedrock-agentcore:us-west-2:555555555555:runtime/support-agent-runtime/runtime-endpoint/DEFAULT \--region us-west-2

Configure an identity-based policy (Example Corp’s account)

Resource-based policies alone aren’t sufficient for cross-account access. Example Corp must also attach an identity-based policy to DeveloperRole and AdminRole in their account (111122223333) that allows the same action on your resources. Without this policy on the tenant side, IAM denies the cross-account request even though your resource-based policies allow it.

Example Corp attaches the following policy to both DeveloperRole and AdminRole:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowInvokeAgentRuntime",
            "Effect": "Allow",
            "Action": "bedrock-agentcore:InvokeAgentRuntime",
            "Resource": [
                "arn:aws:bedrock-agentcore:us-west-2:555555555555:runtime/support-agent-runtime",
                "arn:aws:bedrock-agentcore:us-west-2:555555555555:runtime/support-agent-runtime/runtime-endpoint/DEFAULT"
            ]
        }
    ] 
}

Attach this policy to both DeveloperRole and AdminRole in Example Corp’s account.

AnyCompany operates under HIPAA compliance requirements and mandates that all traffic to your agent stays within a private network path. Like Example Corp, AnyCompany needs cross-account access from account 444455556666—but with an additional constraint, requests must originate from their VPC vpc-health1234 through an interface VPC endpoint. Any request from outside this VPC is denied, even if it comes from AnyCompany’s ApplicationRole.

Resource-based policies (your platform account): To enforce this, you update the resource-based policies on both the AgentCore Runtime and AgentCore Runtime endpoint. Each policy includes an Allow statement that grants ApplicationRole permission to invoke the agent, paired with a Deny statement that blocks any request not originating from vpc-health1234. In the following policy, the Deny statement uses StringNotEquals on aws:SourceVpc . When a request arrives through an interface VPC endpoint, AWS populates this key with the VPC ID. If it doesn’t match vpc-health1234, or if the key is absent because no VPC endpoint was used, the Deny takes effect. Because an explicit Deny overrides any Allow from any policy, this pattern helps ensure that no other identity-based or resource-based policy can inadvertently grant AnyCompany access from outside the VPC. Add the following statements to runtime-policy-v2.json alongside the Example Corp statement from Step 1:

{
  "Sid": "AllowAnyCompanyCrossAccountAccess",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::444455556666:role/ApplicationRole"
  },
  "Action": "bedrock-agentcore:InvokeAgentRuntime",
  "Resource": "arn:aws:bedrock-agentcore:us-west-2:555555555555:runtime/support-agent-runtime"
},
{
  "Sid": "DenyAnyCompanyOutsideVpc",
  "Effect": "Deny",
  "Principal": {
    "AWS": "arn:aws:iam::444455556666:role/ApplicationRole"
  },
  "Action": "bedrock-agentcore:InvokeAgentRuntime",
  "Resource": "arn:aws:bedrock-agentcore:us-west-2:555555555555:runtime/support-agent-runtime",
  "Condition": {
    "StringNotEquals": {
      "aws:SourceVpc": "vpc-health1234"
    }
  }
}

AgentCore Runtime endpoint resource-based policy

Add the equivalent statement to endpoint-policy-v2.json:

{
    "Sid": "AllowAnyCompanyCrossAccountAccess",
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::444455556666:role/ApplicationRole"
    },
    "Action": "bedrock-agentcore:InvokeAgentRuntime",
    "Resource": "arn:aws:bedrock-agentcore:us-west-2:555555555555:runtime/support-agent-runtime/runtime-endpoint/DEFAULT"
},
{
    "Sid": "DenyHealthFirstOutsideVpc",
    "Effect": "Deny",
    "Principal": {
        "AWS": "arn:aws:iam::444455556666:role/ApplicationRole"
    },
    "Action": "bedrock-agentcore:InvokeAgentRuntime",
    "Resource": "arn:aws:bedrock-agentcore:us-west-2:555555555555:runtime/support-agent-runtime/runtime-endpoint/DEFAULT",
    "Condition": {
        "StringNotEquals": {
            "aws:SourceVpc": "vpc-health1234"
        }
    }
}

Because put-resource-policy replaces the entire policy on a resource, your updated policy files must include both the preceding AnyCompany statments and the Example Corp statements from Step 1.

Apply the updated resource-based policies

aws bedrock-agentcore-control put-resource-policy \
	--resource-arn arn:aws:bedrock-agentcore:us-west-2:555555555555:runtime/support-agent-runtime \
	--policy file://runtime-policy-v2.json \
	--region us-west-2 

aws bedrock-agentcore-control put-resource-policy \
	--resource-arn arn:aws:bedrock-agentcore:us-west-2:555555555555:runtime/support-agent-runtime/runtime-endpoint/DEFAULT \
	--policy file://endpoint-policy-v2.json \
	--region us-west-2

Verify the updated policies

After applying the final policies, verify them using the get-resource-policy command:

# Verify Agent Runtime policy 
aws bedrock-agentcore-control get-resource-policy \
	--resource-arn arn:aws:bedrock-agentcore:us-west-2:555555555555:runtime/support-agent-runtime \
	--region us-west-2 

# Verify Agent Runtime Endpoint policy 
aws bedrock-agentcore-control get-resource-policy \
	--resource-arn arn:aws:bedrock-agentcore:us-west-2:555555555555:runtime/support-agent-runtime/runtime-endpoint/DEFAULT \
	--region us-west-2

Identity-based policy (AnyCompany’s account)

AnyCompany must attach an identity-based policy to ApplicationRole in their account 444455556666 that allows the same InvokeAgentRuntime on your resources in account 555555555555. Without this policy on the tenant side, IAM denies the cross-account request even though your resource-based policies allow it.

AnyCompany attaches the following policy to ApplicationRole:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowInvokeAgentRuntime",
            "Effect": "Allow",
            "Action": "bedrock-agentcore:InvokeAgentRuntime",
            "Resource": [
                "arn:aws:bedrock-agentcore:us-west-2:555555555555:runtime/support-agent-runtime",
                "arn:aws:bedrock-agentcore:us-west-2:555555555555:runtime/support-agent-runtime/runtime-endpoint/DEFAULT"
            ]
        }
    ] 
}

The VPC restriction is enforced entirely on resource account through the resource-based policy condition, AnyCompany’s identity-based policy doesn’t need VPC conditions. This keeps the tenant-side configuration straightforward while you maintain centralized network-level control.

OAuth authentication considerations

The policies in this post use SigV4 authentication with specific IAM role principals. If your AgentCore Runtime or AgentCore Gateway is configured with OAuth authentication instead, the principal structure changes. OAuth-authenticated resources require a wildcard principal (“Principal": "*") because the caller identity comes from a JSON Web Token (JWT) validated before policy evaluation. Anonymous or unauthenticated requests are rejected before the policy is evaluated, so the wildcard principal doesn’t grant open access. To restrict OAuth-authenticated requests to a specific VPC, combine the wildcard principal with a VPC condition in the resource-based policy. IAM principal-based condition keys such as aws:PrincipalAccount and aws:PrincipalOrgID aren’t populated in the OAuth authentication context—only supported network-level condition keys (such as aws:SourceVpc, aws:SourceVpce, aws:SourceIp) are available for use in resource-based policies with OAuth. For more details, see Resource-based policies for Amazon Bedrock AgentCore.

Understanding policy evaluation

To understand how AWS evaluates these policies when a request arrives, consider the following scenarios:

In this post, you learned how to use resource-based policies on AgentCore to secure a multi-tenant AI platform with distinct access patterns for each tenant:

• Example Corp gets seamless cross-account integration, their development and admin teams can invoke your AI agent directly from their own AWS account without credential management.
• AnyCompany gets the strict network-level isolation their compliance team requires, the AI agent is accessible only from within their private VPC, ensuring that interactions involving potential PHI — stay within the controlled network boundary

Both tenants share the same underlying AgentCore Runtime and AgentCore Runtime endpoint, yet each has tailored security controls enforced at the resource level. his approach avoids per-tenant infrastructure duplication while satisfying each tenant’s security posture, a challenge you likely face when onboarding tenants with different compliance postures. Resource-based policies complement identity-based IAM policies, giving you layered control over which principals can invoke which agents, and from which network paths.

• Learn more about Amazon Bedrock AgentCore in the official documentation
• Review resource-based policies for Amazon Bedrock AgentCore
• Explore other AWS global condition keys including aws:SourceVpce and aws:PrincipalOrgID
• Visit the AWS Well-Architected Framework Security Pillar for multi-tenancy best practices

If you have feedback about this post, submit comments in the Comments section below.

Managing identities and access across complex environments has become more critical than ever. AWS Directory Service for Managed Microsoft Active Directory, also known as AWS Managed Microsoft AD, has added new capabilities to manage users and groups. Now, you can perform create, read, update, and delete (CRUD) operations on users and groups directly through AWS Command Line Interface (AWS CLI), APIs, and the AWS Management Console. You can use this powerful capability to automate identity lifecycle management and enhance security in your AWS environment. By using these APIs, collectively known as the Directory Service Data APIs, you can perform operations such as:

• Listing users and groups
• Retrieving user and group details
• Disabling and enabling user accounts
• Resetting user passwords
• Managing group memberships

These APIs provide new possibilities for automating identity management tasks and integrating Active Directory management into your existing workflows and applications.

The introduction of these APIs brings several key benefits:

• Automation of the identity lifecycle: You can now programmatically manage user accounts throughout their lifecycle—from creation to deletion—enabling streamlined onboarding and offboarding processes.
• Enhanced security: By integrating these APIs with security services like Amazon GuardDuty, you can create automated responses to potential security threats, such as disabling accounts with inappropriate access.
• Improved compliance: You can use automated user management to help enforce consistent policies and help maintain compliance with various regulatory requirements.
• Operational efficiency: You can automate routine tasks such as user provisioning, deprovisioning, and group management, reducing manual effort and the potential for human error.
• Integration capabilities: By using these APIs, you can seamlessly integrate with existing identity management systems, custom applications, and third-party tools.
• Cost optimization: By automating processes and reducing manual intervention, you can potentially help your organization optimize operational costs associated with identity management.

In this post, we explore these new APIs and demonstrate how you can use them to create an automated solution for detecting and responding to unexpected behavior by Active Directory users. We walk through a practical example that combines GuardDuty, AWS Step Functions, Amazon EventBridge, and the new AWS Directory Service APIs to create a robust security automation workflow.

To demonstrate the power of these new APIs, let’s explore a practical solution that automates the detection and response to unexpected behavior by Active Directory users. This solution combines several AWS services to create a robust security automation workflow:

1. 1. GuardDuty continuously monitors for unexplained behavior of Active Directory users from AWS Managed Microsoft AD. For the example in this post, we’re using Backdoor:Runtime/C&CActivity.B!DNS
   2. An EventBridge rule detects GuardDuty findings related to these users and triggers a Step Functions workflow.

      {
        "detail-type": ["GuardDuty Finding"],
        "source": ["aws.guardduty"],
        "detail": {
          "type": ["Backdoor:Runtime/C&CActivity.B!DNS"]
        }
      }

   3. The Step Functions workflow will:
      1. Extract the Active Directory username from the instance using a run command.
      2. Start an automation that will disable the account using the DisableUser API.

Figure 1: Diagram of the Step Functions workflow showing the process of Systems Manager finding the username and starting the automation to disable the account

4. Finally, another EventBridge rule will monitor the DisableUser API call. It will send an email to the user using Amazon Simple Notification Service (Amazon SNS) notifications.

   {
     "detail-type": ["AWS API Call via CloudTrail"],
     "source": ["aws.ds"],
     "detail": {
       "eventSource": ["ds.amazonaws.com"],
       "eventName": ["DisableUser"]
     }
   }

This solution delivers automated, near real-time remediation of potential security threats — significantly reducing exposure windows and containing the impact of unauthorized account access.

The following figure shows a high-level architecture diagram of the solution.

Figure 2: Diagram showing the workflow of what happens when potentially damaging activity is detected

Note: The solution must be deployed in the primary AWS Region of your directory.

To complete the walkthrough in this post, you must have the following prerequisites in place.

GuardDuty is an automated threat detection service that continuously monitors for unexpected activity and unauthorized behavior to protect your AWS accounts, workloads, and data stored in Amazon Simple Storage Service (Amazon S3).

To activate GuardDuty:

1. Go to the GuardDuty console.
   1. If you’re activating GuardDuty for the first time, under Try threat detection with GuardDuty, select All Features and then choose Get Started.
   2. If you’ve used GuardDuty before, select Runtime Monitoring and then choose Enable under Runtime Monitoring.

Figure 3: Runtime Monitoring enabled

AWS Managed Microsoft AD provides a fully managed service for Microsoft Active Directory (AD) in the AWS Cloud. When you create your directory, AWS deploys two domain controllers that are exclusively yours in separate Availability Zones for high availability. For use cases that require even higher resilience and performance in a specific AWS Region or during specific hours, you can scale AWS Managed Microsoft AD by deploying additional domain controllers to meet your needs. These domain controllers can help load balance, increase overall performance, or provide additional nodes to protect against temporary availability issues. Using AWS Managed Microsoft AD, you can define the correct number of domain controllers for your directory based on your use case.

To deploy a new AWS Managed Microsoft AD:

1. Go to the Directory Service console.
2. Choose Set up directory and select AWS Managed Microsoft AD.
3. Select Standard Edition and enter a directory DNS name and password.
4. Select a virtual private cloud (VPC). For this example, use the Default VPC.
5. Choose Create directory.

# Create the test user
aws ds-data create-user \
 --directory-id "your-directory-id" \
 --sam-account-name "TestUser" \
 --given-name "Test" \
 --surname "User"

# Set a password for the test user 
aws ds reset-user-password \
 --directory-id "your-directory-id" \
 --user-name "TestUser" \
 --new-password "YourSecurePassword123!"

To generate alerts on GuardDuty, you need a domain joined Linux EC2 instance. If you don’t have a domain joined EC2 Linux instance, follow these instructions for joining a Linux instance to an Active Directory domain. This instance will be used to simulate suspicious activity that triggers a GuardDuty finding and initiates the automated remediation workflow.

Let’s walk through the steps to implement this solution in your AWS environment.

1. Download the CloudFormation template
2. Navigate to the CloudFormation console in the AWS account.
3. For Create Stack, choose with new resources (standard).
4. For Template source, choose Upload a template file. Choose Choose file and select the template you downloaded in step 1.
5. Choose Next.
6. For Stack name, enter a stack name (such as CRUD-API-MAD).
7. In the Parameters area, do the following:
   1. For DirectoryID, enter the AWS Active Directory ID.
   2. For NotificationEmail, enter the email address to send the notification to.
8. On the Configure stack options page, choose Next.
9. Select I acknowledge that AWS CloudFormation might create IAM resources with custom names, then choose Submit.

After the page is refreshed, the status of your stack should be CREATE_IN_PROGRESS. When the status changes to CREATE_COMPLETE, proceed to the next section.

To simulate a threat, use a GuardDuty test domain that GuardDuty will recognize as a command and control server.

1. Go to the Amazon EC2 console.
2. Choose Instances from the navigation pane.
3. Select the test EC2 instance that you created earlier.
4. Choose Connect, select the Session Manager tab, and choose Connect.
5. Authenticate with your test user by entering su followed by the test user with the domain name that you created earlier. For example su TestUser@example.com, then enter the password.
6. Enter the command curl guarddutyc2activityb.com.
   You will receive an error because the page won’t resolve, but GuardDuty will have detected concerning events.
7. Go to the GuardDuty console and select Findings from the navigation pane.
8. Within 3–5 minutes, you should see a high severity finding for Backdoor:Runtime/C&CActivity.B!DNS.
9. This will then trigger the automation to disable the account.
   
   

   Figure 4: Account successfully disabled

10. After the account is disabled, an email notification will be sent notifying an administrator that the account was disabled (it might take up to 5 minutes to receive the notification).
    

    Figure 5: AWS notification message showing the username has been disabled

Note: You must archive the GuardDuty finding before running this test again, because the EventBridge rule only runs once against a GuardDuty finding with the same details. To archive the finding, select the check box next to the Backdoor:Runtime/C&CActivity.B!DNS finding, choose Actions (top right), and select Archive.

The new AWS Directory Service APIs for AWS Managed Microsoft AD provide powerful capabilities for programmatically managing Active Directory users and groups. By using these APIs in conjunction with services such as Amazon GuardDuty and AWS Step Functions, you can create sophisticated automation workflows that enhance your security posture and streamline identity management processes.

The solution we’ve explored in this post demonstrates just one of many possible use cases for these new APIs. As you integrate these capabilities into your own environments, you will probably discover numerous opportunities to improve efficiency, security, and compliance in your identity management practices.

For a solution that uses PowerShell Active Directory cmdlets with AWS Systems Manager Run Command to disable users, see How to automatically disable users in AWS Managed Microsoft AD based on GuardDuty findings.

For more information about AWS Directory Service and its APIs, visit the AWS Directory Service documentation.

We’re excited to see how you’ll use these new APIs to innovate and improve your identity management workflows. If you have any questions or want to share your own use cases, leave a comment below or reach out to AWS Support.

Remember, the cloud journey is all about continuous improvement and innovation. Keep exploring, keep learning, and keep pushing the boundaries of what’s possible with AWS.