(@aws-cdk/aws-bedrock-agentcore-alpha): addLambdaTarget() fails due to IAM eventual consistency

[adam-weber]

Describe the bug

gateway.addLambdaTarget() fails with "Gateway execution role lacks permission to invoke Lambda function" even though the IAM policy is correctly created with proper DependsOn ordering.

The BedrockAgentCore service recently added a dry run Lambda invocation during CreateGatewayTarget API. This dry run executes before IAM has propagated the identity-based policy (IAM eventual consistency).

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Library Version

2.233.0-alpha.0

Expected Behavior

addLambdaTarget() should successfully create the gateway target with Lambda permissions.

Current Behavior

Resource handler returned message:

"Gateway execution role lacks permission to invoke Lambda function arn:aws:lambda:us-west-2:XXXX:function:XXXX. Update the permission and retry"

Reproduction Steps

const gateway = new agentcore.Gateway(this, "Gateway", {
  gatewayName: "my-gateway",
  protocolConfiguration: new agentcore.McpProtocolConfiguration({
    supportedVersions: [agentcore.MCPProtocolVersion.MCP_2025_03_26],
  }),
  authorizerConfiguration: agentcore.GatewayAuthorizer.usingCustomJwt({
    discoveryUrl: cognitoDiscoveryUrl,
    allowedClients: [clientId],
  }),
});

const toolLambda = new lambda.Function(this, "ToolLambda", { /* ... */ });

// This fails
gateway.addLambdaTarget("Target", {
  gatewayTargetName: "my-target",
  lambdaFunction: toolLambda,
  toolSchema: agentcore.ToolSchema.fromInline(toolSpec),
});

Possible Solution

LambdaTargetConfiguration.bind() should add a resource-based policy in addition to grantInvoke():

this.lambdaFunction.addPermission('GatewayInvoke', {
  principal: new ServicePrincipal('bedrock-agentcore.amazonaws.com'),
  sourceArn: gateway.gatewayArn,
});

Resource-based policies are validated synchronously by Lambda, avoiding IAM eventual consistency issues.

Additional Information/Context

Confirmed with AgentCore engineering that they recently added dry run Lambda invocation validation to CreateGatewayTarget. The dry run hits IAM before the identity-based policy has propagated.

Workaround: Manually add AWS::Lambda::Permission with bedrock-agentcore.amazonaws.com as principal.

AWS CDK Library version (aws-cdk-lib)

2.233.0

AWS CDK CLI version

2.1029.3

Node.js Version

v23.11.0

OS

macOS

Language

TypeScript

Language Version

No response

Other information

No response

[pahud]

Hi @adam-weber,

Thanks for the detailed bug report. I was able to reproduce this issue.

I also verified that deploying with 2.233.0-alpha.0 today produces the same error, confirming this is a service-side breaking change from BedrockAgentCore (the new dry run Lambda validation) rather than a CDK regression.

The fix is straightforward - LambdaTargetConfiguration.bind() needs to add a resource-based policy via addPermission() in addition to the existing grantInvoke(). Resource-based policies are validated synchronously by Lambda,
avoiding the IAM eventual consistency issue.

Per the AWS documentation on Gateway permissions, resource-based policies are recommended for
Lambda targets. I guess the fix would add:

this.lambdaFunction.addPermission('BedrockAgentCoreInvoke', {
  principal: new ServicePrincipal('bedrock-agentcore.amazonaws.com'),
  sourceArn: gateway.gatewayArn,
});

Confirmed with AgentCore engineering that they recently added dry run Lambda invocation validation to CreateGatewayTarget. The dry run hits IAM before the identity-based policy has propagated.

If you are internal amazon employee, please ping me and let me know relevant ticket ID if any so we can have more context about this issue.

Before we have a PR to get it fixed, a workaround should be like

const gateway = new agentcore.Gateway(this, 'Gateway', { ... });

const toolLambda = new lambda.Function(this, 'ToolLambda', { ... });

// Workaround: Add resource-based policy before addLambdaTarget
toolLambda.addPermission('BedrockAgentCoreInvoke', {
  principal: new iam.ServicePrincipal('bedrock-agentcore.amazonaws.com'),
  sourceArn: gateway.gatewayArn,
});

gateway.addLambdaTarget('Target', {
  gatewayTargetName: 'my-target',
  lambdaFunction: toolLambda,
  toolSchema: agentcore.ToolSchema.fromInline(toolSpec),
});

This should add the resource-based policy that Lambda validates.

Making this a p2 and we welcome PRs.

[pahud]

Thanks to the discussion with @adam-weber and internal feedback, here's a clearer picture of what the fix should look like:

┌─────────────────────────────────────────────────────────────────────────────┐
│                           THE PROBLEM                                       │
└─────────────────────────────────────────────────────────────────────────────┘

  Current flow with grantInvoke() only:

  ┌──────────────────┐     ┌──────────────────┐     ┌──────────────────┐
  │  IAM Policy on   │     │  GatewayTarget   │     │  BedrockAgentCore│
  │  Gateway Role    │     │  Creation        │     │  Dry Run Check   │
  │                  │     │                  │     │                  │
  │  (identity-based)│     │                  │     │                  │
  └────────┬─────────┘     └────────┬─────────┘     └────────┬─────────┘
           │                        │                        │
           ▼                        ▼                        ▼
      ┌─────────┐              ┌─────────┐              ┌─────────┐
      │   IAM   │              │  CFN    │              │ Lambda  │
      │ Service │              │ Creates │              │ Invoke  │
      │         │              │ Target  │              │ Check   │
      └────┬────┘              └────┬────┘              └────┬────┘
           │                        │                        │
           │  ⏱️ Eventual           │                        │
           │  Consistency           │                        │
           │  (seconds-minutes)     │                        │
           ▼                        ▼                        ▼
      ┌─────────┐              ┌─────────┐              ┌─────────┐
      │ Policy  │              │ Triggers│─────────────▶│ Policy  │
      │ Still   │              │ Dry Run │              │ NOT     │
      │ Propagat│              │         │              │ FOUND!  │
      │ -ing... │              │         │              │   ❌    │
      └─────────┘              └─────────┘              └─────────┘

The PR needs to do two things:

┌─────────────────────────────────────────────────────────────────────────────┐
│                           THE SOLUTION                                      │
└─────────────────────────────────────────────────────────────────────────────┘

  1. Add resource-based policy (synchronously validated by Lambda)
  2. Add explicit dependency to ensure correct CloudFormation ordering

  ┌──────────────────┐
  │ Lambda::Permission│  ◀── Step 1: addPermission() creates this
  │ (resource-based) │
  │                  │
  │ Validated sync-  │
  │ ronously by      │
  │ Lambda service   │
  └────────┬─────────┘
           │
           │  DependsOn (explicit)  ◀── Step 2: node.addDependency()
           │
           ▼
  ┌──────────────────┐
  │  GatewayTarget   │
  │                  │
  │  Dry run check   │──────▶ Permission EXISTS ✅
  │  succeeds!       │
  └──────────────────┘

Code Changes

In LambdaTargetConfiguration, modify the bind() method to return the permission resource:

public bind(scope: Construct, gateway: IGateway): TargetConfigurationConfig {
  this.toolSchema.bind(scope);
  this.toolSchema.grantPermissionsToRole(gateway.role);
  this.lambdaFunction.grantInvoke(gateway.role);

  // Add resource-based policy for synchronous validation
  const permission = this.lambdaFunction.addPermission('BedrockAgentCoreInvoke', {
    principal: new ServicePrincipal('bedrock-agentcore.amazonaws.com'),
    sourceArn: gateway.gatewayArn,
  });

  return { 
    bound: true,
    permission,  // Return so GatewayTarget can add dependency
  };
}

Then in the GatewayTarget construct, add the explicit dependency:

const config = props.targetConfiguration.bind(this, gateway);

// Ensure Lambda permission is created before the target
if (config.permission) {
  this.node.addDependency(config.permission);
}

Why Both Are Needed

addPermission() - Creates resource-based policy that Lambda validates synchronously (bypasses IAM eventual consistency)
addDependency() - Ensures CloudFormation creates the permission before the GatewayTarget (guarantees ordering)

Without the dependency, CloudFormation might create resources in parallel, and the dry run could still fail even with the resource-based policy if it hasn't been created yet.

@adam-weber wdyt?

[adam-weber]

Looks good, @pahud.