Advanced Prompt Optimization — Tutorial Notebooks

Four runnable notebooks for Amazon Bedrock Advanced Prompt Optimization (APO): one per evaluator mode, plus local/spot optimization with advpo: tags. All boto3, all explained inline.

Notebook	Mode	Chapters
01_lambda_metric.ipynb	Lambda metric	Text (NESTFUL) + PDF (MMVQA)
02_llm_as_judge.ipynb	LLM-as-Judge	Text (IFBench) + Image (Defactify)
03_steering_criteria.ipynb	Steering criteria	Text (XSum) + Image (MathVista)
04_local_spot_optimization.ipynb	Steering + advpo: tags	Local/spot optimization of a chosen prompt section

Before you start

You need an AWS account configured locally. setup.py discovers everything from your AWS CLI profile — it does not prompt for access keys or take any account info as input. Make sure you have:

• The AWS CLI installed (aws --version works).
• A profile with valid credentials — verify with aws sts get-caller-identity (or aws sts get-caller-identity --profile <name> if you use a named profile). This must return your account ID without error before you run setup.py.
• The AWS principal those credentials belong to must have enough permissions to create an IAM role, an S3 bucket, and a Lambda (or use --skip-perms and have an admin apply the policy setup.py prints).

If aws sts get-caller-identity fails, run aws configure (or aws configure sso, depending on your auth flow) first.

Quick start

# 1. Install Python prerequisites
pip install -r requirements.txt

# 2. Bootstrap your AWS environment (interactive)
python setup.py

# 3. Open any notebook
jupyter notebook 01_lambda_metric.ipynb

That's the whole flow. The setup script reads no static account info — it discovers your account ID, region, bucket, and IAM role from your own AWS profile.

What's in this folder

advanced-prompt-optimization/
├── .env.example                      # Template; the real .env is generated by setup.py
├── README.md                         # This file
├── requirements.txt                  # boto3 1.43.8 + jupyter
├── setup.py                          # Interactive bootstrap (env, bucket, IAM role, mode)
├── utils.py                          # All helper logic (single self-contained module)
├── 01_lambda_metric.ipynb
├── 02_llm_as_judge.ipynb
├── 03_steering_criteria.ipynb
├── 04_local_spot_optimization.ipynb
└── data/                             # Per-example samples, templates, lambda code, multimodal assets
    ├── nestful/, mmvqa/, ifbench/, defactify/, xsum/, mathvista/, spot/
    └── <each>/reference_results.jsonl   # Pre-baked results from a real run (replay mode)

What setup.py does

Run it once per machine / account. It is idempotent — re-running re-confirms each step, defaulting to your prior choices.

1. Verifies AWS credentials via sts:GetCallerIdentity.
2. Auto-detects your account ID.
3. Asks for a region (defaults to your profile's region or us-west-2).
4. Lists existing S3 buckets — pick one or create a new one (default name apo-tutorial-<account>-<region>).
5. Creates the Lambda execution role apo-tutorial-lambda-role if missing.
6. Attaches a narrow inline IAM policy apo-tutorial-permissions to your caller (IAM user or assumed-role) granting only what the notebooks need (see below). If your caller is a federated/SSO session it can't modify, the policy JSON is printed for an admin to apply.
7. Asks for the default notebook mode — live (real APO jobs) or replay (bundled results from a real prior run).
8. Writes ./.env with the resolved values.
9. (live mode only) Deploys the metric Lambdas under data/<example>/lambda_function.py and smoke-tests each with an identity pred/gold pair — fails fast if a Lambda is broken, before any APO job runs.

Non-interactive variants:

python setup.py -y                                       # accept all defaults
python setup.py --region us-west-2 --bucket my-bucket    # mix flags + prompts
python setup.py --mode replay --reuse-env                # flip mode without re-asking
python setup.py --skip-perms                             # don't touch caller IAM

Live mode vs replay mode

Mode	When the notebook calls run_job(...)
Live	Uploads input to S3, submits a real APO job, polls Bedrock for status, downloads the result JSONL. Wall-clock per chapter: 20–50 min.
Replay	Returns the bundled data/<example>/reference_results.jsonl immediately. Full notebook executes in seconds. Great for learning the flow without burning Bedrock tokens.

Default mode is chosen during setup.py and stored in .env as APO_USE_REFERENCE (1 = replay, 0 = live). To temporarily override without re-running setup:

APO_USE_REFERENCE=1 jupyter notebook   # force replay
APO_USE_REFERENCE=0 jupyter notebook   # force live

Shell-set environment variables always win over .env.

Model knobs

Each notebook has a TARGET_MODEL_ID cell near the top (and JUDGE_MODEL_ID in notebook 02). Defaults match the bundled reference run. Edit and re-run to try different target / judge models — the rest of the notebook does not change.

Tearing it all down

When you're done, cleanup.py reverses everything setup.py (and the notebooks) put in your account.

python cleanup.py --dry-run        # preview what would be deleted
python cleanup.py                  # interactive, confirms each step
python cleanup.py -y               # accept all confirmations
python cleanup.py --delete-bucket  # also delete the S3 bucket itself
python cleanup.py --keep-perms     # leave caller IAM policy + Lambda role intact
python cleanup.py --keep-env       # leave the local .env file in place

In order, cleanup removes:

1. APO jobs whose name starts with apo-tutorial- (stops in-flight ones first, then batch-deletes).
2. Lambda functions matching apo-tutorial-* (deployed by setup.py in live mode).
3. S3 objects under the apo-tutorial/ prefix of the bucket from .env.
4. (Only with --delete-bucket) the S3 bucket itself — refuses if it has objects outside the tutorial prefix.
5. The inline IAM policy apo-tutorial-permissions from your caller.
6. The IAM role apo-tutorial-lambda-role (detaches attached policies first).
7. The local .env file (skipped with --keep-env).

Cleanup is idempotent — safe to re-run.

AWS permissions

setup.py attaches a narrow inline policy named apo-tutorial-permissions to your caller. Its scope (after substitution):

Sid	Actions	Resource
AllowAPOAPIs	bedrock:{Create,Get,List,Stop}AdvancedPromptOptimizationJob	*
AllowBucketObjectRW	s3:{Put,Get,Delete}Object	arn:aws:s3:::<your-bucket>/*
AllowBucketLevel	s3:ListBucket, s3:GetBucketLocation	arn:aws:s3:::<your-bucket>
AllowLambdaCRUD	lambda:{Create,Update,Get,Invoke,Delete,AddPermission}Function*	arn:aws:lambda:<region>:<account>:function:apo-tutorial-*
AllowPassLambdaRole	iam:PassRole	apo-tutorial-lambda-role
AllowSTSWhoAmI	sts:GetCallerIdentity	*

Prerequisites for running setup.py itself

setup.py calls APIs that need permissions before the inline policy is in place. You need one of:

• The four AWS-managed policies AmazonBedrockFullAccess + AWSLambda_FullAccess + AmazonS3FullAccess + IAMFullAccess, OR
• An admin to run setup.py --skip-perms, then attach the inline policy by hand from the JSON it prints.

Once the inline policy is attached, you no longer need the broad managed policies for day-to-day notebook usage — the inline scope is enough.