AI Architecture on AWS — A 2026 Cheat Sheet

Scope: design decisions, tradeoffs, cost engineering, prompt/context engineering, customization, evaluation, and use-case patterns for GenAI and agentic systems on AWS.

Audience: platform/ML engineering leads making build-vs-buy and unit-economics calls.

Currency: written July 2026. Prices and service states are point-in-time — treat every number here as an order-of-magnitude anchor and verify against the AWS pricing pages before it lands in a business case.

⚡ The ten-second version. Start deterministic. Escalate only on evidence. Default to the cheap model and prove you need the expensive one. Cache aggressively and structure prompts to make caching possible. Treat context as the scarce resource in agentic systems — truncate, compact, isolate. Measure cost per completed task. Build the eval set before the demo. Enforce policy outside agent code. Pin model versions and rehearse upgrades. Buy the horizontal stuff; build only where your data makes it differentiated.

0. TL;DR decision tree

Reading each terminal node the long way: write code is cheapest, fastest, and testable — take it whenever you can. Deterministic workflows should retry, observe, and cost-model each node independently. Single agent means AgentCore as the runtime and Strands or LangGraph as the framework — or the AgentCore Harness itself if the loop is standard and you want zero orchestration code. Multi-agent is admissible only with isolated contexts, explicit handoff contracts, per-agent budgets, and a trajectory eval suite.

🧭 Model tier default: start every use case on a small/cheap model. Escalate only where evals prove you need to. The price spread across the Bedrock catalog is roughly two orders of magnitude, and “default to frontier” is the single most expensive habit in production AI.

1. The stack in one page

2. The seven architectural decisions (with tradeoffs)

2.1 Bedrock vs SageMaker endpoints vs self-hosted EKS

⚖️ Practical stance: hybrid is the mature answer. Bedrock runs the agent loop on frontier models; a SageMaker endpoint serves one fine-tuned specialist as a tool. Don’t put the whole program on SageMaker — price both ways first.

2.2 Bedrock serving tiers

📏 Rule: classify every workload as interactive / latency-tolerant / async before you write the client. Flex and Batch are free money on anything that isn’t a human waiting.

2.3 Agent platform: managed vs code-first vs roll-your-own

🧱 The thing about SDKs: an SDK is not a production platform. The delta between “agent works in a notebook” and “agent runs safely for 10k users” is identity, session isolation, memory, policy, observability, and eval — that’s exactly the AgentCore surface area. Building it yourself can be a 2–4 engineer-year commitment that you should make deliberately, not by default.

🔀 Also true: model-driven loops (Strands’ default) trade determinism for flexibility. If you need auditable fixed pipelines — and in regulated environments you often do — impose that discipline yourself with explicit state graphs.

2.4 Retrieval strategy

🗂️ Vector store quick pick:

• OpenSearch Serverless (feature-rich, priciest floor)
• Aurora pgvector (Postgres; transactional consistency)
• S3 Vectors (cold/large/cost-sensitive)
• Neptune Analytics (relationships matter more than similarity).

2.5 Single agent vs multi-agent

Default to one agent with well-designed tools. Multi-agent buys you: context isolation, parallelism, and specialization. It costs you: token multiplication, handoff failure modes, non-reproducible trajectories, and much harder evaluation.

🧩 Adopt multi-agent only when you can point at a specific single-agent failure it fixes — usually context window exhaustion or genuinely parallel subtasks. Then enforce: explicit handoff schemas, per-agent step budgets, isolated contexts (sub-agents return summaries, not transcripts), and one orchestrator that owns the final answer.

2.6 Memory

🔁 Long-term memory is basically a subscription; Set retention policy on day one or you will be paying costs on 18-month-old chat trivia.

2.7 Build vs buy at the app layer

Before architecting a coding assistant, a meeting summarizer, a security triage bot, or a doc-modernization pipeline — check whether Kiro, Quick, AWS DevOps Agent, AWS Transform, or Continuum already does it. Differentiated agents are the ones touching your proprietary data and workflows. Everything else is undifferentiated heavy lifting.

3. The escalation ladder (cost, variance, testability)

🧪 The value of this table isn’t the exact multipliers — it’s the reflex: every level up must be justified by an eval, not by a demo.

4. Cost engineering for agentic systems

4.1 The cost equation

💸 Where the money actually goes: model tokens are typically 70–90% of an agentic bill. AgentCore runtime and long-term memory dominate the remainder. Observability is the classic surprise line — traces on every span of every session, at CloudWatch rates, with no built-in cap.

🎯 Unit of measurement: cost per completed task, not cost per token or per request. An agent that costs 3x per call but succeeds first-try instead of on the third attempt is cheaper. Instrument this from day one; it’s the only number that survives a model swap.

4.2 Levers, ranked by impact

1. Model tiering and routing (biggest lever — 100x+ price spread). Route classification, extraction, routing decisions, and short Q&A to the cheap tier. Reserve frontier models for long-context reasoning and agentic planning. A basic routing layer typically cuts blended cost per request by 40–60%. Bedrock’s Intelligent Prompt Routing does this natively within a model family for ~$1 per 1,000 routing requests — do the arithmetic: at 10k req/day that’s ~$300/mo, worth it if you’re saving 30% of a $5k+ inference bill, pointless on a $400 bill. A hand-rolled or simple OSS router with a small classifier model is often cheaper and always more controllable.

2. Prompt caching (up to ~90% off repeated context). Cache reads cost roughly a tenth of standard input; cache writes carry a premium (~1.25x for the short TTL, ~2x for the long TTL). Break-even is about two cache reads inside the TTL window. For agentic loops and long system prompts this is the difference between a $25 and an $80 session. Requirements to remember: minimum tokens per checkpoint, a small maximum number of checkpoints, and a cap on total cached tokens. Track CacheReadInputTokens / CacheWriteInputTokens in CloudWatch — a cache you think is working and isn’t is a common silent cost leak.

3. Context discipline (the agentic-specific lever). Every agent turn re-sends the accumulated history. Token spend grows roughly quadratically with turn count. Countermeasures:

• Truncate tool results at the source. Return IDs, counts, and a summary — not a 40KB JSON blob the model reads once and never uses again.
• Compaction: summarize old turns into a compact state object at a threshold; keep the last N turns verbatim.
• Offload to filesystem/state store: let the agent write intermediate artifacts and re-read only what it needs.
• Sub-agent isolation: delegated work returns a summary, not its transcript.
• Trim the tool manifest. Every tool definition is input tokens on every single call. Twenty tools you “might need” is a permanent tax.

4. Tool design. Fewer, higher-level, well-described tools beat many primitive ones. One get_customer_360(id) beats six calls the model has to sequence itself — fewer turns, fewer tokens, fewer failure modes, and a much better trajectory.

5. Step and token budgets. Hard caps on max turns, max tool calls, and max tokens per task, with graceful degradation and human escalation on breach. Unbounded agents are unbounded bills. Add per-tenant and per-session budget enforcement if you’re multi-tenant.

6. Tier arbitrage. Flex for latency-tolerant online, Batch for async. Half price, minimal code change.

7. Guardrail and policy placement. Guardrails bill per 1,000 text units per filter enabled. Running every filter on every intermediate agent hop is expensive and usually pointless. Apply full filtering at trust boundaries (user input, final output, external tool egress); apply narrower checks internally. Policy authorization charges land per tool call — that’s fine, but it means chatty tool use has a compliance cost as well as a token cost.

8. Semantic caching. For high-volume, low-diversity queries (FAQ deflection, standard lookups), an embedding-similarity cache in front of the model deflects a meaningful share of traffic to near-zero cost. Guard against stale answers with TTL + freshness keys.

9. Distillation for high-volume narrow tasks. Once a task is stable and you have production traces, distill a frontier model’s behavior into a small one. This is where fine-tuning pays for itself — see §6.

10. Reasoning-token control. Extended thinking is billed output. Set thinking budgets per task class; most classification and extraction needs none.

11. Commitment only after measurement. Provisioned Throughput and Reserved tiers are good deals at high, proven, steady duty cycle and terrible ones otherwise. Measure at least 30 days of real traffic first.

12. Memory and observability hygiene. Tune the session idle timeout (the default keeps idle sessions accumulating). Set long-term memory retention. Sample traces — full-fidelity tracing on 100% of production sessions is a five-figure surprise waiting to happen; 100% sampling in dev, 5–20% plus all-errors in production is a sane default.

4.3 Cost governance

• Application inference profiles per team/product/tenant → cost attribution in Cost Explorer and Cost and Usage Reports (CUR). Do this before launch; retrofitting attribution is miserable.
• Tag everything AgentCore-side (Runtime, Gateway, Memory all support tagging).
• Budget alarms on token metrics, not just dollars — dollars lag by hours.
• Publish a cost-per-task dashboard alongside quality metrics. Teams optimize what they can see.

4.4 Cost anti-patterns

5. Prompt engineering (2026 edition)

5.1 What changed

The discipline moved from wording to context engineering: deciding what information enters the window, in what order, in what form, and what gets evicted. Model capability absorbed most clever-phrasing tricks. What still moves the needle: precise task framing, clear output contracts, good tool descriptions, and disciplined context assembly.

5.2 Structure prompts for caching

Caching is a prefix match. The cache key is the exact bytes of the rendered prompt up to each cache breakpoint, so a single byte changed at position N invalidates every breakpoint at or after N. The request renders in a fixed order — tool definitions, then system prompt, then messages — which means a breakpoint on the last system block caches the tool definitions and the system prompt together.

That gives you the task of ordering the context from most-stable to least-stable, so the cacheable prefix runs as long as possible.

[ STATIC       ]  tool definitions, system prompt, role, policies, output contract
[ SEMI-STATIC  ]  few-shot examples, org glossary, schema definitions
[ CACHE POINT  ]
[ DYNAMIC      ]  retrieved documents, user history, current turn

Where the breakpoint shoud go:

Not everything invalidates everything. Caching is tiered, and a change only invalidates its own tier and below:

The practical read: you can flip tool choice or toggle thinking per request without rebuilding the tools+system cache. Only a tool-set edit or a model swap forces a cold start — which is exactly why “just add one more tool mid-session” and “drop to a cheap model for this one sub-task” cost more than they look. Keep the tool manifest fixed for the life of a conversation, and spawn a sub-agent for the cheap model rather than switching models mid-conversation.

Silent invalidators to grep for. Each one plants a per-request byte in the prefix:

• Request IDs, trace IDs, or UUIDs anywhere near the top of the context
• JSON serialized without sorted keys, or a set iterated into a string — the bytes reorder between runs
• User or session IDs interpolated into the system prompt (this also kills cross-user sharing)
• Conditional system sections (if flag: system += ...) — every flag combination is a distinct prefix
• A tool list assembled per user or per request

Forked calls are the subtle one: a summarization pass, a compaction step, or a sub-agent has to reuse the parent’s system prompt, tool definitions, and model verbatim to hit the parent’s cache. Rebuilding them “equivalently” misses entirely.

Two gotchas worth knowing. A prefix below the minimum cacheable length doesn’t error — it just silently doesn’t cache, reporting zero cache-creation tokens. And Bedrock has no automatic breakpoint placement, so every cache point is one you placed deliberately.

Verify rather than assume. The response usage block reports cache-creation tokens, cache-read tokens, and uncached input tokens separately, and the total prompt is the sum of all three. If cache-read tokens stay at zero across requests you believe share a prefix, an invalidator is at work — diff the rendered bytes of two consecutive requests to find it.

⚠️ Self-inflicted wound: injecting a timestamp, request ID, or user name near the top of the system prompt. That invalidates the entire cache prefix on every call.

5.3 System prompt checklist

• Role and scope — what this agent does and explicitly does not do
• Output contract — exact schema/format, with a worked example
• Tool policy — when to call what, when to call nothing, when to ask
• Escalation criteria — the conditions under which it must hand off to a human
• Uncertainty behavior — what to do when evidence is missing (this is where hallucination is prevented, not in a “don’t hallucinate” instruction)
• Grounding rule — cite sources / answer only from provided context, if applicable
• Untrusted-content markers — retrieved documents and tool outputs go in delimited blocks that the prompt explicitly labels as data, never as instructions

5.4 Tool descriptions are prompts

Most “the agent picked the wrong tool” bugs are description bugs. For each tool specify: what it does, when to use it, when not to use it, parameter semantics with units and formats, and what the return looks like. Budget real effort here — it’s higher leverage than another paragraph of system prompt, and it’s the cheapest place to cut wasted turns.

5.5 Structured output

Use the Converse API’s tool/schema mechanism (or JSON-schema-constrained decoding where the model supports it) rather than asking for JSON in prose and parsing hopefully. Validate against the schema; on failure, retry once with the validation error appended, then fail loudly. Never silently repair.

5.6 Few-shot: when it pays

Worth it for: format conformance, edge-case handling, tone, domain vocabulary, subtle classification boundaries. Not worth it for: general capability (the model has it), factual knowledge (use retrieval), or anything where 8 examples become 4,000 permanent input tokens on every call. If you need >8–10 examples to get quality, that’s a fine-tuning signal.

5.7 Injection defense

Prompt injection is an architecture problem, not a prompt problem. Layer it:

1. Structural — untrusted content in labeled delimiters, never concatenated into instructions
2. Guardrails — prompt-attack and sensitive-information filters at ingress
3. AgentCore Policy — enforced outside agent code, so a compromised prompt can’t talk its way past it
4. Least privilege — tool-level IAM scoping; the agent can only do what its role permits
5. Human-in-the-loop on irreversible actions (payments, deletions, external sends)

5.8 Prompts as code

Version prompts in git, not in a console. Bedrock Prompt Management is useful for discovery and A/B, but the source of truth should be reviewable and diffable. Every prompt change is a deploy and must pass the eval gate. Bedrock’s Prompt Optimizer (simple, ~$0.03/1k tokens; and an advanced iterative optimizer) is a decent starting-point generator — treat its output as a draft to evaluate, not an answer.

5.9 Ten rules

1. Specify the output contract before the task description.
2. Put stable content first; never break the cache prefix casually.
3. Say what to do when uncertain — explicitly.
4. Delimit and label all untrusted content.
5. Prefer one precise instruction over three overlapping ones.
6. Examples beat adjectives.
7. Prune tool manifests ruthlessly.
8. Don’t ask for reasoning you won’t read (it’s billed output).
9. Test prompts against a regression set, not against vibes.
10. If the prompt is >2 pages and still failing, the problem is architecture or data.

6. Fine-tuning and customization

This section refers to the customization techniques by acronym throughout:

The division that matters: SFT and LoRA teach the model what to emit — format, tone, a narrow task — while RFT, DPO, and PPO teach it which answer is better. That is why the ladder puts them on separate rungs.

6.1 The ladder — climb in order, stop when quality is met

🪜 Most teams should stop at step 3 or 4. The majority of “we need to fine-tune” conversations are actually retrieval or context-engineering problems.

6.2 What fine-tuning is and isn’t good at

6.3 The distillation play (highest ROI customization in 2026)

The pattern that consistently pays:

1. Ship on a frontier model. Instrument everything.
2. Accumulate production traces on a stable, high-volume, narrow task.
3. Use those traces (teacher outputs) to distill into a small model.
4. Gate the swap on your eval suite with a defined quality floor.
5. Route that task to the small model; keep frontier as the fallback for low-confidence cases.

Result is typically single-digit-percent quality delta at an order-of-magnitude cost reduction and materially lower latency. It only works if you built the eval suite in step 1.

6.4 AWS mechanics to know

• Bedrock fine-tuning — upload to S3, one API call, PEFT applied automatically for supported families. Simplest path; least control.
• Bedrock Reinforcement Fine-Tuning (RFT) — you supply prompts and a reward function; Bedrock runs the generate-score-train loop, billed hourly (currently on select open-weight models). Good when “correct” is verifiable but hard to demonstrate.
• Bedrock Model Distillation — teacher → student inside Bedrock, including Nova Premier → Nova Pro/Lite/Micro.
• Custom Model Import — bring your own weights for supported architectures. Billed per Custom Model Unit-minute in 5-minute windows, with scale-to-zero after ~5 idle minutes and a cold start of tens of seconds. Excellent for bursty custom models; watch the cold start against your latency SLO.
• SageMaker AI recipes — SFT, DPO, PPO, continued pretraining, knowledge distillation; LoRA or full-rank; full hyperparameter control.
• Nova Forge / Nova Forge SDK — start from Nova checkpoints at pre-/mid-/post-training, mix proprietary data with Amazon-curated data to mitigate catastrophic forgetting, integrate reward functions for RFT, deploy to SageMaker or Bedrock. This is the “build your own frontier-class domain model” tier — real budget, real team, real justification required.

6.5 Data rules of thumb

• Quality over quantity. 500 excellent examples beat 50,000 scraped ones. Nearly every failed fine-tune is a data-quality failure.
• Hold out a genuine test set before you start, ideally split by time or entity, not randomly — random splits leak.
• Watch for catastrophic forgetting: always evaluate general capability alongside the target task, not just the target task.
• Version datasets like code. A model you can’t reproduce is a liability.
• Budget the total cost: training + storage per model per month + inference hosting. Custom-model hosting economics, not training cost, are what usually kill the business case.

6.6 Hard gate

🛑 Never promote a customized model without: a holdout eval showing lift on the target task, a regression eval showing no meaningful loss on general capability, a safety eval, a cost-per-task comparison, and a documented rollback to the base model.

7. Evaluations

7.1 Three layers — you need all three

🪤 Output-only evaluation is the classic trap: an agent can call every tool correctly and still fail the task, or produce the right answer via a path you’d never approve.

7.2 Eval type matrix

7.3 LLM-as-judge discipline (where eval programs fail)

• Rubric, not vibes. Explicit criteria at each score level, written down, versioned.
• Use a 0–5 scale. Recent work finds it aligns best with human judgment; 10-point scales add noise without precision. Binary is fine for genuinely binary criteria.
• Calibrate against humans. Label a few hundred examples yourself, measure judge-human agreement (Cohen’s kappa or correlation), and re-calibrate when you change judge model, rubric, or prompt. An uncalibrated judge is an unlabeled ruler.
• Know the biases: position bias in pairwise comparison (randomize order), verbosity bias (longer ≠ better), self-preference (models favor their own family’s output — consider a different family as judge).
• ~500 cases before you trust aggregate metrics. Below that, you’re reading noise.
• Mine production traces for your eval set. Synthetic suites have poor ecological validity — real traffic finds failure modes you’d never invent.
• Ensemble selectively. Full-stream multi-judge is cost-prohibitive; ensemble only on flagged cases (low scores, escalations, novel patterns).
• Judge with a cheaper model where it holds up. Verification is usually easier than generation — validate the downgrade, then bank the savings.
• Treat judge scores as a measurement instrument with known error bars, not an oracle.

7.4 RAG-specific

Evaluate retrieval and generation separately or you’ll chase the wrong bug.

• Retrieval: recall@k, precision@k, MRR, context relevance
• Generation: faithfulness/groundedness (Bedrock’s contextual grounding check is a cheap production-side version), answer relevance, citation accuracy
• Rule: if faithfulness is high and answers are still wrong, it’s a retrieval problem.

7.5 AWS tooling

• AgentCore Evaluations (GA March 2026) — managed, LLM-judge-based, ~13 built-in evaluators plus custom rubrics with your choice of Bedrock judge model and rating scale. Ingests traces via OpenTelemetry/OpenInference from Strands, LangGraph, and others. Same infrastructure serves both on-demand (dev) and online (production monitoring) evaluation — that’s the important part: one system across the lifecycle.
• AgentCore Optimization — failure/intent/trajectory insights across sessions, plus recommendations and A/B testing to close the loop from production traces back into agent changes.
• Bedrock Model Evaluation — model-vs-model comparison, RAG evaluation, human workflows.
• CloudWatch GenAI observability + SageMaker MLflow for experiment tracking.
• Third-party (Langfuse, LangSmith, Deepchecks, RAGAS, DeepEval) all integrate via OTel if you want portability.

7.6 Eval in CI/CD

PR (prompt / tool / model / config change)
  → smoke set (~50 cases, deterministic assertions)      [minutes, blocks merge]
  → regression set (300–1000 cases, judge + assertions)  [pre-merge or nightly]
  → safety + adversarial suite                           [blocks release]
  → canary (1–5% traffic, online eval)                   [auto-rollback on threshold]
  → full production monitoring + weekly human sample     [feeds the eval set]

📌 Non-negotiables: pin model versions (never point production at a floating alias), define a quality floor per gate, and rerun the full suite on every model upgrade. Bedrock’s model lifecycle marks models Active / Legacy / EOL with roughly a six-month window after Legacy — build the upgrade rehearsal into your quarterly cadence rather than discovering it from a deprecation email.

7.7 Keep eval costs sane

Batch tier for offline runs · tiered suites (smoke → regression → full) · cheaper judge models where validated · sample production evaluation rather than scoring 100% · cache eval-set inputs.

8. Business use cases → patterns

⚠️ This is NOT an exhaustive or authorative list, but rather guidelines for architectures for common tasks - your mileage may vary

8.1 Quick reference

8.2 Patterns worth spelling out

Customer support (highest-volume enterprise pattern). Do: intent classification on a cheap model first → deterministic routing → RAG for informational intents → agent with tools only for transactional intents → guardrails on both ends → explicit escalation path with full context handoff. Don’t: put one agent in front of everything. Most support traffic is classification plus lookup, and paying frontier-model agentic prices for “where’s my order” is how deflection programs fail their business case. Measure containment and CSAT together — containment alone incentivizes trapping angry customers.

Intelligent document processing. Do: Bedrock Data Automation for parsing/extraction, schema validation as code, confidence thresholds routing low-confidence pages to human review (A2I), Batch tier for the whole pipeline, and per-field accuracy tracking. Don’t: send a 60-page PDF to a frontier model and ask for JSON. Per-page pricing on BDA is usually far cheaper and far more accurate than token-based parsing, and it gives you page-level auditability.

Enterprise knowledge assistant. The hard part isn’t retrieval, it’s permissions and authority. Two documents disagree; which is canonical? Which user may see which source? Solve access control at retrieval time (filter before ranking, never after), and prefer a governed knowledge layer over a raw vector index once you’re spanning more than a couple of sources.

Text-to-SQL. Always: read-only credentials, row limits, query timeouts, schema + sample rows in context, and a verification pass that re-reads the query against the question. Return the SQL alongside the answer — analysts trust what they can check. Never let generated SQL write.

Regulated-industry agents (finance, healthcare, gov). Non-negotiables: deterministic control flow wherever a regulator will ask “why did it do that”; complete trace retention; policy enforced outside agent code (AgentCore Policy) so prompt compromise can’t bypass it; human sign-off on any irreversible or customer-visible action; model version pinning with documented change control; model risk management artifacts (intended use, limitations, validation evidence, monitoring plan) treated as launch blockers, not paperwork. Bedrock Automated Reasoning checks are worth evaluating where you need provable policy conformance rather than probabilistic filtering.

9. Reference architectures

A. Managed RAG assistant (fastest defensible path)

Weeks to production. Cost dominated by tokens; caching and tiering are the levers.

B. Production agent

The four attached surfaces, spelled out: Gateway fronts the tools (Lambda, MCP servers, internal APIs). Memory holds short-term events plus long-term records under an explicit retention policy. KB / AWS Context supplies grounding. Policy + Guardrails are enforced outside agent code, so a compromised prompt can’t talk its way past them.

Models: cheap router/classifier → mid-tier worker → frontier escalation. Budgets: max turns, max tokens, per-tenant spend caps.

C. High-volume batch pipeline

10. Cross-cutting anti-patterns

1. Agent-first design. Reach for the simplest level of the ladder that passes evals.
2. No evals before scale. If you can’t measure quality, you can’t upgrade models, tune prompts, or defend the system. Build the eval set before the demo, not after the incident.
3. Floating model aliases in production. Pin versions. Test upgrades deliberately.
4. Prompts in a console, not in git.
5. Retrieval as a dumping ground. Chunking, freshness, and permissions decide RAG quality far more than the model does.
6. Treating guardrails as the safety strategy. Guardrails filter content; policy, IAM scoping, and human-in-the-loop prevent actions.
7. Ignoring the trace. No OTel instrumentation = no evaluation, no debugging, no optimization loop.
8. Optimizing token price while ignoring task success rate.
9. Building what AWS ships. Check Kiro/Quick/Transform/DevOps Agent before building horizontal tooling.
10. Skipping the migration rehearsal. Models go Legacy on a schedule. Practice the swap quarterly.

11. 30/60/90 for a new AI platform capability

Days 0–30 — prove it’s worth doing. Pick one use case with a measurable business metric · assemble 100–300 real eval cases from production data · build the thinnest thing that works (level 1 or 2 of the ladder) · instrument traces and cost-per-task from the first commit · establish the quality floor.

Days 31–60 — make it safe and cheap. Add guardrails at trust boundaries · implement caching and model tiering · grow the eval set toward ~500 cases and calibrate the judge against human labels · wire evals into CI · set budgets, alarms, and cost attribution · run the adversarial suite.

Days 61–90 — make it operable. Canary with online evaluation and auto-rollback · publish a cost-per-task and quality dashboard · document model risk artifacts · rehearse a model version upgrade end-to-end · decide from real traces whether distillation is now worth it · write the runbook.

12. Numbers to know (verify before quoting)