I build agentic systems on AWS (Amazon Web Services) for a living, and the hardest part of starting a new one has nothing to do with models. It is that the service list now runs to roughly ninety entries across vector stores, agent runtimes, model catalogues, evaluation harnesses, registries and governance tooling, and it grows every quarter. Teams read that list, conclude that they have ninety things to evaluate, and spend six weeks producing a comparison spreadsheet that answers none of the questions their architecture actually turns on.
Four decisions determine almost everything else: where the vectors live, whether you rent the agent runtime, what binds you to a model provider, and what silicon runs underneath. Each one has two answers that genuinely exclude each other, each one is expensive to reverse, and everything else on the list follows from them or is a line item. This piece is my attempt to state all four precisely, with the numbers I use when I argue about them.
Where the vectors live
Start here, because it is the decision teams make fastest and regret longest. AWS will sell you at least eight places to put an embedding, and they are not variations on a theme. The prescriptive guidance on choosing a vector database publishes a query-latency row that spans three orders of magnitude.
| Store | Stated query latency | Ideal query pattern |
|---|---|---|
| MemoryDB | Sub-millisecond | Millions of requests per day |
| OpenSearch Service | Sub-10 ms (GPU-accelerated) | High-frequency real-time queries |
| DocumentDB | Millisecond | Document queries with semantic search |
| Aurora and RDS (Relational Database Service) PostgreSQL with pgvector | 10–100 ms | Mixed SQL and vector queries |
| Neptune Analytics | Sub-second | Graph traversals with vector search |
| S3 Vectors, on Simple Storage Service (S3) | Sub-100 ms | Infrequent queries (minutes to hours) |
Read the right-hand column rather than the middle one. The useful question is not how fast a store can answer, it is what query cadence it was built to be economical at. AWS describes S3 Vectors as suited to infrequent queries, minutes to hours, and prices it accordingly: the general-availability announcement claims a reduction in the total cost of storing and querying vectors "by up to 90% when compared to specialized vector database solutions", with up to two billion vectors in a single index. MemoryDB sits at the other end and buys its sub-millisecond response by holding vectors in RAM, which is the most expensive place on the price list to hold anything.
So the decision is not really a database choice. It is a judgement about how often your corpus gets queried relative to how large it is, made before you have any traffic to measure it against. My rule is to estimate queries per document per day, and to treat anything under roughly one as an argument for S3 Vectors whatever the demo latency feels like. A knowledge base that each employee touches twice a week is not a low-latency workload, however much it resembles one in a sprint review.
Two constraints tighten this further, and both are easy to discover late.
The first is that the managed pipeline does not accept every store. Amazon Bedrock Knowledge Bases supports OpenSearch Serverless and managed clusters, Aurora PostgreSQL, Neptune Analytics, S3 Vectors, Pinecone, Redis Enterprise Cloud and MongoDB Atlas. MemoryDB and DocumentDB are absent from the Knowledge Bases prerequisites and from AWS's own table of Knowledge Bases storage options, though both appear in its table of individual vector databases. Choose either and you are writing ingestion, chunking, embedding and retrieval yourself. That is a defensible choice; it is not a defensible accident.
The second is narrower and catches people who optimise storage costs at the embedding layer. Binary vector embeddings work only on OpenSearch Serverless and OpenSearch managed clusters. Deciding to quantise your embeddings therefore rules out most of your storage options in a single step, and the two decisions are usually made by different people a month apart.
Dimensionality closes doors the same way. The ceilings differ by an order of magnitude across the same comparison table: MemoryDB indexes up to 32,768 dimensions, OpenSearch up to 16,000, S3 Vectors up to 4,096, and DocumentDB up to 2,000 once indexed. Most production embedding models land between 256 and 1,536, so this rarely bites. When it does, it bites at the point where someone swaps in a longer embedding to chase a retrieval-quality problem and discovers the store will not take it. Changing the embedding model is more expensive than the store's dimension ceiling suggests, for reasons I set out in why embeddings break your audit trail.
One entry in that table is not a latency choice at all. Neptune Analytics is there for GraphRAG, a form of retrieval-augmented generation (RAG) in which retrieval walks the relationships between chunks rather than ranking them by cosine distance, and the reason to pick it is that your corpus has real structure (entities, citations, org charts, parts hierarchies) that flat similarity search throws away. I have seen it justify itself on exactly one class of problem, which is multi-hop questions over documents that reference each other. If your users ask single-hop questions over independent documents, the graph is a cost with no return, and the latency row will not tell you that. What retrieval quality actually turns on is precision and ordering rather than the store, which no comparison table captures.
Whether you rent the agent runtime
This is the largest architectural commitment available on AWS today, and it is a single yes-or-no question. It is routinely mistaken for six separate ones, because Amazon Bedrock AgentCore ships under six product names that surface in six different parts of the console.
There is one service. It reached general availability in October 2025 with nine components released together (Runtime, Memory, Gateway, Identity, Policy, Observability, Evaluations, Browser and Code Interpreter), all of them gaining virtual private cloud (VPC) support, PrivateLink, CloudFormation and resource tagging in that same release. Policy and Evaluations arrived in December 2025, and AWS Agent Registry, a governed catalogue of agents and tools with an approval workflow, entered preview in April 2026.
You adopt the runtime or you do not, and the consequences arrive as a bundle either way. Adopt it and you get session-isolated serverless hosting, an OAuth vault for refresh tokens, per-tool authorisation boundaries and traces, without building any of them. Decline it and every one of those becomes a workstream: a container platform you scale and patch, a session store with your own summarisation strategy, hand-written tool adapters, token rotation you own, authorisation enforced at every call site, and an OpenTelemetry pipeline with its own collector and retention bill.
I have watched teams take the second column deliberately and be right about it, and I have watched teams take it by default because nobody framed it as a decision. The difference is whether portability was a requirement or an assumption. If you genuinely need to run the same agent on another cloud next year, the second column is the price and it is worth paying. If you do not, you are funding a migration nobody has scheduled.
The thing I would push back on is the idea that this is reversible because the components have individual names. It is not. Identity and Policy are meaningful because the Runtime enforces them at the boundary; lifting them out is not a matter of swapping a dependency.
There is a second argument for adoption that only shows up at a certain scale, and it is the one I now lead with when the organisation has more than a handful of teams shipping agents. Gateway turns existing APIs, Lambda functions and Model Context Protocol (MCP) servers into tools with Identity and Access Management (IAM) or OAuth in front, which makes tool access something you grant rather than something each team implements. Agent Registry adds the layer above it: a private catalogue where agents, tools and skills move through draft, pending approval and published, with IAM deciding who can register and who can discover. Neither matters for one team with three agents. Both matter enormously at the point where two teams have independently wrapped the same internal API and neither knows the other did.
What binds you to a model provider
Two sub-decisions sit here, and they are routinely conflated, because both present themselves as "which model do we use". Neither is that question. Which family of model the workload needs is a separate matter that I take up in eight model types and the four design decisions underneath them. What follows is about who you are bound to once you have chosen one.
The first is Bedrock against SageMaker AI, and the clearest evidence that this is a genuine fork rather than a pair of complementary products is that AWS publishes a dedicated decision guide for it. Services you pick between get decision guides. The distinction the guide draws is about who owns the infrastructure. Bedrock is serverless and priced per token, reached through an API, with "minimal infrastructure management" in its phrase. SageMaker AI is where you go when you need "access to the container or orchestration layer to directly manage cost-latency-throughput and other tradeoffs", via dedicated endpoints, training jobs and HyperPod clusters on Elastic Kubernetes Service (EKS) or Slurm, priced by compute hour. Those are different operating models with different staffing implications. The guide is careful to add that they are "not always mutually exclusive" and describes a progressive path: start on Bedrock, move to SageMaker AI as customisation demands grow, import custom models back into Bedrock for serverless inference.
The second sub-decision is whether your agent code can point anywhere at all, and here the interesting artifact is Strands Agents, which AWS open-sourced in May 2025. It is deliberately not Bedrock-bound. It supports any Bedrock model with tool use and streaming, Anthropic's models through the Anthropic API, Llama models through the Llama API, Ollama for local development, and OpenAI and others through LiteLLM. Several AWS teams run it in production, among them Amazon Q Developer, AWS Glue and VPC Reachability Analyzer.
A cloud vendor shipping an agent software development kit (SDK) that does not require its own inference service is worth pausing on. My reading, and it is a reading rather than something AWS has stated, is that this is a concession at one layer bought with a position at another. The framework layer is where LangChain arrived first and where switching costs are low, so AWS competes there with something free and portable. The layer it wants is underneath: the runtime, the gateway, the identity broker, the trace sink. Strands will happily call Anthropic's API. It will not, by itself, hand you session isolation, a token vault and per-tool policy enforcement. Bedrock's own feature list now advertises "framework-agnostic compatibility with LangGraph, CrewAI, and LlamaIndex", which is not the language of a vendor trying to win at the framework layer.
What tends to settle the Bedrock-versus-SageMaker argument in practice is neither capability nor lock-in, but the cost levers each side exposes. On the Bedrock side the guide claims prompt caching reduces cost by up to 90% and latency by up to 85%, intelligent prompt routing up to 30%, batch inference up to 50% for non-real-time work, and distillation produces models up to 500% faster and 75% cheaper with under 2% accuracy loss on RAG-shaped tasks. Those are levers you pull with configuration, though which of them actually move a bill, and which quietly cancel each other out, is a separate question from how large each discount sounds. On the SageMaker side the equivalents are structural: Savings Plans up to 64% against a one- or three-year commitment, HyperPod cutting training time by up to 40%, endpoints that scale to zero when idle. Those are levers you pull with capacity planning. Which set of levers your organisation can actually operate is a better predictor of the right answer than any feature comparison.
The practical consequence is that model portability and runtime portability are separable, and you should decide them separately. Strands over AgentCore is a coherent position: portable model binding, rented substrate. So is the reverse. What is incoherent is assuming that picking an open SDK has bought you freedom at both layers. The same test applies inside the agent itself, where the question is which steps have an oracle outside the model and which only look like they do.
What runs underneath
This decision is the one most often skipped, and skipping it is how unit economics go wrong, in a way no amount of orchestration downstream recovers.
AWS's own rendering of its AI stack is layered rather than clustered. It starts at the bottom with global infrastructure across 39 Regions, then AI chips (Trainium, Graviton, NVIDIA GPUs), then the data foundation, then training and inference with SageMaker and Bedrock, then agent development with AgentCore, then agentic solutions, with security and governance spanning the full height. The ordering carries a claim worth taking seriously: what sits below constrains what sits above.
Trn3 UltraServers went generally available in December 2025 on AWS's first 3nm AI chip, at 2.52 petaflops in 8-bit floating point (FP8) and 144 GB of HBM3e high-bandwidth memory per chip, up to 144 chips per UltraServer, with a claimed 4.4x the performance of Trn2 and 4x better performance per watt. Treat the first-party benchmark figures with the scepticism any vendor's own numbers deserve; the strategic point survives them intact. Custom silicon is the most differentiated asset AWS has in this market and the hardest thing in its portfolio for a competitor to copy, and the trade against NVIDIA instances is the familiar one: better economics against narrower framework support and a smaller pool of people who have debugged it at three in the morning.
The reason this decision hides is that cost is not a service. It is a property of every other choice on the list, which means it never appears as something to select, only as something to discover in arrears. Two stores sitting next to each other in the same console section can differ by two orders of magnitude on the monthly bill, as the latency table above implies and as I have watched a finance team discover in month four.
The order to take them in
The four are not independent, and taking them in the wrong order produces rework. The sequence I use runs against intuition, because it starts with the decision that feels least urgent.
Silicon first, provisionally at least, because it is the only one of the four that is genuinely hard to change later and because it sets the cost envelope everything else has to fit inside. It is also the cheapest to decide early. For most teams the honest answer is NVIDIA instances on managed inference, and the value of deciding is closing the question rather than leaving it open.
Model binding second, because it constrains the runtime decision and not the other way around. Choosing a portable SDK keeps the runtime question open; choosing deep Bedrock integration substantially answers it.
Runtime third, once you know whether portability is a requirement you are funding or one you are only asserting.
Vectors last, which is where most teams start. It feels foundational because retrieval is the first thing you build, but it is the most reversible of the four: re-embedding a corpus into a different store is a weekend for a small corpus and a scripted backfill for a large one, provided you kept the source documents. The decisions above it are not reversible on that timescale. Building the retrieval layer first is fine; committing to a store before you know your query cadence is what costs money, and the cadence is knowable from the product, not from the infrastructure.
What I am still unsure about
Three things, stated plainly, because the honest version of this argument has edges.
I do not know whether the AgentCore bet pays off for teams smaller than about fifteen engineers. The bundle assumes you would otherwise build all six columns, and a four-person team building one agent will not. For them the second column might be a week of work rather than a quarter, which changes the calculation entirely and I have not run that experiment.
I do not know how durable the Strands position is. A vendor-funded open SDK that is genuinely provider-neutral is an unstable arrangement; the historical pattern is that neutrality narrows once adoption is secured. Nothing in the current behaviour suggests that is happening. I would still not build a compliance story on it.
And the agent layer is young enough that any framework for it, including this one, has a short half-life. AgentCore's component list has grown twice since general availability. The Agent Registry is a governance preview for a population of agents most organisations do not yet operate. The four decisions above are the ones I would defend today, and I would expect to be arguing about a fifth within a year. That fifth is most likely about where agent identity lives when agents start calling each other across organisational boundaries, which is a problem nobody has a good answer to yet.