The worst data incident I have worked on produced no alerts. Every job was green for eleven days. The extract ran, the transform ran, the load ran, row counts stayed inside tolerance, and the assistant sitting on top of all of it answered roughly one question in six with something fluent, confident and wrong. The cause was dull: a content management system export had started wrapping each article in its page template, so every chunk carried two hundred words of navigation furniture ahead of anything a user had asked about. By the standards of the pipeline that shipped it, nothing had failed.
That is the part of AI data engineering that is genuinely different, and it is not freshness. Extract, transform, load (ETL) pipelines fail loudly, because the things they produce have types and constraints and a downstream query that breaks when the shape is wrong. An AI pipeline's last transformation is chunking and embedding, and that step turns text into a few hundred floating point numbers that are indistinguishable by inspection from the numbers a good chunk would have produced. A wrong vector is still a valid vector. It has the right dimension, it sits in the same space, it gets retrieved, and it answers.
Everything the pattern literature files under governance, which is to say contracts, quality gates, versioning and lineage, is one response to that single fact. The embedder is a one-way door for your observability. Any property you want to hold about your knowledge base has to be asserted before that step, or carried across it as metadata, because it cannot be recovered afterwards. Once you take that seriously it reorders the list of things worth building, and it demotes the one that teams usually buy first.
Chunking and embedding are one decision, and the vector remembers neither
Chunking gets treated as a preprocessing detail with a sensible default: split on 512 tokens, overlap by 50, move on. The evidence says it is closer to a modelling choice.
Shaukat, Adnan and Kuhn ran 36 chunking methods across six knowledge domains against five embedding models, which is 1,080 configurations scored by graded relevance. Paragraph group chunking reached about 0.459 on normalised discounted cumulative gain at rank 5 (nDCG@5, a retrieval score that rewards putting relevant passages near the top). Fixed-size character chunking came in under 0.244. Close to a factor of two, on the same corpus, the same embedder and the same queries, with nothing changed but where the text was cut.
Two further results from that study matter more than the headline. The ranking of chunking methods stays broadly stable as the embedding model grows, so a good chunker is not a workaround for a weak embedder. But no single method wins everywhere: dynamic token sizing led in biology, physics and health, while paragraph grouping led in legal and mathematical text. Their conclusion is that segmentation effectiveness depends on the structure of the target domain, which in practice means the chunker is tuned against a specific corpus and a specific embedder together.
So the chunker and the embedder are a joint decision, and the artefact they produce records neither of them. A vector in your index does not know what method cut it, at what size, with what overlap, or which model version turned it into numbers. If you want to answer "was this chunk produced by the settings we tuned, or the ones from before the migration", you can only answer it from metadata you chose to write at ingest.
This is where I would spend a data contract, and not where contracts usually get spent. The familiar version guards the producer boundary: a schema, a set of semantics, a promised freshness, versioned so an upstream rename cannot silently change what arrives. That is worth having. But the contract that actually protects a retrieval system is the one covering the transform itself, pinning the chunker settings and the embedding model version, so that a change to either is a deliberate, reviewed, version-bumped event rather than a dependency upgrade somebody merged on a Tuesday.
Freshness is the cheapest guarantee, and the one teams over-buy
The first thing a team builds after a stale answer embarrasses them is a change data capture (CDC) stream. The reasoning is sound as far as it goes. Re-indexing nightly means your index is on average twelve hours behind the source, CDC propagates each change as it commits, and for a catalogue with prices and stock levels the difference between those two is the difference between a usable assistant and a liability.
I have built that pipeline and I would build it again. What I would not do again is treat it as the main lever on answer quality, because the retrieval literature does not support that.
Cuconasu et al. (2024) ran what they describe as the first systematic examination of what a retriever inside a retrieval-augmented generation (RAG) system should actually fetch for the large language model (LLM) downstream of it, varying the relevance of retrieved passages, their position, and how many were included. Their finding, in their words, is that "the retriever's highest-scoring documents that are not directly relevant to the query (e.g., do not contain the answer) negatively impact the effectiveness of the LLM". The near miss is the dangerous case. A passage about the right product from the wrong quarter scores well, reads as relevant, and pulls the model toward a wrong answer harder than an obviously unrelated passage would. That near-miss is a retrieval-time problem as much as an ingest one, and it reframes what retrieval should optimise for.
Their second result is stranger: "adding random documents in the prompt improves the LLM accuracy by up to 35%". I take it as evidence about how much of the outcome is decided by the composition of the retrieved set rather than the recency of its contents, and not as licence to inject noise into production prompts.
Put those two findings together and a five-second-fresh index full of near misses will answer worse than a day-old index that retrieves cleanly. Freshness fixes exactly one failure, which is the source having changed since you last looked. It does nothing about the chunk boundary that severed a clause, the boilerplate that dominated the embedding, or the four almost-right passages crowding out the one right one. Those are the failures I see more often, and CDC does not touch them.
The place CDC earns its cost without argument is where the fact has a short half-life and a user-visible consequence: stock, price, availability, entitlement, anything with a status field. My rule is to ask what the worst answer is if this field is twelve hours old. If the answer is "mildly out of date", nightly batch is fine and the streaming pipeline is a maintenance burden you are choosing. If the answer is "we tell a customer something untrue", stream it.
Batch does not disappear when you add streaming, and its job changes rather than shrinking. It becomes reconciliation: a full pass that catches what the stream dropped, because every CDC pipeline drops things eventually. Schema evolution on the source, a consumer lag that outran retention, a replica failover, a tombstone nobody handled. A nightly job that recomputes what should be there and diffs it against what is there is how you find out. Without it, a stream that has been quietly missing deletes for a month is indistinguishable from one that is working.
Your vector index is a derived store with no version number
Machine learning teams settled one version of this problem a decade ago and gave it a name. Training-serving skew is what happens when the feature a model was trained on and the feature it is served at inference are computed by two different pieces of code that were supposed to agree and drifted. Uber's Michelangelo platform and the feature-store pattern it popularised exist mostly to kill that class of bug, by making one definition the source of both the offline training set and the online serving value.
The discipline that came out of it is worth stating plainly, because it transfers: a derived store is only trustworthy if the code that derives it is versioned and single-sourced, and if you can say which version any given value came from.
A vector index is a derived store with precisely that shape, and almost nobody treats it that way. Which product you run it on is a separate question, and one I have taken up in the four AWS decisions an AI architecture actually turns on; the versioning problem below is the same whichever of them you pick. Every vector in it is a function of three things: the source row at some version, the chunker at some configuration, and the embedding model at some version. Change any one of the three and every vector produced under the old combination is no longer comparable to the ones produced under the new one.
The practical consequence is the one that catches teams mid-migration. You cannot incrementally backfill an embedding model upgrade. Distances between vectors from two different models are not meaningful, so an index half-migrated is not a partially improved index, it is a broken one that still returns results. The only safe move is to build the new index alongside the old, behind a flag, and cut over. None of that is hard engineering, and all of it is expensive in a way nobody budgets for, because "we're upgrading to the newer embedding model" sounds like changing a string in a config file.
Whether that rebuild is cheap or ruinous gets decided much earlier, and this is the case for event sourcing I find actually persuasive. If ingest consumes an immutable log of source changes rather than polling a table for its current state, re-embedding the corpus is replaying the log through a new chunker and a new embedder, at whatever rate you choose, with no load on the system of record. If it does not, a rebuild means going back to the production source and re-reading all of it, which is the conversation with the database owner that nobody wants to have twice a year. Event sourcing is usually sold on audit. The reason I would pay for it is that it turns a full re-embed from an operation into a job you can run on the cheap batch tier, and the pattern above guarantees you will need one.
Two things follow that I now treat as non-negotiable. The embedding model version belongs in the data contract, at the same level of seriousness as a column rename. That applies to the embedder specifically rather than to model choice in general, which sits on a different axis and which I have pulled apart in eight model types and the four design decisions underneath them. And every vector gets written with the triple that produced it: source identifier and row version, chunker configuration, embedder name and version and output dimension. It costs a few bytes per row and it is the difference between a migration you can reason about and one you cannot.
Lineage cannot be reconstructed, and now it has a deadline
"Add lineage later" is the plan that never works, and the reason is the one-way door again. Lineage is not a property you can compute from an index that already exists. It is a set of facts about what happened at write time, and if you did not write them down then, the information is gone. There is no forensic technique that recovers which version of which source row produced a particular vector six months ago.
This used to be an argument about debugging convenience. It is now partly an argument about a deadline, because the European Union's Artificial Intelligence Act (Regulation (EU) 2024/1689) turned several of these habits into obligations for high-risk systems.
Article 12 requires that high-risk AI systems "technically allow for the automatic recording of events (logs) over the lifetime of the system". Article 19 says those logs are kept for a period appropriate to the purpose and "of at least six months, unless provided otherwise in the applicable Union or national law". Article 86 gives an affected person the right to obtain from the deployer "clear and meaningful explanations of the role of the AI system in the decision-making procedure and the main elements of the decision taken", for decisions with legal or similarly significant adverse effects.
Read Article 86 as an engineering requirement rather than a legal one and it says something specific. The right is exercised after the fact, by someone who was affected, about a decision that has already been made, possibly months ago. Whatever you need in order to answer had to have been captured at the moment the decision was taken. A reporting requirement, read carefully, turns out to be a write-path requirement, and that is the part organisations discover late.
Article 10 covers the input side and is blunter than most internal data-quality policies: training, validation and testing datasets "shall be relevant, sufficiently representative, and to the best extent possible, free of errors and complete" for the intended purpose, with an examination for biases likely to affect health, safety or fundamental rights, and measures to mitigate the ones found.
The scope here is narrower than it sounds. These duties attach to high-risk systems as the Act defines them, and a shopping assistant is not a credit decision. Much of what I build falls outside Annex III entirely. The reason the articles are worth reading anyway is that they describe, with more precision than most engineering blogs manage, the minimum set of facts that make an answer explainable: which system was involved, what it was given, when, and on what version of what data. Capturing that is cheap at ingest and impossible afterwards, and the systems I build get reclassified more often than I would like.
The concrete version, for a retrieval system, is short. For each answer, store the chunk identifiers retrieved and their scores, the index version they came from, and the prompt and model version used. For each chunk, store the source identifier and row version, the ingest timestamp, and the transform versions from the section above. Those two records, joined, are the whole chain from an answer back to a row in a source system at a specific point in time. The event log from the previous section earns its keep a second time here. Those two records let you describe what happened; replaying the log lets you rebuild the index as it stood on a given day and reproduce the retrieval instead. Explanation is what Article 86 asks for and the records alone satisfy it. Reproduction is what you want when the explanation is disputed, and only the log gives you that.
Synthetic data: two credible papers pointing opposite ways
Synthetic data is the pattern where the sources actually conflict, and the conflict is worth sitting with rather than splitting the difference.
Shumailov et al., in Nature in 2024, showed that training generative models on generated content "causes irreversible defects in the resulting models, where tails of the original content distribution disappear". They called it model collapse and demonstrated it across language models, variational autoencoders and Gaussian mixture models. The tails going first is the part that should worry anyone building evaluation sets, because rare cases are exactly what you were hoping synthetic data would give you coverage of.
Gerstgrasser et al. (2024) identified an assumption buried in that result. The collapse experiments replace each generation's training data with the previous generation's output. When data accumulates instead, so that synthetic examples are added alongside the original real data rather than in place of it, they show the test error has a finite upper bound independent of the number of iterations, and collapse does not occur.
That is not two papers disagreeing about a fact. It is a finding and a boundary condition on it, and the boundary condition happens to be a data engineering instruction: append, never replace. Keep real data permanently, treat synthetic records as an additive layer, and tag every row with its provenance so that the ratio is a number you can look up rather than a thing you hope about. A corpus that cannot tell you what fraction of itself is generated is a corpus running the replacement experiment by accident.
There is a wrinkle that cuts against reading Shumailov as a reason for caution across the board. Article 10(5) of the same Act permits providers to process special categories of personal data for bias detection only where that detection cannot be done effectively with other data, and names synthetic and anonymised data as the alternatives to try first. A regulator is pointing at synthetic data as the privacy-preserving option, in the same document that demands representative and unbiased datasets. Both things are true, and they apply to different uses. Generating an evaluation set for a rare failure case and recycling model output into training data are not the same activity, and only the second is what collapse is about.
Embeddings are not anonymisation
I have been arguing that the embedder destroys information, which invites a conclusion I want to head off, because I have heard it in a design review and it is wrong.
The conclusion goes: since a vector is opaque, storing vectors instead of text is a privacy control, so the index falls outside the retention rules that govern the source.
Morris, Kuleshov, Shmatikov and Rush (2023) frame embedding inversion as controlled generation, iteratively correcting a candidate text and re-embedding it until it lands near the target point in latent space. Their method "is able to recover 92% of 32-token text inputs exactly", and they demonstrate recovery of full names from a dataset of clinical notes.
So the door is one-way for you and not for an attacker. The embedding discards what your quality gates needed and keeps enough to reconstruct the text, which is the least convenient combination available. Practically: a vector index holding embeddings of personal data is a store of personal data, inherits the retention and deletion obligations of its source, and needs the same access controls. If a source row is deleted under a data subject request, the vectors derived from it have to go too, and the only way to find them is the source identifier you wrote alongside them at ingest. It is the same metadata the migration and lineage arguments already asked for, which is the strongest practical case for writing it.
What I would defend, and what I would not
Three claims here I would defend without hedging. Chunker settings and embedding model version belong in a data contract, because both silently change every vector in the index. Lineage has to be written at ingest, because nothing recovers it later. It is also what makes retrieval usable as the external check that a critic prompt cannot be, since a citation you cannot trace back is not a check at all. Synthetic data should accumulate alongside real data rather than replace it, because the one paper that says collapse is avoidable says so on exactly that condition.
One I hold more loosely. I have argued that freshness is over-bought relative to retrieval quality, and the evidence I lean on is a study of retrieval composition rather than a head-to-head between a stale index and a noisy one. Nobody has run that comparison in a form I would cite. My position comes from the incidents I have actually debugged, where the cause was a chunk boundary or a near-miss passage far more often than a source that had moved. That is a real observation from a small and biased sample.
And one I expect to change. The whole argument rests on the vector being opaque, which is a statement about today's tooling rather than a law of nature. Morris et al. already showed that text is recoverable from short embeddings. If that line of work keeps going and inversion becomes cheap and reliable at production lengths, then some of the checks I have insisted must run upstream could run against the index itself, and the one-way door becomes a turnstile. I would welcome it, and I would still write the metadata, because knowing what a chunk says has never been the same as knowing where it came from.