Data Engineering

Data lineage

A record of where each piece of data came from and what transformations it underwent, from source through every pipeline stage to final use.

Each pipeline stage writes metadata linking its outputs to its inputs and to the code and parameters that produced them. The lineage graph is queryable.

Without lineage, questions about what a model was trained on have no answer. A corpus is the output of a pipeline with dozens of stages, and after the fact the provenance is unrecoverable.

The big idea

Any dataset used for training or evaluation. Lineage is cheap to record at pipeline time and effectively impossible to reconstruct later.

After a dataset has passed through crawlers, filters, joins, deduplication, and labeling, the final record no longer explains where it came from or which code changed it. Data lineage is the record of those sources and transformations as the pipeline runs. The consequence is simple and unforgiving: questions about training data can be answered later only if the pipeline wrote the evidence at the time.

When lineage gets asked

The question gets asked in three situations: a regulator asks whether a specific category of data was used, a rights holder asks whether their work was included, and an engineer asks why the model behaves strangely on a domain.

Access control and retention

Access control, personally identifiable information handling and retention are the operational half of the same problem. Access control means a specific dataset has a named owner, an explicit list of who may read it and an audit log of who did. Detecting personal information at petabyte scale uses pattern matching for structured identifiers, named entity recognition for names and addresses, and classifiers for the rest, and none is close to complete.

Retention policies set how long raw and intermediate data survives: keeping everything supports reproducibility and audit, and it also enlarges the set of data that can be subpoenaed, breached, or found to have been retained past a policy limit.

Deletion and the right to erasure

Data inside trained weights is not stored in any locatable place. It is distributed across billions of parameters as a statistical influence. Removing it exactly requires retraining without it.

Every available answer is partial. Machine unlearning approximates the effect of removal with a fine-tuning procedure. Output filtering blocks the model from emitting specific content. Retraining on a schedule with an accumulated deletion list bounds the delay to a training cycle. The European Data Protection Board’s December 2024 opinion on AI models lists output filters and unlearning techniques among measures supervisory authorities may weigh case by case, without saying that any of them satisfies the obligation.

Licensing and provenance

The record of each document's source and the legal basis for using it, tracked per-document through the pipeline.

Each document carries metadata: source URL or identifier, retrieval date, license or claimed basis, and transformations applied. The provenance is queryable and auditable.

Regulators and rights holders ask what was used. Without provenance, the answer is an assertion rather than evidence. License changes or legal challenges require knowing what to remove.

The big idea

Any corpus built from sources with licenses or copyright. Web text, code, books, and media each carry their own risk and increasingly get their own pipeline.

A corpus rarely comes from one legal category. Web pages, code, books, and media carry different permissions and different ways of proving where they came from. Licensing and provenance names the per-document record that keeps those differences attached as data moves through a pipeline. Without it, a later removal request or license change turns into a search through memory and logs rather than a query over the corpus.

License classes

License classes differ sharply across the four main sources, and the differences are usually flattened in practice. Web text is mostly under implicit copyright with no license granted, and the legal argument for using it rests on fair use or its national equivalents. Code carries explicit licenses, and the copyleft family imposes conditions on derivative works that raise a real and unresolved question about whether a model trained on such code inherits them.

Books are unambiguously copyrighted with an established licensing market. Media adds performance and personality rights on top of copyright. The four sources have different risk profiles and are increasingly handled through separate pipelines.

Opt-out signals

Opt-out signals exist, but their enforceability is weak. The robots exclusion standard has no legal force in most jurisdictions and operates at the level of a path rather than a work. Newer proposals add machine-readable statements of training preferences at the site or asset level, and their adoption is partial. None of these binds a third party who obtained the same content from an aggregator, an archive or a dataset someone else published.

The European Union’s text and data mining exception gives a reservation of rights actual legal weight for commercial use, making opt-out enforceable there in a way it is not elsewhere.

Provenance tracking

Provenance tracking is the artifact regulators and courts ask for. It means that for each document in a corpus there is a record of the source, the retrieval date, the license or claimed basis, the transformations applied and the identity of the pipeline run that produced it. That record allows an organization to answer a subpoena with evidence rather than an assertion, to remove a source when a license changes, and to demonstrate that a stated policy was actually enforced. Organizations that skipped it have found themselves unable to prove what they did or did not use.

Deduplication

Removing repeated or near-repeated documents from a corpus to reduce memorization, improve training efficiency, and prevent evaluation contamination.

Exact duplicates are found by hashing. Near-duplicates are found by MinHash with locality-sensitive hashing. Substring duplicates are found by suffix arrays. Semantic duplicates require embedding and nearest-neighbor search.

Duplicated text is memorized preferentially, since every copy trains the model on the same sequence again. Extraction risk rises steeply with duplication count.

The big idea

Any pretraining corpus. Decontamination against benchmarks is a specific application: n-gram matching removes overlap with evaluation sets.

Scraped corpora repeat themselves at every scale: whole pages mirrored across sites, paragraphs copied into otherwise different documents, and benchmark items quoted online. Deduplication is the work of finding those repeats before training treats each copy as fresh evidence. The payoff is practical as much as statistical: fewer redundant tokens to process, less preferential memorization of repeated text, and a lower chance that evaluation items have leaked into training.

Three grades of duplicates

Duplicates come in three grades and each needs a different method. Exact duplicates are caught by hashing the document, which catches less than people expect because trivial differences in whitespace defeat it.

Near-duplicates, meaning documents sharing most of their content, are the large category. The standard method is MinHash with locality-sensitive hashing: represent each document by the set of its overlapping word sequences, hash them to produce a compact signature whose collision probability equals the set overlap, then bucket signatures so that only plausible pairs are compared.

Suffix arrays

Suffix arrays serve the finer-grained version, finding every substring above a length threshold that appears more than once across the entire corpus. The structure is the sorted list of every suffix of the concatenated corpus, and its usefulness is that identical substrings become neighbors once the suffixes are ordered. A single pass down the sorted array, comparing each suffix with the one before it, exposes every repeated span as a long shared prefix, and a threshold on that shared length filters out the ordinary phrases any large text repeats. Construction runs in time close to linear in the corpus, so the method reaches the scale that document-level hashing already handles.

What suffix arrays catch that the other methods miss is a duplicated passage buried inside two documents that are otherwise different. Hashing compares whole documents and MinHash compares their overall overlap, so both slide past a shared paragraph sitting in unlike surroundings, while the suffix array indexes the corpus as one string and finds the repeat wherever it falls. The cost is exactness: a single inserted token breaks the match, though the unbroken stretches on either side are still found, and the array over an entire corpus is large enough that production systems shard it or compress it.

Semantic duplicates

Semantic duplicates, where the content is the same and the wording differs, require embedding and nearest-neighbor search. The boundary between a paraphrase and an independent treatment of the same subject is genuinely unclear.

Why deduplication matters for memorization

Duplicated text is memorized preferentially: each additional copy is another gradient step on the same sequence. Published extraction work finds verbatim reproduction rises steeply with duplication count.

Sequences appearing only a handful of times are rarely extractable while those appearing thousands of times often are. This makes deduplication a privacy and licensing control rather than only an efficiency measure.

Decontamination

Decontamination removes from the training corpus any text overlapping the benchmarks a model will be evaluated on, and it is never complete. The standard approach matches n-grams from benchmark items against the corpus and removes documents that hit.

Decontamination fails in several directions: a benchmark question discussed in paraphrase survives n-gram matching, and benchmarks released after the training cutoff are contaminated for every subsequent model once discussed online.

Reading benchmark scores

A benchmark score should be read as an upper bound whose looseness is unknown. Contamination biases a score upward, since a model that has seen a test item during training can answer it from memory rather than from the ability the benchmark means to measure. The bias is not a guarantee about any single item (a memorized wrong label hurts rather than helps), but across a benchmark it runs one way. Because decontamination is never complete, some unknown fraction of any public benchmark has leaked into the corpus, and there is no direct way to measure that fraction, because the exhaustive n-gram matching that would find it still slides past paraphrase and translation. So the reported number bounds the true capability from above by an amount nobody can state.

The practical reading follows from where a benchmark came from. A public test set that predates a model’s training cutoff and has circulated online for years is the most exposed, and a near-ceiling score on it carries little information. Benchmarks constructed after the cutoff, or held out and never published, are the ones whose scores approximate real generalization, so a wide gap between a model’s public-benchmark numbers and its performance on fresh private evaluations is the signal to trust the second.

Schemas and data contracts

The explicit interface between data producers and consumers: the structure of the data, the obligations around it, and what happens when either side changes.

A schema defines field names, types, and constraints. A contract adds ownership, freshness guarantees, acceptable ranges, null rates, and notice requirements for breaking changes.

Without an explicit contract, the schema is implicit and any change can break consumers silently. Data that parses but means something different is worse than data that fails to parse.

The big idea

Any dataset with more than one consumer. Schema drift degrades models without any code changing. Validation belongs at the boundary.

A schema is an interface between the system that writes data and every system that reads it, and it deserves the same treatment as a published API.

An unversioned schema is an outage waiting for a deploy. A schema break often returns data that parses successfully and means something different, which is worse than an API break returning an error immediately.

Schema evolution

Schema evolution has three categories worth keeping distinct.

• Backward-compatible. New readers process old data. This covers adding an optional field with a default and widening a type.
• Forward-compatible. Old readers process new data. This covers adding a field that old readers ignore and removing an optional one.
• Neither. Renaming a field, narrowing a type, changing units or changing the meaning of an existing value. No automated compatibility check catches a semantic change.

Data contracts

A data contract makes the interface explicit and adds what a schema alone omits: who owns the dataset, how fresh it will be, what the acceptable ranges and null rates are, what notice will be given before a breaking change, and what the consumer is entitled to do when the terms are violated.

A contract converts an implicit dependency into a stated obligation with a name attached. The parts that can be machine-checked should be checked automatically on every run.

Validation and failure

Validation belongs at the boundary, where data enters the system, and a failed batch should stop loudly. The alternative is a silently wrong column propagating through every downstream table, discovered weeks later with no clean way to determine which outputs were affected.

Type mismatches

Type systems for data at rest do not line up with type systems in programming languages. A warehouse NUMERIC with declared precision behaves differently from a language float. Timestamps carry or lack time zones inconsistently. JSON has one number syntax rather than separate integer and floating-point types, but it does not require a parser to use binary64. JavaScript’s usual JSON.parse loses precision for some integers above $2^{53}$; Python’s default decoder preserves integer tokens as arbitrary-precision integers. Exact identifiers need a representation that every consumer preserves, often a string.

Silent schema drift

A model can degrade steadily while every test passes and nobody changes a line of code.

An upstream team adds a category to an enumeration, changes a unit from dollars to cents, starts populating a field that was previously null, or fixes a bug that alters a distribution. Nothing errors. The model receives inputs from a distribution it was not trained on. The defenses are distributional monitoring on features, contracts that require notice, and lineage good enough to trace a degradation back to the upstream change.

Batch and streaming pipelines

The two processing models for moving data: batch processes bounded chunks on a schedule, streaming processes records as they arrive.

Batch: collect data for an interval, transform, write, repeat. Streaming: ingest continuously, process per-record or per-window, emit results with low latency. Orchestration runs stages as a DAG.

Batch amortizes overhead and makes reruns trivial. Streaming delivers results in seconds. The choice is a latency-cost trade: ask what decision the data supports and how stale it can be.

The big idea

Fraud scoring and ad serving need seconds. Financial reporting and model training do not. Exactly-once semantics require idempotent writes or atomic offset commits.

Some data products fail if a record arrives seconds late, and others care more about replaying a clean history than about immediacy. Batch and streaming pipelines are the two processing answers to that trade. Batch gathers a bounded interval, runs it again when logic changes, and fits scheduled orchestration. Streaming handles records as they arrive, buying low latency while making windows, late data, and exactly-once effects part of the design.

ELT versus ETL

Extract-transform-load and extract-load-transform differ in where the transformation happens, and cheap storage moved the industry decisively to the second. Under the older pattern data was transformed in flight; under the newer, raw data lands first and transformations run inside the warehouse. A bug is fixed by rerunning the transformation against data that is still there, and the logic lives in version control as queries.

The medallion architecture is the common name for this layering, a convention from Databricks that labels the tables a lakehouse promotes data through. Bronze holds the raw landed data with enough metadata to replay and audit it, silver holds cleaned and deduplicated records with conformed schemas and business keys, and gold holds the aggregates a dashboard or a feature store reads, so consumers work from stable curated tables instead of repeating the cleaning themselves. The durable part is the separation of raw retention, conformed records, and serving tables, each under its own contract, rather than the bronze, silver, and gold names. The labels guarantee nothing on their own, since a gold table can still encode wrong business logic, and a team that writes a fresh copy at every tier pays storage and synchronization cost without buying trust.

Batch versus streaming

Streaming delivers sub-second latency at the cost of a continuously running system and harder failure semantics: checkpointing, out-of-order arrival, and deciding when a window is complete. Batch amortizes startup overhead, makes reruns trivial, and fits naturally into scheduled orchestration. Most pipelines are batch because most decisions can tolerate minutes or hours of staleness.

Orchestration and reruns

Orchestration expresses a pipeline as a directed acyclic graph of tasks, letting the scheduler run independent branches in parallel and retry a failed node without rerunning its predecessors. Two properties make reruns safe:

• Idempotency. Running a task twice produces the same result as running it once, achieved by writing to a partition keyed on the logical interval.
• Backfill. The pipeline can be run for an arbitrary past interval, which requires every task to take its interval as an explicit parameter rather than reading the current clock.

Pipelines that call now() internally cannot be backfilled, a defect discovered at the worst moment.

Re-execution and backup tasks

MapReduce, the batch framework Jeffrey Dean and Sanjay Ghemawat of Google described in 2004, recovers from a failed machine by running its tasks again elsewhere. That includes map tasks that had already finished, since their output sat on the failed machine’s local disk, while finished reduce tasks stand because their output is already in the replicated Google File System. Rerunning is safe because each attempt writes to private temporary files and only one attempt’s output is ever committed, for a reduce task by an atomic rename. When the map and reduce functions are deterministic, the result matches what a sequential run with no failures would produce.

The same guarantee lets MapReduce work around stragglers, machines that take unusually long on one of the last tasks, such as one whose bad disk slows reads from 30 MB/s to 1 MB/s. Near the end of a job the scheduler starts backup copies of the tasks still in progress and takes whichever copy finishes first, typically for no more than a few percent more compute. With backups disabled, the paper’s terabyte sort took 1,283 seconds, 44 percent longer, because its last five reduce tasks finished 300 seconds after the rest. Any scheduler that reruns a failed step or duplicates a slow one relies on the step being deterministic and idempotent: the same input yields the same output, and committing that output twice leaves one result.

Exactly-once semantics

Exactly-once is a guarantee about effects and not about deliveries. A message can always be delivered more than once because the acknowledgment can be lost. What a system guarantees is that the observable effect occurs once, achieved either by making the effect idempotent or by committing the effect and the offset atomically. The usual production answer is at-least-once delivery plus idempotent writes.

Windowing and late data

Windowing, watermarks, and late-arriving data are where streaming correctness gets hard. Aggregating over a window requires deciding when it is complete, which requires knowing that no further records will arrive. A watermark is a heuristic assertion that all records with event times before a given point have probably arrived.

Records arriving after the watermark are late, and the system must choose among dropping them, emitting a correction, or holding state longer. Event time (when the thing happened) and processing time (when the record reached the system) diverge under backlog or when a mobile client was offline.

Lambda and kappa

The lambda architecture runs a streaming path for low-latency approximate results alongside a batch path for accurate ones. The problem is that the same business logic exists twice and the copies drift. The kappa architecture removes the batch path and treats everything as a stream, reprocessing history by replaying the log.

Feature pipelines

Training-serving skew is the characteristic failure of feature pipelines. A feature computed one way during training and another at inference will differ, and the model was fitted to the first version.

Common causes: an aggregation window defined differently, a default for missing values that differs between paths, a time zone difference, and a training-time feature unavailable at serving time. The structural fix is a single feature definition executed by both paths.