Judgment and Failed Bets

How the field decides under uncertainty, and what research taste actually is

Research taste

The ability to pick problems worth working on before the answer is known.

Estimate the information value of each possible outcome and whether the knowledge would change what anyone does next. Write down what each result would imply before starting.

Probability of publication selects against genuinely uncertain questions. A problem whose positive confirms what everyone assumed has low information value regardless of difficulty.

A team can be technically strong and still spend a year answering a question whose answer changes nobody’s plan. Doing the work is only part of the problem; the harder part is choosing a question whose possible outcomes would teach the field something useful. Research taste is that selection judgment under uncertainty. It forces the work to earn its budget before it begins, because a hard problem with an unsurprising answer can consume the same effort as one that changes the next decision.

Information value

Choosing a problem by the information it returns is what separates taste from ambition. Ask what will be known afterward that is not known now, and whether it changes what anyone does next. A problem whose positive result confirms the consensus has low information value however hard it is; one whose negative result is equally informative pays out either way.

Most researchers optimize instead for the probability of a publishable positive result, which selects against exactly the questions where the answer is genuinely unknown. The practical form of the discipline is to write down, before beginning, what each possible outcome would imply, and to abandon the problem if the answers are uninteresting under every branch.

Tractability

Importance and tractability must both hold, and most work fails the second. Importance without tractability produces a project that runs for years and returns nothing, and the failure mode is seductive because the problem statement sounds impressive to everyone including the person working on it. Tractability without importance produces steady output that nobody builds on, and this failure is more common because the incentives reward it.

The test for tractability is whether measurable partial progress is achievable within a few months. A problem with no intermediate milestone gives no signal until the entire budget is spent. Peter Medawar called science the art of the soluble; the sharper version is recognizing which important problems have just become soluble, since tractability moves as tools and data arrive and importance mostly does not.

Unexamined assumptions

The highest-value problems sit in what a field has stopped questioning. Assumptions justified when adopted go unrevisited after conditions change, invisible because everyone shares them. The signals: a technique whose original justification nobody can state, a benchmark whose construction nobody has examined recently, a phrase that appears in introductions without a citation.

The productive move is to check whether the original argument still holds under current conditions, which frequently it does not, since the constraint that motivated it has changed by orders of magnitude. Several results on this page have that shape, including the compute-optimal correction, which revisited a fit everyone had adopted, and the emergent abilities critique, which revisited a scoring rule nobody had examined.

Transmission

Taste transmits by apprenticeship rather than publication, which is a bottleneck rather than a romantic claim. Papers record what worked and are silent on the twenty directions abandoned and the reasons. That information exists only in the working process, so it transfers by watching someone choose and being corrected.

Taste therefore concentrates in a few groups and spreads at the speed of people moving between them, far slower than results spread. A field expanding faster than it can train people accumulates capacity to execute without capacity to choose, which shows up as competent work on questions that did not need answering.

Resulting

Judging a decision by its outcome rather than by the information available when it was made.

Outcomes are observed, reasoning is reconstructed in light of them, and the reconstruction is contaminated by knowing the answer.

Variance ensures good decisions can lose and bad ones can win over any short run. A single draw does not identify the distribution it came from.

A review after a launch, hire, or experiment usually starts with the visible result, because that is the part everyone can see. The trouble is that the visible result includes luck, timing, and noise that were not available to the person making the call. Annie Duke calls the habit of grading a decision by its outcome resulting. It makes learning backward: lucky errors get copied, and sound bets that happened to lose get removed from the repertoire.

Separating decision quality from outcome quality produces four cases, and only two of them get judged correctly.

Good decision, good outcome. Rewarded, and the reasoning is treated as validated, often without anyone checking whether it was.
Bad decision, bad outcome. Punished, and the lesson is usually learned.
Good decision, bad outcome. Treated as a mistake, so the process that produced it is abandoned even though it was sound, and the person who made it stops making that class of bet.
Bad decision, good outcome. Treated as insight, so a flawed process is reinforced and repeated until the variance catches up with it.

Variance

A good decision can lose and a bad one can win over any short run, because a single draw does not identify the distribution. A bet with a 90 percent chance fails one time in ten, and the failure says nothing about whether taking it was correct. Distinguishing a good process from a lucky one takes many observations.

In any field with high variance and slow feedback, an individual’s outcome record over a few years carries little information about their judgment, and that is the record hiring and funding decisions use. Outcomes are the noisy signal and reasoning is the observable one, so a reviewer with both should weight the reasoning more heavily than instinct suggests.

Decision journals

Decision journals record reasoning before the outcome arrives, which makes them the only reliable defense against hindsight bias. An entry states what is being decided, what is believed, what probability the main outcomes carry, what would show the decision was wrong, and when it will be reviewed. Memory of prior belief is reconstructed rather than retrieved, and reconstruction is contaminated by knowing the answer.

A written record is not subject to that revision. The second benefit is calibration: with enough entries, the rate at which events assigned 70 percent actually occurred becomes measurable, which is the only way to find out whether stated confidence means anything. The cost is a few minutes per decision, and the practice is rare anyway because the record is uncomfortable to read.

Postmortems

Postmortems on process rather than result apply the same discipline to engineering. Asking what broke fixes one failure; asking what in the decision process permitted it prevents a class. Judging a deployment by whether it caused an outage punishes the engineer who shipped a tested change that failed unforeseeably and rewards the one who shipped an untested change that held.

Both signals are wrong and both change behavior. The correct question is whether the checks were appropriate for the risk, given what was knowable beforehand, and a review that reaches “the process was right and the outcome was bad” has produced a valid finding rather than a failure to find one. Recognizing that as an acceptable conclusion is what makes the process reviewable at all.

Expected value

The probability-weighted average of outcomes.

Weight each outcome by its probability and sum. The formula is the limit of what repeated independent trials converge to.

Maximizing it is correct only when the bet repeats enough for the average to realize and when gains and losses of equal size have equal value. When either fails, the number remains computable but stops being the right guide.

A decision with several possible payoffs needs a single summary, or every comparison becomes a pile of scenarios. Expected value is the probability-weighted average of those payoffs. It is a clean number when the same kind of bet repeats and no single loss changes the decision maker’s ability to keep playing, but it can give dangerous guidance when survival, constraints, or nonlinear utility are the real issue.

E[X] = \sum_{i} p_i x_i

Each outcome weighted by how likely it is. On a single bet the number can mislead because it prices outcomes linearly: a loss that ends the enterprise counts the same per dollar as one that stings, and expectation has no way to say otherwise. What repairs it is a utility that bends, or a constraint that refuses ruin outright.

symbol	meaning
$E[X]$	the expected value of the quantity X
$i$	an index over the possible outcomes
$p_i$	the probability of outcome i
$x_i$	the value of outcome i

When expectation is the right guide

The calculation weights each outcome by its probability and sums, and it is right under two conditions: the bet repeats enough times for the average to be realized, and the stakes are small relative to total resources. When either fails the number is still computable and no longer the right guide.

Ruin risk

A positive expectation is not sufficient when a loss ends participation, because the arithmetic assumes you are there for the next trial. A bet that triples the stake with probability one half and loses everything otherwise returns fifty percent per trial, and repeated at full stake it goes to zero with probability approaching one. The average across parallel players still equals the expectation, and the typical player has nothing.

This is the distinction between the ensemble average and the time average, and it is why the correct rule under repetition with reinvestment maximizes the expected logarithm of wealth rather than expected wealth, which is what the Kelly criterion formalizes. The engineering version of the same point is that a system optimizing expected throughput while accepting a small chance of unrecoverable data loss is not making a favorable trade.

The gap between the averages is widest exactly where the upside is largest. Any strategy whose per-period chance of total loss stays above some fixed level reaches total loss with probability approaching one over enough periods, regardless of expectation, because the survival probability is a product of terms bounded below one.

Structural defenses

The defenses are structural: size positions so no single loss is terminal, cap exposure to correlated risks, and treat removal from the game as a constraint rather than one outcome in the sum. Correlation is the part usually missed, since bets that are independently survivable can be jointly fatal, and the events causing simultaneous failure are the ones nobody modeled. This is why a portfolio’s risk is not the sum of its positions’ risks.

Research portfolios

The same reasoning applies to research portfolios, where most projects are expected to fail by design. Payoffs are heavily skewed, so a few results carry most of the value and the median project returns little, which means judging the aggregate rather than the hit rate. A group with a high success rate is probably choosing problems that were not uncertain.

The constraint that makes this work is that individual failures must be survivable and cheap to detect, which argues for staged commitment: small budgets until a specific signal appears, larger ones after. The ruin condition applies here too, since an organization that bets its existence on a single research direction is making a bet whose expectation is irrelevant, and the correct response is to bound the exposure rather than to improve the estimate.

Position

An informational advantage from the order of moves rather than from the content of what is played.

A player acting later observes earlier actions before committing, so their decision incorporates information those players lacked. The option to observe before committing has value.

Information revealed by earlier moves narrows uncertainty. The same hand is profitable in one seat and unprofitable in another because of this asymmetry alone.

Two people can face the same underlying opportunity and rationally make different moves because one has already watched the other act. The informational edge comes from sequence rather than superior private facts. Position names that edge. It turns timing into part of the payoff calculation, so leading, waiting, and fast-following are not temperaments but choices about what information is worth paying for.

Poker analogy

In the poker sense, acting last is worth more than the cards justify, and the reason is purely informational. A player in late position has observed every earlier action before committing chips, so their decision incorporates information those players did not have when they acted. This permits folding marginal holdings against demonstrated strength, extracting value against demonstrated weakness, and controlling the size of the pot, none of which is available to someone acting first.

The correct action depends on a structural feature of the situation rather than on the cards at all, and this value should be counted when deciding whether to move.

Fast-following

Following a published result is cheaper than producing it, sometimes by a large factor. The follower knows the approach works, roughly what it cost, and which of many directions was productive, so search is replaced by execution. Fast-following is a coherent strategy rather than a failure of ambition, and the correct one where the advantage is execution and distribution rather than discovery.

The follower is structurally behind on timing, cannot differentiate on the thing being copied, and gains no capability to produce the next result, so a group that only follows becomes dependent on others continuing to lead. The strategic question is not which mode is better but which problems justify paying the leader’s premium.

Leading cost

Being early carries an information cost that is often mistaken for a mistake. The first group to attempt something pays for the dead ends, and the record of those dead ends does not appear in anyone’s publication list, so from the outside the early group looks slower and less productive than the followers who skipped them. This is the resulting error from the earlier section applied to timing.

A group that spent two years establishing that three of four approaches do not work has produced real information, and if that information is not published it is captured entirely by whoever inherits the people. The corollary for reading the field is that an approach abandoned early is not thereby refuted, since it may have been abandoned for reasons that no longer hold, and checking why is frequently cheaper than assuming the verdict was permanent.

Skill

The skill is choosing per problem rather than as an identity. The considerations are whether the advantage is in discovery or execution, whether the result stays defensible once published, how long the lead survives replication, and whether the capability built by leading is worth more than the result.

The common error is deciding by disposition. A group that thinks of itself as a research organization leads everywhere, including where the leader’s premium buys nothing; a product organization follows everywhere, including the one problem where being first would have mattered. The disciplined version is an allocation: lead where the capability compounds, follow on the rest, revisit as the field moves.

Sample size and variance

The number of trials needed before a difference is distinguishable from noise.

Standard error falls with the square root of count, so halving a confidence interval requires four times the data. For a proportion, the margin is roughly one over twice the square root of sample size.

Precision is bought at a worsening exchange rate. On 500 items a single-score margin is roughly 4 points, so 71 versus 73 is indistinguishable.

A leaderboard often invites a simple reading: the higher score is the better system. With small evaluation sets, that reading can be mostly noise. Sample size and variance are the check on whether the observed gap is large enough to distinguish from ordinary fluctuation. The consequence is practical and sometimes deflating: collecting or running more examples buys precision slowly, so many reported one-point gains never become evidence.

Square-root relationship

The error on an estimate falls only with the square root of the count, which is the fact that makes evaluation expensive. Four times as many observations halve the interval width; reducing it by a factor of ten requires a hundred times as many. Precision is bought at a rapidly worsening exchange rate.

For a proportion the standard error is at most one over twice the square root of the sample size, and a 95 percent margin is about twice that: 100 items yields a margin around 10 points, 1,000 items about 3, and 10,000 items about 1. The interval on a difference between two systems on the same items narrows when they are correlated, so paired comparison is the right test.

Noise in small samples

A benchmark difference of a point or two on a few hundred items is usually noise, and reporting it as an improvement is the most common error in this area. On 500 items the 95 percent margin on a single score is roughly 4 points, which is why 71 and 73 get called indistinguishable. That reasoning is looser than it looks: overlapping intervals on two separate scores do not settle a comparison between systems run on the same items, where what matters is the count of items they answered differently and which way those went. Two systems agreeing on 480 items and splitting the other 20 fifteen to five are distinguishable at 71 and 73; two that disagree on a hundred items, splitting them fifty-five to forty-five, are not. Compare the pairs, not the intervals, and treat a leaderboard ordering built from unpaired margins as largely an ordering of the noise.

The problem compounds when many systems are compared, since the maximum of many noisy estimates is biased upward and the apparent leader is partly whoever drew favorably. It compounds again under repeated community use, because choices made in response to the score fit the specific items. Report the interval, use a paired test on identical items, and treat differences smaller than the interval as unresolved rather than small.

Seed variance

Variance across random seeds is often larger than the claimed effect and is the source most frequently left unmeasured. Two runs differing only in the seed differ in initialization, data order and every stochastic component, and the spread is several points on many tasks. Comparing two methods at one run each measures the seed as much as the method.

This permits seed selection: the best of several runs reported for the proposal against a single run for the baseline, producing a real-looking improvement from nothing. The remedy is rarely applied. Run each configuration several times, report mean and spread, and state the run count. A result without a seed count and a spread is not interpretable.

Reporting intervals

Confidence intervals on evaluations are the standard, and their absence is a red flag rather than a stylistic omission. Every benchmark score is an estimate from a finite sample of items and runs, so reporting the point estimate alone presents an estimate as a measurement. The cost objection is real for training runs and not for evaluation, since bootstrapping over items costs nothing beyond predictions already computed.

Two distinct sources need reporting, the variation across items and the variation across runs, and the second is usually the larger and the one omitted. When reading a result, the useful sequence is to check the item count, check the run count, compute the approximate margin, and compare it to the claimed effect, which resolves a surprising fraction of reported improvements without any further analysis.

Tilt

Decision quality degrading under emotional load after a loss.

Escalation of commitment following a sunk cost: a decision maker who has invested in a course of action becomes more willing to invest further as evidence against it accumulates.

Abandoning converts the prior investment into an acknowledged loss; continuing preserves the possibility it was not wasted. The discomfort of realizing a loss makes the irrelevant prior investment feel relevant.

After a project has burned time, money, or reputation, the next decision is no longer made in the same emotional state as the first one. The prior investment begins to feel like a reason to continue even though it cannot improve the odds from here. Tilt is that degradation of judgment under the load of a realized or impending loss. Without a rule set before the pain arrives, the person closest to the evidence is often the least able to stop.

Amplifying features

Two features amplify the pattern: personal responsibility for the original choice, and public visibility of that choice. An outside reviewer with no stake reaches the correct conclusion quickly and is resented for it.

Recognizable forms

Recognizable forms include doubling down on an approach after it fails to reproduce. The pattern is a result that worked once and does not work again, followed by a search for the configuration that recovers it rather than an assessment of whether the original was real. Each additional attempt is individually reasonable, since one more hyperparameter sweep is cheap, and the sequence has no natural stopping point because failure is always attributable to the search rather than to the hypothesis.

Related forms are increasing the compute budget on a method whose scaling curve has flattened, adding components to an architecture that is not working rather than removing the ones that were never ablated, and continuing to defend a benchmark result after the evaluation has been shown contaminated. The common signature is that the cost of continuing is compared against the value of the original goal rather than against the value of the next best use of the same resources.

Defenses

Precommitment and stopping rules are the defense, and they must be set before the loss, which is the only time judgment is uncontaminated. A usable rule states what result would end the project, what budget precedes revisiting the decision, and who makes the call, the last because the person running the work should not decide when to stop. What works: a written kill criterion agreed at the start, a review at fixed spend defaulting to stop, and a reviewer with no stake in the outcome.

The design principle is that the default is stopping rather than continuing, since a rule requiring an affirmative decision to stop will not be invoked. Clinical trials institutionalized this with predefined stopping boundaries for exactly this reason, and the analogy is close enough to borrow the structure.

Competitive pressure

Competitive pressure makes this more common rather than less, the opposite of the usual assumption. When results arrive weekly, abandoning a direction risks a competitor making it work next month, so the option value of continuing is inflated by the pace. The same pace shortens the evaluation interval, so the decision is made on less evidence and is more susceptible to whatever is pushing on it.

Public commitment compounds it: an approach announced in a talk or a funding round is harder to abandon than one nobody knows about, which argues for keeping strategic bets quiet until they survive review. The structural response is to make abandoning a direction neutral or rewarded, which is difficult and the only thing that reliably works, since individual discipline is otherwise overcoming an incentive the organization created.

Bitter lesson

Rich Sutton's 2019 observation that general methods scaling with compute have repeatedly beaten methods encoding human domain knowledge.

Computation per unit cost falls exponentially over decades, so a method whose performance improves with available compute eventually overtakes one whose performance is bounded by embedded knowledge.

The advantage of embedded knowledge decays predictably while the general method's advantage grows. The bet on domain knowledge has a known expiry date.

A field can spend years encoding expert knowledge into a system and then watch a blunter method overtake it once more computation is available. The bitter lesson is the pattern Rich Sutton drew from those reversals. It does not say knowledge never helps; it says methods that improve as compute grows have a long-run advantage over methods whose gains are capped by what humans chose to build in.

Historical cases

The historical cases Sutton cites cover chess, Go, speech recognition, and computer vision, and each follows the same arc. In chess, decades of work encoding positional understanding was beaten in 1997 by deep search on specialized hardware, and the researchers who had built the knowledge-based systems regarded the result as unsatisfying rather than instructive. In Go, handcrafted pattern libraries and heuristic evaluation gave way to Monte Carlo tree search and then to self-play with learned evaluation.

In speech recognition, systems built on phonemes, articulatory features, and expert linguistic knowledge lost to statistical methods and then to end-to-end learned models. In vision, edge detectors and engineered feature descriptors representing years of accumulated design were replaced by learned convolutional features. Sutton’s observation is that in every case the human-knowledge approach led initially, that the general method won once compute grew, and that the community resisted the conclusion each time.

Qualifications

The claim is about the long run rather than about any particular year, and this qualification carries most of the weight. The two categories of method Sutton identifies as scaling arbitrarily are search and learning, which is a narrower claim than the one usually attributed to him.

A second qualification is economic. A general method that eventually wins may need resources available to a handful of organizations, so for everyone else the question is which method wins at their budget, frequently the one with more structure. Sutton’s conclusion holds asymptotically, and the asymptote is not where most work is done.

Misreading

The standard misreading treats it as a license to ignore structure entirely, which the essay does not support and the record contradicts. The methods that won these comparisons were not structureless. Convolutional networks encode translation equivariance and locality, attention encodes permutation equivariance with position supplied explicitly, and tree search encodes the game’s move structure.

What distinguishes these from the approaches that lost is that they impose weak general constraints on the problem class rather than specific domain knowledge, and they do not cap what can be learned. The distinction is between structure that is a prior, which data can override, and structure that is a constraint, which it cannot. The essay argues against the second and has been read as arguing against both.

Counterexamples

The counterexample class is problems where data is scarce and the prior is genuinely known. With few examples and an established governing law, encoding it beats learning it, and no amount of compute changes that because the missing input is data. Physical simulation under known conservation laws, structured chemistry and biology, and settings requiring a guarantee rather than an empirical rate all fall here.

Hardware lottery

Sara Hooker's argument that ideas win partly because available hardware happens to suit them.

Ideas waited decades for hardware that made them practical, and the waiting was mistaken for refutation. Accelerators are optimized for the dominant workload, which widens its advantage and biases what is publishable.

Measured comparisons reflect both the merit of the methods and their fit to the substrate. A method with irregular access or dynamic control flow runs at a fraction of peak throughput, so reported gaps include an unmeasured hardware term.

A method can fail in public because it is wrong, or because the machines available at the time make it look slow and awkward. The hardware lottery is the second possibility made systematic. Ideas that match the dominant accelerator get optimized kernels, tooling, and attention; ideas with irregular control flow or memory access pay a penalty that looks like scientific inferiority. Reading the result requires separating the method from the substrate it was tested on.

Historical examples

Backpropagation was described in usable form in the 1980s and its large-scale success arrived in the 2010s, with the intervening decades spent believing that deep networks were untrainable. Convolutional architectures for image recognition existed well before the results that made them standard, and what changed was the arrival of general-purpose graphics processors and a large labeled dataset rather than a change to the idea.

Hooker’s argument is that in each case the idea was correct and unimplementable, that this is indistinguishable from incorrect using the evidence available at the time, and that the reversal came from hardware developed for unrelated commercial reasons. The uncomfortable implication is that the hardware trajectory selected which ideas survived, and it was not chosen with that in mind.

Publishing bias

This biases what is publishable, since an idea that maps badly to current accelerators looks worse than it is. Current accelerators are optimized for dense matrix multiplication at low precision with regular memory access, so a method with that shape runs near peak throughput and one with irregular access, dynamic control flow, or fine-grained sparsity runs at a small fraction of it.

The measured comparison is between an idea running efficiently and an idea running badly, and the reported gap mixes the merit of the methods with the fit of each to the substrate. Unstructured sparsity is the standard illustration: a network with most weights removed has far fewer operations and frequently runs no faster. Nothing in the numbers separates the two contributions.

Self-reinforcement

Hardware is designed for whatever currently wins, which makes the advantage self-reinforcing. Accelerator design targets the dominant workloads, so each generation widens the efficiency advantage of the dominant architecture, which increases its share. Software follows: kernel libraries, compilers, and frameworks are optimized for the common case, and an unusual operation runs a generic path slower by a large factor.

The advantage compounds independently of relative merit and is nearly impossible to overcome from outside, since a challenger must beat an incumbent with years of specialization behind it while running a generic path. Whether the dominant design is best or merely the one that got the specialization is not answerable from current evidence.

Reading negative results

A negative result on an unusual architecture is about the pairing. Ask whether engineering effort was equal, whether the baseline had optimized kernels the new method lacked, and whether the gap would survive comparable specialization. Work on approaches that do not fit current hardware has an expected value depending on a trajectory nobody controls, which argues for a small deliberate allocation rather than zero.

Hooker’s own recommendation follows from the loop, which is that hardware and algorithm research should be less separated than they are, since the cost of the current arrangement is a set of good ideas that were never given a fair test.

Kaplan against Chinchilla

Two scaling law papers that disagreed about how to spend a compute budget, and the correction that changed training practice.

Kaplan 2020 found models should grow faster than data. Hoffmann 2022 found parameters and tokens should grow together. The disagreement traces to how parameters and compute were counted, to warmup length, and to optimizer settings not tuned per model size.

The earlier work counted only non-embedding parameters and fitted smaller models, in which embeddings take a sizable share of all parameters, inflating the exponent. Its warmup period, fixed in length and too long for small models, pushed the same way.

Scaling laws are often treated as neutral curves, but a curve fitted from expensive runs also carries every convention and training choice used to produce it. Kaplan against Chinchilla is the case where those details changed the answer to a budget question. The consequence was not a small bookkeeping correction: the field moved from favoring larger models with fewer tokens toward a different allocation of parameters and data.

Kaplan 2020

Kaplan and coauthors published in 2020 a set of empirical scaling relationships for language models, finding that loss falls as a power law in model size, dataset size, and compute, and that when compute is the binding constraint the model should be grown considerably faster than the data.

The prescription that followed was to train large models on relatively little data and to stop well before convergence, since the marginal return from another parameter exceeded the marginal return from another token. This was adopted widely, and the largest models of the following two years were built on it; several of them had parameter counts far out of proportion to the tokens they saw.

Chinchilla 2022

Hoffmann and coauthors published in 2022 the result known as Chinchilla, from a much larger sweep across model sizes and durations: parameters and tokens should grow together in roughly equal proportion. The demonstration was direct. A model of roughly 70 billion parameters trained on about 1.4 trillion tokens beat one of roughly 280 billion trained on about 300 billion, at matched training compute.

The implication was that the largest models of the period were substantially undertrained, and that for a fixed budget a smaller model given more data was the better allocation. Three separate estimation approaches were used, and the headline guidance of roughly twenty tokens per parameter came from their agreement.

Consequences

The practical consequence was models trained on far more data than the earlier prescription called for. Training runs shifted toward token counts one or two orders of magnitude above what parameter count alone would have suggested, and the industry’s constraint moved from compute toward data acquisition, filtering, and curation, which is where a great deal of subsequent effort went.

A second consequence was a shift in what gets built for deployment, since a smaller model matching a larger one’s quality is cheaper to serve on every request, and serving cost dominates training cost for a widely used system. Reporting practice changed as well, since a parameter count alone stopped being interpretable without the token count beside it.

Source of disagreement

The disagreement was about the fit and the training procedure behind it, not the existence of a power law. Both groups found loss falling as a power law in compute with stable exponents, so the phenomenon was never in question. The discrepancy traces to parameter counting and to the scale of the models fitted: the earlier work counted only non-embedding parameters over comparatively small models, where embeddings are a sizable share of the total, and both choices inflate the exponent relating optimal model size to compute, roughly 0.73 against roughly 0.50 under the later convention.

Pearce and Song showed in 2024 that redoing the later analysis under the earlier convention, at the earlier scale, gives an exponent close to the earlier one, which identifies the accounting rather than the phenomenon as the source. Porian and colleagues, also in 2024, reproduced the earlier law in their own setup and corrected its procedure one step at a time: counting the compute of the output layer, shortening a fixed warmup that was too long for the smallest models, and tuning optimizer settings such as the learning rate and batch size for each model size brought the exponent to about 0.50 with a constant learning rate. The learning rate decay the Chinchilla authors had suspected was not the main cause, since matching it to each run’s token budget moved the exponent only from about 0.60 to 0.57. Decay still lowers the loss substantially, with benefits that persist at larger scales, but it barely moves the compute-optimal allocation.

Chinchilla revisited

Reanalysis of the compute-optimal result finding the published fit harder to reproduce than adoption suggested.

Reconstruct data points from figures, refit the stated model, check if reported parameters and intervals follow. A 2024 reanalysis found one of three estimation approaches did not reproduce cleanly.

The underlying runs are not public, so the check is on the analysis rather than the evidence. Refitting gave a ratio consistent with the paper's other two methods, so the headline survived while one route to it did not.

A result can be influential enough to steer training practice while still being impossible for outsiders to reproduce in the usual sense. Chinchilla revisited is about what can be checked when the runs themselves remain private. The check shifts from rerunning the experiment to reconstructing the analysis, refitting what was published, and asking whether the conclusions depend on the route taken to estimate them.

Replication limits

When training runs are not public, a replication can still reconstruct the data points from published figures, refit the stated model, and check whether the reported parameters follow. A 2024 reanalysis did this and found the third of the paper’s three estimation approaches, a parametric fit to the loss surface, did not reproduce cleanly: the reported confidence intervals were implausibly narrow, and the fitted parameters sat awkwardly against the other two.

What such an attempt cannot establish is whether the underlying experiments were correct, since the runs, the data, and the intermediate measurements are not available. That asymmetry is the general condition for scaling work at this scale, and it is worth stating plainly rather than treating published numbers as verified.

Inference cost

Inference cost changes the optimum, because a model that is served heavily should be smaller than training-optimal. The compute-optimal analysis minimizes loss for a fixed training budget and counts nothing after training, which is the right objective for a research artifact and the wrong one for a deployed system.

Once a model serves a large volume of requests, lifetime compute is dominated by inference, so the right objective minimizes training plus expected serving cost, which favors a smaller model trained past where the training-only analysis stops. The return to additional tokens diminishes but stays positive well beyond the compute-optimal ratio. This is why deployed model sizes have not tracked the prescription, and the divergence is a considered decision.

Data quality

Data quality and repetition are not represented in the original formulation, which treats tokens as interchangeable units. They are not. Deduplication, filtering, and curation change the loss achieved at a fixed token count enough to move the effective position on the curve, so two runs at the same nominal token count are not comparable across corpora.

Repetition sharpens as high-quality text becomes scarce. Repeated data is worth less than fresh, the value decays with each pass, and beyond a modest number of epochs additional passes contribute nothing and eventually hurt. Subsequent work fitted the decay explicitly, which extends the framework rather than replacing it. A token count means little without the corpus and the pass count.

Using scaling laws

A scaling law is an empirical fit within a range, not a law of nature. These are regressions over runs at particular scales with particular architectures, data, and optimizers, and their status is a well supported local approximation. Extrapolating orders of magnitude beyond the fitted range is an assumption, and the fits here were revised when the procedure changed rather than when the physics did.

The productive use is as a planning tool inside the measured range and as a source of hypotheses outside it, and the error to avoid is treating a fitted exponent as a constant of the field. The same caution applies to every successor formulation, including the ones that add data quality and inference cost, since those are also fits.

Scaling stall debate

The recurring argument over whether returns to scale are diminishing.

Both sides agree loss falls smoothly. The dispute is over what that buys, and it is argued on benchmarks that lose resolution at exactly the point the question gets interesting.

Ceiling effects compress measured differences, and contamination inflates scores. A plateau on such a benchmark does not establish that underlying capability has stalled.

People arguing about whether scale has stopped paying often point at different objects while using the same words. Loss can keep improving smoothly while public task scores flatten because the measurement has run out of room. The scaling stall debate is the attempt to separate those claims. Without that separation, a saturated benchmark or a contaminated test set can be mistaken for evidence about the underlying training trend.

Points of agreement

Both sides agree that the loss curve is smooth and the capability curve is not, and starting from the agreement clarifies what is actually disputed. Training loss falls as a power law in compute with remarkable regularity, and nobody in the debate contests this, since it has held across many orders of magnitude and many groups. What is contested is the mapping from loss to anything a user cares about.

Loss is an average over token predictions, and a small improvement in that average can move the completion rate of a multi-step task a great deal or not at all, depending where it lands. The disagreement is not whether scaling reduces loss but whether the remaining reductions convert into capability at a rate worth the cost.

Benchmark saturation

Public benchmarks saturate before capability does, so the measurement fails before the trend does. A benchmark with a ceiling stops discriminating once systems approach it, and the residual difference between a system at 92 and one at 94 is dominated by the benchmark’s own label noise and ambiguous items rather than by the systems.

When the most cited benchmarks are near their ceilings, visible evidence for continued improvement disappears whether or not improvement continues, which produces the appearance of a plateau. Contamination pushes the same way by inflating scores and compressing the range. A flattening curve on a saturated benchmark is uninformative about the underlying trend, and reading it as evidence of one is where public argument here usually goes wrong.

Data constraint

Data availability is a constraint distinct from compute, and conflating the two confuses the debate. Compute continues to grow through capital expenditure and hardware improvement, while the stock of high-quality human-written text is finite and a substantial fraction of the accessible portion has been used. Repetition helps for a limited number of passes and then stops, as noted above.

The candidates are synthetic generation, which compounds the generating model’s errors and works best where outputs can be verified; multimodal sources, which expand the pool at the cost of relevance; and licensed corpora, which are expensive and finite. Whether this binds before compute does is genuinely open, and the parties hold different private information, which is part of why the public argument does not converge.

Reading claims

Reading a claim in this debate means asking what was measured, on what, and by whom. The frontier training runs, their evaluation results, and their internal scaling curves are largely private, so public argument runs on inference from product behavior, selective disclosure, and anonymous reporting, none of which supports a confident conclusion.

Specific questions do most of the work: loss or task metric, benchmark near its ceiling, contamination checked, how many items and runs, and whether the party has a financial position, which is nearly always yes in both directions. As of the middle of the decade the loss trend has continued where measured, public capability measurement is too degraded to settle the conversion question, and anyone confident on either side is reasoning from evidence they have not shown.

Test-time compute bet

Spending more computation during inference rather than only during training.

Allocate inference compute per request according to difficulty: easy questions answered cheaply, hard ones receive more sampling, search, or intermediate steps.

Capability becomes purchasable at the moment it is needed rather than fixed at training time. Gains are largest where a checker exists, since selection requires identifying correct answers.

A system that answers every request with the same amount of work treats an easy lookup and a hard proof search as if they cost the same to solve. The test-time compute bet is to spend extra inference only when the question earns it. That makes capability adjustable at use time, but it also moves part of the product from a fixed model cost into a variable operating cost that depends on the request.

Trade

The trade is a fixed training cost against a per-query cost that scales with difficulty. Training compute is spent once and amortized across every request, behaving as capital expenditure. Inference compute is spent per request and allocated by difficulty: an easy question is answered cheaply, a hard one gets more sampling or search.

Improvements can be delivered without retraining, but the marginal request is more expensive and that expense recurs forever, so the comparison between spending on training and spending on inference depends on the request volume.

Verification requirement

The approach rests on verification being easier than generation. Sampling many candidates helps only if the good ones can be identified more cheaply than generating a correct answer directly. Where a checker exists this holds decisively: a proof assistant validates a proof, a test suite a program, a numerical check a computation.

Where no checker exists the property is assumed rather than established, and the selection falls to a learned verifier or to the model’s own judgment, both of which are themselves fallible and share failure modes with the generator.

Economic implications

Serving cost now varies with the question asked, which breaks previously safe assumptions. Capacity planning on request counts fails when one request consumes many times another’s compute, so the planning unit becomes tokens or accelerator-seconds. Pricing has the same problem: flat per-request pricing is exploitable by whoever sends the hardest questions, and usage-based pricing exposes the customer to unpredictable variance.

Latency becomes variable in a way that is visible to users, since a system that thinks longer on hard questions is slower exactly when the user is most attentive. Operationally this pushes toward explicit budgets per request, routing that sends easy questions to cheap paths, and caching, all of which are engineering responses to a cost structure that the previous generation of systems did not have.

Open questions

Which problems reward extra thinking and which do not is the open question, and the current evidence is uneven. The pattern that holds so far is that gains are large where a solution can be checked, where the problem decomposes into steps that can be independently validated, and where the failure mode is a reasoning error rather than missing knowledge.

Gains are small or absent where the answer depends on information the model does not have, since no amount of deliberation supplies a fact, and where the task is a single judgment with no internal structure to work through. There is also evidence of a reversal at the extreme, where additional deliberation degrades performance on some easy problems by talking a correct initial answer into a wrong one.

Where extra thinking pays

On September 4, 2026, Anthropic reported formalizing Fermat’s Last Theorem with dozens of agents using an internal research model and the Prove2Me proof platform: about six billion output tokens over 11 elapsed days, with human mathematical input limited to occasional high-level instructions. The Lean proof assistant checked the finished proof using only its three standard axioms.

The honest summary is that additional inference compute is not uniformly beneficial, that predicting in advance which requests benefit is itself an unsolved routing problem, and that the systems deployed today mostly handle it with heuristics rather than with a principled estimate.

Emergent abilities as a measurement artifact

The argument that sharp capability jumps are often produced by the choice of metric rather than by a change in the model.

An all-or-nothing score can conceal gradual improvement. Exact match across several tokens gives credit only when every token is correct.

Holding checkpoints fixed while changing the metric can remove apparent jumps. The difference then comes from the measurement.

A model can improve steadily while the graph shown to readers stays flat until a cutoff is crossed. Exact-match grading, thresholds, and small test sets can hide gradual progress until enough partial improvements cross a boundary together. The measurement-artifact account of emergent abilities names that failure mode. It does not rule out sharp internal changes, but it makes the metric itself part of the evidence that must be checked.

Discontinuous scoring

A discontinuous scoring rule turns smooth improvement into an apparent jump. Take a task scored by exact match on a multi-token answer, credit given only if every token is correct. If per-token accuracy improves smoothly with scale, the probability of getting all of them right is that quantity raised to the token count, which stays near zero across a wide range and then rises sharply.

Nothing discontinuous happened to the model, and the discontinuity was introduced by the scoring rule. The same effect follows from any nonlinear or thresholded metric, and it is amplified by small evaluation sets, where a score of zero is common and a handful of newly correct items produces a large apparent jump. Schaeffer, Miranda, and Koyejo made this argument in 2023 and supported it by predicting which metrics would show jumps and which would not.

Empirical core

Continuous metrics show the same runs improving gradually, which is the empirical core of the critique. Replacing exact match with token-level edit distance, or accuracy with the log probability of the correct answer, converts many reported step changes into smooth curves on identical checkpoints. The runs are unchanged, so the difference is attributable to the measurement. The authors also induced apparently emergent behavior in domains known to be smooth simply by choosing a thresholded metric, which is the stronger form of the argument.

Surviving transitions

Some transitions survive the critique, and overstating it is its own error. The argument establishes that a sharp curve is not evidence of a sharp underlying change, not that none occur. Cases that remain interesting after the metric is fixed include transitions where a continuous measure also moves quickly, phase-change behavior in specific internal mechanisms, and grokking.

The distinction worth holding is between a smooth capability crossing a threshold in a scoring rule, which is a measurement effect, and an internal mechanism forming over a short interval of training, which is a real change with observable structure inside the model. Establishing the second requires evidence from the model rather than from the score.

Methodological lesson

The methodological lesson is that a scoring rule is part of the claim, and reporting a result without it is reporting half of the finding. Thresholded metrics are appropriate when the deployment genuinely requires an exactly correct answer, since partial credit does not help a user who needs a correct one, so the objection is not that exact match is invalid.

It is that any claim about the shape of a trend must be accompanied by the metric that produced the shape, and preferably by the same runs measured with a continuous alternative. The habit extends past this case: when a curve does something surprising, check the measurement before theorizing about the mechanism, because the measurement is cheaper to check and is the more common explanation.

Double descent, and the bias-variance picture it broke

Test error falling, rising near the interpolation point, and then falling again as capacity grows past it.

Below the interpolation threshold the model cannot fit the data and error behaves classically. At the threshold there is exactly one way to fit, fully determined by noise. Past it, the optimizer selects among many solutions.

The classical U-curve describes models with fewer parameters than examples. Past the threshold, gradient descent selects a low-complexity interpolating solution rather than an arbitrary one.

Learning theory treats the phenomenon itself. What matters here is that a result taught as settled for decades turned out to describe one regime rather than all of them.

Classical view

The classical U-curve says test error falls as capacity captures more signal, reaches a minimum, then rises as the model fits noise, which prescribes stopping at the bottom of the U. That is correct in the regime where it was derived, models with fewer parameters than training examples, and it stops describing what is observed once capacity passes the point of exact fit, which is where modern networks operate.

Interpolation threshold

The interpolation threshold is where the model has just enough capacity to fit the training data exactly, and test error peaks there. That solution is fully determined by noise as well as signal, so it has no freedom to be well behaved and its norm is enormous. Past the threshold many solutions fit, and the optimizer selects among them, so the question becomes which interpolating solution is chosen. Error falls again, often below the classical minimum. Belkin and coauthors described this in 2019 and showed it outside deep learning, in linear regression and random forests, establishing it as a property of overparameterized fitting.

Variations

Model-wise, sample-wise, and epoch-wise versions of the effect all appear, and they share one explanation. Model-wise is the version just described, with capacity on the horizontal axis. Sample-wise is the counterintuitive case where adding training examples increases test error, which happens when the additional data moves the model toward the interpolation threshold from the overparameterized side, so more data is briefly harmful.

Epoch-wise holds the model fixed and varies training time: error falls, rises as the model reaches interpolation, and falls again, which undermines early stopping on a single validation minimum. All three are the same curve with a different quantity controlling the ratio of capacity to data.

Explanation

The implicit regularization explanation is that gradient descent does not select an arbitrary interpolating solution but a low-complexity one, and what remains unsettled is the precise sense of complexity. For linear models the story is established: gradient descent from zero converges to the minimum-norm interpolating solution, and past the threshold the additional capacity permits a lower-norm fit, which generalizes better.

For deep networks the analogous claim is believed and not proven, and the candidate measures of complexity, including various norms, margin-based quantities, and flatness of the minimum, each explain part of the evidence and each have known counterexamples.

Significance of the peak

The peak’s practical significance is also debated, since with appropriate explicit regularization it can be substantially reduced or removed, which suggests it is a property of underregularized fitting rather than a fundamental barrier. The settled part is that capacity beyond the interpolation threshold is not inherently harmful, and the unsettled part is the account of why.

Grokking, and the stopping rule it broke

Generalization appearing long after training accuracy has saturated.

Train a small network on a fraction of a finite table. Training accuracy reaches 100 percent early; test accuracy stays at chance. Continue training by orders of magnitude more steps; test accuracy then rises sharply.

Two solutions compete: a memorizing one reachable quickly with large weights, and a general one slower to find with smaller weights. Weight decay applies continuous pressure toward the general solution after loss flattens.

Learning theory covers the mechanism. The judgment consequence is that the standard early-stopping criterion was reading a signal that does not mean what the field assumed it meant.

Original setting

The original setting was small algorithmic tasks, principally modular arithmetic, a network trained on a fraction of a finite table and tested on the rest. Training accuracy reaches 100 percent early, where ordinary practice would call the run finished, while test accuracy stays at chance. With training continued for orders of magnitude more steps, test accuracy rises sharply to near perfect.

The delay is the finding: a model can sit at zero training loss for an enormous number of steps while something is still changing that eventually produces generalization. The 2022 report used deliberately small clean settings, which made the effect isolable and raises the question of how far it transfers.

Weight decay and circuit formation

Weight decay is the most effective intervention in the 2022 experiments, though some generalization occurs without it, and the transition runs from memorization to a general circuit. A memorizing solution stores the training pairs and is reachable quickly with large weights; a general one implements the operation and is slower to find but smaller. Both reach zero training loss, so weight decay is what selects the smaller-norm solution.

The transition is a slow migration under a regularizer acting long after the loss has flattened. Interpretability work has identified the general circuit concretely: networks trained on modular addition implement it through periodic representations and trigonometric identities, and the circuit assembles gradually during the plateau rather than appearing when test accuracy moves.

Implications for stopping rules

This is a warning about early stopping criteria, and a specific one. Stopping when training loss flattens or when validation accuracy has not improved for a fixed patience would terminate every one of these runs before the generalization occurred, and would do so with every conventional indicator suggesting the run was finished.

Patience settings encode an assumption about how long a transition can take. A training loss at zero does not mean the parameters have stopped moving, and norm-based diagnostics reveal change that loss curves do not. A stopping rule is a bet rather than a neutral efficiency measure.

Scope at scale

Whether it occurs at scale or only in the constructed setting is genuinely open. The arguments that it is narrow are that the reported cases use small networks on finite algorithmic tasks with clean labels, that the delay is enormous relative to any realistic training budget, and that large runs on natural data have not shown a comparable discontinuity.

It is not merely a curiosity: the competition between memorizing and generalizing solutions is not specific to toy tasks, similar delayed transitions appear in circuit formation during large runs, and at scale the effect may compose from many small transitions that no aggregate metric would show.

Position on the evidence

The defensible position is that the mechanism is established in the small setting, that its scope is not, and that the value of the finding is mostly in what it proves is possible rather than in a claim that it is happening in every large run.

Lottery ticket hypothesis

The claim that a dense network contains a small subnetwork which, trained from its original initialization, matches the full model.

Iterative magnitude pruning: train to convergence, remove smallest weights, reset survivors to their original initialization values, retrain. Repeat, and the surviving subnetwork shrinks while matching accuracy.

A winning ticket includes both a sparse connection pattern and its original weight values. Drawing fresh random values for that pattern loses the observed accuracy.

Pruning a trained network can produce a smaller working model, but that alone does not say much about why the original large model helped. The lottery ticket hypothesis asks whether the large model already contained a sparse subnetwork paired with the right initial values. If so, overparameterization is partly a search over many possible subnetworks, and the important object is not only the final sparse structure but the initialization that let it train.

Procedure

Iterative magnitude pruning is the procedure the result depends on, and the rewinding step is the part that carries the claim. Train the dense network to convergence, remove the smallest-magnitude fraction of weights, reset every surviving weight to the value it held at initialization rather than to a fresh random value, and retrain.

Repeat this cycle, removing a modest fraction each round, and the surviving subnetwork shrinks to a small percentage of the original while matching the dense model’s accuracy in comparable training time. Frankle and Carbin reported this in 2018 and named the surviving subnetwork a winning ticket. Pruning in one large step also finds winning tickets, but iterative pruning finds smaller ones that learn faster and reach higher accuracy, so the gradual schedule is load-bearing rather than incidental.

Why initialization matters

Initialization is what distinguishes the claim from ordinary pruning. Pruning a trained network and fine-tuning was already practice, so a small accurate network is not the finding. The finding is that the same sparse structure, reset to its original random values and trained from scratch, reaches comparable accuracy, while the same structure with fresh values of the same distribution does not. The winning subnetwork is a pairing of connectivity with specific initial values.

The framing that follows is that training a large network is partly a search over many subnetworks with different initial conditions, and that overparameterization helps because it supplies more candidates rather than because the extra parameters are individually used.

Scale challenges

Reproducing it at scale proved difficult, which is where the rewinding variants come from. On larger networks resetting to the original initialization fails. The variant that restores the result rewinds to the weights after a few early training steps rather than to step zero, which works reliably and weakens the original claim, since the ticket is now partly determined by early training.

Learning rate rewinding, which resets only the schedule and keeps the trained weights, matches or beats weight rewinding on many benchmarks and is closer to conventional pruning. The state of the evidence is that the phenomenon is real and that its cleanest form holds mainly in the small-scale setting, with the large-scale version requiring a modification that concedes part of the original hypothesis.

Practical limits

What it does not deliver is a way to find the subnetwork without training the dense model first. The procedure requires full training then repeated cycles of pruning and retraining, so a winning ticket costs several times more than training the dense network once. The efficiency argument applies only if the ticket is reused, and transfer works only partially.

Finding sparse structures at initialization without the training loop produces networks that train but not to the accuracy pruning reaches, and the reported advantage of several such criteria over random pruning has been questioned. The hypothesis is a significant result about why overparameterization helps and has not produced a cheaper way to train, which is what it was expected to deliver.

Neural architecture search

Automating architecture design by search.

Early: train a controller to propose architectures, train each to convergence, update controller from reward. Later: weight sharing trains one supernetwork containing all candidates; differentiable relaxation makes architecture parameters trainable by gradient descent.

Weight sharing recovers parallelism but its ranking of candidates correlates weakly with independent training. Differentiable relaxation is biased toward operations that train quickly.

Designing a neural network by hand mixes insight with a large amount of trial and error. Neural architecture search tried to automate that search over designs. The promise was appealing: let computation explore architectures while researchers specify the space and objective. The difficulty was that the search procedure, training recipe, hardware target, and comparison budget all became part of the result, so the reported architecture could not be judged in isolation.

Early cost

The early compute cost was large enough to limit the work to a few groups, which shaped the subfield’s first several years. The initial formulations trained a controller to propose architectures, trained each proposal to convergence to obtain a reward, and updated the controller from that signal, so a single search consumed thousands of accelerator-days.

Only organizations with that budget could participate, so results went unchecked, comparisons used different search spaces and protocols, and reported numbers included tuning applied to the proposal and not the baselines. The cost also made the one control that mattered, an alternative search method at the same budget, prohibitively expensive.

Efficiency responses

Weight sharing and differentiable relaxations were the efficiency responses, and they cut the cost by orders of magnitude. Weight sharing trains one large supernetwork containing every candidate architecture as a subgraph, so evaluating a candidate means evaluating a path through already-trained weights instead of training from scratch.

Differentiable relaxation goes further by replacing the discrete choice among operations with a continuous mixture weighted by learned coefficients, which makes the architecture parameters trainable by gradient descent jointly with the weights, after which the highest-weighted operation is selected. This brought searches down to a small number of accelerator-days and opened the area to ordinary academic budgets.

Weaknesses of the ranking

The known weaknesses are that the ranking of candidates under shared weights correlates only weakly with their ranking when trained independently, so the search optimizes a proxy that is a poor estimate of the objective, and that the continuous relaxation is biased toward operations that train quickly, including parameter-free ones such as skip connections, which degenerate solutions exploit.

Sobering findings

The finding that random search with an equal budget was often competitive is the sobering part of the record. When the control was finally run properly, with random architectures sampled from the same search space and given the same training protocol and the same total compute, the gap to the sophisticated methods was small and in several cases absent.

Reproducibility studies found reported comparisons frequently differed in training protocol rather than architecture, and that much of the improvement came from the recipe. The reading is that search spaces were designed well enough that most points in them were good, so the field was measuring the space while attributing the result to the method.

Lasting contributions

The lasting contribution was better search spaces rather than better search, and it is a real contribution. The cell-based space, in which a repeated block is searched and stacked, encodes a genuine insight about where architectural variation matters. Several widely used efficient architectures came out of it, particularly for mobile deployment where the objective included latency on a specific device.

Hardware-aware search, optimizing measured latency on a target chip rather than operation counts, remains useful because no human can evaluate that objective by inspection. The general lesson transfers: when an automated method beats a baseline, the control that matters is random search in the same space under the same protocol.

Capsule networks

An architecture replacing scalar activations with pose vectors to preserve spatial relationships that pooling discards.

Each lower-level capsule predicts the pose of each higher-level entity. Routing by agreement assigns outputs to higher capsules whose predictions agree, computed iteratively per forward pass.

Agreement among independent pose predictions is strong evidence for the whole and enforces part-whole geometry. The routing is an iterative, data-dependent inner loop that maps badly to parallel accelerators.

Pooling made convolutional networks robust, but it also threw away information about how detected parts were arranged. Capsule networks were a proposed repair: represent parts with vectors and route them to wholes when their pose predictions agree. The diagnosis pointed at a real weakness in the standard architecture. The cost was an inner loop and data-dependent computation that made the cure hard to run efficiently on the hardware that was winning.

Motivation

The motivation was that pooling discards spatial relationships that matter, and the diagnosis was correct. Max pooling summarizes a region by its strongest activation, which deliberately discards where within the region the feature was, and stacking many such layers yields a representation that records which features are present with progressively less information about their arrangement.

The consequence is a model that responds strongly to the right parts in the wrong configuration, since a face detector built this way weakly constrains the eyes to sit above the mouth. Hinton’s positive proposal was that a network should represent an entity by a vector encoding pose and properties rather than a scalar activation, so the part-whole relationship is a transformation between pose vectors and can be checked.

Costs

Routing by agreement is an iterative inner loop inside every forward pass, requires a transformation matrix per pair of capsules, which grows quadratically, and is sequential and data-dependent. The result ran several times slower than a convolutional network of comparable capability, with the gap widening at higher resolution.

Field’s response

The critique of pooling was accurate and is not disputed, and the field’s response was to address it by other means: architectures that avoid aggressive pooling in favor of strided convolution, attention mechanisms that model relationships between positions directly and retain positional information throughout, and data augmentation that teaches invariance to the transformations that matter rather than building it in.

Each of these addresses part of the same problem at a cost the hardware accommodates. The proposal that identified the problem was not the proposal that solved it, and this ordering is common enough to be worth expecting, which is an argument for separating a critique’s validity from its accompanying remedy when evaluating either.

Hardware lottery connection

The connection to the hardware lottery is direct: routing maps badly to accelerators optimized for large dense matrix multiplications with static shapes. Routing by agreement is an iterative, data-dependent, small-operation loop using a fraction of throughput. The comparisons measured an architecture near peak against one well below it, and no published number separates the idea from the substrate.

Whether capsules would be competitive on hardware designed for dynamic routing is unknown and probably unknowable, since that hardware does not exist and will not be built for an architecture that lost. This is the feedback loop from the earlier section operating on a specific case, and it is why this counts as a clean case study in a failed bet rather than as a refuted idea.

Perceptron limitation

The 1969 demonstration that a single-layer perceptron cannot represent XOR.

Minsky and Papert proved that a perceptron computing a threshold over local predicates cannot represent XOR, connectedness, or parity without predicates whose order grows with input size.

The mathematics is correct and concerns single layers, not neural networks in general. The book treats multilayer machines as an open problem, but the judgment that analysis would not extend usefully was absorbed as a verdict on the direction.

A negative theorem can be exactly right and still become misleading once its scope is forgotten. The perceptron limitation is the classic case. The mathematical result concerned a restricted single-layer machine, but the lesson absorbed by the field was broader than the theorem. The consequence is a durable warning for reading technical failures: the assumptions doing the work may be narrower than the verdict that survives in memory.

Actual result

The actual result concerned a single layer rather than neural networks in general. Minsky and Papert’s 1969 book proved that a perceptron computing a threshold over a fixed set of local predicates cannot represent exclusive or, and more substantially cannot compute predicates such as connectedness and parity without predicates whose order grows with the input, which is a statement about representational capacity under a specific restriction. The mathematics is correct and remains correct.

The book is explicit that the result concerns this restricted class rather than layered networks in general, treating multilayer machines as open. What it did not provide was a method for training multiple layers, and the authors doubted the analysis would extend, which is a judgment rather than a theorem. The field absorbed the judgment.

Reception

The reception went well past the result, and a funding contraction followed. The book arrived when the earlier wave of connectionist work had already promised more than it delivered, so it landed as confirmation rather than as a technical contribution and the qualified conclusion was compressed into the claim that neural networks do not work. Attention and funding moved toward symbolic approaches for roughly fifteen years.

Reconstructing precise causation is difficult, since a general contraction in the field was underway for reasons including the machine translation reviews of the mid-1960s and the broader disappointment with early promises, so the book was one input rather than the sole cause. What is clear is that a narrow theorem about one architecture became a field-wide verdict about a research direction, and that the verdict outlived any evidence for it.

Resolution

The resolution required hidden layers and a way to train them. Hidden layers supply the capacity the theorem shows a single layer lacks; backpropagation supplies the credit assignment that makes them trainable. Backpropagation was derived independently several times and reached wide attention in the mid-1980s, after which the limitation the book described was no longer one. The gap lasted more than fifteen years, and the missing piece was an algorithm rather than an insight.

Lesson

A narrow theorem was generalized into a field-wide verdict, and the pattern is not historical. A precise negative result under specific assumptions gets restated with the assumptions dropped, and the restatement is what circulates. The defense is mechanical: find the original result, read what it quantifies over, and check which assumptions were load-bearing and whether they still hold.

The corresponding discipline for producing results is to state the scope in the abstract, since a paper’s qualifications do not survive summarization and the author is the only party with an incentive to preserve them. This is a specific instance of the point made in the research taste section, which is that a field’s abandoned directions are worth re-examining, because the reason for abandonment is often narrower than the abandonment.

AI winters

Periods when funding and attention collapsed after expectations outran results.

Real capability in a bounded domain was extrapolated to general claims, funding was committed against the general claim, and correction came when the gap became undeniable. The correction then overshot.

Funding can collapse while useful methods retain their capabilities. Lost credibility can drive withdrawal even from the narrow applications that worked.

A technology can work in a bounded setting and still disappoint everyone who funded the generalized story around it. AI winters are the collapses that follow that gap between demonstrated capability and advertised reach. The important point is that credibility, funding, and technical merit can move together without being the same thing. When expectations get repriced, useful techniques can be abandoned along with the claims that oversold them.

First winter

The 1970s contraction followed two reviews that found the field’s results well short of its promises. The 1966 report of the Automatic Language Processing Advisory Committee assessed a decade of machine translation work in the United States and concluded that machine output was slower, less accurate, and more expensive than human translation once the required post-editing was counted, and funding for the area largely ended.

The 1973 report commissioned from James Lighthill by the Science Research Council reached a similar verdict more broadly: methods succeeding on small demonstrations failed on realistic problems as search cost grew combinatorially. Support in the United Kingdom contracted to a few groups. The technical criticism was accurate for the systems reviewed; the inference that the direction was unpromising was stronger than the evidence supported.

Second winter

The late 1980s collapse was commercial rather than academic, and it centered on expert systems. Rule-based systems encoding specialist knowledge had produced genuine successes in narrow domains, an industry grew around them along with specialized hardware for the languages they were written in, and both collapsed within a few years. The causes were specific rather than mysterious.

Maintaining a large rule base grew disproportionately expensive as rules interacted unpredictably, the systems failed abruptly at the edges of their domain, knowledge acquisition stayed a slow manual bottleneck, and general-purpose workstations became fast enough to run the same software, removing the specialized hardware market. Government programs wound down, and the field’s standing fell far enough that researchers adopted other terms.

Common pattern

The common pattern is a real capability oversold as a general one. In each episode something genuinely worked in a bounded domain, the demonstration was extrapolated to the general problem by people with an interest in the extrapolation, funding was committed against the general claim, and the correction came when the gap between the bounded result and the general claim became undeniable.

The correction then overshot, since the same actors who had accepted the inflated claim withdrew from the underlying capability as well, which is the resulting error from the second section of this page applied at the scale of a field. The techniques did not stop working when the funding stopped, and several of them are in production today under different names.

Current situation

What differs now is revenue and deployment; what does not is the gap between demonstration and reliability. Current systems have large paying user bases, revenue independent of future promises, production deployment across many industries, and broad rather than single-domain capability. Those are structural changes, and a collapse of the earlier kind, where the technology fails to work, is not the relevant risk.

The continuity is that the distance between an impressive demonstration and a system trusted without supervision remains large, and public expectation is set by demonstrations. Capital is priced against growth, so the risk that history suggests is a repricing when the reliability gap binds, cutting funding without the capability changing at all. That failure differs from the previous two and would look similar from inside.

From recurrence to attention

The replacement of sequential recurrence by parallel attention.

Attention computes output at each position as a weighted average of values, with weights from query-key similarity. Every position is computed independently, so the entire sequence is processed in one large matrix multiplication.

Recurrence requires each position to wait on the previous one, so training time scales with sequence length in wall-clock terms. Attention is more total arithmetic but arithmetic that can be done at once.

Sequence models once paid for order by processing tokens one after another, which made long inputs slow even when accelerators had unused parallel capacity. The move from recurrence to attention changed the shape of the computation. Attention spends more total arithmetic to compare positions directly, but the work can be batched into large parallel operations. The trade bought shorter training time at the cost of quadratic growth with sequence length.

Recurrent bottleneck

The recurrent bottleneck is a sequential dependency that cannot use a parallel accelerator well. A recurrent network computes each hidden state from the previous one, so position $t$ cannot begin until position $t-1$ has finished, and a sequence of length 1,000 requires 1,000 dependent steps regardless of how many arithmetic units are available.

The operations at each step are small, so the accelerator sits mostly idle while paying the latency of a kernel launch and a memory round trip per step. Gating mechanisms addressed the separate problem of gradients vanishing across long dependencies and did not change the sequential structure. The consequence was that recurrent training time scaled with sequence length in wall-clock terms rather than with total arithmetic, which capped the size of models anyone could afford to train.

Attention advantage

Path length between any two positions becomes one instead of the distance between them, which removes the long-dependency problem directly. The cost is that the attention matrix has an entry for every pair of positions, so compute and memory grow with the square of the length, where recurrence grew linearly. On hardware with abundant parallel throughput and expensive sequential latency, that trade is heavily favorable.

Hardware timing

The transition happened when it did because of the hardware available, which makes this the hardware lottery paying out rather than an exception. Attention existed as an addition to recurrent encoder-decoder models for years before the architecture that removed recurrence entirely arrived in 2017. What made removal attractive was accelerators with parallel throughput and bandwidth adequate for the quadratic term at the sequence lengths then in use, a few hundred to a couple of thousand tokens.

Had memory been tighter or parallel throughput scarcer, the arithmetic would have favored recurrence and the transition would have waited. Later hardware-aware implementations that avoid materializing the attention matrix by tiling the computation to fit in fast on-chip memory extended the practical range considerably, which is another instance of the same principle: a method’s viability was set by an implementation matched to the memory hierarchy rather than by a change to the mathematics.

Return of recurrence

Recurrent and state-space designs returned for long sequences. Structured state-space models and their gated successors compute a linear recurrence evaluable in parallel across the sequence at training time using a scan, recovering the parallelism recurrence lacked, while keeping constant memory per token at inference where attention’s grows with context. Cost is linear rather than quadratic in sequence length.

What they give up is that all history compresses into a fixed-size state, so exact recall of an arbitrary earlier token is harder than for attention, which keeps every token and pays for it. Measured comparisons concentrate the gap on recall-heavy tasks, and hybrids interleaving a few attention layers among many recurrent ones are the common answer. Neither mechanism dominates; the choice follows from whether the workload needs exact recall or long cheap context.

From adversarial training to diffusion

The shift in generative modeling from a minimax game to iterative denoising.

Diffusion trains a model to predict noise added at each step of a fixed forward process, which is supervised regression with a single objective. Sampling starts from noise and applies learned reverse steps.

Adversarial training's loss curves do not indicate progress; mode collapse and instability are intrinsic to the game. Diffusion's objective is a meaningful signal, training is stable, and coverage is good rather than collapsed.

Image generation had a method that could make sharp samples quickly, but its training signal was hard to read and often unstable. The shift from adversarial training to diffusion replaced that game with repeated denoising under a single objective. The payoff was not immediate speed. It was a training process whose progress meant something and whose failures were easier to diagnose, letting later engineering attack the sampling cost.

Adversarial failure modes

Adversarial training pits a generator against a discriminator, and its failure modes follow from the game rather than any implementation detail. The generator produces samples, the discriminator separates them from real data, and at the optimum the pair recovers the data distribution. The optimization is minimax rather than minimization, so no single objective’s decrease indicates progress, and both loss curves are consistent with the system doing well or doing nothing.

Mode collapse is the characteristic pathology: the generator finds a small set of outputs the discriminator cannot currently reject and produces only those, a good response to the opponent and a failure at the task. Training is unstable in the ordinary sense too, the equilibrium easily lost when one network outpaces the other. The samples were sharp and fast, and the approach dominated image generation for years despite all of this.

Diffusion as regression

Diffusion is a stable likelihood-based objective learned by reversing a noising process, and the stability comes from replacing the game with a regression. A fixed forward process adds Gaussian noise to data in many small steps until nothing remains, and the model is trained to predict the noise added at each step, which is a supervised problem with a single objective and no opponent. Sampling starts from noise and applies the learned reverse steps.

The consequences are a loss that signals progress, training stable across settings, and good coverage, since the objective bounds likelihood and penalizes missing regions. Coverage mattered most: a generator producing varied outputs across the full range of prompts beats one producing sharper samples from a narrower set.

Sampling cost and its reduction

The cost is many sequential steps per sample, and reducing it took substantial work. Early formulations required hundreds to a thousand network evaluations to produce one sample, against a single forward pass for the adversarial approach, which made the method orders of magnitude more expensive to serve.

The reductions came from deterministic samplers and higher-order solvers taking larger steps with less error, moving the process into a compressed latent space, and distillation, where a student reproduces in a few steps what the teacher does in many. Several distillation methods reintroduce an adversarial component to sharpen few-step samples, so the two approaches partly converged rather than one replacing the other.

Lesson on stability

A stable objective beats a faster one over a long enough horizon. At the transition the adversarial approach led on sample quality and far ahead on speed; diffusion led only on stability and coverage, which compound. A method that trains reliably can be scaled by anyone and moved to new domains without rediscovering the tricks, so it improves faster even from a worse position.

Speed, meanwhile, is an engineering problem with many available attacks, and it was solved. The transferable form of this is that when comparing two approaches, the properties that determine the rate of improvement deserve more weight than the properties that determine the current position, and stability of optimization is the clearest example of a property in the first category.

Long context against retrieval

The question of whether to put material in the prompt or fetch it on demand.

Long context: attend over all material each query. Retrieval: index once, search per query, place only selected passages in prompt. Prompt caching shifts the comparison when the same material is reused.

Long prompts incur inference cost on every request. Retrieval limits that input and gives the serving system a place to enforce document permissions, track sources, and refresh evidence.

When a system needs outside material, the simplest idea is to put more of it in the prompt. The alternative is to fetch only the pieces needed for the current question. Long context against retrieval is the choice between paying attention over everything each time and paying for an index plus search. The decision affects cost, citations, access control, freshness, and where relevant material sits in the model’s input.

Cost model

The cost model comes first, since attention over a long context is paid on every query. A large body of material in the prompt is attended over each time a question is asked, so cost scales as context length times request volume and recurs indefinitely. Retrieval pays indexing once and a per-query search cost roughly independent of corpus size.

For a corpus larger than the window, retrieval is the only option. For one that fits, the comparison is a large recurring per-query cost against a smaller one plus an indexing pipeline, and the crossover depends on request volume and corpus churn. Prompt caching favors long context when the same material is reused, and does not help when it varies per request.

Position effects

Position effects are measured rather than anecdotal. Studies placing a required fact at varying positions find accuracy highest near the beginning or end and lowest in the middle, reported as lost in the middle. A long context is not a uniform store, so supplying more material can hurt: distracting content degrades performance on what was already there.

Benchmarks asking for a single planted fact are easier than realistic use, since finding one distinctive sentence is a weaker demand than synthesizing across passages, and reported context lengths are usually validated against the easier task. Place the most important material at the boundaries, keep context tight rather than maximal, and measure on the actual task.

Retrieval properties

Retrieval adds provenance, access control, and freshness beyond the cost argument, and these often decide the question on their own. Provenance means each claim can be traced to a specific retrieved document, which permits citation and lets a user verify. A long prompt can carry the same identifiers and be asked to cite them, so the difference is that retrieval makes provenance structural where long context makes it something the system has to be built to preserve and the model has to be trusted to respect.

Access control means retrieval filters by the requesting user’s permissions, so entitlements are enforced at fetch time rather than by asking the model to respect an instruction, which is not a security boundary. Freshness means the index updates as documents change, with no retraining. Both are properties of the architecture rather than the model, which is why retrieval remains standard in enterprise deployments even where the corpus would fit in context.

Combining the approaches

The two are usually combined. The common design retrieves generously, since long context makes it affordable to pass more passages than a short window allowed, and lets the model work across all of them, which relaxes the precision demand on the retriever. Material that is stable, small, and needed on every request belongs in the context, ideally in a cached prefix.

Material that is large, changing, access-controlled, or needs citation belongs in the index. What settles it in practice is measurement on the real task, since the position effects and the retriever’s error rate interact in ways that a general argument does not predict. This is an engineering trade with a workload-dependent answer rather than a question awaiting a result, and treating it as the latter is the common error.

Discounting a contaminated benchmark

Test material appearing in training data, converting evaluation of capability into partial measurement of memorization.

Direct contamination: test items in the corpus. Indirect: exposure to discussion, solutions, and derived datasets. Both inflate scores without improper action, since public material is public.

Overlap detection requires corpus access and misses paraphrases. Canary strings find verbatim copies only. Behavioral tests provide evidence without corpus access but cannot establish cleanliness.

Evaluation covers how contamination happens and what to do about it when you own the benchmark. The question here is narrower and is the one a reader faces: given a reported score you cannot audit, how much should you take off it.

Direct and indirect contamination

Direct and indirect contamination differ in how the material reaches the training data, and the indirect kind is both more common and harder to remove. Direct contamination is the test items themselves appearing in the corpus, which happens because benchmarks are published on the open web and web crawls collect them.

Indirect contamination is exposure to the benchmark rather than its items: forum discussion, solutions in repositories, derived datasets reusing items under another name. A model that has seen extensive discussion of a benchmark’s characteristic problems has an advantage no item-level check detects. Neither form requires improper conduct; public evaluation material is public.

Detection methods

Detection by n-gram overlap and by canary strings is standard and incomplete. Overlap detection finds verbatim copies and misses paraphrases, translations, reformatted versions and the same problem under different wording. It also needs the training corpus, unavailable for most released models, so only the developer can run it.

Canary strings are unique identifiers embedded in a benchmark’s distribution so that a model reproducing the string demonstrably saw the file, which is a genuine positive signal and a weak negative one, since any preprocessing that strips or reformats the text removes the canary while keeping the content.

Behavioral tests

Behavioral tests, such as comparing performance on a benchmark against a freshly written equivalent, or checking whether the model can reproduce a test item given its prefix, provide additional evidence without corpus access, and none of these methods establishes cleanliness. The honest statement is that contamination can be demonstrated and cannot be ruled out.

Structural answers

Held-out and freshly generated evaluations are the structural answer. A private test set never released in any form eliminates the mechanism, at the cost of a trusted party to hold it and run the evaluation, which is what several evaluation organizations now operate.

Benchmarks refreshed from material created after a training cutoff give a clean measurement with a limited lifespan, and dated competition problems serve well. Generating items from templates gives unlimited unseen instances at the cost of a narrower notion of difficulty. Both cost more than downloading a public benchmark, and the public ones persist anyway.

Benchmark decay

A public benchmark has a limited useful life once widely reported, which is a fact of the format rather than a failure. The decay has several sources: items enter training corpora, the community tunes against the reported numbers, and headroom shrinks until the residual differences are label noise.

Treat a benchmark’s age as information, prefer evaluations postdating the training cutoff, expect the most cited benchmarks to be the most contaminated because they are most cited, and plan for replacement. A score on a five-year-old public benchmark measures something, but not capability on unseen problems.

Reproducibility crisis

The gap between published results and independently reproduced ones.

Unreported tuning, seed selection, and comparison against undertuned baselines compound. The proposed method receives extensive search while baselines use defaults.

Expensive training discourages independent reruns, leaving reviewers to assess descriptions rather than check results. Errors can persist through later work that reuses the setup.

A published improvement can be real, lucky, or an artifact of unequal effort, and the paper often does not give enough information to tell which one. The reproducibility crisis in machine learning is that gap between reported numbers and independently checked results. Expensive training makes the problem worse, because the most influential experiments are often the least likely to be rerun by reviewers or competitors.

Sources

The sources are unreported tuning, seed selection and an undertuned baseline, and they compound. Unreported tuning means the reported number is the maximum over an unstated hyperparameter search rather than the method’s expected performance. Seed selection means the best of several runs is reported without the run count, which can manufacture an effect from variance alone.

The undertuned baseline is the largest single source: the comparison runs on default settings while the proposal is tuned, and repeated studies find that baselines given equal effort close most or all of the gap. Recommendation systems, metric learning and language modeling have each had such studies, so assume a comparison is asymmetric unless both tuning budgets are reported.

Why errors persist

Compute cost makes independent replication rare, so errors persist longer than where checking is cheap. Reproducing a large training run costs enough that no reviewer will, so publication rests on the plausibility of the description. Most findings have never been checked, and a mistaken one can be built on for years, since follow-up work inherits the setup rather than testing it.

The Chinchilla reanalysis earlier on this page is an instance of the only kind of checking that is affordable, namely checking the analysis rather than the experiment, and it found something. The asymmetry also biases what gets checked, since small-scale results are verified frequently and the largest results, which are the most influential, are verified least.

Partial fixes

Reproducibility checklists and artifact review have positive, modest effects. Checklists asking whether code is available, how many runs were performed, what the search space was and what compute was used raise the rate at which those items are reported. Artifact evaluation, where a separate committee runs the code, raises the fraction of papers with working code and catches real discrepancies.

What neither does is verify the scientific claim, since running the authors’ code with the authors’ settings reproduces their numbers by construction and says nothing about whether the comparison was fair. The gap between a reproducible artifact and a reliable result is where the remaining problem sits, and closing it requires a reviewer to check the baseline’s tuning, which is work that the current review process does not fund.

Reading a result

Reading a result for what was held fixed is the skill that substitutes for replication. The questions that resolve most papers: what the baseline’s tuning budget was relative to the proposal’s, how many seeds ran and whether spread is reported, whether the evaluation set is one the method was developed against, whether ablations remove one component at a time, and whether the improvement exceeds the interval implied by the evaluation size.

A paper that answers these clearly is making a checkable claim whether or not the code runs. A paper that omits them is reporting a number without the conditions that give it meaning, and the appropriate response is to treat the effect as unestablished rather than to assume the omission is innocent. It usually is innocent, and that does not change what the reader can conclude.

Agent reliability

The compounding of per-step error across a multi-step task.

If each step succeeds independently with probability r and the task has n dependent steps, end-to-end success probability is r to the n. At 95 percent per step and 20 steps, success is about 36 percent.

The exponent is what the user experiences, not the per-step accuracy that gets demonstrated. Improving from 95 to 99 percent per step raises a 20-step task from 36 percent to 82 percent.

An agent demo can look smooth because every visible step worked in that run. A deployed task has to survive the whole chain, including the steps the demo did not show failing. Agent reliability is the compounding of those step-level risks across a dependent sequence. The useful number is end-to-end success, because even a high local accuracy can collapse once many unrecovered errors have a chance to appear.

P = r^{n}

The chance a task with no recovery succeeds end to end. At 95 percent per step and 20 steps this is about 36 percent; verification matters more than raw step accuracy.

symbol	meaning
$P$	the probability the whole task succeeds
$r$	the probability a single step succeeds
$n$	the number of dependent steps

Compounding math

Compounding is unforgiving, and the step count at which a high per-step rate stops being enough is lower than intuition suggests. At 99 percent per step, a 10-step task succeeds 90 percent of the time, a 50-step task 61 percent, a 100-step task 37 percent. At 95 percent: 60 percent, 8 percent, under 1 percent.

Per-step accuracy gets demonstrated; the exponent is what the user experiences. Four points of per-step accuracy is a fivefold cut in per-step failure, from one in twenty to one in a hundred, and on the twenty-step task it takes end-to-end failure from 64 percent to 18 percent, which is why the improvement is so leveraged. The model assumes independent steps with no recovery, which is pessimistic and the right starting point, because it identifies what the system must do to beat it.

Breaking compounding

Error recovery and verification break the compounding; a system without them is bounded by the formula. Recovery means a failed step is detected and retried or routed around, which converts a fatal error into a cost, and the relevant rate becomes the probability that a step fails and the failure goes undetected.

Verification supplies the detection and its quality sets the ceiling. A checker catching most errors turns a long chain into short recoverable segments; one that misses errors silently leaves the compounding intact. This is why verifiable domains support long chains and unverifiable ones do not.

Prefer verifiable steps

The design consequences are to prefer architectures where steps are checkable, to checkpoint state so a retry does not restart the task, and to make failures loud: an undetected wrong step is worse than a detected failed one because it corrupts everything downstream.

Demonstration limits

Demonstrations are a weak evidence base for reliability claims, and the reason is selection. A demonstration shows one successful run of a task the demonstrator chose, from an unreported number of attempts, on an example likely selected for working.

A system succeeding 20 percent of the time produces a flawless demonstration within a handful of attempts, indistinguishable from one produced at 95 percent. That accounts for much of the gap between what agent systems appear to do in public and what they do in deployment. A demonstration cannot convey the success rate, the variance or the failure modes.

What to report

The whole task is the only number a user experiences. Nobody receives 95 percent of an outcome: the task completed correctly or it did not, so the reportable metric is end-to-end success over a representative set, with item count and trials per task stated so the interval can be computed.

Useful reporting adds where failures occur, the rate of silent against detected failures, and the cost of successful runs including retries, since a system that succeeds after twenty attempts is a different product from one that succeeds on the first. The trap is optimizing step accuracy because it is easier to measure and improves smoothly.

Primary literature

Neural Architecture Search with Reinforcement LearningZoph and Le, 2017, ICLR
Used a recurrent controller trained with reinforcement learning to generate network descriptions, producing a CIFAR-10 architecture that matched the best hand-designed ones at 3.65 percent test error.
Deep Double Descent: Where Bigger Models and More Data HurtNakkiran et al., 2019, ICLR 2020
Documented the double-descent curve where test error rises then falls again as model capacity increases past the interpolation threshold.
Denoising Diffusion Probabilistic ModelsHo, Jain and Abbeel, 2020, NeurIPS
Showed that iteratively denoising a sample from Gaussian noise following a learned reverse process generates images competitive with GANs.
Language Models are Few-Shot LearnersBrown et al., 2020, NeurIPS
Showed that a 175-billion-parameter autoregressive model performs tasks from a few examples placed in the prompt, without gradient updates.
Scaling Laws for Neural Language ModelsKaplan et al., 2020
Found power-law relationships between test loss and model size, dataset size, and compute, with model size yielding the steepest returns.
Training Compute-Optimal Large Language ModelsHoffmann et al., 2022, NeurIPS
Showed that earlier scaling-law work had undertrained on data relative to parameters and that compute-optimal training requires scaling tokens and parameters roughly equally.
Reconciling Kaplan and Chinchilla Scaling LawsPearce and Song, 2024, TMLR
Attributed much of the gap between the Kaplan and Chinchilla compute-optimal exponents to Kaplan counting non-embedding rather than total parameters at small scale, since simulating the Chinchilla study under those conditions gave coefficients close to Kaplan's.
Resolving Discrepancies in Compute-Optimal Scaling of Language ModelsPorian et al., 2024, NeurIPS
Traced the Kaplan and Chinchilla discrepancy to uncounted output-layer compute, a warmup too long for small models and optimizer settings not tuned per model size, and recovered the Chinchilla exponent with a constant learning rate, showing that careful learning rate decay is not essential to it.

Sources

The Bitter Lessonincompleteideas.net (Sutton, March 13, 2019)
Chinchilla Scaling: A replication attemptarXiv (Besiroglu, Erdil, Barnett, and You, 2024)
Attention Is All You NeedarXiv (Vaswani, Shazeer, Parmar, and coauthors, 2017)
Reconciling modern machine-learning practice and the classical bias-variance trade-offPNAS 116 (Belkin, Hsu, Ma, and Mandal, 2019)
Grokking: Generalization Beyond Overfitting on Small Algorithmic DatasetsarXiv (Power, Burda, Edwards, Babuschkin, and Misra, 2022)
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural NetworksarXiv (Frankle and Carbin, 2018)
Are Emergent Abilities of Large Language Models a Mirage?arXiv (Schaeffer, Miranda, and Koyejo, 2023)
Language and Machines: Computers in Translation and LinguisticsNational Research Council, Automatic Language Processing Advisory Committee (1966)
Artificial Intelligence: A General SurveyScience Research Council (Lighthill, 1973)
Formalizing Fermat's Last TheoremAnthropic
The Hardware LotteryarXiv (Hooker, 2020)