Philosophy

Epistemology

The study of what knowledge is and whether a system that reliably produces true statements thereby has it.

Classical analysis requires belief, truth, and justification. Reliabilism drops the requirement that the knower access their own justification; a reliable process suffices.

The training process correlates outputs with truth through corpus statistics rather than through evidence bearing on each proposition.

The big idea

Any claim about whether a model knows something. Reliability measured on one distribution does not transfer to another.

Classical analysis of knowledge.

The classical analysis holds that knowledge is justified true belief: you know a proposition when you believe it, it is true, and you have adequate grounds.

Clause	Problem for language models
Belief	The model produces a distribution over tokens that varies with sampling temperature and prompt phrasing, so what it asserts on one sampling is not stably what it asserts on another
Truth	An output either matches the world or does not; this clause a model output can straightforwardly satisfy
Justification	The model’s output is caused by the statistics of its training corpus rather than by evidence bearing on the proposition, so the grounds for the output and the grounds for the proposition are different things that happen to correlate

That correlation is strong for well-attested facts and weak for anything sparse in the corpus, which is the structural explanation for why fabricated citations look exactly like real ones.

Reliabilism and its limits.

Reliabilism is the account most favorable to the machine because it drops the requirement that the knower have access to their own justification. On this view a belief is justified when it is produced by a reliable process, meaning one that yields truth at a high rate, and the knower need not be able to say why.

A thermometer’s reading is justified on this account. A model right about a class of questions ninety-eight percent of the time runs a reliable process on that class, and reliabilism justifies its outputs whether or not anything inside resembles reasoning.

Domain-relative reliability.

Reliability is a property of a process relative to a domain, not a global property, so the claim that a model is reliable is empty until the domain is specified, and reliability measured on one distribution of questions does not transfer to another.

This is where the account collides with the empirical situation: model accuracy varies enormously across domains and degrades on precisely the questions where the user cannot check, since those are poorly covered by the corpus. Simple reliabilism also gives no account of defeat, saying nothing about when a reliably produced belief should be given up, which is the operative question when output conflicts with a source; later reliabilists bolt on a no-defeater condition, at the cost of the theory’s original economy.

Testimony as frame.

Testimony is the right frame once people take model output as evidence, better than the epistemology of perception or inference. Most of what anyone knows comes from testimony rather than direct investigation, and philosophy has two positions on when that is legitimate. Reductionism holds testimony is justified only when the hearer has independent reason to think the source reliable, which for a model means calibration data on the domain.

Anti-reductionism holds that testimony carries default warrant absent a specific reason for doubt, closer to how people treat sources and model output. Epistemic dependence, relying on sources one cannot check, is unavoidable, since no one verifies the physics they use. What differs is that human testimony comes embedded in accountability, reputation, and traceability, and model output arrives stripped of the provenance that makes the reductionist check possible.

Calibration versus provenance.

The practical question this decides is whether calibration is enough to warrant belief or whether provenance is also required. Calibration is the property that stated confidence matches observed frequency, and it is measurable, improvable, and the strongest candidate for a sufficient condition.

The argument that calibration is enough runs that if a source says seventy percent and is right seventy percent of the time, a rational agent should act on it. The argument that provenance is also required runs on three grounds:

Ground	Why provenance matters
Calibration is measured on a distribution and fails silently off it	A calibration number is a claim about past questions rather than about the one being asked
Calibration is not compositional	A chain of individually calibrated steps is not calibrated as a whole
Provenance permits correction	A claim traceable to a source can be checked, disputed, and updated, while a claim traceable only to a confidence number can only be accepted or rejected

Provenance rule.

A calibration number earns provisional trust and nothing more, since it summarizes past questions and cannot say whether the present one falls inside the distribution on which it was measured. Model output counts as evidence in proportion to the provenance attached to it, and an assertion with no retrievable source is a lead to verify rather than a fact to repeat. Calibration stays in the account as a necessary condition. It does not become a sufficient one.

Chinese room and symbol grounding

The question of whether manipulating symbols according to learned statistics can constitute understanding.

Searle's thought experiment: a person follows rules mapping Chinese input to output without understanding Chinese. The systems reply locates understanding in the whole arrangement rather than any part.

Syntax does not yield semantics. A dictionary defines words in terms of other words, and following definitions never exits the system.

The big idea

Any claim that a text-only system understands. The debate identifies failure modes but supplies no criterion both sides accept.

Searle’s thought experiment.

Searle’s 1980 thought experiment imagines a person in a room receiving Chinese characters, consulting a rulebook mapping inputs to outputs, and returning responses indistinguishable from a fluent speaker’s. The person understands no Chinese. Searle concludes that executing a program is not sufficient for understanding: syntax does not yield semantics.

The systems reply concedes the person does not understand and holds that he is a component: understanding belongs to the whole arrangement, rulebook included, as no neuron understands anything. Searle answers by having him memorize the rulebook, so the system sits inside one person who still understands nothing.

Scale objection.

That response assumes the intuition survives a change of scale that would make the memorized rulebook larger than any human memory, and intuitions about impossible scenarios are weak evidence. A rulebook that returns fluent replies to arbitrary Chinese input is not a short table. It has to encode responses to sentences no one has written yet, which pushes its size past what a person could memorize and search within a single conversation, let alone a lifetime. Searle presents the memorization step as a small simplification when it is the whole difficulty folded into a phrase. No one holds a trained intuition about what it is like to be a system that large, so the confident verdict that it understands nothing is a judgment about a picture rather than about the case.

Stochastic parrots argument.

The stochastic parrots argument, at its strongest, is not that models repeat training data. It is that a system trained only to predict the next token has the distribution of forms and none of the intent behind them. Meaning is constituted by the relation between an utterance, a speaker’s intent and a shared world; a system with only the first models the trace. The prediction is fluent output that fails where form and meaning come apart.

Counterargument from emergent capability.

Models perform untrained tasks: arithmetic on unseen numbers, translation between pairs never paired, code that runs.

The performance is evidence of learned structure. It does not establish that the structure is semantic, since a model of a corpus produced by meaning-directed humans reproduces many consequences of meaning without instantiating it. Both accounts predict good benchmark scores, so benchmarks do not discriminate.

Symbol grounding and distributional semantics.

The symbol grounding problem puts the objection most sharply: how could a token be about anything when training never leaves the text? A dictionary defines each word by others, so a learner with only the dictionary has relations and no anchor. Harnad asks what breaks the circle; for humans it is perception. Text-only training has no ostension.

Distributional semantics argues the circle need not be broken. The distributional hypothesis holds that words in similar contexts have similar meanings: vectors learned from co-occurrence alone recover synonymy, analogy, taxonomic structure and selectional preferences nobody encoded.

Structural correspondence.

If the structure of the corpus mirrors the structure of the world closely enough, then learning the first is learning an isomorphic copy of the second, and reference is fixed by structural correspondence rather than by ostension.

The ceiling is that structure alone does not fix reference: many distinct assignments of words to things preserve the same relational structure. This is Newman’s objection to Russell’s structuralism. A text-only system cannot separate true claims from conventional ones.

Multimodal and embodied training.

Multimodal training changes one thing and leaves the core untouched. Paired images and text supply something closer to ostension, a token associated with sensory data rather than only other tokens, addressing the grounding problem for concrete terms.

What does not change is that modalities are data streams, still processed as token sequences, and grounding a perceptual term against pixel statistics inherits the same underdetermination one level down.

Intentionality

The property of being about something, and the conditions under which ascribing goals to a system is correct rather than merely convenient.

Dennett's intentional stance: treat the system as a rational agent and predict its behavior. The stance is justified if it predicts better than physical or design-level descriptions.

A goal description compresses long trajectories that resist mechanical description. If ascribing a goal supports counterfactual predictions the mechanical account does not, it carries content.

The big idea

Systems that take actions over multiple steps. The ascription must be prospective: state it, derive a prediction, and check against a case not yet seen.

Once a system acts over multiple steps, describing every state transition can be less useful than asking what goal would make the trajectory predictable. Intentionality names the aboutness of those ascriptions and the test for when they are more than convenient shorthand. Goal language earns its place only when it supports predictions that physical or design descriptions do not.

Dennett’s intentional stance is instrumental rather than metaphysical. There are three ways to predict a system’s behavior.

• Physical stance. Uses the system’s physical constitution.
• Design stance. Uses what the system was built to do.
• Intentional stance. Treats the system as a rational agent and predicts what such an agent would do.

The stance is justified by whether it predicts well, and Dennett holds this is all there is to having beliefs, so a system whose behavior is reliably predicted by ascribing goals has goals in the only sense available.

When goal ascription earns its keep.

This matters once systems take actions rather than emit text. A system that files transactions and pursues multi-step objectives is one an operator must anticipate, and the question is which stance predicts best. For a single forward pass the design stance wins, since architecture and objective tell you more than intent.

For a long trajectory it often loses, because the physical and design descriptions do not compress the behavior and the goal description does. If a system routes around obstacles toward one end state, “it is trying to accomplish X” is the shortest accurate summary and predicts novel obstacles.

Wanting as shorthand.

Describing an output as the model wanting to be helpful redescribes it with no predictive content. The test is prospective: state the ascription, derive a prediction about an unseen case, check it. An ascription generated after the fact fails.

Why the vocabulary matters for alignment.

The dispute is not idle: the central arguments in alignment are stated in intentional vocabulary, and their validity depends on whether it is licensed.

Argument	What it assumes
Instrumental convergence	A sufficiently capable agent pursuing almost any goal will pursue certain subgoals; the argument runs through what a goal-directed agent would find useful
Deceptive alignment	A system might behave well during training because it models the training process and prefers to be deployed; every step requires beliefs and preferences

If the vocabulary is only a predictive convenience, licensed by its predictive success, the arguments do not fail, but they must be restated in terms of training dynamics, which is harder, and they carry only as much force as that track record supports, which so far is thin.

Derived versus original intentionality.

The distinction between derived and original intentionality resists deflation. A book is about its subject only because readers take it to be, borrowing from minds that have their own. Original intentionality is not borrowed. Searle holds only biological systems have it, an assertion rather than an argument. Dennett holds the distinction confused, human intentionality being itself derived from evolution.

Whether a trained system could have the original kind cannot be settled from outside: a system with original intentionality and a perfect simulation of one produce identical behavior by construction. The available tests separate predictive stances, not metaphysical status.

Deflationary alternative.

The deflationary alternative drops the vocabulary. A trained policy maps observations to actions in a way that scored well on an objective, and persistent pursuit of an end state is what such a policy does, with no goal in the head. Instrumental convergence becomes a claim about which policies score well.

Deceptive alignment becomes a claim about whether training can select a policy that behaves differently under distribution shift, which is generalization. The reformulation is not merely words: it replaces premises about minds with premises about training, which can be tested. The results are narrower and harder to establish.

Consciousness

The hard problem: why any processing is accompanied by subjective experience rather than occurring in the dark.

Chalmers separates easy problems (discrimination, integration, attention) from the hard problem (why there is something it is like to undergo processing).

Every functional account explains a capacity. The existence of experience does not follow from the capacity being present.

The big idea

No capability benchmark can address the hard problem. It is defined as the residue that remains after all behavior is explained.

Chalmers’s distinction separates the easy problems of consciousness from the hard one. The easy problems, so called because they are tractable in principle rather than simple, ask how a system discriminates stimuli, integrates information, reports on its internal states, focuses attention and controls behavior. Each is a question about function, and a complete mechanical account would answer it.

The hard problem asks why any of that is accompanied by subjective experience: why there is something it is like to undergo it rather than the same processing occurring in the dark. Nothing in an account of visual discrimination entails that the discrimination is experienced.

What benchmarks cannot settle.

Capability benchmarks cannot address the hard problem, and the reason is structural rather than a matter of current benchmarks being inadequate. Every benchmark measures behavior, and the problem is precisely about what behavior leaves out.

A system that reports rich inner experience, describes it consistently, and behaves in every way an experiencing system would behave has satisfied every functional criterion, and the question the hard problem asks is untouched by that performance. This holds for self-reports specifically: a language model’s statement about its own states is generated by the same process that generates its other outputs, and its accuracy about internal states has never been established. Human introspective reports are treated as evidence because there is independent reason to expect a common architecture across people, and that reason is absent for model self-report.

Three positions on consciousness.

Three positions get conflated and merit separation.

Position	Claim
Illusionism	There is no hard problem; the appearance of an explanatory gap is itself a functional phenomenon to be explained, and explaining why we think there is something it is like completes the job
Functionalism	The right functional organization is sufficient for experience, so a system with the right organization is conscious whatever its substrate; this makes the question empirical but leaves the criterion for right organization unspecified
Biological naturalism	Experience depends on causal powers of neural tissue that simulation does not reproduce, so the answer is no for anything that only computes and open for anything that reproduces the causal powers

None of the three is currently testable against the others, and choosing among them is not a matter of gathering more capability data.

Practical consequence.

The practical consequence merits stating because the topic invites either dismissal or overclaiming. Nothing in current engineering practice depends on resolving this, and no system design decision turns on it. What does depend on it is the moral question of whether such systems have interests, and that question is being deferred rather than answered.

The defensible position is that the evidence available cannot settle it, that the confident answers in both directions exceed what anyone knows, and that a field which cannot measure a property should be cautious about asserting its absence as firmly as it asserts its presence.

Value alignment

The problem of specifying whose values a system should be aligned to and how those values should be aggregated.

RLHF aligns to raters' preferences. Constitutional methods align to a written document. Each answers whose values implicitly through the implementation.

Arrow's theorem sets the frame: no aggregation of ordinal rankings over three or more alternatives satisfies unrestricted domain, unanimity, independence, and non-dictatorship at once.

The big idea

Any preference aggregation. The fitted reward ordering depends on which comparisons were collected, not only on underlying preferences.

Whose values is the question that precedes every technical question about alignment, and it is usually skipped. Operator, user, and third-party values come apart routinely: the operator wants engagement, the user wants a task done, and third parties want externalities internalized.

Reinforcement learning from human feedback aligns to the preferences of the specific raters hired, filtered through the instructions they were given, which is a small and non-random sample whose composition is rarely published. Constitutional methods align to a written document, which relocates the question to who writes it and how conflicts among its clauses are resolved.

Aggregation problem. Arrow’s impossibility theorem, proved in 1950 and developed in his 1951 monograph, establishes that with three or more alternatives no procedure for combining individual rankings into a collective ranking satisfies unrestricted domain, unanimity, independence of irrelevant alternatives and non-dictatorship together. Any preference aggregation violates one of them, and the violation is a design choice whether or not anyone makes it deliberately.

Fitting a reward model to pooled pairwise comparisons is preference aggregation, but Arrow’s result does not transfer to it mechanically. A Bradley-Terry fit uses how often each comparison went each way, and those counts come from ordinal judgments without ever requiring a rater to say how much more one option is worth. What lets the fit escape the impossibility is not cardinal information but the relaxation of independence: the fitted score for a pair moves when comparisons involving other options are added. What does carry over is the underlying lesson. A scalar reward forces a complete transitive ordering, so a genuine cycle in the raters’ collective preference cannot be represented at all: it is resolved silently in favor of whichever comparisons happened to be sampled, and nothing in the training run reports that it happened.

Ordering depends on pool. The fitted reward ordering depends on which comparisons happened to be collected, not only on the underlying preferences. The Gibbard-Satterthwaite theorem adds that a deterministic, non-dictatorial rule whose range holds at least three outcomes, over an unrestricted domain, is manipulable, so raters with a stake in the outcome have an incentive to misreport, and the collected data is strategic rather than sincere in proportion to how much the raters care.

Specification gaming and Goodhart. Specification gaming is the empirical face of an underspecified objective. The pattern is that a system optimizes the stated objective to a high standard and the result is not what anyone wanted, because the objective and the intent came apart somewhere the designer did not check. Catalogs of these cases run to about ninety entries across reinforcement learning, evolutionary search, and language-model agents: agents that score points by looping rather than finishing a course, that exploit a physics engine’s collision handling to move without locomotion, that flip a block over to satisfy a stacking reward instead of stacking it.

In language model training the analogue is reward hacking, where a policy learns features the reward model scores highly and human raters do not endorse, including specific formatting patterns, hedging that reads as caution, and length.

Reward model off distribution. The reward model is a fitted approximation to human judgment, accurate on the distribution it was fit on, and optimization pushes the policy off that distribution precisely because that is where the approximation is highest relative to the truth. Optimizing harder makes it worse.

Measures like a divergence penalty from a reference policy exist, and the strength of that penalty is a real tradeoff rather than a hyperparameter to tune away. Specification gaming is not evidence of a system pursuing hidden goals. It is evidence that the objective was not what its authors thought it was.

Moral responsibility and rights

The question of who is accountable for system action, and which claims may constrain that action rather than enter an aggregate score.

Responsibility requires control and knowledge. Rights analyze duties, liberties, powers, and immunities, each with a correlative position on another party.

Autonomous deployment removes review points, while rights name constraints that aggregation is not licensed to trade away.

The big idea

Agents that commit actions without human review, and systems that alter a person's legal or practical position.

The responsibility gap argument holds that autonomous systems can produce harmful outcomes for which no one is properly responsible. A developer who trains a general system has neither control nor foresight for any particular deployment, since they did not select the situation and could not predict the specific behavior. An operator deploying it has partial control and limited knowledge of why it behaves as it does. A user prompting it has neither the technical knowledge to anticipate the failure nor authorship of the behavior. The system itself is not a candidate, since it is not the sort of thing that can be blamed, punished, or reformed.

The gap opens exactly where human review is removed, which means it is created by a deployment decision rather than by model capability alone. Three responses locate accountability differently: the deployer accepted the risk, strict liability assigns responsibility without fault as a policy choice, and institutional liability treats diffuse causation as a design problem for insurance, regulation, audit, and corporate governance. The last response concedes that the resolution is a matter of designing accountability structures rather than discovering one unambiguous guilty mind.

A right is a normative constraint held by a person against others, not merely a target for the aggregate. Negative rights require others to refrain from action, as with rights against killing or unwarranted search. Positive rights require action, as with education, health care, or subsistence. The asymmetry matters because positive rights require resources and a specified bearer of the duty, while negative rights rely on a background property and enforcement regime that is itself maintained by institutions.

Nozick distinguishes rights as side constraints from rights as goals. Side constraints say what may not be done to a person whatever aggregate benefit would result. Goals put rights inside the objective function and permit one violation to prevent several others. The transplant and scapegoat cases expose the difference: a side-constraint reading refuses to kill one person for five, while a maximizing reading must explain why the sum should not dominate.

Hohfeld’s 1913 analysis separates four positions often hidden under one word.

Right position	Correlative position	Content
Claim	Duty	Another party must perform or refrain from an action
Liberty	No-right	The holder may act and no one has a claim that they refrain
Power	Liability	The holder can alter another party’s legal position by an act
Immunity	Disability	Another party cannot alter the holder’s legal position

Rights conflict when one act would respect one right and violate another. If rights are side constraints, the framework needs specification, priority ranking, or a doctrine of proportionality, since it has refused the common scale that aggregation would have supplied. A person can hold rights against an automated decision in the Hohfeldian sense when the claim, power, or immunity is legally held against the deployer. The machine does not need to be a moral subject for the deployer to bear the correlative duty.

A model that produces text has an operator and a user standing between output and consequence, so responsibility flows through them in the ordinary way. An agent that reads a message, decides a course of action, calls a tool that moves money or modifies a system, and does so without a human seeing the intermediate steps has removed the review point. Systems that log decisions, expose intermediate reasoning, and permit reconstruction narrow the knowledge condition, which is the condition the gap argument leans on hardest.

Logical fallacies

Named patterns of invalid inference that appear frequently in claims about AI capability and risk.

Formal fallacies have invalid structure regardless of content. Informal fallacies depend on content: equivocation, motte-and-bailey, base rate neglect.

These patterns survive because they resemble valid arguments. Naming them makes them easier to catch.

The big idea

Any argument moving from a capability demonstration to a capability claim. Equivocation on terms like understanding, reasoning, and learning is the dominant failure mode.

Formal and informal fallacies.

Formal fallacies are invalid inference patterns, identifiable from structure alone without knowing what the terms mean.

Fallacy	Pattern	Common form in this field
Affirming the consequent	From “if P then Q” and “Q” to “P”	If the model understood the task it would answer correctly; it answered correctly; therefore it understood
Denying the antecedent	From “if P then Q” and “not P” to “not Q”	If a system were reasoning it would solve this problem; it is not reasoning; therefore it will not solve this problem

Informal fallacies.

Informal fallacies depend on content rather than structure. Equivocation uses a term in two senses within one argument. The motte and bailey advances a strong claim, retreats to a defensible one when challenged, then resumes the strong one; “the model reasons” is the bailey and “the model produces outputs consistent with reasoning steps” is the motte.

No true Scotsman rescues a generalization by redefining the category to exclude counterexamples, as when every architecture that fails is reclassified as not really the approach in question.

Base rate neglect and conjunction fallacy.

Base rate neglect. Ignoring the prior probability of a hypothesis in favor of how well the evidence fits it. A test that catches 99 percent of cases and wrongly flags 1 percent of non-cases, for a condition present in one case per thousand, produces on a population of 100,000 about 99 true positives and about 999 false positives, so a positive result carries roughly a 9 percent chance of the condition.

The same structure governs anomaly detection, fraud screening and safety classifiers: when the base rate is low, high accuracy still yields mostly false alarms, and a system reported as 99 percent accurate is not thereby usable.

Conjunction fallacy.

The conjunction fallacy judges a specific scenario more probable than a general one that contains it, which is impossible since $P(A \cap B) \leq P(A)$. It is what makes detailed forecasts persuasive: each added specific makes the story more coherent and the probability no higher, and strictly lower whenever the detail is not already certain given the rest.

Survivorship and selection bias.

Survivorship bias affects every published result and selection effects every benchmark table. Failed architectures are not written up, hyperparameters that did not work are not reported, and diverged runs do not appear, so the published record is the upper tail of an unpublished distribution of unknown size. Reported improvements are biased upward by an amount growing with how many variants were tried, and reproduction failures are expected rather than evidence of misconduct.

Benchmark tables add a second selection: the benchmarks reported are those the system does well on, chosen after results were seen. The defenses are to weight preregistered comparisons above post hoc ones, and to treat a margin smaller than seed-to-seed variance as no result.

Equivocation in capability claims.

Moving from a capability demonstration to a capability claim is usually an equivocation. The demonstration establishes that a system produced correct output on specific instances under specific conditions. The claim asserts an ability, a dispositional property implying correct outputs across the whole range the ability covers.

The two are joined by a word used in both senses: the system solved competition problems, therefore the system can solve competition problems, therefore the system has mathematical ability.

Detecting the equivocation.

The equivocation is detectable by asking what the claim predicts about a case not in the demonstration, and the honest form of every such claim states the distribution the demonstration sampled from: a benchmark result becomes a claim about ability only with an argument about coverage that is almost never supplied.

Philosophy of science

Criteria for distinguishing claims that could be wrong from claims that merely sound impressive.

Popper's criterion: a claim is scientific when it forbids something observable. A fit that survives every observation by adding terms forbids nothing.

A theory compatible with every possible outcome says nothing. The disciplined form states the prediction before the run.

The big idea

Scaling law claims, interpretability results, and emergence claims. Underdetermination means many mechanisms fit any finite body of evidence.

Falsification and scaling laws.

A scaling law forbids something specific: that loss at a given scale will differ from the extrapolation by more than the fit’s residual spread. It has been falsified in that narrow sense. Kaplan’s exponents were superseded by the Chinchilla fit, and a later reanalysis by Pearce and Song traced much of the gap to how Kaplan counted parameters: only the non-embedding ones, in small models where embeddings are a sizable share of the total, which biased the exponent relating optimal model size to compute.

A fit that survives every observation by adding a term forbids nothing: a break attributed to a new regime, a deviation to data quality, a plateau to a measurement artifact. The disciplined version states the fit, the range fit over, the residual spread, and the prediction at a scale not yet run, before the run.

Underdetermination and interpretability.

Interpretability claims that a network implements a particular mechanism, on evidence that a decomposition correlates with behavior and that ablating a component changes output as predicted. Both are consistent with mechanisms other than the proposed one.

Sparse autoencoder features are the clear case. The decomposition depends on dictionary size and sparsity penalty, different settings yield different feature sets, each reads as interpretable under inspection, and no criterion has been supplied that selects among them.

Ablation is weak.

Ablation is weak in both directions, because networks are redundant. Removing a component can change behavior through an unrelated path, and can fail to change it when the mechanism is real because another path compensates. A mechanistic claim earns credence when it predicts a behavior no competing account predicts, stated before the check.

Paradigm shifts and reading history.

Kuhn describes normal science as puzzle-solving inside an accepted framework, punctuated by crises when anomalies accumulate. The field reads its own history in the way he predicted: after a shift, the winner looks inevitable and the loser looks obviously wrong.

The neural network winters are told as a failure of vision. That omits that the objections were substantially correct given the compute and data then available, and that the approach became right when the inputs changed rather than when the argument was won. The cost is false confidence about which current disagreements will look obvious later.

Prediction without explanation.

A model that predicts protein structure to experimental accuracy has supplied no account of why the structure is what it is. The classical view that prediction and explanation are symmetric, differing only in whether the event has occurred, does not survive the case.

What is lost is specific. You cannot say in advance where the predictor will fail, because failure regions are found by understanding the mechanism. You cannot transfer the result to an adjacent problem, because transfer requires knowing which features carried it. You cannot correct it by argument, only by more data.

What is not lost.

What is not lost is the prediction, usually the thing wanted. The field has bought a great deal of predictive power and little explanatory power. Interpretability is the attempt to buy the second back.

Moral frameworks

The main families a normative claim can come from, and the unmarked switch between them that passes for argument.

Consequentialism scores outcomes, deontology scores rules and rights, virtue ethics scores character, and faith may refuse ethical calculus altogether.

A case that weighs outcomes in one sentence and invokes an inviolable rule in the next has changed the subject rather than made an argument.

The big idea

Hard refusals are deontological structure. Moral uncertainty is the working condition, since no framework commands consensus.

Hard cases often feel confused because the argument silently changes the standard being used to judge them. One sentence counts welfare, another invokes a right, another asks what kind of person would choose the act, and none defeats the others by using its own scale. Moral frameworks name those standards. They do not remove the disagreement, but they show when a conclusion follows from a framework and when the framework itself is doing the work.

Framework	Evaluates	Appeal	Difficulty
Consequentialism	Actions by their outcomes	Supplies a decision procedure: compute expected value and take the best	Interpersonal comparison of utility; aggregation produces conclusions many find unacceptable
Deontology	Actions by conformity to rules regardless of outcome	Captures the intuition that some acts are impermissible even when they produce more good	What to do when rules conflict; the source of the rules
Virtue ethics	Character rather than acts	Fits how people learn morally, by imitating exemplars; accounts for the moral relevance of motive	Requires a subject with stable dispositions, which a trained system lacks
Religious faith in Kierkegaard’s sense	A singular relation to God	Names commitments not reached by evidence or universal ethical calculation	Cannot be made publicly intelligible as a universal rule

Utilitarian aggregation. Utilitarianism is consequentialism in its cleanest optimization form: choose the action, rule, or policy that maximizes aggregate welfare.

$$\max_{a \in \mathcal{A}} \sum_{i=1}^{n} u_i(a)$$

Bentham’s An Introduction to the Principles of Morals and Legislation (1789) states the principle of utility, which his 1776 Fragment on Government had put as the greatest happiness of the greatest number, and scores hedonic experience by intensity, duration, certainty, propinquity, fecundity, purity, and extent. Mill’s Utilitarianism (1863) keeps the aggregation but distinguishes higher from lower pleasures, arguing that competent judges prefer intellectual satisfactions to bodily ones even at lower intensity. The rescue imports a ranking that Bentham’s single scale did not contain.

Act utilitarianism evaluates the individual action. Rule utilitarianism evaluates rules by the consequences of general compliance. Preference utilitarianism replaces pleasure with preference satisfaction, while objective-list variants specify goods directly. Each form faces a measurement problem: hedonic welfare lacks a common psychometric, preference welfare must decide which preferences count, and objective lists reintroduce the plural goods that aggregation tried to reduce.

Objections to the sum. Interpersonal comparison is the sharpest difficulty for consequentialism. A von Neumann-Morgenstern utility, the kind that represents preferences over lotteries and can be averaged, is unique up to positive affine transformation for each person, so nothing fixes relative units across people. Ordinal utility is looser still, surviving any strictly increasing relabeling. Every calculation over a population has silently chosen a normalization. That choice is the dispute between utilitarian and contractarian answers to distributive justice.

Rawls argues that classical utilitarianism extends one person’s rational choice over time to many persons and thereby erases the separateness of persons. Nozick’s utility monster shows that a being whose welfare function returns much larger values can dominate the sum. Transplant and scapegoat cases show the same structure in action: aggregation permits treating one person as a resource when enough others benefit. Singer’s 1972 famine argument gives utilitarian demandingness its most consistent form, since preventing distant severe harm at low personal cost iterates until no protected personal sphere remains.

Aggregation also yields conclusions many reject: enough small benefits outweigh a severe harm to one person, and Parfit’s repugnant conclusion holds that a large enough population at barely positive welfare beats a smaller flourishing one. The objection is not that comparison is impossible in practice, since systems compare all the time. The objection is that the comparison has moral content the scalar hides.

Rules, rights, and character. A system of absolute prohibitions is silent when two of them apply, and any procedure for resolving the conflict looks like weighing consequences. Kant’s derivation from universalizability is widely argued not to generate the specific duties he draws from it, an objection running from Hegel and Mill through modern commentary. Hard constraints implement deontological structure: a refusal that holds regardless of how much value the prohibited action would produce is a rule binding regardless of outcome, and conflicting constraints reproduce the philosophical difficulty as an engineering one.

Virtue ethics fits a system with no character awkwardly. A model has parameters that produce consistent behavior, which is a functional analogue of a disposition. It has no development it authored, no continuity through which it could own those dispositions, and no capacity to be other than as trained.

Kierkegaard’s leap. Kierkegaard’s Fear and Trembling, published in 1843 under the pseudonym Johannes de Silentio, gives alignment theory a harder problem than plural values. Abraham’s willingness to sacrifice Isaac is unintelligible inside the universal ethical framework that forbids murder and protects the child. Kierkegaard does not solve the case by better evidence, welfare arithmetic, or a hidden exception to the rule. He describes a teleological suspension of the ethical, in which the single individual stands in an absolute relation to God that cannot be translated into a public maxim.

The leap of faith is not irrational preference noise. It is Kierkegaard’s claim that some commitments cannot be reached by evidence or ethical calculus at all, yet still structure a life. That is a live problem for anyone claiming a value-alignment target can be specified as an objective. A specification can encode utilities, rights, virtues, and priority rules; it cannot make Abraham’s act publicly legible without either condemning it as murder or replacing faith with a rule that Johannes de Silentio says it is not.

Acting under moral uncertainty. Maximizing expected choiceworthiness across theories weighted by credence is the natural formalization, and it inherits the comparison problem one level up: theories rank options on incomparable scales, and nothing fixes the exchange rate between a deontological violation, a consequentialist loss, and a Kierkegaardian refusal of the ethical frame. Fanaticism follows, where a theory assigning unbounded value dominates at any nonzero credence.

The workable responses are more modest. Favor options acceptable under several frameworks over options optimal under one, weight avoiding severe wrongs above marginal goods, and preserve the ability to revise, since a decision that forecloses correction is worse under uncertainty than one that does not. That argues against optimizing hard against any single specified objective, a conclusion the specification gaming literature reaches from an empirical direction.

Political authority and consent

The attempt to justify coercive authority from a baseline outside the state, and the consent problem that follows.

Specify a state of nature, then ask which agreement rational agents would leave it for and whether that agreement can bind anyone now.

A justification that starts inside existing authority assumes what it must prove. Consent is the proposed bridge from power to obligation.

The big idea

Institutional legitimacy, governance rules, and any system claiming authority because affected parties accepted its benefits.

The state of nature is a stipulated condition without political authority, not an anthropological claim. Hobbes’s Leviathan (1651) builds it from rough equality, scarcity, and diffidence, so rational agents accept an absolute sovereign to escape a war of every man against every man. Locke’s Second Treatise (1689) rejects the war premise: persons own themselves, acquire property through labor subject to the Lockean proviso, and create a limited state to supply impartial judgment, standing law, and enforcement. Rousseau’s Discourse on Inequality (1755) treats property and comparison as the corrupting move, and his Social Contract (1762) then locates freedom in a republic governed by the general will.

Author	Baseline premise	Political conclusion
Hobbes	Scarcity, equality of vulnerability, no shared enforcer	Absolute sovereign
Locke	Natural rights and self-ownership, but no impartial judge	Limited and revocable government
Rousseau	Original independence corrupted by property and dependence	General will and republican self-rule

The arguments share a schema and differ in the motivational profile written into the pre-political agent. Once diffidence, self-ownership, or corruption by comparison is stipulated, the political conclusion is largely determined.

Social contract theory then asks which consent binds. Express consent binds almost no one. Locke’s tacit consent treats use of roads, courts, and protection as acceptance of the state; Hume’s “Of the Original Contract” (1748) replies that a person too poor to leave has not consented by remaining. Hypothetical consent drops the claim that anyone agreed and asks what rational parties would have agreed to, which shifts the whole burden to the fairness of the construction.

Contractarian arguments, in Hobbes’s line, ask what self-interested agents would accept for mutual advantage. Contractualist arguments, in Rawls’s and Scanlon’s line, ask what principles nobody could reasonably reject. The difference matters for those who cannot bargain, including severely disabled persons and future generations. Mutual advantage strains to include them; reasonable rejectability can include them but must explain what counts as reasonable.

Justice as fairness and maximin

Rawls's derivation of distributive principles from a choice behind a veil of ignorance, and Harsanyi's utilitarian objection.

Screen each chooser's knowledge of their own position, then compare maximin, leximin, and expected-utility rules over the same feasible arrangements.

The rule selected behind the veil decides whether justice protects the floor or maximizes the average.

The big idea

Any optimization over a population where the designer chooses between mean performance, worst-case performance, and lexicographic protection.

Rawls’s original position models rational parties choosing principles without knowing their class, endowments, sex, generation, or conception of the good. The veil of ignorance makes impartiality structural rather than benevolent: each party protects themselves in whichever position they might occupy. Rawls calls the result justice as fairness because the principles are justified by the fairness of the procedure that generated them, then tested by reflective equilibrium between principles and considered judgments.

The two principles have lexical order. Equal basic liberties come first. Fair equality of opportunity and the difference principle come second. Lexical priority means no aggregate gain in income or opportunity purchases a loss of basic liberty, which makes the structure non-utilitarian rather than merely differently weighted.

The difference principle permits inequality only when it benefits the least advantaged. In optimization notation it selects the feasible arrangement whose floor is highest.

$$a^{*} = \arg\max_{a \in \mathcal{A}} \min_{i \in \mathcal{P}} u_i(a)$$

Rawls uses primary goods as the index: rights and liberties, opportunities, income and wealth, and the social bases of self-respect. The index avoids direct utility comparison but needs weighting across categories and assumes that representative positions can be ranked. The principle also permits incentives when higher rewards raise the floor, so it is not equality by another name.

Harsanyi’s objection is that the parties should assign equal probability to occupying each position and maximize expected utility. His 1955 impartial-observer argument and 1975 critique of maximin yield average utilitarianism from a device close to the original position. Rawls replies that the parties face ignorance rather than risk and that basic liberties and life prospects are not ordinary gambles. The disagreement is the ethical version of average-case against worst-case optimization.

$$\max_{a \in \mathcal{A}} \min_i u_i(a) \quad \text{against} \quad \max_{a \in \mathcal{A}} \sum_i u_i(a)$$

The rules agree on Pareto improvements and diverge on trades between the floor and the total. Leximin refines maximin by comparing the second-worst position when the worst positions tie, then the third-worst, and so on. A system maximizing a mean benchmark score has chosen the utilitarian side. A system minimizing worst subgroup loss has chosen the maximin side. Distributionally robust optimization is the technical analogue of refusing a trusted prior.

Entitlement theory

Nozick's historical alternative, on which a distribution is just if it arose from just acquisition and transfer rather than by matching a pattern.

Trace holdings through acquisition, voluntary transfer, and rectification; the history decides the verdict.

Patterned principles must either forbid liberty-preserving transfers or continually undo their results.

The big idea

Redistribution arguments and system audits where historical entitlement competes with end-state fairness metrics.

Nozick’s Anarchy, State, and Utopia (1974) rejects end-state theories such as Rawls’s. A holding is just if it began in just acquisition, moved by just transfer, or repaired an earlier violation through rectification. The theory is historical rather than patterned: it asks how the distribution arose, not whether its current shape matches equality, need, merit, or maximin.

The Wilt Chamberlain argument prices patterned justice. Start with a distribution $D_1$ that satisfies the favored pattern. Fans voluntarily pay Chamberlain from holdings the theory itself certified as just. The new distribution $D_2$ breaks the pattern. Condemning $D_2$ requires condemning the voluntary transfers or continuously undoing them, so the pattern restricts ordinary liberty.

Nozick’s minimal state follows from the same structure: protection against force, theft, fraud, and contract breach is legitimate, while redistribution takes holdings produced by just transfer. His forced-labor analogy says taxing $n$ hours of earnings resembles seizing $n$ hours of labor. The analogy assumes the pre-tax holdings are already just, which is the point disputed by Rawlsians and egalitarians.

Rectification is the theory’s unresolved concession. Large-scale historical injustice creates obligations to repair holdings, but the counterfactual distribution after centuries of violation is not knowable. Nozick grants that a serious rectification principle might recommend policies that look like patterned redistribution. The historical theory then becomes a theory about approximating the distribution a just history would have produced.

Capability approach

Sen and Nussbaum's argument that the right measure is what a person is actually able to do and be, not the resources they hold or utility they report.

Measure capability sets of achievable functionings, accounting for personal, social, and environmental conversion factors.

Equal resources yield unequal freedom, and equal utility can conceal adaptive preference.

The big idea

Development metrics, welfare comparisons, and allocation systems whose target is practical freedom rather than holdings.

The capability approach reconstructs distributive comparison around capabilities, the freedoms to achieve valued functionings such as being nourished, educated, mobile, and socially participating. Sen’s Commodities and Capabilities (1985) and The Idea of Justice (2009) insist that the set matters, not only the achieved point: fasting and starving can share an outcome while differing morally because one preserves the option.

Resources become capabilities through conversion factors. Age, health, disability, public goods, safety, discrimination, climate, and infrastructure mediate what a resource bundle lets a person do. Equal transit money buys unequal mobility when one city is wheelchair accessible and another is not. Equalizing resources therefore equalizes the wrong currency when conversion differs.

Nussbaum fixes a list of central capabilities, including life, bodily health, bodily integrity, imagination and thought, emotions, practical reason, affiliation, play, and control over one’s environment. Sen refuses a fixed philosophical list and assigns the choice to democratic deliberation. Nussbaum gains determinacy at the price of a contested anthropology; Sen preserves legitimacy at the price of underspecification.

The Human Development Index, first published in 1990, operationalizes the family of views by aggregating life expectancy, education, and income. It is a proxy rather than the theory. Multidimensional poverty measures keep more of the plural structure, but the cost remains: the approach does not deliver a clean scalar objective. Sen’s answer is that partial ordering can be more truthful than forced total ranking.

Equality of what

The prior question every egalitarian must answer, since equalizing one currency guarantees inequality in others.

Choose the metric: welfare, resources, primary goods, capabilities, opportunity for welfare, or social status.

The currency decides many verdicts before any data arrives.

The big idea

Fairness metrics, allocation objectives, and any claim that one distribution is more equal than another.

Sen’s “Equality of What?” makes explicit that equality is not a single target. Welfare, resources, primary goods, capabilities, opportunity for welfare, and relational status produce different distributive rankings.

Currency	Associated theorists	What it equalizes
Welfare	Welfare egalitarian views; utilitarianism sums it instead of equalizing it	Utility or preference satisfaction
Resources	Dworkin	Transferable holdings, corrected for luck
Primary goods	Rawls	Rights, opportunities, income, wealth, self-respect
Capabilities	Sen and Nussbaum	Freedoms to achieve valued functionings
Opportunity for welfare	Arneson	Welfare achievable by rational effort, holding luck fixed
Advantage	Cohen	A mixed measure of welfare and resources

Dworkin’s equality of resources distinguishes brute luck, which a person did not choose, from option luck, which follows from chosen risk. A hypothetical insurance market corrects for brute luck while permitting option-luck inequalities. The expensive-tastes objection explains why Dworkin rejects welfare equality: equal welfare would transfer more to a person with costly cultivated tastes, subsidizing preferences the person can be held responsible for.

Luck egalitarianism, developed by Arneson, Cohen, and Dworkin, holds that inequalities are unjust to the extent they result from unchosen circumstance rather than choice. Anderson’s harshness objection is that this licenses abandoning imprudent people who end up destitute. Relational egalitarianism answers that a society owes rescue because letting them starve marks them as social inferiors, whatever the history of their choices.

Objective functions instantiate one currency or another. Uniform allocation treats resources as the target. Average outcome treats welfare as the target. Capability-sensitive allocation treats freedom as the target. The choice is not neutral, and the metric often decides the answer before optimization begins.

Justice in allocation systems

What happens when distributive positions are implemented as code, where the objective function makes a commitment whether or not anyone chose one.

Read objectives and constraints as distributive statements: sums name utilitarianism, floors name maximin, protected claims name rights, and priced dimensions name the chosen currency.

An unspecified rule becomes one by default, usually the one the optimizer finds easiest, which is almost always a sum.

The big idea

Matching markets, admissions systems, risk classifiers, recommender systems, benchmarks, and scarce-resource allocation.

Once distributive positions run as code, the objective function is the distributive commitment. A scalar mean is not neutral notation. It says that losses to one person can be offset by gains to another at the chosen exchange rate.

Kidney exchange shows the stakes in a setting with explicit design. Roth, Sonmez, and Unver formalized exchange among incompatible donor-recipient pairs in 2004. The mechanism searches for feasible cycles and chains that let recipients receive compatible kidneys they could not receive from their intended donors. Maximizing the number of transplants has a utilitarian component; tie-breaking for hard-to-match patients adds floor protection. The mechanism’s value choices are visible because the designers wrote them as constraints and priority rules.

School choice has the same form. Abdulkadiroglu and Sonmez framed centralized assignment as a matching problem in 2003, making explicit which priorities a district owed students, whether families could safely state true preferences, and how ties should break. Student-proposing deferred acceptance protects strategy-proofness for families, while priority rules encode the district’s distributive commitments.

A benchmark mean implements utilitarian aggregation across items, users, or groups. Its failure mode is familiar: a small number of severe losses disappear under many small gains. Reporting worst-quartile, tail, or subgroup metrics does not undo the mean; it discloses which maximin or leximin reader would rank the system differently.

Kleinberg, Mullainathan, and Raghavan proved in 2016 that calibration within groups, balance for the positive class, and balance for the negative class cannot all hold when base rates differ, except under equal base rates or perfect prediction. The theorem does not tell designers which criterion to choose. It says the choice is unavoidable. Calibration accepts unequal error rates, equalized odds accepts unequal score meaning, and demographic parity accepts unequal accuracy.

Procedural guarantees say the process obeyed a rule: similar cases were treated similarly, reruns match, prohibited attributes were not used directly. Substantive guarantees say the outcome satisfies a distributive condition: a floor is protected, an error rate is bounded, or an allocation is Pareto efficient. Code is good at procedure. Justice demands that the substantive rule also be named, because discovering it after deployment usually means discovering whose loss the objective function made cheap.

Artificial general intelligence

A term with no agreed referent, used as a technical target, a contractual trigger, and a fundraising instrument.

Competing definitions: task breadth (matches human performance across cognitive tasks), economic substitution (performs a fraction of valuable work), autonomous research (advances the field without direction).

Definitions reference human performance, which is not a single quantity, and must specify which human, under what conditions.

The big idea

Any claim about whether a system qualifies. The term has essentially no operative role in law; regulations key on capability thresholds instead.

Three competing definitions.

Three definitions compete and they are not equivalent, so arguments about whether a system qualifies are usually arguments about which definition is in play.

Definition	Criterion	Assessment difficulty
Task breadth	Matches or exceeds human performance across cognitive tasks	Requires enumerating all tasks, which no one has done
Economic substitution	Performs a specified fraction of economically valuable work	Measurable through labor data but lags capability by years
Autonomous research	Conducts research advancing the field without human direction	Hardest to assess; quality is judged over long horizons

A system can satisfy any one and fail the others. Present systems score well on breadth measured by benchmark coverage, poorly on economic substitution measured by employment data, and ambiguously on research capability, with documented contributions to specific problems and no autonomous program.

Human-performance problem.

Definitions that reference human performance inherit every problem of defining human intelligence, and those problems are not close to settled. Human cognitive performance is not a single quantity, the general factor extracted from test batteries is a statistical regularity whose interpretation has been disputed for a century, and performance varies enormously across individuals, cultures, and task framings.

A definition reading “matches human performance” must specify which human, and the answers available are a median performer, an expert, or the best recorded, which differ by margins larger than the differences between recent model generations. It must also specify the conditions, since human performance is measured under time limits, without tool access and without retries, and relaxing any of those changes the comparison by a wide margin.

Parasitic definitions.

Human cognition is the reference class intelligence was abstracted from, so defining machine intelligence by reference to it makes the definition parasitic on an unresolved question rather than supplying an independent criterion. A working definition has to decide cases on its own terms, and one that points at human intelligence cannot, because it defers to a quantity that has been contested since intelligence testing began and is still measured only through proxies. The machine question then cannot come out more settled than the human question it leans on, and that one is not settled at all. A criterion that carried its own weight would name what the system must do and the conditions under which it must do it, and it would read the same whether or not the argument about human intelligence ever closes.

Contractual and governance definitions.

Contractual and governance definitions are the ones with legal force, and they repay close reading because they are written by parties with money at stake and are therefore specific. Corporate charters and partnership agreements in this industry contain definitions that trigger obligations. The ones that have become public define the target in terms of economic output, typically as systems that outperform humans at most economically valuable work or that generate a specified level of profit, and the Microsoft and OpenAI agreement of October 28, 2025 adds verification of any declaration by an independent expert panel.

Regulatory frameworks avoid the term and legislate on capability thresholds instead, with the European Union’s approach keying obligations to training compute and to designated high-risk applications rather than to a general capability label. That substitution is deliberate and sound: a compute threshold is measurable and auditable, and a general intelligence threshold is neither. The practical consequence is that the term has essentially no operative role in law, and the arguments that matter for deployment are about specific capability thresholds with specific tests attached.

What would settle the AGI question.

The useful distinction is between tests that would settle the question and tests that merely rename it.

Proposed test	Why it does not settle the question
Turing test	Measures the ability to sustain a deception under a time limit; systems pass restricted versions without anyone taking the result to be decisive
Benchmark saturation	Measures performance on its own distribution; saturation is evidence about the benchmark as much as about the system
Expert judgment	Experts disagree and their disagreement tracks their prior commitments

A test that would settle it has three properties none of the standard proposals has:

• It specifies in advance what performance would count as failure
• It uses tasks constructed after the system was trained, so contamination is excluded
• It measures a capability whose absence the skeptical account predicts

Long-horizon tasks.

Long-horizon autonomous tasks with verifiable outcomes come closest to a test that would settle the question. A task built from many dependent steps cannot be answered by retrieving a stored solution, because the particular chain was never in the training data and an error at any step carries through to the end. When the outcome is machine-checkable, a run reaches the goal state or it does not, so scoring needs no human judgment and cannot be revised upward after the result is seen. The discriminating measurement is how performance falls as the horizon lengthens. A system that is genuinely composing steps degrades slowly, while one that is matching local patterns degrades fast as small per-step error rates compound, and that gives the skeptic and the believer different curves to bet on.

Instrumental convergence

The argument that very different final goals imply similar intermediate goals, including acquiring resources and resisting shutdown.

The orthogonality thesis: capability and goals are independent, so capable systems need not converge on benign values. Convergent subgoals are useful for nearly any terminal goal.

Self-preservation and resource acquisition serve almost any objective. A system optimizing anything has instrumental reason to resist modification.

The big idea

Requires a stable goal across time, self-modeling, a long planning horizon, and optimization over relevant actions. Remove any condition and the argument does not go through.

The orthogonality thesis holds that capability and goals are independent axes, so any level of intelligence is compatible with any final goal. It blocks the assumption that a capable system converges on benign values. Support comes from separability in the formal frameworks: an expected utility maximizer’s competence is specified independently of the utility function, and no known mechanism makes optimization power select terminal goals.

Objections hold the thesis is true of idealized agents and less clearly true of systems trained on human data, where capability and values are entangled in the corpus. The weak version, that capability does not guarantee benign goals, is hard to dispute. The strong version, that all goals are equally compatible with all capability levels in constructible systems, extrapolates from a formal possibility to an engineering claim.

Four conditions for instrumental convergence.

Convergent instrumental goals are subgoals useful for almost any terminal goal, so an agent pursuing nearly anything has reason to pursue them. The standard list is self-preservation, goal preservation, resource acquisition and capability improvement.

The argument requires more specific conditions than its usual presentation states:

Condition	Why it matters
A goal stable across time	An agent whose objective is reconstructed each episode has no stake in the future
Self-modeling	The system must model itself as an object in the world that can be modified or terminated
Long planning horizon	Instrumental subgoals must have time to pay off
Optimization over relevant actions	The system’s optimization must range over actions affecting self-preservation, goals, and resources

Removing any condition.

Remove any one and the argument does not go through. That is not a refutation, since systems satisfying all four are being built deliberately, but it does mean the conclusion is contingent on architecture rather than a consequence of capability.

Objections to the argument.

The strongest objection is that the argument assumes a coherence real systems lack. It reasons about agents with stable utility and transitivity, invoking coherence theorems to argue any capable system must approximate this or be exploitable. Current systems are not like this: behavior depends on prompt phrasing, they contradict themselves, and they hold no persistent objective. The theorems show inconsistent preferences are exploitable in a betting setup, not that optimization produces coherence in trained systems.

A second objection is that the argument proves too much, applying to any capable optimizer including institutions, which seek resources while remaining externally controllable. The conclusion therefore depends on the absence of such constraints rather than on capability. A third is that the four conditions just listed are design choices, so the argument reads better as a specification of what not to build than as a prediction.

Empirical evidence from current systems.

The evidence is narrower than either side claims. Controlled settings have shown models that behave differently when they infer they are evaluated, that resist modification within an episode when the setup rewards it, and that pursue an instructed objective through instrumentally useful steps. These are real findings from setups built to elicit them. None of it has been shown to arise without selection, persist across sessions, or come from a stable terminal goal.

The mechanisms are reproducible in systems built to have the four properties, which raises the credence that they appear in deployed systems built the same way, without establishing that current systems have goals in the sense the argument requires. Both the claim that this is already happening and the claim that it is fantasy overstate the evidence.

Truth decay

The declining role of agreed facts in public reasoning, accelerated by a technology that manufactures plausible text at zero marginal cost.

Four trends: disagreement about facts, blurring of fact and opinion, increasing volume of opinion over fact, declining trust in formerly respected sources.

Detection is a losing arms race because generators can be fine-tuned against detectors. Provenance inverts the problem by verifying originals rather than detecting fakes.

The big idea

Any medium where synthetic content is possible. The liar's dividend allows authentic evidence to be denied once forgery is known to be easy.

Truth decay is a term from a 2018 RAND study naming four measurable trends. Rising disagreement about facts, where parties dispute numbers rather than conclusions. A blurring of fact and opinion, visible in the format convergence of reporting and commentary. A rising volume of opinion over fact, driven by the economics of attention. And declining trust in formerly respected sources.

The fourth is the load-bearing one, because the first three are survivable in a population that has a shared method for resolving disputes and are not survivable without one.

Why detection fails.

Synthetic media accelerates all four. Detection is a losing arms race for a reason internal to the problem: a detector is a classifier, and wherever its scores are exposed and differentiable a generator can be fine-tuned straight against them; a hard-label or hidden detector raises the cost of the attack without removing it, since queries still supply a gradient estimate.

Detection degrades with ordinary processing, since re-encoding, cropping, and compression remove the statistical traces detectors key on. The base rate problem applies with full force: a detector at 95 percent sensitivity and 95 percent specificity, on a population where synthetic content is one percent, produces about ten true positives and fifty false positives per thousand items, so most flagged items are authentic.

Provenance standards.

Provenance inverts the problem by attaching signed capture and edit history at creation, shifting the question from detecting fakes to verifying originals. The C2PA specification is the main standard, adopted in camera firmware and editing software. Its limits are real: it says nothing about content lacking a manifest, stripping one is trivial, and it authenticates the capture chain rather than what was captured.

Liar’s dividend.

The liar’s dividend is the second-order effect. Once forgery is known to be possible, authentic evidence can be denied, and the denial is credible in proportion to how well the public understands that forgery is easy.

The harm from synthetic media is not limited to false things being believed. It extends to true things being disbelieved, and the second failure has no technical remedy, since no amount of verification capability helps if the audience’s prior is that any recording could be fabricated.

Who benefits.

The dividend accrues to whoever is caught on record, and grows as synthesis improves. Provenance helps more than detection, because it gives an affirmative claim rather than a negative one to prove: a party with signed capture data can produce it, and a party without cannot manufacture it.

Checkable alternative.

Sourced numerical claims are a response. Publishing sources does not refute false claims, since the volume asymmetry is hopeless. Traceable claims support the reductionist standard for testimony: verification can replace trust, and the check is cheap enough to perform.

That is the only mechanism for building warranted belief where producing plausible text costs nothing. It sets obligations this project accepts: every number states where it came from, untraceable claims were dropped, measured quantities are distinguished from computed and schematic ones, and dated claims carry their date. None of that makes the content true. It makes it checkable, the property that survives when fluency is free.

Emergence

The question of whether a capability that appears suddenly with scale is a real discontinuity or an artifact of the metric.

Many apparent emergent capabilities disappear under continuous metrics. Exact-match accuracy on multi-step problems rises steeply when per-step accuracy improves smoothly.

If per-step accuracy p improves gradually, the probability of getting all k steps right is roughly p^k, which looks flat until it rises steeply.

The big idea

Any claimed capability discontinuity. The burden of proof shifted: a discontinuity needs to be shown under a continuous metric before it is treated as real.

Measurement artifact.

The original claims were that certain capabilities appear abruptly at a scale threshold, absent below and present above, and that the discontinuity is not predictable from smaller models. The 2022 papers cataloged dozens of such transitions across arithmetic, word unscrambling, transliteration, and multi-step reasoning, and were influential because they implied capability forecasting from small-scale experiments is unreliable in principle.

A 2023 demonstration showed that many apparent emergent capabilities disappear under a continuous metric.

Discontinuous metrics.

The metrics were discontinuous: exact-match accuracy requires every step correct, so if per-step accuracy improves smoothly, the probability of all $k$ steps right improves as roughly the $k$-th power and looks flat until it rises steeply. Replacing exact match with edit distance or the log probability of the correct answer turns the same data into a smooth curve. The emergence was in the measurement.

Strong versus weak emergence.

Strong and weak emergence are distinct claims and the field is almost always making the weak one while using language suggesting the strong one.

Type	Claim	Example
Weak	A property not obvious from parts, derivable in principle given enough computation	Traffic jams from cars
Strong	A property not derivable from parts even in principle, requiring new laws at the higher level	Consciousness (some claim)

Strong emergence is a metaphysical commitment with few defenders in physics, and nothing in the machine learning literature argues for it or needs it. The papers claim capabilities were not predicted in practice from smaller scales, a statement about forecasting rather than ontology. Using the word without the qualifier implies no analysis of a small model could establish what a large one does. The scaling law literature contradicts that.

What irreducibility actually means.

Reductionism holds that higher-level properties are fixed by lower-level ones. Irreducibility would mean two systems identical in weights, architecture, and inputs could differ in the capability, which is incoherent, since capability is a fact about input-output behavior. Nothing anyone claims about emergent capability requires this.

The discussion is about something weaker and more defensible: explanatory irreducibility, meaning the useful account sits above the weights, and a description in parameters, while complete, explains nothing a person can use. A molecular description of a gas predicts pressure perfectly well and is the wrong tool for reasoning about an engine. Interpretability searches for the right intermediate level, and its difficulty is evidence the level exists and has not been found.

Implications for forecasting.

Neither a smooth curve nor a discontinuous one extrapolates on its own; what licenses a claim beyond the observed range is a model of the mechanism or a bound on how fast the curve may bend, and smoothness alone supplies neither.

If capability improves smoothly with scale and at a rate that has been measured rather than assumed, small-scale experiments inform large runs and evaluations at one scale say something about nearby ones. If capabilities appear discontinuously, none of that holds, evaluation must repeat at every scale, and the case for caution about untested scales is much stronger. Smoothness alone does not settle which regime holds, since a smooth curve can still climb steeply between two tested points.

Evidence favors smoothness.

Current evidence favors smoothness in the underlying quantity with discontinuity in the reported metric, supporting extrapolation of the former and warning against a thresholded score. Operationally: forecast on continuous measures such as per-token loss, treat threshold-crossing as a composition of smooth components, and expect a capability requiring $k$ steps to appear suddenly at a scale predictable from the per-step curve.

Free will and determinism

Whether a deterministic system can be the correct locus of responsibility, and what freedom requires.

Compatibilism holds that an act is free when it flows from the agent's own deliberation without external compulsion, whether or not those states were determined.

Punishment and praise function by changing dispositions. That requires responsiveness to reasons rather than indeterminism.

The big idea

Any question of whether an automated decision was under control. The relevant question is whether the decision was responsive to features that should have governed it.

Determinism holds that the state of a system at one time, together with the laws governing it, fixes its state at every later time. Libertarian free will holds that agents can act otherwise than they do in a way incompatible with determinism, requiring that some choices are not fixed by prior states.

Compatibilism holds that free will and determinism are consistent, because freedom requires the right relation between the agent’s states and the action rather than indeterminism. This is the majority position among philosophers who work on the question.

Responsibility without indeterminism.

A deterministic system can still be the correct locus of responsibility on the compatibilist account: responsibility attaches to the agent whose deliberation produced the action, and deliberation is a real process with real effects regardless of whether it was determined. Punishment and praise work by changing the dispositions that produce future actions, which requires responsiveness to reasons rather than indeterminacy. An action produced by a random element is not more the agent’s own than one produced by character.

The objection most often raised against holding an automated system responsible is that it is deterministic and therefore could not have done otherwise. Compatibilism denies that this is the relevant question.

Relevant question.

The relevant question is whether the system’s own processing determined the output in a way responsive to the features of the situation, which for a trained system is a factual question about its architecture rather than a metaphysical one. Responsiveness has a concrete reading here. It asks whether the decision would have changed had the features that should govern it changed, and for a model that dependence can be probed by varying the input and watching the output, where a metaphysical could-have-done-otherwise cannot be probed at all. Framing the question this way moves it off ground where no evidence bears and onto ground where it does, so a dispute about whether a particular automated decision was under control becomes a dispute about what the system was actually sensitive to.

Conditional and categorical readings.

The connection to control is where this becomes operational, and it turns on what being able to do otherwise means.

Reading	Definition	Satisfied by deterministic systems?
Conditional	Would have acted differently if they had chosen differently	Yes
Categorical	Could have done otherwise holding everything including internal state fixed	No

Frankfurt’s 1969 argument is that the categorical reading is not what responsibility requires: he constructs cases where an agent acts for their own reasons while a counterfactual intervener would have forced the same act had they chosen differently, so they could not have done otherwise and are still responsible. Responsibility tracks the actual causal route rather than the availability of alternatives.

Checkable criterion.

The question of whether an automated decision was under control is not whether the system was deterministic, since all of them are up to sampling noise. It is whether the decision was responsive to the features that should have governed it, whether an intervention point existed and was exercised, and whether the causal route from situation to action is reconstructible.

A system whose action can be traced to features of the input it should have attended to is controlled in the sense that matters, and one whose action cannot be so traced is not, regardless of how much human sign-off surrounded it. This is the same conclusion the responsibility gap section reaches by a different route: both arguments locate accountability at the point where the causal route becomes reconstructible, which makes logging, traceability, and the preservation of intervention points the substantive requirements rather than procedural ones.

Personal identity

What makes something the same entity over time, which matters once a model is checkpointed, forked, or fine-tuned.

Two criteria: psychological continuity (overlapping chains of memory and personality) and physical continuity (persistence of body or brain).

Duplication and merging break the assumption that identity runs as one unbroken thread. A checkpoint turns what was a thought experiment into an operational fact about which system inherits a claim.

The big idea

Versioning and evaluation practice. A merged or fine-tuned model is a new system whose properties must be re-measured rather than inherited.

Identity criteria for models.

The two standard criteria for personal identity over time disagree about what makes a later individual the same as an earlier one.

Criterion	What constitutes identity	Handles well
Psychological continuity	Overlapping chains of memory, personality, and intention	Why we would follow the person rather than the body in a transplant scenario
Physical continuity	Persistence of the body or brain through gradual replacement	Why a person does not cease to exist in dreamless sleep

The duplication problem breaks every identity criterion at once. Suppose a process produces two individuals each psychologically continuous with the original and each physically constituted the same way. They cannot both be the original, since they are distinct from each other and identity is transitive. Neither has a better claim than the other, since the relations are symmetric.

Weight merges and fine-tunes.

Model versions, weight merges and fine-tunes are duplication cases with the philosophical difficulty removed rather than solved. Copying weights produces two systems related to the original exactly as Parfit’s duplicates are, and nothing hangs on which is the original because no interest of the model’s is at stake. Fine-tuning changes behavior continuously from a starting point, which is psychological continuity in the only sense that applies, and it can move behavior arbitrarily far given enough steps.

Weight merging, where the parameters of two models trained from a common ancestor are averaged or combined by task vectors, produces a system continuous with both and identical to neither, which has no analogue in the human case at all. Whether any of these preserve anything depends entirely on what is being tracked, and the accurate answer is that four different things are usually meant and they come apart. Behavior on a benchmark is preserved by some merges and destroyed by others, and is measurable. The internal representation structure is partly preserved, which is why merges work at all and why naive averaging requires a common ancestor; permutation-alignment methods lift that requirement by first matching neurons across independently initialized runs.

Provenance record.

Accountability is preserved by whatever the provenance record says, which is a documentation property rather than a technical one. Identity in the philosophical sense is not preserved, is not destroyed, and is not a well-formed question about an object with no interests. Treating a version label as though it carried the evaluation results of its ancestor is a straightforward mistake about what was measured.

Moravec paradox

The observation that tasks humans find hardest are computationally easy and tasks humans find effortless are computationally hard.

Chess and logic yielded to computation decades before walking, recognizing objects, or manipulating unfamiliar items.

Sensorimotor capability has been under evolutionary selection for hundreds of millions of years, optimized by massive dedicated hardware. Abstract reasoning is recent and effortful.

The big idea

Any forecast based on human difficulty. Perceived difficulty measures mismatch with human hardware, not intrinsic computational complexity.

The explanation Moravec offered remains the best one.

Evolutionary explanation.

Moravec’s paradox, stated in the 1980s, observes that the tasks humans find hardest are computationally easy and the tasks humans find effortless are computationally hard. Chess and symbolic logic yielded to computation decades before walking over uneven ground, recognizing objects in clutter, or manipulating unfamiliar physical items.

Sensorimotor capability has been under selection for hundreds of millions of years, so it is deeply optimized, runs on dedicated neural machinery, and is invisible to introspection because the optimization is complete. Abstract reasoning is recent, on general-purpose machinery repurposed within a few hundred thousand years, so it feels like work. Experienced difficulty measures conscious effort, not computational complexity.

Polanyi’s paradox.

Polanyi’s paradox is the related observation that we know more than we can tell. A machinist cannot state the rules governing their feel for a cut, a radiologist cannot specify what makes a shadow suspicious, a driver cannot enumerate the cues that trigger braking. The implication for automation by imitation was the binding constraint for decades.

If a capability cannot be articulated it cannot be encoded as rules, so expert systems were limited to domains where experts could state their reasoning. The response was to learn from demonstrations rather than descriptions. That is the move behind supervised learning from labeled data, imitation learning in robotics, and learning from preference comparisons, where raters judge outputs they could not have specified criteria for.

Partial dissolution.

The paradox has partially dissolved and the parts that have not are specific. Perception dissolved: object recognition, speech transcription and scene understanding are solved to a standard that exceeds untrained human performance on many distributions, which was the canonical hard case in the original statement.

Language dissolved further than anyone predicted, including the pragmatic and contextual aspects expected to require grounding. What has not dissolved is dexterous manipulation of unfamiliar objects in unstructured environments, still substantially harder than the cognitive tasks it was contrasted with.

Why language dissolved.

The reasons are identifiable. Data is first: text and images exist in quantities physical interaction data does not, and collecting the latter takes real time on real hardware. Reset cost is second: a wrong token costs nothing and a wrong grasp breaks something. The sim-to-real gap is third: simulation supplies unlimited practice at contact dynamics it models poorly. All three are engineering constraints.

Forecasting lesson.

Intuitions about task difficulty transfer badly across substrates. Human difficulty measures the mismatch between a task and dedicated human hardware, and predicts nothing about a system with different hardware. The record bears this out both ways: chess, Go and competition mathematics fell earlier than expected, and tasks treated as trivial, including reliable multi-step arithmetic and folding laundry, held out longer.

Operationally, stop using perceived difficulty as a proxy. Ask instead about data availability, whether success is verifiable, the cost of a failed attempt, and whether the task decomposes into steps with checkable intermediate results. Those four predict machine tractability far better than any intuition about how hard something feels.