Control Theory

Feedback and open loop

Two ways to drive a system: execute a precomputed input with no measurement, or correct the input continuously from the measured gap between what the system does and what it should do.

Open loop maps a reference to an input through a model and commits. Feedback subtracts the measured output from the reference and drives that error toward zero through a compensator, so disturbance and model error get canceled as they appear.

A model is always wrong and disturbances are unmeasured, so an input computed once drifts. Correcting from the error absorbs whatever the model missed, at the cost of a stability question that open loop never has to ask.

The big idea

Any system with a sensor and a disturbance. Feedback cannot help when the measurement lags the dynamics or the actuator saturates, where feedforward from a model still has to carry the command.

A command computed from a model works only while the model and the disturbance assumptions keep holding. The moment load, friction, latency, or demand shifts, a plan executed blind starts to drift. Open loop is that blind commitment, while feedback measures the gap between desired and actual output and corrects from the error. The benefit is disturbance rejection, bought at the price of a stability question that did not exist before the measurement was fed back.

Plan executed blind

Open loop runs a plan without ever looking at the result. A stepper motor counting pulses, a microwave heating for ninety seconds, and a trajectory sent to a robot joint are all correct exactly when the model is correct and nothing pushes on the system. The appeal is that there is no stability question at all, because nothing is fed back to destabilize.

Feedforward is the honest name for open loop used well: compute the input a good model says you need, and let a separate feedback term clean up the residual. When the model captures most of the physics, feedforward carries the command and feedback handles only the part the model missed.

Error correction as uncertainty absorption

Closing the loop turns the controller into a machine for canceling ignorance. For a static loop gain $L$ relating input to output, reference $r$, and disturbance $d$, the output $y$ splits into a tracking term and a rejection term.

$$y = \frac{L}{1 + L}\, r + \frac{1}{1 + L}\, d$$

As the loop gain $L$ grows, the first coefficient approaches one, so the output $y$ follows the reference $r$, and the second coefficient approaches zero, so the disturbance $d$ is suppressed. High gain buys accuracy and rejection without a model of where the disturbance came from.

Steady-state accuracy against stability margin

The same gain that suppresses disturbance erodes the distance to instability. Raising loop gain shrinks steady-state error, but every real system carries phase lag, and past some frequency the correction arrives late enough to reinforce the error rather than cancel it. That is the trade a control designer lives inside: accuracy and disturbance rejection pull toward high gain, and stability margin pulls the other way.

Positive feedback and runaway

Negative feedback subtracts the error and stabilizes. Positive feedback adds it and amplifies, and a system that rewards its own output has no damping term to stop it. A recommender trained on the clicks it generated is a positive loop nobody wrote down: the model promotes an item, the promotion produces engagement, and the engagement trains the model to promote it harder.

Runaway is the generic outcome when a learned system consumes its own influence on the world.

Autoscaler as a controller

A cluster autoscaler that adds replicas in proportion to observed queue depth is a proportional controller, whether or not its authors used the word. Load rises, the measured error grows, and the controller commands more capacity. The dynamics are the same first-order loop a mechanical engineer would recognize, and the same rules govern it.

Set the gain too high and add the delay of a container that takes a minute to boot, and the autoscaler oscillates: it overprovisions, sees the queue drain, tears capacity down, and watches the queue refill. The cure is the control engineer’s cure, lower gain and explicit handling of the delay, not more capacity.

Retraining loop nobody designed

A production model retrained on logs of its own decisions is a closed loop whose designer never studied its stability. The model shapes the traffic, the traffic becomes the training set, and the next model inherits a world its predecessor bent. The same closed-loop reasoning governs an agent acting in an environment, which reinforcement learning studies from the learning side.

State-space representation

A model of a dynamical system as a first-order vector recurrence in a hidden state, whose next value depends on its current value and the input, with the output read separately from the state.

Gather the variables whose present values determine the future into a state vector. Write its derivative, or its next-step value, as one matrix times the state plus another times the input, and read the output through an output matrix.

Every stability, reachability, and optimality result in the field is stated over this form. A high-order scalar equation hides the structure that a first-order vector recurrence hands directly to linear algebra.

The big idea

Linear systems exactly, and nonlinear systems near an operating point. Far from that point the matrices are a local fiction, and the true vector field is what actually governs the motion.

A system can be written as one high-order equation relating an output to its own derivatives, and that form hides almost everything useful about it. Stacking the quantities that determine the future into a vector turns it into a first-order equation instead. State-space representation is that rewrite, with the measurement treated as a separate readout rather than as the thing being modeled. Nearly every result about stability, reachability and optimality is stated in this form, because a first-order vector recurrence hands the problem straight to linear algebra.

State, input, output, and four matrices

The state $x$ collects the quantities whose current values, together with future inputs $u$, determine the entire future. The measured output $y$ is a readout of the state, and four matrices carry the linear dynamics.

$$\dot{x}(t) = A\,x(t) + B\,u(t), \qquad y(t) = C\,x(t) + D\,u(t)$$

symbol	meaning
$x$	state vector, dimension $n$
$u$	input or control vector
$y$	measured output vector
$A$	dynamics matrix, how the state moves on its own
$B$	input matrix, how the control enters
$C$	output matrix, what the sensors expose
$D$	direct feedthrough from input to output
$t$	continuous time

The dynamics matrix $A$ sets the natural motion, the input matrix $B$ sets how the control reaches the state, the output matrix $C$ sets what a sensor sees, and $D$ is the rare direct path from input to output.

Continuous against discrete time

Physical systems evolve in continuous time, but a computer reads sensors and writes actuators on a clock, so the continuous model is discretized at a sampling period $T$. Exact discretization of a linear system produces a matrix exponential.

$$x_{k+1} = A_d\,x_k + B_d\,u_k, \qquad A_d = e^{A T}$$

Here $x_k$ is the state at step $k$, the discrete dynamics matrix $A_d$ is the matrix exponential of $A$ scaled by the period $T$, and $B_d$ is the corresponding input matrix obtained by integrating over one sample. The eigenvalues transform through the same exponential, which is why the stability boundary moves from a half plane in continuous time to a circle in discrete time, a point the stability section makes precise.

Reduction of high-order dynamics

A single differential equation of order $n$ becomes a first-order system in $n$ dimensions by stacking the variable and its derivatives into a state. The companion-form matrix that results carries the original coefficients in its bottom row and shifts the rest of the state up by one derivative per step.

This reduction is why the whole theory needs only first-order machinery.

Linearization about an operating point

Real dynamics $\dot{x} = f(x, u)$ are nonlinear, and the linear model is their first-order Taylor expansion about an operating point where the system is meant to run. The two matrices are Jacobians evaluated at that point.

$$A = \left.\frac{\partial f}{\partial x}\right|_{(x^{*}, u^{*})}, \qquad B = \left.\frac{\partial f}{\partial u}\right|_{(x^{*}, u^{*})}$$

The dynamics matrix $A$ and input matrix $B$ are the partial derivatives of the vector field $f$ with respect to state $x$ and input $u$ at the operating point $(x^{*}, u^{*})$. The linear model is trustworthy in a neighborhood and misleading outside it.

Relation to hidden Markov models and Kalman filtering

Add Gaussian process and measurement noise to a linear state-space model and you have exactly the object a Kalman filter estimates, with the hidden state playing the role the latent variable plays in a hidden Markov model. The control model is the deterministic skeleton, and the stochastic layer that turns it into an estimation problem belongs to stochastic processes.

This page treats the state as something to steer, taking estimation as given, while the recursive least-squares machinery that produces the estimate from noisy outputs is developed alongside Markov chains and filtering.

Naming collision with sequence-model state space models

The phrase state space model means two different things. Here the state is a physical or estimated latent, and the matrices come from dynamics or system identification. In modern sequence modeling the state space models named by the S4 architecture of Gu, Goel, and Re in 2021 and the Mamba architecture of Gu and Dao in 2023 are learned linear recurrences used as sequence operators, parameterized for long-range memory on token streams.

The algebra is shared and the purpose is not. A control engineer identifies $A$, $B$, and $C$ to match a plant and then proves something about the closed loop, while a sequence-model designer initializes a structured recurrence and trains it by gradient descent to move information across a context window.

Controllability and observability

Two structural tests: whether the inputs can drive the state anywhere, and whether the outputs reveal the state. A failure of either is built into the wiring rather than into the tuning.

Stack powers of the dynamics matrix against the input matrix to form the controllability matrix, and against the output matrix to form the observability matrix. Full rank in each certifies reachability and reconstructability.

A mode that no input excites can never be steered, and a mode that no output shows can never be estimated. No choice of gains repairs either, so these tests gate what any controller could possibly do.

The big idea

Linear or linearized models. The binary rank test is often too coarse in floating point, where a nearly unreachable mode passes the test and defeats the design anyway.

Controllability asks whether inputs can reach every state, and observability asks whether outputs can reveal it. Both are properties of the matrices rather than of the tuning.

Controllability matrix and its rank

A system is controllable when some finite input sequence can drive the state from any start to any target. The test stacks the input matrix $B$ against its images under the dynamics matrix $A$.

$$\mathcal{C} = \begin{bmatrix} B & AB & A^{2}B & \cdots & A^{n-1}B \end{bmatrix}$$

The controllability matrix $\mathcal{C}$ is built from the input matrix $B$ and the dynamics matrix $A$ up to power $n-1$, where $n$ is the state dimension, and the system is controllable exactly when $\mathcal{C}$ has rank $n$. If the rank falls short, some direction in state space is unreachable by any input, and that direction is fixed by the wiring of $A$ and $B$.

Observability matrix and duality

Observability is the mirror image: a system is observable when the output history determines the initial state uniquely. Its test stacks the output matrix $C$ above its compositions with the dynamics.

$$\mathcal{O} = \begin{bmatrix} C \\ CA \\ CA^{2} \\ \vdots \\ CA^{n-1} \end{bmatrix}$$

The observability matrix $\mathcal{O}$ uses the output matrix $C$ and the dynamics matrix $A$, and full rank $n$ certifies that the state can be reconstructed from measurements. Duality makes the two tests one: the pair given by $A$ and $C$ is observable exactly when the pair given by the transpose of $A$ and the transpose of $C$ is controllable, so every reachability result has an estimation twin for free.

Luenberger observer as placement’s dual

An estimator reconstructs the state a controller needs but cannot measure. The Luenberger observer runs a copy of the model driven by the mismatch between predicted and actual output, correcting the estimate $\hat{x}$ through an observer gain $L$ applied to the output residual.

$$\dot{e} = (A - L C)\,e, \qquad e = x - \hat{x}$$

The estimate error $e$, the gap between the true state $x$ and the estimate $\hat{x}$, evolves under the matrix $A - L C$, so choosing the observer gain $L$ to place the eigenvalues of $A - L C$ is the exact transpose of choosing a feedback gain $K$ to place the eigenvalues of $A - B K$. Observability is what makes that placement possible, detectability is the weaker condition that suffices, and this estimator is the half of the separation principle that certainty-equivalent design takes for granted.

Stabilizability and detectability

Full controllability is more than a design usually needs. Stabilizability is the weaker and sufficient condition that every unstable mode is controllable, leaving stable but unreachable modes alone because they decay on their own.

Detectability is the estimation counterpart: every unstable mode must be observable, while stable hidden modes can be ignored because their contribution to the output dies out. Most real designs certify stabilizability and detectability rather than the full-rank conditions.

Kalman decomposition of the modes

Kalman’s decomposition splits any linear system into four parts by whether each mode is controllable and whether it is observable. The part that is both is the piece a controller and an estimator can actually work with, and the transfer function from input to output sees only that piece.

Every quadrant other than the controllable-and-observable one is hidden from the transfer function, and the worst is the one that is neither controllable nor observable, a mode that evolves invisibly and unreachably inside the system. If such a mode is unstable, the input-output behavior can look perfectly well behaved while the internal state diverges.

Numerical rank and conditioning

Rank is a yes-or-no fact in exact arithmetic and a fiction in floating point. A system can be controllable on paper while its controllability matrix is nearly singular, and the honest measure is the controllability Gramian, whose small eigenvalues mark directions that need enormous inputs to reach.

Pole placement that ignores the conditioning will demand gains that saturate every actuator and amplify every sensor noise, so the conditioning is the number to watch, not the binary verdict.

Sensor and actuator placement

Where to put actuators and sensors is a design decision these conditions govern before any gain is chosen. Adding an actuator that couples strongly to a weakly controllable mode, or a sensor that observes it directly, is often cheaper than fighting a badly conditioned Gramian with brute gain later.

The same judgment appears in software, where deciding which service to instrument so an incident stays diagnosable is observability wearing a different name.

Stability and pole placement

The question of whether a system returns to equilibrium after a disturbance, read off the eigenvalues of its dynamics, together with the fact that feedback relocates those eigenvalues when the system is controllable.

Compute the eigenvalues of the closed-loop dynamics. Place them by state feedback that subtracts a gain times the state, and choose the gain so the closed-loop eigenvalues sit in the stable region.

A linear system decays exactly when every eigenvalue sits in the stable region, a half plane in continuous time and a disk in discrete time. Controllability guarantees that feedback can move them anywhere the designer likes.

The big idea

Linear or linearized dynamics. Arbitrary placement is available in principle and expensive in reality, because fast eigenvalues cost actuator effort and amplify noise, and time delay quietly destroys the margin.

After a disturbance, the practical question is whether the motion dies out or grows until something saturates, breaks, or oscillates. For linear or linearized dynamics, stability and pole placement turn that question into eigenvalue locations, then ask how feedback can move them. The useful qualification is that arbitrary relocation needs controllability, and even then the chosen poles still trade speed against actuator effort, noise amplification, and delay margin.

Eigenvalues and the stability boundary

A linear system decays to equilibrium exactly when its eigenvalues lie in the stable region, and the region depends on whether time runs continuously or in steps.

$$\operatorname{Re}(\lambda_i) < 0 \;\;\text{(continuous)}, \qquad |\lambda_i| < 1 \;\;\text{(discrete)}$$

Each eigenvalue $\lambda_i$ of the dynamics matrix must have negative real part in continuous time, or magnitude below one in discrete time, for its mode to decay. A single eigenvalue outside the boundary is enough to make the whole system diverge, since its mode grows without bound and eventually dominates every trajectory.

Bounded input against internal stability

Bounded-input bounded-output stability asks only that a bounded input produce a bounded output, and it can hide a divergence. When a controller cancels an unstable pole with a matching zero, the transfer function looks stable while an internal mode grows unmeasured, the same neither-seen-nor-steered trap the Kalman decomposition warned about.

Internal stability is the stronger and correct requirement that every state, not merely the measured output, stays bounded. Certifying it means checking the eigenvalues of the full closed-loop dynamics rather than trusting the input-output map, because pole-zero cancellation of an unstable mode is a bug that reads as a clean frequency response.

Pole placement by state feedback

Feedback that subtracts a gain times the state changes the dynamics matrix from $A$ to $A - BK$, and choosing the gain $K$ sets the closed-loop eigenvalues.

$$u = -K\,x \;\Longrightarrow\; \dot{x} = (A - B K)\,x$$

With input $u$ set to the negative of gain $K$ times state $x$, the closed-loop dynamics become the matrix $A - BK$, whose eigenvalues the designer places. When the pair given by $A$ and $B$ is controllable and the system has a single input, Ackermann’s formula computes the one gain $K$ that puts the eigenvalues at any chosen locations. With several inputs the placement is still available but the gain is no longer unique, and the leftover freedom is spent on robustness or on control effort.

Cost of placing poles far left

Arbitrary placement is a license, not a free action. Pushing eigenvalues far into the stable region makes the system respond fast, and fast response demands large control effort that saturates actuators and amplifies measurement noise straight into the input.

The sensible target is fast enough and no faster. Every decade of extra bandwidth multiplies the noise the controller injects and the authority the actuator must supply, so the eigenvalue locations are an economic choice about effort and noise.

Loop transfer function and Nyquist encirclement

The frequency-domain view collapses the plant and controller into one loop transfer function $L(s) = G(s)\,C(s)$, the product of the plant $G(s)$ and the controller $C(s)$ evaluated at the complex frequency $s$. Harry Nyquist showed in 1932 that closed-loop stability can be read from how the locus of $L(j\omega)$, traced as the frequency $\omega$ runs up the imaginary axis, winds around the critical point at $-1$.

$$Z = N + P$$

The count $Z$ of closed-loop poles in the right half plane equals the clockwise encirclements $N$ of the point $-1$ by the locus plus the count $P$ of open-loop right-half-plane poles, so a loop whose open-loop transfer is already stable, meaning $P$ is zero, stays stable closed exactly when the locus makes no encirclement. On this picture the robustness margins become distances and angles to the critical direction: gain margin is how far the locus falls short of $-1$ along the negative real axis at the phase crossover, and phase margin is the angular gap between the negative real axis and the ray from the origin to the locus at the gain crossover where the magnitude $|L(j\omega)|$ equals one, both equally visible on the Bode plot.

Gain margin and phase margin

Margins are the honest measure of how much reality a design tolerates before it rings. Gain margin is how much the loop gain can grow before instability, and phase margin is how much extra phase lag the loop absorbs before its own correction turns from cancellation into reinforcement.

High gain shrinks steady-state error and shrinks phase margin at the same time, so a number like sixty degrees of phase margin is a compact statement of how much the designer gave back to stay safe.

Delay as the margin killer

Time delay is the quiet assassin of phase margin. A pure delay adds phase lag that grows with frequency while leaving the gain untouched, so it eats margin exactly where a fast loop needs it and gives no warning in the magnitude response. Every real loop has some delay, and underestimating it is a common path to an oscillation nobody predicted.

A loop that closes over a remote procedure call, a message queue, or a congested link carries variable latency that behaves like a delay with a distribution, and a controller tuned as if the measurement were instant will ring the moment the network hiccups.

Lyapunov functions

An energy-like scalar that decreases along every trajectory, whose existence proves stability without ever solving the dynamics, which is why the method survives into nonlinear and learning settings.

Propose a positive scalar of the state that is zero only at equilibrium, then show its time derivative along the dynamics is never positive. Downhill motion of the scalar forces the state toward equilibrium.

A quantity that only decreases and is bounded below cannot wander, which is what makes the equilibrium stable. Driving the state to it needs strict decrease, or LaSalle's invariant-set argument, since a conserved quantity is nonincreasing while the state orbits forever. The argument needs no closed-form trajectory, which is what lets it reach systems no one can integrate.

The big idea

Nonlinear and linear systems, local or global depending on where the conditions hold. The method proves stability when it finds a certificate and stays silent when it fails, since a bad candidate proves nothing either way.

For a nonlinear plant, solving the trajectories may be impossible while the safety question still has to be answered. A Lyapunov function gives a certificate instead: find an energy-like scalar that is positive away from equilibrium and does not increase along the actual motion. The consequence depends on where those conditions hold and how strict the decrease is, so the result may be local, may need LaSalle’s argument for convergence, and proves nothing when the candidate fails.

Direct method and its two conditions

The direct method asks for a scalar function $V$ of the state $x$ that is positive everywhere except at equilibrium and never increases along the motion. Two conditions carry the whole argument.

$$V(x) > 0 \;\text{ for } x \neq 0, \qquad \dot{V}(x) = \nabla V(x)^{\top} f(x) \le 0$$

The candidate $V$ is positive definite, vanishing only at the origin, and its derivative along the dynamics $f$, computed as the gradient of $V$ dotted with the vector field, is non-positive. A quantity that is bounded below and never rises must level off, so the state cannot escape, and strict decrease forces it to the equilibrium.

Quadratic candidates and the Lyapunov equation

For a linear system the natural candidate is a quadratic form, the state weighted by a positive definite matrix, and its decrease condition becomes a matrix equation. Solving that equation is how linear stability gets certified without eigenvalues.

$$A^{\top} P + P A = -Q, \qquad P = P^{\top} \succ 0$$

Given a chosen positive definite matrix $Q$, the Lyapunov equation is solved for a symmetric matrix $P$, and a positive definite solution exists exactly when the dynamics matrix $A$ is stable. The quadratic form built from $P$ is then a Lyapunov function, so a linear-algebra solve replaces the search for a certificate.

Region of attraction

Nonlinear stability is almost always local, and the honest deliverable is a region of attraction rather than a global claim. A bounded sublevel set of the Lyapunov function on which the derivative stays negative away from the equilibrium is a region from which every trajectory converges, and the largest such set the certificate can prove is the guaranteed basin.

Estimating that basin conservatively is the working engineer’s task.

LaSalle invariance principle

When the derivative is only non-positive rather than strictly negative, the basic theorem stalls, because the state might coast along a surface where the function stops decreasing. LaSalle’s invariance principle, published by Joseph LaSalle in 1960, closes the gap: trajectories that stay in a compact positively invariant set converge to the largest invariant set contained in the region where the derivative is zero.

Establishing that the only invariant set inside the zero-derivative surface is the equilibrium upgrades a non-strict certificate into a genuine convergence proof, without demanding strict decrease that the physics does not provide.

Finding a candidate by sum-of-squares

Verifying a Lyapunov function is a calculation, and finding one is the hard part. For systems with polynomial dynamics, sum-of-squares programming searches over polynomial candidates by requiring the function and the negative of its derivative to be expressible as sums of squares, a constraint a semidefinite program in convex optimization can enforce.

The method turns an open-ended search into a structured feasibility problem, at the price of a candidate class fixed in advance. A certificate the solver returns is genuine, and a failure to find one within the chosen degree proves nothing.

Lyapunov arguments in learning

A convergence proof for an optimizer usually exhibits a potential, the loss or a distance to the optimum, and shows each step decreases it, which is a Lyapunov argument by another name and the backbone of the guarantees in optimization.

Safe learning uses the same certificate to constrain a policy. A control Lyapunov function that a learned controller is required to decrease gives a stability guarantee that a purely learned policy cannot supply on its own, which is one concrete way control contributes what reinforcement learning lacks: a proof that the closed loop stays bounded.

PID control

A controller built from three responses to the tracking error: a term proportional to the present error, a term summing past error, and a term reacting to the error's rate of change.

Weight the present error, its integral, and its derivative, then sum the three into the control command. Tune the three weights to trade speed of response against overshoot and noise sensitivity.

The proportional term supplies the corrective push, the integral erases steady offset, and the derivative anticipates. The combination needs no model of the plant, so it deploys where identifying a model would cost more than it returns.

The big idea

Single-loop regulation of slow physical processes, where it still runs most of the installed base. It struggles with strong coupling, hard constraints, and long dead time, which is where model-based methods start to earn their cost.

PID control sums three terms driven by present error, accumulated error, and the rate of error, and it runs most of the physical world because it needs no model.

Three terms and the error signal

The controller acts on the error, the reference minus the measurement, through three weighted terms whose sum is the command at time $t$.

$$u(t) = K_p\, e(t) + K_i \int_{0}^{t} e(\tau)\, d\tau + K_d\, \dot{e}(t)$$

symbol	meaning
$u$	control command sent to the actuator
$e$	tracking error, reference minus measurement
$K_p$	proportional gain on present error
$K_i$	integral gain on accumulated error
$K_d$	derivative gain on the rate of error

The proportional gain $K_p$ sets the immediate push, the integral gain $K_i$ accumulates past error $e$ over a dummy variable $\tau$ to erase any steady offset, and the derivative gain $K_d$ reacts to how fast the error is changing.

Integral windup and its remedies

The integral term is the source of the most common field failure. While an actuator sits saturated at its limit, the error persists and the integral keeps accumulating, so when the error finally reverses the controller stays pinned to the limit until the oversized integral unwinds, producing a large overshoot.

Clamping stops the integral from accumulating once the actuator saturates, and back-calculation feeds the difference between commanded and delivered actuation back into the integrator to bleed it down.

Derivative kick and measurement differentiation

Differentiating the error creates a hidden hazard at setpoint changes. A step in the reference is a step in the error, and its derivative is an impulse, so the derivative term slams the actuator on any operator command, an artifact called derivative kick that has nothing to do with the process.

Since the error and the measurement differ only by the reference, and the reference derivative is the offending impulse, differentiating the measured output alone keeps the useful anticipation and drops the kick.

Ziegler-Nichols and relay tuning

Ziegler and Nichols published tuning rules in 1942 that set the three gains from a couple of measured features of the loop, such as the gain and period at which sustained oscillation begins. The rules are famous because they need almost no analysis, and notorious because they aim for a fast response with little margin, giving a loop that tracks quickly and rings under disturbance.

Relay auto-tuning automates the measurement by driving the loop with a relay to provoke a controlled oscillation and reading the critical gain and period from it. The result is convenient and inherits the same aggressiveness, so tuned values are a starting point to detune toward margin rather than a finished design.

Derivative action in noisy loops

Derivative action is often switched off, leaving a proportional-integral loop, because differentiation amplifies high-frequency noise. Sensor hash that the proportional and integral terms average away gets multiplied by the derivative term and injected straight into the actuator, so a noisy measurement makes the derivative a liability rather than an asset.

When anticipation is genuinely needed, the derivative runs on a low-pass filtered measurement to tame the noise gain. The filter adds phase lag that eats some of the margin the derivative was meant to buy.

Baseline a learned controller must beat

A two-parameter proportional-integral loop is the baseline any learned controller has to beat, and beating it is harder than it sounds. The classical loop is cheap to compute, trivial to certify, and understood by whoever maintains it at three in the morning, so a learned policy has to justify its opacity with a margin the simple loop cannot reach.

A learned controller earns its keep on strongly coupled, constrained, or nonlinear plants where tuning three gains cannot express the right behavior, and it should be measured against a well-tuned baseline rather than a strawman, a discipline the evaluation of any policy in reinforcement learning needs just as badly.

Linear quadratic regulator

The optimal state-feedback law for linear dynamics under a quadratic cost on state and input, available in closed form as a constant gain, the one corner of optimal control that is genuinely easy.

Write a cost that adds a state penalty and an input penalty over time. Solve the algebraic Riccati equation for a single matrix, and read off the constant feedback gain that minimizes the cost.

The quadratic cost and linear dynamics make the value function quadratic, so its minimization collapses to a matrix equation rather than a search. The resulting law is linear feedback, and state feedback carries strong stability margins for free.

The big idea

Linear or linearized systems whose objective is honestly quadratic. Add an estimator for unmeasured state and the guarantees weaken, because the margins the state-feedback law enjoys do not survive the observer.

Most of optimal control ends at a numerical solver. One case does not. When the dynamics are linear and the cost is quadratic in state and input, the best policy can simply be written down: a fixed matrix multiplying the current state. The linear quadratic regulator is that case. A single matrix equation replaces a search over trajectories, and the design work shrinks to deciding how much a state deviation should cost against how much control effort should.

Quadratic cost and its weights

The objective adds a running penalty on state deviation and a running penalty on control effort, integrated over time $t$.

$$J = \int_{0}^{\infty} \left( x(t)^{\top} Q\, x(t) + u(t)^{\top} R\, u(t) \right) dt$$

The cost $J$ weights the state $x$ by a positive semidefinite matrix $Q$ and the input $u$ by a positive definite matrix $R$. The ratio between $Q$ and $R$ is the single design knob: a large $Q$ demands tight regulation and spends control freely, while a large $R$ conserves actuation and tolerates larger state excursions.

Algebraic Riccati equation and constant gain

Minimizing the quadratic cost over the linear dynamics produces a matrix equation for the cost-to-go, and the optimal gain reads directly off its solution.

$$A^{\top} P + P A - P B R^{-1} B^{\top} P + Q = 0, \qquad K = R^{-1} B^{\top} P$$

The algebraic Riccati equation is solved for the matrix $P$, built from the dynamics matrices $A$ and $B$ and the weights $Q$ and $R$. A unique stabilizing solution exists when $(A, B)$ is stabilizable and $(A, Q^{1/2})$ is detectable, and the optimal feedback gain $K$ is then a formula in $P$. The control law is the constant feedback that sets the input to the negative of $K$ times the state, with the quadratic form built from $P$ serving as a Lyapunov function for the closed loop.

Discrete-time regulator and its Riccati equation

Every digital controller regulates the sampled-data model rather than the continuous plant, so the regulator that ships solves a discrete counterpart of the algebraic Riccati equation, with the cost summed over steps instead of integrated over time.

$$P = A_d^{\top} P A_d - A_d^{\top} P B_d\,(R + B_d^{\top} P B_d)^{-1} B_d^{\top} P A_d + Q$$

The discrete algebraic Riccati equation solves for the cost-to-go matrix $P$ from the sampled dynamics $A_d$ and $B_d$ together with the state weight $Q$ and input weight $R$, and the optimal gain is again a formula in $P$. Sampling faster brings the discrete gain toward its continuous value, while sampling too slowly can destabilize a loop the continuous design certified, so the sample rate is part of the regulator rather than an afterthought.

Guaranteed margins for state feedback

When the full state is measured, a single-input loop, or each channel of a multi-input loop with diagonal $R$, has an infinite gain margin upward, tolerates the loop gain being halved, and holds at least sixty degrees of phase margin.

The guarantees are a property of feeding back the actual state, and the moment the state is estimated rather than measured, the argument that produced them no longer applies, which is the subject two subsections down.

LQG as regulator plus filter

Real systems rarely expose the full state, so the linear quadratic Gaussian design pairs the LQR gain with a Kalman filter that estimates the state from noisy outputs. The separation principle, given its stochastic form by Wonham in 1968, states the convenient result precisely: the optimal controller runs the optimal estimator and feeds its estimate to the gain computed as if the state were known, and the estimator and regulator are designed independently by two separate Riccati equations.

Certainty equivalence is the name for this structure, and it is genuinely optimal for the linear Gaussian quadratic problem rather than an approximation. The estimator solves one Riccati equation for the filter gain and the regulator solves another for the control gain, and their combination minimizes the expected cost.

No guaranteed margins for LQG

Doyle proved in 1978, in a note whose one-line abstract states that there are none, that the LQG regulator inherits no guaranteed stability margins at all. His construction takes a benign plant and choices of noise intensity that drive the gain and phase margins of the LQG loop arbitrarily close to zero, so the sturdy margins of state-feedback LQR simply do not survive the observer.

An optimal controller can be fragile, because optimality is defined against the exact model and says nothing about a wrong one, and the loop transfer recovery technique that partially rebuilds the lost margin does so by deliberately detuning the filter away from its optimal setting. Anyone deploying an estimator-based optimal controller has to check the margin directly, since the optimality certificate does not carry it.

Inverse optimal control

Optimal control runs from cost to behavior, and inverse optimal control runs the other way, inferring the cost that makes observed behavior optimal. Given a stabilizing controller, one can ask which state and input weights would make it the LQR solution.

The idea is the control ancestor of inverse reinforcement learning, which recovers a reward from demonstrations. Both face the same ambiguity, that many costs rationalize the same behavior, and both are useful anyway. The reward-inference side is treated in reinforcement learning.

Model predictive control

A controller that solves a finite-horizon optimal control problem at every step, applies only the first action of the resulting plan, then discards the rest and re-solves at the next step from the new state.

At each step, optimize a sequence of future inputs against a model and a cost subject to constraints, execute the first input, advance one step, and repeat the whole optimization from the measured state.

Re-solving from the current state folds feedback into optimization and lets hard limits on state and input enter the problem directly, so constraints are respected by construction rather than clipped after a law computes an infeasible command.

The big idea

Systems fast enough to tolerate the per-step solve and slow enough that a solver finishes in time. The binding limit is compute, which is why fast loops precompute the law and slow processes solve online.

Model predictive control solves a finite-horizon optimization at each step, applies the first action, and re-solves at the next, which is how constraints get handled directly rather than clipped after the fact.

Receding horizon and the first action

At each step the controller optimizes a whole sequence of future inputs against a model and a cost, then throws almost all of it away.

$$\min_{u_0, \dots, u_{N-1}} \; \sum_{k=0}^{N-1} \ell(x_k, u_k) + V_f(x_N) \;\;\text{s.t.}\;\; x_{k+1} = f(x_k, u_k), \; x_k \in \mathcal{X}, \; u_k \in \mathcal{U}$$

The problem minimizes a stage cost that depends on state $x$ and input $u$ over a horizon $N$, plus a terminal cost $V_f$, subject to the model $f$ and to the state set $\mathcal{X}$ and input set $\mathcal{U}$. Only the first input is applied, and at the next step the whole optimization runs again from the freshly measured state, so re-solving is what turns an open-loop plan into feedback.

Constraints as the reason over LQR

The reason to pay for an online solver instead of a constant gain is constraints. LQR minimizes the same kind of cost but knows nothing of limits, so it will happily command an actuator past its stop or a state past a safety bound, and clipping the result afterward voids the optimality and can destabilize. Predictive control puts the actuator limits and the state bounds inside the optimization, so every command it issues is feasible.

For linear dynamics with a convex quadratic cost and polyhedral limits, each step is a quadratic program, and the constrained optimization machinery of convex optimization, including the Karush-Kuhn-Tucker conditions, is exactly the solver’s engine. A convex cost that is not quadratic leaves a convex program, solvable by the same machinery but not by a QP solver.

Terminal cost and terminal set

A finite horizon threatens stability, because a controller that optimizes only a few steps ahead can walk the state into a corner it cannot escape. The standard fix, formalized by Mayne, Rawlings, Rao, and Scokaert in 2000, adds a terminal cost that is a local control Lyapunov function and a terminal set that is invariant under a known stabilizing law.

With those ingredients the finite-horizon value function is itself a Lyapunov function for the closed loop, and the receding-horizon controller inherits a genuine stability guarantee rather than a hope that a longer horizon behaves.

Solve time as the binding constraint

Predictive control lives or dies on whether the solver finishes before the next sample. The optimization runs every step, so the horizon length, the model size, and the constraint count all trade against the achievable loop rate, and a controller that needs ten milliseconds of solve cannot run a one-millisecond loop.

For fast loops, explicit predictive control precomputes the optimal law offline as a piecewise-affine function of the state, turning the online solve into a table lookup. Real-time iteration schemes take the opposite tack, doing one solver iteration per step and accepting a suboptimal command to hit the deadline.

Robust and stochastic variants

The nominal formulation trusts its model, and the robust variants do not. Tube predictive control keeps the true state inside a tube around a nominal trajectory by tightening the constraints enough to absorb the worst-case disturbance, buying a guarantee at the price of a smaller feasible region.

Stochastic predictive control replaces the worst case with a distribution and enforces chance constraints that may be violated with small probability, which recovers some of the performance the worst case gives up.

Overlap with model-based planning

Predictive control and model-based planning are the same computation viewed from two fields. Optimizing a sequence of actions against a known model over a horizon and committing only the first is what a planner does, and the model-based branch of reinforcement learning learns the model first and then plans through it exactly this way.

Sampling-based predictive control makes the kinship concrete by evaluating many random input sequences through the model and weighting them by cost, a procedure that reads equally as control or as planning.

Robust control

Designing a controller against a whole set of plants rather than a single nominal model, the formal version of the working engineer's knowledge that the model in hand is wrong in ways that matter.

Bound the uncertainty as a set around the nominal model, then synthesize a controller and prove stability for every plant in that set, typically by keeping a loop gain product below one.

Guaranteeing performance for one model guarantees nothing for the real system, which differs from it. Certifying the entire uncertainty set converts an unstated hope about modeling error into a proof that survives it.

The big idea

Systems whose modeling error can be bounded, in norm or in structure. The guarantee costs nominal performance, since insuring against the worst case forces caution the typical case never needed.

A controller tuned to one identified plant can look perfect on paper and fragile on the real machine, because the real plant differs in ways the nominal model did not include. Robust control makes that uncertainty part of the design object by replacing the single plant with a bounded set of possible plants. The consequence is a certificate that survives every member of that set, paid for by giving up performance the nominal model alone would have allowed.

Uncertainty as additive, multiplicative, parametric

Robust design begins by naming the shape of the ignorance. Additive uncertainty adds an unknown bounded term to the plant, multiplicative uncertainty scales the plant by a factor near one, and parametric uncertainty admits that specific physical coefficients are known only within intervals.

Frequency-domain norm bounds capture unmodeled high-frequency dynamics well and physical parameter intervals capture a mass or a stiffness known only roughly, and picking the description that matches the ignorance is what keeps the resulting guarantee meaningful rather than decorative.

Small-gain theorem as the core condition

The workhorse sufficient condition for robust stability is the small-gain theorem, which certifies a feedback interconnection of two stable systems when the product of their gains stays below one.

$$\|M\|_{\infty}\, \|\Delta\|_{\infty} < 1$$

Here the nominal closed loop is captured by the stable operator $M$ and the uncertainty by the stable perturbation $\Delta$, and the interconnection is guaranteed stable whenever the product of their infinity norms is below one. The condition is conservative, since it ignores the phase and structure of $\Delta$ and asks only about magnitude, but its generality is the point: it holds for any perturbation inside the norm bound.

Sensitivity and complementary sensitivity

Two closed-loop transfer functions carry most of the design information. The sensitivity $S(s)$ maps disturbance and reference into the tracking error, and the complementary sensitivity $T(s)$ maps sensor noise into the output, both written from the same loop transfer function $L(s)$.

$$S(s) = \frac{1}{1 + L(s)}, \qquad T(s) = \frac{L(s)}{1 + L(s)}, \qquad S(s) + T(s) = 1$$

Because the sensitivity $S$ and the complementary sensitivity $T$ sum to one at every frequency, they cannot both be small at the same complex frequency $s$, so the tradeoff is an algebraic identity rather than a rule of thumb. Disturbance rejection and tracking want $S$ small at low frequency, while noise rejection and robustness to unmodeled dynamics want $T$ small at high frequency, so a design assigns each its own band and pays for the crossover between them.

Loop shaping and the Bode integral

Loop shaping designs the magnitude of $L(j\omega)$ across frequency directly: high gain where disturbances live, low gain where noise and uncertainty dominate, and a controlled slope through the crossover. What no shape can escape is a conservation law that Hendrik Bode derived and that James Freudenberg and Douglas Looze sharpened for unstable plants in 1985.

$$\int_{0}^{\infty} \ln|S(j\omega)|\,d\omega = \pi \sum_{k} \operatorname{Re}(p_k)$$

The integral of the log sensitivity magnitude $|S(j\omega)|$ over frequency $\omega$ is fixed, equal to $\pi$ times the sum of the real parts of the plant’s unstable poles $p_k$, provided the loop rolls off with relative degree at least two. For a stable plant the right side is zero, so every band pushed below one in sensitivity buys a band forced above it, the waterbed effect, and each right-half-plane pole raises the floor the design cannot get under.

H-infinity synthesis

H-infinity synthesis makes the small-gain idea constructive by minimizing the worst-case gain from disturbance to error, the infinity norm of the closed-loop map. Doyle, Glover, Khargonekar, and Francis gave the clean state-space solution in 1989, reducing the synthesis to a pair of Riccati equations much like the ones the regulator and estimator already solve.

Shaping the frequency weights that define the error and disturbance channels is the engineering, and the Riccati solve that follows is mechanical, so H-infinity design feels like weighted LQG with the worst case in the objective rather than the average.

Structured singular value analysis

The small-gain bound treats the uncertainty as one unstructured block, which throws away real information when the ignorance is actually several independent pieces. The structured singular value, written mu and introduced by Doyle in 1982, sharpens the test by accounting for the block-diagonal structure of the perturbation, giving a far less conservative stability margin.

Evaluating mu exactly is intractable in general, so practice works with upper and lower bounds that bracket it, and the bounds are usually tight enough to certify or reject a design.

Conservatism cost

Every robust guarantee is bought with nominal performance. A controller that must stay stable for every plant in a set cannot tune itself to the one plant it will actually face, so it responds more cautiously and settles more slowly than a design that trusted the nominal model completely.

A tight, honest uncertainty set costs little caution and certifies what matters, while a loose set drowns performance in insurance against plants that will never occur, so the discipline is to bound the uncertainty no more loosely than the evidence demands.

Distributionally robust optimization as the same idea

The robust move reappears in machine learning as distributionally robust optimization, which minimizes expected loss against the worst distribution inside an ambiguity set rather than against the empirical distribution alone. Plant uncertainty becomes distributional uncertainty, the norm-bounded set becomes a divergence ball around the data, and the min-max structure is identical.

A robust controller insures against a set of dynamics and a distributionally robust model insures against a set of data-generating distributions, and both trade average-case performance for a worst-case guarantee inside a set they had to choose. The treatment of worst-case objectives in optimization develops the same pattern on the learning side.

System identification

Fitting a dynamics model from measured inputs and outputs, which is supervised learning with the extra requirement that the input has to excite the modes the model is meant to capture.

Excite the system with an input rich enough to reveal its modes, collect input and output trajectories, fit parameters of a chosen model structure, and validate the model on data it was never fit to.

A controller is only as good as its model, and a model is only as good as the data that identified it. An input that fails to excite a mode leaves that mode unmeasured, so no fitting procedure can recover it.

The big idea

Systems that can be probed with a designed input, in open or closed loop. Closed-loop data biases naive fits, because feedback correlates the input with the very noise the fit is trying to average out.

A controller is designed against a model, and the model usually has to be measured rather than derived. Most of that is an ordinary fitting problem. The part with no analogue in supervised learning is that the engineer also chooses the inputs, and a system never stirred in a given way never reveals how it responds to it. System identification is fitting dynamics from measured input and output, with the design of the probing signal treated as part of the method rather than as data collection that happened first.

Persistent excitation

A model can only learn what the data reveals, and revealing a mode requires an input that stirs it. Persistency of excitation of a given order is the precise condition that the input carries enough distinct frequencies to identify a model of that order, made exact by a rank condition on a Hankel matrix built from the input.

$$\operatorname{rank} \begin{bmatrix} u_0 & u_1 & \cdots & u_{L-1} \\ u_1 & u_2 & \cdots & u_{L} \\ \vdots & & & \vdots \\ u_{n-1} & u_{n} & \cdots & u_{L+n-2} \end{bmatrix} = n$$

Full row rank $n$ of this Hankel matrix of the input samples $u_k$ is the definition of persistency of excitation of order $n$, and it is necessary but not sufficient to determine an order-$n$ model: the data span every trajectory of length $T$ of an order-$n$ system only when the input is persistently exciting of order $T + n$. A smooth or constant input fails this by a wide margin and identifies almost nothing, a result Willems, Rapisarda, Markovsky, and De Moor sharpened in 2005 into the foundation of data-driven control.

Model structures ARX, ARMAX, output-error

Where the noise is assumed to enter determines the model structure and the estimator that fits it. The ARX structure regresses the output on past outputs and inputs and is the easiest to fit, but it assumes a noise model that colors the estimate when reality disagrees. ARMAX adds a moving-average noise term, and output-error models the noise as acting only on the measurement.

Choosing among them is a bias decision, since each structure is unbiased only when the true noise matches its assumption. The prediction-error framework unifies them and connects directly to the estimation methods in statistics.

Subspace identification

Subspace methods recover a state-space model directly from input-output data without first choosing a scalar structure, using projections and a singular value decomposition to estimate the state sequence and then the matrices. The N4SID algorithm of Van Overschee and De Moor, published in 1994, is the canonical version, and its appeal is that it is numerically robust and needs little tuning.

Subspace methods deliver a state-space model in one non-iterative computation, which suits high-order multivariable systems, at the cost of less direct control over the noise assumptions than the prediction-error structures give.

Model order selection

Deciding how many states the model should have is the same bias-against-variance judgment that governs every fit. Too few states and the model cannot represent the dynamics, biasing every prediction, while too many states fit the noise and give a model that matches the training trajectory and generalizes poorly.

Information criteria that penalize order, and inspecting where the subspace singular values fall off, both estimate the point where added states stop buying genuine dynamics and start buying noise. The reasoning is identical to capacity control in statistics.

Closed-loop identification

Identifying a plant while a controller is running introduces a bias that a naive fit walks straight into. Feedback makes the input a function of past output and therefore of past noise, so the input and the noise are correlated, and an estimator that assumes them independent attributes some of the noise to the dynamics and returns a skewed model.

Handling it requires methods built for the correlation, whether by injecting a known external excitation that the noise cannot mimic or by modeling the controller explicitly in the fit. Closed-loop data is often unavoidable, because the plant cannot be run open loop safely.

Validation on held-out trajectories

A model is validated on trajectories it was never fit to, simulated forward over many steps rather than scored on one-step-ahead prediction. One-step error flatters a model, because predicting the next sample from the true past is easy even for a model whose errors compound over a horizon.

Multi-step simulation on held-out data is the honest test, the identification twin of a train-and-test split, and it exposes the compounding error that one-step scoring hides.

Adaptive control

Adjusting the controller online as the plant changes, estimating the plant or the controller parameters while the loop runs, which is powerful, occasionally unstable, and the classical relative of continual learning.

Run an estimator alongside the controller and update the control law from its current estimate, either by identifying the plant and recomputing the law or by adapting the controller parameters directly toward a reference behavior.

A plant whose parameters drift or are unknown at startup defeats a fixed controller, and adapting online tracks the change a fixed gain cannot. The same adaptation that tracks a real change can chase noise into instability.

The big idea

Slowly varying or initially unknown plants with enough excitation to keep the estimates honest. When excitation dies, the estimates drift, and unmodeled dynamics can turn a stable-looking scheme unstable without warning.

A fixed gain is a poor bargain when the plant starts unknown or drifts while the loop is running. Adaptive control updates estimates or controller parameters online, using the same operation of the loop to learn the behavior it must regulate. The upside is tracking a change without stopping for a new identification campaign; the danger is that weak excitation, noise, or unmodeled dynamics can make the adaptation chase artifacts into instability.

Direct against indirect adaptation

Indirect adaptation identifies the plant parameters online and then recomputes the controller from the current estimate, a clean two-stage structure that inherits the identification pitfalls of excitation and closed-loop bias.

Direct adaptation skips the plant and adjusts the controller parameters straight toward the desired behavior, folding estimation and control into one update. The two are duals more than rivals, and the choice turns on which parameterization is easier to write down.

Model reference adaptive control and the MIT rule

Model reference adaptive control specifies the desired behavior as a reference model and adjusts the controller parameters to drive the plant’s response to match it. The oldest update, the MIT rule from the late 1950s, moves each parameter down the gradient of the squared tracking error.

$$\dot{\theta} = -\gamma\, e\, \frac{\partial e}{\partial \theta}$$

The parameter vector $\theta$ moves at a rate set by the adaptation gain $\gamma$ times the error $e$ times the sensitivity of the error to the parameter. The rule is intuitive and has no stability guarantee, since a large adaptation gain $\gamma$ makes the parameters chase fast and can destabilize the loop.

Parameter drift and bursting

Adaptation needs excitation to stay honest, and when the excitation dies the estimates wander. With the reference quiet and the loop near its target, the update sees mostly noise, and the parameters drift on a random walk that the data no longer constrains, a failure mode called parameter drift.

Drift sets up bursting. The parameters wander until they reach a region where the loop is poorly behaved, a burst of oscillation erupts, the oscillation re-excites the system and briefly corrects the estimates, and the cycle repeats. A loop that looks settled for a long stretch and then erupts without an external cause is showing the signature of drift and bursting.

Rohrs counterexamples

The counterexamples that Rohrs, Valavani, Athans, and Stein published in 1985 broke the confidence of the field. They took the adaptive algorithms of the late 1970s, which had been proven stable under an exact-order model assumption, and showed that a small unmodeled high-frequency mode plus a bounded disturbance drives them unstable on plants that look entirely benign.

The result located the flaw precisely: the stability proofs assumed the true plant order, and real plants always carry unmodeled dynamics above the modeled band, so the guarantee rested on an assumption reality never satisfies. Robust adaptive control grew directly out of this, adding the excitation monitoring, normalization, and leakage terms that keep the estimates from drifting into the regime the counterexamples exploited.

L1 adaptive control

L1 adaptive control answers the Rohrs problem by separating how fast the controller adapts from how robust it stays. It permits very fast adaptation, which ordinarily courts the high-gain instability the MIT rule warned about, and then inserts a low-pass filter in the control path that keeps the fast adaptation from injecting energy at the high frequencies where the unmodeled dynamics live.

Hovakimyan and Cao developed the theory in their 2010 monograph, and its contribution is a knob that decouples two things earlier schemes bound together. The adaptation rate can be pushed high for fast response while the filter bandwidth sets the robustness margin independently.

Guardrails for online-updating models

An online-updating production model is an adaptive controller wearing different clothes, and it inherits the same failure modes. A model that updates on live traffic can drift when the traffic goes quiet, burst when the drift reaches a bad region, and destabilize on dynamics it never modeled. The guardrails are also the classical ones: bound the update size, monitor that the incoming data still excites what is being estimated, and hold a fixed fallback to revert to.

A system that keeps learning from a world it is simultaneously changing is a closed adaptive loop, and the decades of adaptive control experience with drift, bursting, and unmodeled dynamics are a direct warning to anyone deploying a model that updates itself. The online settings of reinforcement learning confront the same lesson from the learning direction.

Optimal control

The general problem of choosing an input trajectory that minimizes a cost over the dynamics, together with the two solution routes, dynamic programming and the maximum principle, that every method in the field descends from.

Either solve backward for a value function that satisfies a recursive optimality equation, or solve for a trajectory and a costate that satisfy the maximum principle's boundary value problem, then discretize whichever route into a solver.

Feedback laws, predictive controllers, and reinforcement learners are all approximations to this one problem. Naming the exact problem shows what each method approximates and where the approximation is expected to break.

The big idea

Deterministic or stochastic dynamics with a well-posed cost. The exact solution meets the curse of dimensionality, which is what forces the approximations that the rest of control and reinforcement learning supply.

Choosing a feedback gain, tuning a predictive controller and training a policy look like three different activities, and all three are approximations to one problem: pick the input trajectory that minimizes a cost, subject to the dynamics. Optimal control is that problem stated exactly. Two routes solve it, one working backward through the cost still to come and one solving for a trajectory and its shadow price together. Naming the exact problem is what shows which corner each practical method is cutting.

Dynamic programming and the Bellman equation

Dynamic programming solves the problem backward through a value function, the cost-to-go from each state under optimal play, which satisfies a recursive optimality equation.

$$V(x) = \min_{u} \; \left\{ \ell(x, u) + V\big(f(x, u)\big) \right\}$$

The value $V$ at state $x$ is the smallest sum of the immediate stage cost, which depends on state $x$ and input $u$, and the value at the next state given by the dynamics $f$. This is the same fixed-point equation that anchors reinforcement learning, where the Bellman backup and the value function are the objects of study, and its curse is dimensionality: the state space grows exponentially with its dimension.

Pontryagin maximum principle

The maximum principle solves for a trajectory rather than a field, introducing a costate that carries the sensitivity of the cost to the state and a Hamiltonian the optimal input must minimize.

$$\dot{p} = -\frac{\partial H}{\partial x}, \qquad H(x, u, p) = \ell(x, u) + p^{\top} f(x, u)$$

The costate $p$ evolves backward driven by the state gradient of the Hamiltonian $H$, which combines the stage cost with the costate weighting of the dynamics $f$, and the optimal input $u$ minimizes $H$ at each instant. The state runs forward from its start and the costate runs backward from a terminal condition, so the principle produces a two-point boundary value problem rather than a feedback law.

Direct against indirect methods

Indirect methods discretize the maximum principle’s boundary value problem and solve for the trajectory and costate that satisfy it, inheriting accuracy near the optimum and a notorious sensitivity to the initial guess for the costate, which has no physical meaning to guess from.

Direct methods discretize the trajectory and the input into variables and hand the whole thing to a nonlinear program, and collocation, which enforces the dynamics as equality constraints between sampled points, dominates practice. The direct route connects control to the solvers of optimization and tolerates constraints and poor initial guesses that break the indirect route.

Bang-bang solutions

When the input enters the dynamics and cost linearly, the Hamiltonian is linear in the input, and minimizing a linear function over a bounded input set puts the optimum at a limit. The optimal input then jumps between its extremes, a bang-bang solution, switching at the instants where the costate-dependent coefficient changes sign.

Minimum-time problems are the classic case, where reaching a target as fast as possible means full effort one way, then full effort the other. Bang-bang structure is a genuine feature of the optimum rather than a numerical artifact, and recognizing it tells a designer to solve for switching times instead of a smooth input.

Correspondence with reinforcement learning

Optimal control and reinforcement learning are the same problem under different assumptions about the model. With the dynamics and cost known, the problem is optimal control and gets solved by dynamic programming or the maximum principle; with the model unknown and only samples available, the same problem is reinforcement learning, and Benjamin Recht’s 2019 survey lays out the correspondence from the continuous-control side.

The map between the vocabularies is exact. The value function and the Bellman equation are shared outright, the cost is a negated reward, dynamic programming is the model-based ideal that temporal-difference methods such as Q-learning approximate from samples, and the maximum principle’s costate is the adjoint that policy-gradient methods estimate.

Differentiable simulators and gradients through dynamics

When the dynamics are written in a differentiable simulator, the gradient of the cost with respect to the inputs can be computed by backpropagation through the rollout, which is the maximum principle’s adjoint sweep implemented by automatic differentiation. The costate that the indirect method struggled to guess becomes the reverse-mode gradient the simulator produces for free.

The technique powers learned control on contact-rich systems, where analytic gradients through a physics engine train policies that sampling alone would learn slowly; robotics develops this where learned control meets friction and contact. The same gradients drive trajectory optimization and policy search alike, so a differentiable model collapses the distance between control and learning. Quantitative teams building execution and hedging policies in quantitative finance exploit this when their environment is simulated rather than merely observed.