PPO on CartPole-v1, against A2C and DQN

Proximal Policy Optimization implemented from scratch in PyTorch — clipped surrogate objective, Generalized Advantage Estimation, advantage normalisation, entropy bonus and gradient clipping — compared against A2C and DQN on CartPole-v1. All three run 5 trials of 10⁶ steps across 5 parallel environments, evaluated every 10,000 steps over 10 episodes in a separate environment.

PPO's advantage is variance, not ceiling. A2C reaches roughly the same peak — about 490 at 430,000 steps — but oscillates between 400 and 490 for the rest of training with a standard deviation band spanning several hundred reward. PPO gets there by 200,000 steps and holds a flat band of about ±10 for the remaining 800,000. On a task where three of three algorithms can find a near-optimal policy, the interesting difference is which one keeps it.

Full write-up: report.pdf

A2C's curve is the one worth a second look. It sits below 100 reward until 250,000 steps — flat, doing nothing — then climbs almost vertically to 490 within 180,000 steps. Whatever it needs to discover, it discovers late and all at once. PPO's curve has no such phase.

Why PPO is steadier. Three mechanisms in the implementation push the same way. The clipped ratio caps how far a single update can move the policy, so a bad advantage estimate cannot destroy a working policy. GAE with λ = 0.95 blends multi-step returns rather than trusting a one-step TD error. And advantages are normalised to zero mean and unit variance before each update, so gradient magnitude does not scale with how long episodes happen to be running. A2C has none of these — its advantage is a raw one-step estimate and its update is unbounded.

The DQN baseline here is weaker than the same DQN elsewhere. In dqn-variants-cartpole, the target-network plus replay agent ends near 490. Here it ends near 350. The algorithm is the same; the settings are not — learning rate dropped from 5e-4 to 1e-4 and batch size from 256 to 64. The report attributes the gap to off-policy stability, but the retuning is the simpler explanation, and it matters for reading the comparison: DQN is not being shown at its best.

Sample efficiency has two meanings here. PPO reuses each batch of 640 transitions for 10 gradient epochs, so per environment step it does far more optimisation work than A2C's single pass. It reaches a good policy in fewer environment steps and more compute. On CartPole the environment is cheap and compute is not, which is the opposite of the setting PPO was designed for.

• Evaluation is not consistent across algorithms. evaluate_policy_actor_critic samples from the policy distribution, so PPO and A2C are evaluated with their stochastic policies; DQN's evaluate_policy_dqn takes the argmax. The report describes evaluation as excluding exploration noise, which holds only for DQN, and the entropy bonus keeps the actor-critic policies stochastic throughout.
• PPO does not use mini-batching, despite the report's abstract saying it does. The 10 optimisation epochs each run over the full 640-sample batch; the report's own Future Work section lists mini-batch processing as an improvement still to make.
• Seeding is partial. env.reset(seed=42 + run) seeds the environments per run, but torch, numpy and random are left unseeded, so network initialisation and action sampling still vary.
• Hyperparameters differ between algorithms — PPO uses a 256-unit hidden layer and lr 1e-3, A2C 128 units and 7e-4, DQN 128 units and 1e-4 — so this compares three separately configured agents, not three algorithms at matched capacity.
• Standard deviation is computed over 5 runs of a stochastic evaluation, which is why the A2C and DQN bands reach the plot limits.
• compare_all.py writes all1.png, not all.png, and needs the three .npy files produced by the individual scripts to be present first.

gymnasium
numpy
pandas
torch
matplotlib

pip install -r requirements.txt
python src/ppo.py
python src/a2c.py
python src/dqn.py
python src/compare_all.py

PPO settings: lr 1e-3, γ 0.99, 256-unit hidden layer, 128-step rollouts × 5 environments, 10 epochs per update, clip ε 0.2, GAE λ 0.95, entropy coefficient 0.01, value coefficient 0.5, gradient norm clipped at 0.5. Each script writes a learning-curve PNG and a *_eval_rewards.npy file; compare_all.py reads all three, so run it last. Scripts use CUDA when available. A full 10⁶-step run of all three takes several hours — reduce max_steps or num_runs in the config block for a quick check.

This is the third of three CartPole studies. The others are dqn-variants-cartpole and policy-gradient-methods-cartpole.

• Nallathambi Vethiappan
• Godwin Addetsi
• Federico Cucinotta