MountainCar Q-Learning

A tabular reinforcement-learning project that trains a Q-learning agent to solve Gym's MountainCar-v0 environment. The project studies continuous-state discretization, epsilon-greedy exploration, Q-table updates, policy evaluation, value-function visualization, and reward shaping.

The repository is prepared from a completed course assignment. The original code cells and their stored outputs were preserved exactly; only the notebook's explanatory Markdown was translated, reorganized, and polished for an English-language GitHub presentation.

• Implements tabular Q-learning from scratch.
• Maps continuous position and velocity observations to discrete state buckets.
• Compares constant, linear, logarithmic, and exponential epsilon schedules.
• Demonstrates the trade-off between coarse and fine discretization grids.
• Evaluates the learned policy over 100 episodes with exploration disabled.
• Visualizes the learned state-value function as a three-dimensional surface.
• Compares a sparse baseline reward, two shaped rewards, and a deliberately hacked objective.
• Includes saved results from the original experiments for reproducible inspection without retraining.

MountainCar-v0 contains an underpowered car in a valley. The car cannot drive directly up the right hill, so it must first move left, accumulate potential energy, and then reverse direction with enough momentum to reach the flag.

The observation contains two continuous variables:

The agent can select one of three actions: accelerate left, coast, or accelerate right. The Q-table is updated with

$$ Q(s,a) \leftarrow Q(s,a) + \alpha\left[r + \gamma\max_{a'}Q(s',a') - Q(s,a)\right]. $$

The principal agent uses a 40 x 40 state grid, exponential epsilon decay, a learning rate of 0.05, a discount factor of 0.99, and seed 400.

The stored training log reports its first solution event at episode 288,136, with a 100-episode moving average of -109.99. The environment's solution threshold is -110.

The trained agent was evaluated for 100 episodes with epsilon = 0.

The best result demonstrates that the Q-table contains an efficient momentum-building policy. Variation across evaluation episodes shows that some initial position and velocity combinations still require additional oscillations.

Four epsilon strategies were compared for 100,000 episodes using the same 40 x 40 grid and seed.

Under this configuration, logarithmic decay provides the strongest and most stable evaluation results. Constant epsilon occasionally produces an excellent trajectory, but permanent random actions increase its variability. These results are empirical and depend on the selected hyperparameters and seed.

Individual diagnostic plots are also available in assets/results/.

The project compares an overly coarse grid with an overly fine grid.

The 5 x 5 grid suffers from state aliasing: one table entry represents physically different observations that may require conflicting actions. The 100 x 100 grid preserves more detail but distributes experience across many entries, so 100,000 episodes are insufficient for reliable state visitation and convergence.

The notebook visualizes $V(s)=\max_a Q(s,a)$ only for visited states. Regions with negative velocity can receive high value because moving left is a useful stage of the optimal strategy: it stores energy that is later converted into rightward momentum.

Four reward modes were trained for 2,000 episodes. Success is measured independently by whether the car reaches the flag, rather than by the shaped reward used for learning.

Both shaped variants learn much faster than the sparse baseline in the stored experiment. The hacked objective encourages high-speed oscillation but never reaches the goal, demonstrating specification gaming: maximizing the written reward can fail to satisfy the intended task.

• The experiments initialize NumPy and Gym action-space randomness with seed 400.
• Environment resets derive deterministic episode-specific seeds from the base seed.
• Runtime and exact learning curves can still vary across operating systems and library versions.
• Gym emits maintenance and deprecation warnings in the saved output; these do not alter the recorded experiment results.
• Packaging for this repository changed no code cell and no saved code output in the submitted notebook.

The final notebook section contains ten ordered questions and answers covering:

• continuous-state discretization;
• state aliasing and the curse of dimensionality;
• epsilon-greedy and alternative exploration policies;
• the effects of learning rate and discount factor;
• constant exploration and greedy evaluation;
• sparse state visitation;
• the MountainCar energy interpretation; and
• reward hacking and its common forms.

Faeze Mohammadi - 810103510