Implements Monte Carlo prediction and Monte Carlo control (on-policy and off-policy) for learning value functions and policies from sampled episodes.
Demonstrates every-visit MC, exploring starts (ES), ε-soft on-policy control, and importance sampling for off-policy learning.

• Python ≥ 3.10
• pip install -r requirements.txt (use the repo-root requirements.txt)

# Monte Carlo prediction demo
python -m ch5_mon­te_carlo.examples.mc_prediction_demo

# On-policy MC control in GridWorld
python -m ch5_mon­te_carlo.examples.mc_control_onpolicy_gridworld

# Exploring starts (ES) control in GridWorld
python -m ch5_mon­te_carlo.examples.mc_control_es_gridworld

# Off-policy MC with importance sampling demo
python -m ch5_mon­te_carlo.examples.mc_offpolicy_is_demo

ch5_monte_carlo/
├─ __init__.py
├─ examples/
│  ├─ mc_control_es_gridworld.py
│  ├─ mc_control_onpolicy_gridworld.py
│  ├─ mc_offpolicy_is_demo.py
│  └─ mc_prediction_demo.py
└─ tests/
   ├─ __init__.py
   ├─ test_mc_control.py
   └─ test_offpolicy_is.py

• Estimates state-value and action-value functions by averaging returns from sampled episodes.

• Exploring Starts (ES): ensures sufficient exploration by starting from all state–action pairs.
• On-Policy Control: ε-soft policies improve gradually from data generated by the same policy.
• Off-Policy Control: learns optimal policy from data generated by a different behavior policy using importance sampling.

pytest -q ch5_monte_carlo/tests

Covers:

• Convergence of MC prediction
• Correctness of ES and on-policy MC control
• Stability of off-policy MC with importance sampling

• Chapter 2 (RL Problem Formulation): foundation in MDPs and Bellman equations
• Chapter 3 (Multi-Armed Bandits): exploration strategies
• Chapter 4 (Dynamic Programming): exact solutions with known models
• Chapter 6 (Temporal-Difference Learning): bootstrapping methods bridging DP and MC