Mario_PPO_E3B

Playing Super Mario Bros using PPO and Exploration via Elliptical Episodic Bonuses (E3B)

My PyTorch Proximal Policy Optimization (PPO) and Exploration via Elliptical Episodic Bonuses (E3B) implement to playing Super Mario Bros. There are PPO paper and E3B paper.

Results

Recently, I have read and learned about some intrinsic reward methods. I find E3B to be a good theory with an amazing report in their paper. I want to implement this algorithm to test it on Super Mario Bros to compare it with other intrinsic reward algorithms like RND, DRND, NGU, NovelD, ...

You can use my notebook for training and testing agent very easy:

• Train your model by running all cell before session test
• Test your trained model by running all cell except agent.train(), just pass your model path to agent.load_model(model_path)

Or you can use train.py and test.py if you don't want to use notebook:

• Train your model by running train.py: For example training for stage 1-4: python train.py --world 1 --stage 4 --num_envs 8
• Test your trained model by running test.py: For example testing for stage 1-4: python test.py --world 1 --stage 4 --pretrained_model best_model.pth --num_envs 2

You can find trained model in folder trained_model

Below is a detailed hyperparameter table for full E3B. This will work for all stages.

• num_envs = 32, the same as previous projects.
• update_proportion = 0.1, tunning (1 not work and I try 0.1 --> it work)
• int_adv_coef, ext_adv_coef: 0.5 and 1, as in previous projects.
• gamma, gamma_int: 0.99, like previous projects.
• entropy_coef = 0.01: it just work (I don't need to tune this param)
• learn_step = 512, batchsize = 256, lambda = 0.95, epoch = 10, lr = 7e-5, target_kl = 0.05, clip_param = 0.2, max_grad_norm = 0.5, norm_adv = false, V_coef = 0.5, as in previous projects.
• norm_int_reward = rms: Normalizing intrinsic reward: I tried min-max scaling and dividing by running std. I find that RMS works better than min-max scaling. Especially, min-max scaling may not work for these algorithms. I tried not normalizing the intrinsic reward and it didn't work at all.
• lambda_C = 0.1: as in paper.
• embedding_dim = 256: as in paper.

• About Hyperparameters

  • I'm using this set of hyperparameters based on the ones I'm familiar with from previous projects. This isn't a standard or optimal set of hyperparameters. You can tune them.
  • Some hyperparameters are correlated; if you want to tune one hyperparameter, you need to check the others, for example: learning rate - batchsize, update_proportion - learning rate - batchsize - learn_step, gamma - gamma_int, ...
  • I see that E3B is very sensitive to hyperparameters. When I try update_proportion = 1 or normalize the intrinsic reward with min-max scaling, it doesn't work well. Maybe this project isn't good enough just because my hyperparameters are not tuned enough! Poor hyperparameters. I tried not normalizing the intrinsic reward and it didn't work at all.

• About reward normalization:

  • First, I didn't normalize the intrinsic reward, similar to NovelD, and it did not work at all.
  • Then, I compared min-max scaling and RMS on several stages:
    • 4-3, 4-4, 5-3, 6-3: Both worked.
    • 1-3, 8-4: Min-max scaling did not work. RMS completed 1/2 runs.
    • Therefore, RMS seems to work better.
  • I used RMS to complete all other stages.

• About E3B reward:

  • I see that E3B more sensitive with hyperparameters (same as RND, poor hyperparameters --> not work). NovelD and NGU working without tunning hyperparameters.
  • The algorithm is not as good as NGU and NovelD because stages 1-3 and 8-4 were only completed 1/2 times in the experiment. Stage 8-4 was completed partly because the agent learned a rather unique jumping technique instead of discovering the hidden brick.
  • The reward of E3B is not stable:
    • The range of intrinsic reward values ​​varies greatly, from tens to thousands. This might be why the choice of normalization method has a greater impact than other algorithms. See figures below.
    • When the episode is reset, the intrinsic reward for state 1 (the next state after the reset) is always very high (almost outlier).

World-Stage	E3B	RND	NGU	NovelD
1-3
5-3
8-1
8-4

The data is based on my old projects (NGU, NovelD). Note: because I don't log the reward in RND, I use the reward in the NGU project.

• python 3>3.6
• gym==0.25.2
• gym-super-mario-bros==7.4.0
• imageio
• imageio-ffmpeg
• cv2
• pytorch
• numpy

With my code, I can completed all 32/32 stages of Super Mario Bros.

• facebookresearch e3b
• CVHvn PPO_RND
• Stable-baseline3 PPO
• lazyprogrammer A2C
• jcwleo RND
• DI-engine RND
• vwxyzjn cleanrl/ppo_rnd_envpool.py