avatarCaleb M. Bowyer, Ph.D. Candidate

Summary

This article discusses three strategies for decaying epsilon in epsilon-greedy reinforcement learning: linear decay, exponential decay, and discrete interval decay.

Abstract

The exploration-exploitation dilemma is a fundamental aspect of reinforcement learning problems. The epsilon-greedy training strategy is a common approach to addressing this dilemma, where the parameter epsilon represents the probability of selecting a random control. At the beginning of training, epsilon starts at 1.0, and near the end, it should be a very small value. This article presents three methods for decaying epsilon effectively: linear decay, exponential decay, and discrete interval decay. Linear decay involves reducing epsilon linearly over time, while exponential decay involves reducing epsilon exponentially over time. Discrete interval decay involves reducing epsilon at specific intervals. The article provides Python code for implementing each of these strategies.

Bullet points

  • The exploration-exploitation dilemma is fundamental to reinforcement learning problems.
  • The epsilon-greedy training strategy is a common approach to addressing this dilemma.
  • Epsilon represents the probability of selecting a random control.
  • Epsilon starts at 1.0 at the beginning of training and should be a small value near the end.
  • Three methods for decaying epsilon effectively are presented: linear decay, exponential decay, and discrete interval decay.
  • Linear decay involves reducing epsilon linearly over time.
  • Exponential decay involves reducing epsilon exponentially over time.
  • Discrete interval decay involves reducing epsilon at specific intervals.
  • Python code for implementing each of these strategies is provided.

Strategies for Decaying Epsilon in Epsilon-Greedy

Photo by Dids on Pexels

The exploration-exploitation dilemma is fundamental to Reinforcement Learning (RL) problems. Early on in training an agent has not learned anything meaningful in terms of associating higher Q-values to certain controls in different states mainly because it hasn’t collected enough experience yet. Later on, after “enough” experience has been collected, it should begin exploiting its knowledge through the Q-values to act optimally in the environment.

How to Decay Epsilon During Training?

There are a few basic things that must be stated about decaying epsilon for the epsilon-greedy training strategy. In epsilon-greedy the parameter epsilon is our probability of selecting a random control. At the beginning of a training simulation epsilon starts at 1.0 and near the end it should be a very small value, e.g. 0.001 for reasons already discussed.

The reason epsilon should be probability one for selecting a random control starting out training is that we want the agent to explore many different controls across the state space (exploration). The reason we want epsilon small near the end of the training horizon is that we want the agent to exploit what it has learned.

In this article, I present three ways of decaying epsilon effectively and provide the necessary Python code you can add to your training simulation code. I also assume one million training steps, and fixed ending epsilon values, but these values an be played with in your own experiments. The three methods I present for decaying epsilon are:

  • Linear Decay
  • Exponential Decay
  • Discrete Interval Decay

Linear Decay

Exponential Decay

Discrete Interval Decay

That’s it for this story. Start learning about RL, Python, and other high-value topics today by firstly following this blog; if you want to stay in the loop and never miss a story of mine then subscribe to my email list. Consider becoming a Medium member to gain unlimited access to my writing and the writing of other authors:

Also, if you’re interested in learning specifically more about Reinforcement Learning (RL), then take a look at this list of stories I have curated:

Until next time,

Caleb

Reinforcement Learning
AI
Machine Learning
Artificial Intelligence
Learning
Recommended from ReadMedium
avatarAlexander Nguyen
The resume that got a software engineer a $300,000 job at Google.

1-page. Well-formatted.

4 min read
avatarFellipe Marcellino
Temporal-difference learning: from SARSA to Q-Learning

An introduction to temporal-difference methods: Sarsa, Expected Sarsa, Q-Learning and Double Q-Learning.

15 min read
avatarWouter van Heeswijk, PhD
Proximal Policy Optimization (PPO) Explained

The journey from REINFORCE to the go-to algorithm in continuous control

13 min read
avatarRakshit Kalra
The Difference Between Model-Based and Model-Free Reinforcement Learning

Understand when to use model-based or model-free approach for your RL problem

5 min read
avatarRenu Khandelwal
Unlocking the Secrets of Actor-Critic Reinforcement Learning: A Beginner’s Guide

Understanding Actor-Critic Mechanisms, Different Flavors of Actor-Critic Algorithms, and a Simple Implementation in PyTorch

6 min read
avatarEdwina Gu
Gentle Introduction to RL — MC Policy Control

from scratch in Python by playing Hangman Game

5 min read