Deep Reinforcement Learning & Meta-Learning Series

Summary

The website content provides a comprehensive series on Deep Reinforcement Learning and Meta-Learning, offering insights into various algorithms, techniques, and applications within the field.

Abstract

The provided web content outlines a series of articles dedicated to Deep Reinforcement Learning (DRL) and Meta-Learning, emphasizing the complexity and rewards of making decisions through algorithms that learn from interactions with an environment. The articles cover a range of topics from the basics of DRL, such as value learning and policy gradients, to advanced concepts like Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO). The series also delves into model-based RL, meta-learning, and practical applications, including a discussion on AlphaGo Zero. The author aims to explain these concepts with clarity, avoiding overly fancy talk, and includes equations where necessary to deepen understanding. The series is designed to be a resource for readers to start their journey into DRL and Meta-Learning, with a promise of more articles to be released in the future.

Opinions

The author values simplicity in explaining complex DRL concepts, aiming to provide clear understanding rather than impressing with technical jargon.
There is an emphasis on the importance of understanding equations to gain a deeper grasp of DRL topics, suggesting that a balance between theoretical and practical knowledge is crucial.
The author acknowledges the efficiency of human learning compared to current RL methods and discusses techniques that aim to narrow this gap.
The series takes a comprehensive approach, covering both the breadth of DRL topics and the depth of specific algorithms and techniques.
The author expresses a commitment to continuing education in the field, with plans to publish additional articles to further assist readers in their learning journey.
The content reflects the author's belief in the significance of meta-learning, suggesting that learning how to learn is a fundamental aspect of intelligence and a key area of focus in the field.
The series includes a critical examination of various RL algorithms, providing a comparison and tips for selecting the most appropriate one for different scenarios.
The author recognizes the contributions of other researchers and educators in the field, expressing gratitude for the collective knowledge that has informed their work.

Deep Reinforcement Learning & Meta-Learning Series

Deep Reinforcement Learning is about making the best decisions for what we see and what we hear. It sounds simple but making a decision is never easy. This subject is one of the hardest and one most rewarding. I try to explain things with an easy to understand angle. I don’t want to fill my readers with fancy talks that feel good but learn nothing. In reality, simplicity makes me see through the subject in better clarity. But I don’t want to skip the equations either. It just needs to be introduced in the proper manner. Understand them helps us to go deeper.

While there are still many articles need to be reviewed before publishing, the published one should give you enough details to start your journey. For the remaining articles, I will try to release them in 2019 Spring. So stay tuned.

Credit and references

Reinforcement learning is a huge topic and I owe a lot of debt to many professors, researchers, and bloggers. It is impossible to quote all videos, classes, research papers, and blog that I read. In fact, there are other university courses that help me a lot but I cannot recall the institutes anymore.

For here, I want to list a few that has the biggest impacts on me.

But I want to single out the UC Berkeley Reinforcement Learning course which offers every year for now. I start watching it in 2015. It is a tough course. The lesson on LQR almost made me give up RL. But with some perseverance, that makes the biggest impact on me. I hope it can have the same impact on you too.

Deep Reinforcement Learning & Meta-Learning Series

Overview

RL— Introduction to Deep Reinforcement Learning

Deep reinforcement learning is about taking the best actions from what we see and hear. Unfortunately, reinforcement…

RL — Deep Reinforcement Learning (Learn effectively like a human)

A human learns much efficient than RL. In this article, we will study other methods that may narrow this gap.

RL — Transfer Learning (Learn from the Past)

Humans are explorers and we do it smartly. In reinforcement learning RL, model-free methods search the solution space…

Value-learning

RL — Value Learning

Value learning is a fundamental concept in reinforcement learning. It is as basic as the fully connected network in…

RL — Value Fitting & Q-Learning

We can learn the value function and the Q-value function iteratively. However, it cannot scale well to large state…

Monte Carlo Tree Search (MCTS) in AlphaGo Zero

AlphaGo Zero uses MCTS to select the next move in a Go game.

Q-learning

RL — DQN Deep Q-network

Can computers play video games like a human? In 2017, a professional team beats a DeepMind AI program in Starcraft 2…

Policy Gradients

RL — Policy Gradients Explained

Policy Gradient Methods (PG) are frequently used algorithms in reinforcement learning (RL). The principle is very…

RL — Policy Gradients Explained (Part 2)

In the first part of the Policy Gradients article, we cover the basic. In the second part, we continue on the Temporal…

RL — Actor-Critic Methods: A3C, GAE, DDPG, Q-prop

undefined

RL — Natural Policy Gradient Explained

Policy Gradient methods PG are popular in reinforcement learning RL. PG increases the chance of taking actions that…

RL — Trust Region Policy Optimization (TRPO) Explained

TRPO, one of the most popular Policy Gradient methods (PG), addresses the convergence problem by introducing the…

RL — Trust Region Policy Optimization (TRPO) Part 2

After discussing the basic concepts, we will discuss the detail of TRPO in Part 2. (TRPO is one of the most popular…

RL — Appendix: Proof for the article in TRPO & PPO

Difference of discounted rewards

RL — Actor-Critic using Kronecker-Factored Trust Region (ACKTR) Explained

In a previous article, we explain how Natural Policy Gradient allows the Policy Gradient methods to converge better by…

RL — Proximal Policy Optimization (PPO) Explained

A quote from OpenAI on PPO:

RL — The Math behind TRPO & PPO

TRPO Trust Region Policy Optimization & Proximal Policy Optimization PPO are based on the Minorize-Maximization MM…

Model-based RL

RL — LQR & iLQR Linear Quadratic Regulator

Reinforcement learning can be divided into Model-free and Model-based learning. Model-free learning emphasizes heavily…

RL — Model-based Reinforcement Learning

In reinforcement learning RL, we maximize the rewards for our actions, which depend on the policy and the system…

RL — Guided Policy Search (GPS)

With Guided Policy Search GPS, a robot learns each skill in the video in 20 minutes. If it is trained by the Policy…

RL — Guided Policy Search (A walkthrough)

In the previous article, we discuss the concept of the Guided Policy Search. Now we look into how it is trained.

RL — Model-Based Learning with Raw Videos

Vision is a critical part of intelligence and the decision-making process. Many toy experiments avoid raw image…

Technologies

RL — Imitation learning

Imitation is a key part in the human learning. In the high-tech world, if you are not an innovator, you want to be a…

RL — Transfer Learning

We learn from past experiences. We apply learned knowledge to solve new tasks. In Deep Learning, training a deep…

RL — Inverse Reinforcement Learning

It is a major challenge for reinforcement learning (RL) to process sparse and long-delayed rewards. It is difficult to…

RL — Exploration

undefined

RL — Prediction

How can we learn better? This is something we struggle with in real life also. Besides meta-learning, humans make…

RL — PLATO Policy Learning using Adaptive Trajectory Optimization

Imitation plays a major role in learning. In RL, it reduces the amount of time in searching for solutions and it is…

Comparison & Tips

RL — Reinforcement Learning Algorithms Overview

We have examined many Reinforcement Learning (RL) algorithms in this series, for instance, Policy Gradient methods for…

RL — Tips on Reinforcement Learning

undefined

RL — Reinforcement Learning Algorithms Comparison

Choosing an RL algorithm can be confusing. In this article, we will focus on different decision factors in choosing…

Meta-learning

Meta-Learning (Learn how to Learn)

People learn continuously. We recall relevant skills and adjust them accordingly in handling new tasks. Overall…

Meta-Learning (Bayesian Meta-Learning & Weak Supervision)

In part 2 of our Meta-Learning article, we will discuss Bayesian Meta-Learning, Unsupervised Learning, and Weak…

RL — Meta-Learning

Many deep learning classifiers demonstrate superhuman performance but human still learns far more efficient than deep…

Neural Turing Machines: a fundamental approach to access memory in deep learning

Memory is a crucial part of the brain and the computer. In some areas of deep learning, we extend the capabilities of…

Cheat sheet