Hacking Chess with Decision Making Deep Reinforcement Learning

Creating a Chess Champion with Machine Learning

The image below is a checkmate applied by an Artificial Intelligence algorithm on a Chess.com engine that simulates a 2650 (which is quite high) Grandmaster — Daniel Naroditsky “Danya”.

In this article I will introduce the concept developed for this Deep Reinforcement Learning algorithm and how it won this fierce machine fight!

“Oooof.. well, you have defeated Danya. I hope you feel good about yourself :)” — Danya engine

Chess is a very old strategic board game and its tradition has already transformed it into a sport, an art and finally a science.

Due to its complexity, the human being has tried for many years to create models to hack chess and discover once and for all a way to become invincible. Today, with the great computational advance, we have models that can calculate several steps ahead and achieve excellent results like Google’s alpha-zero model.

Passionate about chess and without Google huge resources, it’s my time to show that knowing few things about chess but a lot about algorithms we could be able to create a Chess Champion, and try to challenge some of the strongest algorithms in the area.

Decision Making Deep Reinforcement Learning

Decision Making Deep Reinforcement Learning is a concept that uses Deep Reinforcement Learning to optimize decision making through previously defined strategies. The main idea is that from a diverse set of good strategies, the algorithm can choose the best of them for each situation faced.

A Deep Reinforcement Learning model observes each state within an environment and uses a neural network to choose a particular action. This action is then taken in the environment, which is changed, and then the agent makes a new observation of the new state and prepares the next action. Each action generates a reward for the agent and the agent’s goal is always to choose an action that maximizes this reward in each state.

And with this mix of chess games, Deep Q Learning, LSTM and engine, CharlesChessAI is born!

Simulation Environment — Python Chess and Gym-AI

The first step is to create a simulation environment so that the model can observe the states and take actions, basically a playground for playing chess. For this I made use of several libraries combined in Python: chess, gym and gym chess. These libraries allowed me to reproduce and observe all the movements on the chess board, simulating a game, besides obtaining analytical scores and good definitions of the states and movements, with check, checkmate, possible movements and more.

CharlesChessAI — The Intelligence

CharlesChessAI’s intelligence is based on 3 predefined strategies:

• id: 0 — Human Long Term Memory: Uses a sequence of plays within a base of 20,000 games played by humans and filtered only with games completed in checkmate. This option is focusing in the long term because it always ends in a checkmate for the algorithm.
• id: 1 — Human Short Term Memory: Next Word Predictor with LSTM previously trained on a large game base considering a sequence of 10 moves. The model uses chess movements as a string and interprets the movements within a game as words in a sentence. Chess behaving like a conversation between players. This option is focusing in the short term because it is bases in the last 10 steps and generates only the next one.
• id: 2 — Engine: The use of Engine Stockfish. Stockfish is a free and open source chess engine that analyzes the game and calculates some moves ahead to choose the best move.

CharlesChessAI x Danya

Below is a complete video showing all the movements of CharlesChessAI’s match against Danya that lasted about 60 moves and 10 minutes.

During the match with Danny the model stores all of his decisions and we can analyze them in the graph below and understand what was the behavior and decision making during the match.

At the beginning of the game, until move 25, we see the model making a well-balanced switch between strategies but using a lot of short-term memory strategy with LSTM. In the middle of the game, until move 45, we see the model using the engine more than at the beginning and alternating more with long-term human memory, which guarantees good moves to take the game to an advantageous end. At the end of the game, we see that the model uses a lot of the human memory option to reach the checkmate.

The code, still in development, is in my GitHub repository below:

Conclusion

The decision making model makes the algorithm choose the best strategy for each game situation, and the portfolio of diversified strategies brings extra creativity from great players to an engine, an excellent combination between man and machine. This algorithm is also extremely dynamic and adaptive, since according to the model playing more games and facing new situations it learns how to optimize the decision for each situation.

In the same way that this type of model was applied to Chess, it can be applied in other diverse environments for decision making.

I leave you with another chess game played between CharlesChessAI (in white) and a Windows application engine called Chess Lv100 at its maximum difficulty. For now the model is undefeated both with matches against engines and humans and I will make more matches with them and make available on the same channel. A good name for the match below “The dance of the towers” and you can better follow the video.

CharlesChess to win!

I hope it was a good read! I will be providing more information and CharlesChess matches and will be available for contact on LinkedIn: https://www.linkedin.com/in/octavio-b-santiago/