RTD3 (Recurrent twin-delayed deep deterministic policy gradient)

Summary

The video explains how the creator upgraded a predator–prey reinforcement-learning project from a basic recurrent actor–critic approach to RTD3 (Recurrent Twin Delayed Deep Deterministic Policy Gradient).

Starting Point: A Simple Predator–Prey Task

• The predator receives an image of its surroundings and chooses how to turn.
• An actor selects actions, while a critic estimates the rewards those actions will produce.
• The predator is rewarded for catching the prey and punished otherwise.
• In the original setup, the prey was stationary, the predator moved at a fixed speed, and the predator had a time limit.
• Training could produce a predator that caught the prey, but the creator wanted a more capable and realistic system.

Making the Task More Complex

The creator made two major changes to the environment:

• Give the predator control over its speed. Moving faster consumes more energy, creating a trade-off between sprinting and conserving energy.
• Replace the arbitrary time limit with an energy limit. The predator loses when it runs out of energy.

These changes give the predator meaningful decisions, but they also made training unreliable. Sometimes the predator learned to stop moving because doing so seemed to reduce its punishment.

Giving the Recurrent Model the Full History

The earlier system fed the last eight observations into the model. A recurrent layer can instead process observations one at a time while maintaining a hidden state that summarizes the past:

1. Begin with an initial hidden state of zeros.
2. Feed in the latest observation and the current hidden state.
3. Use the layer’s output for the next part of the model, and keep its updated hidden state.
4. Repeat for each new observation so the hidden state carries information from the entire sequence.

Both the actor and critic use recurrent layers, so each has its own hidden state. The observation was also expanded to include the image and additional information, such as speed, energy, and the previous action.

What RTD3 Changes

RTD3 adds several techniques to the recurrent actor–critic setup:

• Off-policy learning and experience replay: The predator stores experience and can train on it repeatedly, rather than learning only from the latest episode.
• Sequential replay: Because recurrent models need context, the replay buffer stores whole episodes and replays them from beginning to end. The creator avoids saving hidden states with individual memories because those states were produced by older versions of the model and may no longer fit the updated model.
• Two critics: The more pessimistic critic is used when estimating value, reducing the chance that the actor and critic reinforce overly optimistic actions.
• Delayed actor updates: The critic is trained more often than the actor, giving it more time to evaluate actions before the actor changes.
• Noise during actor training: Noise is added as part of the training procedure.
• Target models: Copies of the actor and critics are maintained for off-policy training.

Together, these additions make the system substantially more complex. With an actor, two critics, and three target models, the predator must track six recurrent hidden states. Because the LSTM used by the creator has two kinds of hidden state, that amounts to twelve hidden-state values or structures to manage.

Demonstrating Behavior and Remaining Challenges

The creator shows a manually controlled simulation in which the predator searches for the moving prey, adjusts its direction and speed, and eventually catches it. The demonstration illustrates how the environment works and how a person can provide initial experience for the predator.

The creator says that manually supplying some of the predator’s first memories made training more reliable, though RTD3 still did not succeed every time.

The video ends with ideas for future, more elaborate simulations: collisions might drain energy rather than end the game; there could be multiple predators and prey that cooperate or compete; and the simulation could include offspring and generations of agents.

Speakers and Sources Featured

• The video’s narrator and creator, who explains the project and demonstrates the simulation.
• The narrator’s father, who says that once a program works, people come up with ideas to improve it—and those ideas initially do not work.
• Professor Joontani at OIST, credited with suggesting that the predator control its speed and have an energy system.
• An unnamed PhD student at OIST, credited with suggesting RTD3.
• “Recurrent Off-Policy Baselines for Memory-Based Continuous Control,” the paper cited as the description of RTD3.