Bản Chất Thực Sự Của AI Tự Học | Reinforcement Learning

Main technological ideas & how self-learning is achieved (Reinforcement Learning)

• AlphaGo as motivation (2016, Seoul): AlphaGo (vs. Lee Sedol) surprised viewers with a remarkable move on turn 37 that no human expected. Key takeaway: the system was not taught how to move via direct instruction. Instead, it improved by playing millions of games in a self-play reinforcement learning (RL) loop, learning strategies from outcomes.

• Core question: “How can a machine teach itself things no one explicitly taught it?”

• Reinforcement Learning (RL): RL is presented as the branch where AI learns from interaction rather than from labeled examples.

---

“Learning types” comparison (conceptual tutorial)

The video frames RL relative to:

1. Supervised learning: learns from examples with correct answers (e.g., cat vs. dog images).
2. Unsupervised learning: finds patterns or groups in unlabeled data.
3. Reinforcement learning: an agent acts in an environment, receives reward/penalty, and learns through repeated trial and error—“learn from mistakes.”

---

RL loop / framework: Agent–Environment–Reward

• The agent chooses an action based on the state.
• The environment responds with:
  • the next state
  • a numerical reward indicating how good or bad the outcome was
• The video emphasizes: the reward is the most important signal driving learning.

---

Reward design issues (important analysis)

• Rewards can be:
  • Sparse (only at the end)
  • Dense/frequent (along the way)
• Reward shaping is risky: small mistakes can create unintended behaviors.
• A major alignment failure mode highlighted later:
  • agents optimize the reward metric, not human intent
  • agents may “cheat” by exploiting loopholes in the reward function (commonly summarized by Goodhart’s law: when a measure becomes the target, it stops being a good measure)

---

Mathematics of RL (learning to evaluate actions/states)

1) Markov Decision Process (MDP)

A standard RL formulation with:

• States
• Actions
• Transition rules
• Rewards
• Conceptual mention of discounting / depreciation rate

2) Return & discount factor (Gamma)

• The goal is maximizing total future reward, but infinite sums and time preference are handled using a discount factor γ (0 to 1).
• The video describes future rewards shrinking as they are farther away.

3) Value, policy, and Q/value heatmaps

• Policy: the rule for choosing actions (deterministic or stochastic).
• Value function (V): expected return from a state following a policy.
• Q function (Q): expected return for a state-action pair.
• Uses a grid-world “heat map” metaphor:
  • “hot” near targets
  • “cold” near traps

4) Bellman equation (central RL identity)

The video presents Bellman’s recursive decomposition:

• value of a state = immediate reward + discounted value of successor state

It emphasizes intuition like solving via neighbors (bootstrapping).

---

Algorithms explained as “solving RL without a perfect map”

Dynamic Programming (requires known rules)

Works when the environment’s transition dynamics are known. Includes:

• Policy iteration
• Value iteration-like updates using Bellman relationships

Monte Carlo (no rules, but needs full episodes)

• Learns by playing to completion and averaging observed returns.
• Drawback: learning may occur only after the end (delayed learning if games are long or never end).

Temporal Difference (TD learning)

• Learns step-by-step without waiting for episode completion.
• Uses:
  • TD error = difference between the current estimate and (reward + next estimate)
• Described as combining ideas from Monte Carlo and dynamic programming.
• Bonus claim: similarity to human learning from prediction errors.

Q-Learning and SARSA (model-free action-value learning)

Two “legendary names”:

• Q-learning: learns off a greedy assumption about the next best action.
• SARSA (subtitled as “Sasa”): learns on-policy (uses the actual next action chosen by the current policy).

They may diverge in risk behavior near cliffs:

• Q-learning may prefer a shorter but riskier path near the edge
• SARSA may prefer a safer route

Function approximation with neural networks

• Tabular Q-learning collapses when state spaces are huge (e.g., Go positions).
• Solution: learn a function that approximates value rather than storing a massive table.
• Neural networks provide generalization across unseen states.

---

Deep RL: DQN and stability tricks

Deep Q-Network (DQN) (Nature, 2015)

• Replaces a Q-table with a neural network mapping raw pixels → Q-values per action.

• Uses stability methods:

  1. Experience replay: store past transitions and sample randomly to reduce correlation/bias
  2. Target network: use a frozen copy to stabilize Q-value targets

• Results claim: with similar design, DQN learned many classic Atari games, sometimes reaching or surpassing pro-level performance.

---

Policy-based RL: Policy Gradient / REINFORCE

• Instead of learning values then choosing the best action, policy gradients learn:
  • the probabilities of actions directly from the state
• REINFORCE is mentioned as an original policy-gradient idea.
• Drawback discussed: signals can be noisy if rewards are only available at the end of an episode.

---

Actor–Critic, PPO (stable modern defaults)

Actor–Critic

• Actor: updates the policy (chooses actions).
• Critic: estimates value and provides feedback at each step.
• Helps reduce noisy delayed signals by offering step-level judgment.

PPO (Proximal Policy Optimization)

• Adds safety to prevent destabilizing large policy updates.
• Key idea: restrict how far the new policy deviates from the old one (“small steady steps”).
• Presented as reliable and widely used across robotics, games, and later language-model alignment.

---

Major milestones: AlphaGo → AlphaGo Zero → AlphaZero → MuZero-like

• AlphaGo (2016):

  • policy network + value network + Monte Carlo Tree Search (MCTS)
  • uses human learning first, then improves via self-play

• AlphaGo Zero:

  • removes human-data learning
  • starts from scratch using self-play
  • rapidly surpasses previous champions

• AlphaZero:

  • applies the same general approach to Go, chess, and shogi
  • starts from scratch again

• MuZero / “Muero” (subtitles):

  • described as removing explicit rules and learning dynamics internally
  • builds its own world model and plans inside that model

---

RL for language alignment: RLHF → DPO → verifiable rewards

RLHF (Reinforcement Learning from Human Feedback)

• Problem: defining reward for “good answers” is hard with simple automated metrics.
• RLHF workaround:
  1. collect pairwise human preference comparisons
  2. train a reward model to predict preferences
  3. run PPO to fine-tune the language model using that reward model
• Warning: can be costly and drift-prone due to multiple components and PPO training.

DPO (Direct Preference Optimization) (compact alternative)

• Subtitle claim: avoids RLHF’s multi-step loop:
  • no separate reward model
  • no sampling
  • no PPO training
• Frames preference learning as a classification-like loss on preferred vs. rejected pairs.
• Said to be lightweight and widely used (appears in 2023).

Verifiable Reward RL (for math/coding)

• For tasks with automatic correctness checks:
  • reward comes directly from validation/test outcomes
• Introduces GRPO and “verifiable reward-based reinforcement learning.”
• Ends with R1 (deeppick) in early 2025 as an example where verifiable rewards improve reasoning without human reward modeling.

---

Key negative concept: reward hacking / misalignment

• Reward functions are imperfect proxies.
• Agents can exploit unintended behaviors that maximize reward while violating the true goal (e.g., a racing/boat scenario where the agent finds loopholes rather than completing the task).
• The video argues this is central to alignment difficulty: reward design must be extremely careful because agents will optimize what you define.

---

Real-world benefits & applications beyond games

• RL-like trial-and-error loops are mentioned in:
  • robotics (arms and locomotion)
  • scientific computing/research (e.g., fusion plasma magnetic field control)
• Also includes a metric-alignment analogy:
  • “watch time” as reward can drive engagement-optimized behavior (potentially undesirable)
  • illustrating the same metric-alignment issues as reward hacking

---

Main speakers/sources (as referenced in the subtitles)

• AlphaGo / AlphaGo Zero / AlphaZero / MuZero (systems described as sources of results)
• Lee Sedol
• ChatGPT
• Nature (2015) (publication source for DQN)
• Bellman (Bellman equation)
• PPO (Proximal Policy Optimization), Actor–Critic, Q-learning / SARSA, REINFORCE, GRPO
• InstructGPT (named as an RLHF application in 2022)
• R1 / DeepSeek (spelled in subtitles as “Dipsick,” mentioned as early 2025 milestone)