RL Framework and Applications

Main ideas / lessons

• Core loop of Reinforcement Learning (RL):

  • An agent learns by interacting with an environment.
  • The agent observes the current state of the environment.
  • It chooses an action based on that state.
  • The action changes the environment’s state.
  • The agent must not only pick actions that are good now, but actions that lead to better future states.

• “Optimal behavior” is about long-term reward, not immediate gain:

  • Example concept: taking a “high immediate reward” move (e.g., capturing a queen) can still be bad if it leads to a losing position later.
  • RL formalizes the objective as maximizing cumulative reward over time.

• Rewards/evaluations are often modeled as a numeric signal (scalar), but this is a modeling convenience:

  • In mathematical RL, the environment provides an evaluation/reward signal.
  • In real biological learning, there may not be a clean “reward channel”; instead, organisms interpret sensory inputs as reward/punishment.
  • Reward can be:
    • Scalar (single number) like +5, -100, +20
    • Vector-valued, which introduces trade-offs and multi-objective / Pareto optimality complexity.
  • The speaker notes an upcoming exercise to argue for/against scalar rewards.

• Why RL differs from supervised/unsupervised learning:

  • Not supervised learning:
    • Supervised learning has an input → output mapping with a known target, enabling an error signal to train from.
    • In RL, there is no target output; the agent receives evaluation as feedback.
    • Learning relies on trial-and-error exploration because the direction of improvement isn’t known.
  • Not unsupervised learning:
    • Unsupervised learning primarily discovers patterns in data.
    • RL focuses on choosing actions to affect outcomes.

• Temporal Difference (TD) learning intuition: predicting future outcomes improves with time:

  • RL often resembles predicting the eventual reward/outcome (e.g., win probability).
  • TD learning uses a “bootstrap” idea:
    • Prediction at time t+1 is typically more accurate than at time t.
    • When reality reveals that the next-state prediction differs, update the earlier estimate.

• Exploration is necessary (exploit vs explore):

  • If the agent always picks the currently best action, it may only visit a small part of the state/action space and never learn better alternatives.
  • RL uses a strategy to explore sometimes and exploit known good actions sometimes.
  • This leads to the exploration–exploitation dilemma, related to simplified multi-armed bandit problems.

• Two ways of learning value estimates in Tic-Tac-Toe (as presented):

  1. Wait-until-end approach:
     • Play a complete game repeatedly.
     • After the game ends, update earlier states based on whether the final outcome was win/loss/draw.
  2. Temporal Difference (TD) approach:
     • Update values during the game by using the value/prediction of the next state rather than waiting for the final outcome.

• General RL algorithm taxonomy (high-level):

  • Dynamic programming (offline, uses repeated structure / known model aspects).
  • Temporal-difference family / online approximate dynamic programming (e.g., TD, TD(λ), Q-learning, SARSA, actor-critic).
  • Policy search methods (optimize behavior policies directly).
  • The speaker emphasizes RL methods often relate to these families.

---

Methodology / instruction-like content

RL interaction and decision-making loop

• Observe state from environment.
• Select an action based on state.
• Apply the action to the environment.
• Receive reward/evaluation (possibly delayed/noisy).
• Update the agent so that future choices lead to higher long-run return (not just immediate reward).

Modeling reward as scalar (as described)

Convert outcomes into a single numeric scale, e.g.:

• hurt → -100
• food → +5
• win → +20
• capturing a piece → +5

Then optimize by maximizing expected cumulative scalar reward over time.

Supervised vs RL training signal

• Supervised learning:
  • input → output compared to a target
  • compute an error
  • use error gradients to update parameters
• Reinforcement learning:
  • input/state → action
  • receive evaluation directly (no target)
  • use trial-and-error; sometimes update using exploration outcomes

Temporal Difference learning (conceptual update rule described)

• Maintain a prediction (e.g., probability of winning / expected reward) for each state.
• When moving from state sT to sT+1:
  • compare predicted value at sT versus the “better” estimate implied by sT+1
  • update sT accordingly:
    • if best-next-state estimate is lower than what you predicted, reduce the earlier prediction
    • if higher, increase it
• Key notion: predictions improve as you get closer to the end, so you bootstrap from the next step.

Tic-Tac-Toe RL formulation (as used in the talk)

• Define:
  • Agent: player X
  • Environment/opponent: player O
  • State: board positions
  • Actions: placing X in available squares
• Reward scheme (given):
  • win → 1
  • lose → 0
  • (draw handling discussed as making learning non-informative if opponent is perfect)
• Repeat many games to estimate:
  • expected reward / win probability from each state
• Choose next move:
  • evaluate possible next states
  • move to the one with the highest estimated expected value (e.g., highest win probability)

Exploration requirement in RL policy learning

If the agent always chooses the currently best action:

• it may overfit to visited parts of the game tree
• it may not discover better actions

Therefore:

• occasionally choose non-greedy actions (e.g., actions with lower estimated win probability)
• learn from those trials to correct value estimates

Related topic introduced:

• explore–exploit dilemma
• bandit problems as a simplified case

---

Speaker/source list (as featured in the subtitles)

People / organizations

• Course instructor / lecturer (unnamed; appears to lead the class, ask questions to the audience, and present RL concepts)

References / works mentioned

• David Silver (implied “Deep Mind” context; “Deep Mind paper/Deep Mind fellows” referenced)
• Tom Mitchell (mentioned for a short introduction to RL in his book)
• Russell & Norvig (mentioned for Artificial Intelligence)
• Sutton & Barto (mentioned as the main RL textbook, 2nd edition)
• Puterman (mentioned for MDP-related groundwork: “Mark of Dynamic processes by Puterman” / MDPs)
• Bertsekas & Tsitsiklis (mentioned as a mathematically grounded RL introduction)
• Dian and Abbott (mentioned regarding neuroscience/behavioral psychology history; exact first names unclear in subtitles)
• Dean and/or colleagues at University of Alberta and DeepMind/Google DeepMind (Atari learning environment + DeepMind agent described; specific individual names not clearly given)
• Pavlov (referenced via “Pavlov’s dog” experiment)
• Groundhog Day (movie referenced as an analogy)
• IBM (mentioned as a contrast—Google/DeepMind highlighted as the more recent hot source)