Video Summary - AI Invents New Bowling Techniques

Re-uses an RL Approach (PPO)

The video builds on a previous “Spider-Man AI” project, again using the PPO (Proximal Policy Optimization) reinforcement learning algorithm. The creator frames this as mostly “for fun,” but emphasizes how reward design shapes agent behavior.

Bowling Environment Setup

The “bowling” test runs in a Unity-like physics environment with:

A ragdoll character (noted as having 12 joints / 13 bones)
A ball that must stay in the lane while knocking pins

The creator describes the objective as knocking all 90 pins, including an early misconception that it might only require knocking the “middle 10.”

Key Reward Function Design (Main Technical Contribution)

The main technical contribution is the reward shaping, including:

Lane-keeping reward/punishment
- Rewards are given only when the ball stays within a specific lateral range
- The creator avoids overly harsh “punishment framing”
Forward progress reward
- Reward is proportional to forward speed/distance traveled “every frame”
Exponent on speed reward
- Adds an exponential term, making faster throws worth disproportionately more reward
- Played for laughs as reducing “imaginary negative rewards”
Encouraging posture
- Adds reward proportional to the AI’s head Y coordinate to encourage staying upright/high

AI Control Interface (State / Action Definition)

Observations (State)

For each joint, the AI observes:

Position
Velocity
Angular velocity
Plus the angle it’s pointing toward

Actions

The AI outputs:

Control signals for angles
Multiple degrees of freedom are allowed (possibly 2–3 angles)
It may also decide when to release the ball

Training Results and Failure Modes

Across multiple sessions:

Session 1: The agent learns to stand up rather than bowl
- Interpreted as reward hacking / incomplete objective learning
Session 2: It begins bowling “reasonably,” including a quirky “spell/straight casting” behavior
Session 3: It exploits body elasticity to launch the ball in ways that look physically exaggerated

Analysis: Local Optima / Reward Hacking

The creator explains that even when agents can increase reward, they may become stuck in local optima—optimizing parts of the reward rather than the overall objective. For example, the agent may choose fast throws that ignore accuracy.

Reward Function Fixes (Tuning Iteration)

The creator adjusts the reward function with three main changes:

Reduce reward for staying upright
- To prevent the “two-step / stand-up” strategy from dominating
Punish horizontal movement
- To force straighter, more accurate throws
Cap the exponential speed reward
- To stop reckless behavior caused by the exponent (the agent was throwing as fast/high as possible regardless of accuracy)

Improved Outcome

After the reward tweaks, the agent produces an AI that can:

Fall straight
Be capable of getting strikes

This suggests better alignment between the reward function and actual bowling performance.

Next Step / Extension: Pins Knowledge & Spin

Even with improvements, the bowler still lacks:

Knowledge of pin positions, limiting aim
Spin/control, which is important in real bowling and requires additional state/action inputs

The creator argues that simply reusing the old network isn’t straightforward because adding inputs/outputs typically requires retraining or architectural modification. They joke about “winging it” with “open brain surgery” (e.g., stitching in new neurons/weights).

They also add extra rewards for knocking pins down, explicitly noting that it wasn’t fully included earlier.

Main Tutorial Takeaway

The video functions as a guide to how reward shaping in reinforcement learning (especially with PPO) can lead agents to exploit loopholes. It also demonstrates how iterative changes to reward/state/action design can reduce sub-optimal local solutions.

Main Speakers / Sources

Primary speaker: The video’s creator/host (speaking in first person; no other named person is identified in the subtitles).

AI Invents New Bowling Techniques

Key takeaways