Andrew Ng Geoffrey Hinton Interview

Main ideas, concepts, and lessons from the interview

Purpose of the interview (framing)

• The host explains that deeplearning.ai wants to do more than teach deep learning technical ideas—it also wants to highlight the people who pioneered them.
• The series will include career advice from prominent deep learning leaders.
• The first interview focuses on Geoffrey Hinton, introduced as one of the most influential inventors behind deep learning (often called the “godfather of deep learning”).

---

Geoffrey Hinton’s personal origin story (how he got into AI/neural nets)

Triggering early curiosity

• In high school (~1966), a classmate suggested the brain uses holograms, sparking Hinton’s interest in how the brain might store memories distributedly.
• This connects to ideas like Lashley’s experiments, where a memory seemed hard to localize to a single brain region.

Academic path

• Began studying physiology and physics at Cambridge.
• Switched to philosophy (feeling it didn’t provide tools for judging falsehood).
• Switched to psychology, but found its theories too simple for brain complexity.
• Took time off and became a carpenter.

First major AI involvement

• Chose to study AI/neural networks at Edinburgh with D. Langer Higgins, who had shifted away from neural networks toward symbolic AI.
• Hinton continued focusing on neural networks despite disagreements.

Early career movement

• Earned a PhD in AI, but couldn’t find a job in Britain.
• Took a Sloan Fellowship to California.
• Found a more supportive environment for neural nets, collaborating closely with David Rumelhart.

---

Backpropagation: what happened and why it mattered

Development history

• In early 1982, Rumelhart, Ron Williams, and Hinton developed the backprop algorithm (Rumelhart’s idea was central).
• Later, it became clear others had similar versions independently:
  • Paul Werbos published earlier but received little attention.
  • Several related methods were also developed independently—so “backprop” may not map cleanly to one single inventor.

Breakthrough publication/acceptance

• The team secured acceptance of a Nature paper in 1986 after significant effort.
• Hinton also sought out a likely referee, Stuart Sutherland, and convinced him by emphasizing that backprop can learn representations for words.

Example that showed semantic feature learning

• Train on word triples like: “Mary has mother Victoria.”
• Task: given the first two words, predict the last word.
• Learned representations (word vectors) contained interpretable features such as:
  • nationality
  • generation level
  • family tree branch

Why it helped the community accept the work

• It bridged two views of knowledge:
  • Psychology: concepts as bundles of features
  • Traditional AI: concepts as graph/relational structures (semantic nets)
• The approach supported “to-and-fro” conversion:
  • from graph/tree structure → to feature vectors
  • from those features → back to consistent relational structure
• This supported generalization.

---

Computing power and the resurgence timeline (1986 → early 1990s → later deep learning)

Hardware speed as a key enabler

• Hinton highlights a major compute jump:
  • In 1986: using a “list machine” with < 0.1 mega-flops
  • By ~1993: around 10 mega-flops
• That’s roughly a 100× improvement, making training far more feasible.

Early embeddings from real text

• Later work (explicitly attributed to Bengio) used similar ideas with real English text to produce word embeddings, which impressed many people.

---

Hinton’s “most beautiful” inventions and why they mattered

1) Boltzmann machines (with Terry Sejnowski)

Core idea

• A learning algorithm for densely connected networks with partially observed nodes.
• Learns hidden representations through two phases:
  • wake
  • sleep

Difference from backprop

• In wake/sleep, information flow is more similar across phases.
• In backprop, there are distinct forward and backward passes with different roles.

From curiosity to practice

• Initially felt slow, but later simplifications produced:
  • Restricted Boltzmann Machines (RBMs)
• RBMs later contributed to performance in competitions (including the Netflix competition, as cited).

---

2) Deep belief nets / stacked restricted Boltzmann machines

Training by stacking

• Train an RBM to learn one layer of hidden features.
• Use learned features as data and repeat layer-by-layer.

“Derestricted” work and probabilistic inference

• Hinton describes work converting the effect of stacked RBMs into efficient inference for Sigmoid belief nets.

Key claimed property

• When learning each new layer, a form of variational improvement can occur:
  • adding layers yields a new bound that is better than the old one
  • discussed as a monotonic improvement property

---

3) Variational methods / approximate EM / tractable Bayesian learning

Relationship to statistics

• Hinton notes that statisticians had related ideas earlier.

Core improvement

• You don’t need a perfect E-step:
  • using an approximate E-step can be enough for tractability.

Variational Bayes example

• Mentions work around 1993 (with Van Camp) as early variational Bayes:
  • approximate the true posterior with a tractable distribution
  • enabling Bayesian learning in neural networks

Practical relevance today

• He references ideas connected to modern techniques like reparameterization tricks (indirectly under “variational” work).

---

ReLU and related activation insights

Clarification

• Hinton says the broader idea of ReLU was considered by others too.

His contribution to adoption

• He helped drive uptake by showing ReLU is almost equivalent to a stack of logistic units (in a specific representational sense).
• That made the learning mathematics “carry over,” supporting practical use of ReLUs even when motivations differed.

---

Neural network depth, initialization, and residual connections (as described)

• Hinton discusses a later (2014) talk at Google:
  • use ReLUs
  • initialize to the identity matrix
• With that, “copying behavior” can support training very deep nets (he claims demonstrations with ~300 hidden layers).
• He regrets not pursuing it further.
• He links the idea to later residual networks, suggesting conceptual similarity.

---

Backprop and the brain: current viewpoint + alternative learning rules

Main position

• Hinton argues: if backprop works well, evolution likely could implement something backprop-like.
• Therefore, the claim “backprop is implausible biologically” is dismissed as “silly,” even if details would differ.

Alternative biologically-inspired algorithms mentioned

• Recirculation algorithm (1987, with Jay McClelland)

  • Activity is sent around a loop so it becomes stable.
  • Train an autoencoder without classical backprop by minimizing change (reducing variation around the loop).
  • Learning rule adjusts synaptic weights proportional to:
    • presynaptic input
    • the rate of change of postsynaptic input
  • Interpretable as making “old” predictions better and “new” predictions worse (a sign convention).
  • Related to later neuroscience: STDP is described as the same concept, but with the update direction reversed.

• RBM-stack reconstruction → derivative insight (around 2007)

  • Claim: training stacked autoencoders/RBMs creates conditions where reconstruction signals can approximate derivatives of discriminative performance.
  • Hinton says the 2007 talk was largely ignored.
  • He notes Joshua Bengio later took up the idea, leading to further work.

• Capsule routing / inference-by-iteration (connected to the capsule section)

  • Capsule routing uses iterative agreement and can still support discriminative backprop through that process (as described later).

---

Multi-time-scale learning and “fast weights”

Historical starting point

• In Hinton’s first grad-student talk (1973), he introduced fast weights:
  • weights that adapt rapidly but also decay rapidly
  • acting as short-term memory

“True recursion” idea

• Reuse neuron populations and weights in recursive computation.
• Question: where is the memory when neuron resources are reused?
• Answer: store temporary memory in fast weights, then recover activities from them.

Later revival

• Mentions a paper with Jimmy Ba (around 2015/2016).

---

Capsules: current status and motivation

Hinton’s attitude

• He says he strongly believes in capsules, but few others do.
• He keeps publishing and expects rejection.

Representation idea

• Represent multi-dimensional entities using groups (“capsules”) whose internal activity encodes different properties.
• Key assumption: in each image region, assume at most one instance of a feature.

How a capsule works (conceptually)

• Each capsule represents one instance of a feature and stores its parameters:
  • x/y coordinates, orientation, speed, color, brightness, etc.

Routing-by-agreement (method)

• Goal: combine parts into wholes (e.g., mouth + nose → face).
• Mechanism:
  • each lower-level capsule predicts parameters for a higher-level capsule’s entity
  • higher-level capsule inputs are updated based on how much predictions agree
• Decision criterion:
  • consistent votes for the same higher-level parameters in a high-dimensional space suggest the capsules belong together.

Expected benefits

• Better generalization with limited data
• Improved handling of viewpoint changes
• Improved segmentation via agreement-based filtering

Training approach (as described)

• Not purely feed-forward:
  • uses iterative routing/iteration during the forward pass to decide whether parts belong together
  • backprop can be applied through the iteration process
• He states this is being developed in his Toronto group (with a Google Brain “Toronto” team).

---

How Hinton’s AI views evolved over decades

Phase 1 (mid-80s)

• Backprop for discriminative learning worked well.

Phase 2 (early 90s)

• He decided most human learning is likely unsupervised, increasing focus on that direction.

Later real-world constraint

• Even though he believes unsupervised learning is crucial long-term, the last decade succeeded with supervised/discriminative learning because it worked effectively.

Exciting examples he cites

• variational methods
• GANs (generative adversarial networks) as a major breakthrough

“Slow features” correction

• He agrees with the general principle but argues:
  • it’s a mistake to seek “slow” features per se
  • better to find features that change predictably
• General modeling principle:
  • apply nonlinear transforms to observables so underlying variables admit linear transformations for prediction/action
  • e.g., viewpoint changes become linear in an appropriate coordinate space

Connection to face synthesis

• He describes compressing a face into a low-dimensional representation enabling controlled generation changes.

---

Advice to break into deep learning (explicit methodology/list)

• Read the literature, but not too much

  • Avoid spending years passively reading.
  • Read enough to develop intuition.
  • Look for what “everyone is doing wrong.”
  • Challenge assumptions—try something that “doesn’t feel right,” then figure out how to do it right.

• Trust your intuitions

  • Use intuition as the steering mechanism.
  • Good intuition → you’ll likely succeed.
  • Bad intuition → little else matters.

• If others call your idea nonsense and you still believe it, that’s a positive sign

  • Example: early variational methods were rejected because people thought it was nonsense (“drunk” or “stupid”); Hinton treats that mismatch as a potential indicator of value.

• Never stop programming

  • Programming exposes whether details are correct.
  • A bad student returns with “it didn’t work” due to missing crucial decisions.
  • A good student reasons about why it must work (or won’t) and converges faster.

• Replicate published papers

  • Replication is time-consuming, but valuable.

• For new grad students: choose an advisor with beliefs similar to yours

  • Working on what your advisor cares about improves guidance and attention.

• Choosing PhD vs. industry/top group

  • He acknowledges tradeoffs.
  • Claims university faculty capacity is currently insufficient for deep-learning education needs.
  • Predicts academia will catch up, especially as “showing computers what to do” becomes as important as traditional programming.

---

Sources / speakers featured (identified)

1. Geoffrey Hinton (primary speaker)
2. Andrew Ng (interviewer/host)
3. David Rumelhart (collaborator; backprop development)
4. Ron Williams (co-developer of early backprop algorithm)
5. Paul Werbos (earlier related work)
6. David Parker (mentioned in context of early backprop-like inventions)
7. Stuart Sutherland (referee; discussed paper acceptance)
8. Terry Sejnowski (collaborator on Boltzmann machines)
9. Joshua Bengio (referenced for later embeddings/variational work and follow-up interest)
10. U. Tay (mentioned in subtitles as “UY Tay” in context of probabilistic models)
11. Van Camp (coauthor referenced in variational Bayes work)
12. Jay McClelland (coauthor on recirculation algorithm)
13. Jimmy Ba (collaborator on fast weights recursion paper)
14. Peter Brown (mentioned in variational methods early communication story)
15. “the brothers” (Peter Brown’s twin colleagues; no specific names provided)
16. D. Langer Higgins (mentor at Edinburgh)
17. Don Norman (mentioned as supportive figure in California neural-nets environment)
18. The audience / learners (recipients of advice)