POS Tagging | Part of Speech Tagging in NLP | Hidden Markov Models in NLP | Viterbi Algorithm in NLP

POS tagging tutorial — HMMs and Viterbi (video lecture)

Overview

This video explains part-of-speech (POS) tagging and demonstrates both practical tooling (spaCy) and the statistical core behind taggers (Hidden Markov Models and the Viterbi algorithm). It includes a hands-on demo, a worked toy example, and practical advice for building and inspecting taggers.

POS tagging: assigning a POS tag to every word in a sentence (coarse- and fine-grained tags).

---

What the video covers (high-level)

• Definition and importance
  • POS tagging assigns POS tags to each word in a sentence (coarse- and fine-grained).
  • Common preprocessing step for NER, information retrieval, question-answering, word-sense disambiguation, chatbots, and other NLP pipelines.
• Tools / demo
  • Hands-on spaCy demo: installing spaCy, loading en_core_web_sm, creating a Doc, accessing token.text, token.pos_, token.tag_, using spacy.explain(), looping through tokens, and visualizing tags with displaCy.
  • Shows coarse vs fine-grained tags and context-dependent tag examples (e.g., words like “left” and “read” that change tag by context).
• Core algorithms
  • Statistical POS tagging via Hidden Markov Models (HMMs): train on labeled data, compute emission (observation) probabilities and transition probabilities (use start/end tokens), and use these probabilities to score tag sequences.
• Brute-force vs optimized decoding
  • Brute-force enumeration of all tag sequences grows as |T|^n and is intractable for realistic sentences.
  • The Viterbi algorithm uses dynamic programming: at each timestep keep the most probable path to each state, then backtrack to recover the best tag sequence.
• Worked example
  • Manual toy dataset: compute counts → normalize to emission/transition probabilities → compute sequence probabilities for candidate tag assignments.
  • Shows how zero or small probabilities prune/penalize paths and how Viterbi reduces computation while selecting the most probable tag path.
• Practical tips
  • In practice, spaCy performs tagging and visualization.
  • The HMM/Viterbi walkthrough helps understand what libraries do internally.
  • When building taggers, experiment and inspect probabilities; smoothing is important to handle zeros.

---

Key technical concepts explained

• Emission probabilities
  • P(word | tag) computed from counts and normalized across words for each tag.
• Transition probabilities
  • P(tag_t | tag_{t-1}) computed from tag sequence counts; include start/end markers to model sentence boundaries.
• Sequence scoring
  • The probability of a tag sequence = product of emission and transition probabilities for the sequence; choose the sequence with maximum probability.
• Viterbi algorithm
  • Dynamic programming approach to avoid exponential brute-force enumeration.
  • Maintain the best predecessor and probability per tag per position; backtrack from the end to recover the tags.
• Visualization
  • displaCy options (distance, colors, options dictionary) to render POS annotations clearly.

---

Code / commands demonstrated (spaCy)

Install and load:

pip install spacy
python -m spacy download en_core_web_sm

Python demo:

import spacy
nlp = spacy.load("en_core_web_sm")
doc = nlp("I will Google about Facebook")

# token access
for token in doc:
    print(token.text, token.pos_, token.tag_, spacy.explain(token.tag_))

displaCy rendering:

from spacy import displacy
displacy.render(doc, style="dep")      # dependency visualization
displacy.render(doc, style="ent")      # entity visualization
# displaCy supports options such as colors and spacing via an options dict

Notes:

• Access token attributes with doc[i].text, doc[i].pos_, doc[i].tag_, and spacy.explain(doc[i].tag_).
• Loop through tokens to print coarse/fine tags and explanations.

---

Worked example (toy dataset)

• Start with a few labeled sentences and count tag-to-tag transitions and tag-to-word emissions.
• Convert counts into emission and transition probabilities (normalization).
• Compute probabilities for candidate tag sequences. Low or zero probabilities eliminate or penalize candidate sequences.
• Apply Viterbi to efficiently find the highest-probability tag sequence by keeping the best path to each state at each timestep and backtracking at the end.

---

Practical tips

• Use spaCy for production and visualization; it handles tagging efficiently.
• Study HMM/Viterbi to understand internal behaviors of taggers and to debug or customize models.
• Inspect raw counts and probabilities; apply smoothing where appropriate to avoid zero probabilities.
• Run experiments to observe how context affects tagging (e.g., words like “left”, “read”).

---

Recommended resources

• The presenter suggests watching a dedicated video on Hidden Markov Models before the detailed HMM/Viterbi explanation. The referenced channel name in captions may be garbled, but look for a clear HMM tutorial (examples: university lecture videos or concise HMM overviews from reliable tutorial channels).

---

Notes on caption inaccuracies

• Several auto-generated subtitles contained garbled words/names (e.g., “Twitter”, “Iodine”, “Marg”) that are likely mis-transcriptions of technical terms such as “trigram”, “smoothing”, etc.
• Despite subtitle errors, the technical core (spaCy demo, emission/transition probabilities, Viterbi optimization) is consistent and should be the focus.

---

Main speaker / sources

• Presenter: “Mike” (name appears in captions; likely the channel instructor).
• Libraries/tools referenced: spaCy (en_core_web_sm), displaCy visualizer.
• External tutorial recommended: an HMM/HMM-to-Viterbi explanatory video (caption referenced something like “Neso Academy” / “No Academy” — seek a clear HMM tutorial prior to deeper explanations).