The Story of Information Theory: from Morse to Shannon to ENTROPY

Scientific concepts, discoveries, and nature/technology phenomena

Wartime / optical communication innovation (1794)

• Optical semaphore: an early long-range communication system that used a shape-shifting structure to represent information with distinct symbols.
  • Symbols / positions: 92 positions
  • Encoded message combinations: derived via combinatorial encoding

Combinatorial encoding

• Primary symbols: 92 possible configurations
• Secondary symbols (two consecutive primary symbols):
  • (92 \times 92 = 8{,}464) combinations

Code book concept

• The system used an indexing idea (a “code book”) so that specific commands or words could be mapped to symbol combinations (e.g., “page/entry → symbol sequence”).

---

Electromagnetism enabling hidden long-distance signaling (1820)

• Oersted’s discovery: an electric current can deflect a magnetic needle, directly linking electricity and magnetism.
• Repeatability and basic circuit setup: battery + conductor + compass.
• This phenomenon enables electrical pulses through conductors, supporting long-range telegraphy.

---

Telegraph systems and encoding

Morse telegraph (1838 / 1844 public demonstration)

• Dots and dashes: letters and numbers were encoded as sequences of electrical pulses over wires.
• Required practical engineering, including a pulse-impressing mechanism.

Wheatstone–Cook needle telegraph (UK)

• Used multiple needles driven by parallel wires to provide faster visual (often letter-by-letter) representation.

Morse limitation

• A simpler one-wire design, but effectively serial transmission—“one pulse at a time.”

---

Source coding via symbol frequency (information-theoretic precursor)

• Alfred Vail’s intuition:
  • Assign shorter codes to more frequent letters (e.g., E)
  • Assign longer codes to less frequent letters (e.g., Z)
• Result: reduced the average number of pulses needed to transmit messages.

---

Digital-style fixed-length encoding (1876)

• Baudot / Baudureau code (described as the “Bordeaux code”)
  • Fixed-length 5-bit codes
  • Maps each letter/number to one of 32 possibilities
  • Designed to support automation via teleprinter keyboard I/O
• Efficiency note: treated as less efficient than variable-length coding because it ignores probability differences (introducing redundancy).

---

Quantifying information (1927–1938 foundations)

Ralph Hartley (1927)

• Proposed a quantitative measure based on the number of possible symbol sequences.
• Uses a logarithm to prevent exponential blow-up:
  • information grows roughly linearly with message length
• Conceptual form:
  • Information = log of number of choices, scaled by alphabet size and consecutive selections.

Nyquist contribution (signal & bandwidth)

• Connects discrete signaling to analog signal components.
• Identifies a bandwidth-related limit on how many distinct signaling events can occur per second.

Shannon era (1930s onward; theory formalized 1948)

Frames a communication system as:

• information source
• transmitter
• channel / medium
• receiver
• destination

---

Entropy (Shannon’s core definition of information)

• Entropy measures uncertainty resolved, not just the raw “number of possibilities.”
• Biased probabilities reduce effective information per symbol compared to the uniform case.
• Defined as:
  • the average “surprise” weighted by probability
  • maximized when outcomes are equiprobable

Compression bound

• A source can be compressed down to its entropy, but not reliably below it.

---

Typical sequences and proof sketch of optimal compression

• For long sequences, outcomes concentrate into a “typical set.”
• The number of typical sequences grows exponentially, with growth rate determined by the source’s entropy.
• Extends beyond independent sources: even with dependence (e.g., Markov-chain-like structure), typical sets still emerge characterized by entropy.

---

Noisy channels and conditional entropy

Shannon’s correction-channel idea

• Noise introduces uncertainty at the receiver.
• Conditional entropy represents the missing information—what the receiver cannot infer about what was sent.

Example logic

• Even if the transmitter is correct most of the time, the receiver may not know when errors occur, so uncertainty persists.

---

Channel coding theorem (reliable communication near capacity)

• With appropriate coding, it’s possible to communicate at (or near) channel capacity with arbitrarily small error probability for sufficiently large block lengths.
• Mechanism (high level):
  • Choose a rate (R) below capacity
  • Randomly map messages to channel input sequences
  • For long blocks, ensure that any received sequence is very unlikely to correspond to multiple messages.

---

Hartley–Nyquist–Shannon unification

• Extends to continuous signals to yield a single capacity expression for channels constrained by:
  • power
  • noise
• Final theme: the ultimate speed limit for reliable communication—capacity of a noisy channel.

---

Methodologies / step-like ideas mentioned

Semaphore encoding approach

1. Choose symbol positions (primary symbols).
2. Combine two consecutive positions to form a larger symbol set (secondary symbols).
3. Use a code book mapping commands/words to secondary symbol sequences.

---

Variable-length coding based on symbol frequency (Vail-like strategy)

• Compute letter frequencies in a language.
• Assign short codes to high-frequency symbols.
• Assign long codes to low-frequency symbols.
• Goal: minimize average bits/pulses per character.

---

Entropy-based compression (Shannon’s limit)

• Model the source with probabilities (including possible dependence such as Markov structure).
• Compress using code lengths aligned with symbol “surprise.”
• For long sequences, rely on typical sets so that:
  • the average bits per symbol approaches entropy.

---

Noise handling via coding

• Model receiver uncertainty using conditional entropy.
• Use coding with rate (R <) capacity.
• Decode reliably with low probability of confusion as block length increases.

---

Researchers / sources featured

• Claude Shannon
• Ralph Hartley
• Harry Nyquist (referred to as “Nyquist”)
• Hans Christian Ørsted (subtitles contain a spelling error: “Urststead”)
• Claude Chappe (optical semaphore inventor; subtitles contain variant spellings: “Chapper/Chaper/Chaper”)
• Samuel Morse
• Charles Thomas Jackson
• Alfred Vail
• Émile Baudot (subtitles contain a spelling error: “Emile Bordeaux”)
• Cook (as in Cook and Wheatstone telegraph; first name not given)
• Wheatstone (as in Cook and Wheatstone; first name not given)
• Alexander the Great (historical reference in message-speed context)
• Benjamin Franklin (historical reference in message-speed context)