LECTURE 20

Summary

This lecture continues the discussion of CNNs for semantic segmentation, focusing on fully convolutional networks (FCNs), training and evaluation considerations, and the U-Net architecture.

Fully Convolutional Networks (FCNs)

• Classification-oriented CNNs typically end in fully connected layers, which discard spatial layout and produce image-level predictions. Semantic segmentation instead requires a class prediction for every pixel.
• An FCN replaces fully connected layers with convolutional operations, preserving spatial feature maps and allowing the network to accept images of varying sizes.
• Convolutional encoders still reduce spatial resolution as they extract features. To produce a segmentation map at the input image’s resolution, the network must upsample its low-resolution features.
• The lecture introduces transposed convolution as one way to perform this upsampling. A stride-2 example illustrates how a small input can produce a larger output. Students are encouraged to calculate the result for different kernels and compare it with a Python implementation.
• An FCN can be trained end to end: an encoder extracts features, an upsampling stage restores spatial dimensions, and a pixel-wise loss updates the network.
• FCNs are trainable, support dense prediction for arbitrary image sizes, and provide a foundation for later segmentation architectures.

Practical Considerations for Segmentation

• Class imbalance and sampling: Classes may occur at very different frequencies. For example, vehicle pixels may be much rarer than building pixels in an urban image. Flood datasets may have the reverse imbalance if flood scenes are deliberately oversampled. Data sampling, model design, and loss selection should account for these distributions.
• Ground-truth annotation: Labeling every pixel is time-consuming and costly, especially when annotators must mark fine boundaries between neighboring categories. Weak supervision is mentioned as a potential research direction.
• Evaluation metrics: The lecture names pixel accuracy, Intersection over Union (IoU), mean IoU, and Dice score. IoU measures overlap between prediction and ground truth, is generally computed per class, and penalizes both false positives and false negatives. Mean IoU averages the class-wise IoUs.

U-Net and Skip Connections

Upsampling can restore the dimensions of a feature map, but it cannot by itself recover spatial information lost during repeated downsampling. U-Net addresses this by combining:

• An encoder (contracting path) that extracts semantic features while reducing spatial resolution and increasing feature depth.
• A decoder (expanding path) that upsamples features to produce dense, pixel-wise predictions.
• Skip connections that carry fine-grained spatial features from encoder stages to corresponding decoder stages.

These connections help recover localization and object boundaries, produce sharper segmentation maps, and improve gradient flow during training. In the described arrangement, decoder stages connect to encoder stages at matching levels, proceeding from deeper encoder blocks toward shallower ones as the decoder expands.

Speakers and Sources Featured

• Speaker: An unidentified lecturer; no individual name is given in the subtitles. The lecture is from NPTEL IIT Delhi.
• Paper cited: “Dense Semantic Labeling of Subdecimeter Resolution Images with Convolutional Neural Networks” (2017), presented as an example of work following a vanilla-FCN approach.