LECTURE 14

Summary

The lecture covers ways to reuse models across datasets or domains, then introduces deep clustering and self-supervised learning as methods for learning useful representations without human-provided labels.

1. Reusing a Pretrained Model

A model trained with supervision on one dataset or domain can be applied to a different target domain in three broad ways:

• Use it as a feature extractor: Keep the pretrained model fixed and use its internal representations for a target task.
• Fine-tune it: Update some or all of the model using labeled examples from the target domain.
• Use unsupervised domain adaptation: Use a model trained on a labeled source domain together with unlabeled target-domain data to produce a model better suited to the target.

Choosing Features from the Network

There is no universally best layer to extract features from. The choice depends on both the source–target difference and the intended task:

• Later layers may be useful when source and target images are relatively similar—for example, transferring from ImageNet to high-resolution RGB Earth-observation imagery.
• Earlier layers may be more suitable when the target is substantially different, such as coarse-resolution imagery or synthetic aperture radar (SAR), where lower-level features such as edges may be more transferable.
• The required output also matters:
  • For an image-level task, a fully connected layer can provide one feature vector per image.
  • For a dense, pixel-level task, features should come from convolutional layers, since fully connected layers lose spatial layout.
• Convolutional feature maps often have lower spatial resolution than the input image. Upsampling or interpolation can restore the dimensions; features can also be repeated across pixels when appropriate.

If validation data is available, experiments can help determine which layers work best. Another option is a hypercolumn: combine features from several layers rather than choosing only one. This can reduce the layer-selection problem, but produces high-dimensional representations. Selecting or ranking a subset of features from each layer can help control dimensionality.

2. Fine-Tuning on Target Data

Fine-tuning is a straightforward way to adapt a pretrained model when some labeled target examples are available. For example, a model trained on one season or sensor can be adapted to another using a smaller target-domain dataset.

A common practice is to:

1. Keep the early layers frozen, on the assumption that they capture broadly useful features.
2. Train the later layers on the target data.
3. Consider fine-tuning the whole network if the task warrants it.

The appropriate choice depends on the target task and how different the target domain is from the source.

3. Unsupervised Domain Adaptation

In unsupervised domain adaptation, the target data has no labels. The setup uses:

• A model trained on a labeled source domain.
• Unlabeled examples from a target domain.

For example, a model trained on summer Sentinel-2 imagery could be adapted using unlabeled winter Sentinel-2 imagery. The aim is to obtain a model that works better on the target domain without requiring target labels. The lecture introduces this problem setting but does not go into specific adaptation architectures.

4. Deep Clustering

Deep clustering learns representations by alternating between clustering data and training a neural network to predict those clusters. The clusters act as automatically generated pseudo-labels; they are not necessarily the true semantic classes.

Basic Procedure

1. Pass input images through a CNN to extract features.
2. Optionally reduce the feature dimensionality, for example with PCA.
3. Cluster the features, such as with k-means.
4. Assign each cluster a pseudo-label.
5. Train the CNN to predict those pseudo-labels.
6. Repeat the clustering and network-training steps iteratively, progressively refining the features and clusters.

The goal is primarily to learn useful embeddings—representations that capture meaningful structure—not to treat clustering itself as the final objective.

The lecture notes two important observations:

• Using more clusters than semantic classes can improve representations, because a broad class such as “building” may contain meaningful subtypes.
• A key challenge is collapse or a trivial solution, where many or all data points end up assigned to the same cluster.

5. Self-Supervised Learning

Self-supervised learning resembles supervised learning in how the network is trained, but its labels are derived from the data itself rather than supplied by human annotators. It often relies on a pretext task: an artificial task used to make the network learn visual representations that may help with later, downstream tasks.

An example is predicting image rotation:

1. Rotate an image by a known angle, such as 90°, 180°, or 270°.
2. Train the network to identify the rotation.
3. Use the learned representation for a later task.

The underlying assumption is that performance on the pretext task is related to performance on downstream tasks.

Choosing Pretext-Task Difficulty

The relationship between task difficulty and representation quality is not simply linear:

• If the pretext task is too easy, the network receives little incentive to learn useful features.
• Increasing difficulty can encourage the network to learn richer representations.
• If the task becomes so difficult that it is effectively impossible, it provides little useful learning signal and effectiveness may fall.

Thus, the task should be challenging enough to promote learning but still learnable.

Speakers and Sources

• Speaker: One unnamed lecturer, presenting the lecture.
• Source: NPTEL IIT Delhi (the channel identified in the video metadata).