LECTURE 27

Main Ideas

Combining Spectral and Spatial Features

• Hyperspectral image analysis depends on both spectral information—how a material responds across bands—and spatial information—the arrangement of pixels and objects.
• A hybrid attention architecture can apply spectral (channel) attention and spatial attention together, then use a 3D CNN to produce a classification map.
• The key design principle is to account for both types of information rather than relying on either alone.
• Because hyperspectral bands form an ordered sequence, recurrent neural networks are another possible approach to analyzing them.

Self-Supervised Learning for Hyperspectral Images

Detailed hyperspectral labels are difficult and costly to obtain, partly because classification may require distinguishing fine-grained categories. Self-supervised learning (SSL) can learn useful representations from unlabeled data, which can then be fine-tuned using a smaller labeled set.

Models pretrained on ImageNet generally cannot be transferred directly in the usual way: RGB images have only a few channels, whereas hyperspectral images may have many bands and a substantially different input structure.

A possible contrastive SSL pipeline is:

1. Generate two augmented views of each hyperspectral patch.
2. Train the model to learn representations that remain consistent across the views using a contrastive objective.
3. Fine-tune the pretrained representation with a limited labeled dataset.

Potential augmentations and pretext tasks include:

• Spatial cropping: Create different crops from an image. This requires care: at coarse spatial resolution, crops may depict different concepts, and many hyperspectral datasets contain only a small number of scenes.
• Spectral masking: Hide selected bands, for example by setting them to zero, so the model learns from incomplete spectral information.
• Spectral perturbation or noising: Add noise to some or all bands to create altered views.
• Band reordering: Shuffle spectral bands and train the model to detect whether they were reordered or identify the reordering applied.
• Masked spectral modeling: Hide bands and train the network to predict their missing information, encouraging it to learn dependencies among bands.

Augmentations should be different enough to make the learning task meaningful, but not so different that the views no longer represent related data.

Semi-Supervised Learning with Graph Neural Networks

In practical settings, a scene may have only scattered labeled pixels, with many unlabeled pixels. This makes semi-supervised learning useful.

A graph neural network (GNN) can represent pixels or superpixels as nodes and their relationships as edges. Edges can reflect spatial proximity, spectral similarity, or both, so the graph’s adjacency structure captures spatial and spectral relationships. The GNN propagates information between connected nodes, helping it use the unlabeled parts of the scene alongside the labeled samples.

Hyperspectral Datasets and Domain Differences

• Indian Pines: An agricultural scene captured by the AVIRIS sensor, with about 224 spectral bands. Papers may use fewer bands after preprocessing, such as removing bands affected by water absorption. Example classes include soybean varieties, illustrating how hyperspectral tasks can require fine-grained distinctions.
• Pavia University: An urban scene in Italy captured by the ROSIS sensor, with about 103 bands. Example classes include asphalt, meadows, trees, and buildings.

In addition to geographical and seasonal domain shifts, hyperspectral analysis faces substantial sensor-related shifts: sensors can differ in their number of bands and other characteristics.

Hyperspectral Anomaly Detection

Anomaly detection seeks spectrally unusual targets that differ from their surrounding background, without requiring a known target signature. If a known target signature is available, the task is more properly described as target detection.

Hyperspectral data supports anomaly detection because materials often have distinctive spectral signatures, allowing subtle deviations—and potentially small or concealed objects—to be identified. Applications include surveillance, environmental monitoring, defense, and mineral exploration.

Anomaly detection is generally more challenging than supervised classification or target detection because the target is unknown in advance.

Speakers and Sources Featured

• Speaker: One unnamed lecturer, presenting the lecture for NPTEL IIT Delhi.
• Sources and examples referenced: A paper cited on the lecture slide for the hybrid-attention architecture (not named in the subtitles), the Indian Pines dataset (AVIRIS), and the Pavia University dataset (ROSIS).