LECTURE 11

Summary

The lecture presents the main challenges—and corresponding opportunities—of applying artificial intelligence to Earth observation. Earth-observation problems vary widely: the application, sensor, data quality, labels, and geographic or seasonal context all affect how a model should be designed.

1. Many applications require different problem formulations

Earth observation supports diverse tasks, including environmental monitoring, forestry, infrastructure monitoring, and disaster assessment. Even within one area, different tasks can require different data and models.

Examples from forestry include:

• Species classification: The modeling approach depends on assumptions about how stable species are over time. When multiple satellite images are available, clouds or other issues may make it useful to combine them rather than rely on a single image. The model must then determine how to fuse information from multiple dates into one result.
• Deforestation detection: This can be framed as an image-based task, a time-series task, or—commonly—as change detection using images from two dates. The framing affects the architecture. For example, a time-series approach might use an LSTM, while two-date change detection might use a Siamese CNN or deep change vector analysis.
• Other forestry tasks: Biomass estimation and insect-infection detection pose distinct modeling challenges.

The task definition matters as much as the choice of architecture:

• Image classification assigns one label to an entire image, such as “building” or “water.”
• Semantic segmentation assigns a label to each pixel. It may be multiclass or binary, with a foreground class of interest and a background.
• A task might also be modeled as change detection, particularly when prior maps or images are available.

Infrastructure monitoring illustrates the same diversity. Post-disaster assessment, monitoring urban sprawl or informal settlements, and tracking construction progress all have different requirements. For example, monitoring construction progress may require very high-resolution images and repeated observations. Solar-panel monitoring is another relevant application.

2. Sensors differ, so models may not transfer directly

Earth-observation data comes from many sensor types, including multispectral and hyperspectral instruments, synthetic aperture radar (SAR), and lidar. Even sensors of the same broad type can differ in:

• Spatial resolution
• Spectral resolution or band characteristics
• Temporal resolution

As a result, a model trained on one sensor cannot necessarily be applied directly to another without adaptation. A model trained on ordinary RGB imagery may be a useful starting point when only a small dataset is available, but it may be unsuitable when the band combination is different or the data has many bands, as in hyperspectral imagery.

3. Labels can be complex, ambiguous, and costly to obtain

Earth-observation scenes commonly contain many objects at once—such as buildings, roads, and cars—rather than a single clear category. This is one reason pixel-level semantic segmentation is often useful.

Creating accurate labels can also require substantial domain expertise. In applications such as interpreting mining areas, it may be difficult for untrained annotators to label every pixel reliably. Consequently, large-scale datasets can be hard to produce for specialized tasks.

Weak labels may offer an alternative, but they have limitations. For example, OpenStreetMap data can provide building-function labels, yet coverage may be incomplete and map features may not align accurately with the imagery. Misalignment is especially concerning with lower-resolution images.

4. Large quantities of mostly unlabeled data motivate other learning approaches

Many satellites collect large amounts of data, including freely available imagery at regular intervals. However, most of it is unlabeled, and labeling it can be difficult and expensive. This motivates approaches beyond standard supervised learning, including:

• Semi-supervised learning
• Unsupervised learning
• Self-supervised learning

5. Combining sensors and other data sources is useful but challenging

Multiple sensors can complement one another, but their data may have different characteristics and may not be processed effectively with the same network or architecture.

A disaster example illustrates the issue: pre-event information may be available from both optical and SAR sensors, while post-event optical imagery may be obstructed by clouds. A system might therefore need to compare pre-event optical and SAR data with post-event SAR data. This raises questions about how to combine inputs that differ in modality and characteristics.

Other data sources can also supplement imagery. Geotagged tweets, for instance, may help with building-type classification, but their locations may be difficult to associate accurately with the relevant places on the ground.

Architecture design may need to account for the properties of the data. In hyperspectral classification, a model may use separate branches to capture spatial and spectral features. For time-series analysis, sensors with different revisit schedules might be combined to obtain more frequent observations, while still accounting for differences between sensors and temporal behavior.

6. Models must handle domain shifts and recognize when they may fail

A model may perform poorly when applied to data that differs from its training data. Such differences can result from:

• A different sensor
• A different season
• A different geographic region

For example, a building-detection model trained on imagery from Munich may encounter substantially different building styles when applied in Hong Kong. The lecture relates this to the distinction between in-domain training data and out-of-distribution test data.

Depending on the scenario, relevant approaches include:

• Unsupervised domain adaptation, when some unlabeled data from the target domain is available
• Domain generalization, when the model must work across domains it was not trained on
• Out-of-distribution detection, to identify inputs that differ from the training distribution and may lead to model failure

Overall lesson

There is no single AI solution for Earth observation. Before selecting a model, practitioners need to define the task, understand the sensor and data characteristics, decide how to use observations across time, assess the quality and availability of labels, and anticipate shifts between training and deployment conditions.

Speakers and sources featured

• Speaker: One lecturer, whose name is not stated in the subtitles.
• Named data sources, platforms, and examples: ImageNet, Sentinel-2, Landsat 8, OpenStreetMap, Amazon Mechanical Turk, geotagged tweets, and Munich and Hong Kong as examples of different geographic domains.
• Sensor types and methods: Multispectral and hyperspectral imagery, SAR, lidar, LSTM, Siamese CNNs, deep change vector analysis, and separate spatial/spectral model branches.