Lecture 16 | Adversarial Examples and Adversarial Training

Summary

1. What adversarial examples are

An adversarial example is an input deliberately modified to make a machine-learning model misclassify it. The modification can be nearly imperceptible to a person while causing a model to make a highly confident wrong prediction. For example, a tiny, carefully calculated change to a panda image made a convolutional network classify it as a gibbon with 99.9% confidence.

The lecturer emphasizes that this is not unique to deep networks: linear classifiers, logistic regression, SVMs, decision trees, and other models can also be vulnerable.

2. Why they occur

The lecture’s central explanation is that many models behave approximately linearly with respect to their inputs, even when the overall model is complex. ReLU networks are piecewise linear, and other common architectures also have substantial linear behavior.

In high-dimensional inputs, small changes across many pixels can combine into a large effect on a model’s prediction. An adversarial perturbation is chosen to align with a direction that strongly increases the model’s error; random noise usually does not have that alignment and often has little effect.

The lecturer contrasts two possible explanations:

• Overfitting: The model makes arbitrary mistakes away from the training data.
• Linearity and extrapolation: The model’s decision boundaries extend in systematic directions beyond the data it has seen.

Evidence presented favors the second explanation: different models often make similar mistakes on the same examples, and a perturbation found for one input can often be applied to other inputs.

The lecturer also describes these errors as occurring outside the narrow region of input space occupied by ordinary data. A model may perform well on naturally occurring test examples while behaving unreliably elsewhere.

3. Adversarial regions, transfer, and noise

Adversarial examples are not isolated, fragile points. They often lie in broad, connected regions separated from correctly classified examples by roughly linear decision boundaries. Once an input crosses into such a region, nearby inputs may also be misclassified.

The lecture reports that adversarial regions on MNIST had, on average, about 25 adversarial dimensions, though the lecturer says the evidence came from only a limited number of datasets. A larger adversarial subspace makes it more likely that different models’ vulnerable regions will overlap, helping explain why attacks can transfer between models.

Random noise is generally different from an adversarial perturbation: noise usually does not systematically change the prediction, while an adversarial perturbation is specifically selected to do so. The speaker also shows that adding noise to an already adversarial example often does not restore the correct classification.

4. Fast Gradient Sign Method (FGSM)

FGSM is presented as a fast way to construct adversarial examples. It uses the gradient of the training loss with respect to the input and changes each pixel in the direction that increases the loss.

Basic procedure:

1. Start with an input image and its label.
2. Compute the gradient of the model’s loss with respect to the input pixels.
3. Take the sign of each gradient component.
4. Add or subtract a small amount, (\epsilon), at each pixel according to that sign.
5. Keep each pixel’s change within the chosen maximum per-pixel bound.

This per-pixel constraint—described in the lecture as a max-norm constraint—helps prevent the attack from simply changing the image into a visibly different class. FGSM is useful because it is fast, including when adversarial examples are generated during training. However, the lecturer warns that defeating FGSM alone does not demonstrate a robust defense; stronger, iterative attacks can be more effective.

5. Security implications and black-box attacks

Adversarial examples create practical security risks because an attacker may not need direct access to a target model’s parameters.

Transfer-based black-box attack procedure:

1. Train a substitute model for the same task, using either:
   • a separately collected and labeled dataset, or
   • examples sent to the target model and the labels or outputs returned by it.
2. Generate adversarial examples against the substitute model.
3. Apply those examples to the target model, where they may still cause misclassification.

The lecture reports transfer between different model types and datasets. It also describes improving transfer by optimizing an example to fool an ensemble of models; examples that fool the ensemble are more likely to fool another model.

Applications discussed include attacks on hosted image-classification services, malware detectors, and reinforcement-learning agents. For a reinforcement-learning policy, the attack alters a frame so that the policy chooses a different action. The lecturer notes that this can make an agent perform badly, though directing it to carry out a specific, complex alternative task remains a harder problem.

6. Physical-world attacks and human perception

The lecturer describes experiments in which adversarial images were printed and then photographed by a phone camera. The camera-based recognition system could still be fooled, despite the effects of printing, camera capture, and using a different model from the one used to create the attack. This suggests that attacks may work even when an attacker cannot edit a system’s digital input directly.

Humans are also vulnerable to some perceptual illusions, but the machine-learning adversarial examples discussed generally do not fool people in the same way they fool models. The lecturer suggests that understanding this difference could help improve machine learning.

7. Defenses and their limitations

The lecture characterizes robust defense as an open research problem. Many proposed defenses have failed against stronger attacks, and traditional regularization or a generative model alone is not necessarily sufficient.

A model must estimate not just what inputs are likely, but the appropriate posterior over labels given an input. Different generative models can assign similar likelihoods to inputs yet behave very differently in the class probabilities they produce away from the training data. Approximation errors can also cause a model to resume overly linear extrapolation.

8. Adversarial training

Adversarial training adds perturbed examples to the training process, teaching the model to preserve the correct label despite those changes.

Basic procedure:

1. Take a labeled training example.
2. Generate an adversarial perturbation intended to increase the model’s loss.
3. Train the model to assign the original label to the perturbed input as well as to the clean one.
4. Repeat as the model changes, generating fresh adversarial examples during training.

The lecture reports that this approach can substantially reduce error on the same type of attack used during training. In one example, training against FGSM brought adversarial test error to below 1%. It also improved clean-data performance in that experiment, apparently by acting as a regularizer against overfitting. But the gains do not necessarily transfer to stronger iterative attacks or to attacks generated by different methods. Training is data- and computation-intensive because new adversarial examples must be generated as the model updates.

9. Virtual adversarial training

Virtual adversarial training, introduced by Takeru Miyato, extends the idea to unlabeled data.

Basic procedure:

1. Ask the model for its output distribution on an input, even if no ground-truth label is available.
2. Find a small perturbation that changes the model’s output distribution.
3. Train the model to keep its output distribution similar for the original and perturbed inputs.

This supports semi-supervised learning, since unlabeled examples can contribute to training. The lecture says it improved results on several text-classification tasks.

10. Broader significance: model-based optimization

Adversarial vulnerability also matters when a model is used to guide the design of something new. If an optimizer searches for inputs that maximize a model’s predicted outcome—for example, a fast car design or a useful drug molecule—it may find an input that exploits the model rather than a genuinely good design.

The lecturer argues that solving adversarial robustness could enable more reliable model-based optimization for engineering and scientific tasks, including designing molecules, genes, or circuits.

Speakers and sources featured

Speakers

• Ian Goodfellow — guest lecturer and primary speaker.
• Unidentified audience members — asked questions during the Q&A.
• Unidentified host or organizer — closed the session and thanked the speaker.

Researchers, works, and examples mentioned

• Battista Biggio — earlier work on neural-network attacks.
• Christian Szegedy — independent discovery of adversarial examples and input-optimization experiments.
• David Warde-Farley — work with Goodfellow on decision-boundary visualizations.
• Florian Tramer — collaboration on measuring adversarial subspaces.
• Bob Sturm — essay Clever Hans, Clever Algorithms; the lecture also recounts the Clever Hans horse story.
• Nicholas Carlini — stronger iterative attacks and analysis of proposed defenses.
• Takeru Miyato — virtual adversarial training.
• Nicolas Papernot — transferability research and attacks on hosted machine-learning services.
• Dawn Song’s group at UC Berkeley — ensemble-based attacks that improve transfer.
• Alexey Kurakin and Samy Bengio — collaboration with Goodfellow on physical-world, camera-based attacks.
• Andrej — ImageNet fooling-attack example involving a class-weight perturbation.
• Catherine Gross — work on evading malware detectors.
• Sara Sabour and collaborators — Adversarial Manipulation of Deep Representations.
• MalGAN — a GAN-based approach to generating adversarial examples for malware detectors.
• MetaMind, Amazon, Google, and Clarifai — machine-learning services cited in examples of transfer-based attacks.