A Discussion of ‘Adversarial Examples Are Not Bugs, They Are Features’: Two Examples of Useful, Non-Robust Features Ilyas et al. define a feature as a function fff that takes xxx from the data distribution (x,y)∼D(x,y) \sim \mathcal{D}(x,y)∼D into a real number, restricted to have mean zero and unit variance. A feature is said to be […]

The hypothesis in Ilyas et. al. is a special case of a more general principle that is well accepted in the distributional robustness literature — models lack robustness to distribution shift because they latch onto superficial correlations in the data. Naturally, the same principle also explains adversarial examples because they arise from a worst-case analysis of distribution

On May 6th, Andrew Ilyas and colleagues published a paper outlining two sets of experiments. Firstly, they showed that models trained on adversarial examples can transfer to real data, and secondly that models trained on a dataset derived from the representations of robust neural networks seem to inherit non-trivial robustness. They proposed an intriguing interpretation

We demonstrate that there exist adversarial examples which are just “bugs”: aberrations in the classifier that are not intrinsic properties of the data distribution. In particular, we give a new method for constructing adversarial examples which: Do not transfer between models, and Do not leak “non-robust features” which allow for learning, in the sense of

A figure in Ilyas, et. al. that struck me as particularly interesting was the following graph showing a correlation between adversarial transferability between architectures and their tendency to learn similar non-robust features. Adversarial transferability vs test accuracy of different architectures trained on ResNet-50′s non-robust features. One way to interpret this graph is that it shows

Ilyas et al. report a surprising result: a model trained on adversarial examples is effective on clean data. They suggest this transfer is driven by adverserial examples containing geuinely useful non-robust cues. But an alternate mechanism for the transfer could be a kind of “robust feature leakage” where the model picks up on faint robust

We want to thank all the commenters for the discussion and for spending time designing experiments analyzing, replicating, and expanding upon our results. These comments helped us further refine our understanding of adversarial examples (e.g., by visualizing useful non-robust features or illustrating how robust models are successful at downstream tasks), but also highlighted aspects of

By some metrics, research on Generative Adversarial Networks (GANs) has progressed substantially in the past 2 years. Practical improvements to image synthesis models are being made almost too quickly to keep up with: Odena et al., 2016 Miyato et al., 2017 Zhang et al., 2018 Brock et al., 2018 However, by other metrics, less has

Even if you have spent some time reading about machine learning, chances are that you have never heard of Gaussian processes. And if you have, rehearsing the basics is always a good way to refresh your memory. With this blog post we want to give an introduction to Gaussian processes and make the mathematical intuition

Memorization in Recurrent Neural Networks (RNNs) continues to pose a challenge in many applications. We’d like RNNs to be able to store information over many timesteps and retrieve it when it becomes relevant — but vanilla RNNs often struggle to do this. Several network architectures have been proposed to tackle aspects of this problem, such as Long-Short-Term