This article is one of two Distill publications about graph neural networks. Take a look at Understanding Convolutions on Graphs to understand how convolutions over images generalize naturally to convolutions over graphs. Graphs are all around us; real world objects are often defined in terms of their connections to other things. A set of objects, […]

Over the past five years, Distill has supported authors in publishing artifacts that push beyond the traditional expectations of scientific papers. From Gabriel Goh’s interactive exposition of momentum, to an ongoing collaboration exploring self-organizing systems, to a community discussion of a highly debated paper, Distill has been a venue for authors to experiment in scientific

Contents This article is part of the Differentiable Self-organizing Systems Thread, an experimental format collecting invited short articles delving into differentiable self-organizing systems, interspersed with critical commentary from several experts in adjacent fields. Self-Organising Textures This article makes strong use of colors in figures and demos. Click here to adjust the color palette. In a

This article is part of the Circuits thread, an experimental format collecting invited short articles and critical commentary delving into the inner workings of neural networks. Branch Specialization Introduction Open up any ImageNet conv net and look at the weights in the last layer. You’ll find a uniform spatial pattern to them, dramatically unlike anything

This article is part of the Circuits thread, an experimental format collecting invited short articles and critical commentary delving into the inner workings of neural networks. Visualizing Weights Weight Banding Introduction If we think of interpretability as a kind of “anatomy of neural networks,” most of the circuits thread has involved studying tiny little veins

Acknowledgments We are deeply grateful to Sandhini Agarwal, Daniela Amodei, Dario Amodei, Tom Brown, Jeff Clune, Steve Dowling, Gretchen Krueger, Brice Menard, Reiichiro Nakano, Aditya Ramesh, Pranav Shyam, Ilya Sutskever and Martin Wattenberg. Author Contributions Gabriel Goh: Research lead. Gabriel Goh first discovered multimodal neurons, sketched out the project direction and paper outline, and did

Contents This article is part of the Differentiable Self-organizing Systems Thread, an experimental format collecting invited short articles delving into differentiable self-organizing systems, interspersed with critical commentary from several experts in adjacent fields. Self-classifying MNIST Digits Adversarial Reprogramming of Neural Cellular Automata Neural Cellular Automata (NCA We use NCA to refer to both Neural Cellular

This article is part of the Circuits thread, an experimental format collecting invited short articles and critical commentary delving into the inner workings of neural networks. Curve Circuits Branch Specialization Introduction The problem of understanding a neural network is a little bit like reverse engineering a large compiled binary of a computer program. In this

Author Contributions As we mentioned in Curve Detectors, our first investigation into curve neurons, it’s hard to separate author contributions between different papers in the Circuits project. Much of the original research on curve neurons came before we decided to separate the publications into the behavior of curve neurons and how they are built. In

This article is part of the Circuits thread, an experimental format collecting invited short articles and critical commentary delving into the inner workings of neural networks. Naturally Occurring Equivariance in Neural Networks Curve Circuits Introduction Some of the neurons in vision models are features that we aren’t particularly surprised to find. Curve detectors, for example,