Testing methods of neural systems understanding

Neuroscientists apply a range of analysis tools to recorded neural activity in order to glean insights into how neural circuits drive behavior in organisms. Despite the fact that these tools shape the progress of the field as a whole, we have little empirical proof that they are effective at identifying the mechanisms of interest. At the same time, deep learning systems are trained to produce intelligent behavior using neural networks, and the resulting models are impressive but also largely impenetrable. Can the tools of neuroscience be applied to artificial neural networks (ANNs) and if so what would this process tell us about ANNs, brains, and – most importantly – the tools themselves? Here we argue that applying analysis methods from neuroscience to ANNs will provide a much-needed test of the abilities of these tools. It would also encourage the development of a unified field of neural systems understanding, which can identify shared concepts and methods for studying distributed information processing in artificial and biological systems. To support this argument, we review methods commonly used in neuroscience, along with work that has demonstrated how these methods can be applied to ANNs and what we learn from this, and related efforts from interpretable AI.