Conference on Cognitive Computational Neuroscience

Deep neural networks (DNNs) can explain a large portion of variance in image-evoked cortical responses by accounting for the highest variance latent dimensions in neural data, such as dimensions corresponding to animacy, aspect ratio, and curvature. However, there is a long tail of low-variance latent dimensions in image-evoked cortical responses that may nonetheless be critical to human vision. We wondered if these low-variance dimensions are meaningful and whether current DNNs are successful in explaining them. To answer these questions, we estimated the number of reliable dimensions in visual cortex in a large-scale human fMRI dataset and assessed how well DNNs performed at explaining these dimensions. We found that hundreds of dimensions contained reliable stimulus-relevant information. However, standard DNN encoding models explain a much smaller number of these dimensions—often an order of magnitude smaller than the number of reliable dimensions in the data. Our findings demonstrate the surprisingly low-dimensional nature of explained variance in computational models of visual cortex, and reveal the long-tail of complex, stimulus-relevant information in cortical responses that remains to be explained.