Conference on Cognitive Computational Neuroscience

Although both use reward-based learning, it is still unclear why deep reinforcement learning (RL) agents, in general, perform worse than the brain in novel out-of-distribution (OOD) tasks. Here, we propose one reason: generalization requires task-appropriate modular neural architectures like the brain; inferior generalization abilities result from using architectures without task-appropriate modules. To verify this hypothesis, we used a spatial navigation task to train RL agents with different neural architectures varying in modularity, then compared their generalization abilities in a novel task presenting novel sensorimotor mappings. We found that, although all agents could master the training task, only those with highly modular architectures separating computations of task variables learned a robust state representation, which supported generalization in the novel task. Our work exemplifies the rationale of the architectural modularity, i.e., supporting generalization.