Conference on Cognitive Computational Neuroscience

In our daily lives, we quickly and effortlessly perceive features of others’ interactions. Extracting these social details is crucial for deciding how to act in the social world, but little is understood about how this is solved in the mind and brain. Recent work has identified a visually-selective region for the presence of a social interaction in posterior superior temporal sulcus (pSTS), but whether and how diverse features of a social interaction (ranging from visual to high-level) are presented in the pSTS or elsewhere in the brain is unknown. To answer this question, we showed participants 250 3-second video clips of naturalistic two-person interactions while undergoing fMRI. We used an encoding model approach to model where visual and social features of the videos are represented in the brain. We replicate known preference for scene and object features in visual cortex. We also find preference for facing bodies in EBA and joint action in pSTS, extending prior findings with controlled stimuli to natural settings. Finally, we identify regions along the extent of the STS that show a preference for third-party communication. Together, these results suggest a hierarchy of visual to social feature processing along the lateral surface of the brain.