P-3.18

Using Massive Individual fMRI Movie Data to Align Artificial and Brain Representations in an Auditory Network

Maëlle Freteault, Université de Montréal, IMT Atlantique, Canada; Basile Pinsard, Julie Boyle, Centre de Recherche de l'Institut Universitaire de Gériatrie de Montréal, Canada; Pierre Bellec, Université de Montréal, Canada; Nicolas Farrugia, IMT Atlantique, France

Session:
Posters 3 Poster

Track:
Cognitive science

Location:
Pacific Ballroom H-O

Presentation Time:
Sat, 27 Aug, 19:30 - 21:30 Pacific Time (UTC -7)

Abstract:
Artificial neural networks have opened new ways to investigate the brain activity evoked by rich stimuli such as movies. The traditional approach is to compare the activity of the brain with an artificial neural network pre-trained on a vision or language task, e.g. image classification in ImageNet. However, a recent study in the language domain suggests that brain encoding can be substantially improved by fine-tuning a pre-trained network to directly predict brain activity. In this study, we develop such an end-to-end fine-tuning of a pre-trained auditory network (called Soundnet) using a massive individual fMRI dataset of movie watching from the Courtois Neuromod Project. We found that fine-tuning led to consistent improvement of brain encoding, in particular in the auditory cortex but also in visual cortices and other distributed brain areas. This work establishes a new approach to build computational models of human auditory processing.

Manuscript:
License:
Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License.
DOI:
10.32470/CCN.2022.1311-0
Publication:
2022 Conference on Cognitive Computational Neuroscience
Presentation
Discussion
Resources
No resources available.
Session P-3
P-3.1: The role of semantics in similarity judgements of scene stimuli
Katerina Marie Simkova, Jasper van den Bosch, University of Birmingham, United Kingdom; Ian Charest, Université de Montréal, Canada
P-3.2: Voxel-wise Encoding Models with Hierarchical Task-optimized Brain Atlas
Huzheng Yang, Shi Gu, University of Electronic Science and Technology of China, China; Yuanning Li, ShanghaiTech University, China
P-3.3: Hidden knobs: Representations for flexible goal-directed decision-making
Romy Froemer, Amitai Shenhav, Brown University, United States; Sebastian Gluth, University of Hamburg, Germany
P-3.4: AI-based modeling of brain and behavior: Combining neuroimaging, imitation learning and video games
Anirudha Kemtur, Francois Paugam, Basile Pinsard, Pravish sainath, Yann Harel, Maximilien Le clei, Julie Boyle, Karim Jerbi, Pierre Bellec, University of montreal, Canada
P-3.5: Intrinsic ionic dynamics, oscillations, and resonance are reflected in and can be extracted from neuronal spike-train cross-correlations
Rodrigo FO Pena, Horacio G Rotstein, New Jersey Institute of Technology and Rutgers University, United States; Martín V Ibarra, Universidad Nacional de la Patagonia San Juan Bosco & CONICET, Argentina
P-3.6: Prediction of brain regions from single channel ECoG signals by deep learning
Ryosuke Negi, Tsukuba Univesity, Japan; Masaru Kuwabara, Ryota Kanai, Araya, Inc, Japan
P-3.7: Image Embeddings Informed by Natural Language Significantly Improve Predictions and Understanding of Human Higher-level Visual Cortex
Aria Wang, Michael Tarr, Leila Wehbe, Carnegie Mellon University, United States
P-3.8: An Efficient Search for Novel Behavioral Strategies in a Vast Program Space
Tzuhsuan Ma, Ann Hermundstad, HHMI Janelia Research Campus, United States
P-3.9: The Individualized Neural Tuning Model: Precise and generalizable cartography of functional architecture in individual brains
Ma Feilong, Guo Jiahui, Yaroslav O. Halchenko, James V. Haxby, Dartmouth College, United States; Samuel A. Nastase, Princeton University, United States; M. Ida Gobbini, University of Bologna, Italy
P-3.10: Brain State Dynamics Underlying False Alarms
Bikash Sahoo, Adam Snyder, University of Rochester, United States
P-3.11: Responses in an orientation recall task are generated by taking expectations of distributional beliefs
Peter Vincent, Athena Akrami, Sainsbury Wellcome Centre, United Kingdom; Maneesh Sahani, Gatsby Computational Neuroscience Unit, United Kingdom
P-3.12: The best advice you can give.
Sevan Harootonian, Mark Ho, Nastasia Klevak, Yael Niv, Princeton University, United States
P-3.13: Modality specificity and generality in the hierarchical levels of cognitive control
Taehyun Yoo, Hyeon-Ae Jeon, Daegu Gyeongbuk Institute of Science and Technology (DGIST), Korea (South)
P-3.14: Sequential object-based attention for robust visual reasoning
Hossein Adeli, Seoyoung Ahn, Gregory Zelinsky, Stony Brook University, United States
P-3.15: Deep convolutional neural networks fail to classify images ‘in the wild’
Michelle Greene, Jennifer Hart, Bates College, United States
P-3.16: Modeling pain in the brain with conditional variational autoencoder
Sungwoo Lee, Jihoon Han, Choongwan Woo, Sungkyunkwan Univercity / Institute for Basic Science, Korea (South)
P-3.18: Using Massive Individual fMRI Movie Data to Align Artificial and Brain Representations in an Auditory Network
Maëlle Freteault, Université de Montréal, IMT Atlantique, Canada; Basile Pinsard, Julie Boyle, Centre de Recherche de l'Institut Universitaire de Gériatrie de Montréal, Canada; Pierre Bellec, Université de Montréal, Canada; Nicolas Farrugia, IMT Atlantique, France
P-3.19: Conversion of ConvNets to Spiking Neural Networks With Less Than One Spike per Neuron
Javier López-Randulfe, Nico Reeb, Alois Knoll, Technical University of Munich, Germany
P-3.20: Brain-to-Brain Linguistic Coupling in Natural Conversations
Zaid Zada, Samuel Nastase, Ariel Goldstein, Uri Hasson, Princeton University, United States
P-3.21: Optimal encoding of prior information in noisy working memory systems
Hua-Dong Xiong, The University of Arizona, United States; Xue-Xin Wei, The University of Texas at Austin, United States
P-3.22: Angular gyrus responses show joint statistical dependence with brain regions selective for different categories
Mengting Fang, University of Pennsylvania, United States; Aidas Aglinskas, Stefano Anzellotti, Boston College, United States; Yichen Li, Harvard University, United States
P-3.23: Different computational strategies for different reinforcement learning problems
Pieter Verbeke, Tom Verguts, Ghent University, Belgium
P-3.24: Investigating individual differences in structure learning
Avinash Vaidya, David Badre, Brown University, United States
P-3.26: Revealing dimensions underlying the organization of observed actions
Zuzanna Kabulska, Angelika Lingnau, University of Regensburg, Germany
P-3.27: Human-like capacity limitation in multi-system models of working memory
Yudi Xie, Christopher Cueva, Guangyu Robert Yang, Massachusetts Institute of Technology, United States; Yu Duan, Aohua Cheng, Tsinghua University, China; Pengcen Jiang, University of Science and Technology of China, China
P-3.28: Are there “affect detectors” in the human limbic system? A multivariate analysis of intracranial single cell recordings
Alexander Lawriw, Christopher Cox, Louisiana State University, United States
P-3.29: Do Convolutional Neural Networks Model Inferior Temporal Cortex Because of Perceptual or Semantic Features?
Anna Truzzi, Rhodri Cusack, Trinity College Dublin, Ireland
P-3.30: Fixation duration variability increases with mind wandering during scene viewing
Kevin O'Neill, Kristina Krasich, Felipe De Brigard, Duke University, United States; Samuel Murray, Providence College, United States; James Brockmole, University of Notre Dame, United States; Antje Nuthmann, Kiel University, Germany
P-3.31: Individual auto-regressive models for long-term prediction of BOLD fMRI signal
François Paugam, Guillaume Lajoie, Pierre Bellec, Université de Montréal, Canada; Basile Pinsard, Centre de Recherche de l'Institut Gériatrique de Montréal, Canada
P-3.32: Face Pareidolia Selectivity in Macaque Face-Cells Does Not Reflect Perceived Faceness
Saloni Sharma, Kasper Vinken, Margaret Livingstone, Harvard Medical School, United States
P-3.33: Do deep convolutional neural networks accurately model representations beyond the ventral stream?
Dawn Finzi, Daniel Yamins, Kalanit Grill-Spector, Stanford University, United States; Kendrick Kay, University of Minnesota, United States
P-3.34: Rationalizing Behavior during Virtual Reality Navigation
Yizhou Chen, Baylor college of medicine, United States; Paul Schrater, University of Minnesota, United States; Dora Angelaki, New York University, United States; Xaq Pitkow, Rice University, Baylor college of medicine, United States
P-3.35: Neural Mechanisms of Credit Assignment for Inferred Relationships in a Structured World
Phillip Witkowski, Seongmin Park, Erie Boorman, University of California, Daivs, United States
P-3.36: Developmental differences in social brain responses during movie viewing
Angira Shirahatti, Leyla Isik, Johns Hopkins University, United States
P-3.37: Transfer learning in a 3D-CNN is beneficial for small sample sizes in HCP task data
Philipp Seidel, Jens V. Schwarzbach, Regensburg University, Germany
P-3.38: Gaze-centered spatial representations in human hippocampus
Zitong Lu, Julie Golomb, The Ohio State University, United States; Anna Shafer-Skelton, University of Pennsylvania, United States
P-3.39: Large Scale Resting-State Network Connectivities Predict Verbal Suggestibility
Yeganeh Farahzadi, Zoltan Kekecs, Eötvös Loránd University, Hungary
P-3.41: Lateral Inhibition Facilitates Sequential Learning in a Hippocampus-Inspired Auto-Associator
Benjamin Midler, James McClelland, Stanford University, United States
P-3.42: Latent dimensionality scales with the performance of deep learning models of visual cortex
Eric Elmoznino, Michael Bonner, Johns Hopkins University, United States
P-3.43: Towards Precise and Robust Hippocampus Segmentation using Self-Supervised Contrastive Learning
Kassymzhomart Kunanbayev, Donggon Jang, Jeongwon Lee, Dae-Shik Kim, KAIST, Korea (South)
P-3.44: Effects of Predictability and Controllability on Pain Perception
Marie Habermann, Christian Büchel, University Medical Center Hamburg-Eppendorf, Germany
P-3.45: A mathematical framework for bridging Marr’s levels
Anja Meunier, Moritz Gosse-Wentrup, University of Vienna, Austria
P-3.46: Precision-weighted evidence integration predicts time-varying influence of memory on perceptual decisions
Maria Khoudary, Megan Peters, Aaron Bornstein, University of California, Irvine, United States
P-3.47: How Well Do Contrastive Learning Algorithms Model Human Real-time and Life-long Learning?
Chengxu Zhuang, Violet Xiang, Daniel Yamins, Stanford University, United States; Yoon Bai, James DiCarlo, MIT, United States; Xiaoxuan Jia, Allen Institute, United States
P-3.48: Different Brain Mechanisms of Time Estimation Depending on Situational Information
Jungtak Park, Hyeon-Ae Jeon, Daegu Gyeongbuk Institute of Science and Technology (DGIST), Korea (South)
P-3.49: Uncovering the Spatiotemporal Dynamics of Goal-driven Efficient Coding with a Brain-supervised Sparse coding Network
Bruce Hansen, Isabel Gephart, Victoria Gobo, Colgate University, United States; Michelle Greene, Bates College, United States; David Field, Cornell University, United States
P-3.50: A New Computational Framework for Estimating Spatio-temporal Population Receptive Fields in Human Visual Cortex
Insub Kim, Eline Kupers, Kalanit Grill-Spector, Stanford University, United States; Garikoitz Lerma-Usabiaga, Basque Center on Cognition, Brain and Language, Spain; Won Mok Shim, Sungkyunkwan University, Korea (South)
P-3.51: Economically expanding internal models in human density estimation
Tianyuan Teng, Hang Zhang, Peking University, China; Li Kevin Wenliang, University College London, United Kingdom
P-3.52: The geometry of cognitive maps under dynamic cognitive control
Seongmin Park, Maryam Zolfaghar, Jacob Russin, Douglas Miller, Randall O’Reilly, Erie Boorman, University of California, Davis, United States
P-3.53: Reconstruction of line illusion from human brain activity
Fan Cheng, Tomoyasu Horikawa, Advanced Telecommunications Research Institute International(ATR), Japan; Kei Majima, Yukiyasu Kamitani, Kyoto University, Japan
P-3.54: Statistical inference on representational geometries
Heiko Schütt, Alexander D. Kipnis, Nikolaus Kriegeskorte, Columbia University, United States; Jörn Diedrichsen, Western University, Canada
P-3.55: Network Architecture of Cortex and Cerebellum for Supporting Super-learning
Serhat Çağdaş, Yalova University, Turkey; Ismail Akturk, Ozyegin University, Turkey; N. Serap Şengör, İstanbul Technical University, Turkey
P-3.56: Humans learning a complex task are picky and sticky
Tiago Quendera, Zachary F. Mainen, Champalimaud Foundation, Portugal; Dongrui Deng, Xi’an Jiaotong University, China; Mani Hamidi, University of Tubingen, Germany; Mattia Bergomi, Veos Digital, Italy; Gautam Agarwal, Claremont Colleges, United States
P-3.57: Leaving alternatives behind: A theoretical and experimental investigation of the role of mutual inhibition in shaping choice
Xiamin Leng, Romy Frömer, Thomas Summe, Amitai Shenhav, Brown University, United States
P-3.58: Orthogonal neural encoding of targets and distractors supports cognitive control
Harrison Ritz, Amitai Shenhav, Brown University, United States
P-3.59: Identifying transfer learning in the reshaping of inductive biases
Anna Székely, Wigner Research Centre for Physics // Budapest University of Technology, Hungary; Balázs Török, Mozalearn Ltd., Hungary; Dávid Gergely Nagy, Gergő Orbán, Wigner Research Centre for Physics, Hungary; Mariann M. Kiss, Dezső Németh, Eötvös Lóránd University, Hungary; Karolina Janacsek, University of Greenwich, United Kingdom
P-3.60: The Neural Representation of Real-World Object Size in Natural Images
Andrew Luo, Leila Wehbe, Michael Tarr, Margaret Henderson, Carnegie Mellon University, United States
P-3.61: Component Activity States Underlying Memory Reactivation in the Posterior Medial Cortex
Yoonjung Lee, Hongmi Lee, Janice Chen, Johns Hopkins University, United States
P-3.62: Time cell encoding is decoupled from time perception in deep reinforcement learning agents
Ann Zixiang Huang, Dongyan Lin, Blake Richards, McGill University, Quebec AI Institute (Mila), Canada
P-3.63: Syntax in working memory using a simple plastic attractor
Lin Sun, Imperial College London, United Kingdom; Sanjay G. Manohar, University of Oxford, United Kingdom
P-3.64: Relating covariability in visual cortex to natural image statistics
Amirhossein Farzmahdi, Ruben Coen-Cagli, Albert Einstein College of Medicine, United States
P-3.65: Learning efficient attractor-based working memory representations in heterogeneous environments
Tahra L Eissa, Zachary P Kilpatrick, University of Colorado Boulder, United States
P-3.66: Attractor dynamics account for decision uncertainty in macaque prefrontal cortex
Siyu Wang, Rossella Falcone, Barry Richmond, Bruno Averbeck, National Institute of Mental Health, United States
P-3.67: A Characterization of the Neural Representation of Confidence during Probabilistic Learning
Tiffany Bounmy, NeuroSpin, CEA, INSERM, Université Paris-Saclay, Université de Paris, France; Evelyn Eger, Florent Meyniel, NeuroSpin, CEA, INSERM, Université Paris-Saclay, France
P-3.68: Representational dynamics of listened and imagined musical sound sequences
David Ricardo Quiroga-Martinez, Robert Knight, Helen Wills Neuroscience Institute and the Department of Psychology, UC Berkeley, United States; Leonardo Bonetti, Center for Eudaimonia and Human Flourishing, Linacre College & Department of Psychiatry, University of Oxford, United Kingdom; Peter Vuust, Center for Music in the Brain, Aarhus University and the Royal Academy of Music, Denmark, Denmark
P-3.69: Analysis of Transformer attention in EEG signal classification
Philipp Thölke, Karim Jerbi, University of Montreal, Canada
P-3.70: Superstitious learning of abstract order from random reinforcement
Yuhao Jin, Jacqueline Gottlieb, Vincent Ferrera, Columbia University, United States; Greg Jensen, Reed College, United States
P-3.71: Overcoming the Failure of Neoclassical Economics to Capture Excessive Demand: A Learning-to-Neuroforecast Experimental Approach
John Haracz, Indiana University, United States
P-3.72: Probing population codes and circuit dynamics of probabilistic learning
Nuttida Rungratsameetaweemana, The Salk Institute for Biological Studies, United States; Shruti Kumar, Javier Garcia, US Combat Capabilities Development Command Army Research Laboratory, United States
P-3.73: Occluded object completion occurs in full across human visual cortex but emerges gradually across layers of CORnet-S
David Coggan, Frank Tong, Vanderbilt, United States