SPECIAL ISSUE:
Cognitive Computational Neuroscience of Language

Computational Language Modeling and the
Promise of in Silico Experimentation

Shailee Jain1

, Vy A. Vo3, Leila Wehbe4,5, and Alexander G. Huth1,2

1Departamento de Ciencias de la Computación, University of Texas at Austin, austin, Texas, EE.UU
2Department of Neuroscience, University of Texas at Austin, austin, Texas, EE.UU
3Brain-Inspired Computing Lab, Intel Labs, Hillsboro, O, EE.UU
4Machine Learning Department, Carnegie Mellon University, pittsburgh, Pensilvania, EE.UU
5Neuroscience Institute, Carnegie Mellon University, pittsburgh, Pensilvania, EE.UU

Palabras clave: computational neuroscience, aprendizaje profundo, encoding models, experimental design,
natural language processing, naturalistic stimuli

ABSTRACTO

Language neuroscience currently relies on two major experimental paradigms: controlled
experiments using carefully hand-designed stimuli, and natural stimulus experiments. Estos
approaches have complementary advantages which allow them to address distinct aspects of
the neurobiology of language, but each approach also comes with drawbacks. Here we
discuss a third paradigm—in silico experimentation using deep learning-based encoding
models—that has been enabled by recent advances in cognitive computational neuroscience.
This paradigm promises to combine the interpretability of controlled experiments with the
generalizability and broad scope of natural stimulus experiments. We show four examples of
simulating language neuroscience experiments in silico and then discuss both the advantages
and caveats of this approach.

INTRODUCCIÓN

One major goal of language neuroscience is to characterize the function of different brain
regions and networks that are engaged in language processing. A large body of work has inves-
tigated different aspects of language processing—such as semantic knowledge representation
(Binder et al., 2009; Huth et al., 2016; Mitchell et al., 2008), syntactic processing (Friederici
et al., 2000), and phonological mapping (Chang et al., 2010)—and characterized the proper-
ties of the language network like the processing timescale (Lerner et al., 2011), convergence
with different sensory systems (Popham et al., 2021), role in bilingual representations (chan
et al., 2008), y más. To study these questions, language neuroscientists have developed a
suite of experimental designs, ranging from highly specific controlled experiments to natural
stimulus experiments and, more recently, deep learning-based approaches for computational
modelado.

Each experimental design can be thought of as an investigative tool for understanding the
brain’s response Rv = fv(S ), where fv is the function that some brain element v (p.ej., a single
neurona, voxel, brain area, or magnetoencephalography [MEG] sensor) computes over a given
language stimulus S to produce responses Rv. Some experimental designs—like contrast-based
studies—aim to directly compare certain aspect of fv, such as the response to different word
categories. Others—like experiments with complex stimuli that are paired with encoding

un acceso abierto

diario

Citación: jainista, S., Vo, Vy A., Wehbe, l.,
& Huth, A. GRAMO. (2023). computacional
language modeling and the promise of
in silico experimentation. Neurobiología
of Language. Publicación anticipada.
https://doi.org/10.1162/nol_a_00101

DOI:
https://doi.org/10.1162/nol_a_00101

Recibió: 28 Febrero 2022
Aceptado: 18 Enero 2023

Conflicto de intereses: Los autores tienen
declaró que no hay intereses en competencia
existir.

Autor correspondiente:
Alexander G. Huth
huth@cs.utexas.edu

Editor de manejo:
Alessandro Lopopolo

Derechos de autor: © 2023
Instituto de Tecnología de Massachusetts
Publicado bajo Creative Commons
Atribución 4.0 Internacional
(CC POR 4.0) licencia

La prensa del MIT

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
norte
oh

/

yo
/

yo

a
r
t
i
C
mi
–
pag
d

F
/

d
oh

i
/

yo

/

.

/

1
0
1
1
6
2
norte
oh
_
a
_
0
0
1
0
1
2
0
7
4
5
3
9
norte
oh
_
a
_
0
0
1
0
1
pag
d

.

/

yo

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

In silico language neuroscience

In silico experimentation:
Simulating experiments by predicting
brain responses with a computational
model to test generalizability and do
efficient hypothesis testing.

models—approximate fv using computational tools, and this allows for the prediction of activ-
ity related to new stimuli. In this paper we describe an alternative to existing paradigms:
in silico controlled experimentation using computational models of naturalistic language pro-
cesando. This hybrid approach combines the strengths of controlled and naturalistic paradigms
to achieve high ecological generalizability, high experimental efficiency and reusability, alto
interpretability, and sensitivity to individual participant effects.

We first compare and contrast experimental designs based on their effectiveness and effi-
ciency for revealing fv. Then we introduce the in silico experimentation paradigm with deep
learning models. We discuss four different neuroimaging studies that use this paradigm to
investigate different linguistic phenomena in the brain. And finally, we discuss the potential
of this approach to alleviate the problems of reproducibility in language neuroscience, también
as caveats and pitfalls of in silico experimentation.

EXPERIMENTAL DESIGNS IN LANGUAGE NEUROSCIENCE

Controlled Experimental Design: Contrast-Based Studies

Language is a rich and complex modality that humans are uniquely specialized to process.
Given this complexity, neuroscientists have traditionally broken language down into specific
processes and properties and then designed controlled experiments to test each separately
(Binder et al., 2009; Friederici et al., 2000). Consider the example of investigating which areas
of the brain are responsible for encoding specific types of semantic categories like “actions”
(Kable et al., 2002; Noppeney et al., 2005; Wallentin et al., 2005). A simple and effective
approach is to collect and compare brain responses to action words and pair them with min-
imally different words, perhaps similar length and frequency “object” words. If some brain
element v responds more to stimuli containing the property being tested than the control
stimuli—that is, fv(“action” words) > fv(“object” words)—the experimenter concludes that v
is involved in processing action words. Similarmente, the N400 effect (Kutas & Hillyard, 1984)
is assessed by testing whether an elements’ fv reflects surprise with respect to some context.
If fv(expected word|contexto) < fv(unexpected word|context), it would suggest that the brain element is capturing word surprisal. In order for a contrast-based study to be interpretable, it is vital to remove any confounds that could corrupt observed responses and lead to false positives. Binder et al. (2009) charac- terize three types of confounds: the main and control conditions could differ in low-level pro- cessing demands (phonological/orthographic); the main and control conditions could differ in working memory demands, attention demands, and so forth; and, in passive tasks, the partic- ipants might engage in different mental imagery or task-related thoughts in the two conditions. If such confounds are controlled effectively, one can assume that the observed brain response will be identical in all respects unless v specifically captures the property being studied. For example, if the action and object words are matched on all other properties, fv(“action” words) and fv (“object” words) will only differ if v selectively encodes action or object concepts. Con- sequently, the contrast-based paradigm has high interpretability, as any variations in observed response can be attributed to the hypothesis. This clear and direct relationship between hypothesis and result ensures that the experiment has scientific value even when a hypothesis or theory is incorrect. The controlled experimental design has thus been fundamental in revealing many important aspects of brain function, such as the specialization of parts of tem- poral cortex for speech processing (reviewed in S. K. Scott, 2019) and distinct neural systems for concrete versus abstract concepts (Binder et al., 2005, 2009). Neurobiology of Language 2 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u n o / l / l a r t i c e - p d f / d o i / l / . / 1 0 1 1 6 2 n o _ a _ 0 0 1 0 1 2 0 7 4 5 3 9 n o _ a _ 0 0 1 0 1 p d . / l f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 In silico language neuroscience Naturalistic stimuli: Stimuli that subjects could be exposed to in real life; not artificially constructed for an experiment. While this paradigm has been hugely influential and effective in language neuroscience, it is not without flaws. Perhaps the biggest drawback of most contrast-based designs is the lack of ecological generalizability (Hamilton & Huth, 2018; Matusz et al., 2019). To avoid confounds, controlled experiments often employ the simplest linguistic constructions required to demon- strate an effect, such as single words in the action versus object contrast. While we are fully capable of identifying action words in isolation, it is not necessary that the brain employs the same networks to understand such words in real-world settings (Matusz et al., 2019), for exam- ple, as used in a conversation or a story. In contrast to such studies, those using naturalistic stimuli have found more engagement and activation in higher order cortical regions, likely due to the incorporation of long-range structure (Deniz et al., 2021; Lerner et al., 2011). Further- more, due to practical limitations, controlled studies typically use small stimulus sets that span a limited domain. For example, neuroimaging studies of the action contrast often use fewer than 100 words in each condition. This raises the probability that there is something peculiar or nonrepresentative about the experimental stimuli, making it more difficult to reproduce the effect or establish generalizability to a broader stimulus domain (Yarkoni, 2022). Small stimu- lus sets can also artificially inflate the observed statistical significance (Westfall et al., 2017). While controlled studies offer a very clear and direct relationship between the hypothesis and experimental result, their value depends entirely on the quality of the hypothesis. In many cases, narrowing the experimental hypothesis to focus on contrasts of a particular stimulus property may be misleading, and may fail to account for interactions between several other stimulus properties. For example, standard statistical models for assessing the “action” contrast assume that brain response is identically distributed for any subcategorization of this semantic concept. However, studies such as Hauk et al. (2004) have found that different regions across cortex selectively encode hand-related, foot-related, or mouth-related actions. This type of subcategory specificity decreases the statistical power of the overall action contrast, thereby increasing the probability of false negatives. Worse, if the overall action contrast has unevenly sampled these subcategories, the statistical power to detect action selectivity will vary in an unexpected and unknown fashion between brain areas. This issue can occur in any contrast- based experiment and is difficult or even impossible to detect by the experimenter. One poten- tial solution would be to combine data across different contrast-based experiments, which could reveal interactions between effects. However, separate controlled experiments often do not share analysis methods, stimulus sets, or subjects, making it difficult to combine data or compare effect sizes across experiments. Lastly, for each language property that one wishes to investigate using a controlled experiment, one needs to design specific controls and repeat- edly measure Rv. This results in limited reusability of experimental data, slowing down the process of scientific discovery. Naturalistic Stimuli To combat the lack of stimulus generalization and limited reusability, there has been a rising trend toward naturalistic experimental paradigms (Brennan et al., 2012; Hamilton & Huth, 2018; Hasson et al., 2008; Lerner et al., 2011; Regev et al., 2013; Shain et al., 2020). With the development of better neuroimaging/recording technology, we now have access to high quality brain recordings of humans while they perceive engaging, ecologically valid stimuli like podcasts (Huth et al., 2016; Lerner et al., 2011; J. Li et al., 2022; Nastase et al., 2021; S. Wang et al., 2022), fictional books (Bhattasali et al., 2020; Wehbe, Murphy, et al., 2014), and movies (J. Chen et al., 2017)—all examples of stimuli humans encounter or seek out in their everyday lives. Recent work has further developed this naturalistic paradigm to incorpo- rate communication and social processing, beyond passive perception (Bevilacqua et al., Neurobiology of Language 3 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u n o / l / l a r t i c e - p d f / d o i / l / / . 1 0 1 1 6 2 n o _ a _ 0 0 1 0 1 2 0 7 4 5 3 9 n o _ a _ 0 0 1 0 1 p d . / l f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 In silico language neuroscience Ecological validity: Determiniation whether an experiment is likely to faithfully reflect and generalize to situations encountered in real life. Linearized encoding model: Model that learns to predict elicited response in a brain element as a linear function of features of interest extracted from stimuli. 2019; Redcay & Moraczewski, 2020). Naturalistic stimulus data sets are easier to construct and often larger than controlled stimuli. For example, J. Chen et al. (2017) publicly released a data set collected on a 50 min movie, Wehbe, Murphy, et al. (2014) released data collected on an entire chapter from the Harry Potter books, comprising more than 5,000 words, and LeBel et al. (2022) released data collected on over 5 hr of English podcasts per participant. These stimuli also provide a diverse test bed of linguistic phenomena—from a broad array of semantic concepts to rich temporal structure capturing discourse-level information. Further- more, they do not directly constrain the hypotheses the experimenter can test and thus facil- itate high reusability of the data. However, this also means that natural stimulus data have low statistical power with respect to any specific hypothesis, and it is necessary to carefully design analyses to control for confounding effects. This makes interpretation of the observed effects much more challenging than contrast-based experiments. Naturalistic Experimental Design: Controlled Manipulations of Naturalistic Stimuli To reap the benefits of both interpretable controlled experiments and generalizable naturalistic stimuli, some studies have deployed a hybrid experimental design (Chien & Honey, 2020; Deniz et al., 2019; Lerner et al., 2011; Overath et al., 2015; Yeshurun et al., 2017). Here, natural stimuli are manipulated to change or remove some specific language cue or property (e.g., scrambling the words in a story) and the sensitivity of different brain regions to this manipulation is measured, for example, fv(intact story) vs. fv(scrambled story). This can reveal properties across the brain like the timescale of information represented (Lerner et al., 2011, 2014) or specificity to the type of naturalistic stimulus, such as human speech (Overath et al., 2015). This experimental design accounts for ecological validity by restricting analyses to brain regions that robustly respond to the naturalistic stimuli. Furthermore, it has the same advantage of controlled experiments when it comes to interpretation: Assuming effective con- trol of confounds, any observed change in brain activity is likely to be an effect of the stimulus manipulation. However, this approach also has disadvantages: The manipulated stimuli are often unnatural (like reversed or scrambled speech) and restrict the types of interactions the experimenter can observe. For example, the scrambled story experiment assumes that all regions processing short timescale information will behave identically. The manipulated stim- uli also limit the reusability of the experiment, meaning that a new experiment needs to be designed for each effect of interest. Naturalistic Experimental Design: Predictive Computational Modeling Encoding models are an alternative computational approach for leveraging naturalistic exper- imental data (Bhattasali et al., 2019; Caucheteux & King, 2022; Goldstein et al., 2021; Huth et al., 2016; Jain et al., 2020; Jain & Huth, 2018, p. 20; Schrimpf et al., 2021; Wehbe, Vaswani, et al., 2014). These predictive models learn to simulate elicited brain responses Rv = fv(S ) to natural language stimuli S by building a computational approximation to the function fv for each brain element v, typically in every participant individually. Here, R can be captured by any neuroimaging or neural recording technique. Given limitations on data set sizes, the search for fv is typically constrained to linearized encoding models, gv(Ls(S )) (M. C.-K. Wu et al., 2006), where gv is a linear combination of features extracted from the stimulus by a function Ls. While gv is termed a linear model, of particular interest is the linearizing transform Ls. Contrast-based experimental designs test a hypothesis by comparing responses elicited by different conditions. Each condition is composed of stimuli that share some features (e.g., all words that describe actions). Encoding models can test the same hypothesis by incorporating these features into Ls. For example, for every word in the natural stimulus, one could create an Neurobiology of Language 4 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u n o / l / l a r t i c e - p d f / d o i / l . / / 1 0 1 1 6 2 n o _ a _ 0 0 1 0 1 2 0 7 4 5 3 9 n o _ a _ 0 0 1 0 1 p d / . l f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 In silico language neuroscience Neural language models: Types of artificial neural networks that learn to predict the next word in a sequence from past context. indicator feature Iaction that is 1 if the word describes an action and 0 otherwise. Feature spaces consisting of 1s and 0s are equivalent to a contrast-based experimental design, assuming other confounds have been eliminated. Encoding models can also adopt much more complex and high-dimensional functions for Ls. This makes it possible to account for multiple, interacting stimulus properties that may affect the response Rv. For example, Ls could indicate multiple levels of semantic categories. In the example of action and object words, the feature space could indicate that hand-related, foot-related, and mouth-related words were all types of actions, and distinguish all action words from multiple subcategories of objects. One recent example of such a high-dimensional feature space that captures semantic similarity (Mikolov et al., 2013; Pennington et al., 2014) is word embeddings, which have been used to characterize semantic language representations across the human brain (de Heer et al., 2017; Huth et al., 2016; Wehbe, Murphy, et al., 2014; Wehbe, Vaswani, et al., 2014). With a suitably rich linearizing transform Ls, this approach vastly expands the set of hypotheses that can be reasonably explored with a limited data set. The expandable feature space also allows encoding models great flexibility to test addi- tional hypotheses without collecting new data, leading to high reusability. Estimating the brain response as a function of the nonlinear feature space is made possible by collecting large data sets that are partitioned into a portion for training (estimating) the model and a portion for testing the model on unseen data. Typically, regularized linear regression is used to estimate the linear relationship gv based on the feature space Ls. This is used to predict new responses ^Rv ¼ gv Ls Sð Þ ð Þ to unseen stimuli. Finally, the model is evaluated by measuring how well it predicts brain responses, ρ(fv(Snew), g(Ls(Snew))). Thus, unlike other approaches, encoding models explicitly measure generalizability by testing on new, naturalistic stimuli. In contrast-based designs, a gen- eralization test is usually achieved through replication with an independent data set, often from a different lab where protocols and analysis details may differ. With encoding models, the same experimenter usually runs their own generalization test and directly estimates how much of the neural response Rv is explained by the model, holding all other variables constant. Encoding models can also be used to investigate if the same brain region under different tasks have the same tuning. For example, Deniz et al. (2019) show that semantic tuning is preserved between reading and listening, while Çukur et al. (2013) show that the tuning of different regions in visual cortex when attending to a given category is biased toward the attended category. Encoding models can also be used to compare tuning of two different regions (Toneva, Williams, et al., 2022 ). Artificial Neural Networks as a Rich Source of Linguistic Features The most important choice that an experimenter makes when using encoding models is that of the linearizing transform. To find useful linearizing transforms, neuroscience has mostly followed advances in computational linguistics or natural language processing (NLP) where, in recent years, deep learning (DL) models trained using self-supervision have seen great suc- cess. One such cornerstone model is the neural language model—a self-supervised artificial neural network (ANN) that learns to predict the next word in a sequence, wt+1, from the context provided by previous words (w1, w2 … wt). Several recent studies have shown that representa- tions derived from LMs capture many linguistic properties of the preceding sequence (w1, w2 … wt) like dependency parse structure, semantic roles, and sentiment (see Mahowald et al., 2020, for a review; Clark et al., 2019; Conneau et al., 2018; Gulordava et al., 2018; Haber & Poesio, 2021; Hewitt & Liang, 2019; Hewitt & Manning, 2019; Lakretz et al., 2019; B. Z. Li et al., 2021; Linzen & Leonard, 2018; Marvin & Linzen, 2018; Prasad et al., 2019; Tenney, Das, et al., 2019; Neurobiology of Language 5 Q1 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u n o / l / l a r t i c e - p d f / d o i / l / . / 1 0 1 1 6 2 n o _ a _ 0 0 1 0 1 2 0 7 4 5 3 9 n o _ a _ 0 0 1 0 1 p d . / l f b y g u e s t Q2 t o n 0 7 S e p e m b e r 2 0 2 3 In silico language neuroscience Tenney, Xia, et al. 2019). While this by no means is a complete representation of phrase mean- ing (Bender & Koller, 2020), using a language model as a linearizing transform has been shown to effectively predict natural language responses in both the cortex and cerebellum, with dif- ferent neuroimaging techniques and stimulus presentation modalities (Abnar et al., 2019; Anderson et al., 2021; Caucheteux & King, 2022; Goldstein et al., 2021; Jain et al., 2020; Jain & Huth, 2018; Kumar et al., 2022; LeBel et al., 2021; Reddy & Wehbe, 2020; Schrimpf et al., 2021; Toneva, Mitchell, et al., 2022 ; Toneva & Wehbe, 2019; Wehbe, Murphy, et al., 2014; Wehbe, Vaswani, et al., 2014; S. Wang et al., 2020). Moreover, these models easily outperform earlier word embedding encoding models that use one static feature vector for each word in the stimulus and thus ignore the effects of context (Antonello et al., 2021; Caucheteux & King, 2022; Jain & Huth, 2018). Deep LMs have also been used to investigate the mapping between ANN layer depth and hierarchical language processing (Jain & Huth, 2018). Along similar lines and at a lower-level, supervised and self-supervised models of speech acoustics have been used to develop the best current models of auditory processing in human cortex to date (Kell et al., 2018; Y. Li et al., 2022; Millet et al., 2022; Millet & King, 2021; Vaidya et al., 2022). The unprecedented success of DL-based approaches over earlier encoding models can likely be attributed to several important factors. First, features extracted from the DL-based models have the ability to represent many different types of linguistic information, as discussed above. Second, DL-based models serially process words from a language stimulus to generate incremental features. This mimics causal processing in humans and thus offers an important advantage over static representations like word embeddings, which cannot encode contextual properties. Third, recent work has shown that these models often recapitulate human errors and judgments, such as effectively predicting behavioral data of human reading times (Aurnhammer & Frank, 2018; Futrell et al., 2019; Goodkind & Bicknell, 2018; Merkx & Frank, 2021; Wilcox et al., 2021). This again suggests some isomorphism between human language processing and DL-based models. The next word prediction objective also enables language models to perform well on psycholinguistic diagnostics like the cloze task, although there is substantial room for improvement (Ettinger, 2020; Pandia & Ettinger, 2021). Finally, self- supervised ANNs, that is, networks that predict the next word or speech frame, transfer well to downstream language tasks like question answering and coreference resolution, and to speech tasks like speaker verification and translation across languages (Z. Chen et al., 2022; A. Wu et al., 2020). This suggests that the self-supervised networks are learning representations of language that are useful for many tasks that humans may encounter. These factors have contributed to the increasing popularity of DL-based encoding models as an investigative tool of brain function. This approach has revealed aspects of how the brain represents compositional meaning (Toneva, Mitchell, et al., 2022), provided fine-grained esti- mates of processing timescales across cortex (Jain et al., 2020), and uncovered new evidence for the cerebellum’s role in language understanding (LeBel et al., 2021). Yet despite these successes, DL-based encoding models are hard to interpret. The represen- tations produced by language models are entirely learned by black-box neural networks, and thus cannot be understood with the same ease as the indicator features described in the above section . While the representations themselves are opaque, one potential avenue is to interpret the success of a DL-based model at predicting some brain area as suggesting a commonality between that brain area and the objective that model was trained for (e.g., word identification (Kell et al., 2018) or 3D vision tasks (A. Wang et al., 2019)). However, the fact that similar representations can be derived from DL-based models that are trained for different objectives puts this type of interpretation on shaky ground (Antonello & Huth, 2022; Guest & Martin, 2023). These difficulties have left the field at something of an impasse: We know that DL-based Neurobiology of Language 6 Q3 Q4 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u n o / l / l a r t i c e - p d f / d o i / l / / . 1 0 1 1 6 2 n o _ a _ 0 0 1 0 1 2 0 7 4 5 3 9 n o _ a _ 0 0 1 0 1 p d / . l f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 Q5 In silico language neuroscience models are extremely effective at predicting brain responses, but we are unsure why and unsure what these models can tell us about how the brain processes language. Pièce De Résistance: In Silico Experimentation With DL-Based Encoding Models Controlled experiments and encoding models using naturalistic stimuli both have distinct advantages and disadvantages. However, it may be possible to combine these paradigms in a way that avoids the disadvantages and retains the advantages. To this end, we present an experimental design that combines these two paradigms: in silico controlled experimentation using encoding models. This paradigm first trains encoding models on an ecologically valid, highly generalizable naturalistic experiment. Then, it uses the encoding models to simulate brain activity to controlled stimulus variations or contrasts. Notably, this does not require addi- tional data to be collected for every condition. The first use of in silico experimentation is to test if effects discovered in controlled, non- ecologically valid setups generalize to naturalistic stimuli. This experimental design also facil- itates quick and efficient hypothesis testing. Experimenters can prototype new controlled experiments and narrow down the desired contrasts or stimuli without having to repeatedly measure in vivo. While this is a complement to and not a substitute for in vivo experiments that should follow the prototyping phase, in silico experimentation can greatly reduce the cost of generalizability and hypothesis testing, and accelerate scientific discovery. In Figure 1, we present a controlled experimental design with its in silico counterpart. Figure 1A shows an experimental paradigm that was designed to understand linguistic l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u n o / l / l a r t i c e - p d f / d o i / l / / . 1 0 1 1 6 2 n o _ a _ 0 0 1 0 1 2 0 7 4 5 3 9 n o _ a _ 0 0 1 0 1 p d . / l f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 Figure 1. Example of an in silico adaptation of a controlled experiment. (A) The original MEG study investigated composition over two-word phrases (Bemis & Pylkkänen, 2011). This was done by presenting three different types of phrases to participants to solve a picture matching task. By contrasting the elicited brain responses in the composition condition with the responses in the list and non-word conditions, the authors could infer which brain regions are engaged in compositional processing of two-word phrases. (B) This experimental paradigm can be conceptually simulated with LM-based fMRI encoding models of naturalistic stimuli. The composition and list conditions can be tested by using the learned encoding model to predict each voxel’s response to a large, diverse corpus of phrases. The non-word condition can be simulated by replacing the first word in a phrase with a non-word, extracting new ablated features of the phrase from the LM and using the encoding model to predict the brain’s response to the ablated phrase. If a voxel’s response is highly sensitive to the removal of the first word, it would suggest that the voxel combines information over both words to arrive at meaning. This provides a data-efficient way to test for compositional processing across diverse types of phrase constructions. fMRI = functional magetic resonance imaging; LM = language model; MEG = magnetoencephalography. Neurobiology of Language 7 In silico language neuroscience composition of two-word phrases (Bemis & Pylkkänen, 2011). Participants were presented with phrases in which meaning can be composed across constituent words and contrasting conditions where it cannot (word list and non-word). This experiment can be conceptually simulated in silico, as shown in Figure 1B (Jain & Huth, 2023). Instead of collecting separate neuroimaging data for each type of phrase construction, the in silico experiment was done with DL-based encoding models trained on two-word sequences. The learned models were first used to predict brain responses to a large, diverse corpus of phrases that contained both noun–noun and adjective–noun constructs among others. Next, the non-word condition was simulated by replacing the first word in the phrase with a non-word, extracting a new ablated feature, and finally predicting each functional magnetic resonance imaging (fMRI) voxel’s response to the ablated phrase. Assuming that the DL-based encoding model captures com- positional effects, this in silico experiment can ameliorate the disadvantages of both controlled and encoding model-based experimental designs. First, since simulating responses is trivial in both time and cost, the simulated experiment can use thousands or even millions of two-word phrases instead of the hundreds that can be tested in vivo. This ameliorates problems that arise with limited stimulus sets that may fail to account for key properties or generalize to natural- istic contexts. Second, by simulating and then comparing responses under conditions that are derived from linguistic theory (composition vs. single word, or word list), this in silico exper- iment provides results that are easily and explicitly interpretable, unlike encoding models with natural stimuli. However, one major concern raised by this approach is whether the encoding model can capture how the brain responds to the language properties of interest. To address this it is important to verify both that the encoding model is highly effective at predicting brain activity, and that it is sufficiently complex to capture the desired property. Similar in silico experimentation has recently become popular in vision neuroscience. There, DL-based encoding models of the visual system are first trained on ethologically valid tasks like object recognition. Then they are probed to understand brain function (Yamins & DiCarlo, 2016). For example, Bashivan et al. (2019) used DL-models to synthesize images that maximally drive neural responses. This provided a noninvasive in silico technique to control and manipulate internal brain states. Similarly, Ratan Murty et al. (2021) synthesized images from end-to-end DL models trained on brain data to provide stronger evidence for the cate- gorical selectivity of different brain regions. In silico experimentation with explicit computa- tional models has also been used in studies of the medial temporal lobe. In Nayebi et al. (2021), computational models of the medial entorhinal cortex were used to investigate the functional specialization of heterogeneous neurons that do not have stereotypical response profiles. By doing ablation in silico, they found that these types of cells are equally important for downstream processing as are grid- and border-cells. Each of these studies first relied on the encoding model’s ability to generalize to new stimuli. This was an indication that the features learned by the DL-based models encoded similar information to the brain regions that they predicted well. Second, these studies leveraged the predictive ability of encoding models to simulate brain activity in new, controlled conditions as a lens into brain function. This enabled the researchers to explore aspects of brain function that would otherwise be highly data inten- sive or impossible to do. In language, in silico experimentation is a promising area that is under development, bol- stered by the successes in vision neuroscience and growing efforts to understand artificial lan- guage systems. One of its earliest uses is the BOLDpredictions simulation engine (Wehbe et al., 2018, 2021), an online tool that allows the user to simulate language experiments that contrast two conditions, each defined by a list of isolated words. BOLDpredictions relies on an encoding model from a natural listening experiment that predicts brain activity as a function of Neurobiology of Language 8 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u n o / l / l a r t i c e - p d f / d o i / l . / / 1 0 1 1 6 2 n o _ a _ 0 0 1 0 1 2 0 7 4 5 3 9 n o _ a _ 0 0 1 0 1 p d / . l f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 In silico language neuroscience Generalizability testing: Testing to see if effects observed on a particular data set extend to a new data set not used for model estimation. individual word embeddings (Huth et al., 2016). In the following sections, we review in silico adaptations of four different language experiments based on four separate data sets. Each of these in silico experiments uses a single naturalistic experiment to train the encoding models, illustrating how a single data set and experimental approach can provide a flexible test bed for many different hypotheses about natural language. The first experiment uses the BOLDpredic- tions engine to simulate a semantic contrast comparing concrete and abstract words (Binder et al., 2005), testing its generalizability to naturalistic settings. The next experiment focuses on a contrast-based study of composition in two-word phrases (Bemis & Pylkkänen, 2011), testing generalizability over a broader, more diverse stimulus set. The third experiment adopts con- trasts from a group-level study investigating the temporal hierarchy for language processing across cortex by manipulating naturalistic stimuli (Lerner et al., 2011). This simulation checks if effects persist at the individual-level and demonstrates how a successful replication can be used to validate computational model constructs themselves. Finally, the last experiment con- ceptually replicates a study on forgetting behavior in the cortex that also uses controlled manipulations of naturalistic stimuli (Chien & Honey, 2020). This simulation demonstrates the possibility of misinterpretation with the in silico approach, arising from fundamental com- putational differences between neural language models and the human brain. In the experimental simulations described below, voxelwise encoding models were fit to fMRI data collected from a naturalistic speech perception experiment. Participants listened to natural, narrative stories from The Moth Radio Hour (Allison, 2009–) while their whole- brain BOLD responses were recorded (N = 8 for study 1; N = 7 for studies 2 and 3). In each study, encoding models were fit for each voxel in each subject individually using ridge regres- sion. The learned models were then tested on one held-out story that was not used for model estimation, and encoding performance was measured as the linear correlation between pre- dicted and true BOLD responses. Statistical significance of the encoding performance was measured using temporal blockwise permutation tests (p < 0.001, false discovery rate (FDR) corrected; Benjamini & Hochberg, 1995). Finally, in silico analyses were conducted on voxels that were significantly predicted by the encoding model, broadly covering the temporal, pari- etal, and frontal lobes. Semantic contrasts: Wehbe et al. (2018) Binder et al. (2005) investigated the brain regions responsive to abstract and concrete con- cepts. Subjects read individual stimulus strings and pressed one of two buttons to indicate whether each one was a word or a non-word. The study reported that concrete words acti- vated bilateral language regions such as the angular gyrus more than abstract words, and abstract words activated left inferior frontal regions more than concrete words. In total, the authors found 15 cluster peaks. Wehbe et al. (2018) evaluated the reproducibility of these results using an encoding model trained on naturalistic stimuli. They simulated a contrast between the lists of concrete words and abstract words that were kindly shared by Binder et al. (2005). Figure 2 shows the signif- icance map for subject 1 and the group-averaged significance map showing the number of subjects for which the null hypothesis is rejected. The reported regions of interest (ROIs) are shown as an overlay on the flattened cortical maps. Each ROI is originally reported as a single coordinate in brain space and is estimated to have a radius of 10 mm. For every one of the eight subjects, many voxels were significantly more activated for concrete words over abstract words (with p < 0.05, FDR corrected permutation test over the words in each condition), spe- cifically in areas bordering the visual cortex and parts of the inferior frontal gyrus. Some reported ROIs had a high overlap with the significance map (specifically in the angular gyri, Neurobiology of Language 9 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u n o / l / l a r t i c e - p d f / d o i / l / . / 1 0 1 1 6 2 n o _ a _ 0 0 1 0 1 2 0 7 4 5 3 9 n o _ a _ 0 0 1 0 1 p d . / l f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 In silico language neuroscience Figure 2. Generalizability test using BOLDpredictions of the concrete vs. abstract contrast of Binder et al. (2005). The authors compared fMRI activity when subjects processed concrete and abstract words. Wehbe et al. (2016) used the published stimulus to simulate the contrast for each subject and run a permutation test. After MNI space transformation, the number of subjects for which the null hypothesis was rejected is computed at each voxel. The simulated statistical maps are shown on flattened maps and inflated 3D hemispheres. Results for subject 1 are shown in subject 1’s native cortical space. Results for the average of eight subjects are shown in the MNI space. Published ROIs are estimated as 10 mm radius spheres, shown in red hatch on the flatmaps (distortion due to the flattening process). A comparison of the overlap of the reported ROIs and the statistical maps reveals that Wehbe et al. (2016) achieve a relatively high overlap for specific ROIs (in the angular gyri, in the posterior cingulate gyri, the right precuneus, and the middle temporal gyri) and not for others. Therefore, BOLDpredictions predicts that the contrast from Binder et al. (2005) generalizes to naturalistic conditions, to a certain extent. MNI = Montreal Neurological Institute; ROIs = regions of interest. l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u n o / l / l a r t i c e - p d f / d o i / l . / / 1 0 1 1 6 2 n o _ a _ 0 0 1 0 1 2 0 7 4 5 3 9 n o _ a _ 0 0 1 0 1 p d / . l f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 in the posterior cingulate gyri, the right precuneus, and the middle temporal gyri). The signif- icant effect in those ROIs can be considered to be replicated by BOLDpredictions. However, the reported ROIs and the significance map did not always agree, with the effect in some regions being reported only by Binder et al. (2005) or only by Wehbe et al. (2018). There are many possible reasons for non-generalizability of individual reported ROIs, including the stochasticity of brain activity, variations in experimental paradigms and analysis techniques, and lack of reproducibility. The authors of BOLDpredictions (Wehbe et al., 2018, 2021) note that any scientific finding needs to be reproduced in a series of experiments that would create a body of evidence toward this finding, and the in silico experimentation using BOLDpredictions is one additional piece of evidence. The authors also note that expanding the engine to different data sets, models, and so forth will establish the robustness of the in silico effects and help determine if the original contrast-based experiment lacks reproducibility (Wehbe et al., 2018). Semantic composition contrasts: Jain and Huth (2023) In the second in silico experiment, Jain and Huth (2023) simulated and expanded on studies of combinatorial processing in 2-word phrases across cortex, first described in Bemis and Pylkkänen (2011). The original controlled experiment consisted of participants reading two word adjective–noun phrases (“red boat”) and doing a picture matching task while brain responses were recorded using MEG. To contrast this compositional condition, a list control was introduced wherein participants were presented with two-word noun–noun phrases (“cup boat”) along with a non-word control consisting of a non-word and a word (“xkq boat”). Note that participants were instructed to avoid composing meaning in the word list, but no explicit control was introduced. To isolate regions involved in two-word composition, the study con- trasted the adjective–noun condition with the controls. The experimenters tested 25 base nouns, six color adjectives, and six non-words. Overall, they found that areas in ventral medial Neurobiology of Language 10 In silico language neuroscience prefrontal cortex (vmPFC) and left anterior temporal lobe both selectively responded to the composition condition. Jain and Huth (2023) conceptually replicated the original study by building encoding models that approximate every voxel’s response to a naturalistic two-word phrase as a non- linear function of the words in the phrase. For each (overlapping) two-word phrase in the natural language stimuli, features were first extracted from a powerful language model, the generative pretrained transformer (GPT; Radford et al., 2018). Then, voxelwise encoding models were trained to learn a linear function from the phrase features to the elicited response after the second word. Using the encoding models, each voxel’s response to a large corpus of over 147,000 two-word phrases was predicted and ranked. This stimulus set comprised both adjective–noun phrases like “red boat” and noun–noun phrases like “picnic table.” Next, for a given phrase selected by a voxel, the first word was replaced with a non-word (i.e., the word was ablated) and the ablated phrase feature was extracted from GPT. Using the learned encod- ing model, the voxel’s response to the ablated phrase was predicted. Finally, the sensitivity of the voxel to the presence of the ablated word was measured. If the ablated word is important for the voxel to process the phrase, removing it should notably change its response and give high sensitivity. This was done to simulate the compositional versus non-word condition in the original study. The resultant ablation sensitivity of voxels across cortex is visualized in Figure 3. Overall, the in silico experimentation produced similar results to the original study in vmPFC and left anterior temporal lobe—both of these regions exhibit sensitivity to the presence of a l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u n o / l / l a r t i c e - p d f / d o i / l . / / 1 0 1 1 6 2 n o _ a _ 0 0 1 0 1 2 0 7 4 5 3 9 n o _ a _ 0 0 1 0 1 p d . / l f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 In silico adaptation of a study examining compositional processing in two-word phrases. The original study compared the MEG Figure 3. responses of participants with three different types of two-word phrases: adjective–noun, noun–noun and non-word–noun (Bemis & Pylkkänen, 2011). The in silico simulation of the first two conditions was done by constructing a larger diverse corpus of phrases and using LM-based encoding models to predict fMRI voxel responses to each phrase. The non-word–noun condition was simulated by replacing the first word in a phrase with a non-word (i.e., word ablation), extracting new phrase features from the LM, and then predicting the voxel’s response to the ablated phrase. A large change in a voxel’s response upon word ablation indicated its sensitivity to the first word in the phrase and suggested that the voxel relied on the first word to process the phrase. Similar to the original study, the in silico experiment revealed high sensitivity in ventral medial prefrontal cortex and left anterior temporal lobe. However, the experiment also found that several other areas across cortex combined meaning over the two words in a phrase and moreover, captured diverse semantic concepts arising from the composition. fMRI = functional magetic resonance imaging; LM = language model; MEG = magnetoencephalography. Neurobiology of Language 11 In silico language neuroscience compositional word. Beyond areas reported originally, other regions like right inferior parietal and dorsal prefrontal also showed high sensitivity. This finding corroborates other studies of phrase composition (e.g., Boylan et al., 2015; Graves et al., 2010). The in silico study was able to analyze two-word composition in broader regions of cortex by simulating activity for each voxel independently and over a much larger stimulus set that comprises diverse concepts and constructions. While the simulation does not guarantee causal involvement of any region in two-word composition, it demonstrates the utility of broadly sampling stimuli and raises the possibility that many more regions are involved in this process. Moreover, in the in silico study Jain and Huth (2023), this paradigm was extended to much longer phrases (10 words) to understand the relationship between semantic representations and word-level integration across cortex. This would be difficult to implement in real-world settings as doing single-word ablations on increasingly longer phrases is combinatorially explosive. Construction timescale contrast: Vo et al. (2023) In the third in silico experiment, Vo et al. (2023) tested whether voxelwise encoding models based on features from a neural LM can capture the timescale hierarchy observed during human natural language comprehension. In Lerner et al. (2011), subjects listened to a first- person narrative story that was either intact, reversed, or temporally scrambled at the word level, sentence level, or paragraph level. The scrambling manipulations altered the temporal coherence of the natural language stim- ulus, and allowed the researchers to measure the reliability of fMRI responses to each condi- tion using intersubject correlation. This revealed an apparently hierarchical organization of temporal receptive windows, with information over short timescales processed in auditory cor- tex and long timescales processed in parietal and frontal cortex. For the in silico adaptation, the authors trained a multi-timescale long short-term memory (MT-LSTM) network as a lan- guage model. Then they used the features from the MT-LSTM to predict fMRI responses for each voxel using the data set described above. To mimic the manipulations of the original study, they generated 100 scrambled versions of a held-out test story. This enabled the authors to examine the predicted fMRI responses within each voxel in each subject. Rather than mea- suring intersubject reliability, they chose to measure an analogous intrasubject reliability value, testing whether the scrambling condition caused a significant drop in this value across conditions. The authors show through simulations that their metric (based on the variance in the simulated fMRI response) is directly analogous to intersubject correlation measures, which is supported by other work (Blank & Fedorenko, 2017; Hasson et al., 2009). The results of this experiment compared to a schematized version of the original results are shown in Figure 4. This in silico experiment reproduced the pattern of the temporal hierarchy along the temporoparietal axis. It did find that some regions in frontal cortex appear to inte- grate over shorter timescales than the original work, similar to a later replication of the work (Blank & Fedorenko, 2020) and to a different in-silico replication of the experiment that used GPT-2, rather than a MT-LSTM language model (Caucheteux et al., 2021). Furthermore, the fine-grained resolution of the single-voxel analyses revealed substantial variability across sub- jects. Taken together, the in silico results suggest that timescales are not as uniform across broad regions as previously reported. This is in agreement with single-neuron studies that show a heterogeneity of intrinsic timescales within a brain region (Cavanagh et al., 2020). Forgetting timescale contrast: Vo et al. (2023) In the last experiment, the authors used the same MT-LSTM encoding models as experiment 3 to simulate how different brain regions forget information in naturalistic narrative stimuli Neurobiology of Language 12 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u n o / l / l a r t i c e - p d f / d o i / l . / / 1 0 1 1 6 2 n o _ a _ 0 0 1 0 1 2 0 7 4 5 3 9 n o _ a _ 0 0 1 0 1 p d / . l f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 In silico language neuroscience In silico adaptation of a study mapping the hierarchy of temporal receptive windows. (Top) original results adapted from Lerner Figure 4. et al. (2011). The authors played an audio clip of a narrative story, either intact, reversed, or scrambled at different temporal scales. The figure shows an overlay of several intersubject correlation maps, which measured the cross-subject reliability of the fMRI response in each condition. (Bottom) Results from the in silico experiment of temporal receptive windows, shown for every significantly predicted voxel on a single subject. The in silico experiment suggests that temporal processing windows for different brain regions are not as uniform as previously reported. CS = central sulcus; IPS = intrapariental sulcus; LS = lateral sulcus; mPFC = medial prefrontal cortex; TPJ = temporoparietal junction. (Chien & Honey, 2020). While Chien and Honey found that all brain regions forget informa- tion at the same rate (Figure 5A), the in silico results suggested that low-level regions such as auditory cortex forget information at a faster rate than high-level regions like the precuneus (Figure 5B). To better understand this discrepancy, the authors investigated forgetting behavior in the MT-LSTM itself. The results first indicated that every unit in MT-LSTM forgot information at a specific rate tied to its processing timescale (Figure 5C). The authors further hypothesized that the discrepancy could stem from the MT-LSTM’s inability to forget information, even if the preceding context is noisy/uninformative (Figure 5D). To test this, they measured the language model’s cross entropy (lower is better) for a paragraph in three conditions: preceded by the correct paragraph (actual context), preceded by no paragraph (no context) and preceded by random paragraphs in the story (box plot of 100 different incorrect contexts). The story was scrambled by dividing it into non-overlapping chunks of 9, 55, 70, 80, or 200 words or at the actual sentence and paragraph boundaries (hand-split). Overall, smaller differences were observed between the conditions as the scrambled context became longer (increased chunk size) and closer to the intact story. With fixed-size chunks, the model performed better when it had no context than when it had access to incorrect information. In contrast, with actual sentences/paragraphs, the model had better performance with incorrect context than no con- text at all. In both cases, the type of context influences the model performance suggesting that the model retains information from the past. Second, it retains this context even if it is not useful, as in the fixed-chunk conditions. The model could have simply ignored the wrong con- text to perform better but it did not (or was unable to). This highlights the language model’s inability to forget information that is then reflected in the encoding model results. The authors Neurobiology of Language 13 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u n o / l / l a r t i c e - p d f / d o i / l . / / 1 0 1 1 6 2 n o _ a _ 0 0 1 0 1 2 0 7 4 5 3 9 n o _ a _ 0 0 1 0 1 p d / . l Q6 f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 In silico language neuroscience In silico adaptation of a study on forgetting behavior during natural language comprehension. In the original study, Chien and Figure 5. Honey (2020) scrambled paragraphs in a story and analyzed how quickly different brain regions forgot the incorrect context preceding each paragraph. The in silico adaptation used the MT-LSTM based encoding model to predict brain activity at different points in a paragraph when it was preceded by incorrect context. (A) The original study reported that each brain region (denoted by different colored lines) forgot information at a similar rate, despite differences in construction timescales. (B) In contrast, the in silico replication estimated that regions with longer construction timescales also forgot information slowly. (C) Within the MT-LSTM itself, the forgetting rate of different units was related to its attributed timescale. (D) Next, the MT-LSTM’s language modeling abilities were tested on shuffled sentences or paragraphs. The DL model achieved better performance at next-word prediction by using the incoherent, shuffled context as opposed to no context at all. This shows that the DL model retains the incoherent information, possibly because it helps with the original language modeling task it was trained on or because the model has no explicit mechanism to flush-out information when context changes (at sentence/paragraph boundary). The com- putational model’s forgetting behavior thus differs from the brain, revealing specific flaws in the in silico model that could be improved in future versions, such as a modified MT-LSTM. DL = deep learning; MT-LSTM = multi-timescale long short-term memory; rSI = correlation between scrambled and intact chunks. hypothesized that with hand-split sentences and paragraphs, the incorrect context still pro- vides relevant information to predict the next word, leading to better performance than no context at all. DISCUSSION Advantages of the in Silico Experimental Design In the following sections, we discuss the advantages of using in silico experimental design with DL-based encoding models and its potential impact on language neuroscience. Hypothesis development and testing Each of the studies above conceptually replicated controlled analyses of different linguistic properties using voxelwise encoding models fit on a single naturalistic fMRI data set. Overall, the studies reproduced several effects reported in the original experiments. Interestingly, how- ever, the in silico experiments also found new effects that had not been explored originally. For example, the first experiment suggested that regions in inferior frontal gyrus were more active for concrete words than abstract words. In the second experiment, the investigation of com- position in phrases was expanded to a much larger stimulus set and longer phrases. This cor- roborates earlier results, but the in silico paradigm also enables experimenters to explore Neurobiology of Language 14 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u n o / l / l a r t i c e - p d f / d o i / l / / . 1 0 1 1 6 2 n o _ a _ 0 0 1 0 1 2 0 7 4 5 3 9 n o _ a _ 0 0 1 0 1 p d . / l f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 In silico language neuroscience interactions between the effect of interest (here, linguistic composition) and other important linguistic properties, such as semantic category. The third experiment found more diversity in timescales in regions like prefrontal cortex than previously reported, closely matching more recent studies of timescale distribution across cortex (Blank & Fedorenko, 2020; Caucheteux et al., 2021) and single-neuron studies (Cavanagh et al., 2020). This demonstrates how the in silico experimental design can be used not only to reproduce and test the generalizability of controlled studies, but also to conduct large-scale exploratory and data-driven analyses that can reveal new aspects of brain function. Beyond these examples, it is possible to test new hypotheses using in silico experiments before collecting data for a controlled experiment. Wehbe et al. (2016) showcase how BOLD- predictions and in silico experimentation can be used to design new experiments. While the in silico results may not precisely match the eventual data collected on a human population, they would reveal areas where the underlying DL model has failed to match human-level process- ing and present possible areas of improvement. This has the potential to advance our under- standing of both neural network language models and biological language processing. In particular, the in silico paradigm can both draw upon large-scale multidisciplinary efforts to build tools and methods for interpreting neural network language models (Ettinger et al., 2018; Hewitt & Manning, 2019; Ravfogel et al., 2020; Sundararajan et al., 2017), as well as con- tribute to them by providing a human neural benchmark. Furthermore, more interpretable models allow for novel causal intervention experiments that perturb and control ANNs in ways that biological neural networks cannot be perturbed (Zhang et al., 2022). Testing for generalizability of effects and experimental construct validity One way to ensure the observed effects of the in silico experiments are not due to the specific task design is to test the generalizability of effects across model architectures, training tasks, neuroimaging data sets and modalities. Unlike reproducibility tests in traditional neuroimaging experiments, these tests do not rely on laborious and time-consuming data collection. More- over, there are increasingly more tools and techniques to interpret DL models (Clark et al., 2019; Ettinger, 2020), and we can target investigations to these. For example, in the forgetting experiment, the authors checked how the model represented the cognitive process itself. We note that some drawbacks of DL models persist across architectures and tasks. For instance, current language models still perform poorly on common sense reasoning and struggle with capturing long-term dependencies in language. However, with technological advancements, the types of inferences we can make with the in silico paradigm will greatly improve. A case in point is the modeling of contextual processing in the brain. Until recently, language encoding models were largely restricted to static word embeddings that made it difficult to analyze how the brain processed word sequences. However, with the advent of neural language models, this has changed dramatically. In vision neuroscience, the functional profile of the fusiform face area was established through contrast experiments that evolved over a long period of time (Anzellotti et al., 2014; Gauthier et al., 2000; Kanwisher et al., 1997; Liu et al., 2010; Sergent et al., 1992; Suzanne Scherf et al., 2008; Xu, 2005). Each new experiment was designed to address a con- found that was not accounted for previously. Today, however, in silico experiments with vision models have enabled neuroscientists to efficiently contrast large, diverse sets of stimuli and establish the functional specificity of different regions (Ratan Murty et al., 2021). Similarly, in language neuroscience, encoding models have been used to evaluate the semantic selec- tivity of many regions going beyond semantic contrasts that are tested for a handful of condi- tions at a time (Huth et al., 2016; Mitchell et al., 2008). This demonstrates how the in silico Neurobiology of Language 15 Q7 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u n o / l / l a r t i c e - p d f / d o i / l / . / 1 0 1 1 6 2 n o _ a _ 0 0 1 0 1 2 0 7 4 5 3 9 n o _ a _ 0 0 1 0 1 p d / . l f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 In silico language neuroscience Construct validity: Determinination whether a theoretical, experimental, or computational construct faithfully reflects the true phenomena. paradigm allows scientists to quickly design and test multiple experiments that get at the same underlying question. This means that in silico experiments, despite using similar manipulations to controlled experiments, can provide an additional way to test the construct validity of the experiment (Yarkoni, 2022). When coupled with generalizability testing, we run a lower risk of over-claiming or over-generalizing. Establishing the validity of model constructs The in silico approach uniquely facilitates experimenters to evaluate and improve the design of computational models based on observed in silico behavior, going beyond good prediction performance. For example, in the forgetting experiment, the authors identified that the MT-LSTM language model does not instantaneously switch context, and this could influence the observed effects. One possible solution to the nonreplication would be to then train the language model on a new task that encourages forgetting. Alternatively, it could prompt the need for designing alternate architectures that have a built-in forgetting mechanism closer to observed human behavior. Artificial neural networks can be investigated through causal inter- vention experiments and perturbations, whereas it is very difficult to impossible to do this for human language systems. By analyzing the behavior of DL models in many in silico experi- ments, we can create a check-and-correct cycle to build better computational models of the brain and establish the validity of model constructs. An analogous paradigm has also risen in popularity in NLP. Moving beyond better perfor- mance with larger language models, there has been a growing effort toward curating diverse language tasks like syntactic reasoning and multistep inference to understand the limitations of current models and establish a benchmark for future innovation. In the same vein, we believe that many different in silico experiments can be used together to establish the validity of dif- ferent model constructs and provide a benchmark to test future innovations in computational language modeling. We hope that this pushes the field past solely testing encoding model performance on different architectures. Ultimately, this paradigm is a bridge between compu- tational models and experimental designs in neuroscience, such that we can make joint infer- ences on both and improve them in tandem. l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u n o / l / l a r t i c e - p d f / d o i / l / . / 1 0 1 1 6 2 n o _ a _ 0 0 1 0 1 2 0 7 4 5 3 9 n o _ a _ 0 0 1 0 1 p d . / l f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 Preserving individual participant differences One potential advantage of in silico encoding model experiments is that the models are typically estimated with single-subject data, allowing experiments to test for effects in individuals rather than over a group average. While group averaging is a common method to improve the signal- to noise ratio (SNR) of neuroimaging studies, it can lead to an underestimation of effect size (Fedorenko, 2021; Handwerker et al., 2004) and hide effects that can be seen with more fine- grained functional anatomical data (Popham et al., 2021). Finally, individual participant analysis does not preclude the estimation of how prevalent an effect is at the population level (Ince et al., 2021); however, it does enable experimenters to account for individual differences, which can be critical to establish links between brain and behavior (Hedge et al., 2018). Consequently, there has been a rising trend toward language studies that analyze participants individually and report consistency of effects across the group (Blank & Fedorenko, 2020; Huth et al., 2016; Wehbe, Vaswani, et al., 2014). While this requires the experimenter to collect more samples per subject to improve the SNR, this approach does not make assumptions about anatomical alignment and preserves spatial resolution important for inferring brain function. The improved sensitivity provides better control for Type 1 errors (by allowing the experimenter to see which effects replicate across participants) and Type 2 errors (by allowing a flexible mapping that can identify important regions in each participant, even if they do not match perfectly in anatomy). Neurobiology of Language 16 In silico language neuroscience However, the individual-participant analytic approach raises important questions about how to isolate functionally consistent regions across participants and infer consistency of effects. One solution is to use a common set of functional localizer stimuli across participants to isolate functionally homologous networks. For example, the auditory and visual language localizers developed by Fedorenko et al. (2010) and T. L. Scott et al. (2017) have been shown to robustly identify regions across cortex that are important for language processing. This approach enables future studies to consistently isolate language processing regions and char- acterize their function. Modeling approaches such as hyperalignment (Haxby et al., 2020) and probabilistic mapping of the cortical surface (Huth et al., 2015) offer solutions to compute group-level maps from functional data of individual participants. Nevertheless, these approaches do not provide a computational framework to model individual-participant effects. Encoding models, on the other hand, learn a different function for each brain element in each subject. This enables them to effectively model individual participants and retain high spatial resolution. Improving reproducibility in language neuroscience There has been an increasing concern in the sciences about the lack of reproducibility for many types of experiments (Pashler & Harris, 2012; Simmons et al., 2011), a problem to which neuroscience is not immune. Several papers have discussed the prevalence of analysis vari- ability, software errors, nontransparent reporting of methods, and lack of data/code sharing as primary causes for low reproducibility and generalizability in neuroscience (see Barch & Yarkoni, 2013, and Poldrack et al., 2020, for introductions to special issues on reproducibility in neuroimaging; Button et al., 2013; Evans, 2017). These studies have also identified issues in statistical analyses, like low statistical power (and, consequently, inflated effect sizes), HARK- ing, and p-hacking. We believe that the in silico experimentation paradigm can help alleviate some of these issues by providing access to and encouraging open tools for scientific research. When combined with open access to naturalistic data, preprocessing methods, and analysis code, the in silico paradigm can enable scientists to use a standard setup as they test a variety of different hypotheses and thus reduce the “researcher degrees of freedom.” Platforms such as BOLDpredictions can help with this. Indeed, BOLDpredictions is intended as a community tool to allow easy in silico experimentation and generalization testing. It is intended to allow other researchers to contribute their encoding models for other experiments (even outside of language) so that in silico experiments can be available to all. Furthermore, competitions such as Brain-Score (https://www.brain-score.org/competition/) and the SANS’22 Naturalistic fMRI data analysis challenge (https://compsan.org/sans_data_competition/content/intro.html) can align scientific work toward a common goal and facilitate verifiability. Since naturalistic exper- iments broadly sample the stimulus space, the in silico paradigm can also act as a test bed for generalizability. Caveats and the Possibility of Overinterpretation The in silico paradigm leverages the advantages of both controlled and naturalistic experimen- tal designs with DL-based encoding models. However, it is important to recognize the caveats of this approach so as to minimize the risk of overinterpretation. Here we discuss a number of potential issues. Limitations in the natural language stimulus One critical advantage of naturalistic stimuli over controlled designs is the access to many diverse examples of language use. However, this also means that the experimenter has little Neurobiology of Language 17 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u n o / l / l a r t i c e - p d f / d o i / l . / / 1 0 1 1 6 2 n o _ a _ 0 0 1 0 1 2 0 7 4 5 3 9 n o _ a _ 0 0 1 0 1 p d / . l f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 In silico language neuroscience Fine-tuning: A secondary learning procedure that modifies an already trained artificial neural network to adapt to a new task or data set. control over the rate of occurrence of different types of linguistic features. Word frequency is an example of uncontrolled variation in natural stimuli (e.g., high frequency of words describe everyday objects like “table” and “book” as opposed to low-frequency words like “artillery” and “democracy”). This presents an important challenge in naturalistic paradigms as the rare variables will have low power and could lead to incorrect or incomplete inferences of brain function. For example, if a voxel encodes semantic concepts related to politics and gover- nance, but this category is not well represented in the naturalistic stimuli, the experimenter runs the risk of incorrectly inferring the voxel function. This can be addressed by building larger, freely available data sets collected from diverse sources and encouraging replication of effects on them. Another issue with naturalistic paradigms is that they currently rely on the passive percep- tion of language. Many studies have shown that turn-taking in conversation is an important, universal aspect of communication and has implications on how we learn, understand and generate language (Levinson, 2016). Despite a rising trend toward stimuli that take into account social and contextual information, we are still far from studying truly natural use of language with neuroimaging. Some work has investigated aspects of conversational commu- nication (Bögels et al., 2015; Gisladottir et al., 2015; Magyari et al., 2014; Sievers et al., 2020), but the field is still behind in modeling these effects with encoding models or ANNs. Richer data sets will be key to developing these approaches, such as the real-world communication data collected in Bevilacqua et al. (2019) or multimodal movie stimuli discussed in Redcay and Moraczewski (2020). This is an important future direction for the naturalistic paradigm to understand the brain mechanisms of language processing in ethological settings. Limitations in the DL-based feature space Perhaps the most important factor guiding the in silico experimental design is the success of DL models at predicting brain activity. This paradigm allows neuroscientists to inspect brain function by conducting simulations on the computational model instead, which is easier to perturb, interpret, and control. However, this also means that the types of effects we can observe are limited by the capabilities of the DL model. For example, the forgetting experiment by Vo et al. (2023) demonstrates how the computational model has different behavior than the human brain, affecting the observed in silico behavior. Domain shift presents another common issue for neural networks, although recent studies has proposed that fine-tuning on the target domain/task (Radford et al., 2018) and dynamic data selection during training (Aharoni & Goldberg, 2020; van der Wees et al., 2017) can greatly alleviate this problem for language models. Several encoding model studies explicitly fine-tuned the language model to operate in set (Jain et al., 2020) or trained the language model on a corpus specifically curated to resemble the experimental task (Jain & Huth, 2018; Wehbe, Vaswani, et al., 2014). Further- more, while ANNs like language models have been successfully employed for a wide range of tasks, their syntactic, common sense, and logical reasoning abilities are still far from those of humans (Ettinger, 2020; Linzen, 2020; Pandia & Ettinger, 2021; Wilcox et al., 2021). Overall, it is important to note that building good encoding models of brain activity and understanding brain function with the in silico paradigm are both contingent on better artificial models of language processing. Limitations in computational modeling Another source of confounds in encoding models and the in silico paradigm is incorrect modeling assumptions. For example, Jain et al. (2020) highlight that many fMRI encoding models rely on a downsampling technique that incorrectly transforms slowly varying features, Neurobiology of Language 18 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u n o / l / l a r t i c e - p d f / d o i / l / . / 1 0 1 1 6 2 n o _ a _ 0 0 1 0 1 2 0 7 4 5 3 9 n o _ a _ 0 0 1 0 1 p d / . l f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 In silico language neuroscience Zone generalization: Determination whether two brain regions process stimuli similarly. making them highly correlated with local word rate. Consequently, an experimenter may (incorrectly) conclude that a brain region that is well predicted by the downsampled features is selective for the slowly varying information (e.g., discourse) it captures, when, in fact, the brain region merely responds to the rate of words. In other cases, it may be important to model several different sources of noise, which has been pursued in other work simulating fMRI data (Ellis et al., 2020). Current neuroimaging modalities also have low SNR, limiting the predictive performance of computational models. Because all modeling approaches likely have caveats and edge cases for which their assumptions fail, it is important to clearly articulate and discuss these issues in future work. Inappropriate causality and zone generalization Unlike contrast-based experiments and encoding models with simple interpretable features like indicator variables, DL-based encoding models rely on ANNs that are themselves hard to interpret. To this end, any significant correlation observed between brain activity and model predictions leaves many possibilities for interpretation. An experimenter may conclude that the task or objective function the DL model was trained on closely resembles a task the brain solves, when this may not be the case. For example, one might falsely infer that the brain does predictive coding for language because it is well predicted by features from a language model that is trained to predict the next word in a sequence. Guest and Martin (2023) elaborate on this issue by discussing the logical fallacies in inferences drawn between brain behavior or activity, and DL models of language. Specifically, they highlight that studies analyzing paral- lels between the brain and computational models of language often attribute inappropriate causality by assuming that predictive ability is sufficient to claim task similarity or model equivalence. On the contrary, the direction of causality should be that if an artificial model closely resembles the brain, it can mimic brain behavior and activity, or that a lack of predic- tion abilities clearly indicates a lack of model equivalence. This is a pertinent issue for in silico experimentation as the paradigm uses computational models of language processing in the brain to simulate its behavior. However, it is important to note that in all of the in silico exam- ples presented here, the authors were using the generalizability of the encoding models to predict brain responses in different conditions. This only suggests that the encoding models can effectively capture the brain’s behavior for language tasks but is not a sufficient account to conclude model equivalence. Another issue with logical inference in DL-based encoding models relates to the functional equivalence of two brain regions that are both well predicted by a given feature space. In their recent study, Toneva, Williams, et al. (2022) discuss this issue in detail for language encoding models and provide a computational framework to analyze the extent to which brain regions share computational mechanisms solely based on their encoding performance. The three main sources of confounds—naturalistic stimuli, DL-based feature spaces, and modeling assumptions—can intersect in interesting ways and raise the probability of incorrect interpretation. False causality stemming from spurious correlations are a problem for in silico experiments, much like controlled experiments. To this end, it is important to emphasize trans- parent analysis methods, better interpretability tools for DL models, and rigorous tests of repro- ducibility with diverse data sources. Although reproducibility is traditionally viewed as the replication of effects across partici- pant pools/data sets, with in silico experimentation we can add another layer of replicability, across different models that have different intrinsic biases (architectures, training objectives, etc.), and learn different types of representation. Neurobiology of Language 19 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u n o / l / l a r t i c e - p d f / d o i / l / / . 1 0 1 1 6 2 n o _ a _ 0 0 1 0 1 2 0 7 4 5 3 9 n o _ a _ 0 0 1 0 1 p d / . l f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 In silico language neuroscience Important Factors of Consideration Before doing in silico experimentation, one important consideration is determining if the encoding model is “good enough.” While there is no quantitative threshold above which a model can be considered suitable, we suggest the following. Statistical significance of encoding model performance on a diverse, held-out test set It is imperative that experimenters test whether encoding model performance is statistically significant at the individual brain element level. Any in silico experimentation should only be done on brain elements that are significantly predicted by the computational model. A well-established approach in language encoding modes is to correlate the true and predicted responses for a held-out test set. Following this, a permutation test can be done to check if the correlation is statistically significant. We also emphasize the importance of using diverse test sets to effectively gauge generali- zation. If a brain element is selective for features that are not present in the test set, then it may falsely be labeled as poorly predicted. One feasible solution is to use a leave-one-out testing procedure. This can be done by fitting an ensemble of encoding models, each of which excludes one unique set of training data in the model estimation. Statistical significance can then be measured for encoding model predictions on all held-out data. This procedure increases diversity in the test set and improves statistical power. l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u n o / l / l a r t i c e - p d f / d o i / l . / / 1 0 1 1 6 2 n o _ a _ 0 0 1 0 1 2 0 7 4 5 3 9 n o _ a _ 0 0 1 0 1 p d / . l f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 Feature-space selection Given the diversity of function across the brain, it is possible that no one feature space or computational model best predicts all brain regions. Thus, experimenters should test several different features spaces and models (Nunez-Elizalde et al., 2019) and individually choose the one with best held-out set performance for each brain element. This is especially important for DL models as different neural language models or their layers predict different brain regions well. In this case, we would use the neural language model (layer) that best predicts held-out stories for each element and, further, passes construct validity tests (ie., has well-understood behavior to the controlled manipulation). For example, in the in silico semantic composition experiment, Jain and Huth (2023) found that the lower layers of the neural language model were generally indifferent to ablating words farther in the past (Khandelwal et al., 2018). Con- sequently, these layers cannot be used to conduct the ablation study, as they do not respond to the manipulation in the first place. Interpreting the DL-models To establish the validity of computational model constructs, we suggested the use of interpret- ability tools and techniques to understand how the DL-model itself represents a cognitive pro- cess. This would allow the experimenter to directly investigate sources of confounds. It is also important to consider the types of questions the in silico paradigm is most suited to answer. As demonstrated here, this paradigm can be used to estimate functional properties in the brain, such as selectivity to different word categories or the processing timescale. It cannot, however, be used to test the causal involvement of a brain area or the exact computational mechanism. For example, many regions in the experiments above are shown to capture semantic properties in language. Whether these regions play a causal role in semantic tasks, can only be determined by an in vivo measurement. Neurobiology of Language 20 In silico language neuroscience Conclusion and Future Directions In this article, we highlight the promises of in silico experimentation and detail how it brings together the advantages of controlled experiments and naturalistic experiments paired with encoding models. We showcase four different in silico experiments that all rely on naturalistic language experiments to simulate four different previous studies. We survey the advantages and potential caveats of in silico experimentation and highlight how it can take advantage of recent work in DL to simulate experiments with diverse types of language stimuli. Current work on DL-based encoding models for language is largely restricted to self- supervised models. This is expected since self-supervised models have been trained on large amounts of data and consequently learn highly useful and transferable linguistic representa- tions. However, it remains to be seen if task-based experimental designs in neuroscience can be simulated and adapted with more goal-directed artificial language networks. Additionally, it is also important to investigate and characterize which types of neuroscientific results can be explored with self-supervised models and what aspects of language meaning are beyond the scope of the next-word-prediction objective. Lastly, DL-based language encoding models rely on feature extraction from language or speech ANNs (linearizing transform) and learn a linear function atop the features. We believe that the in silico paradigm can become more powerful if language encoding models directly update the parameters of the ANN itself, resulting in an end-to-end system. While this has been popularized in vision (e.g., Bashivan et al., 2019), it is yet to be explored for language. This approach can potentially introduce diversity into the computational mechanisms of the ANNs, such as recurrence, linear readout from a memory store, and so forth, to integrate pro- cessing in different brain structures (hippocampus, cortex, etc.). This could allow us to under- stand parallel mechanisms like linguistic function, working memory access, and attention using this same approach. ACKNOWLEDGMENTS Research reported in this article was also supported by the National Institute on Deafness and Other Communication Disorders of the National Institutes of Health as part of the Collabora- tive Research in Computational Neuroscience (CRCNS) program. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. We thank Nicole Beckage and Javier Turek for useful discussions on this work. FUNDING INFORMATION Alexander Shailee Jain, Foundations of Language Fellowship, William Orr Dingwall Foundation. G. Huth, Burroughs Wellcome Fund (https://dx.doi.org/10.13039/100000861). Alexander G. Huth, Intel Corporation (https://dx.doi.org/10.13039/100002418). Alexander G. Huth, National Institute on Deafness and Other Communication Disorders https://dx.doi.org/10.13039 /100000055), Award ID: R01DC020088. Leila Wehbe, National Institute on Deafness and Other Communication Disorders https://dx.doi.org/10.13039/100000055), Award ID: R01DC020088. AUTHOR CONTRIBUTIONS Shailee Jain: Conceptualization: Lead; Writing – original draft: Lead; Writing – review & editing: Lead. Vy A. Vo: Conceptualization: Supporting; Writing – original draft: Supporting; Writing – review & editing: Supporting. Leila Wehbe: Conceptualization: Supporting; Funding Neurobiology of Language 21 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u n o / l / l a r t i c e - p d f / d o i / l . / / 1 0 1 1 6 2 n o _ a _ 0 0 1 0 1 2 0 7 4 5 3 9 n o _ a _ 0 0 1 0 1 p d . / l f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 Q8 In silico language neuroscience acquisition: Equal; Supervision: Supporting; Writing – original draft: Supporting; Writing – review & editing: Supporting. Alexander G. Huth: Conceptualization: Supporting; Funding acquisition: Equal; Supervision: Lead; Writing – original draft: Supporting; Writing – review & editing: Supporting. REFERENCES Abnar, S., Beinborn, L., Choenni, R., & Zuidema, W. (2019). Black- box meets blackbox: Representational similarity & stability anal- ysis of neural language models and brains. In Proceedings of the 2019 ACL workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP (pp. 191–203). Association for Compu- tational Linguistics. https://doi.org/10.18653/v1/ W19-4820 Aharoni, R., & Goldberg, Y. (2020). Unsupervised domain clusters in pretrained language models. In Proceedings of the 58th annual meeting of the Association for Computational Linguistics (pp. 7747–7763). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.692 Allison, J. (Producer). (2009–). The moth radio hour [Radio pro- gram]. The Moth; Atlantic Public Media/PRX. Anderson, A. J., Kiela, D., Binder, J. R., Fernandino, L., Humphries, C. J., Conant, L. L., Raizada, R. D. S., Grimm, S., & Lalor, E. C. (2021). Deep artificial neural networks reveal a distributed corti- cal network encoding propositional sentence-level meaning. Journal of Neuroscience, 41(18), 4100–4119. https://doi.org/10 .1523/JNEUROSCI.1152-20.2021, PubMed: 33753548 Antonello, R., & Huth, A. (2022). Predictive coding or just feature discovery? An alternative account of why language models fit brain data. Neurobiology of Language, 1–16. https://doi.org/10 .1162/nol_a_00087 Antonello, R., Turek, J. S., Vo, V. A., & Huth, A. (2021). Low- dimensional structure in the space of language representations is reflected in brain responses. In A. Beygelzimer, Y. Dauphin, P. Liang, & J. W. Vaughan (Eds.), Advances in neural information processing systems. NeurIPS. https://openreview.net/forum?id =UYI6Sk_3Nox Anzellotti, S., Fairhall, S. L., & Caramazza, A. (2014). Decoding representations of face identity that are tolerant to rotation. Cere- bral Cortex, 24(8), 1988–1995. https://doi.org/10.1093/cercor /bht046, PubMed: 23463339 Aurnhammer, C., & Frank, S. L. (2018). Comparing gated and sim- ple recurrent neural network architectures as models of human sentence processing. PsyArXiv. https://doi.org/10.31234/osf.io /wec74 Barch, D. M., & Yarkoni, T. (2013). Introduction to the special issue on reliability and replication in cognitive and affective neurosci- ence research. Cognitive, Affective, & Behavioral Neuroscience, 13(4), 687–689. https://doi.org/10.3758/s13415-013-0201-7, PubMed: 23922199 Bashivan, P., Kar, K., & DiCarlo, J. J. (2019). Neural population control via deep image synthesis. Science, 364(6439), Article eaav9436. https://doi.org/10.1126/science.aav9436, PubMed: 31048462 Bemis, D. K., & Pylkkänen, L. (2011). Simple composition: A mag- netoencephalography investigation into the comprehension of minimal linguistic phrases. Journal of Neuroscience, 31(8), 2801–2814. https://doi.org/10.1523/JNEUROSCI.5003-10.2011, PubMed: 21414902 Bender, E. M., & Koller, A. (2020). Climbing towards NLU: On meaning, form, and understanding in the age of data. In Proceedings of the 58th annual meeting of the Association for Computational Linguistics (pp. 5185–5198). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl -main.463 Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discov- ery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society B (Methodological), 57(1), 289–300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x Bevilacqua, D., Davidesco, I., Wan, L., Chaloner, K., Rowland, J., Ding, M., Poeppel, D., & Dikker, S. (2019). Brain-to-brain syn- chrony and learning outcomes vary by student–teacher dynamics: Evidence from a real-world classroom electroencephalography study. Journal of Cognitive Neuroscience, 31(3), 401–411. https:// doi.org/10.1162/jocn_a_01274, PubMed: 29708820 Bhattasali, S., Brennan, J., Luh, W.-M., Franzluebbers, B., & Hale, J. (2020). The Alice datasets: fMRI & EEG observations of natural language comprehension. In Proceedings of the 12th language resources and evaluation conference (pp. 120–125). European Language Resources Association. https://aclanthology.org/2020 .lrec-1.15 Bhattasali, S., Fabre, M., Luh, W.-M., Al Saied, H., Constant, M., Pallier, C., Brennan, J. R., Spreng, R. N., & Hale, J. (2019). Loca- lising memory retrieval and syntactic composition: An fMRI study of naturalistic language comprehension. Language, Cognition and Neuroscience, 34(4), 491–510. https://doi.org/10.1080 /23273798.2018.1518533 Binder, J. R., Desai, R. H., Graves, W. W., & Conant, L. L. (2009). Where is the semantic system? A critical review and meta- analysis of 120 functional neuroimaging studies. Cerebral Cortex, 19(12), 2767–2796. https://doi.org/10.1093/cercor/ bhp055, PubMed: 19329570 Binder, J. R., Westbury, C. F., McKiernan, K. A., Possing, E. T., & Medler, D. A. (2005). Distinct brain systems for processing con- crete and abstract concepts. Journal of Cognitive Neuroscience, 17(6), 905–917. https://doi.org/10.1162/0898929054021102, PubMed: 16021798 Blank, I. A., & Fedorenko, E. (2017). Domain-general brain regions do not track linguistic input as closely as language-selective regions. Journal of Neuroscience, 37(41), 9999–10011. https://doi.org/10 .1523/JNEUROSCI.3642-16.2017, PubMed: 28871034 Blank, I. A., & Fedorenko, E. (2020). No evidence for differences among language regions in their temporal receptive windows. NeuroImage, 219, Article 116925. https://doi.org/10.1016/j .neuroimage.2020.116925, PubMed: 32407994 Bögels, S., Magyari, L., & Levinson, S. C. (2015). Neural signatures of response planning occur midway through an incoming ques- tion in conversation. Scientific Reports, 5(1), Article 12881. https://doi.org/10.1038/srep12881, PubMed: 26242909 Boylan, C., Trueswell, J. C., & Thompson-Schill, S. L. (2015). Compositionality and the angular gyrus: A multi-voxel similarity analysis of the semantic composition of nouns and verbs. Neuropsy- chologia, 78, 130–141. https://doi.org/10.1016/j.neuropsychologia .2015.10.007, PubMed: 26454087 Neurobiology of Language 22 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u n o / l / l a r t i c e - p d f / d o i / l / / . 1 0 1 1 6 2 n o _ a _ 0 0 1 0 1 2 0 7 4 5 3 9 n o _ a _ 0 0 1 0 1 p d . / l f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 In silico language neuroscience Brennan, J., Nir, Y., Hasson, U., Malach, R., Heeger, D. J., & Pylkkänen, L. (2012). Syntactic structure building in the anterior temporal lobe during natural story listening. Brain and Language, 120(2), 163–173. https://doi.org/10.1016/j.bandl.2010.04.002, PubMed: 20472279 Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365–376. https://doi.org /10.1038/nrn3475, PubMed: 23571845 Caucheteux, C., Gramfort, A., & King, J.-R. (2021). Model-based analysis of brain activity reveals the hierarchy of language in 305 subjects. In Findings of the Association for Computational Linguisitcs: EMNLP 2021 (pp. 3635–3644). Association for Computational Linguistics. https:// hal.archives-ouvertes.fr/ hal -03361430. https://doi.org/10.18653/v1/2021.findings-emnlp .308 Caucheteux, C., & King, J.-R. (2022). Brains and algorithms partially converge in natural language processing. Communications Biology, 5(1), Article 134. https://doi.org/10.1038/s42003-022 -03036-1, PubMed: 35173264 Cavanagh, S. E., Hunt, L. T., & Kennerley, S. W. (2020). A diversity of intrinsic timescales underlie neural computations. Frontiers in Neural Circuits, 14, 615626. https://doi.org/10.3389/fncir.2020 .615626, PubMed: 33408616 Chan, A. H. D., Luke, K.-K., Li, P., Yip, V., Li, G., Weekes, B., & Tan, L. H. (2008). Neural correlates of nouns and verbs in early bilinguals. Annals of the New York Academy of Sciences, 1145(1), 30–40. https://doi.org/10.1196/annals.1416.000, PubMed: 19076387 Chang, E. F., Rieger, J. W., Johnson, K., Berger, M. S., Barbaro, N. M., & Knight, R. T. (2010). Categorical speech representation in human superior temporal gyrus. Nature Neuroscience, 13(11), 1428–1432. https://doi.org/10.1038/nn.2641, PubMed: 20890293 Chen, J., Leong, Y. C., Honey, C. J., Yong, C. H., Norman, K. A., & Hasson, U. (2017). Shared memories reveal shared structure in neural activity across individuals. Nature Neuroscience, 20(1), 115–125. https://doi.org/10.1038/nn.4450, PubMed: 27918531 Chen, Z., Chen, S., Wu, Y., Qian, Y., Wang, C., Liu, S., Qian, Y., & Zeng, M. (2022). Large-scale self-supervised speech representa- tion learning for automatic speaker verification. In ICASSP 2022 —IEEE international conference on acoustics, speech and signal processing (ICASSP) (pp. 6147–6151). IEEE. https://doi.org/10 .1109/ICASSP43922.2022.9747814 Chien, H.-Y. S., & Honey, C. J. (2020). Constructing and forgetting temporal context in the human cerebral cortex. Neuron, 106(4), 675–686. https://doi.org/10.1016/j.neuron.2020.02.013, PubMed: 32164874 Clark, K., Khandelwal, U., Levy, O., & Manning, C. D. (2019). What does BERT look at? An analysis of BERT’s attention. In Pro- ceedings of the 2019 ACL workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP (pp. 276–286). Associ- ation for Computational Linguistics. https://doi.org/10.18653/v1 / W19-4828 Conneau, A., Kruszewski, G., Lample, G., Barrault, L., & Baroni, M. (2018). What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th annual meeting of the Association for Computational Linguistics ( Volume 1: Long papers) (pp. 2126–2136). Associa- tion for Computational Linguistics. https://doi.org/10.18653/v1 /P18-1198 Çukur, T., Nishimoto, S., Huth, A. G., & Gallant, J. L. (2013). Atten- tion during natural vision warps semantic representation across the human brain. Nature Neuroscience, 16(6), 763–770. https:// doi.org/10.1038/nn.3381, PubMed: 23603707 de Heer, W. A., Huth, A. G., Griffiths, T. L., Gallant, J. L., & Theunissen, F. E. (2017). The hierarchical cortical organization of human speech processing. Journal of Neuroscience, 37(27), 6539–6557. https://doi.org/10.1523/JNEUROSCI.3267-16.2017, PubMed: 28588065 Deniz, F., Nunez-Elizalde, A. O., Huth, A. G., & Gallant, J. L. (2019). The representation of semantic information across human cerebral cortex during listening versus reading is invariant to stimulus modality. Journal of Neuroscience, 39(39), 7722–7736. https://doi.org/10.1523/ JNEUROSCI.0675-19.2019, PubMed: 31427396 Deniz, F., Tseng, C., Wehbe, L., & Gallant, J. L. (2021). Semantic representations during language comprehension are affected by context. bioRxiv. https://doi.org/10.1101/2021.12.15.472839 Ellis, C. T., Baldassano, C., Schapiro, A. C., Cai, M. B., & Cohen, J. D. (2020). Facilitating open-science with realistic fMRI simula- tion: Validation and application. PeerJ, 8, Article e8564. https:// doi.org/10.7717/peerj.8564, PubMed: 32117629 Ettinger, A. (2020). What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8, 34–48. https:// doi.org/10.1162/tacl_a_00298 Ettinger, A., Elgohary, A., Phillips, C., & Resnik, P. (2018). Assessing composition in sentence vector representations. In Proceedings of the 27th international conference on computational linguistics (pp. 1790–1801). Association for Computational Linguistics. https://aclanthology.org/C18-1152 Evans, S. (2017). What has replication ever done for us? Insights from neuroimaging of speech perception. Frontiers in Human Neuroscience, 11, 41. https://doi.org/10.3389/fnhum.2017.00041, PubMed: 28203154 Fedorenko, E. (2021). The early origins and the growing popularity of the individual-subject analytic approach in human neurosci- ence. Current Opinion in Behavioral Sciences, 40, 105–112. https://doi.org/10.1016/j.cobeha.2021.02.023 Fedorenko, E., Hsieh, P.-J., Nieto-Castañón, A., Whitfield-Gabrieli, S., & Kanwisher, N. (2010). New method for fMRI investigations of language: Defining ROIs functionally in individual subjects. Journal of Neurophysiology, 104(2), 1177–1194. https://doi.org /10.1152/jn.00032.2010, PubMed: 20410363 Friederici, A. D., Opitz, B., & von Cramon, D. Y. (2000). Segregat- ing semantic and syntactic aspects of processing in the human brain: An fMRI investigation of different word types. Cerebral Cortex, 10(7), 698–705. https://doi.org/10.1093/cercor/10.7 .698, PubMed: 10906316 Futrell, R., Wilcox, E., Morita, T., Qian, P., Ballesteros, M., & Levy, R. (2019). Neural language models as psycholinguistic subjects: Representations of syntactic state. In Proceedings of the 2019 conference of the North American chapter of the Association for Computational Linguistics: Human language technologies ( Volume 1: Long and short papers) (pp. 32–42). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1004 Gauthier, I., Skudlarski, P., Gore, J. C., & Anderson, A. W. (2000). Expertise for cars and birds recruits brain areas involved in face recognition. Nature Neuroscience, 3(2), 191–197. https://doi.org /10.1038/72140, PubMed: 10649576 Gisladottir, R. S., Chwilla, D. J., & Levinson, S. C. (2015). Conver- sation electrified: ERP correlates of speech act recognition in underspecified utterances. PLOS ONE, 10(3), Article e0120068. https://doi.org/10.1371/journal.pone.0120068, PubMed: 25793289 Neurobiology of Language 23 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u n o / l / l a r t i c e - p d f / d o i / l / / . 1 0 1 1 6 2 n o _ a _ 0 0 1 0 1 2 0 7 4 5 3 9 n o _ a _ 0 0 1 0 1 p d / . l f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 In silico language neuroscience Goldstein, A., Zada, Z., Buchnik, E., Schain, M., Price, A., Aubrey, B., Nastase, S. A., Feder, A., Emanuel, D., Cohen, A., Jansen, A., Gazula, H., Choe, G., Rao, A., Kim, S. C., Casto, C., Fanda, L., Doyle, W., Friedman, D., … Hasson, U. (2021). Thinking ahead: Spontaneous prediction in context as a keystone of language in humans and machines. bioRxiv. https://doi.org/10.1101/2020.12 .02.403477 Goodkind, A., & Bicknell, K. (2018). Predictive power of word sur- prisal for reading times is a linear function of language model quality. In Proceedings of the 8th workshop on cognitive model- ing and computational linguistics (CMCL 2018) (pp. 10–18). Association for Computational Linguistics. https://doi.org/10 .18653/v1/ W18-0102 Graves, W. W., Binder, J. R., Desai, R. H., Conant, L. L., & Seidenberg, M. S. (2010). Neural correlates of implicit and explicit combinato- rial semantic processing. NeuroImage, 53(2), 638–646. https://doi .org/10.1016/j.neuroimage.2010.06.055, PubMed: 20600969 Guest, O., & Martin, A. E. (2023). On logical inference over brains, behaviour, and artificial neural networks. Computational Brain & Behavior. https://doi.org/10.1007/s42113-022-00166-x Gulordava, K., Bojanowski, P., Grave, E., Linzen, T., & Baroni, M. (2018). Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 conference of the North American chapter of the Association for Computational Linguistics: Human language technologies ( Volume 1: Long papers) (pp. 1195–1205). Association for Computational Linguistics. https://doi.org/10 .18653/v1/N18-1108 Haber, J., & Poesio, M. (2021). Patterns of polysemy and homonymy in contextualised language models. In Findings of the Association for Computational Linguistics: EMNLP 2021 (pp. 2663–2676). Association for Computational Linguistics. https://doi.org/10 .18653/v1/2021.findings-emnlp.226 Hamilton, L. S., & Huth, A. G. (2018). The revolution will not be controlled: Natural stimuli in speech neuroscience. Language, Cognition and Neuroscience, 35(5), 573–582. https://doi.org/10 .1080/23273798.2018.1499946, PubMed: 32656294 Handwerker, D. A., Ollinger, J. M., & D’Esposito, M. (2004). Vari- ation of BOLD hemodynamic responses across subjects and brain regions and their effects on statistical analyses. Neuro- Image, 21(4), 1639–1651. https://doi.org/10.1016/j.neuroimage .2003.11.029, PubMed: 15050587 Hasson, U., Avidan, G., Gelbard, H., Vallines, I., Harel, M., Minshew, N., & Behrmann, M. (2009). Shared and idiosyncratic cortical activation patterns in autism revealed under continuous real-life viewing conditions. Autism Research, 2(4), 220–231. https://doi.org/10.1002/aur.89, PubMed: 19708061 Hasson, U., Yang, E., Vallines, I., Heeger, D. J., & Rubin, N. (2008). A hierarchy of temporal receptive windows in human cortex. Journal of Neuroscience, 28(10), 2539–2550. https://doi.org/10 .1523/JNEUROSCI.5487-07.2008, PubMed: 18322098 Hauk, O., Johnsrude, I., & Pulvermüller, F. (2004). Somatotopic representation of action words in human motor and premotor cortex. Neuron, 41(2), 301–307. https://doi.org/10.1016/S0896 -6273(03)00838-9, PubMed: 14741110 Haxby, J. V., Guntupalli, J. S., Nastase, S. A., & Feilong, M. (2020). Hyperalignment: Modeling shared information encoded in idio- syncratic cortical topographies. ELife, 9, Article e56601. https:// doi.org/10.7554/eLife.56601, PubMed: 32484439 Hedge, C., Powell, G., & Sumner, P. (2018). The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences. Behavior Research Methods, 50(3), 1166–1186. https://doi.org/10.3758/s13428-017-0935-1 , PubMed: 28726177 Hewitt, J., & Liang, P. (2019). Designing and interpreting probes with control tasks. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP–IJCNLP) (pp. 2733–2743). Association for Computa- tional Linguistics. https://doi.org/10.18653/v1/D19-1275 Hewitt, J., & Manning, C. D. (2019). A structural probe for finding syntax in word representations. In Proceedings of the 2019 conference of the North American chapter of the Association for Computational Linguistics: Human language technologies ( Volume 1: Long and short papers) (pp. 4129–4138). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19 -1419 Huth, A. G., de Heer, W. A., Griffiths, T. L., Theunissen, F. E., & Gallant, J. L. (2016). Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532(7600), 453–458. https://doi.org/10.1038/nature17637, PubMed: 27121839 Huth, A. G., Griffiths, T. L., Theunissen, F. E., & Gallant, J. L. (2015). PrAGMATiC: A probabilistic and generative model of areas tiling the cortex. arXiv. https://doi.org/10.48550/arXiv.1504.03622 Ince, R. A., Paton, A. T., Kay, J. W., & Schyns, P. G. (2021). Bayesian inference of population prevalence. ELife, 10, Article e62461. https://doi.org/10.7554/eLife.62461, PubMed: 34612811 Jain, S., & Huth, A. (2018). Incorporating context into language encoding models for fMRI. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, & R. Garnett (Eds.), Advances in Neural Information Processing Systems 31 (10 pp.). NeurIPS. Jain, S., & Huth, A. G. (2023). Discovering distinct patterns of semantic integration across cortex using natural language encod- ing models for fMRI [Manuscript in preparation]. Departments of Computer Science & Neuroscience, University of Texas at Austin. Jain, S., Vo, V. A., Mahto, S., LeBel, A., Turek, J. S., & Huth, A. (2020). Interpretable multi-timescale models for predicting fMRI responses to continuous natural speech. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, & H. Lin (Eds.), Advances in Neural Information Processing Systems 33 (pp. 13738–13749). NeurIPS. Kable, J. W., Lease-Spellmeyer, J., & Chatterjee, A. (2002). Neural substrates of action event knowledge. Journal of Cognitive Neuroscience, 14(5), 795–805. https://doi.org/10.1162 /08989290260138681, PubMed: 12167263 Kanwisher, N., McDermott, J., & Chun, M. M. (1997). The fusiform face area: A module in human extrastriate cortex specialized for face perception. Journal of Neuroscience, 17(11), 4302–4311. https://doi.org/10.1523/ JNEUROSCI.17-11-04302.1997, PubMed: 9151747 Kell, A. J. E., Yamins, D. L. K., Shook, E. N., Norman-Haignere, S. V., & McDermott, J. H. (2018). A task-optimized neural network replicates human auditory behavior, predicts brain responses, and reveals a cortical processing hierarchy. Neuron, 98(3), 630–644. https://doi.org/10.1016/j.neuron.2018.03.044, PubMed: 29681533 Khandelwal, U., He, H., Qi, P., & Jurafsky, D. (2018). Sharp nearby, fuzzy far away: How neural language models use context. In Pro- ceedings of the 56th annual meeting of the Association for Com- putational Linguistics ( Volume 1: Long papers) (pp. 284–294). Association for Computational Linguistics. https://doi.org/10 .18653/v1/P18-1027 Kumar, S., Sumers, T. R., Yamakoshi, T., Goldstein, A., Hasson, U., Norman, K. A., Griffiths, T. L., Hawkins, R. D., & Nastase, S. A. (2022). Reconstructing the cascade of language processing in the Neurobiology of Language 24 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u n o / l / l a r t i c e - p d f / d o i / l / . / 1 0 1 1 6 2 n o _ a _ 0 0 1 0 1 2 0 7 4 5 3 9 n o _ a _ 0 0 1 0 1 p d . / l f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 In silico language neuroscience brain using the internal computations of a transformer-based language model. bioRxiv. https://doi.org/10.1101/2022.06.08 .495348 Kutas, M., & Hillyard, S. A. (1984). Brain potentials during reading reflect word expectancy and semantic association. Nature, 307(5947), 161–163. https://doi.org/10.1038/307161a0, PubMed: 6690995 Lakretz, Y., Kruszewski, G., Desbordes, T., Hupkes, D., Dehaene, S., & Baroni, M. (2019). The emergence of number and syntax units in LSTM language models. In Proceedings of the 2019 con- ference of the North American chapter of the Association for Computational Linguistics: Human language technologies ( Volume 1: Long and short papers) (pp. 11–20). Association for Computational Linguistics. https://doi.org/10.18653/v1/ N19 -1002 LeBel, A., Jain, S., & Huth, A. G. (2021). Voxelwise encoding models show that cerebellar language representations are highly conceptual. Journal of Neuroscience, 41(50), 10341–10355. https://doi.org/10.1523/ JNEUROSCI.0118-21.2021, PubMed: 34732520 LeBel, A., Wagner, L., Jain, S., Adhikari-Desai, A., Gupta, B., Morgenthal, A., Tang, J., Xu, L., & Huth, A. G. (2022). A natural language fMRI dataset for voxelwise encoding models. bioRxiv. https://doi.org/10.1101/2022.09.22.509104 Lerner, Y., Honey, C. J., Katkov, M., & Hasson, U. (2014). Temporal scaling of neural responses to compressed and dilated natural speech. Journal of Neurophysiology, 111(12), 2433–2444. https://doi.org/10.1152/jn.00497.2013, PubMed: 24647432 Lerner, Y., Honey, C. J., Silbert, L. J., & Hasson, U. (2011). Topographic mapping of a hierarchy of temporal receptive windows using a narrated story. Journal of Neuroscience, 31(8), 2906–2915. https://doi.org/10.1523/JNEUROSCI.3684-10.2011, PubMed: 21414912 Levinson, S. C. (2016). Turn-taking in human communication: Origins and implications for language processing. Trends in Cognitive Sciences, 20(1), 6–14. https://doi.org/10.1016/j.tics .2015.10.010, PubMed: 26651245 Li, B. Z., Nye, M., & Andreas, J. (2021). Implicit representations of meaning in neural language models. In Proceedings of the 59th annual meeting of the Association for Computational Linguistics and the 11th international joint conference on natural language processing ( Volume 1: Long papers) (pp. 1813–1827). Associa- tion for Computational Linguistics. https://doi.org/10.18653/v1 /2021.acl-long.143 Li, J., Bhattasali, S., Zhang, S., Franzluebbers, B., Luh, W.-M., Spreng, R. N., Brennan, J. R., Yang, Y., Pallier, C., & Hale, J. (2022). Le Petit Prince multilingual naturalistic fMRI corpus. Scientific Data, 9(1), Article 530. https://doi.org/10.1038 /s41597-022-01625-7, PubMed: 36038567 Li, Y., Anumanchipalli, G. K., Mohamed, A., Lu, J., Wu, J., & Chang, E. F. (2022). Dissecting neural computations of the human audi- tory pathway using deep neural networks for speech. bioRxiv. https://doi.org/10.1101/2022.03.14.484195 Linzen, T. (2020). How can we accelerate progress towards human-like linguistic generalization? In Proceedings of the 58th annual meeting of the Association for Computational Linguistics (pp. 5210–5217). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.465 Linzen, T., & Leonard, B. (2018). Distinct patterns of syntactic agreement errors in recurrent networks and humans. arXiv. https://doi.org/10.48550/arXiv.1807.06882 Liu, J., Harris, A., & Kanwisher, N. (2010). Perception of face parts and face configurations: An fMRI study. Journal of Cognitive Neuroscience, 22(1), 203–211. https://doi.org/10.1162/jocn .2009.21203, PubMed: 19302006 Magyari, L., Bastiaansen, M. C. M., de Ruiter, J. P., & Levinson, S. C. (2014). Early anticipation lies behind the speed of response in conversation. Journal of Cognitive Neuroscience, 26(11), 2530–2539. https://doi.org/10.1162/jocn_a_00673, PubMed: 24893743 Mahowald, K., Kachergis, G., & Frank, M. C. (2020). What counts as an exemplar model, anyway? A commentary on Ambridge (2020). First Language, 40(5–6), 608–611. https://doi.org/10 .1177/0142723720905920 Marvin, R., & Linzen, T. (2018). Targeted syntactic evaluation of language models. In Proceedings of the 2018 conference on empirical methods in natural language processing (pp. 1192–1202). Association for Computational Linguistics. https://doi.org/10 .18653/v1/D18-1151 Matusz, P. J., Dikker, S., Huth, A. G., & Perrodin, C. (2019). Are we ready for real-world neuroscience? Journal of Cognitive Neuro- science, 31(3), 327–338. https://doi.org/10.1162/jocn_e_01276, PubMed: 29916793 Merkx, D., & Frank, S. L. (2021). Human sentence processing: Recurrence or attention? In Proceedings of the workshop on cognitive modeling and computational linguistics (pp. 12–22). Association for Computational Linguistics. https://doi.org/10 .18653/v1/2021.cmcl-1.2 Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient esti- mation of word representations in vector space. arXiv. https://doi .org/10.48550/arXiv.1301.3781 Millet, J., Caucheteux, C., Orhan, P., Boubenec, Y., Gramfort, A., Dunbar, E., Pallier, C., & King, J.-R. (2022). Toward a realistic model of speech processing in the brain with self-supervised learning. arXiv. https://doi.org/10.48550/arXiv.2206.01685 Millet, J., & King, J.-R. (2021). Inductive biases, pretraining and fine-tuning jointly account for brain responses to speech. arXiv. https://doi.org/10.48550/arXiv.2103.01032 Mitchell, T. M., Shinkareva, S. V., Carlson, A., Chang, K.-M., Malave, V. L., Mason, R. A., & Just, M. A. (2008). Predicting human brain activity associated with the meanings of nouns. Science, 320(5880), 1191–1195. https://doi.org/10.1126/science .1152876, PubMed: 18511683 Nastase, S. A., Liu, Y.-F., Hillman, H., Zadbood, A., Hasenfratz, L., Keshavarzian, N., Chen, J., Honey, C. J., Yeshurun, Y., Regev, M., Nguyen, M., Chang, C. H. C., Baldassano, C., Lositsky, O., Simony, E., Chow, M. A., Leong, Y. C., Brooks, P. P., Micciche, E., … Hasson, U. (2021). The “Narratives” fMRI dataset for eval- uating models of naturalistic language comprehension. Scientific Data, 8(1), Article 250. https://doi.org/10.1038/s41597-021 -01033-3, PubMed: 34584100 Nayebi, A., Attinger, A., Campbell, M. G., Hardcastle, K., Low, I. I. C., Mallory, C. S., Mel, G. C., Sorscher, B., Williams, A. H., Ganguli, S., Giocomo, L. M., & Yamins, D. L. K. (2021). Explaining heterogeneity in medial entorhinal cortex with task-driven neural networks. bioRxiv. https://doi.org/10.1101/2021.10.30.466617 Noppeney, U., Josephs, O., Kiebel, S., Friston, K. J., & Price, C. J. (2005). Action selectivity in parietal and temporal cortex. Cogni- tive Brain Research, 25(3), 641–649. https://doi.org/10.1016/j .cogbrainres.2005.08.017, PubMed: 16242924 Nunez-Elizalde, A. O., Huth, A. G., & Gallant, J. L. (2019). Voxel- wise encoding models with non-spherical multivariate normal priors. NeuroImage, 197, 482–492. https://doi.org/10.1016/j .neuroimage.2019.04.012, PubMed: 31075394 Overath, T., McDermott, J. H., Zarate, J. M., & Poeppel, D. (2015). The cortical analysis of speech-specific temporal structure Neurobiology of Language 25 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u n o / l / l a r t i c e - p d f / d o i / l . / / 1 0 1 1 6 2 n o _ a _ 0 0 1 0 1 2 0 7 4 5 3 9 n o _ a _ 0 0 1 0 1 p d / . l f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 In silico language neuroscience revealed by responses to sound quilts. Nature Neuroscience, 18(6), 903–911. https://doi.org/10.1038/nn.4021, PubMed: 25984889 Pandia, L., & Ettinger, A. (2021). Sorting through the noise: Testing robustness of information processing in pre-trained language models. In Proceedings of the 2021 conference on empirical methods in natural language processing (pp. 1583–1596). Asso- ciation for Computational Linguistics. https://doi.org/10.18653 /v1/2021.emnlp-main.119 Pashler, H., & Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7(6), 531–536. https://doi.org/10.1177/1745691612463401, PubMed: 26168109 Pennington, J., Socher, R., & Manning, C. (2014). GloVe: Global Vectors for Word Representation. In Proceedings of the 2014 con- ference on empirical methods in natural language processing (EMNLP) (pp. 1532–1543). Association for Computational Linguistics. https://doi.org/10.3115/v1/D14-1162 Poldrack, R. A., Whitaker, K., & Kennedy, D. (2020). Introduction to the special issue on reproducibility in neuroimaging. NeuroImage, 218, Article 116357. https://doi.org/10.1016/j.neuroimage.2019 .116357, PubMed: 31733374 Popham, S. F., Huth, A. G., Bilenko, N. Y., Deniz, F., Gao, J. S., Nunez-Elizalde, A. O., & Gallant, J. L. (2021). Visual and linguis- tic semantic representations are aligned at the border of human visual cortex. Nature Neuroscience, 24(11), 1628–1636. https:// doi.org/10.1038/s41593-021-00921-6, PubMed: 34711960 Prasad, G., van Schijndel, M., & Linzen, T. (2019). Using priming to uncover the organization of syntactic representations in neural language models. In Proceedings of the 23rd conference on computational natural language learning (CoNLL) (pp. 66–76). Association for Computational Linguistics. https://doi.org/10 .18653/v1/K19-1007 Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving language understanding by generative pre-training [Preprint]. Papers With Code. Ratan Murty, N. A., Bashivan, P., Abate, A., DiCarlo, J. J., & Kanwisher, N. (2021). Computational models of category- selective brain regions enable high-throughput tests of selectivity. Nature Communications, 12(1), 5540. https://doi.org/10.1038 /s41467-021-25409-6, PubMed: 34545079 Ravfogel, S., Elazar, Y., Gonen, H., Twiton, M., & Goldberg, Y. (2020). Null it out: Guarding protected attributes by iterative null- space projection. In Proceedings of the 58th annual meeting of the Association for Computational Linguistics (pp. 7237–7256). Association for Computational Linguistics. https://doi.org/10 .18653/v1/2020.acl-main.647 Redcay, E., & Moraczewski, D. (2020). Social cognition in context: A naturalistic imaging approach. NeuroImage, 216, Article 116392. https://doi.org/10.1016/j.neuroimage.2019.116392, PubMed: 31770637 Reddy, A. J., & Wehbe, L. (2020). Can fMRI reveal the representa- tion of syntactic structure in the brain?. bioRxiv. https://doi.org/10 .1101/2020.06.16.155499 Regev, M., Honey, C. J., Simony, E., & Hasson, U. (2013). Selective and invariant neural responses to spoken and written narratives. Journal of Neuroscience, 33(40), 15978–15988. https://doi.org /10.1523/JNEUROSCI.1580-13.2013, PubMed: 24089502 Schrimpf, M., Blank, I. A., Tuckute, G., Kauf, C., Hosseini, E. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2021). The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 118(45), Article e2105646118. https://doi.org/10 .1073/pnas.2105646118, PubMed: 34737231 Scott, S. K. (2019). From speech and talkers to the social world: The neural processing of human spoken language. Science, 366(6461), 58–62. https://doi.org/10.1126/science.aax0288, PubMed: 31604302 Scott, T. L., Gallée, J., & Fedorenko, E. (2017). A new fun and robust version of an fMRI localizer for the frontotemporal language sys- tem. Cognitive Neuroscience, 8(3), 167–176. https://doi.org/10 .1080/17588928.2016.1201466, PubMed: 27386919 Sergent, J., Ohta, S., & MacDonald, B. (1992). Functional neuro- anatomy of face and object processing. A positron emission tomography study. Brain: A Journal of Neurology, 115(1), 15–36. https://doi.org/10.1093/brain/115.1.15, PubMed: 1559150 Shain, C., Blank, I. A., van Schijndel, M., Schuler, W., & Fedorenko, E. (2020). fMRI reveals language-specific predictive coding during naturalistic sentence comprehension. Neuropsychologia, 138, Article 107307. https://doi.org/10.1016/j.neuropsychologia .2019.107307, PubMed: 31874149 Sievers, B., Welker, C., Hasson, U., Kleinbaum, A. M., & Wheatley, T. (2020). How consensus-building conversation changes our minds and aligns our brains. PsyArXiv. https://doi.org/10.31234 /osf.io/562z7 Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False- positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psycho- logical Science, 22(11), 1359–1366. https://doi.org/10.1177 /0956797611417632, PubMed: 22006061 Sundararajan, M., Taly, A., & Yan, Q. (2017). Axiomatic attribu- tion for deep networks. In D. Precup & Y. W. Teh (Eds.), Proceed- ings of the 34th international conference on machine learning ( Volume 70) (pp. 3319–3328). Association for Computing Machinery. Suzanne Scherf, K., Behrmann, M., Minshew, N., & Luna, B. (2008). Atypical development of face and greeble recognition in autism. Journal of Child Psychology and Psychiatry, 49(8), 838–847. https://doi.org/10.1111/j.1469-7610.2008.01903.x, PubMed: 18422548 Tenney, I., Das, D., & Pavlick, E. (2019). BERT rediscovers the classical NLP pipeline. arXiv. https://doi.org/10.48550/arXiv .1905.05950 Tenney, I., Xia, P., Chen, B., Wang, A., Poliak, A., McCoy, R. T., Kim, N., Durme, B. V., Bowman, S. R., Das, D., & Pavlick, E. (2019). What do you learn from context? Probing for sentence structure in contextualized word representations [Preprint]. Open Review. https://openreview.net/forum?id=SJzSgnRcKX Toneva, M., Mitchell, T. M., & Wehbe, L. (2022). Combining com- putational controls with natural text reveals new aspects of mean- ing composition. Nature Computational Science, 2(1), 745–757. https://doi.org/10.1038/s43588-022-00354-6, PubMed: 36777107 Toneva, M., & Wehbe, L. (2019). Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). In H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, & R. Garnett (Eds.), Advances in Neural Information Processing Systems 32. NeurIPS. Toneva, M., Williams, J., Bollu, A., Dann, C., & Wehbe, L. (2022). Same cause; different effects in the brain. Proceedings of the First Conference on Causal Learning and Reasoning, 177, 787–825. Vaidya, A. R., Jain, S., & Huth, A. (2022). Self-supervised models of audio effectively explain human cortical responses to speech. Proceedings of the 39th International Conference on Machine Learning, 162, 21927–21944. https://proceedings.mlr.press /v162/vaidya22a.html van der Wees, M., Bisazza, A., & Monz, C. (2017). Dynamic data selection for neural machine translation. In Proceedings of the Neurobiology of Language 26 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u n o / l / l a r t i c e - p d f / d o i / l / / . 1 0 1 1 6 2 n o _ a _ 0 0 1 0 1 2 0 7 4 5 3 9 n o _ a _ 0 0 1 0 1 p d / . l f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 In silico language neuroscience 2017 conference on empirical methods in natural language processing (pp. 1400–1410). Association for Computational Linguistics. https://doi.org/10.18653/v1/D17-1147 Vo, V. A., Jain, S., Beckage, N., Chien, H.-Y. S., Obinwa, C., & Huth, A. G. (2023). A unifying computational account of temporal processing in natural speech across cortex [Manuscript in prepara- tion]. Departments of Computer Science & Neuroscience, Univer- sity of Texas at Austin. Wallentin, M., Østergaard, S., Lund, T. E., Østergaard, L., & Roepstorff, A. (2005). Concrete spatial language: See what I mean? Brain and Language, 92(3), 221–233. https://doi.org/10.1016/j.bandl .2004.06.106, PubMed: 15721955 Wang, A., Tarr, M., & Wehbe, L. (2019). Neural taskonomy: Infer- ring the similarity of task-derived representations from brain activity. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, & R. Garnett (Eds.), Advances in Neural Information Processing Systems 32. NeurIPS. https:// p r o c e e d i n g s . n e u r i p s . c c / p a p e r / 2 0 1 9 / h a s h /f490c742cd8318b8ee6dca10af2a163f-Abstract.html Wang, S., Zhang, J., Lin, N., & Zong, C. (2020). Probing brain activation patterns by dissociating semantics and syntax in sentences. Proceedings of the AAAI Conference on Artificial Intelligence, 34(5), 9201–9208. https://doi.org/10.1609/aaai .v34i05.6457 Wang, S., Zhang, X., Zhang, J., & Zong, C. (2022). A synchronized multimodal neuroimaging dataset for studying brain language processing. Scientific Data, 9(1), Article 590. https://doi.org/10 .1038/s41597-022-01708-5, PubMed: 36180444 Wehbe, L., Huth, A. G., Deniz, F., Gao, J., Kieseler, M.-L., & Gallant, J. L. (2021). BOLDpredictions [Software]. https://github .com/boldprediction Q9 Wehbe, L., Huth, A. G., Deniz, F., Gao, J., Kieseler, M.-L., & Gallant, J. L. (2018). BOLD predictions: Automated simulation of fMRI experiments [Poster]. 2018 Conference on Cognitive Computational Neuroscience, Philadelphia, Pennsylvania. https://doi.org/10.32470/CCN.2018.1123-0 Wehbe, L., Murphy, B., Talukdar, P., Fyshe, A., Ramdas, A., & Mitchell, T. (2014). Simultaneously uncovering the patterns of brain regions involved in different story reading subprocesses. PLOS ONE, 9(11), Article e112575. https://doi.org/10.1371 /journal.pone.0112575, PubMed: 25426840 Wehbe, L., Vaswani, A., Knight, K., & Mitchell, T. (2014). Aligning context-based statistical models of language with brain activity during reading. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 233–243). Association for Computational Linguistics. https://doi.org/10.3115 /v1/D14-1030 Westfall, J., Nichols, T. E., & Yarkoni, T. (2017). Fixing the stimulus-as-fixed-effect fallacy in task fMRI. Wellcome Open Research, 1, 23. https://doi.org/10.12688/wellcomeopenres .10298.2, PubMed: 28503664 Wilcox, E., Vani, P., & Levy, R. (2021). A targeted assessment of incremental processing in neural language models and humans. In Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint confer- ence on natural language processing ( Volume 1: Long papers) (pp. 939–952). Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.acl-long.76 Wu, A., Wang, C., Pino, J., & Gu, J. (2020). Self-supervised represen- tations improve end-to-end speech translation. Interspeech 2020, 1491–1495. https://doi.org/10.21437/Interspeech.2020-3094 Wu, M. C.-K., David, S. V., & Gallant, J. L. (2006). Complete func- tional characterization of sensory neurons by system identifica- tion. Annual Review of Neuroscience, 29, 477–505. https://doi .org/10.1146/annurev.neuro.29.051605.113024, PubMed: 16776594 Xu, Y. (2005). Revisiting the role of the fusiform face area in visual expertise. Cerebral Cortex, 15(8), 1234–1242. https://doi.org/10 .1093/cercor/bhi006, PubMed: 15677350 Yamins, D. L., & DiCarlo, J. J. (2016). Eight open questions in the computational modeling of higher sensory cortex. Current Opin- ion in Neurobiology, 37, 114–120. https://doi.org/10.1016/j.conb .2016.02.001, PubMed: 26921828 Yarkoni, T. (2022). The generalizability crisis. Behavioral and Brain Sciences, 45, Article e1. https://doi.org/10.1017 /S0140525X20001685, PubMed: 33342451 Yeshurun, Y., Nguyen, M., & Hasson, U. (2017). Amplification of local changes along the timescale processing hierarchy. Proceed- ings of the National Academy of Sciences, 114(35), 9475–9480. https://doi.org/10.1073/pnas.1701652114, PubMed: 28811367 Zhang, X., Wang, S., Lin, N., Zhang, J., & Zong, C. (2022). Probing word syntactic representations in the brain by a feature elimina- tion method [Poster]. Proceedings of the 36th AAAI conference on artificial intelligence. Association for the Advancement of Artifical Intelligence. https://aaai-2022.virtualchair.net/poster _aaai7935 Neurobiology of Language 27 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u n o / l / l a r t i c e - p d f / d o i / l / . / 1 0 1 1 6 2 n o _ a _ 0 0 1 0 1 2 0 7 4 5 3 9 n o _ a _ 0 0 1 0 1 p d / . l f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3

Descargar PDF