Medial Orbitofrontal Cortex, Dorsolateral Prefrontal
Kortex, and Hippocampus Differentially
Represent the Event Saliency
Anna Jafarpour1,2, Sandon Griffin1, Jack J. Lin3, und Robert T. Knight1
Abstrakt
■ Two primary functions attributed to the hippocampus and
prefrontal cortex (PFC) network are retaining the temporal
and spatial associations of events and detecting deviant events.
It is unclear, Jedoch, how these two functions converge into
one mechanism. Hier, we tested whether increased activity
with perceiving salient events is a deviant detection signal or
contains information about the event associations by reflecting
the magnitude of deviance (d.h., event saliency). We also tested
how the deviant detection signal is affected by the degree of
anticipation. We studied regional neural activity when people
watched a movie that had varying saliency of a novel or an
anticipated flow of salient events. Using intracranial electro-
encephalography from 10 Patienten, we observed that high-frequency
Aktivität (50–150 Hz) in the hippocampus, dorsolateral PFC,
and medial OFC tracked event saliency. We also observed that
medial OFC activity was stronger when the salient events were
anticipated than when they were novel. These results suggest
that dorsolateral PFC and medial OFC, as well as the hippo-
campus, signify the saliency magnitude of events, reflecting
the hierarchical structure of event associations. ■
EINFÜHRUNG
“I was waiting at home for my friend. I made some tea,
washed the cups, and poured hot water. Then I felt
everything shaking. It was an earthquake. I put the cup
down and waited to see if there was an aftershock. Nur
about then, my friend arrived.” We experience the world
as a sequence of events, but we remember them as seg-
mented sequences. During encoding, perceiving unusual
events separates the flow of events, which is an event
segmentation process (Zacks, Speer, Swallow, Mutiger, &
Reynolds, 2007; Zacks & Swallow, 2007), so that each seg-
ment contains the relationship of events that occurred in
a similar circumstance (Zwaan & Radvansky, 1998). In
this process, sequences of events are organized to create
a hierarchical relationship (Kurby & Zacks, 2008; Zacks &
Swallow, 2007; Zwaan & Radvansky, 1998), where a se-
quence of less notable events are embedded in an over-
riding structure of segmented sequences (Hanson &
Hirst, 1989). In this example, making some tea was em-
bedded in a friend’s visit. We hypothesized that a basic
model to support the hierarchical relationship of se-
quences relies on the magnitude of event deviancy. In
principle, the hierarchical relationship can be recon-
structed from event saliency where less salient events
are more temporally associated with prior events than
1Universität von Kalifornien, Berkeley, 2Universität Washington,
3Universität von Kalifornien, Irvine
© 2019 Massachusetts Institute of Technology
more salient new events (Abbildung 1C; also see Yeung,
Yeo, & Liu, 1996).
The magnitude of deviance of events is referred to as
“event saliency” and is quantified by crowdsourcing.
Previously, event saliency has been used to determine
the probability of a deviant event in a linguistic experi-
ment (Coulson, King, & Kutas, 1998), but recently, mit
the availability of videos and advances in computer vi-
sion, this term is used for quantifying the magnitude of
deviation in a flow of a movie. Event saliency is not deter-
mined by changes in a visual scene but relies on follow-
ing the movies and noting significant changes in the flow
of events (Zhang, Han, Jiang, Ye, & Chang, 2017; Rosani,
Boato, & De Natale, 2015). Entsprechend, event saliency is
measured by either asking an independent group to
identify the boundaries or, in linguistics, studying the
transition from one word to another, which is extracted
from a large body of literature.
Both event association and deviancy detection are
linked to the hippocampus, OFC, and dorsolateral pre-
frontal cortex (PFC; Paz et al., 2010; Zacks et al., 2007;
Nobre, Coull, Frith, & Mesulam, 1999; Ritter, 1996).
We reasoned deviant detection reflects event saliency; Das
signal would also reflect a temporal association of events.
Hier, we tested the prediction that high-frequency neural
activity in subregions of PFC and the hippocampus tracked
event saliency.
PFC–hippocampal neural network is also engaged in
prospective coding ( Jafarpour, Piai, Lin, & Ritter, 2017;
Zeitschrift für kognitive Neurowissenschaften 31:6, S. 874–884
doi:10.1162/jocn_a_01392
l
D
Ö
w
N
Ö
A
D
e
D
F
R
Ö
M
H
T
T
P
:
/
/
D
ich
R
e
C
T
.
M
ich
T
.
e
D
u
/
J
/
Ö
C
N
A
R
T
ich
C
e
–
P
D
l
F
/
/
/
/
3
1
6
8
7
4
1
7
8
8
4
5
0
/
J
Ö
C
N
_
A
_
0
1
3
9
2
P
D
.
F
B
j
G
u
e
S
T
T
Ö
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
l
D
Ö
w
N
Ö
A
D
e
D
F
R
Ö
M
H
T
T
P
:
/
/
D
ich
R
e
C
T
.
M
ich
T
.
e
D
u
/
J
/
Ö
C
N
A
R
T
ich
C
e
–
P
D
l
F
/
/
/
/
3
1
6
8
7
4
1
7
8
8
4
5
0
/
J
Ö
C
N
_
A
_
0
1
3
9
2
P
D
.
F
B
j
G
u
e
S
T
T
Ö
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
Figur 1. A hierarchical structure of the movie can be extracted from event saliency. (A) Patients passively watched a (∼3 min) muted
animation that they did not see before. The movie had a mixture of novel and anticipated new events. (B) We chunked the movie into
129 interchangeable epochs of 1.5 Sek. The epochs had a range of event saliency, defined as the proportion of an independent group of
Teilnehmer (n = 80) that determined event boundaries in the movie epochs. (C) Event saliency was used to construct the hierarchical
relationship of epochs, with distance being the sum of event saliency between epochs. The heat map shows the sum of saliency of the events
that occurred between each pair of events (ranged between 0 Und 18.8, which is the largest sum of saliency of a pair of events). The row
and column order have been reordered based on the hierarchical clustering results. All the epochs are displayed in rows and columns (jeden
third column is numbered in the illustration), and temporally adjacent events are next to each other. The right and left branches of the
hierarchy do not imply any order and can be flipped because the plot is symmetric. (D) Zoomed in view of the first 10.5 sec of the movie
(marked branches in C). The graph shows the structure of event associations at the start of the movie.
Brown et al., 2016; Hindy, Ng, & Turk-Browne, 2016;
Hsieh & Ranganath, 2015; Hsieh, Gruber, Jenkins, &
Ranganath, 2014), which enhances event segmentation
(Schütz-Bosbach & Prinz, 2007). Note that anticipated
salient events are different from novel events. For exam-
Bitte, a salient event like a friend’s planned visit is antic-
ipiert, whereas a salient event such as an earthquake is
novel. Hier, we predicted that the neural representation
of event saliency would be different for novel and pre-
dictable salient events.
Generating sequences of novel events in an experi-
mental condition is challenging (Zarcone, van Schijndel,
Vogels, & Demberg, 2016). Previous studies have used a
discrete experimental design, comparing the neural
activity at the time of perceiving deviant events, das ist,
event boundaries (Kurby & Zacks, 2008), to the neural
activity at the time of perceiving nonboundary events
( Whitney et al., 2009; Speer, Zacks, & Reynolds, 2007;
Zacks et al., 2007). Jedoch, after encountering the first
few deviant events, participants anticipate the new
Veranstaltungen; daher, the later deviant events are no longer novel;
stattdessen, they become anticipated salient events. A flow of
Veranstaltungen, such as observed in a movie, has numerous event
boundaries, with a range of anticipated or novel saliency
Jafarpour et al.
875
providing an ideal experimental design to address this
issue.
We recorded local field potential using intracranial
electroencephalography (z.B) aus 10 patients with
epilepsy who had electrodes implemented for clinical
Zwecke. Patients passively watched a movie (Figur 1).
The analysis focused on the local activity captured as the
power in the high-frequency activity (HFA; 50–150 Hz)
that serves as a metric for local neural activation (Reich &
Wallis, 2017; Lachaux, Axmacher, Mormann, Halgren, &
Crone, 2012; Jacobs & Kahana, 2009; Belitski et al.,
2008; Ray, Crone, Niebur, Franaszczuk, & Hsiao, 2008).
epileptic activity, which were excluded from the analysis
so that all electrodes included in the analysis were
nonpathological and free of epileptogenic spikes. Any
segment where focal spikes spread to other brain re-
gions were also excluded from the analysis (Tisch 1).
Electrode coverage included the medial-temporal lobe
and the PFC, depending on their clinical requirements
(Figur 2). Electrodes were localized in the patient’s
native space and then transferred to MNI space for vi-
sualizing the group coverage. We studied electrodes
in three ROIs: the lateral PFC, the OFC, and the hippo-
campus (Tisch 1 and Figure 2).
METHODEN
Approval
The study protocol was approved by the Office for the Pro-
tection of Human Subjects of the University of California,
Berkeley, and the University of California, Irvine. Alle
participants provided written informed consent before
participating.
Behavioral Experiment
A control group of 80 healthy adults participated in a
movie segmentation test (53 Frauen, Durchschnittsalter = 28 Jahre,
SD = 14, age range = 18–68 years). Thirty-one participants
were 18–21 years old, 29 participants were 21–30 years old,
Und 20 participants were older than 30 Jahre alt.
Teilnehmer
Intrakranielles EEG
Ten epileptic patients who had stereotactically implanted
depth electrodes to localize the seizure onset zone for sub-
sequent surgical resection participated in this study (vier
Frauen, Durchschnittsalter = 37 Jahre, SD = 11, age range = 22–
58 Jahre; Tisch 1). The electrodes were placed at the
Universität von Kalifornien, Irvine, Medical Center, with 5-mm
interelectrode spacing. All patients had normal (or cor-
rected) vision. No seizure occurred during task admin-
istration. Two independent neurologists inspected the
neural activity and identified the electrodes with an
Experimental Design
Salient events can occur frequently in a flow of events,
such as serving customers in a busy café, or they can
be infrequent, such as driving along a desert highway.
Participants watched a short mute animation (∼3 min
long) that had frequent salient events. The movie was a
short version of the animation designed by Ali Derakhshi,
named “Wildlife” or “Hayat-e Vahsh,” the episode on
lions. The movie was selected so that participants have
not watched it before, the visual angle was kept similar
(Figure 1A), and events with various magnitudes of sa-
liency occurred in a short period. Critically, the storyline
of the movie changed so we could test the effect of novel
Tisch 1. Patient Electrode Coverage
Patient
Alter
1
2
3
4
5
6
7
8
9
10
50
46
34
31
40
58
34
22
33
23
Sex
Male
Male
Female
Male
Female
Female
Male
Female
Male
Male
Dominant Hand
OFC
Hippocampus
Lateral PFC
NEIN. of trials
10
11
3
3
3
2
2
3
3
10
Rechts
Rechts
Rechts
Links
Links
Rechts
Rechts
Rechts
Rechts
Rechts
15
17
15
20
20
2
16
10
33
112
125
104
110
129
123
129
122
126
129
This table contains patients’ age (in years); Sex (male or female); dominant hand (right or left); the number of bipolar-referenced electrodes in any
region of the OFC, hippocampus, or lateral PFC; and the number of trials (out of 129 total trials).
876
Zeitschrift für kognitive Neurowissenschaften
Volumen 31, Nummer 6
l
D
Ö
w
N
Ö
A
D
e
D
F
R
Ö
M
H
T
T
P
:
/
/
D
ich
R
e
C
T
.
M
ich
T
.
e
D
u
/
J
/
Ö
C
N
A
R
T
ich
C
e
–
P
D
l
F
/
/
/
/
3
1
6
8
7
4
1
7
8
8
4
5
0
/
J
Ö
C
N
_
A
_
0
1
3
9
2
P
D
.
F
B
j
G
u
e
S
T
T
Ö
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
“whenever something new happened.” We clarified that
“we want to segment this movie into episodes.” After the
segmentation task, they performed a target detection
task with targets displayed at random intervals. Par-
ticipants were instructed to press a key as soon as they
perceived the target. This part of the experiment mea-
sured participants’ RT for normalizing the timing of
event boundaries across participants.
Behavioral Analysis
We recorded the timing of keypresses during segmenta-
tion and target detection tasks. Participants’ RT during
the target detection task was measured as the difference
between the target onsets and the responses. We ex-
cluded consecutive keypresses for segmentation if the
interval was less than 100 ms (double key registration).
The averaged RT per participant was subtracted from the
timing of keypresses for movie segmentation to normal-
ize the timing of event boundaries. The number of
segmented events was accumulated across participants
at 1.5-sec epochs. The number of events in the 1.5-sec
epochs reflected the saliency of events (see also Ben-
Yakov & Henson, 2018). Entsprechend, an event was more
salient if more people reported it as an event boundary,
and an event was considered less salient if fewer people
marked them as an event boundary (see Figure 1B for
the range of saliency scale). The autocorrelation analysis
of the event saliency was tested using the Ljung–Box test
that was implemented in R (R Development Core Team,
2014). There was no significant autocorrelation in the
saliency magnitude in 1.5-sec epochs (χ2 = 1.922, df =
1, p = .1656). Likewise, autocorrelation in the epochs of
1.5-sec HFAs was negligible, allowing to using permuta-
tion test and applying the event saliency magnitude for
statistical analysis as outlined below.
A hierarchical clustering of event relationships
(Figur 1) was constructed by applying a binary hierarchi-
cal clustering algorithm in R (R Development Core Team,
2014) using distances between events. The distance be-
tween two events was measured by adding the saliency
magnitude of all the events between the two events.
Entsprechend, the events that had many highly salient
events in between them had larger distance than the
events with less salient events between them.
iEEG Data Collection and Preprocessing
iEEG data were acquired using the Nihon Kohden record-
ing system, analog-filtered above 0.01 Hz and digitally
sampled at 5 kHz or 10 kHz. A photodiode recorded
the luminance of a corner of the screen to track the tim-
ing of the movie presentation. Two independent neurol-
ogists selected the electrodes that showed both epileptic
activities and epochs with seizure spread. Only elec-
trodes in nonpathological regions were included in the
Analyse.
Jafarpour et al.
877
Figur 2. Patients’ electrode coverage is color-coded by the patients’
number. (A–D) shows PFC and OFC coverage from (A) the right
sagittal, (B) the left sagittal, (C) the coronal, Und (D) the inferior view.
(E) Hippocampal coverage in a 3-D glass hippocampus from the
superior anterior view.
versus anticipated salient events (link to the movie:
https://www.youtube.com/watch?v=Q_guH9vA0sk).
The movie had an overarching cliché love triangle
Geschichte. It starts by showing a few animal couples going
back and forth in a park (this part gets repetitive after
two repetitions). Dann, there is a small lion that looks
heartbroken. The lion sees a lioness, but there is a bigger
lion that also wants to meet the lioness. The two lions
fight for the lioness’s attention through a series of
matches. After each match, the score is shown on board
(this part is repetitive and predictable). Jedoch, Die
score is not immediately shown after an eating contest.
After the bigger lion wins the eating contest, it eats the
small lion’s food too. Dann, the score is shown, und das
matches continue. The small lion loses the competition
and moves on. The end of the movie shows that the
small lion meets a lioness again (this part was repetitive).
This movie had periods with an anticipated flow. Für
Beispiel, after each game that the bigger lion wins, Die
scoreboard is shown.
The iEEG group passively watched the muted movie,
but the behavioral control group watched the muted
movie and concurrently segmented the movies into epi-
sodes. We instructed the behavioral group to press a key
l
D
Ö
w
N
Ö
A
D
e
D
F
R
Ö
M
H
T
T
P
:
/
/
D
ich
R
e
C
T
.
M
ich
T
.
e
D
u
/
J
/
Ö
C
N
A
R
T
ich
C
e
–
P
D
l
F
/
/
/
/
3
1
6
8
7
4
1
7
8
8
4
5
0
/
J
Ö
C
N
_
A
_
0
1
3
9
2
P
D
.
F
B
j
G
u
e
S
T
T
Ö
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
All EEG analyses were run in R, MATLAB 2015a, Und
Fieldtrip toolbox (Oostenveld, Fries, Maris, & Schoffelen,
2011) offline. We applied a 2-Hz-wide stopband Butterworth
notch filter at 60-Hz line power noise and harmonics
and then down-sampled the data to 1 kHz using re-
sample() MATLAB function via Fieldtrip. The function
applies an antialiasing finite impulse response lowpass
filter and compensates for the delay introduced by the fil-
ter. All electrodes were re-referenced to a neighboring
electrode (d.h., bipolar reference). The continuous signal
was then cropped in 1.5-sec-long epochs with no over-
laps. The epochs were bandpass filtered for HFA (50–
150 Hz) using padding and a Hamming window. Der
Hilbert transformation was applied to the filtered data
for extracting the power.
statistical permutation test because the nonoverlapping
1.5-sec epochs of HFA were interchangeable. Beachten Sie, dass
the magnitudes of saliency and HFA epochs were not sig-
nificantly autocorrelated. The null distribution was made
aus 1000 iterations of surrogated trial labels. In each
iteration, the maximum correlation between HFA and
saliency magnitude was taken across all electrodes in an
ROI (nämlich, the lateral PFC, the OFC, and the hippo-
campus) für jeden Teilnehmer. The proportion of HFA–
saliency magnitude correlation coefficients in the null
distribution that was more than the observed correlation
coefficient yielded the nonparametric corrected p value
for the observed correlation. A p value of <.05 was con-
sidered significant.
Correlation Analysis between HFA and
Event Saliency
The correlation between event saliency and HFA in each
electrode was calculated using the Spearman correlation.
For estimating the p value, we used a nonparametric
Effects of Anticipating Salient Events
We tested if HFA–saliency magnitude correlation changed
with anticipating salient events using a linear mixed-effects
model. We recalculated the correlations between HFA and
saliency magnitudes in each brain region in sliding win-
dows of 15 sec with overlaps of 7.5 sec (24 bins). We used
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
e
d
u
/
j
/
o
c
n
a
r
t
i
c
e
-
p
d
l
f
/
/
/
/
3
1
6
8
7
4
1
7
8
8
4
5
0
/
j
o
c
n
_
a
_
0
1
3
9
2
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
Figure 3. Event saliency was
tracked in the dorsolateral PFC,
the medial OFC, and the
hippocampus. (A) HFA in the
dorsolateral PFC and the medial
OFC increased with increasing
event saliency. The electrodes
are color-coded by the Spearman
correlation coefficient (R)
between HFA and event saliency
across 1.5-sec epochs, which
ranged between −0.3 and 0.7.
(B) The saliency of each epoch
(gray line) and HFA across
epochs in a left medial OFC
electrode for example (black
line). For demonstration, HFA
and saliency were smoothed by a
10-episode window (equal to
15 sec). The underlying data
were not autocorrelated. (C) The
same as B but for a right lateral
PFC electrode. (D) HFA in the
hippocampus also correlated
with event saliency magnitude.
(E) An example of HFA in the
hippocampus and the epochs
saliency magnitude (same as B
and C). (A and D) The black
outlines highlight the electrodes
that showed the effect (cluster-
corrected p < .05). The thickness
of the black outline reflects the
effect significance (see Table 2
for the exact p values < .05); the
arrows show which electrode
corresponds to the Plots B, C,
and E, which were from Patient 6.
878
Journal of Cognitive Neuroscience
Volume 31, Number 6
Table 2. Statistical Results
Table 2. (continued )
Patient
r
p
x
y
z
Patient
r
Hippocampus
1
1
5
5
5
6
6
OFC
1
1
4
6
6
6
.257
.251
.273
.217
.230
.246
.283
.305
.271
.263
.410
.456
.241
Dorsolateral PFC
2
4
4
4
4
6
6
6
6
6
6
6
6
6
6
6
6
9
9
9
.280
.350
.391
.352
.349
.369
.279
.602
.539
.409
.308
.268
.302
.291
.397
.465
.336
.231
.247
.215
.021
.025
.006
.044
.031
.011
.002
.003
.005
.004
< .001
< .001
.009
.012
.003
.001
.003
.003
.001
.013
< .001
< .001
< .001
.005
.022
.005
.005
< .001
< .001
.001
.013
.008
.025
−32.42
−32.85
22.62
35.49
22.39
25.13
33.78
16.26
20.11
19.01
−14.80
−17.17
−18.54
−17.61
−33.69
−15.30
−16.88
−29.53
−21.11
−20.60
45.28
44.06
47.55
34.05
36.55
39.42
−45.65
−0.92
26.82
32.04
33.25
−32.37
30.23
32.44
37.97
37.01
36.53
36.32
−40.57
−26.79
−40.40
−37.94
−45.67
−52.64
18.2
−37.13
−42.09
41.96
23.65
23.40
25.08
40.62
43.05
1.86
7.27
13.01
18.54
13.40
29.02
27.15
−12.53
−12.97
−13.45
7.735
14.50
15.50
−13.84
−5.65
−18.68
−14.58
−9.728
−13.12
−11.82
−22.26
−19.80
−8.65
−27.56
−20.63
−12.64
32.36
51.74
59.05
63.07
60.27
−2.91
4.80
30.73
36.46
41.85
47.24
44.16
26.59
30.77
42.31
43.13
43.82
49.02
52.60
53.60
10
10
.245
.258
p
.024
.013
x
21.50
−35.98
y
39.55
13.74
z
32.72
25.19
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
e
d
u
/
j
/
o
c
n
a
r
t
i
c
e
-
p
d
l
f
/
/
/
/
3
1
6
8
7
4
1
7
8
8
4
5
0
/
j
o
c
n
_
a
_
0
1
3
9
2
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
The table lists the patient’s number, Spearman correlation coefficient r,
cluster-corrected p value, and the MNI coordinates of the electrode
in millimeters and RAS (positive x = right, positive y = anterior, and
positive z = superior). The electrodes are sorted according to their
localization into the hippocampus, OFC, or PFC, as observed in the
MRI scan in native space.
a linear mixed-effects model to test the effect of novel
(n = 13) or anticipated periods of salient events (n =
11; rep = 0 for novel and 1 for repetitive storylines;
Figure 4) from the three ROIs (the correlation coeffi-
cients [R] in each electrode region; the hippocampus,
OFC, and dorsolateral PFC). We used the linear mixed-
effects model in MATLAB ( fitlme) to account for the dif-
ferent number of electrodes in each ROI of a subject and
the nested effect of time: formulated as R ∼ rep + (rep |
subject : electrodes) + (1 | rep : time_bin). ANOVA was
applied for the result of F tests for the fixed-effect term in
the linear mixed-effects model.
Electrode Localization and Visualization
Electrode locations were reconstructed and visualized in
MATLAB using the Fieldtrip toolbox (Stolk et al., 2018).
We manually selected electrodes on the postimplantation
CT, which was coregistered to the preimplantation MRI
using SPM (Ashburner & Friston, 1997) to maximize the
accuracy of the reconstructions. A neurologist identified
the electrodes’ locations. We then normalized each pa-
tient’s preimplantation MRI to the MNI-152 template
brain using SPM to obtain the electrode positions in
MNI space (Ashburner & Friston, 1999). If electrode loca-
tions in MNI space did not correspond to electrode loca-
tions in native (participant) space after normalization (e.g.,
an electrode is within hippocampus in native space but
appears outside the hippocampus in MNI space after nor-
malization), then electrode locations were manually ad-
justed to represent their true locations in native space.
Electrode locations for bipolar re-referenced channels
were calculated as the midpoint between the two elec-
trodes (Burke et al., 2013, 2014; Long, Burke, & Kahana,
2014). Representations of the cerebral cortex were gener-
ated using FreeSurfer (Dale, Fischl, & Sereno, 1999) and
representations of the hippocampus were generated from
the Desikan–Killiany atlas (Desikan et al., 2006) using
Fieldtrip. Brodmann’s areas were inferred from Bioimage
Suite package (bioimagesuite.yale.edu/).
RESULTS
We identified the magnitude of salient events in the
movie by studying a separate group of adults (n = 80).
Jafarpour et al.
879
This group indicated when, during the movie, a new ep-
isode started (i.e., perceiving an event boundary). The
metric of event saliency magnitude was the proportion
of identified event boundaries in a short time window
of the movie (about 20 frames or 1.5 sec; Figure 1B; total
movie time was 3 min). Epochs of 1.5 sec resulted in sa-
liency magnitude of 0–0.6 (1 would be the maximum sa-
liency magnitude when every participant agrees that,
during the same 1.5 sec, an event boundary occurred).
A distance matrix was constructed from the sum of the
saliency of events that occurred between pairs of events
and was used for binary hierarchical event clustering (see
the Methods section; Figure 1). We tested the hypothesis
that the magnitude of event saliency was tracked in the
targeted regions. Ranked (Spearman) correlation and
nonparametric permutation tests for statistical results
were applied. The interchangeable nonoverlapping HFA
in 1.5-sec epochs allowed using nonparametric permuta-
tion testing. Cluster-corrected p values are reported.
We observed that HFA correlated with the magnitude
of event saliency was captured by the behavior of the in-
dependent rating group. The neural effect was clustered
in dorsolateral PFC (BA 6, BA 8, BA 9, and BA 10; in five of
seven patients; seven of nine patients with lateral PFC
electrodes had dorsolateral PFC coverage). The effect
was also detected in the hippocampus (BA 54; in three
of three patients) and the medial OFC (BA 11; in three
of three patients with medial OFC coverage; three of
seven patients with OFC electrodes had medial PFC cover-
age; Figure 3). See Figure 3 for the correlation coefficient
of all electrodes and Table 2 for statistical results of elec-
trodes that showed a significant correlation (the R value
of electrodes with p > .05 is color-coded in Figure 3).
We conducted a planned analysis on the electrodes
that showed a significant correlation (d.h., task relevant)
to assess the effects of anticipation (Tisch 2). We recal-
culated the correlation coefficient between HFA and
event saliency magnitude in 15-sec-long sliding windows
(7.5-sec overlaps), ergebend 24 tested windows, von
welche 11 had repetitive storylines and 13 were novel.
The flow of salient events in 46% of the sliding windows
was anticipated. A storyline was predictable if the same
type of event reoccurred more than twice, such as rep-
etition of animals going back and forth or repetition of
scoring in a competition. The results of a linear mixed-
effect model showed that the correlation coefficient
between HFA and saliency magnitude was higher in
the OFC when the salient events were anticipated than
Figur 4. Representation of
event saliency during repetitive
and novel storylines. HFA–
saliency magnitude correlation
coefficients (R) in sliding
windows of 15 Sek (with 7.5-sec
overlaps) throughout the
movie. (A) The solid lines show
the mean R in the hippocampus
(in magenta), medial OFC (In
Blau), and dorsolateral PFC (In
cyan) in the y-axis. The x-axis is
the time in minutes. Der
shaded lines show the SEM. Der
dashed line shows 1 für die
periods with repetitive
storylines (anticipated salient
Veranstaltungen) Und 0 for novel periods.
(B) The R across the time bins is
color-coded in each electrode
that was included in the
planned test. The x-axis is the
time in minutes. Each row
shows R in an electrode. Der
patients’ number, showing the
owner of the electrode, Ist
written on the y-axis. The R in
the OFC was higher for
anticipated salient events than
for novel events ( P < .05).
880
Journal of Cognitive Neuroscience
Volume 31, Number 6
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
e
d
u
/
j
/
o
c
n
a
r
t
i
c
e
-
p
d
l
f
/
/
/
/
3
1
6
8
7
4
1
7
8
8
4
5
0
/
j
o
c
n
_
a
_
0
1
3
9
2
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
when they were novel (Figure 4A; OFC, F(1, 142) = 4.3,
p = .039), and this effect was observed in all patients
with task-relevant electrodes (Figure 4B). There was
no difference between novel and anticipated salient
events in the hippocampus, F(1, 166) = 0.39, p = .52,
or the dorsolateral PFC, F(1, 526) = 1.26, p = .26.
DISCUSSION
During encoding, a sequence of events is segmented to
construct a hierarchical representation of event associa-
tions (Kurby & Zacks, 2008; Zacks & Swallow, 2007;
Zwaan & Radvansky, 1998), with clusters of associated
events represented in lower levels of a hierarchy and
the associations of the clusters of events represented in
higher levels of the hierarchy. The construction of such a
hierarchical association requires linking relevant events
and separating events that occur in different circum-
stances. For instance, a circumstance changes with per-
ceiving a deviant event. Detecting the magnitude of
event saliency can also contribute to establishing the struc-
ture of associations. When the newly perceived event is
not salient, the event is closely associated with the pre-
ceding events; however, if the new event is highly salient,
it should be separated from the preceding events. Here,
we report distributed neural regions that detect the mag-
nitude of deviance (i.e., event saliency) in a flow of events,
including dorsolateral PFC and hippocampus, and further
show that anticipating the deviant events affects the OFC
activities.
We used event segmentations of a large control popu-
lation (behavior group) who watched the silent movie to
infer the event segmentation in another group with intra-
cranial electrodes (iEEG group) who watched the same
movie. The behavior group’s event segmentation pro-
vided the event saliency of the entire movie. The RT of
each participant in this group was estimated from a target
detection task and used for normalizing the timing of
event boundaries (see the Methods section). We inferred
the saliency from the proportion of people that reported
an event boundary in each movie epoch. The 1.5-sec win-
dows provided interchangeable epochs of data for using
correlation and permutation tests (see the Methods
section).
The iEEG group passively watched a movie and did not
know about the segmentation task, allowing us to study
spontaneous and naturalistic neural processing during
parsing a continuous flow of events. We observed that
the HFA that is linked to nearby single neural activity
(Rich & Wallis, 2017; Lachaux et al., 2012; Jacobs &
Kahana, 2009; Belitski et al., 2008; Ray, Crone, et al.,
2008) increased proportionally with event saliency in
the hippocampus, dorsolateral PFC, and medial OFC.
HFA in the dorsolateral PFC in the iEEG group tracked
event saliency magnitude. Dorsolateral PFC is critical for
guiding attention (Corbetta & Shulman, 2002; Hopfinger,
Buonocore, & Mangun, 2000; Kastner, Pinsk, De Weerd,
Desimone, & Ungerleider, 1999; Paus, 1996), and in-
creased HFA may, in part, be due to attention to novel
events (Ray, Niebur, Hsiao, Sinai, & Crone, 2008; Zacks
et al., 2001). Dorsolateral PFC is also engaged in cogni-
tive control and conflict monitoring (Miller & Cohen,
2001; MacDonald, Cohen, Stenger, & Carter, 2000) by de-
tecting new associations of categories and exemplars
(Dolan & Fletcher, 1997). Accordingly, the observed ad-
ditional correlation between HFA and event saliency in
dorsolateral PFC reflects the demand for event segmen-
tation and updating the event circumstance (Reynolds,
Zacks, & Braver, 2007; Zacks et al., 2007).
Event saliency also correlated with HFA in the hippo-
campus. Hippocampal activity has been linked to the rep-
resentation of event associations (Mack, Love, & Preston,
2017; Quiroga, 2012; Quiroga, Reddy, Kreiman, Koch, &
Fried, 2005; Ekstrom et al., 2003). The hippocampal rep-
resentation changes with salient changes in the environ-
ment (Shapiro, Tanila, & Eichenbaum, 1997), and its
activity increases with detecting salient events (Chen,
Cook, & Wagner, 2015; Chen et al., 2013; Axmacher
et al., 2010; Kumaran & Maguire, 2007; Wittmann,
Bunzeck, Dolan, & Düzel, 2007; Lisman & Otmakhova,
2001; Knight, 1996). Recent studies showed that hippo-
campal representations reflect the scale of topological sa-
liencies of an environment, such as changes in the spatial
closeness of streets or the centrality of the streets (Javadi
et al., 2017), and the scale of deviance from expectation
(Chen et al., 2015). Also, the hippocampal BOLD signal
tracked the saliency of event boundaries when people
watched movies (Ben-Yakov & Henson, 2018). Here,
we propose that small deviance induces only a minor
change in the hippocampal representation so that close
events share more similar hippocampus representations
than far events (Ezzyat & Davachi, 2014). These results
expand the pattern separation mechanism for distin-
guishing similar visual associations attributed to the
hippocampus (Yassa & Stark, 2011) to a mechanism for
identifying the scale of event separation. Pattern separa-
tion for visual stimuli engages dentate gyrus in the hippo-
campus (Baker et al., 2016; Berron et al., 2016), but what
the subregion of the human hippocampus contributes to
the deviant detection is unknown (see Lisman & Grace,
2005, for the novelty signal in the rodent’s subiculum and
Knierim & Neunuebel, 2016, for mismatch signal in
subregions of rodent’s hippocampus).
We also observed a similar saliency magnitude effect
in the medial OFC (BA 11 but not in the lateral OFC),
with increased HFA for highly salient events. This obser-
vation is akin to the representation of saliency in the
nonhuman primates’ OFC, captured by HFA (Rich &
Wallis, 2016, 2017). In humans, breaching expectations in-
creases the OFC activity (Mikutta et al., 2015; Duarte,
Henson, Knight, Emery, & Graham, 2009; Nobre et al.,
1999). OFC also represents the saliency of anticipated
events (Metereau & Dreher, 2015; Bechara, Tranel,
Damasio, & Damasio, 1996). The reflection of the saliency
Jafarpour et al.
881
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
e
d
u
/
j
/
o
c
n
a
r
t
i
c
e
-
p
d
l
f
/
/
/
/
3
1
6
8
7
4
1
7
8
8
4
5
0
/
j
o
c
n
_
a
_
0
1
3
9
2
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
magnitude suggests that OFC represents the structure of
the event association. Notably, anticipation is a critical
feature for encoding sequences of events because the
gist of a previous experience shapes the expected con-
text (Reynolds et al., 2007; Purcell, 1986). Here, the
HFA in medial OFC tracked the event saliency better
when the salient events were anticipated than during a
novel intrusion of events, suggesting that the anticipated
structure of event associations is represented in the
medial OFC.
An important question concerns the dynamics of inter-
action in the neural network for representing the structure
of event associations. For example, disturbing the input
from the hippocampus to OFC impairs representing task
structures in rodents ( Wikenheiser, Marrero-Garcia, &
Schoenbaum, 2017). It is not clear whether hippocampal
deviancy detection is essential for constructing the OFC
signal in humans or whether other brain regions such as
midbrain structures contribute to detecting the magnitude
of deviance (Dürschmid et al., 2016; Wittmann et al., 2007).
For instance, hippocampal response to deviant events is
associated with activity in the substantia nigra and ventral
tegmental area (Murty & Adcock, 2014; Wittmann et al.,
2007). It is also suggested that the hippocampus provides
a novelty signal to the nucleus accumbens (Dürschmid
et al., 2016). Although hippocampus activity is linked to
anticipation ( Jafarpour et al., 2017; Hindy et al., 2016;
Hsieh et al., 2014), an outstanding question is whether
the prediction is made by the hippocampus or is under
control of other brain regions, such as the PFC. System-
atically, comparison of regional activity, however, re-
quires simultaneous recordings from both regions in a
patient. A caveat of this iEEG study is that all the pa-
tients did not have sufficient coverage from multiple
task-relevant regions to definitively study the dynamics
of the network.
Event segmentation requires tracing the association
of a new event to the preceding events (Zwaan &
Radvansky, 1998). Detecting a new event’s deviance mag-
nitude helps with an accurate association of the event to
the preceding sequence. Accordingly, segmented se-
quences that are separated by small surprises are more
associated in comparison to a sequence separated by a
big surprise. Detecting deviant events is known to in-
crease neural activity in the PFC and the hippocampus
(Long, Lee, & Kuhl, 2016; Axmacher et al., 2010;
Bunzeck, Dayan, Dolan, & Duzel, 2010; Kumaran &
Maguire, 2007; Strange, Duggins, Penny, Dolan, &
Friston, 2005; Knight, 1996). Both brain regions, albeit
differently, represent the associations in an experimental
setup ( Wikenheiser & Schoenbaum, 2016; Wilson,
Takahashi, Schoenbaum, & Niv, 2014; O’Keefe & Nadel,
1978; Tolman, 1948). Here, we showed that the hippocam-
pus and PFC regions tracked the scale of event saliency in a
movie, and in the medial OFC, this effect is stronger when
the salient events were anticipated than for novel events.
We propose that a core function of the hippocampus,
dorsolateral PFC, and medial OFC network is to con-
struct the event association structure, akin to a task
structure ( Wikenheiser & Schoenbaum, 2016; Wilson
et al., 2014).
Acknowledgments
This research used statistical consulting resources provided by
the Center for Statistics and the Social Sciences, University of
Washington. This work was sponsored by the James S.
McDonnell Foundation, National Institute of Neurological
Disorders grant R37 NS21135 (to R. T. K.), and the University
of California, Irvine School of Medicine Bridge Fund (to J. J. L.).
The authors are indebted to the patients for their participation.
We thank Jie Zheng and members of the Knightlab for helping
with data collection. We also thank Prof. Elizabeth Buffalo for
helpful discussions.
Reprint requests should be sent to Anna Jafarpour, Department
of Physiology and Biophysics, University of Washington, 1705
NE Pacific St., HSB Box 357290, Seattle, WA 98195, or via e-mail:
annaja@uw.edu.
REFERENCES
Ashburner, J., & Friston, K. J. (1997). The role of registration
and spatial normalization in detecting activations in functional
imaging. Clinical MRI/Developments in MR, 7, 26–28.
Ashburner, J., & Friston, K. J. (1999). Nonlinear spatial
normalization using basis functions. Human Brain Mapping,
7, 254–266.
Axmacher, N., Cohen, M. X., Fell, J., Haupt, S., Dümpelmann, M.,
Elger, C. E., et al. (2010). Intracranial EEG correlates of expectancy
and memory formation in the human hippocampus and
nucleus accumbens. Neuron, 65, 541–549.
Baker, S., Vieweg, P., Gao, F., Gilboa, A., Wolbers, T., Black,
S. E., et al. (2016). The human dentate gyrus plays a
necessary role in discriminating new memories. Current
Biology, 26, 2629–2634.
Bechara, A., Tranel, D., Damasio, H., & Damasio, A. R. (1996).
Failure to respond autonomically to anticipated future
outcomes following damage to prefrontal cortex. Cerebral
Cortex, 6, 215–225.
Belitski, A., Gretton, A., Magri, C., Murayama, Y., Montemurro,
M. A., Logothetis, N. K., et al. (2008). Low-frequency local
field potentials and spikes in primary visual cortex convey
independent visual information. Journal of Neuroscience,
28, 5696–5709.
Ben-Yakov, A., & Henson, R. (2018). The hippocampal film-editor:
Sensitivity and specificity to event boundaries in continuous
experience. Journal of Neuroscience, 38, 10057–10068.
Berron, D., Schütze, H., Maass, A., Cardenas-Blanco, A., Kuijf,
H. J., Kumaran, D., et al. (2016). Strong evidence for pattern
separation in human dentate gyrus. Journal of
Neuroscience, 36, 7569–7579.
Brown, T. I., Carr, V. A., LaRocque, K. F., Favila, S. E., Gordon, A. M.,
Bowles, B., et al. (2016). Prospective representation of
navigational goals in the human hippocampus. Science,
352, 1323–1326.
Bunzeck, N., Dayan, P., Dolan, R. J., & Duzel, E. (2010). A
common mechanism for adaptive scaling of reward and
novelty. Human Brain Mapping, 31, 1380–1394.
Burke, J. F., Long, N. M., Zaghloul, K. A., Sharan, A. D., Sperling,
M. R., & Kahana, M. J. (2014). Human intracranial high-
frequency activity maps episodic memory formation in space
and time. Neuroimage, 85, 834–843.
882
Journal of Cognitive Neuroscience
Volume 31, Number 6
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
e
d
u
/
j
/
o
c
n
a
r
t
i
c
e
-
p
d
l
f
/
/
/
/
3
1
6
8
7
4
1
7
8
8
4
5
0
/
j
o
c
n
_
a
_
0
1
3
9
2
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
Burke, J. F., Zaghloul, K. A., Jacobs, J., Williams, R. B., Sperling,
M. R., Sharan, A. D., et al. (2013). Synchronous and
asynchronous theta and gamma activity during episodic
memory formation. Journal of Neuroscience, 33, 292–304.
Chen, J., Cook, P. A., & Wagner, A. D. (2015). Prediction
strength modulates responses in human area CA1 to
sequence violations. Journal of Neurophysiology, 114,
1227–1238.
Chen, J., Dastjerdi, M., Foster, B. L., LaRocque, K. F., Rauschecker,
A. M., Parvizi, J., et al. (2013). Human hippocampal
increases in low-frequency power during associative
prediction violations. Neuropsychologia, 51, 2344–2351.
Corbetta, M., & Shulman, G. L. (2002). Control of goal-directed
and stimulus-driven attention in the brain. Nature Reviews
Neuroscience, 3, 201–215.
Coulson, S., King, J. W., & Kutas, M. (1998). Expect the unexpected:
Event-related brain response to morphosyntactic violations.
Language and Cognitive Processes, 13, 21–58.
Dale, A. M., Fischl, B., & Sereno, M. I. (1999). Cortical surface-
based analysis, I. Segmentation and surface reconstruction.
Neuroimage, 9, 179–194.
Desikan, R. S., Ségonne, F., Fischl, B., Quinn, B. T., Dickerson,
B. C., Blacker, D., et al. (2006). An automated labeling system
for subdividing the human cerebral cortex on MRI scans into
gyral based regions of interest. Neuroimage, 31, 968–980.
Dolan, R. J., & Fletcher, P. C. (1997). Dissociating prefrontal and
hippocampal function in episodic memory encoding. Nature,
388, 582–585.
Duarte, A., Henson, R. N., Knight, R. T., Emery, T., & Graham,
K. S. (2009). Orbito-frontal cortex is necessary for temporal
context memory. Journal of Cognitive Neuroscience, 22,
1819–1831.
Dürschmid, S., Zaehle, T., Hinrichs, H., Heinze, H. J., Voges, J.,
Garrido, M. I., et al. (2016). Sensory deviancy detection
measured directly within the human nucleus accumbens.
Cerebral Cortex, 26, 1168–1175.
Ekstrom, A. D., Kahana, M. J., Caplan, J. B., Fields, T. A., Isham,
E. A., Newman, E. L., et al. (2003). Cellular networks
underlying human spatial navigation. Nature, 425, 184–188.
Ezzyat, Y., & Davachi, L. (2014). Similarity breeds proximity:
Pattern similarity within and across contexts is related to later
mnemonic judgments of temporal proximity. Neuron, 81,
1179–1189.
Hanson, C., & Hirst, W. (1989). On the representation of
events: A study of orientation, recall, and recognition.
Journal of Experimental Psychology. General, 118, 136–147.
Hindy, N. C., Ng, F. Y., & Turk-Browne, N. B. (2016). Linking
pattern completion in the hippocampus to predictive coding
in visual cortex. Nature Neuroscience, 19, 665–667.
Hopfinger, J. B., Buonocore, M. H., & Mangun, G. R. (2000).
The neural mechanisms of top–down attentional control.
Nature Neuroscience, 3, 284–291.
Hsieh, L. T., Gruber, M. J., Jenkins, L. J., & Ranganath, C. (2014).
Hippocampal activity patterns carry information about
objects in temporal context. Neuron, 81, 1165–1178.
Hsieh, L. T., & Ranganath, C. (2015). Cortical and subcortical
contributions to sequence retrieval: Schematic coding of
temporal context in the neocortical recollection network.
Neuroimage, 121, 78–90.
Jacobs, J., & Kahana, M. J. (2009). Neural representations of
individual stimuli in humans revealed by gamma-band
electrocorticographic activity. Journal of Neuroscience, 29,
10203–10214.
Jafarpour, A., Piai, V., Lin, J. J., & Knight, R. T. (2017). Human
hippocampal pre-activation predicts behavior. Scientific
Reports, 7, 5959.
Javadi, A.-H., Emo, B., Howard, L. R., Zisch, F. E., Yu, Y., Knight,
R., et al. (2017). Hippocampal and prefrontal processing of
network topology to simulate the future. Nature
Communications, 8, 14652.
Kastner, S., Pinsk, M. A., De Weerd, P., Desimone, R., &
Ungerleider, L. G. (1999). Increased activity in human visual
cortex during directed attention in the absence of visual
stimulation. Neuron, 22, 751–761.
Knierim, J. J., & Neunuebel, J. P. (2016). Tracking the flow of
hippocampal computation: Pattern separation, pattern
completion, and attractor dynamics. Neurobiology of
Learning and Memory, 129, 38–49.
Knight, R. (1996). Contribution of human hippocampal region
to novelty detection. Nature, 383, 256–259.
Kumaran, D., & Maguire, E. A. (2007). Match mismatch
processes underlie human hippocampal responses to
associative novelty. Journal of Neuroscience, 27,
8517–8524.
Kurby, C. A., & Zacks, J. M. (2008). Segmentation in the
perception and memory of events. Trends in Cognitive
Sciences, 12, 72–79.
Lachaux, J. P., Axmacher, N., Mormann, F., Halgren, E., &
Crone, N. E. (2012). High-frequency neural activity and
human cognition: Past, present and possible future of
intracranial EEG research. Progress in Neurobiology, 98,
279–301.
Lisman, J. E., & Grace, A. A. (2005). The hippocampal-VTA
loop: Controlling the entry of information into long-term
memory. Neuron, 46, 703–713.
Lisman, J. E., & Otmakhova, N. A. (2001). Storage, recall, and
novelty detection of sequences by the hippocampus:
Elaborating on the SOCRATIC model to account for normal
and aberrant effects of dopamine. Hippocampus, 11,
551–568.
Long, N. M., Burke, J. F., & Kahana, M. J. (2014). Subsequent
memory effect in intracranial and scalp EEG. Neuroimage,
84, 488–494.
Long, N. M., Lee, H., & Kuhl, B. A. (2016). Hippocampal
mismatch signals are modulated by the strength of neural
predictions and their similarity to outcomes. Journal of
Neuroscience, 36, 12677–12687.
MacDonald, A. W., III, Cohen, J. D., Stenger, V. A., & Carter, C. S.
(2000). Dissociating the role of the dorsolateral prefrontal
and anterior cingulate cortex in cognitive control. Science,
288, 1835–1838.
Mack, M. L., Love, B. C., & Preston, A. R. (2017). Building
concepts one episode at a time: The hippocampus and
concept formation. Neuroscience Letters, 680, 31–38.
Metereau, E., & Dreher, J. C. (2015). The medial orbitofrontal
cortex encodes a general unsigned value signal during
anticipation of both appetitive and aversive events. Cortex,
63, 42–54.
Mikutta, C. A., Dürschmid, S., Bean, N., Lehne, M., Lubell, J.,
Altorfer, A., et al. (2015). Amygdala and orbitofrontal
engagement in breach and resolution of expectancy: A case
study. Psychomusicology: Music, Mind, and Brain, 25,
357–365.
Miller, E. K., & Cohen, J. D. (2001). An integrative theory of
prefrontal cortex function. Annual Review of Neuroscience,
24, 167–202.
Murty, V. P., & Adcock, R. A. (2014). Enriched encoding: Reward
motivation organizes cortical networks for hippocampal
detection of unexpected events. Cerebral Cortex, 24,
2160–2168.
Nobre, A. C., Coull, J. T., Frith, C. D., & Mesulam, M. M. (1999).
Orbitofrontal cortex is activated during breaches of
expectation in tasks of visual attention. Nature
Neuroscience, 2, 11–12.
O’Keefe, J., & Nadel, L. (1978). The hippocampus as a
cognitive map. Oxford: Oxford University Press.
Jafarpour et al.
883
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
e
d
u
/
j
/
o
c
n
a
r
t
i
c
e
-
p
d
l
f
/
/
/
/
3
1
6
8
7
4
1
7
8
8
4
5
0
/
j
o
c
n
_
a
_
0
1
3
9
2
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
Oostenveld, R., Fries, P., Maris, E., & Schoffelen, J.-M. (2011).
Stolk, A., Griffin, S., van der Meij, R., Dewar, C., Saez, I., Lin, J. J.,
FieldTrip: Open source software for advanced analysis of MEG,
EEG, and Invasive electrophysiological data. Computational
Intelligence and Neuroscience, 2011, 156869.
Paus, T. (1996). Location and function of the human frontal eye-
field: A selective review. Neuropsychologia, 34, 475–483.
Paz, R., Gelbard-Sagiv, H., Mukamel, R., Harel, M., Malach,
R., & Fried, I. (2010). A neural substrate in the human
hippocampus for linking successive events. Proceedings of
the National Academy of Sciences, U.S.A., 107, 6046–6051.
et al. (2018). Integrated analysis of anatomical and
electrophysiological human intracranial data. Nature
Protocols, 13, 1699–1723.
Strange, B. A., Duggins, A., Penny, W., Dolan, R. J., & Friston,
K. J. (2005). Information theory, novelty and hippocampal
responses: Unpredicted or unpredictable? Neural Networks,
18, 225–230.
Tolman, E. C. (1948). Cognitive maps in rats and men.
Psychological Review, 55, 189–208.
Purcell, A. T. (1986). Environmental perception and affect: A
Whitney, C., Huber, W., Klann, J., Weis, S., Krach, S., &
schema discrepancy model. Environment and Behavior, 18,
3–30.
Quiroga, R. Q. (2012). Concept cells: The building blocks of
Kircher, T. (2009). Neural correlates of narrative shifts
during auditory story comprehension. Neuroimage, 47,
360–366.
declarative memory functions. Nature Reviews
Neuroscience, 13, 587–597.
Quiroga, R. Q., Reddy, L., Kreiman, G., Koch, C., & Fried, I.
(2005). Invariant visual representation by single neurons in
the human brain. Nature, 435, 1102–1107.
Ray, S., Crone, N. E., Niebur, E., Franaszczuk, P. J., & Hsiao,
S. S. (2008). Neural correlates of high-gamma oscillations
(60–200 Hz) in Macaque local field potentials and their
potential implications in electrocorticography. Journal of
Neuroscience, 28, 11526–11536.
Ray, S., Niebur, E., Hsiao, S. S., Sinai, A., & Crone, N. E. (2008).
High-frequency gamma activity (80–150Hz) is increased in
human cortex during selective attention. Clinical
Neurophysiology, 119, 116–133.
R Development Core Team. (2014). R: A language and
environment for statistical computing. Vienna, Austria: R
Foundation for Statistical Computing. Retrieved from https://
www.R-project.org/.
Reynolds, J. R., Zacks, J. M., & Braver, T. S. (2007). A
computational model of event segmentation from perceptual
prediction. Cognitive Science, 31, 613–643.
Rich, E. L., & Wallis, J. D. (2016). What stays the same in
orbitofrontal cortex. Nature Neuroscience, 19, 768–770.
Rich, E. L., & Wallis, J. D. (2017). Spatiotemporal dynamics of
information encoding revealed in orbitofrontal high-gamma.
Nature Communications, 8, 1139.
Rosani, A., Boato, G., & De Natale, F. G. B. (2015).
EventMask: A game-based framework for event-saliency
identification in images. IEEE Transactions on
Multimedia, 17, 1359–1371.
Schütz-Bosbach, S., & Prinz, W. (2007). Prospective coding in
event representation. Cognitive Processing, 8, 93–102.
Shapiro, M. L., Tanila, H., & Eichenbaum, H. (1997). Cues that
hippocampal place cells encode: Dynamic and hierarchical
representation of local and distal stimuli. Hippocampus, 7,
624–642.
Speer, N. K., Zacks, J. M., & Reynolds, J. R. (2007). Human brain
activity time-locked to narrative event boundaries.
Psychological Science, 18, 449–455.
Wikenheiser, A. M., & Schoenbaum, G. (2016). Over the river,
through the woods: Cognitive maps in the hippocampus and
orbitofrontal cortex. Nature Reviews Neuroscience, 17,
513–523.
Wikenheiser, A. M., Marrero-Garcia, Y., & Schoenbaum, G.
(2017). Suppression of ventral hippocampal output impairs
integrated orbitofrontal encoding of task structure. Neuron,
95, 1197.e3–1207.e3.
Wilson, R. C., Takahashi, Y. K., Schoenbaum, G., & Niv, Y.
(2014). Orbitofrontal cortex as a cognitive map of task space.
Neuron, 81, 267–279.
Wittmann, B. C., Bunzeck, N., Dolan, R. J., & Düzel, E.
(2007). Anticipation of novelty recruits reward system and
hippocampus while promoting recollection. Neuroimage,
38, 194–202.
Yassa, M. A., & Stark, C. E. (2011). Pattern separation in the
hippocampus. Trends in Neurosciences, 34, 515–525.
Yeung, M., Yeo, B.-L., & Liu, B. (1996). Extracting story units
from long programs for video browsing and navigation.
In Proceedings of the Third IEEE International Conference
on Multimedia Computing and Systems, pp. 296–305.
Zacks, J. M., Braver, T. S., Sheridan, M. A., Donaldson, D. I.,
Snyder, A. Z., Ollinger, J. M., et al. (2001). Human brain
activity time-locked to perceptual event boundaries. Nature
Neuroscience, 4, 651–655.
Zacks, J. M., Speer, N. K., Swallow, K. M., Braver, T. S., &
Reynolds, J. R. (2007). Event perception: A mind-brain
perspective. Psychological Bulletin, 133, 273–293.
Zacks, J. M., & Swallow, K. M. (2007). Event segmentation.
Current Directions in Psychological Science, 16, 80–84.
Zarcone, A., van Schijndel, M., Vogels, J., & Demberg, V. (2016).
Salience and attention in surprisal-based accounts of
language processing. Frontiers in Psychology, 7, 844.
Zhang, D., Han, J., Jiang, L., Ye, S., & Chang, X. (2017).
Revealing event saliency in unconstrained video collection.
IEEE Transactions on Image Processing, 26, 1746–1758.
Zwaan, R. A., & Radvansky, G. A. (1998). Situation models in
language comprehension and memory. Psychological
Bulletin, 123, 162–185.
884
Journal of Cognitive Neuroscience
Volume 31, Number 6
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
e
d
u
/
j
/
o
c
n
a
r
t
i
c
e
-
p
d
l
f
/
/
/
/
3
1
6
8
7
4
1
7
8
8
4
5
0
/
j
o
c
n
_
a
_
0
1
3
9
2
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
