Tracking Your Mind’s Eye during Recollection: - IA de Investigación especializada en el MIT

Tracking Your Mind’s Eye during Recollection:
Decoding the Long-Term Recall of
Short Audiovisual Clips

Christelle Larzabal1,2, Nadège Bacon-Macé1,2,
Sophie Muratot1,2, and Simon J. Thorpe1,2

D
oh
w
norte
yo
oh
a
d
mi
d

Abstracto

■ Unlike familiarity, recollection involves the ability to recon-
struct mentally previous events that results in a strong sense of
reliving. According to the reinstatement hypothesis, this spe-
cific feature emerges from the reactivation of cortical patterns
involved during information exposure. Over time, the retrieval
of specific details becomes more difficult, and memories be-
come increasingly supported by familiarity judgments. The mul-
tiple trace theory (MTT) explains the gradual loss of episodic
details by a transformation in the memory representation, a
view that is not shared by the standard consolidation model.
en este estudio, we tested the MTT in light of the reinstatement
hypothesis. The temporal dynamics of mental imagery from

long-term memory were investigated and tracked over the pas-
sage of time. Participant EEG activity was recorded during the
recall of short audiovisual clips that had been watched 3 semanas,
1 día, or a few hours beforehand. The recall of the audiovisual
clips was assessed using a Remember/Know/New procedure,
and snapshots of clips were used as recall cues. The decoding
matrices obtained from the multivariate pattern analyses re-
vealed sustained patterns that occurred at long latencies
(>500 msec poststimulus onset) that faded away over the reten-
tion intervals and that emerged from the same neural processes.
En general, our data provide further evidence toward the MTT and
give new insights into the exploration of our “mind’s eye.” ■

INTRODUCCIÓN

Episodic memory involves the recollection of a unique
event that occurred in a specific context. It is different
from semantic memory, which reflects the ability to re-
trieve general concepts in the absence of contextual
details (Tulving, 1972). The Remember/Know (R/ K)
paradigm was originally developed by Tulving (1985) a
distinguish between episodic (“Remember” responses)
and semantic memories (“Know” responses). Over time
and with its extensive use in recognition memory,
“Remember” and “Know” responses became associated
with recollection and familiarity processes, respectivamente
( Yonelinas, 2002). A key difference between “remember-
ing” and “knowing” lies in the ability to create mental im-
ages of the information to be retrieved. En efecto, episodic
memories would be associated with conscious mental im-
siglos, an attribute that would not be shared by semantic
memories (Gardiner & Richardson-Klavehn, 2000;
Cervecero & Pani, 1996; Tulving, 1983).

According to the reinstatement hypothesis, the re-
trieval of specific episodic details would emerge from
the reactivation of cortical patterns that were involved
during the encoding of the event (Rugg, Johnson, Parque,

1Université de Toulouse, 2CNRS, Tolosa

& Uncapher, 2008; Damasio, 1989). Much experimental
evidence supports the reinstatement hypothesis by
showing that the brain regions that are active during ep-
isodic retrieval overlap with those that were involved at
the time of the encoding of the information (para una revisión,
see Danker & anderson, 2010). In the absence of sensory
información, top–down mechanisms originating from
the prefrontal and lateral parietal cortex would trigger
reactivations in the sensory areas (Dijkstra, Zeidman,
Ondobaka, Van Gerven, & Friston, 2017; Mechelli,
Precio, Friston, & Ishai, 2004; Ishai, Ungerleider, &
Haxby, 2000). Por lo tanto, the retrieval of content-specific
information from the sensory cortical regions ( Johnson
& Rugg, 2007; Woodruff, Johnson, Uncapher, & Rugg,
2005; Nyberg, Habib, McIntosh, & Tulving, 2000; Rodador,
Petersen, & Buckner, 2000) mediated by content-
independent retrieval effects observed in the medial-
temporal lobe, in the pFC, and in the lateral parietal cortex
would contribute to the reactivation of a unitary episodic
representation that would be directly accessible to
consciousness (Rugg & Vilberg, 2013).

En tono rimbombante, the retrieved episodic content is not an
exact copy of the encoded information, and some trans-
formations are already at hand a day after exposure (xiao
et al., 2017). Important reorganizations occur at the sys-
tem level, resulting in the gradual disengagement of the

© 2019 Instituto de Tecnología de Massachusetts. Published under a
Creative Commons Attribution 4.0 Internacional (CC POR 4.0) licencia.

Revista de neurociencia cognitiva 32:1, páginas. 50–64
https://doi.org/10.1162/jocn_a_01468

/
j

t
t

F
/

i
t
.

:
/
/

F
r
oh
metro
D
oh
h
w
t
norte
t
pag
oh
:
a
/
d
/
mi
d
metro
i
F
r
t
oh
pag
metro
r
C
h
.
pag
s
i
yo
d
v
i
r
mi
mi
r
C
t
C
.
metro
h
a
i
mi
r
d
.
tu
C
oh
oh
metro
C
norte
/
j
a
oh
r
t
C
i
C
norte
mi
/
–
a
pag
r
d
t
i
3
2
C
yo
1
mi
5
–
0
pag
d
1
9
F
3
/
1
3
1
2
9
/
9
1
/
oh
5
C
0
norte
/
_
a
1
_
8
0
6
1
1
4
4
6
1
8
0
pag
/
d
j
oh
b
C
y
norte
gramo
_
tu
a
mi
_
s
0
t
1
oh
4
norte
6
0
8
8
.
pag
S
d
mi
F
pag
mi
b
metro
y
b
mi
gramo
r
tu
2
mi
0
s
2
t
3

/
j

oh
norte

0
5

METRO
a
y

2
0
2
1

hippocampus within a few weeks of encoding (Frankland
& Bontempi, 2005). This is reflected in terms of subjec-
tive reports where retrieval becomes increasingly sup-
ported by familiarity judgments (Piolino, Desgranges,
& Eustache, 2009; Herbert & Burt, 2004; Conway,
Gardiner, Perfect, anderson, & cohen, 1997). The grad-
ual loss of episodic details is explained by the multiple
trace theory (MTT) under the transformation hypothesis
(para una revisión, see Winocur & Moscovitch, 2011). It posits
that episodic memories fade away over time in such a
way that most of the transformed memories contain no
more than a schematic version of the original memory:
the “gist.” This view differs from the standard consol-
idation model (Squire & Alvarez, 1995) para cual (1)
episodic memories and semantic memories are con-
sidered as two separate entities and (2) episodic
memories would not undergo a qualitative change
over the consolidation process. En tono rimbombante, a pesar de
these “semanticized” memories would not rely on the
hippocampus, remote memories that contain some
episodic details would still be hippocampus dependent
according to the MTT. De nuevo, this conclusion is not
shared by the standard consolidation model for which
both remote semantic and episodic memories would
not require hippocampal activations.

The reinstatement hypothesis suggests that cortical
reinstatement is specific to recollection. Hasta ahora, sin embargo,
the patterns of memory reinstatement found in the fMRI
studies were similar for recollection and familiarity judg-
ments in R/K paradigms ( Thakral, Wang, & Rugg, 2017;
Johnson, McDuff, Rugg, & Norman, 2009). Esto indica
that the observed spatial patterns could not differentiate
between the two distinct phenomenological experiences
of remembering and knowing. en este estudio, we decided
to focus on the temporal patterns associated to memory
reinstatement as specific ERP signatures have been al-
ready identified.

The late positive component (LPC) kicks in between
500 y 800 msec in the left parietal region and has been
linked to recollection. Converging evidence suggests that
activations of the inferior parietal cortex correspond to
the LPC (para una revisión, see Vilberg & Rugg, 2008).
Curiosamente, the LPC persists over retention intervals
ranging from several seconds (Nessler & Mecklinger,
2003) to several minutes and up to 1 día (Wolk et al.,
2006; Curran & Friedman, 2004) but significantly de-
creases after a retention interval of a week (Roberts,
Tsivilis, & Mayes, 2013) hasta 4 semanas (Tsivilis et al.,
2015). This late ERP component is generally contrasted
with the FN400, a midfrontal negativity occurring within
300–500 msec after stimulus onset, which has been linked
to familiarity judgment (Duarte, Ranganath, Winward,
Hayward, & Caballero, 2004; Curran, 2000; Tsivilis, Otten,
& Rugg, 2001; Düzel, Yonelinas, Mangun, Heinze, &
Tulving, 1997; para una revisión, see Rugg & Curran, 2007).
Unlike the LPC, the FN400 appears insensitive to long
delays.

ERP data provide meaningful information concerning
the time course of long-term retrieval processes but do
not propose an integrated view of the complete dynam-
ics during the reinstatement of episodic memories.
Multivariate pattern analyses (MVPAs) on time-resolved
signals have been proposed in this sense (Rey &
Dehaene, 2014). This decoding technique was recently
used to characterize the temporal dynamics associated
with mental
imagery arising from STM (Dijkstra,
Mostert, DeLange, Bosh, & van Gerven, 2018). Data re-
vealed a late sustained pattern that relied on activations
in the sensory areas as well as in the frontal and parietal
regiones. This sustained pattern suggests that a single pro-
cess was involved during mental imagery reinstatement
and contrasted with the sequential ordering observed
during perception.

en este estudio, we decided to test the transformation
hypothesis proposed by the MTT in light of the cortical
reinstatement theory. Para tal fin, we proposed to char-
acterize the temporal dynamics of mental imagery from
long-term memories and track these dynamics as memo-
ries aged. Three predictions were made: (1) the recollec-
tion of episodic details should be associated with a clear
pattern like the one obtained during mental imagery
from STM (Dijkstra et al., 2018), (2) this pattern should
vanish over the passage of time reflecting the loss of ep-
isodic details, y (3) this recollection pattern should rely
on the same neural processes irrespective of the age of
the memory. Para tal fin, we investigated participant
EEG activity (norte = 11) during the recall of short audiovi-
sual clips seen 3 semanas, 1 día, and a few hours before-
mano. Recall was elicited by a short presentation of
snapshots of previously seen or new audiovisual clips
and assessed by a Remember/Know/New (R/K/N) para-
digm. The reason audiovisual clips were used was to en-
able participants to perform rich and dynamic mental
imagery. ERP and decoding analyses were performed.

MÉTODOS

Participantes

D
oh
w
norte
yo
oh
a
d
mi
d

/
j

t
t

F
/

i
t
.

:
/
/

/
j

Eleven healthy participants (six women, edad media =
24 años, DE = 1.9 años) gave written informed consent
and participated in the experiment approved by the
INSERM ethical evaluation committee.

oh
norte

0
5

METRO
a
y

2
0
2
1

Estímulos

Seven-second audiovisual clips (norte = 750) downloaded
from the Internet were used in the experiment. The clips
contained no speech and were selected to be diversified
and unknown to the participants. Half of them were
shown to a group of participants (norte = 6) durante el
learning sessions, and the other half of the clips were
shown to the second group of participants (norte = 5). Para
all the audiovisual clips, one frame of the 7-sec clip was

Larzabal et al.

chosen as a representative snapshot of the clip and pre-
sented during the recall phase. The same 750 snapshots
were presented for all the participants and counterba-
lanced between participant groups.1

Procedimiento

The experiment included two phases: an explicit learning
phase during which audiovisual clips were presented to
the participants and a recall test of these clips (Cifra 1).
The explicit learning phase was composed of three view-
ing sessions occurring respectively 3 semanas (3W., significar
interval = 21 días 2 hr 43 mín., DE = 16 hr 54 mín.),
1 día (1D, mean interval = 1 día 2 hr 21 mín., DE = 2 hr
39 mín.), and several hours (HOURS, mean interval = 4 hr
37 mín., DE = 1 hr 6 mín.) before the recall test. In every
session, 125 audiovisual clips (duration = 7 segundo, size =
320 × 240) were shown to participants (norte = 11) en un
computer screen. Every session was split in two blocks
de 65 y 60 videos. All the clips were presented twice
within a block but were not presented again in any other
session blocks. To control for participant attention dur-
ing viewing, participants were asked to make a button-
press response every time a clip was shown for the
second time. Participants could also indicate that they al-
ready knew a clip by pressing a key assigned to this pur-
pose during the clip presentation, in which case the
corresponding trials were discarded from further analysis.
En general, participants watched 375 audiovisual clips,

presented twice, during the three learning sessions. El
duration of a learning session was about 50 mín.. Partic-
ipants were instructed that a recall test on the audiovisual
clips would be performed 3 weeks after the first learning
session. The testing phase performed 3 weeks after the first
learning session was a cued-recall task of 750 trials and
during which participant EEG activity was recorded. Cada
trial started with a fixation cross of varying duration (600–
1000 mseg) followed by either a snapshot from a clip seen
during the learning sessions (OLD, norte = 375) or a new
uno (NEW, norte = 375), which was presented for 400 mseg.
After picture offset, participants were asked to press one
of the three keys corresponding to Remember (“R”),
Know (“K”), and New (“N”) judgments (randomized be-
tween participants), as fast and as accurately as possible,
with the same hand. Before testing, participants were
instructed about the meaning of these three response
opciones. The definitions given to the participants were
based on those proposed by Gardiner and Richardson-
Klavehn (2000) and were the following: “R”: You identify
the picture as previously seen in one of the three learn-
ing sessions and you can replay the related clip in your
mind by anticipating the following scenes of the short
scenario; “K”: You identify the picture as previously seen
in one of the three learning sessions but you are unable
to retrieve the clip it belongs to; “N”: The picture does
not belong to any of the audiovisual clips shown during
the three learning sessions. Participants were then in-
vited to judge their response on a 5-point confidence scale

Cifra 1. Diseño experimental.
Short audiovisual clips were
shown to participants during
three learning sessions: 3 semanas
(Aprendiendo 1), 1 día (Aprendiendo 2),
and a few hours (Aprendiendo 3)
before the recall test. Para
each learning session, el
audiovisual clips (norte = 125) eran
shown twice on screen, y
participants were asked to make
a button-press response every
time a video was presented for
the second time. durante el
cued-recall test, snapshots of
clips seen during the learning
sessions (norte = 375) or new
unos (norte = 375) were briefly
presented on screen
(400 mseg). After picture
offset, participants were invited
to make an R/K/N judgment
as quickly and accurately
as possible and rate the
confidence in their response
on a five-point scale. Participant
EEG activity was only recorded
during the recall test.

Revista de neurociencia cognitiva

Volumen 32, Número 1

D
oh
w
norte
yo
oh
a
d
mi
d

/
j

t
t

F
/

i
t
.

:
/
/

/
j

oh
norte

0
5

METRO
a
y

2
0
2
1

(1 = not sure at all, 2 = not so sure, 3 = fairly sure, 4 = very
sure, 5 = completely sure). The duration of the recall test
was about 1 hr 15 mín.. The learning and recall phases
were programmed using the Psychtoolbox, a MATLAB
(The MathWorks, Cª) toolbox.

EEG Acquisition and Analysis

All of the 11 participants were included in the EEG
análisis.

EEG Acquisition and Preprocessing

During the recall test, participant brain activity was re-
corded with a 64-channel cap connected to a BioSemi
EEG amplifier (5 kHz) and with a sampling frequency
de 1024 Hz. A finite impulse response Blackman band-
pass filter was applied to the data with a lower cutoff fre-
quency of 0.1 Hz to avoid filtering artifacts (Acunzo,
Mackenzie, & van Rossum, 2012), a higher cutoff fre-
quency of 100 Hz and a transition bandwidth = 1. Datos
were down-sampled to 256 Hz. Trials were then epoched
from −1 to 2 sec relative to picture onset, their baseline
removed (−0.2 to 0 segundo) and rereferenced based on the
average reference. Independent component analyses
were performed on the whole data, and the components
were visually inspected to remove the ones that did not
correspond to EEG signal (eye and jaw movements, corazón
actividad). For all the participants, trials with extreme
values for a channel (>100 or < −100 μV), an abnormal trend (slope max: 75 μV, R2 > .3), or an abnormal distri-
bution (kurtosis outside 5 SDs of the kurtosis for each
single electrode) were excluded from the analysis after
visual inspection. All the preprocessing steps were done
with EEGlab (Delorme & Makeig, 2004).

Importantly and following the recommendations in
VanRullen (2011), the analysis of the EEG data was re-
stricted to the conditions that were manipulated by the
experimenter. En efecto, introducing subjective responses
in the comparisons such as participant memory judg-
ment or selecting only the correct responses would likely
produce biases in the analysis. Por lo tanto, in the following
ERP and MVPAs, trials were split according to the four re-
tention intervals: HOURS, 1D, 3W., and NEW for snap-
shots respectively seen a few hours, 1 día, y 3 semanas
beforehand, or never seen in the learning sessions.

ERP Analysis

The ERP time courses for the four retention intervals
were analyzed on average across the participants for
t = −200 to 800 msec after picture onset. The topo-
graphical representations of the ERPs were analyzed ev-
ery 50 mseg. Differential activity was also computed on a
2 × 2 analysis by contrasting trials with snapshots seen
during a learning session and trials with NEW snapshots.
The topographical representations of the ERP contrasts

were analyzed every 50 mseg. To ensure that the activity
observed was not mainly driven by some motor activity
linked to the button press, the ERPs for the four reten-
tion intervals were aligned to participant response onset.
All trials answered within 1.8 sec after picture onset were
incluido (HOURS: mean number = 113, DE = 7; 1D:
mean number = 105, DE = 12; 3W.: mean number = 91,
DE = 21; NEW: mean number = 289, DE = 51). Estos
response-locked ERPs were analyzed on average across
the participants for t = −800 to 200 msec after response
onset. The differential activity between snapshots previ-
ously seen and NEW snapshots was also computed on a
2 × 2 analysis for the response-locked ERPs.

Decoding

Decoding analyses were performed by using an open
source script that was used in the studies of Crouzet,
Busch, and Ohla (2015) and Cauchoix, Crouzet, Fize,
and Serre (2016).2

MVPAs were conducted on the same data used for the
ERP analyses. The classification was performed on two
classes: one class corresponding to snapshots seen dur-
ing a specific learning session (HOURS, 1D, or 3W) y
the other class corresponding to NEW snapshots. Trials
were randomly split 120 times according to the Monte
Carlo cross-validation (CV) procedimiento: 120 CVs. For each
CV, the number of trials was equalized between the clas-
ses: 90% of the trials were used by the algorithm as a
training set, and the remaining 10% were used as a test
colocar. To reduce the effect of outlier values in the signal,
the activity of each electrode was normalized across the
ensayos: For each CV, the signal was scaled between 0 y 1
by using parameters that were estimated from the train-
ing set. A linear classifier (L2-regularized logistic regres-
sión; Admirador, Chang, Wang, & lin, 2008) was trained on
individual trials at each time point from −200 to 800 mseg
and tested on individual trials on the same time points
and all the other time points (temporal generalization).
The performance of the classifier for every participant
and at each test time point was evaluated by using the
area under the curve. The average decoding performance
across participants resulted in a decoding matrix where
the x and y axes represented the training and testing
veces, respectivamente. Chance-level decoding was calcu-
lated by performing the same classification on randomly
permuted labels. This generalization across time was
further extended to generalization across conditions
where a classifier trained on one condition was tested
on its ability to generalize to another condition (p.ej.,
trained on HOURS and NEW trials and tested on 1D
and NEW trials). Again for each condition, one class cor-
responded to snapshots never seen before (NEW) y
one class to snapshots seen during a specific learning
session (HOURS, 1D, or 3W). This led to six different
time-and-condition generalization decoding matrices.
These six decoding matrices were obtained using the

Larzabal et al.

D
oh
w
norte
yo
oh
a
d
mi
d

/
j

F
/

t
t

i
t
.

:
/
/

/
j

oh
norte

0
5

METRO
a
y

2
0
2
1

same procedure as the previous time generalization de-
coding matrices except for the number of CV which was
lowered to 20 to reduce computational time.

test whether the number of clips between the two presen-
tations of the same clip significantly affected participant
actuación.

Estadísticas

Recall Phase

The same statistical procedure was applied for the EEG anal-
yses described above: time course representation (canales
by time points), topographical representation (channels at
specific time points), and decoding matrices (time points by
time points). Statistical significance was assessed using non-
parametric cluster permutation tests, norte(permutations) =
4,000, pag < .05, to control for the family-wise error rate (Maris & Oostenveld, 2007). This was performed by adding Fieldtrip functionalities (Oostenveld, Fries, Maris, & Schoffelen, 2011) to the EEGlab toolbox. RESULTS On average, 8 of the 375 videos presented (SD = 9.8) were considered already known by the participants. For each participant, the corresponding trials were discarded from the behavioral and the EEG analysis. Behavioral Results Learning Phase Three learning sessions were done respectively 3 weeks (3W), 1 day (1D), and a few hours (HOURS) before the recall test. To make sure that participants were paying at- tention to the videos, they were asked to make a button- press response whenever a video was presented for the second time. Accordingly, a correct and a wrong button press respectively corresponded to a hit (HIT) and a false alarm (FA) within the signal detection theory framework. The percentage of HITs and FAs was close to optimal per- formance and was stable over the different learning ses- sions: 3W: HITs = 96.7% (SD = 3.2, range = 90.2–100%) and FAs = 0.3% (SD = 0.4, range = 0–0.8%); 1D: 96.3% (SD = 4.7, range = 86.5–100%) and FAs = 0.5% (SD = 0.6, range = 0–1.7%); HOURS: 95.9% (SD = 5.5, range = 83.1– 100%) and FAs = 0.2% (SD = 0.4, range = 0–0.8%). These results show that the encoding of the audiovisual clips was similar across the three learning sessions (one-way ANOVA for HITs: F(2, 30) = 0.1, CI95 difference (Learn- ing 1 vs. Learning 2) [−4.4, 5.3], CI95 difference (Learn- ing 1 vs. Learning 3) [−4.0, 5.8], CI95 difference (Learning 2 vs. Learning 3) [−4.4, 5.3]). For each learning session, the audiovisual clips were presented twice inside a block of 65 or 60 clips. On average across participants, the average median number of clips between the two presentations of the same clip was 30.4 (SD = 0.7, range = 30–32) and varied between 0.5 (SD = 0.5, range = 0–1) and 112.4 (SD = 3.4, range = 110–121) for each participant. Because the num- ber of misses was very small compared with the number of HITs (participants were almost at ceiling), we could not During the recall phase, snapshots from audiovisual clips seen by the participant (n = 375) or NEW pictures (n = 375) were briefly presented on screen. Participants were asked to make an R/K/N judgment as quickly and accu- rately as possible and to rate the confidence in their response on a 5-point scale. In this Performance over the retention intervals. section, “R” and “K” responses were merged to calculate participant performance for the identification of the snapshots. Participants were 90.1% (SD = 9.5), 85.1% (SD = 10.3), and 64.5% (SD = 18.1) correct in identifying snapshots seen respectively a few hours (HOURS), 1 day (1D), and 3 weeks (3W) before and were 79.7% (SD = 13.3) correct at identifying NEW pictures. The percentage of correct responses for snapshots seen 3W before was sig- nificantly different from the performance for snapshots seen HOURS (CI95 difference [10.4, 40.7]) and 1D before (CI95 difference [5.4, 37.7]; one-way ANOVA: F(3, 40) = 7.68; Figure 2A). Participant average confidence was 4.7 (SD = 0.3), 4.6 (SD = 0.4), and 3.9 (SD = 0.8) for the HOURS, 1D, and 3W conditions, respectively, and 3.5 (SD = 0.9) for the NEW snapshots. The HOURS condition was significantly different from the 3W condition (CI95 difference [0.0, 1.5]) and the NEW condition (CI95 difference [0.4, 1.9]; one-way ANOVA: F(3, 40) = 7.64). These results show that participants were able to identify the snapshots ex- plicitly and that they could estimate accurately the prob- ability of being correct. Participant performance was strongly negatively cor- related (r = −.65, Pearson’s correlation coefficient) with the speed of the recall. Participant median RTs increased over the retention intervals: 1.04 sec (SD = 0.12), 1.14 sec (SD = 0.16), and 1.53 sec (SD = 0.34) for snapshots cor- rectly identified and respectively seen in the HOURS, 1D, and 3W conditions. Participant median RT was 1.27 sec (SD = 0.20) for NEW snapshots. The 3W condition was significantly different from the HOURS condition (CI95 difference [0.23, 0.74]) and 1D condition (CI95 difference [0.13, 0.64]; one-way ANOVA: F(3, 40) = 9.5; Figure 2B). R/K/N analysis. Overall, participant subjective responses were distributed as follows: “R” = 33.9% (SD = 11.4), “K” = 15.8% (SD = 7.4), and “N” = 50.3% (SD = 11.8). The significant difference between “R” and “K” responses shows that the snapshots were good cues for eliciting the re- collection of the audiovisual clips (paired t test: t(10) = 3.9, p < .01, CI95 difference [7.9, 28.3]). Participant average performance was high for “R” and “N” responses with 54 Journal of Cognitive Neuroscience Volume 32, Number 1 D o w n l o a d e d l l / / / / j t t f / i t . : / / f r o m D o h w t n t p o : a / d / e d m i f r t o p m r c h . p s i l d v i r e e r c t c . m h a i e r d . u c o o m c n / j a o r t c i c n e / - a p r d t i 3 2 c l 1 e 5 - 0 p d 1 9 f 3 / 1 3 1 2 9 / 9 1 / o 5 c 0 n / _ a 1 _ 8 0 6 1 1 4 4 6 1 8 0 p / d j o b c y n g _ u a e _ s 0 t 1 o 4 n 6 0 8 8 . p S d e f p e b m y b e g r u 2 e 0 s 2 t 3 / j t f . / o n 0 5 M a y 2 0 2 1 D o w n l o a d e d l l / / / / j f / t t i t . : / / f r o m D o h w t n t p o : a / d / e d m i f r t o p m r c h . p s i l d v i r e e r c t c . m h a i e r d . u c o o m c n / j a o r t c i c n e / - a p r d t i 3 2 c l 1 e 5 - 0 p d 1 9 f 3 / 1 3 1 2 9 / 9 1 / o 5 c 0 n / _ a 1 _ 8 0 6 1 1 4 4 6 1 8 0 p / d j o b c y n g _ u a e _ s 0 t 1 o 4 n 6 0 8 8 . p S d e f p e b m y b e g r u 2 e 0 s 2 t 3 / j . / f t o n 0 5 M a y 2 0 2 1 Figure 2. Participant behavioral performance. (A) Participant performance for the retention intervals: hours (HOURS), 1 day (1D), 3 weeks (3W), and the NEW condition (mean and standard deviation in percentage). (B) Participant RTs for the correct responses over the retention intervals (average of the medians and standard deviation in seconds). (C) Repartition of the “R” and “K” judgments for correct responses over the three retention intervals (in percentage). (D) RTs for correct “R” and correct “K” responses over the three retention intervals (average of the medians and standard deviation in seconds). respectively 93.0% (SD = 6.0) and 81.4% (SD = 7.4) of correct responses and significantly lower for “K” responses: 55.3% (SD = 20.4; one-way ANOVA: F(2, 30) = 24.34, CI95 dif- ference [24.0, 51.3], CI95 difference [12.5, 39.8]). Partici- pant average confidence was significantly the highest for “R” responses: 4.7 (SD = 0.2) and lower for “N”: 3.4 (SD = 0.9) and “K” responses: 3.0 (SD = 0.5; one-way ANOVA: F(2, 30) = 23.19, CI95 difference [1.1, 2.4], CI95 difference [0.6, 1.9], CI95 difference [−1.1, 0.2]). The average number of “R,” “K,” and “N” judgments across participants is presented in Table 1. The values seem to indicate that correct “R” responses decrease over the retention interval whereas correct “K” responses in- crease. To test whether this trend was significant, we cal- culated the proportion of correct “R” responses compared with correct “K” responses: the correct “R/K” ratio. This ratio was calculated for each participant and over the three retention intervals: HOURS, 1D, and 3W. Interestingly, the Table 1. Average Response Number of “R,” “K,” and “N” Judgments across the Retention Intervals NEW HOURS 1D 3W N K R N K R N K R N K R 299 55 21 12 13 97 18 18 86 43 32 47 average correct “R/K” ratio across the participants de- creased over the retention intervals: 87.4% (SD = 15.7), 81.4% (SD = 16.4), 55.6% (SD = 23.31) for snapshots re- spectively seen in the HOURS, 1D, and 3W conditions with the 3W condition significantly different from the HOURS condition (CI95 difference [12.0, 51.6]) and 1D condition (CI95 difference [5.9, 45.5]; one-way ANOVA: F(30, 2) = 8.85; Figure 2C). Participant median RTs for correct “R” responses were on average 1.01 sec (SD = 0.1), 1.07 sec (SD = 0.2), and 1.30 sec (SD = 0.3) for snapshots seen in the HOURS, 1D, and 3W conditions, respectively (Figure 2D), with a signif- icant difference between the HOURS and 3W conditions (CI95 difference [0.04, 0.52]; one-way ANOVA: F(2, 30) = 4.69). Participant RTs for remembering the clips were negatively correlated with participants performance (r = −.58). In contrast, correct “K” responses were stable over retention intervals: 2.08 sec (SD = 0.7), 2.11 sec (SD = 0.7), and 1.99 sec (SD = 0.7) for the HOURS, 1D, or 3W conditions, respectively, F(2, 29) = 0.09. Overall, partici- pant RTs were slower for “K” (mean = 2.00 sec, SD = 0.7) compared with “R” (mean = 1.07 sec, SD = 0.1, CI95 dif- ference [0.49, 1.37]) and “N” responses (mean = 1.27 sec, SD = 0.2, CI95 difference [0.29, 1.17]; one-way ANOVA: F(2, 30) = 15.17). Slower RTs for “K” responses are due to our experimental design that pushes participants to make fast “R” responses preferentially. Late “K” responses simply indicate a failure of recollection after about two sec. Larzabal et al. 55 Overall, the behavioral results indicate that participant ability to replay the audiovisual clips in their mind de- creased over the retention intervals. This had a direct impact on the percentage of correct discrimination be- tween the old and new snapshots and on the timing of the decision. The gradual shift from “Remember” to “Know” responses is in agreement with the transforma- tion hypothesis proposed by the MTT. This should be directly reflected by differences of brain dynamics. We now propose to investigate these temporal dynamics by analyzing participant EEG activity during the recall test. EEG Analysis As explained in the Methods section, subjective reports were not used in the EEG analysis as they could bias the results ( VanRullen, 2011). Only the objective HOURS, 1D, 3W, and NEW conditions were considered in the following ERP and MVPAs. Stimulus-locked ERP Analyses ERP topographies. First, the topographical maps corre- sponding to the ERP activity elicited by the snapshots presented in the three watching sessions (OLD) and by snapshots never seen before (NEW) were compared ev- ery 50 msec (paired t test, p < .05, cluster corrected; Figure 3). This provides a global overview of the tempo- ral dynamics involved during memory recall at the scalp level. Significant differences were found from 350 msec poststimulus onset onward and involved a large and dis- tributed network of channels. Over time, a gradual shift from frontocentral to centroparietal locations could be observed along with significant differences in the occipi- tal regions. To track how these dynamics evolved over the passage of time, the OLD condition was split accord- ing to the three retention intervals: HOURS, 1D, and 3W. The corresponding topographical representations are presented in Figure 4A. The activity for these three reten- tion intervals was respectively compared with the NEW condition. The statistical maps with the exact p values (paired t tests, cluster corrected) are presented in Figure 4B. For the three contrasts, significant activations were found at t = 450 msec and t = 650 msec poststim- ulus onset. At t = 450 msec, significant channels were ob- served in the frontocentral area. Occipital channels were also involved for the HOURS and 1D conditions. At t = 650 msec, the difference of activity mainly relied on cen- troparietal channels for the two recent intervals and on frontocentral channels for the 3W condition. Overall, significant activations for the two recent intervals covered an extended time period and involved a vast network of channels as previously observed for the main OLD/NEW contrast. This was different for the 3W condition for which differences of activity remained located in the frontocentral region and did not extend over long time periods. The topographical representations allowed a clear identification of the frontocentral, central, parietal, and occipital regions as key regions involved during the retrieval of the audiovisual clips. ERP time course analyses. The channels FC1, C1, P3, and P10 were representative of the respective fronto- central, central, parietal, and occipital regions that were involved during memory retrieval. The activity mod- ulation of these channels for the HOURS, 1D, 3W, and NEW retention intervals is presented in Figure 5. The channels FC1 and C1 exhibited significant activity from t = 400 msec to t = 650 msec poststimulus onset (one- way ANOVA, p < .05, cluster corrected). Although the P3 channel was found to be significant at t = 550 msec, t = 600 msec, t = 750 msec, and t = 800 msec post- stimulus onset for the OLD/NEW contrast, no significant activation modulation was found for this channel when considering the four conditions tested over all chan- nels and time points (one-way ANOVA, p < .05, cluster corrected). It is interesting to note that the P10 channel was found significant from the stimulus offset (t = 400 msec) to the end of the epoch (t = 800 msec), which was demonstrative of the activity modulation in the oc- cipital area. Overall, the modulation of the ERP activity Figure 3. ERP topography elicited by the presentation of snapshots seen in the three learning sessions (OLD) and snapshots never seen before (NEW). Figures represent the average activity across the participants for t = −200 to 800 msec poststimulus onset. Red dots correspond to significant channels at the specified time (paired t test, p < .05, cluster corrected). 56 Journal of Cognitive Neuroscience Volume 32, Number 1 D o w n l o a d e d l l / / / / j f / t t i t . : / / f r o m D o h w t n t p o : a / d / e d m i f r t o p m r c h . p s i l d v i r e e r c t c . m h a i e r d . u c o o m c n / j a o r t c i c n e / - a p r d t i 3 2 c l 1 e 5 - 0 p d 1 9 f 3 / 1 3 1 2 9 / 9 1 / o 5 c 0 n / _ a 1 _ 8 0 6 1 1 4 4 6 1 8 0 p / d j o b c y n g _ u a e _ s 0 t 1 o 4 n 6 0 8 8 . p S d e f p e b m y b e g r u 2 e 0 s 2 t 3 / j / f . t o n 0 5 M a y 2 0 2 1 Figure 4. ERP topography for snapshots seen in the three learning sessions (HOURS, 1D, and 3W). (A) Average activity across the participants for t = −200 to 800 msec poststimulus onset. (B) Statistical probability maps of the average differential ERP activity for t = −200 to 800 msec poststimulus onset. The exact p values are displayed (paired t test, cluster corrected). D o w n l o a d e d l l / / / / j f / t t i t . : / / f r o m D o h w t n t p o : a / d / e d m i f r t o p m r c h . p s i l d v i r e e r c t c . m h a i e r d . u c o o m c n / j a o r t c i c n e / - a p r d t i 3 2 c l 1 e 5 - 0 p d 1 9 f 3 / 1 3 1 2 9 / 9 1 / o 5 c 0 n / _ a 1 _ 8 0 6 1 1 4 4 6 1 8 0 p / d j o b c y n g _ u a e _ s 0 t 1 o 4 n 6 0 8 8 . p S d e f p e b m y b e g r u 2 e 0 s 2 t 3 / j f . t / o n 0 5 M a y 2 0 2 1 Figure 5. ERP time course elicited by snapshots seen in the three learning sessions (HOURS, 1D, and 3W) and snapshots never seen before (NEW). Figures represent the average activity across the participants for t = −200 to 800 msec poststimulus onset. The representative FC1, C1, P3, and P10 channels are presented. The location of the channel is circled in red. Gray areas correspond to significant time points (one-way ANOVA, p < .05, cluster corrected). Larzabal et al. 57 D o w n l o a d e d l l / / / / j f / t t i t . : / / f r o m D o h w t n t p o : a / d / e d m i f r t o p m r c h . p s i l d v i r e e r c t c . m h a i e r d . u c o o m c n / j a o r t c i c n e / - a p r d t i 3 2 c l 1 e 5 - 0 p d 1 9 f 3 / 1 3 1 2 9 / 9 1 / o 5 c 0 n / _ a 1 _ 8 0 6 1 1 4 4 6 1 8 0 p / d j o b c y n g _ u a e _ s 0 t 1 o 4 n 6 0 8 8 . p S d e f p e b m y b e g r u 2 e 0 s 2 t 3 / j t f / . o n 0 5 M a y 2 0 2 1 Figure 6. Response-locked ERP. (A) Average ERP time course for the representative channels: FC1, C1, P3, and P10. (B) Average ERP topography for the HOURS and NEW conditions. observed for these representative channels closely matches the memory components classically found in the literature. The frontal activations observed at t = 400–500 msec and the late parietal component found at t = 600–800 msec resemble the FN400 and the LPC, respectively. Response-locked ERPs The ERP analyses were performed between t = −200 msec and t = 800 msec poststimulus onset. This time window was selected to analyze the components associated to memory retrieval without including participant median button response timing (1.04–1.53 sec). But even with these precautions it might be possible that the significant activations observed at the end of the epoch were directly driven by some motor activation linked to the button press. To test for a potential motor effect, the ERPs of the HOURS, 1D, 3W, and NEW conditions were aligned to the participant response onset (t = −800 to 200 msec) and analyzed on average across the participants. Statistical testing (one-way ANOVA, p < .05, cluster corrected) across all channels and time points revealed no significant differences in the time course of the 64 channels. The time course of the representative channels FC1, C1, P3, and P10 are presented in Figure 6A. Late effects were mainly observed for the HOURS/NEW and 1D/NEW contrasts. We therefore tested whether sig- nificant differences could be detected for these two con- trasts when ERPs were aligned to participant response onset (paired t test, p < .05, cluster corrected). Again, no significant differences were observed. The topograph- ical representation of the response-locked ERPs for the HOURS and NEW conditions is shown in Figure 6B. The analysis of the response-locked ERPs did not reveal any significant motor component that would be respon- sible for the ERP effects observed earlier. Overall, the ERP data showed that the brain processes involved in the recall of the audiovisual clips differed according to the retention interval. For the clips seen 3 weeks beforehand, the ERP activations were transient and focused on the frontocentral area, whereas for the recent clips a large network of channels was activated for an extended time period. These dynamics might be reflected by different patterns. In the next section, we use MVPA techniques to characterize such temporal patterns. Multivariate Pattern Analysis We performed MVPA on our EEG data by comparing the activity for snapshots seen in one of the three learning sessions (HOURS, 1D, or 3W) and NEW snapshots. For each participant, classifier performance was measured for the trial tests at the same time point as for the training sets (diagonal matrix) or for any other time points (off-diagonal matrix). Performance was averaged across the 11 participants and calculated using the area under the curve where chance level corresponded to the same classification performed using random labels (Figure 7A). Classification performance was found to be significantly different (paired t test, p < .05, cluster corrected) from chance level from around t = 500 msec poststimulus onset and until the end of the epoch for the HOURS/NEW comparison. This resulted in a clear square-shaped pattern. Similar findings were observed for the 1D/NEW contrast, 58 Journal of Cognitive Neuroscience Volume 32, Number 1 although no generalization was found when the decoder was trained at t ∼ 500 msec and tested at t = 600–800 msec and vice versa. For the 3W/NEW contrast, significant time points were observed within the same timing range and formed small connected clusters resulting in a degraded square-shaped pattern. Similarity analyses on the differential correct response percentages and on the decoding matrices were performed for each participant. Interestingly, we found that the differ- ential correct response percentages for the HOURS/NEW, 1D/NEW, and 3W/NEW conditions were correlated with the respective decoding matrix performance (mean: r = .57, SD = 0.7, range = −0.99 to 1.00, Pearson’s correlation coefficient). This suggests that it might be possible to infer participant performance on very long term memories from the matrix decoding obtained through MVPA. The above time generalization decoding matrices re- vealed important features concerning the recollection of long-term memories in regard to the predictions made in the introduction. Recollection was associated with a clear square-shaped pattern that faded away over the re- tention intervals. To test whether these patterns relied on the same neural processes, we decided to test how a clas- sifier trained on a specific retention interval could gener- alize to another retention interval. This resulted in six time-and-condition generalization decoding matrices. Significant time points (paired t test, p < .05, cluster cor- rected) for these six time-and-condition generalization D o w n l o a d e d l l / / / / j t t f / i t . : / / f r o m D o h w t n t p o : a / d / e d m i f r t o p m r c h . p s i l d v i r e e r c t c . m h a i e r d . u c o o m c n / j a o r t c i c n e / - a p r d t i 3 2 c l 1 e 5 - 0 p d 1 9 f 3 / 1 3 1 2 9 / 9 1 / o 5 c 0 n / _ a 1 _ 8 0 6 1 1 4 4 6 1 8 0 p / d j o b c y n g _ u a e _ s 0 t 1 o 4 n 6 0 8 8 . p S d e f p e b m y b e g r u 2 e 0 s 2 t 3 / j t f . / o n 0 5 M a y 2 0 2 1 Figure 7. Average decoding matrices for the two-class conditions: HOURS/NEW, 1D/NEW, and 3W/NEW. (A) Generalization across time. Training and test time correspond respectively to the x and y axis. The black bar from 0 to 400 msec corresponds to the snapshot presentation. Significant values are contoured in black (paired t test, p < .05, cluster corrected). (B) Generalization across time and conditions. Training and test conditions are shown respectively on the horizontal and vertical axis of the 3 × 3 condition map. Each of the nine entries of the map is a specific time-and-condition generalization matrix. For each matrix, significant time points are plotted in orange (paired t test, p < .05, cluster corrected). The decoding matrices numbers 1, ,5 and 9 correspond to the temporal decoding matrices shown in Figure 7A for which the classifier was trained and tested on the same two-class condition. Larzabal et al. 59 matrices are shown in Figure 7, along with the significant time points obtained for the three previous time gener- alization matrices. Each of these nine decoding matrices corresponded to a specific entry in the 3 × 3 training ver- sus test condition map. The three time generalization matrices for which the classifier was trained and tested for the same condition correspond to the decoding matrices numbers 1, 5, and 9. Interestingly, significant clusters where found for all the six time-and-condition generalization matrices (numbers 2, 3, 4, 6, 7, and 8). This means that common neural processes were shared across the different retention intervals and could be used by the classifier to perform above chance level. However, the generalization across condition varied according to the condition trained and tested. In particular, the clas- sifier performed particularly well when trained on HOURS and NEW trials and tested on 1D and NEW trials (decoding matrix number 4). Classification per- formances were not perfectly symmetrical when the training and test conditions were inverted (e.g., training on HOURS/NEW and tested on 1D/NEW: decoding matrix number 4 and training on 1D/NEW and tested on HOURS/NEW: decoding matrix number 2). Indeed, more information would be available in the HOURS trials for the classifier to generalize to 1D trials rather than the opposite. Importantly, although less information was expected to be contained in the 3W trials, a classifier trained on 3W trials was still able to generalize to HOURS and 1D trials (decoding matrices numbers 3 and 6). DISCUSSION Using MVPA techniques on EEG signal, we were able to characterize the brain dynamics involved during the long- term recollection of audiovisual clips. The time generali- zation decoding matrices obtained for each retention in- terval revealed sustained patterns that emerged 500 msec after the presentation of the snapshots. This sustained ac- tivity was the most stable for the clips seen a few hours beforehand and faded away over the passage of time. Generalization across time and condition also revealed that the same processes were involved during the re- collection of long-term memories independently of their age. According to the ERP analyses, the patterns observed emerged from the interaction between the cen- tral channels and the frontal region at early latencies (∼450 msec poststimulus onset) and between the central channels and the parietal sites later in time (t = 600– 800 msec poststimulus onset). Late effects could have been the result of motor initi- ation but no significant motor components were found when participant ERPs were time-locked to the re- sponse onset. The correlation between the decoding performance and participant performance over the re- tention intervals demonstrated a close link between the brain activity used by the algorithm and participant ability to recall the audiovisual clips. Overall our findings are in agreement with the MTT and the transformation hypothesis. Many fMRI investigations have been carried out to solve the ongoing debate between the standard consolidation model and the MTT. More specifically, studies tested whether the hippocampus was still involved during the re- trieval of remote episodic memories. In addition to classical univariate approaches, (e.g., Harand et al., 2012; Gilboa, Winocur, Grady, Hevenor, & Moscovitch, 2004; Maguire & Frith, 2003; Maguire, Henson, Mummery, & Frith, 2001) MVPA techniques were used (Bonnici & Maguire, 2018; Bonnici, Chadwick, & Maguire, 2013; Bonnici et al., 2012; Chadwick, Hassabis, Weiskopf, & Maguire, 2010). These studies revealed that the hippocampus was still ac- tivated during the retrieval of remote episodic details al- though some changes were observed over the passage of time reflecting some transformation of the memory trace. Indeed, remote episodic memories involved preferen- tially the posterior part of the hippocampus and were par- ticularly visible in the ventromedial pFC compared with recent episodic memories. Similarly to our study, the con- trasts used to find specific spatial patterns in these fMRI studies were based on the timing between the learning of the information and its retrieval. This differs from other fMRI studies that used subjective “R” and “K” contrasts (Rissman, Reggente, & Wagner, 2016; Rissman, Greely, & Wagner, 2010) and that could be more prone to inter- pretation biases ( VanRullen, 2011). We found that the correct “R/K” ratio decreased over the retention intervals. As previously observed in the lit- erature (Piolino et al., 2009; Herbert & Burt, 2004; Conway et al., 1997), this shift from “R” to “K” responses shows that the recollection of the audiovisual content was becoming more difficult as time elapsed. Inter estingly, the speed of “R” responses also increased over the retention intervals. Classical speeded R/K paradigms usually compare “R” and “K” participants’ RTs. In addition to this comparison, our design allowed us to analyze the speed of “R” and “K” responses according to the reten- tion intervals. Since the early 1950s and the development of the sequential analysis method, the speed of a deci- sion has been linked to the amount of evidence accumu- lated toward the decision ( Wald, 1947). The difference of timing observed for the “R” responses over the retention intervals would then directly reflect the difference of evidence that had been accumulated. The recollection of contextual details, which would result from the accumula- tion of information retrieved over the time of the decision, would be achieved quickly for recent memories but would take more time for remote memories. Accordingly in our design, “K” responses would indicate a failure to retrieve contextual information for snapshots previously seen. Indeed, “K” responses were associated with the longest latencies in our experiment. This view of considering memory retrieval based on an evidence accumulation model is in agreement with the 60 Journal of Cognitive Neuroscience Volume 32, Number 1 D o w n l o a d e d l l / / / / j f / t t i t . : / / f r o m D o h w t n t p o : a / d / e d m i f r t o p m r c h . p s i l d v i r e e r c t c . m h a i e r d . u c o o m c n / j a o r t c i c n e / - a p r d t i 3 2 c l 1 e 5 - 0 p d 1 9 f 3 / 1 3 1 2 9 / 9 1 / o 5 c 0 n / _ a 1 _ 8 0 6 1 1 4 4 6 1 8 0 p / d j o b c y n g _ u a e _ s 0 t 1 o 4 n 6 0 8 8 . p S d e f p e b m y b e g r u 2 e 0 s 2 t 3 / j t / f . o n 0 5 M a y 2 0 2 1 diffusion model proposed by Ratcliff that accounts for percep- tual and memory processes (Ratcliff & Mckoon, 2008; Ratcliff, 1978). Although this perspective is now commonly held in the perception domain (Dehaene, 2009; Norris & Kinoshita, 2008; Gold & Shadlen, 2007; Ploran et al., 2007), it is not widely shared within the memory community to account for high level of retrieval such as familiarity and recollection processes (Shadlen & Shohamy, 2016; Moulin & Souchay, 2015; Koriat & Levy-Sadot, 2001). Indeed, instead of such a dynamic model of memory retrieval, most of the literature on recollection and familiarity relies on a debate between two static recognition models: the dual-process model and the single-process model. According to the dual-process models, familiarity and re collection are two distinct processes ( Yonelinas, 2002; Gardiner & Richardson-Klavehn, 2000; Mandler, 1980; Juola, Fischler, Wood, & Atkinson, 1971), whereas the single-process models suggest that familiarity and recol lection can be explained using a single strength-based measure ( Wixted, 2009; Dunn, 2004; McClelland & Chappell, 1998; Shiffrin & Steyvers, 1997; Hintzman, 1988; Eich, 1982). In most of the R/K paradigms, “R” and “K” responses are associated with recollection and familiarity processes. However, depending on the paradigms used, “K” re- sponses can also reflect retrieval from semantic memory. In our study, participants were asked to recall specific au- diovisual clips from the presentation of snapshots. Two re- sponse judgments were therefore possible and referred to the amount of source information retrieved (Gardiner & Richardson-Klavehn, 2000): The amount of information re- trieved was sufficient to recall the clip (“R” response), and the identification of an old snapshot was not followed by the retrieval of the clip (“K” response). Therefore, “R” re- sponses corresponded to the retrieval of item plus as- sociative information whereas “K” responses reflect the retrieval of item-only information (Mickes, Seale-Carlisle, & Wixted, 2013). Accordingly, our design cannot provide a direct comparison between familiarity and recollection processes, although familiarity judgment might emerge from the retrieval of item-only information (Mickes et al., 2013). The sustained activity that we observed could be ex- plained by the episodic buffer account proposed by Baddeley (2000). According to this view, the key role of the episodic buffer would be to provide temporary storage for the information retrieved from long-term memory. Indeed, the information would be bound into a unitary ep- isodic representation that would be directly accessible to consciousness. Recent evidence shows that the parietal cortex might play this buffer role (Kuhl & Chun, 2014; Vilberg & Rugg, 2008; Wagner, Shannon, Kahn, & Buckner, 2005). Other regions would be also involved such as the visual areas, as supported by our data. Interestingly, sustained decoding patterns were recently observed when participants were involved in a short-term visual imagery task and differed from the sequential decod- ing patterns observed during visual perception (Dijkstra et al., 2018). However, and as raised by the authors, the task and the stimuli used in their study were not the best to ob- serve the constructive process involved during memory re- instatement. In our task, recollection corresponded to participants’ ability to replay short audiovisual clips in their mind. Even in such a dynamical context, it is worth men- tioning that a sustained pattern emerged as well. Again, the role of the episodic buffer, which is part of the working memory model, would explain the similarity between im- agery from short-term and long-term memory. In line with the evidence accumulation model de- scribed earlier, the difference of signal strength for the sustained patterns would be directly linked with the amount of information retrieved in memory. Indeed, studies showed that the amplitude of the LPC was higher if recollection required the retrieval of several contextual details ( Vilberg, Moosavi,& Rugg, 2007; Wilding, 2000; Wilding & Rugg, 1996). The same effect was also ob- served when comparing the LPC of remote and recent memories (Tsivilis et al., 2015). Another possibility would be that, over the consolida- tion process, long-term memories might become sup- ported by a smaller and more specialized neuronal network to prevent their loss (Thorpe, 1989, 2011). The decrease of signal would therefore reflect the in- volvement of a more localized pattern of neurons. Importantly, sleep might have played a crucial role in the change of sustained activity found for the different re- tention intervals. Indeed, numerous studies have shown that, during sleep, a memory trace can be reactivated sev- eral times and therefore be consolidated into a stable neu- ronal pattern (Girardeau, Benchenane, Wiener, Buzsáki, & Zugaro, 2009; Peyrache, Khamassi, Benchenane, Wiener, & Battaglia, 2009; Sirota, Csicsvari, Buhl, & Buzsáki, 2003; Buzsáki, 1989). However, in our design, we could not isolate this variable to measure its impact. Overall, we showed that imagery from long-term memory was associated with a sustained pattern of activity that faded away over the passage of time. This provides further evidence toward the MTT. Further in- vestigations should be carried out to compare these data with patients suffering from memory impairment. Notes 1. All the audiovisual clips and snapshots used in this experi- ment are freely available for research purposes and can be downloaded using the following link: https://data.mendeley. com/datasets/nh8xsmr289/draft?a=1813589b-7737-442d-9fcb- 89fa0f3bee5d. 2. The open source script can be downloaded at https:// github.com/scrouzet/classifyEEG. Acknowledgments This research was supported by the European Union’s Seventh Framework Programme (FP7/2007-2013)/ERC grant agreement Larzabal et al. 61 D o w n l o a d e d l l / / / / j t t f / i t . : / / f r o m D o h w t n t p o : a / d / e d m i f r t o p m r c h . p s i l d v i r e e r c t c . m h a i e r d . u c o o m c n / j a o r t c i c n e / - a p r d t i 3 2 c l 1 e 5 - 0 p d 1 9 f 3 / 1 3 1 2 9 / 9 1 / o 5 c 0 n / _ a 1 _ 8 0 6 1 1 4 4 6 1 8 0 p / d j o b c y n g _ u a e _ s 0 t 1 o 4 n 6 0 8 8 . p S d e f p e b m y b e g r u 2 e 0 s 2 t 3 / j t . / f o n 0 5 M a y 2 0 2 1 no. 323711 to S. J. T. The authors also acknowledge the support of the French Agence Nationale de la Recherche under grant ANR-12-BSH2-0010 (project ELMA). We thank Emmanuel Barbeau for helpful discussions and Estelle Bonin for her assis- tance in the stimuli preparation. We are also very grateful for the feedback that the two anonymous reviewers gave us during the revision process. Reprint requests should be sent to Christelle Larzabal, Pavillon Baudot, CHU Purpan, Toulouse 31052, France, or via e-mail: christelle.larzabal@cnrs.fr. REFERENCES Acunzo, D. J., Mackenzie, G., & van Rossum, M. C. (2012). Systematic biases in early ERP and ERF components as a result of high-pass filtering. Journal of Neuroscience Methods, 209, 212–218. Baddeley, A. (2000). The episodic buffer: A new component of working memory? Trends in Cognitive Sciences, 4, 417–423. Bonnici, H. M., Chadwick, M. J., Lutti, A., Hassabis, D., Weiskopf, N., & Maguire, E. A. (2012). Detecting representations of recent and remote autobiographical memories in vmPFC and hippocampus. Journal of Neuroscience, 32, 16982–16991. Bonnici, H. M., Chadwick, M. J., & Maguire, E. A. (2013). Representations of recent and remote autobiographical memories in hippocampal subfields. Hippocampus, 23, 849–854. Bonnici, H. M., & Maguire, E. A. (2018). Two years later— Revisiting autobiographical memory representations in vmPFC and hippocampus. Neuropsychologia, 110, 159–169. Brewer, W. F., & Pani, J. R. (1996). Reports of mental imagery in retrieval from long-term memory. Consciousness and Cognition, 5, 265–287. Buzsáki, G. (1989). Two-stage model of memory trace formation: A role for noisy brain states. Neuroscience, 31, 551–570. Cauchoix, M., Crouzet, S. M., Fize, D., & Serre, T. (2016). Fast ventral stream neural activity enables rapid visual categorization. Neuroimage, 125, 280–290. Chadwick, M. J., Hassabis, D., Weiskopf, N., & Maguire, E. A. (2010). Decoding individual episodic memory traces in the human hippocampus. Current Biology, 20, 544–547. Conway, M. A., Gardiner, J. M., Perfect, T. J., Anderson, S. J., & Cohen, G. M. (1997). Changes in memory awareness during learning: The acquisition of knowledge by psychology undergraduates. Journal of Experimental Psychology: General, 126, 393–413. Crouzet, S. M., Busch, N. A., & Ohla, K. (2015). Taste quality decoding parallels taste sensations. Current Biology, 25. 890–896. Curran, T. (2000). Brain potentials of recollection and familiarity. Memory and Cognition, 28, 923–938. Curran, T., & Friedman, W. J. (2004). ERP old/new effects at different retention intervals in recency discrimination tasks. Cognitive Brain Research, 18, 107–120. Damasio, A. R. (1989). Time-locked multiregional retroactivation: A systems-level proposal for the neural substrates of recall and recognition. Cognition, 33, 25–62. Danker, J. F., & Anderson, J. R. (2010). The ghosts of brain states past: Remembering reactivates the brain regions engaged during encoding. Psychological Bulletin, 136, 87–102. Delorme, A., & Makeig, S. (2004). EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. Journal of Neuroscience Methods, 134, 9–21. Dijkstra, N., Mostert, P., DeLange, F. P., Bosh, S., & van Gerven, M. A. (2018). Differential temporal dynamics during visual imagery and perception. eLife, 29, e33904. Dijkstra, N., Zeidman, P., Ondobaka, S., Van Gerven, M. A. J., & Friston, K. (2017). Distinct top–down and bottom–up brain connectivity during visual perception and imagery. Scientific Reports, 7, 5667. Duarte, A., Ranganath, C., Winward, L., Hayward, D., & Knight, R. T. (2004). Dissociable neural correlates for familiarity and recollection during the encoding and retrieval of pictures. Cognitive Brain Research, 18, 255–272. Dunn, J. C. (2004). Remember-know: A matter of confidence. Psychological Review, 111, 524–542. Düzel, E., Yonelinas, A. P., Mangun, G. R., Heinze, H. J., & Tulving, E. (1997). Event-related brain potential correlates of two states of conscious awareness in memory. Proceedings of the National Academy of Sciences, U.S.A., 94, 5973–5978. Eich, J. M. (1982). A composite holographic associative recall model. Psychological Review, 89, 627–661. Fan, R.-E., Chang, K.-W., Wang, X.-R., & Lin, C.-J. (2008). LIBLINEAR: A library for large linear classification. Journal of Machine Learning, 9, 1871–1874. Frankland, P. W., & Bontempi, B. (2005). The organization of recent and remote memories. Nature Reviews Neuroscience, 6, 119–130. Gardiner, J. M., & Richardson-Klavehn, A. (2000). Remembering and knowing. In E. Tulving & F. I. M. Craik (Eds.), The Oxford handbook of memory (pp. 229–244). Oxford: Oxford University Press. Gilboa, A., Winocur, G., Grady, C. L., Hevenor, S. J., & Moscovitch, M. (2004). Remembering our past: Functional neuroanatomy of recollection of recent and very remote personal events. Cerebral Cortex, 14, 1214–1225. Girardeau, G., Benchenane, K., Wiener, S. I., Buzsáki, G., & Zugaro, M. B. (2009). Selective suppression of hippocampal ripples impairs spatial memory. Nature Neuroscience, 12, 1222–1223. Gold, J. I., & Shadlen, M. N. (2007). The neural basis of decision making. Annual Review of Neuroscience, 30, 535–5574. Harand, C., Bertran, F., La Joie, R., Landeau, B., Mézenge, F., Desgranges, B., et al. (2012). The hippocampus remains activated over the long term for the retrieval of truly episodic memories. PLoS One, 7, e43495. Herbert, D. M. B., & Burt, J. S. (2004). What do students remember? Episodic memory and the development of schematization. Applied Cognitive Psychology, 18, 77–88. Hintzman, D. L. (1988). Judgments of frequency and recognition memory in a multiple-trace memory model. Psychological Review, 95, 528–551. Ishai, A., Ungerleider, L. G., & Haxby, J. V. (2000). Distributed neural systems for the generation of visual images. Neuron, 28, 979–990. Johnson, J. D., McDuff, S. G., Rugg, M. D., & Norman, K. A. (2009). Recollection, familiarity, and cortical reinstatement: A multivoxel pattern analysis. Neuron, 63, 697–708. Johnson, J. D., & Rugg, M. D. (2007). Recollection and the reinstatement of encoding-related cortical activity. Cerebral Cortex, 17, 2507–2515. Juola, J. F., Fischler, I., Wood, C. T., & Atkinson, R. C. (1971). Recognition time for information stored in long-term memory. Perception & Psychophysics, 10, 8–14. Dehaene, S. (2009). Conscious and nonconscious processes: King, J. R., & Dehaene, S. (2014). Characterizing the dynamics Distinct forms of evidence accumulation. Séminaire Poincaré XII, 89–114. of mental representations: The temporal generalization method. Trends in Cognitive Sciences, 18, 203–210. 62 Journal of Cognitive Neuroscience Volume 32, Number 1 D o w n l o a d e d l l / / / / j t t f / i t . : / / f r o m D o h w t n t p o : a / d / e d m i f r t o p m r c h . p s i l d v i r e e r c t c . m h a i e r d . u c o o m c n / j a o r t c i c n e / - a p r d t i 3 2 c l 1 e 5 - 0 p d 1 9 f 3 / 1 3 1 2 9 / 9 1 / o 5 c 0 n / _ a 1 _ 8 0 6 1 1 4 4 6 1 8 0 p / d j o b c y n g _ u a e _ s 0 t 1 o 4 n 6 0 8 8 . p S d e f p e b m y b e g r u 2 e 0 s 2 t 3 / j f t . / o n 0 5 M a y 2 0 2 1 Koriat, A., & Levy-Sadot, R. (2001). The combined contributions of the cue-familiarity and accessibility heuristics to feelings of knowing. Journal of Experimental Psychology: Learning, Memory, and Cognition, 27, 34–53. Kuhl, B. A., & Chun, M. M. (2014). Successful remembering elicits event-specific activity patterns in lateral parietal cortex. Journal of Neuroscience, 34, 8051–8060. Maguire, E. A., & Frith, C. D. (2003). Lateral asymmetry in the hippocampal response to the remoteness of autobiographical memories. Journal of Neuroscience, 23, 5302–5307. Maguire, E. A., Henson, R. N., Mummery, C. J., & Frith, C. D. (2001). Activity in prefrontal cortex, not hippocampus, varies parametrically with the increasing remoteness of memories. NeuroReport, 12, 441–444. Mandler, G. (1980). Recognizing: The judgment of previous occurrence. Psychological Review, 87, 252–271. Maris, E., & Oostenveld, R. (2007). Nonparametric statistical testing of EEG- and MEG-data. Journal of Neuroscience Methods, 164, 177–190. McClelland, J. L., & Chappell, M. (1998). Familiarity breeds differentiation: A subjective-likelihood approach to the effects of experience in recognition memory. Psychological Review, 105, 724–760. Mechelli, A., Price, C. J., Friston, K. J., & Ishai, A. (2004). Where bottom–up meets top–down: Neuronal interactions during perception and imagery. Cerebral Cortex, 14, 1256–1265. Mickes, L., Seale-Carlisle, T. M., & Wixted, J. T. (2013). Rethinking familiarity: Remember/know judgments in free recall. Journal of Memory and Language, 68, 333–349. Moulin, C., & Souchay, C. (2015). An active inference and epistemic value view of metacognition. Cognitive Neuroscience, 6, 221–222. Nessler, D., & Mecklinger, A. (2003). ERP correlates of true and false recognition after different retention delays: Stimulus- and response-related processes. Psychophysiology, 40, 146–159. Rissman, J., Greely, H. T., & Wagner, A. D. (2010). Detecting individual memories through the neural decoding of memory states and past experience. Proceedings of the National Academy of Sciences, U.S.A., 107, 9849–9854. Rissman, J., Reggente, N., & Wagner, A. D. (2016). Decoding fMRI signatures of real-worlds autobiographical memory retrieval. Journal of Cognitive Neuroscience, 28, 604–620. Roberts, J. S., Tsivilis, D., & Mayes, A. R. (2013). The electrophysiological correlates of recent and remote recollection. Neuropsychologia, 51, 2162–2171. Rugg, M. D., & Curran, T. (2007). Event-related potentials and recognition memory. Trends in Cognitive Science, 11, 251–257. Rugg, M. D., Johnson, J. D., Park, H., & Uncapher, M. R. (2008). Encoding-retrieval overlap in human episodic memory: A functional neuroimaging perspective. In W. S. Sossin, J.-C. Lacaille, V. F. Castellucci, & S. Belleville (Eds.), Progress in brain research (pp. 339–352). Rugg, M. D., & Vilberg, K. L. (2013). Brain networks underlying episodic memory retrieval. Current Opinion in Neurobilogy, 23, 255–260. Shadlen, M. N., & Shohamy, D. (2016). Decision making and sequential sampling from memory. Neuron, 90, 927–939. Shiffrin, R. M., & Steyvers, M. (1997). A model for recognition memory: REM—retrieving effectively from memory. Psychonomic Bulletin & Review, 4, 154–166. Sirota, A., Csicsvari, J., Buhl, D., & Buzsáki, G. (2003). Communication between neocortex and hippocampus. Proceedings of the National Academy of Sciences, U.S.A., 100, 2065–2069. Squire, L. R., & Alvarez, P. (1995). Retrograde amnesia and memory consolidation: A neurobiological perspective. Current Opinion in Neurobiology, 5, 169–177. Thakral, P. P., Wang, T. H., & Rugg, M. D. (2017). Decoding the content of recollection within the core recollection network and beyond. Cortex, 91, 101–113. Thorpe, S. J. (1989). Local vs. distributed coding. Intellectica, 2, 3–40. Norris, D., & Kinoshita, S. (2008). Perception as evidence Thorpe, S. J. (2011). Grandmother cells and distributed accumulation and Bayesian inference: Insights from masked priming. Journal of Experimental Psychology: General, 137, 434–455. Nyberg, L., Habib, R., McIntosh, A. R., & Tulving, E. (2000). Reactivation of encoding-related brain activity during memory retrieval. Proceedings of the National Academy of Sciences, U.S.A., 97, 11120–11124. Oostenveld, R., Fries, P., Maris, E., & Schoffelen, J. M. (2011). FieldTrip: Open source software for advanced analysis of MEG, EEG, and invasive electrophysiological data. Computer Intelligence and Neuroscience, 2011, 156869. Peyrache, A., Khamassi, M., Benchenane, K., Wiener, S. I., & Battaglia, F. P. (2009). Replay of rule-learning related neural patterns in the prefrontal cortex during sleep. Nature Neuroscience, 12, 919–926. Piolino, P., Desgranges, B., & Eustache, F. (2009). Episodic autobiographical memories over the course of time: Cognitive, neuropsychological and neuroimaging findings. Neuropsychologia, 47, 2314–4329. Ploran, E. J., Nelson, S. M., Velanova, K., Donaldson, D. I., Petersen, S. E., & Wheeler, M. E. (2007). Evidence accumulation and the moment of recognition: Dissociating perceptual recognition processes using fMRI. Journal of Neuroscience, 27, 11912–11924. Ratcliff, R. (1978). A theory of memory retrieval. Psychological Review, 85, 59–108. Ratcliff, R., & Mckoon, G. (2008). The diffusion decision model: Theory and data for two-choice decision tasks. Neural Computation, 20, 873–922. representations. In N. Kriegeskorte & G. Kreiman (Eds.), Visual population codes: Toward a common multivariate framework for cell recording and functional imaging (pp. 23–51). Cambridge, MA: MIT Press. Tsivilis, D., Allan, K., Roberts, J., Williams, N., Downes, J. J., & El-Deredy, W. (2015). Old–new ERP effects and remote memories: The late parietal effect is absent as recollection fails whereas the early mid-frontal effect persists as familiarity is retained. Frontiers in Human Neuroscience, 9, 532. Tsivilis, D., Otten, L., & Rugg, M. (2001). Context effects on the neural correlates of recognition memory: An electrophysiological study. Neuron, 31, 497–505. Tulving, E. (1972). Episodic and semantic memory. In E. Tulving & W. Donaldson (Eds.), Organization of memory (pp. 382–402). New York: Academic Press. Tulving, E. (1983). Elements of Episodic Memory. Oxford, United Kingdom: Oxford University Press. Tulving, E. (1985). Memory and consciousness. Canadian Psychology, 26, 1–12. VanRullen, R. (2011). Four common conceptual fallacies in mapping the time course of recognition. Frontiers in Psychology, 2, 365. Vilberg, K. L., Moosavi, R. F., & Rugg, M. D. (2007). Recollection and amount of information retrieved. Brain Research, 1122, 161–170. Vilberg, K. L., & Rugg, M. D. (2008). Memory retrieval and the parietal cortex: A review of evidence from a dual-process perspective. Neuropsychologia, 46, 1787–1799. Larzabal et al. 63 D o w n l o a d e d l l / / / / j f / t t i t . : / / f r o m D o h w t n t p o : a / d / e d m i f r t o p m r c h . p s i l d v i r e e r c t c . m h a i e r d . u c o o m c n / j a o r t c i c n e / - a p r d t i 3 2 c l 1 e 5 - 0 p d 1 9 f 3 / 1 3 1 2 9 / 9 1 / o 5 c 0 n / _ a 1 _ 8 0 6 1 1 4 4 6 1 8 0 p / d j o b c y n g _ u a e _ s 0 t 1 o 4 n 6 0 8 8 . p S d e f p e b m y b e g r u 2 e 0 s 2 t 3 / j f / t . o n 0 5 M a y 2 0 2 1 Wagner, A. D., Shannon, B. J., Kahn, I., & Buckner, R. L. (2005). Parietal lobe contributions to episodic memory retrieval. Trends in Cognitive Sciences, 9, 445–453. Wald, A. (1947). Sequential analysis. New York: Dover Publications. Wheeler, M. E., Petersen, S. E., & Buckner, R. L. (2000). Memory’s echo: Vivid remembering reactivates sensory- specific cortex. Proceedings of the National Academy of Sciences, U.S.A., 97, 11125–11129. Wilding, E. L. (2000). In what way does the parietal ERP old/new effect index recollection? International Journal of Psychophysiology, 35, 81–87. Wilding, E. L., & Rugg, M. D. (1996). An event-related potential study of recognition memory with and without retrieval of source. Brain, 119, 889–905. Wixted, J. T. (2009). Remember/know judgments in cognitive neuroscience: An illustration of the underrepresented point of view. Learning & Memory, 16, 406–412. Wolk, D. A., Schacter, D. L., Lygizos, M., Sen, N. M., Holcomb, P. J., Daffner, K. R., et al. (2006). ERP correlates of recognition memory: Effects of retention interval and false alarms. Brain Research, 1096, 148–162. Woodruff, C. C., Johnson, J. D., Uncapher, M. R., & Rugg, M. D. (2005). Content-specificity of the neural correlates of recollection. Neuropsychologia, 43, 1022–1032. Xiao, X., Dong, Q., Gao, J., Men, W., Poldrack, R. A., & Xue, G. (2017). Transformed neural pattern reinstatement during episodic memory retrieval. Journal of Neuroscience, 37, 2986–2998. Winocur, G., & Moscovitch, M. (2011). Memory transformation Yonelinas, A. P. (2002). The nature of recollection and and systems consolidation. Journal of International Neuropsychological Society, 17, 766–780. familiarity: A review of 30 years of research. Journal of Memory and Language, 46, 441–517. D o w n l o a d e d l l / / / / j f / t t i t . : / / f r o m D o h w t n t p o : a / d / e d m i f r t o p m r c h . p s i l d v i r e e r c t c . m h a i e r d . u c o o m c n / j a o r t c i c n e / - a p r d t i 3 2 c l 1 e 5 - 0 p d 1 9 f 3 / 1 3 1 2 9 / 9 1 / o 5 c 0 n / _ a 1 _ 8 0 6 1 1 4 4 6 1 8 0 p / d j o b c y n g _ u a e _ s 0 t 1 o 4 n 6 0 8 8 . p S d e f p e b m y b e g r u 2 e 0 s 2 t 3 / j / f t . o n 0 5 M a y 2 0 2 1 64 Journal of Cognitive Neuroscience Volume 32, Number 1 Tracking Your Mind’s Eye during Recollection: imagen

Descargar PDF