Seeing Sounds and Hearing Sights: The Influence of
Prior Learning on Current Perception
Jennifer D. Ryan1,2, Sandra N. Moses1, Melanie L. Ostreicher1,
Timothy Bardouille1, Anthony T. Herdman3, Lily Riggs1,2,
and Endel Tulving1,2
D
o
w
n
je
o
un
d
e
d
Abstrait
& It is well known that previous perceptual experiences alter
subsequent perception, but the details of the neural under-
pinnings of this general phenomenon are still sketchy. Ici,
we ask whether previous experiences with an item (tel que
seeing a person’s face) leads to the alteration of the neural
correlates related to processing of the item as such, ou si
it creates additional associative connections between such
substrates and those activated during prior experience. À
address this question, we used magnetoencephalography
(MEG) to identify neural changes accompanying subjects’
viewing of unfamiliar versus famous faces and hearing the
names of unfamiliar versus famous names. We were interested
in the nature of the involvement of auditory brain regions in
the viewing of faces, and in the involvement of visual regions
in the hearing of names. Evoked responses from MEG re-
cordings for the names and faces conditions were localized
to auditory and visual cortices, respectivement. Sans surprise,
peak activation strength of evoked responses was larger for
famous versus nonfamous names within the superior temporal
gyrus (STG), and was similar for famous and nonfamous
faces in the occipital cortex. More relevant to the issue of
experience on perception, peak activation strength in the STG
was larger for viewed famous versus nonfamous faces, et
peak activation within the occipital cortex was larger for heard
famous versus nonfamous names. Critique, these experience-
related responses were present within 150–250 msec of stimulus
début. These findings support the hypothesis that prior experi-
ences may influence processing of faces and names such that
is imparted on
perception encompasses more than what
the senses. &
INTRODUCTION
What we see or hear is determined not only by the sen-
sory stimuli and their context but also by our previous
experiences with the stimuli. In Principles of Psychol-
ogie, William James (1890) enunciated his famous Gen-
eral Law of Perception: ‘‘whilst part of what we perceive
comes through our senses from the object before us; un-
other part (and it may be the larger part) always comes
. . . out of our own head.’’ Now, more than a hundred
years later, research is outlining how our previous
experiences serve to influence and alter the very nature
de perception. The idea expressed in this statement is
now universally accepted, and a rich literature exists on
top–down and bottom–up processes in perception.
How are our perceptions changed by prior events? Pour
instance, imagine that you meet someone for the first
time and engage in conversation, and then you meet that
person again a month later. Is your perceptual experience
of the person’s face exactly the same a month later as it
was when you first met? If not, what changes? Un
possibility is that the processing of the face remains the
1The Rotman Research Institute, Baycrest, 2Université de
Toronto, 3Simon Fraser University, Toronto, Canada
same between both meetings, but the knowledge you
have learned about the person becomes available upon
viewing the face. Another possibility is that the knowl-
edge you have learned about the person actually changes
the way in which you perceive his or her face.
One way to address these questions is to examine neu-
ral responses and assess whether such activity can be
altered by previous experience during the time at which
perception is occurring. Ici, we consider ‘‘percep-
tion’’ to refer to the construction of a coherent repre-
sentation regarding sensory input and which can bring
to bear stored information regarding the item itself,
whereas ‘‘perceptual processing’’ may refer to the un-
derlying mechanisms involved in such construction. Le
issue here is whether prior learning experiences are cou-
pled with input from the senses to form such a percep-
tual representation. En effet, there is some indication
that the neural regions associated with perceptual pro-
cessing are altered by our experiences. Par exemple,
Nyberg, Habib, McIntosh, and Tulving (2000) exposed
their subjects to pairs of visually presented words and
auditorily presented sounds, and then showed them
these words alone, one at a time, while using functional
magnetic resonance imaging (IRMf) to look for brain
D 2008 Massachusetts Institute of Technology
Journal des neurosciences cognitives 20:6, pp. 1030–1042
je
je
/
/
/
/
j
t
t
F
/
je
t
.
:
/
/
F
r
o
m
D
h
o
t
w
t
n
p
o
:
un
/
d
/
e
m
d
je
F
t
r
o
p
m
r
c
h
.
s
p
je
je
d
v
je
e
r
e
r
c
c
t
.
h
m
un
je
r
e
.
d
toi
c
o
o
m
c
/
n
j
un
o
r
c
t
je
n
c
/
e
un
–
p
r
d
t
je
2
c
0
je
6
e
–
1
p
0
d
3
F
0
/
1
2
9
0
3
/
6
6
8
/
9
1
2
0
o
3
c
0
n
/
1
2
0
7
0
5
8
9
5
2
5
0
7
0
7
/
5
j
o
p
c
d
n
.
b
oui
2
0
g
0
toi
8
e
.
s
t
2
o
0
n
0
7
0
5
7
.
S
p
e
d
p
F
e
m
b
oui
b
e
g
r
toi
2
0
e
2
s
3
t
/
j
.
F
.
/
.
t
o
n
1
8
M.
un
oui
2
0
2
1
activation in visual and auditory regions. Initially, le
visual stimuli did not elicit activation within auditory
régions. Cependant, by virtue of pairing the words with
sounds, the visual presentation of the previously en-
countered words alone then led to activity in the au-
ditory regions. Similar findings were obtained by Wheeler,
Petersen, and Buckner (2000): The pairing of items, chaque
presented in its unique modality, later produced activa-
tion in neural regions underlying processing in an item’s
own presentation modality but also in regions of the as-
sociated item’s modality. These findings are intriguing,
but although these fMRI results illuminate the neural cor-
relates of the venerable phenomenon and concept of as-
sociation, questions remain regarding the association itself.
The purpose of the study reported here was to probe
more deeply into the nature of the kind of ‘‘association’’
demonstrated in the Nyberg et al. (2000) and Wheeler
et autres. (2000) études. Does the neural activity in the au-
ditory regions when the subject views a visually pre-
sented stimulus signify a neural ‘‘connection’’ (underlying
an acquired association or a mental ‘‘link’’) between the
previously presented stimuli, the ages-old standard view
(par exemple., Anderson, 1976)? Or does it perhaps signify a change
in the visual item’s processing, and a change in how it is
now ‘‘perceived’’? Autrement dit, does it perhaps tell
us something about the neural underpinnings of William
James’s Law of Perception?
Although these fMRI studies can tell us which brain
regions are related to associative processing of sensory
inputs, they did not throw much light on the issue we
are raising here: Can previous experience with an object
change the way the object is subsequently ‘‘perceived’’?
Although the above studies demonstrate that the pro-
cessing of stimuli is altered through prior experience, it is
not clear what stage of processing is affected. Is percep-
tual processing altered through prior experience or is
associated information activated following perceptual
traitement? The current work used a methodology that
provides information about the temporal dynamics of
the neural connection between the brain regions that
represent the association. Such an investigation would
illuminate the question we are discussing here, involving
‘‘auditory’’ neural activity upon the presentation of a vi-
sual stimulus. If we had information about the time when
the auditory region becomes active in relation to the
physical presentation of the visual stimulus, and vice-versa,
we would be in a better position to evaluate the issue of
whether our real-life experiences result in ‘‘changed
perception’’ versus ‘‘acquired associations.’’ That is, audi-
tory neural activity elicited by a visual stimulus that occurs
with the same time course as activity in visual regions may
suggest that prior experience alters our perceptual repre-
sentations. D'autre part, auditory activity elicited by
a visual stimulus that follows, in time, activity in visual
regions may suggest that our prior experiences are stored
as acquired associations and are merely reactivated upon
viewing a previously encountered stimulus.
Recently, researchers have provided initial evidence
that begins to address this issue. Moses, Martine, Houck,
Ilmoniemi, and Tesche (2005) presented their subjects
with delay and trace conditioning paradigms in which an
auditory noise burst was paired with a visual presenta-
tion of a geometric figure. Encore, presentation of the vi-
sual stimuli alone did not elicit activity within auditory
régions, cependant, following several trials of training in
which the noise and visual stimulus were associated,
the presentation of the geometric figure alone elicited a
response within the auditory cortex. Because the re-
searchers used magnetoencephalography (MEG), lequel
provides superior temporal
information compared to
IRMf, they were able to discern that this response within
the auditory cortex occurred early (environ 50 msec)
following the offset of the visual stimulus, at the time when
the presentation of the noise would have ordinarily been
expected. In a further analysis, Moses et al. (2007) noted
that following training, early responses were observed in
the amygdala within 200 msec following the onset of the
visual stimulus that had been previously paired with the
noise compared to visual stimuli that had never been
paired with the noise. This suggests, at the very least, que
the experiences learned by the subjects within the labora-
tory altered their subsequent neural responses, and pos-
sibly, perception of the items.
Cependant, do such early changes in neural activity rep-
resent a general and obligatory change in perception?
Such findings may merely reflect a temporary change, un
artifact of the experiment. Note that the above studies all
provided the learning within a single experimental ses-
sion. The observed changes in neural activity may only be
transient changes, lequel, although present during the
learning situation, may not occur at a later date even if the
knowledge regarding the link between the noise and the
picture can be brought forth. En plus, the observed
neural changes may not reflect an obligatory change
in perception because, for some of the above studies
(Nyberg et al., 2000; Wheeler et al., 2000), the learned
associations could have been explicitly recruited in an
effort to perform the tasks that were given to the subjects,
thereby resulting in increased activation in associated
neural regions. Par exemple, to recall whether a picture
had been previously viewed, it would be advantageous to
recall other elements that were present during the time
of the original learning episode in order to verify whether
one’s judgment of ‘‘previously studied’’ versus ‘‘novel’’ is
indeed accurate. Definitive evidence that our experiences
create changes to perception would require a demon-
stration that such neural changes are long-lasting and
occur even when the associated information may not be
relevant for the task at hand. For these reasons, nous
employed a paradigm here in which the ‘‘previous learn-
ing episodes’’ occurred in real
vie, and in which the
learned associated information was not presented at any
time during the study and was not needed in order to
complete the task we gave to the subjects.
Ryan et al.
1031
D
o
w
n
je
o
un
d
e
d
je
je
/
/
/
/
j
F
/
t
t
je
t
.
:
/
/
F
r
o
m
D
h
o
t
w
t
n
p
o
:
un
/
d
/
e
m
d
je
F
t
r
o
p
m
r
c
h
.
s
p
je
je
d
v
je
e
r
e
r
c
c
t
.
h
m
un
je
r
e
.
d
toi
c
o
o
m
c
/
n
j
un
o
r
c
t
je
n
c
/
e
un
–
p
r
d
t
je
2
c
0
je
6
e
–
1
p
0
d
3
F
0
/
1
2
9
0
3
/
6
6
8
/
9
1
2
0
o
3
c
0
n
/
1
2
0
7
0
5
8
9
5
2
5
0
7
0
7
/
5
j
o
p
c
d
n
.
b
oui
2
0
g
0
toi
8
e
.
s
t
2
o
0
n
0
7
0
5
7
.
S
p
e
d
p
F
e
m
b
oui
b
e
g
r
toi
2
0
e
2
s
3
t
/
j
F
.
.
/
.
t
o
n
1
8
M.
un
oui
2
0
2
1
information;
This study, alors, is the first exploration of whether
real-life experiences change perception. Like the Moses
et autres. (2005) étude, we used MEG to outline when differ-
ences in neural activity could be observed as a result of
prior experiences. Subjects viewed faces or heard names
of famous and nonfamous people. We examined the ex-
tent to which visual regions were activated upon hear-
ing famous versus nonfamous names, and likewise, le
extent to which auditory regions were activated upon
viewing famous versus nonfamous faces. We did not ask
subjects to reflect upon the contents of their memories
during the task or to make use of associated information
in any way; rather we solely instructed them to judge
whether a given face/name was male or female. If our
previous experiences, en effet, change our perceptual rep-
resentations, famous stimuli should elicit early neural
responses in regions which are typically activated during
the processing of associated information. Par exemple,
viewing a famous face should elicit early activity from re-
gions ordinarily related to the processing of auditory/
verbal
de même, hearing a famous name
should elicit early activity from regions ordinarily related
to the processing of visual/facial information. In order to
suggest that prior experiences result in a change in per-
réception, such responses across multiple sensory regions
should occur within the time window that is generally
assumed to be related to perceptual processing rather
than later stages of processing which may include con-
ceptual processing or semantic elaboration. Using find-
ings from the event-related brain potential (ERP) literature
as a guide, we suggest that differences in neural activity
between famous and nonfamous conditions that occur
prior to 300 msec may predominantly reflect changes in
perceptual processing rather than changes in postpercep-
tual processes, such as the access to semantic or concep-
tual information (par exemple., Pickering & Schweinberger, 2003;
Schweinberger, Pickering, Burton, & Kaufmann, 2002), al-
though precise temporal delineation between perceptual
and postperceptual processes remains controversial (Itier,
Herdman, Georges, Cheyne, & Taylor, 2006). En plus,
we would take findings of simultaneous (as opposed to
successive) responses within visual and auditory regions
for the famous conditions as evidence of altered percep-
tual representations rather than as the activation of dis-
tinct, associated information. To return to the example of
meeting someone new, such findings of early neural
changes as a result of prior experience would show that
learning something about the person behind the face al-
ters the very manner in which you perceive the face when
you meet him or her again.
participated in exchange for monetary compensation.
None of those who participated in the study had a history
of any psychiatric illness. Consent was obtained and the
rights of the participants were protected, and the guide-
lines of the Toronto Academic Health Science Council
were upheld.
Stimuli and Design
Subjects were presented with visual and auditory stim-
uli. The visual stimuli were 160 pictures of famous and
160 pictures of nonfamous faces that have been used
in previous studies (Hannula, Ryan, Tranel, & Cohen,
2007; Ryan, Hannula, & Cohen, 2007). Equal numbers
of female and male stimuli were presented across the
famous and nonfamous conditions. The faces were se-
lected from a larger set of faces, all of which had as-
sociated rating scores from previous norming studies
regarding the ‘‘famousness’’ and ‘‘nameability’’ of each
face. Selected famous faces were those which had an
average score of 3 or higher on a 5-point scale of ‘‘fa-
mousness’’ (1 = nonfamous, 5 = famous) and an
average score of 3 or higher on a 5-point scale of
‘‘nameability’’ (1 = no associated name, 5 = first and
last names readily available). Selected nonfamous faces
were those that had an average score of less than 2 sur
both scales. Faces were equated for brightness and
placed on a 1024 par 768 black background with the
eyes of each face centered in approximately the same
location. The screen width measured 16.5 dans. and sub-
tended approximately 298 of visual angle when partic-
ipants were seated 32 dans. from the screen. The faces
comprised approximately 5 dans. of the screen width or 98
of visual angle when viewed from a distance of 32 dans.
The auditory stimuli consisted of spoken names of fa-
mous and nonfamous men and women. All names were
recorded by a professional voice artist. The famous names
corresponded to the set of famous faces as outlined
au-dessus de. Nonfamous male and female names were con-
structed such that the first and last names of a given
nonfamous name began with the same starting phoneme
and had the same syllable length as a corresponding first
and last names of a famous person (par exemple., ‘‘Vern Kilston’’
was the nonfamous counterpart to ‘‘Val Kilmer’’). Fa-
mous names were, on average, 1088 msec in length
(SD = 138 msec); nonfamous names were, on average,
1159 msec in length (SD = 201 msec). The nonfamous
names were uniquely associated with individual nonfa-
mous faces.
MÉTHODES
Sujets
Ten adults (3 men; 9 right-handed), aged 21–30 years
(âge moyen = 23.86 années), from the Toronto community,
Procedure
Prior to MEG recordings, a survey was administered to
assess the subjects’ familiarity with entertainment celeb-
rities (see Appendix). Following this, a subset of the
1032
Journal des neurosciences cognitives
Volume 20, Nombre 6
D
o
w
n
je
o
un
d
e
d
je
je
/
/
/
/
j
t
t
F
/
je
t
.
:
/
/
F
r
o
m
D
h
o
t
w
t
n
p
o
:
un
/
d
/
e
m
d
je
F
t
r
o
p
m
r
c
h
.
s
p
je
je
d
v
je
e
r
e
r
c
c
t
.
h
m
un
je
r
e
.
d
toi
c
o
o
m
c
/
n
j
un
o
r
c
t
je
n
c
/
e
un
–
p
r
d
t
je
2
c
0
je
6
e
–
1
p
0
d
3
F
0
/
1
2
9
0
3
/
6
6
8
/
9
1
2
0
o
3
c
0
n
/
1
2
0
7
0
5
8
9
5
2
5
0
7
0
7
/
5
j
o
p
c
d
n
.
b
oui
2
0
g
0
toi
8
e
.
s
t
2
o
0
n
0
7
0
5
7
.
S
p
e
d
p
F
e
m
b
oui
b
e
g
r
toi
2
0
e
2
s
3
t
/
j
t
.
/
.
F
.
o
n
1
8
M.
un
oui
2
0
2
1
auditory stimuli was played backward to the subjects to de-
tect their threshold level of hearing. Auditory stimuli were
then presented at 40 dB above this threshold level. Visual
and auditory stimuli were presented using Presentation.
The MEG data collection was synchronized with the
onset of visual and auditory stimulus presentation by
recording the luminance of the screen and the audio
sound envelope, respectivement. Four blocks of stimuli
were presented, two of which were composed of visual
stimuli only, and two were composed of auditory stimuli
only. Each block contained 80 stimuli, 40 famous and 40
nonfamous, presented in random order. Equal numbers
of male/female stimuli were presented across conditions
and across blocks. Subjects either saw the face or heard
the name of a given famous person (and their nonfa-
mous counterpart) during the course of the experiment;
Par exemple, participants either saw Val Kilmer’s (or Vern
Kilston’s) face or heard his name, but not both. Half of
the subjects were exposed to an auditory block first, et
half were presented with a visual block at the beginning
of the experiment. Subjects were asked to judge whether
the presented stimulus was either male or female. Re-
sponses were made via button presses. Although success-
ful performance of male/female judgments may require
retrieval of semantic knowledge pertaining to the typical-
ity of male/females names or the features of male versus
female faces, such retrieval of general semantic knowl-
edge should occur similarly for famous and nonfamous
stimuli. Sex and recognition judgments have been disso-
ciated through neuropsychological studies of prosopag-
nosic patients, (Sergent & Villemure, 1989); and may occur
through separate neural routes (Mouchetant-Rostaing,
Giard, Bentin, Aguera, & Pernier, 2000; Bruce & Jeune,
1986). Sex judgments do not require the retrieval of iden-
tity or of other specific associated information related to
identité; de même, retrieval of identity does not require the
categorization of sex (Wild et al., 2000; Bruce, Élise, Gibling,
& Jeune, 1987 ). Donc, any differences between the
famous and nonfamous condition will be taken as evi-
dence for unintentional retrieval of associated information
specific to the stimulus.
Data Acquisition
All MEG data were recorded in a magnetically shielded
room using a 151-channel whole-head first-order gradi-
ometer system (VSM Medtech, Port Coquitlam, BC, Canada)
with detection coils spaced 31 mm apart. Data ac-
quisition was 625 Hz. To monitor head movement, three
small coils of wire were attached to the participant’s
head and energized with small currents before and after
the task. This determined the location of the partic-
ipant’s head with respect to the MEG array. For each
sujet, a structural MRI was obtained using a 1.5-T
Siemens Magnetom system at Sunnybrook’s Women’s
Hospital.
Data Analysis
Preparation for Source Analysis
Eye-movement and blink artifacts were removed off-line
from single-trial MEG data using principle components
analyse (Kobayashi & Kuriki, 1999). This approach has
been shown effective for removing artifacts with ampli-
tudes larger than the brain signals of interest (Lagerlund,
Sharbrough, & Busacker, 1997 ). Principal components
with amplitudes larger than 1 pT at any time in any MEG
sensor were removed from the single-trial magnetic field
data. Data were then averaged over 2000-msec epochs,
time locked to the onset of the stimuli. The averages
were low-pass filtered at 50 Hz, and high-pass filtered at
2 Hz. A baseline for the measured response was selected
à (cid:1)100 à 0 msec prior to stimulus onset. A minimum
norm least squares (MNLS; L2 norm) algorithm from
the Curry 5.05 software package was used to estimate
the current distribution in the brain from the MEG data.
A realistically shaped boundary element model for the
conducting volume of the brain was extracted from in-
dividual subjects’ MRIs for magnetic field computations
in the MNLS analysis.
Extractions of Regions of Interest
The MNLS algorithm permits detailed modeling of dis-
tributed activity with no a priori assumptions about the
number of active regions. The following procedure was
used to select specific regions of interest (ROI) depuis
the MNLS inversion. (1) Whole-brain current flow distri-
bution maps were thresholded at 80% of the maximum
value at each time slice. Using this thresholded map, dans-
spection of the MNLS inversions revealed that current
flow in the bilateral superior temporal gyrus (STG) et
the bilateral occipital cortex appeared as disjoint vol-
umes of activation upon visual inspection (par exemple., McDowell,
Kissler, Berger, et coll., 2005). (2) ROIs were selected that
encompassed regions of high activation that overlapped
the left and right STG and the left and right occipital cor-
tex by 1 cm, and were retained for further analysis. ROIs in
and near the auditory cortex were selected from the non-
famous names condition. ROIs in and near the visual cor-
tex were selected from the nonfamous faces condition. Si
sources appeared ambiguous, the threshold was raised
à 90% of maximum to obtain a smaller ROI, and/or the
famous names/faces condition was consulted for similar
sources. (3) Source strength waveforms over time were ex-
tracted for the ROIs from the MNLS output using MatLab.
Analysis of Waveform Amplitude Peaks and Latencies
Latencies and amplitudes of evoked responses were de-
termined from the waveforms for the first 400 msec
following stimulus onset. Amplitudes of responses were
computed at each sample (1.6 msec) within the 400-msec
time window. The single highest amplitude peak for this
Ryan et al.
1033
D
o
w
n
je
o
un
d
e
d
je
je
/
/
/
/
j
t
t
F
/
je
t
.
:
/
/
F
r
o
m
D
h
o
t
w
t
n
p
o
:
un
/
d
/
e
m
d
je
F
t
r
o
p
m
r
c
h
.
s
p
je
je
d
v
je
e
r
e
r
c
c
t
.
h
m
un
je
r
e
.
d
toi
c
o
o
m
c
/
n
j
un
o
r
c
t
je
n
c
/
e
un
–
p
r
d
t
je
2
c
0
je
6
e
–
1
p
0
d
3
F
0
/
1
2
9
0
3
/
6
6
8
/
9
1
2
0
o
3
c
0
n
/
1
2
0
7
0
5
8
9
5
2
5
0
7
0
7
/
5
j
o
p
c
d
n
.
b
oui
2
0
g
0
toi
8
e
.
s
t
2
o
0
n
0
7
0
5
7
.
S
p
e
d
p
F
e
m
b
oui
b
e
g
r
toi
2
0
e
2
s
3
t
/
j
.
.
F
t
.
/
o
n
1
8
M.
un
oui
2
0
2
1
epoch was selected for analysis. If several peaks occurred
in the waveform during this epoch, the largest was se-
lected for analysis.
Peak amplitudes for the identical STG sources were
determined for the famous and nonfamous names con-
dition, as well as the famous and nonfamous faces
condition. De la même manière, peak amplitudes for identical bilat-
eral occipital sources were determined for the famous
and nonfamous faces condition, as well as the famous
and nonfamous names condition. The peak amplitude of
the ROIs in the famous conditions was compared to
those in the nonfamous conditions within ±30 msec of
the peak selected for the famous condition. These cri-
teria for selecting activity peaks in MEG data for com-
parison across conditions are similar to those used in
previous MEG studies of auditory activation (Hanlon et al.,
2005; Moses et al., 2005; Thoma et al., 2003, 2005). Le
peak amplitudes for the famous condition were greater
than or equal to three standard deviations above the mean
baseline amplitude.
Sources that yielded waveforms that appeared to be
extremely noisy showing no clear peaks were excluded
from analysis. En outre, occasionally, sources were not
found over specific regions for particular subjects. These
situations resulted in three missing data points for right
occipital sources in the names condition, and one right
occipital source in the faces condition.
Statistical Analysis
Peak amplitudes and latencies for bilateral STG and oc-
cipital sources found in the names and faces conditions,
respectivement, were examined using repeated measure
analysis of variance (ANOVA). Autrement dit, we exam-
ined auditory activity in the names condition and visual
activity in the faces condition. Within-subjects factors in-
cluded ‘‘source’’ (STG/occipital), ‘‘hemisphere’’ (right/
gauche), and ‘‘stimulus’’ (famous/nonfamous). Peak ampli-
tudes and latencies for the identical auditory and visual
sources found in the faces and names conditions, concernant-
spectively, were also compared using similar ANOVA,
c'est, we examined auditory activity in the faces con-
dition and visual activity in the names condition. Peak
amplitudes of greater than two standard deviations away
from the group mean for each condition were consid-
ered outliers and were removed from analysis. This re-
sulted in the exclusion of one subject’s left occipital source
found for the faces condition, the same subject’s right STG
source found for the faces condition, and another sub-
ject’s left STG source found for the faces condition.
RÉSULTATS
Familiarity with Entertainment Celebrities
see Appendix). Subjects viewed about two movies per
month (mean score of 5.07 ). Celebrity magazines and
Websites were read once every few months (mean
scores of 3.00 et 3.36, respectivement). Dans l'ensemble, sujets
expressed a fair knowledge of celebrities (mean score
de 4.21).
Behavioral Responses
Measures of response time and accuracy were analyzed
in a repeated measures ANOVA with the within-subjects
factors of modality type (face, nom), famousness (fa-
mous, nonfamous), and sex of the presented stimulus
(male, female). Due to recording error, response data
were obtained for all conditions for 7 of the 10 sujets.
The results that follow are for those seven subjects. UN
significant main effect of modality type was observed for
response time [F(1, 6) = 29.29, p < .01] but not for
accuracy (F < 1), whereas responses were faster for the
faces than for the names, responses were equivalent on
accuracy across the auditory and visual conditions.
There was a significant effect of fame on the speed of
response [F(1, 6) = 80.63, p < .001] and accuracy [F(1,
6) = 27.16, p < .01]; responses to famous stimuli were
faster and more accurate than responses to the non-
famous stimuli. A significant effect of sex was also ob-
served; responses were faster [F(1, 6) = 21.05, p < .01]
and more accurate [F(1, 6) = 15.34, p < .01] for men
than women. There was a significant interaction be-
tween modality and fame of the presented stimuli for
response time [F(1, 6) = 36.1, p = .001] and accuracy
[F(1, 6) = 19.1, p < .01]; response times and accuracy
were similar across famous and nonfamous faces, but
responses for famous names were faster and more
accurate than responses for nonfamous names. Critically,
the effects of sex did not interact with the effects of fame
on either measure. This suggests that subjects were
responding to the sex discrimination task in a similar fash-
ion for famous and nonfamous stimuli.
D
o
w
n
l
o
a
d
e
d
l
l
/
/
/
/
j
f
/
t
t
i
t
.
:
/
/
f
r
o
m
D
h
o
t
w
t
n
p
o
:
a
/
d
/
e
m
d
i
f
t
r
o
p
m
r
c
h
.
s
p
i
l
d
v
i
e
r
e
r
c
c
t
.
h
m
a
i
r
e
.
d
u
c
o
o
m
c
/
n
j
a
o
r
c
t
i
n
c
/
e
a
-
p
r
d
t
i
2
c
0
l
6
e
-
1
p
0
d
3
f
0
/
1
2
9
0
3
/
6
6
8
/
9
1
2
0
o
3
c
0
n
/
1
2
0
7
0
5
8
9
5
2
5
0
7
0
7
/
5
j
o
p
c
d
n
.
b
y
2
0
g
0
u
8
e
.
s
t
2
o
0
n
0
7
0
5
7
.
S
p
e
d
p
f
e
m
b
y
b
e
g
r
u
2
0
e
2
s
3
t
/
j
t
.
.
/
.
f
o
n
1
8
M
a
y
2
0
2
1
Auditory Activation for Names and Visual
Activation for Faces
We examined strength and latency of the strongest
peaks for STG sources in the famous and nonfamous
names condition, and for occipital sources in the famous
and nonfamous faces condition. Figure 1 shows sensor
waveforms for the nonfamous faces and the nonfamous
names conditions. Figure 2 shows the STG sources for
the nonfamous names condition for a representative sub-
ject and occipital sources from the nonfamous faces con-
dition for a representative subject, as identified using the
MNLS algorithm.
On average, subjects watched approximately 5 to 6 hr of
television per week (mean score of 3.86 on a 1–7 scale;
The amplitude of STG activation was greater for the
famous compared to the nonfamous names (see Fig-
1034
Journal of Cognitive Neuroscience
Volume 20, Number 6
Figure 1. (A) Sensor
waveforms for a representative
subject for the nonfamous
faces conditions. (B) Sensor
waveforms for a representative
subject for the nonfamous
names condition.
D
o
w
n
l
o
a
d
e
d
l
l
/
/
/
/
j
t
t
f
/
i
t
.
:
/
/
f
r
o
m
D
h
o
t
w
t
n
p
o
:
a
/
d
/
e
m
d
i
f
t
r
o
p
m
r
c
h
.
s
p
i
l
d
v
i
e
r
e
r
c
c
t
.
h
m
a
i
r
e
.
d
u
c
o
o
m
c
/
n
j
a
o
r
c
t
i
n
c
/
e
a
-
p
r
d
t
i
2
c
0
l
6
e
-
1
p
0
d
3
f
0
/
1
2
9
0
3
/
6
6
8
/
9
1
2
0
o
3
c
0
n
/
1
2
0
7
0
5
8
9
5
2
5
0
7
0
7
/
5
j
o
p
c
d
n
.
b
y
2
0
g
0
u
8
e
.
s
t
2
o
0
n
0
7
0
5
7
.
S
p
e
d
p
f
e
m
b
y
b
e
g
r
u
2
0
e
2
s
3
t
/
j
.
t
/
.
.
f
o
n
1
8
M
a
y
2
0
2
1
ure 3), as illustrated by a main effect of ‘‘stimulus’’ in
planned comparisons [F(1, 9) = 12.9, p = .005], and
separate t tests on the left [t(9) = 3.3, p < .01] and right
[t(9) = 3.1, p = .01] hemispheres. The strongest peaks
of STG activity occurred following presentation of the
famous names at an average of 211 msec (SD = 81)
in the left hemisphere and 204 msec (SD = 60) in the
right, and following the nonfamous names at 208 msec
(SD = 88) in the left and 207 msec (SD = 59) in the
right. No significant differences in latency were detected
between the famous and nonfamous names for STG
peaks.
There were no differences in amplitude of occipital
peaks for the famous and nonfamous faces (see Fig-
ure 4). The strongest peaks of occipital activity occurred
following presentation of the famous faces at an aver-
age of 135 msec (SD = 61) in the left hemisphere and
117 msec (SD = 29) in the right, and following the non-
famous faces at 137 msec (SD = 59) in the left and 124
(SD = 28) in the right. No significant differences in
latency were detected between the famous and non-
famous faces for occipital peaks. The strongest peaks in
occipital sources elicited by the faces occurred earlier
than the peaks in the STG sources elicited by the names
[F(1, 8) = 17.0, p = .003].
Auditory Activation for Faces and Visual Activation
for Names
We examined strength and latency of the strongest peaks
for STG sources in the famous and nonfamous faces
Ryan et al.
1035
the left hemisphere and 171 msec (SD = 56) in the
right, and following the nonfamous faces at 195 msec
(SD = 54) in the left and 172 msec (SD = 53) in the
right.
Amplitudes of occipital peaks were greater for the
famous compared to the nonfamous names (see Fig-
ure 6), as illustrated by separate t test for each hemi-
sphere [left: t(9) = 2.8, p < .05; right: t(6) = 2.8, p < .05].
No differences in hemisphere were found for occipital
peaks.
The strongest peaks of occipital activity occurred fol-
lowing presentation of the famous names at an average
of 208 msec (SD = 67 ) in the left hemisphere and
201 msec (SD = 51) in the right, and following the non-
famous names at 210 msec (SD = 68) in the left and
198 msec (SD = 48) in the right. No significant differ-
ences in latency were detected between the famous and
nonfamous conditions for STG or occipital peaks when
responses to faces and names were combined. How-
ever, when the latencies for the strongest peak within
occipital and STG regions were examined separately
D
o
w
n
l
o
a
d
e
d
l
l
/
/
/
/
j
f
/
t
t
i
t
.
:
/
/
f
r
o
m
D
h
o
t
w
t
n
p
o
:
a
/
d
/
e
m
d
i
f
t
r
o
p
m
r
c
h
.
s
p
i
l
d
v
i
e
r
e
r
c
c
t
.
h
m
a
i
r
e
.
d
u
c
o
o
m
c
/
n
j
a
o
r
c
t
i
n
c
/
e
a
-
p
r
d
t
i
2
c
0
l
6
e
-
1
p
0
d
3
f
0
/
1
2
9
0
3
/
6
6
8
/
9
1
2
0
o
3
c
0
n
/
1
2
0
7
0
5
8
9
5
2
5
0
7
0
7
/
5
j
o
p
c
d
n
.
b
y
2
0
g
0
u
8
e
.
s
t
2
o
0
n
0
7
0
5
7
.
S
p
e
d
p
f
e
m
b
y
b
e
g
r
u
2
0
e
2
s
3
t
/
j
f
/
.
.
t
.
o
n
1
8
M
a
y
2
0
2
1
Figure 3. (A) Strength of response over time to famous and
nonfamous names for a source overlying the left STG in a
representative participant. (B) Strength of peak activation from left and
right STG sources following presentation of famous and nonfamous
names averaged across participants. Peak amplitude of STG activation
was greater for the famous compared to the nonfamous names.
Figure 2. Auditory response to nonfamous names (top). Regions of
activity obtained over the left (A) and right (B) STG using the MNLS
algorithm superimposed onto an individual subject’s MRI; coronal and
axial slices are displayed. Visual response to nonfamous faces
(bottom). Regions of activity obtained over the left (C) and right (D)
occipital cortex superimposed onto an individual subject’s MRI; sagittal
and axial slices are displayed.
condition, and for occipital sources in the famous and
nonfamous names condition.
The amplitude of STG and occipital activation was
greater for the famous compared to the nonfamous
conditions (see Figure 5). This effect is illustrated by a
main effect of ‘‘stimulus’’ in the overall ANOVA [F(1,
5) = 9.8, p < .05], and by a separate t test on the STG
[left: t(8) = 3.0, p < .05; right: t(8) = 2.8, p < .05].
Additionally, STG activation in the right hemisphere
was stronger than in the left, collapsed across condi-
tions [F(1, 7) = 12.7, p < .01]. The strongest peaks
of STG activity occurred following presentation of the
famous faces at an average of 201 msec (SD = 50) in
1036
Journal of Cognitive Neuroscience
Volume 20, Number 6
whom the phenomenon does not depend on experi-
ence (Ramachandran & Hubbard, 2001). Here, these re-
sponses were likely driven by information that, through
experience, had come to be associated with the pre-
sented item, such as the name or occupation that ac-
companies a particular face. We observed these effects
early during processing, within 150–250 msec, which
is within the time window during which largely per-
ceptual processes are thought to occur as opposed to a
later time window during which conceptual/semantic
processes and/or the retrieval of associated information
are largely purported to occur (e.g., Itier et al., 2006;
Schweinberger et al., 2002; Donaldson & Rugg, 1998,
1999).
Our findings not only add to the growing claims that
the neural regions that are on-line during processing are
altered through previous experience (Prince, Daselaar,
& Cabeza, 2005), but our work further suggests that such
changes are obligatory. Previous work that outlined
the changes in neural activation during processing as a
function of prior learning explicitly required subjects to
D
o
w
n
l
o
a
d
e
d
l
l
/
/
/
/
j
t
t
f
/
i
t
.
:
/
/
f
r
o
m
D
h
o
t
w
t
n
p
o
:
a
/
d
/
e
m
d
i
f
t
r
o
p
m
r
c
h
.
s
p
i
l
d
v
i
e
r
e
r
c
c
t
.
h
m
a
i
r
e
.
d
u
c
o
o
m
c
/
n
j
a
o
r
c
t
i
n
c
/
e
a
-
p
r
d
t
i
2
c
0
l
6
e
-
1
p
0
d
3
f
0
/
1
2
9
0
3
/
6
6
8
/
9
1
2
0
o
3
c
0
n
/
1
2
0
7
0
5
8
9
5
2
5
0
7
0
7
/
5
j
o
p
c
d
n
.
b
y
2
0
g
0
u
8
e
.
s
t
2
o
0
n
0
7
0
5
7
.
S
p
e
d
p
f
e
m
b
y
b
e
g
r
u
2
0
e
2
s
3
t
/
j
t
.
f
/
.
.
o
n
1
8
M
a
y
2
0
2
1
Figure 5. (A) Strength of response over time to famous and
nonfamous faces for a source overlying the left STG in a representative
participant. (B) Strength of peak activation from left and right STG
sources following presentation of famous and nonfamous faces
averaged across participants. Peak amplitude of STG activation was
greater for the famous compared to the nonfamous faces.
Ryan et al.
1037
Figure 4. (A) Strength of response over time to famous and
nonfamous faces for a source overlying the left occipital cortex in a
representative participant. (B) Strength of peak activation from left and
right occipital cortex sources following presentation of famous and
nonfamous faces averaged across participants. There were no
differences in amplitude of occipital peaks for the famous and
nonfamous faces.
for the names and faces conditions, there was a non-
significant effect of ‘‘source’’ for the names condition
[F(1, 6) = 1.33, p > .25], but a significant effect of
‘‘source’’ for the faces condition [F(1, 8) = 7.18, p < .05].
Therefore, the strongest peak of activity within STG
and occipital regions occurred at similar latency for the
names, but latency of the strongest peak in the occipital
cortex preceded that from the STG when faces were
presented.
DISCUSSION
Our prior experiences alter our perceptions. Notably,
responses within the auditory cortex were larger for
visually presented famous versus nonfamous faces.
Likewise, activation within the visual cortex was larger
for auditorily presented famous versus nonfamous
names. The cross-modal activity pattern seen here is
not unlike that found in synesthetic individuals in
auditorily presented words elicited greater activation,
as revealed by fMRI, within the STG compared to novel
words. Such modulation of responses is apparent early
in processing; here, peak differences were observed
between famous and nonfamous names in the STG
approximately 200 msec following stimulus onset. These
latency findings are consistent with other MEG find-
ings from Marinkovic et al. (2003) and Simos, Breier,
Zouridakis, and Papanicolaou (1998). Simos et al. (1998)
observed neural responses to a continuous recognition
task of spoken words localized to the STG that arose
within 150–200 msec of stimulus onset. Marinkovic et al.
found differential responses within the superior tem-
poral plane for repeated versus novel words approxi-
mately 225 msec following stimulus onset. Therefore, in
these previous studies and the current work, auditory
responses distinguished previously learned (famous
names) from novel (nonfamous names) auditory stimuli
early in processing.
Although the auditory names were not presented
along with the visual faces, neural responses within the
STG were also significantly greater for famous faces com-
pared to nonfamous faces. This is consistent with the
work from Nyberg et al. (2000) and Wheeler et al. (2000)
which demonstrated activation in auditory regions upon
presentation of a lone visual stimulus which had been
previously associated with an auditory item. It is not un-
reasonable to expect activation in neural regions associ-
ated with auditory processing as a function of prior
learning, even in the absence of an acoustic stimulus,
when findings from auditory imagery are considered.
For instance, Kraemer, Macrae, Green, and Kelley (2005)
had subjects passively listen to familiar and unfamiliar
excerpts of music in which a silent gap was inserted.
Greater activation was observed in the primary auditory
cortex and auditory association regions during the silent
gap of familiar versus unfamiliar songs. Altogether, re-
sponses in the auditory cortex may be revealed during
processing of a visual target item if that visual item had
been associated with auditory information during a prior
learning event.
Responses within the occipital cortex did not distin-
guish famous from nonfamous faces. Nevertheless, the
findings here of increased neural responses to famous
compared to nonfamous names within the occipital cor-
tex harken back to findings from work on visual imagery,
and suggest that neural activity within the occipital cor-
tex can be affected through prior experience. Kosslyn,
Thompson, Kim, and Alpert (1995) demonstrated in-
creased activation in the primary visual cortex as revealed
by positron emission tomography when subjects were
asked to visualize previously studied objects in response
to an auditory prompt. From this, Kosslyn et al. (1995) sug-
gested that although primary visual areas may not be the
site of storage for visual representations, higher-level areas
which are involved in the storage of visual information
may induce differential activity in these primary sensory
Figure 6. (A) Strength of response over time to famous and
nonfamous names for a source overlying the left occipital cortex in a
representative participant. (B) Strength of peak activation from left and
right occipital cortex sources following presentation of famous and
nonfamous names averaged across participants. Peak amplitude of
occipital activation was greater for the famous compared to the
nonfamous names.
access their memories and/or to use any remembered
associated information to aid in the recognition memory
judgments (Nyberg et al., 2000; Wheeler et al., 2000).
However, in our study, associated information was not
present during the experimental session in the current
work and was not needed to aid in task performance.
Thus, the differences in neural responses between the
famous and nonfamous stimuli were observed even
though we did not require subjects to remark on the
contents of their memories. As such, we propose that
prior experiences not only influence processing but that
they do so in a compulsory manner. Below, we describe
the modulation of neural responses within the STG and
the occipital cortex as a function of prior learning in the
context of a variety of findings, from visual imagery to
word meaning, which ultimately serves to address the
fundamental question of how our prior experiences may
fundamentally alter our perceptions.
Here, neural responses within the STG were modu-
lated by prior learning. This is consistent with Saykin
et al. (1999), who demonstrated that previously studied
1038
Journal of Cognitive Neuroscience
Volume 20, Number 6
D
o
w
n
l
o
a
d
e
d
l
l
/
/
/
/
j
t
t
f
/
i
t
.
:
/
/
f
r
o
m
D
h
o
t
w
t
n
p
o
:
a
/
d
/
e
m
d
i
f
t
r
o
p
m
r
c
h
.
s
p
i
l
d
v
i
e
r
e
r
c
c
t
.
h
m
a
i
r
e
.
d
u
c
o
o
m
c
/
n
j
a
o
r
c
t
i
n
c
/
e
a
-
p
r
d
t
i
2
c
0
l
6
e
-
1
p
0
d
3
f
0
/
1
2
9
0
3
/
6
6
8
/
9
1
2
0
o
3
c
0
n
/
1
2
0
7
0
5
8
9
5
2
5
0
7
0
7
/
5
j
o
p
c
d
n
.
b
y
2
0
g
0
u
8
e
.
s
t
2
o
0
n
0
7
0
5
7
.
S
p
e
d
p
f
e
m
b
y
b
e
g
r
u
2
0
e
2
s
3
t
/
j
.
.
f
/
t
.
o
n
1
8
M
a
y
2
0
2
1
regions. Therefore, neural activity in the occipital cortex,
like the STG, may be altered due to prior experience and
stored information.
Our findings of neural activation within the occipital
cortex approximately 150–200 msec following stimulus
onset of famous or nonfamous faces are consistent with
Iidaka, Matsumoto, Haneda, Okada, and Sadato (2006),
who demonstrated through fMRI and ERP experiments
conducted on the same group of subjects that the N170,
a negative deflection in ERP occurring approximately
170 msec (140–200 msec; Itier et al., 2006) following the
onset of face stimuli, is correlated with hemodynamic
responses within occipital cortices. Our findings of larger
neural responses for famous versus nonfamous names,
at approximately 200 msec following the onset of the
name, suggest that the famous name had become as-
sociated with visual
information, presumably a face,
through prior learning episodes. When considered with
the imagery findings mentioned above, we find that the
activation observed in the occipital cortex is altered due
to past experiences.
It should be noted that famous and nonfamous names
each elicited some activation within the occipital cortex,
and that famous and nonfamous faces each elicited some
activation within the STG. This may be surprising as one
does not expect visual information to be associated with
nonfamous names; likewise, auditory/verbal information
is not assumed to be associated with nonfamous faces.
In the current paradigm, famous and nonfamous stimuli
were intermixed, thereby likely prompting subjects to
expect famous and nonfamous stimuli; thus, we may be
observing initial ‘‘false fame’’ responses to the nonfa-
mous stimuli. The nonfamous faces were obtained from
magazines and catalogs and therefore appear similar in
picture quality to pictures of famous people, or perhaps
resemble other people the subjects had encountered
in real life. Also, the names were controlled such that
the initial phonemes of the first and last names were
the same across famous and nonfamous stimuli. In fact,
results from a subsequent behavioral paradigm, in which
subjects were played either a 200-msec truncated or
full version of the famous and nonfamous name stim-
uli used here, revealed that previously stored memories
were accessed even during the presentation of nonfa-
mous names (Ryan, Moses, Ostreicher, & Riggs, in prep-
aration). Although subjects classified famous truncated
names as ‘‘famous’’ more often than the nonfamous
counterparts, subjects were still more likely to classify
the nonfamous truncated names as ‘‘famous’’ than
‘‘nonfamous,’’ and even generated a famous name to
the nonfamous truncated names on over half of the
trials when asked to guess what the full name might
have been. By contrast, full versions of the names were
accurately classified as famous/nonfamous. Therefore,
the nonfamous stimuli (regardless of whether they are
faces or names) in the current paradigm may have in-
advertently activated a search through a similar set of
representations of famous (or otherwise, known) peo-
ple which are then perhaps not maintained in an active
state upon further processing of the auditory or visual
stimuli. Nevertheless, the responses observed in the
occipital cortex were larger for famous versus nonfa-
mous names, and the responses observed in the STG
were larger for famous versus nonfamous faces, suggest-
ing that our prior experiences alter the responses within
associated neural regions that are observable early in
processing.
Although it is clear that processing is modified
through prior learning, a question remains regarding
how (or even whether) associated information is exert-
ing its influence on perception. In particular, do the
present findings reflect the cortical reactivation of sep-
arate, but associated, information, or is perception itself
fundamentally altered through experience? The stan-
dard view has proposed that through prior learning
episodes, associated information may be added as a
separate link to the target item, without necessarily
affecting the nature of the representation of the target
item itself (Anderson, 1976). Indeed, the current work
could be taken to support the notion that the process-
ing of an item results in the rapid access and activation
of distinct, but associated information, through previ-
ously established links or connections. However, pro-
cessing was affected early by associated information. In
particular, prior experiences altered neural responses
during a time at which it is hypothesized that the
perceptual representation is formed (Roisson et al.,
2000; Bentin, Allison, Puce, Perez, & McCarthy, 1996).
It should be noted that there are indications that con-
ceptual or identification processes could occur to some
extent within this time window as well ( Jemel, Pisani,
Calabria, Crommelinck, & Bruyer, 2003; Itier & Taylor,
2002). Thus, although we speculate here that prior ex-
periences may alter perception itself, further work re-
mains to rule out the contribution of conceptual/semantic
processes to the observed effects.
Further evidence from the current work speaks to
the notion that perception may be fundamentally al-
tered through prior experience. We found no differ-
ences in the peak latency of responses between the
STG and the occipital cortex for auditory names. When
considered in isolation, this finding would suggest that
rather than the auditory stimulus reactivating associated
visual
information, the representation of the famous
person encompasses both the visual and auditory infor-
mation within a single representation. However, upon
presentation of a visual face, the peak latency within the
occipital cortex occurred earlier than the peak response
from the STG. This may be taken to support the al-
ternate account noted above that presentation of an
item activates distinct, associated information early in
processing.
It would appear that the latency findings from faces
versus names may be at odds with one another, each
Ryan et al.
1039
D
o
w
n
l
o
a
d
e
d
l
l
/
/
/
/
j
f
/
t
t
i
t
.
:
/
/
f
r
o
m
D
h
o
t
w
t
n
p
o
:
a
/
d
/
e
m
d
i
f
t
r
o
p
m
r
c
h
.
s
p
i
l
d
v
i
e
r
e
r
c
c
t
.
h
m
a
i
r
e
.
d
u
c
o
o
m
c
/
n
j
a
o
r
c
t
i
n
c
/
e
a
-
p
r
d
t
i
2
c
0
l
6
e
-
1
p
0
d
3
f
0
/
1
2
9
0
3
/
6
6
8
/
9
1
2
0
o
3
c
0
n
/
1
2
0
7
0
5
8
9
5
2
5
0
7
0
7
/
5
j
o
p
c
d
n
.
b
y
2
0
g
0
u
8
e
.
s
t
2
o
0
n
0
7
0
5
7
.
S
p
e
d
p
f
e
m
b
y
b
e
g
r
u
2
0
e
2
s
3
t
/
j
/
.
.
f
.
t
o
n
1
8
M
a
y
2
0
2
1
supporting an alternate account of how past experience
alters current perceptions. However, recent work by Bar
et al. (2006) and Bar (2003, 2004) may shed some light
on this issue. In short, Bar and colleagues suggest that,
upon presentation of a visual stimulus,
information
regarding low-level spatial frequencies is rapidly pro-
jected from early visual regions to the orbito-frontal cor-
tex. From this sparse information, the orbito-frontal
cortex activates a small set of representations that could
serve to match the perceptual input, thereby facilitating
eventual recognition. In that case, the early response
seen here within the occipital cortex for the faces may
reflect the projection of low spatial frequency informa-
tion to frontal regions, whereas the subsequent re-
sponse for famous faces within the STG may reflect
the activation of the representation in memory that
matches the perceptual input. In this case, the repre-
sentation of the object (face or name) may include other
information that had been present during the previous
learning episode and serves to distinguish the object
from other items, either novel or familiar.
A similar phenomenon may be occurring for the
names as well (Marinkovic, 2004; Gold & Buckner,
2002), even though there were no detectable differences
in the peak latency between STG and occipital responses
for the famous names. Although there is a clear and
consistent onset of visual information for the presented
faces, the ‘‘onset’’ of auditory information is less dis-
crete. Although the first phonemes were the same
across famous and nonfamous names, the duration of
the first phoneme varied from trial to trial across the
names, thereby potentially reducing our signal-to-noise
ratio for the mean STG response. Although timing
differences exist between the occipital cortex and the
STG for faces (and possibly even for names), the audi-
tory and visual information may, nonetheless, be com-
bined into a representation of the famous person that is
activated upon processing of initial, sparse information.
Thus, the current findings support the account that
suggests that an object is composed of more than what
it imparts upon the senses.
This notion receives converging support from recent
findings on word meaning. For instance, Pulvermuller,
Shytrov, and Ilmoniemi (2005) presented subjects with
action words and pseudowords, whereas neural re-
sponses were recorded with MEG. The action words
referred to those that could either be performed with
the face (e.g., eat) or the leg (e.g., kick). Action words,
relative to the pseudowords, elicited somatotopic activ-
ity in the motor cortex. Such activity was evident within
170–200 msec following the onset of the second syllable
which, due to the design of the stimuli, was the first
moment when the action words could be disambiguated
from the pseudowords. Likewise, reading words that
depict odors not only activates the same neural regions
that are related to language processing as does reading
neutral words but reading odor words also activates
regions within the olfactory cortex (Gonzalez et al.,
2006). Gonzalez et al. (2006) and Pulvermuller et al.
(2005) interpreted their findings as suggesting that
words are processed by distributed cell assemblies or
systems, including those that are involved in processing
associated semantic and sensory information; that is, one
cannot read, or perceive, the word, without automati-
cally and obligatorily accessing its meaning. An alternate
interpretation proposed by James and Gauthier (2003)
suggests that semantic information is stored in what are
considered to be largely perceptual regions. Similar to
that position, findings of increased activation in early
visual association cortices, as revealed through positron
emission tomography, in response to naming auditory
led Tranel et al. (2003) to argue that such
stimuli,
activation may reflect the retrieval of conceptual knowl-
edge that is pertinent to the task at hand. With respect
to the current work, we suggest that the above studies
indicate that our prior experiences may change our
stored representations, leading to an altered perceptual
experience. The ‘‘meaning’’ of the famous face, for
example,
information
but also in the information that has come to be associ-
ated with it over time (e.g., name, occupation). That is,
the identification and semantic analysis of an item may
be achieved through evaluation of the associated infor-
mation that has altered, and is present within, the
perceptual representation.
is found not only in the visual
In this sense, experience provides a sort of perceptual
plasticity by which the processing of an item is ever-
changing. It may be these modifications that are enacted
upon the stored representation that allow one to dis-
tinguish a previously encountered item from a novel
one. That is, perceptual plasticity may induce the requi-
site changes to a maintained representation that are
needed in order to accurately judge whether a pre-
sented item has been previously encountered, whether
it is by modifying the existing representation of the item,
or linking distinct, associated information to that item
representation. In either event, the onset of stronger
responses within additional sensory modalities, other
than the modality in which the external stimulus is
presented, may feed-forward to higher-order brain re-
gions, and culminate in neuronal markers of familiarity/
recognition that typically occur around 250–600 msec, as
revealed through electroencephalogram/ERP and MEG
studies (Marinkovic et al., 2003; Paller et al., 2003).
The notion of perceptual plasticity—that prior learn-
ing affects changes in or to the representation of a target
item—makes contact with William James’ (1890) asser-
tion that perception of an item is a combination of what
is externally presented and what is associated with that
item within the brain/mind. Regardless of how memory
exerts its influence on perception, it is clear that (to
borrow from Heraclitus) just as we cannot step twice
into the same river, we cannot perceive twice the same
form.
1040
Journal of Cognitive Neuroscience
Volume 20, Number 6
D
o
w
n
l
o
a
d
e
d
l
l
/
/
/
/
j
t
t
f
/
i
t
.
:
/
/
f
r
o
m
D
h
o
t
w
t
n
p
o
:
a
/
d
/
e
m
d
i
f
t
r
o
p
m
r
c
h
.
s
p
i
l
d
v
i
e
r
e
r
c
c
t
.
h
m
a
i
r
e
.
d
u
c
o
o
m
c
/
n
j
a
o
r
c
t
i
n
c
/
e
a
-
p
r
d
t
i
2
c
0
l
6
e
-
1
p
0
d
3
f
0
/
1
2
9
0
3
/
6
6
8
/
9
1
2
0
o
3
c
0
n
/
1
2
0
7
0
5
8
9
5
2
5
0
7
0
7
/
5
j
o
p
c
d
n
.
b
y
2
0
g
0
u
8
e
.
s
t
2
o
0
n
0
7
0
5
7
.
S
p
e
d
p
f
e
m
b
y
b
e
g
r
u
2
0
e
2
s
3
t
/
j
f
.
.
.
/
t
o
n
1
8
M
a
y
2
0
2
1
APPENDIX
(1) On average, how many hours of television do you watch per week?
1
0 hours
2
3
4
5
6
7
1–2 hours
3–4 hours
5–6 hours
7–8 hours
9–10 hours
10+ hours
(2) How often do you go to the movies/rent a movie/watch a movie on T V?
1
never
2
3
4
5
6
7
once a year
every few months once a month twice a month once a week
a few times per week
(3) How often do you read celebrity/fashion magazines?
1
never
2
3
4
5
6
7
once a year
every few months once a month twice a month once a week
a few times per week
(4) Approximately how many hours do you spend on the Internet visiting sites with information regarding celebrities or
downloading various T V shows or movies?
1
2
3
4
5
6
7
0 hours
once a year
every few months once a month twice a month once a week
a few times per week
(5) Overall, how would you rate your knowledge of celebrities?
1
2
3
none whatsoever
4
fair
5
6
7
extremely knowledgeable
Acknowledgments
We thank Guy Earle for creating the auditory stimuli. We also
thank Bernhard Ross for comments and assistance with artifact
rejection. This work was supported by grants from the Natural
Sciences and Engineering Research Council of Canada (J. D. R.,
E. T.) and the Canada Chairs Program (J. D. R.).
Reprint requests should be sent to Jennifer D. Ryan, Rotman
Research Institute, Baycrest, 3560 Bathurst St., Toronto, Ontario,
Canada M6A 2E1, or via e-mail: jryan@rotman-baycrest.on.ca.
REFERENCES
Anderson, J. R. (1976). Language, memory and thought.
Hillsdale, NJ: Erlbaum.
Bar, M. (2003). A cortical mechanism for triggering top–down
facilitation in visual object recognition. Journal of Cognitive
Neuroscience, 15, 600–609.
Bar, M. (2004). Visual objects in context. Nature Reviews
Neuroscience, 5, 617–629.
Bar, M., Kassam, K. S., Ghuman, A. S., Boshyan, J., Schmidt,
A. M., Dale, A. M., et al. (2006). Top–down facilitation of
visual recognition. Proceedings of the National Academy of
Sciences, 103, 449–454.
Bentin, S., Allison, T., Puce, A., Perez, E., & McCarthy, G.
(1996). Electrophysiological studies of face perception in
humans. Journal of Cognitive Neuroscience, 8, 551–565.
Bruce, V., Ellis, H. D., Gibling, F., & Young, A. W. (1987).
Parallel processing of the sex and familiarity of faces.
Canadian Journal of Psychology, 41, 510–520.
Bruce, V., & Young, A. W. (1986). Understanding face
recognition. British Journal of Psychology, 77, 305–327.
Donaldson, D. I., & Rugg, M. D. (1998). Recognition memory
for new associations: Electrophysiological evidence for the
role of recollection. Neuropsychologia, 36, 377–395.
Donaldson, D. I., & Rugg, M. D. (1999). Event-related potential
studies of associative recognition and recall:
Electrophysiological evidence for context dependent
retrieval processes. Cognitive Brain Research, 8, 1–16.
Gold, B. T., & Buckner, R. L. (2002). Common prefrontal
regions coactivate with dissociable posterior regions during
controlled semantic and phonological tasks. Neuron, 35,
803–812.
Gonzalez, J., Barros-Loscertales, A., Pulvermuller, F., Meseguer,
V., Sanjuan, A., Belloch, V., et al. (2006). Reading cinnamon
activates olfactory brain regions. Neuroimage, 32, 906–912.
Hanlon, F. M., Miller, G. A., Thoma, R. J., Irwin, J. G., Jones, A.,
Moses, S. N., et al. (2005). Distinct M50 and M100 auditory
gating deficits in schizophrenia. Psychophysiology, 42,
417–427.
Hannula, D. E., Ryan, J. D., Tranel, D., & Cohen, N. J. (2007).
Rapid onset relational memory effects are evident in eye
movement behavior, but not in hippocampal amnesia.
Journal of Cognitive Neuroscience, 19, 1690–1705.
Iidaka, T., Matsumoto, A., Haneda, K., Okada, T., & Sadato, N.
(2006). Hemodynamic and electrophysiological relationship
involved in human face processing: Evidence from a
combined fMRI–ERP study. Brain and Cognition, 60,
176–186.
Itier, R. J., Herdman, A. T., George, N., Cheyne, D., & Taylor, M.
(2006). Inversion and contrast-reversal effects on face
processing assessed by MEG. Brain Research, 1115,
108–120.
Itier, R. J., & Taylor, M. J. (2002). Inversion and contrast
polarity reversal affect both encoding and recognition
processes of unfamiliar faces: A repetition study using ERPs.
Neuroimage, 21, 1518–1532.
James, T. W., & Gauthier, I. (2003). Auditory and action
semantic features activate sensory-specific perceptual brain
regions. Current Biology, 13, 1792–1796.
James, W. (1983). The principles of psychology. Cambridge:
Harvard University Press. (Original work published 1890.)
Ryan et al.
1041
D
o
w
n
l
o
a
d
e
d
l
l
/
/
/
/
j
f
/
t
t
i
t
.
:
/
/
f
r
o
m
D
h
o
t
w
t
n
p
o
:
a
/
d
/
e
m
d
i
f
t
r
o
p
m
r
c
h
.
s
p
i
l
d
v
i
e
r
e
r
c
c
t
.
h
m
a
i
r
e
.
d
u
c
o
o
m
c
/
n
j
a
o
r
c
t
i
n
c
/
e
a
-
p
r
d
t
i
2
c
0
l
6
e
-
1
p
0
d
3
f
0
/
1
2
9
0
3
/
6
6
8
/
9
1
2
0
o
3
c
0
n
/
1
2
0
7
0
5
8
9
5
2
5
0
7
0
7
/
5
j
o
p
c
d
n
.
b
y
2
0
g
0
u
8
e
.
s
t
2
o
0
n
0
7
0
5
7
.
S
p
e
d
p
f
e
m
b
y
b
e
g
r
u
2
0
e
2
s
3
t
/
j
.
/
f
t
.
.
o
n
1
8
M
a
y
2
0
2
1
Jemel, B., Pisani, M., Calabria, M., Crommelinck, M., & Bruyer,
Prince, S. E., Daselaar, S. M., & Cabeza, R. (2005). Neural
R. (2003). Is the N170 for faces cognitively penetrable?
Evidence from repetition priming of Mooney faces of familiar
and unfamiliar persons. Brain Research, Cognitive Brain
Research, 17, 431–446.
Kobayashi, T., & Kuriki, S. (1999). Principal component
elimination method for the improvement of S/N in evoked
neuromagnetic field measurements. IEEE Transactions on
Biomedical Engineering, 46, 951–958.
Kosslyn, S. M., Thompson, W. L., Kim, I. J., & Alpert, N. M.
(1995). Topographical representations of mental images in
primary visual cortex. Nature, 378, 496–498.
Kraemer, D. J. M., Macrae, C. N., Green, A. E., & Kelley, W. M.
(2005). Sound of silence activates auditory cortex. Nature,
434, 158.
Lagerlund, T. D., Sharbrough, F. W., & Busacker, N. E. (1997).
Spatial filtering of multichannel electroencephalographic
recordings through principal component analysis by singular
value decomposition. Journal of Clinical Neurophysiology,
14, 73–82.
Marinkovic, K. (2004). Spatiotemporal dynamics of word
processing in the human cortex. The Neuroscientist, 10,
142–152.
Marinkovic, K., Dhond, R. P., Dale, A. M., Glessner, M., Carr, V.,
& Halgren, E. (2003). Spatiotemporal dynamics of modality-
specific and supramodal word processing. Neuron, 38,
487–497.
McDowell, J. E., Kissler, J. M., Berg, P., Dyckman, K. A., Gao, Y.,
Rockstroh, B., et al. (2005). Electroencephalography/
magnetoencephalography study of activities preceding
prosaccades and antisaccades. NeuroReport, 16, 663–668.
Moses, S. N., Houck, J. M., Martin, T., Hanlon, F. M., Ryan, J. D.,
Thoma, R. J., et al. (2007). Dynamic neural activity recorded
from human amygdala during fear conditioning using
magnetoencephalography. Brain Research Bulletin, 71,
452–460.
Moses, S. N., Martin, T., Houck, J. M., Ilmoniemi, R. J., &
Tesche, C. D. (2005). The C50m response: Conditioned
magnetocerebral activity recorded from the human brain.
Neuroimage, 27, 778–788.
Mouchetant-Rostaing, Y., Giard, M.-H., Bentin, S., Aguera, P.-E.,
& Pernier, J. (2000). Neurophysiological correlates of face
gender processing in humans. European Journal of
Neuroscience, 12, 303–310.
Nyberg, L., Habib, R., McIntosh, A. R., & Tulving, E. (2000).
Reactivation of encoding-related brain activity during
memory retrieval. Proceedings of the National Academy of
Sciences, 97, 1120–1124.
Paller, K. A., Ranganath, C., Gonsalves, B., LaBar, K. S., Parrish,
T. B., Gitelman, D. R., et al. (2003). Neural correlates of
person recognition. Learning and Memory, 10, 253–260.
Pickering, E. C., & Schweinberger, S. R. (2003). N200, N250r,
and N400 event-related brain potentials reveal three loci of
repetition priming for familiar names. Journal of
Experimental Psychology: Learning, Memory, and
Cognition, 6, 1298–1311.
correlates of relational memory: Successful encoding and
retrieval of semantic and perceptual associations. Journal of
Neuroscience, 25, 1203–1210.
Pulvermuller, F., Shytrov, Y., & Ilmoniemi, R. (2005). Brain
signatures of meaning access in action word recognition.
Journal of Cognitive Neuroscience, 17, 884–892.
Ramachandran, V., & Hubbard, E. M. (2001). Synaesthesia—A
window into perception thought and language. Journal of
Consciousness Studies, 8, 3–34.
Roisson, B., Gauthier, I., Tarr, M. J., Despland, P., Bruyer, R.,
Linotte, S., et al. (2000). The N170 occipito-temporal
component is delayed and enhanced to inverted objects: An
electrophysiological account of face-specific processes in the
human brain. NeuroReport, 11, 69–74.
Ryan, J. D., Hannula, D. E., & Cohen, N. J. (2007). The
obligatory effects of memory on eye movements. Memory,
15, 508–525.
Ryan, J. D., Moses, S. N., Ostreicher, M., & Riggs, L. (in
preparation). Accessibility of stored memory representations.
Saykin, A. J., Johnson, S. C., Flashman, L. A., McAllister, T. W.,
Sparling, M., Darcey, T. M., et al. (1999). Functional
differentiation of medial temporal and frontal regions
involved in processing novel and familiar words. Brain, 122,
1963–1971.
Schweinberger, S. R., Pickering, E. C., Burton, A. M., &
Kaufmann, J. M. (2002). Human brain potential correlates of
repetition priming in face and name recognition.
Neuropsychologia, 40, 2057–2073.
Sergent, J., & Villemure, J. G. (1989). Prosopagnosia in a right
hemispherectomized patient. Brain, 12, 975–995.
Simos, P. G., Breier, J. J., Zouridakis, G., & Papanicolaou, A. C.
(1998). Identification of language-specific brain activity using
magnetoencephalography. Journal of Clinical and
Experimental Neuropsychology, 20, 706–722.
Thoma, R. J., Hanlon, F. M., Moses, S. N., Edgar, J. C., Huang, M.,
Weisend, M. P., et al. (2003). Lateralization of auditory sensory
gating and neuropsychological dysfunction in schizophrenia.
American Journal of Psychiatry, 160, 1595–1605.
Thoma, R. J., Hanlon, F. M., Moses, S. N., Ricker, D., Huang, M.,
Edgar, C., et al. (2005). M50 sensory gating predicts negative
symptoms in schizophrenia. Schizophrenia Research, 73,
311–318.
Tranel, D., Damasio, H., Eichhorn, G. R., Grabowski, T., Ponto,
L. L. B., & Hichwa, R. D. (2003). Neural correlates of naming
animals from their characteristic sounds. Neuropsychologia,
41, 847–854.
Wheeler, M. E., Petersen, S. E., & Buckner, R. L. (2000).
Memory’s echo: Vivid remembering reactivates sensory-
specific cortex. Proceedings of the National Academy of
Sciences, 97, 1125–1129.
Wild, H. A., Barrett, S. E., Spence, M. J., O’Toole, A. J., Cheng,
Y. D., & Brooke, J. (2000). Recognition and sex
categorization of adults’ and children’s faces: Examining
performance in the absence of sex-stereotyped cues.
Journal of Experimental Child Psychology, 77, 269–291.
1042
Journal of Cognitive Neuroscience
Volume 20, Number 6
D
o
w
n
l
o
a
d
e
d
l
l
/
/
/
/
j
f
/
t
t
i
t
.
:
/
/
f
r
o
m
D
h
o
t
w
t
n
p
o
:
a
/
d
/
e
m
d
i
f
t
r
o
p
m
r
c
h
.
s
p
i
l
d
v
i
e
r
e
r
c
c
t
.
h
m
a
i
r
e
.
d
u
c
o
o
m
c
/
n
j
a
o
r
c
t
i
n
c
/
e
a
-
p
r
d
t
i
2
c
0
l
6
e
-
1
p
0
d
3
f
0
/
1
2
9
0
3
/
6
6
8
/
9
1
2
0
o
3
c
0
n
/
1
2
0
7
0
5
8
9
5
2
5
0
7
0
7
/
5
j
o
p
c
d
n
.
b
y
2
0
g
0
u
8
e
.
s
t
2
o
0
n
0
7
0
5
7
.
S
p
e
d
p
f
e
m
b
y
b
e
g
r
u
2
0
e
2
s
3
t
/
j
t
.
.
/
f
.
o
n
1
8
M
a
y
2
0
2
1
