Rapid and Accumulated Modulation of - Am MIT spezialisierte KI-Forschung

Rapid and Accumulated Modulation of
Action-Effects on Action

Liyu Cao, Wilfried Kunde, and Barbara Haendel

Abstrakt

■ Auditory feedback to a keypress is used in many devices to
facilitate the motor output. The timing of auditory feedback is
known to have an impact on the motor output, yet it is not known
if a keypress action can be modulated on-line by an auditory feed-
back or how quick an auditory feedback can influence an ongoing
keypress. Außerdem, it is not clear if the prediction of auditory
feedback already changes the early phase of a keypress action in-
dependent of sensory feedback, which would suggest that such
prediction changes the motor plan. In the current study, partici-
pants pressed a touch-sensitive device with auditory feedback in a
self-paced manner. The auditory feedback was given either after a
short (60 ms) or long (160 ms) delay, and the delay was either

predictable or not. Our results showed that the keypress peak
force was modulated by the amount of auditory feedback delay
even when the delay was unpredictable, thus demonstrating an
on-line modulation effect. The latency of the on-line modulation
was suggested to be as low as 70 ms, indicating a very fast sen-
sory to motor mapping circuit in the brain. When the auditory
feedback delay was predictable, a change in the very early phase
of keypress motor output was found, suggesting that the predic-
tion of sensory feedback is crucial to motor control. daher,
even a simple keypress action contains rich motor dynamics,
which depend on expected as well as on-line perceived sensory
Rückmeldung. ■

EINFÜHRUNG

Actions are accompanied by sensory feedback. For exam-
Bitte, we hear our own voices when we speak. The impor-
tance of auditory feedback in regulating motor control
has been recognized since the 1950s using the delayed
auditory feedback paradigm (van Vugt & Tillmann, 2015;
Sasisekaran, 2012; Pfordresher & Dalla Bella, 2011;
Chase, Harvey, Standfast, Rapin, & Sutton, 1959; Kalmus,
Denes, & Fry, 1955; Black, 1950; Lee, 1950). In most natural
situations, the auditory feedback from actions is usually in-
stant (a counterexample is when we speak in a large room;
see Black, 1950). In the delayed auditory feedback para-
digm, the latency of auditory feedback is experimentally
manipulated so that the auditory feedback follows actions
with a small amount of delay. A typical finding is that
delayed auditory feedback has adverse effects on the per-
formance of actions in various systems. Zum Beispiel, Wann
people hear their own voice with a 200-msec delay, Sie
speak louder and slower.

In a manual tapping task, delayed auditory feedback leads
to increased tapping peak force (z.B., Chase et al., 1959). In
some recent demonstrations, participants were asked to
make keypresses, and they pressed more lightly when a
sound followed the press without any delay as compared
to when the sound was delayed or when no sound was pres-
ent (Cao, Steinborn, Kunde, & Haendel, 2020; Neszmélyi &
Horváth, 2018; see also Chase, Rapin, Gilden, Sutton, &

Julius-Maximilians-Universität Würzburg, Deutschland

Guilfoyle, 1961). Although it is clear that finger tapping
behavior can be modulated by auditory feedback, the un-
derlying mechanism is not clear. A motor response starts
with a motor program and can be modified by on-line feed-
back (Shadmehr, Schmied, & Krakauer, 2010; Todorov, 2004;
Wolpert, Miall, & Kawato, 1998). Two questions are relevant
Hier. The first question is how quick the auditory feedback
can modulate an ongoing keypress, or stated as whether an
on-line modulation is possible for a keypress. A keypress is a
very quick movement, which lasts about 300 ms. Daher, Es
may be programmed to be rather ballistic, and an on-line
modulation is not possible. The answer to the question can-
not be derived from existing studies as the feedback delay in
a series of keypresses is constant. When a series of key-
presses is followed by auditory feedback of the same delay,
das ist, the feedback delay is predictable, the prediction of
the auditory feedback may change the motor output
(Shadmehr et al., 2010; Flanagan, Vetter, Johansson, &
Wolpert, 2003). daher, it is not known if a keypress
peak force increase because of a delayed auditory feedback
results from the auditory feedback of the current keypress
or learning from previous trials. Jedoch, if an unpredict-
able auditory feedback delay also leads to a modulation of
keypress peak force (z.B., the force is higher for delay vs. NEIN
delay or for long delay vs. short delay), an on-line modula-
tion process can be assumed to be at work.

The second question is closely related to the first ques-
tion, das ist, whether a motor-prediction-related signal can
be detected in the keypress behavior when the auditory
feedback is constantly delayed. Motor prediction refers to

Zeitschrift für kognitive Neurowissenschaften 32:12, S. 2333–2341
https://doi.org/10.1162/jocn_a_01633

D
Ö
w
N
l
Ö
A
D
e
D

/
J

T
T

F
/

ich
T
.

:
/
/

F
R
Ö
M
D
Ö
H
w
T
T
N
P
Ö
:
A
/
D
/
e
M
D
ich
F
R
T
Ö
P
M
R
C
H
.
S
P
ich
l
D
v
ich
R
e
e
R
C
T
C
.
M
H
A
ich
e
R
D
.
u
C
Ö
Ö
M
C
N
/
J
A
Ö
R
T
C
ich
C
N
e
/
–
A
P
R
D
T
ich
3
C
2
l
1
e
2
–
2
P
3
D
3
F
3
/
2
3
0
2
1
/
4
1
1
2
2
/
0
2
3
Ö
3
C
3
N
_
/
A
1
_
8
0
6
1
2
6
2
3
6
3
6
P
/
D
J
Ö
B
C
j
N
G
_
u
A
e
_
S
0
T
1
Ö
6
N
3
0
3
8
.
S
P
D
e
F
P
e
B
M
j
B
e
G
R
u
2
e
0
S
2
T
3

/
J

Ö
N

0
5

M
A
j

2
0
2
1

the process of predicting the sensory feedback from own
Aktionen (reafferent inputs), and it is deemed as a part of
motor program ( Wolpert & Flanagan, 2001). In the time
course of a keypress, the dependent variable of keypress
peak force used in existing studies occurs too late to be a
motor-prediction-related signal if it receives on-line mod-
ulation. For motor-prediction-related signals, one needs to
focus on the very early phase of motor output in which an
influence of sensory feedback is physiologically impossible
(because of sensory feedback delay). daher, a success-
ful detection of a modulation in the early phase of keypress
force trajectory because of feedback predictability reflects
a change in motor program and the related motor predic-
tion. This may open up a possibility to study motor control
processes within keypressing behavior, a very common
response in cognitive studies, in a very fine-grained fashion.
To answer the above two questions, a keypress experi-
ment was performed with a manipulation of the predictability
of auditory feedback delay. The keypress force was recorded
as the dependent variable. We show that the keypress peak
force is modulated by the auditory feedback delay when the
delay is predictable, und am wichtigsten, also when the
delay is unpredictable (evidence for on-line modulation).
We also show that the early phase of keypress force trajectory
is modulated by the predictability of the auditory feedback
delay (evidence for a change in motor program).

METHODEN

Teilnehmer

Forty-nine healthy participants (28 Frauen; Durchschnittsalter =
28.7; SD = 8.8) volunteered to participate in the study. Von
all the participants, 23 were colleagues and friends, und das
rest were recruited from a local participant pool. Only exter-
nal participants received monetary compensation for partic-
ipation. Forty-one participants were included in the peak
force analysis and 39 in the force trajectory analysis (sehen
Data Exclusion section). Informed consent was obtained
from all participants. The study was approved by the local
ethics committee (Institute of Psychology, Faculty for
Human Sciences, University of Würzburg; Project No. GZ
2018-27) and conducted in accordance with the
Declaration of Helsinki and the General Data Protection
Regulation in Europe.

Task, Stimuli, and Apparatus

Participants sat in front of a laptop and completed a self-
paced keypress task. The keypress, in our case, was made
against a flat surface similarly to making a keypress on a touch
screen without any movement of the key itself (Figure 1A).
The auditory feedback after a keypress was manipulated
(Figure 1B). There were two predictable auditory feedback

D
Ö
w
N
l
Ö
A
D
e
D

/
J

T
T

F
/

ich
T
.

:
/
/

/
J

Ö
N

0
5

M
A
j

2
0
2
1

Figur 1. Experiment details. (A) The key used in the study (links), an example of force trajectory of a keypress (Mitte), and the empirically measured
relationship between the sensor measured raw force value and force value in Newton (Rechts). The three dots in the example force trajectory (Mitte) vertreten
the force start point (d.h., the last point with a force value of 0 before the keypress force escalates; Point 1), the keypress start point (raw force value at 200; Point 2),
and the peak force point (Point 3). Each circle in the right plot shows a measurement point in assessing the mapping between sensor-measured raw force and
force in Newton. A polynomial fitting curve is overlaid. (B) Experimental conditions. Each participant completed the experiment in one of the four orders, mit dem
included number of participants in the final analysis shown on the right. Each box represents a testing block (50 Versuche). NS = No sound; Mix = Mix60 and Mix160.

2334

Zeitschrift für kognitive Neurowissenschaften

Volumen 32, Nummer 12

delay conditions, Fix60 and Fix160. In the Fix60 condition
(100 keypresses), a keypress was always followed by a
60-msec delayed sound. In the Fix160 condition (100 key-
presses), a keypress was always followed by a 160-msec
delayed sound. Fix60 and Fix160 conditions were presented
in separate testing blocks so that the same delay was pre-
sented in a testing block. There were two unpredictable
auditory feedback delay conditions, Mix60 and Mix160. In
the Mix60 condition (100 keypresses), a keypress was always
followed by a 60-msec delayed sound. In the Mix160 condi-
tion (100 keypresses), a keypress was always followed by a
160-msec delayed sound. The Mix60 and Mix160 conditions
were presented in the same testing block (‘Mix’ in Abbildung
1B). To make sure that the feedback delay was unpredict-
able, the feedback delays in this testing block were ran-
domized using the randperm function in MATLAB (Der
MathWorks). The presentation order of predictable and un-
predictable auditory feedback conditions was counterba-
lanced among participants. Zusätzlich, a NoSound
condition (100 keypresses) was included in separate testing
blocks, in which no auditory feedback was provided after a
keypress. The NoSound condition served as a baseline con-
dition in which the general effect of auditory feedback on
keypress can be compared (Neszmélyi & Horváth, 2018).
For each participant, 500 keypresses (or trials) were col-
lected in blocks of 50 keypresses (10 blocks in total), und ein
self-paced break was given between blocks. During each
testing block, participants were asked to fixate the screen
center, where the block number was constantly presented.
Participants were told to make a keypress after about every
2 Sek. Of the 41 participants included in the final analysis
(see Data Exclusion section), the average interkeypress
interval was 2.54 (SD = 0.98) sec in NoSound condition,
2.34 (SD = 0.99) in mix condition, 2.36 (SD = 1.07) In
Fix60 condition, Und 2.52 (SD = 1.03) in Fix160 condition.
No significant difference was found between conditions in
the interkeypress interval with a within-subject one-way
ANOVA (F(3, 120) = 1.47, p = .24, ηp

2 = 0.04).

The auditory feedback was a 1000-Hz tone (50 msec in
Dauer; 5-msec rise/fall ramp), which was presented via
headphones (Vic Firth SIH1) at a comfortable volume level
(same volume for all participants). Participants wore the
headphones throughout the experiment (d.h., einschließlich
the NoSound condition). The force information of a key-
press was collected by a force sensing resistor (FSR;
Modell 402, Interlink Electronics Inc.). The FSR, welches ist
also the key being pressed, has a circular active area with
a diameter of ∼1.3 cm. A 10-kΩ resistor, from which the
voltage information was collected, was connected in series
with the FSR with an input voltage of 5V. With increased
force applied on the active area of the FSR, the FSR would
reduce its resistance, which would result in increased cur-
rent flow in the circuit and increased voltage on the 10-kΩ
resistor. Analog to digital conversion was achieved using a
microcontroller ATMega16 (10-bit precision; Microchip
Technology Inc.). Each digit in the recorded data repre-
sents ∼5 mV (5V/1023). The force was sampled at

500 Hz. The relationship between applied force and
recorded voltage (d.h., the raw FSR force value) is mono-
tonic but not linear (Figure 1A, Rechts). During the testing,
participants put the index finger of their dominant hand
directly on the FSR (softly in contact). The index finger
was always in contact with the FSR, resting very softly on
the sensor before and after each keypress so that no force
was detected by the FSR, even during the break between
blocks. This contact with the FSR was most often below
the FSR force detection threshold (around 0.3 N). Es
brought two advantages with the finger always being in
contact with the device: (A) No auditory feedback from
the keypress itself (d.h., physical vibrations) could be heard
by participants, Und (B) the relative position between the
finger and the FSR did not change throughout the exper-
iment, which made the FSR output comparable among all
keypresses. In conditions with auditory feedback, the tim-
ing of auditory feedback was controlled by the FSR output.
A sound was played after the FSR output reached a thresh-
old value of 200 (∼0.87 N), which was referred to as the
keypress start point. A keypress was made by briskly de-
pressing the index finger. The measured delay between
the keypress start and physical sound wave output was
60.6 ms (SD = 6.0; 100 sound measurements) Und
161.3 ms (SD = 6.8; 100 sound measurements) für die
intended 60- and 160-msec delay, jeweils.

Participants were given a few practice keypresses
(between four and eight keypresses) to get familiarized
with the key (FSR) before data collection. During the prac-
tice, participants were instructed on the correct way to
make a keypress. Participants always had to keep the finger
in contact with the key, briskly depress the key, and then
wait for about 2 sec before the next keypress. Each success-
ful keypress was followed by a letter “k” on the screen im-
mediately after the keypress start point (no visual feedback
was given in the formal testing). At the end of the formal
testing, participants were asked to make three additional
strong keypresses. These three keypresses gave larger
peak FSR output values than keypresses from any testing
condition, indicating that the keypress force collected dur-
ing the testing was in the working range of FSR. Nach dem
Datenerfassung, participants were asked if they realized
anything wrong with respect to the sound. No one men-
tioned the delay of the sound.

Data Exclusion

For each keypress, the associated force trajectory and the
peak force were extracted (see Figure 1A for an example).
In our experimental setup, the data stream of FSR output
was not available for controlling the auditory feedback until
1 sec after the start of each trial. daher, in conditions
with auditory feedback, if a keypress was made immedi-
ately after the preceding keypress (innerhalb 1 Sek), the in-
tended timing manipulation of auditory feedback based
on the FSR output cannot be achieved. These trials were
excluded. Trials that had double peaks in the keypress

Cao, Kunde, and Haendel

2335

D
Ö
w
N
l
Ö
A
D
e
D

/
J

T
T

F
/

ich
T
.

:
/
/

/
J

Ö
N

0
5

M
A
j

2
0
2
1

force trajectory with at least 40 msec of raw force value
lower than 200 between the two peaks were also excluded
as there was a pause during the keypress, and the manip-
ulation of auditory feedback timing may also not be accu-
rate. For each participant, the number of excluded trials
was calculated for each testing block. A testing block
was excluded if more than 20% of trials (d.h., 10 Versuche)
were excluded. A participant was excluded if no blocks
remained after the block exclusion procedure for one or
more auditory feedback conditions (Mix60, Mix160, Fix160,
Fix60). In Summe, eight participants were excluded after this
step, das ist, 41 participants were included in the peak
force analysis. Among the remaining participants, 13
blocks (3.2% of all blocks) were excluded from seven
Teilnehmer. On average, 96.3 (SD = 12.0) trials were left
in the NoSound condition, 93.0 (SD = 18.1) trials in Mix60,
91.8 (SD = 16.6) trials in Mix160, 93.8 (SD = 15.4) trials in
Fix60, Und 97.3 (SD = 3.3) trials in Fix160. No significant
difference in the number of trials was found between con-
ditions (one-way within-subject ANOVA: F(4, 160) = 1.93,
p = .14, ηp

2 = 0.05).

In the force trajectory analysis, the force trajectory of
each trial was aligned to the force start point and averaged
across trials. For each keypress included in the peak force
Analyse, the force start point was defined as the last time
point with a force value of 0 within the 400-msec time win-
dow ending at the peak force point. If no points with a force
value of 0 were found, the trial was excluded. Two more
participants were excluded from further analysis as no more
als 15 trials were left in at least one condition. The average
number of trials included in the force trajectory analysis was
86.5 (SD = 22.9) in the NoSound condition, 88.1 (SD = 17.7)
in Mix60, 84.8 (SD = 17.9) in Mix160, 85.7 (SD = 20.6) In
Fix60, Und 81.8 (SD = 23.8) in Fix160. There was no signifi-
cant difference in the number of trials between conditions
in the force trajectory analysis (one-way within-subject
ANOVA: F(4, 152) = 0.83, p = .47, ηp

2 = 0.02).

Data Analysis

All the results were reported with force values converted to
Newton in the main text. Jedoch, qualitatively similar
results were obtained from the data analysis with raw FSR
force values. Because the keypress peak force was not nor-
mally distributed, robust statistical measurements based
on randomization were used, and individual data were
always presented for clarity. The data analysis was per-
formed with MATLAB (The MathWorks Inc.). The p value
von .05 was taken as the statistical significance cutoff, Und
corrections for multiple comparisons were made and
mentioned where relevant.

To answer Question 1, the critical test was to compare
the keypress peak force between Mix60 and Mix160. Wenn ein
difference in the auditory feedback delay can lead to a
difference in the peak force when the delay was not pre-
dictable, the difference in the peak force could only be be-
cause of an on-line modulation. We performed a two-level

nonparametric test. Erste, a randomization test was made to
test whether Mix160 had a higher peak force than Mix60
for each participant so that each participant had a p value
(referred to as “prand” later). One-tailed test was used here
because it has been shown that a long auditory feedback
delay led to a higher keypress force than a short keypress
delay (Neszmélyi & Horváth, 2018; Ruhm & Cooper,
1962). The rationale of the randomization test is that, Wenn
there are no differences between two conditions, values from
the two conditions should be exchangeable. daher, ein
individual prand value was obtained in the following four
Schritte: 1. Compare Mix160 and Mix60 using an unpaired
t test, and keep the t value as the original t. 2. Take all
the force values from Mix160 and Mix60, and randomize
the order of those force values before assigning them back
to the two conditions. Compare the two conditions after
randomization, and get a t value as a randomized t. 3.
Repeat Step 2 für 1,000 times so that there are 1,000
randomized t values. 4. Check the proportion of random-
ized t values that are bigger than the original t, welches ist
then the individual prand value. We report all prand values
as prand < 0.001 when prand value is 0 after 1,000 random- izations. Note that, in this case, a small prand value indicates that Mix160 is higher than Mix60, and a large prand value indicates the opposite. Second, the group level prand value for Mix160 having higher peak force than Mix60 was obtained by comparing the individual prand values to 0.5 (chance level) with the randomization test. The procedure was similar to the procedure of getting individual prand values except that the randomization only took place within participants, that is, the condition relabeling was performed for each participant separately so that each participant always had the same pair of data points. The same random- ization test was also used to make comparisons between, for example, Fix160 and Fix60 (see the Results section). To answer Question 2, the average keypress force trajec- tory was analyzed. This analysis allows us to examine the early phase of motor output that is free from the influence of sensory feedback. The force trajectory of each keypress was selected from 200 msec before the force start point to 800 msec after. Within-subject two-tailed t tests were made on the average force trajectory at each time point (500 time points in total) of the force trajectory between Mix60 and Mix160, between Fix60 and Fix160, between the average of Mix60 and Mix160 (“Mix” in Figure 3A legend) and the average of Fix60 and Fix160 (“Fix” in Figure 3A legend), and between the NoSound condition and the average of the four conditions with auditory feedback (“WithSound” in Figure 3A legend). Multiple comparisons over time points were corrected for each between-condition com- parison using a cluster based permutation method with a cutoff p value at .05 (Maris & Oostenveld, 2007). Data and Code Availability The original data and MATLAB analysis code are freely avail- able on-line at: doi.org/10.6084/m9.figshare.12146271.v1. 2336 Journal of Cognitive Neuroscience Volume 32, Number 12 D o w n l o a d e d l l / / / / j f / t t i t . : / / f r o m D o h w t t n p o : a / d / e m d i f r t o p m r c h . s p i l d v i r e e r c t c . m h a i e r d . u c o o m c n / j a o r t c i c n e / - a p r d t i 3 c 2 l 1 e 2 - 2 p 3 d 3 f 3 / 2 3 0 2 1 / 4 1 1 2 2 / 0 2 3 o 3 c 3 n _ / a 1 _ 8 0 6 1 2 6 2 3 6 3 6 p / d j o b c y n g _ u a e _ s 0 t 1 o 6 n 3 0 3 8 . S p d e f p e b m y b e g r u 2 e 0 s 2 t 3 / j t / . f o n 0 5 M a y 2 0 2 1 RESULTS The Classic Delayed Auditory Feedback Effect The manipulation of the auditory feedback from keypress had a significant influence on the keypress (Figure 2, left). Consistent with previous studies on the effect of delayed auditory feedback, all the four conditions in which an auditory feedback followed the keypress had significantly weaker peak force than the condition in which no auditory feedback was provided (all the four between-condition comparisons had prand < 0.001 with one-tailed randomiza- tion test; NoSound: mean = 5.56 N, SD = 6.21 N; Mix60: mean = 2.07 N, SD = 3.05 N; Mix160: mean = 2.19 N, SD = 3.34 N; Fix60: mean = 1.49 N, SD = 1.17 N; Fix160: mean = 1.79 N, SD = 1.05 N). A long fixed delay (Fix160) led to higher keypress peak force than a short fixed delay (Fix60; prand < 0.001, one-tailed randomization test; Figure 2, middle). On-Line Modulation Revealed by Peak Force Analysis To test the effect of an on-line modulation of auditory feed- back on the keypress action, a comparison of peak force was made between the 60-msec delayed sound and the 160-msec delayed sound when both sounds were pre- sented in a random order (i.e., unpredictable). Indeed, a 160-msec delayed sound still led to a stronger keypress force than a 60-msec delayed sound in this case (Mix160 >
Mix60, prand < 0.001, one-tailed randomization test; Figure 2, right), which showed that auditory feedback could influence the triggering action on the fly. Excluding the two participants showing a large difference between Mix160 and Mix60 (Figure 2, right) had no influence on the statistical results. Similar results were obtained with the data analysis on raw force values. However, the on-line modulation from auditory feedback can only occur if the auditory feedback arrived at a time point earlier than the peak force. Therefore, a sanity check was made. The mean duration between the keypress start and the peak force point (Figure 1A) was 95.53 msec (Mix60) and 108.29 msec (Mix160; 176.45 msec in NoSound, 83.21 msec in Fix60, and 121.25 msec in Fix160). The 95.53-msec delay of peak force in the Mix60 condition may seem a bit short to be in- fluenced by a 60-msec delayed sound (35.53-msec reaction latency from sound onset to motor response), but it was impossible for a 160-msec delayed sound to have a direct influence on the peak force at 108.29 msec in the Mix160 condition. This peak force delay was an average of all the 41 participants included in the data analysis, and not every- one showed an on-line modulation effect. If the participants showing a strong on-line adjustment effect were considered ( prand value smaller than 0.05; n = 7), the average peak force delay was 133.02 msec in the Mix60 and 183.01 msec in the Mix160. If participants were grouped based on the prand value, half of the participants with small prand values ( prand < 0.3; n = 20) had an average peak force delay of 112.84 msec in the Mix60 and 133.28 msec in the Mix160; the other half with large prand values ( prand > 0.3; n = 21)
had an average peak force delay of 79.03 msec in the Mix60
Und 84.49 msec in the Mix160. daher, there is enough
time for a 60-msec delayed sound to exert its on-line mod-
ulation effect on motor output. The individual difference in
the on-line modulation effect strongly depends on the
idiosyncratic keypress behavior.

Accumulated Modulation Revealed by Force
Trajectory Analysis

The modulation effect of delayed auditory feedback on
peak force was stronger when the delay was predictable
(Fix160-Fix60; mean = 0.30 N; SD = 0.46 N) than when
the delay was not predictable (Mix160-Mix60; mean =
0.13 N; SD = 0.38 N, prand = 0.04, two-tailed randomization
test). This suggests that there are other sources of modula-
tion in the Fix conditions besides on-line modulation. Wir
tested the idea that the extra modulation effect might even
be detectable in the early phase of the keypress force trajec-
tory beyond the influence of any sensory feedback. To do
Das, single trial keypress force trajectories were aligned

D
Ö
w
N
l
Ö
A
D
e
D

/
J

F
/

T
T

ich
T
.

:
/
/

/
J

Ö
N

0
5

M
A
j

2
0
2
1

Figur 2. Peak force results. Mean peak force in each condition (links), and individual peak force comparison between Fix60 and Fix160 (Mitte) Und
between Mix60 and Mix160 (Rechts). Each line represents the data of an individual participant (note that the mean force value from the contrasted
conditions was subtracted for clear visualization, d.h., the demeaned force as shown on the y axis). Black lines show a difference in the same direction
as the group average, and red lines show the opposite. Vertical lines indicate ± 1 standard error. NoSound: keypress without auditory feedback; Mix:
160- and 60-msec delayed auditory feedback were mixed; Fix60 and Fix160: nur 60- or 160-msec delayed sound was presented in one testing block.

Cao, Kunde, and Haendel

2337

D
Ö
w
N
l
Ö
A
D
e
D

/
J

T
T

F
/

ich
T
.

:
/
/

/
J

Figur 3. The results of force trajectory analysis. (A) Auf der Linken, the average keypress force trajectory from each condition between −200 and 800 ms,
with time 0 being the force start point. The horizontal black bars show the time windows of a significant difference in the comparison indicated on the
left after applying the cluster correction for multiple comparisons. Numbers above the horizontal bar indicate the time window covered by the bar.
Vor allem, sound predictability led to a very early modulation of keypress force trajectory (Mix vs Fix, Fix60 vs. Fix160). The empty triangle shows the
average onset time of the 60-msec delayed sound calculated from the force start point, and the filled triangle shows the onset time of the 160-msec
delayed sound. Auf der rechten Seite, a clearer plot of the first 100 msec from the force start point. (B) A difference plot is shown for better visualization of the
comparisons made in (A). Zum Beispiel, the blue line shows the result of subtracting the average force trajectory in Mix60 from Mix160. (C–F) Individuell
results from four significant time windows. Each line represents an individual participant and shows the average force value in the selected time window
(the mean force value from the contrasted conditions was subtracted for clear visualization, d.h., the demeaned force as shown on the y axis). Black lines
show a difference in the same direction as the group average, and red lines show the opposite. NoSound: keypress without auditory feedback;
WithSound: the average of the keypress with auditory feedback conditions; Mix60 and Mix160: 60- and 160-msec delayed sounds were presented in
random; Fix60 and Fix160: nur 60- or 160-msec delayed sound was presented in one testing block; Mix: the average of Mix160 and Mix60; Fix: Die
average of Fix160 and Fix60.

Ö
N

0
5

M
A
j

2
0
2
1

2338

Zeitschrift für kognitive Neurowissenschaften

Volumen 32, Nummer 12

to the force start point (Figure 1A, Mitte), averaged like
an event-related-potential study in electroencephalo-
Gramm, and compared between conditions (Abbildung 3A,
B). Interessant, significant differences in the average
force trajectory were found between the Fix60 and the
Fix160 conditions in two clusters. In the first cluster
(starting at 20 msec after the force start point), Die
Fix60 condition had a higher force than the Fix160 con-
dition (see Figure 3B for a difference plot). Jedoch, In-
dividual results suggested that this pattern was not stable
as more than 45% of participants showed the effect in the
other direction (Figure 3C). Außerdem, there was no
such cluster when the data analysis was performed with
the raw force values. The second cluster (starting at
72 ms) showed a reliable pattern that the Fix160 con-
dition had a higher force than the Fix60 condition (38 von 39
participants showed results in this direction; Figure 3D).
Because the starting point of a cluster cannot be taken
as the point where statistical difference emerges (sehen
Sassenhagen & Draschkow, 2019), the Fix60 and Fix160
conditions were compared with a within-subject t test at
each time point in the entire time window. Das war
followed by a false discovery rate adjustment. The first sig-
nificant point from this analysis was at 80 ms, which was
much earlier than the delivery of the 60-msec delayed au-
ditory feedback. The average delivery time of the 60-msec
delayed sound calculated from the force start point was at
um 117 ms (the average sound delivery time was
marked with triangles in Figure 3A, B). daher, the dif-
ference in force trajectory between the Fix60 and Fix160
conditions started much before the sound delivery and
cannot be explained by auditory feedback. The early differ-
ence can only be because of different motor commands.
The same comparison between the Mix60 and Mix160
conditions led to a significant cluster in the time window
zwischen 182 Und 520 ms. The earliest difference point
as identified through a whole time window comparison
followed by a false discovery rate adjustment was at
192 ms, das ist, after the delivery of the 60-msec delayed
Klang. This difference can be interpreted as a feedback
modulation, with a short delayed sound leading to a
weaker force as compared to a long delayed sound. More
strikingly, when the average force trajectory of the Mix con-
ditions (Mix60, Mix160) was compared to the average force
trajectory of the Fix conditions (Fix60, Fix160), a significant
cluster was found between 4 Und 114 ms, das ist, im
very early stage of the keypress action. As can be seen on
the right of Figure 3A, force trajectories from the condi-
tions with auditory feedback formed two bundles in the
early phase. The first bundle, with a shallow slope, War
formed by the keypresses with predictable feedback timing
(Fix60, Fix160). The second bundle, with a steep slope, War
formed by the keypresses with unpredictable feedback
zeitliche Koordinierung (Mix60, Mix160). Similar results were obtained
with the data analysis on raw force values. Results from
the force speed (slope of force value over time) und Kraft
acceleration (slope of force speed over time) analysis also

showed a similar pattern, that the Fix conditions had lower
speed and acceleration than the Mix conditions in the very
beginning from the keypress force start (results not
shown). When there was no auditory feedback (NoSound
condition), the keypress force was higher than when the
auditory feedback was provided, across the entire analysis
time window.

DISKUSSION

The human motor system generates actions, welche, innerhalb
given task constraints, are optimized with respect to
certain variables such as effort, comfort, or variability
(Todorov, 2004). When interacting with force sensitive
devices, where only the exerted force matters, es scheint
natural that the motor system aims to reduce unnecessary
forces and, daher, minimize metabolic effort. It has been
shown that the latency and the availability of auditory feed-
back plays an important role in regulating the motor output
during a keypress (Cao et al., 2020; Neszmélyi & Horváth,
2018; van Vugt & Tillmann, 2015; Pfordresher & Dalla Bella,
2011; Chase et al., 1959, 1961; Kalmus et al., 1955). Im
current study, we showed that auditory feedback can exert
an on-line modulation on keypress behavior before the
keypress is complete. We also showed that predictable
delays in auditory feedback can lead to a change in motor
Programm.

On-line control suggests that force output is increased
until extra sensory feedback, which signals that enough
force has been exerted, comes in. This process was clearly
confirmed by demonstrating that the keypress peak force
was reduced with early (60 ms) compared to late
(160 ms) auditory feedback when both were equally un-
predictable. In motor control studies, there is an ongoing
debate on how quick sensory feedback can influence the
motor output. Zum Beispiel, human-goal-directed move-
gen (z.B., pointing to an object with the finger: Elliott,
Helsen, & Chua, 2001; Woodworth, 1899) probably require
a latency of at least 100 msec before a visual signal can in-
fluence the motor output (Elliott et al., 2001). In the cur-
rent study, the average latency from the onset of auditory
feedback to the time point of keypress peak force, where a
significant modulation from auditory feedback was found,
War 73 ms (133–60) in the Mix60 condition for the seven
participants showing a strong on-line modulation effect.
This suggests that the latency for an on-line modulation
of motor output from sensory feedback may be overesti-
gepaart, at least in the auditory domain. Interessant, unser
results are consistent with reports of fast orienting re-
sponse to visual stimulus. Neck muscles in monkeys
(Corneil & Munoz, 2014; Corneil, Olivier, & Munoz,
2004) and arm muscles in humans (Pruszynski et al.,
2010) show a very short latency response to visual stimuli
(55–95 msec in monkeys). A very recent study in monkeys
also showed that at least for the initiation of smooth pursuit
eye movements, a latency as short as 50 msec was enough

Cao, Kunde, and Haendel

2339

D
Ö
w
N
l
Ö
A
D
e
D

/
J

F
/

T
T

ich
T
.

:
/
/

/
J

Ö
N

0
5

M
A
j

2
0
2
1

for a visual signal to show an impact on the oculomotor
output (Buonocore, Skinner, & Hafed, 2019). The current
Studie, to the best of our knowledge, is the first to demon-
strate that a self-generated auditory feedback can have an
almost comparably fast influence on motor output (finger
Bewegung).

The motor program change was observed in the early
phase of keypress force trajectory. A long predictable audi-
tory feedback delay (160 ms) led to a steeper increase of
keypress force than a short predictable auditory feedback
delay (60 ms) in a time window far earlier than the arrival
of auditory feedback. daher, the early force difference
cannot be explained by sensory feedback and can only be
explained by a change in the motor program. The motor
program change is related to the predictability of the audi-
tory feedback delay. Supporting this, the early force differ-
ence was not observed when the auditory feedback delay
was not predictable (Mix60 vs. Mix160). There may be at
least two explanations for the motor program change.
One explanation is that the effect from on-line modulation
accumulates, and the motor program change is a passive
adaptation to the auditory feedback delay. According to
this explanation, the order of the steepness of early force
trajectory should be (from high to low): Fix160, Mix160,
Mix60, Fix60. This is because a long feedback delay leads
to a higher force than a short feedback delay, and a higher
force should be associated with a steeper slope (see the
force trajectory of NoSound condition; Abbildung 3A). Fix160
had a 160-msec feedback delay in all trials and should have
the steepest slope. Mix160 also received a 160-msec feed-
back delay in all trials but was interrupted by some 60-msec
feedback trials in between, so it should have the second
steepest slope. Following the same rationale, it may not
be difficult to figure out that Mix60 and Fix60 should have
the third and the fourth steepest slope, jeweils.
Jedoch, the results clearly showed that the force trajecto-
ries of both the Mix160 and Mix60 conditions were above
the Fix160 and Fix60 conditions (Abbildung 3A, Rechts).
Außerdem, the average of the Mix160 and Mix60 condi-
tions had a statistically significant steeper slope than the
average of the Fix160 and Fix60 conditions in the time
window between 4 Und 114 msec from the keypress force
start. daher, an explanation of passive motor adapta-
tion is highly unlikely for the early keypress force slope
ändern (note that we do not exclude the possibility that
a passive motor adaptation may exist). The other explana-
tion is that the motor program change is a result of an
active motor adaptation, which involves the prediction of
the auditory feedback. This explanation is consistent with
neurophysiological findings that the brain can learn the
auditory feedback delay (Elijah, Le Pelley, & Whitford,
2018; Cao, Veniero, Thut, & Brutto, 2017) and that the
prediction of sensory feedback is crucial to motor control
(Kunde, Koch, & Hoffmann, 2004; Flanagan et al., 2003;
Wolpert & Flanagan, 2001). daher, the change in the
early phase of keypress force trajectory may well be a sign
of an updated motor prediction.

The approach of averaging force trajectories over trials is
not without caveats. Zum Beispiel, the difference between
the Mix60 and Mix160 conditions in the force trajectory
only showed up from 182 ms, which was after the peak
of the average force trajectory. It was already shown that
the peak force was different between the two conditions
from previous analyses. Daher, trial averaging smears the
timing information of between-condition differences
because of an intertrial variation in the force trajectory.
Jedoch, the early component of the force trajectory can
be considered as a realization of the motor command
behind, and trial-averaging should work to reveal the signal
in a way similar to the ERP technique in human neuroim-
aging studies. There are surely other approaches to the
analysis of force profiles (z.B., looking at the skewness
and kurtosis of each keypress, see Ulrich, Rinkenauer, &
Müller, 1998). The trial-averaging approach, as we dis-
cussed, is theoretically sound for the purpose of the cur-
rent study. We conclude that the force profile difference
between the Fix60 and Fix160 conditions, between the
average of the Mix conditions and the average of the Fix
Bedingungen, in the very early time window clearly indicates
an open-loop component of motor control, whereas the
force profile difference between the Mix60 and Mix160
conditions should be interpreted as a sign of a closed-loop
Kontrolle (vgl. Wing, 1977).

Gesamt, our findings suggest that the prediction of the
auditory feedback changes the motor program, which indi-
cates that the prediction of sensory feedback is crucial to
motor control. Gleichzeitig, unpredictable auditory
feedback can lead to an on-line modulation of motor out-
put at a very low latency (∼70 msec).

Danksagungen

We thank Normann Mangold for technical support, Lisa von
Boros for assistance with data collection, Michael Steinborn for
helpful comments on an earlier draft, and Victoria Nicholls
for English editing. L. C. and B. H. were supported by the
European Research Council (gewähren 677819, awarded to B. H.).

Reprint requests should be sent to Liyu Cao, Department of
Psychologie (III), Julius-Maximilians-Universität Würzburg,
Röntgenring 11, 97070, Würzburg, Deutschland, oder per E-Mail:
liyu.cao@uni-wuerzburg.de.

VERWEISE

Black, J. W. (1950). The effect of room characteristics upon vocal
intensity and rate. Journal of the Acoustical Society of America,
22, 174–176. DOI: https://doi.org/10.1121/1.1906585

Buonocore, A., Skinner, J., & Hafed, Z. M. (2019). Eye position
error influence over “open-loop” smooth pursuit initiation.
Zeitschrift für Neurowissenschaften, 39, 2709–2721. DOI: https://doi.
org/10.1523/JNEUROSCI.2178-18.2019, PMID: 30709895,
PMCID: PMC6445996

Cao, L., Steinborn, M., Kunde, W., & Haendel, B. (2020). Action
force modulates action binding: Evidence for a multisensory
information integration explanation. Experimental Brain
Forschung, 238, 2019–2029. DOI: https://doi.org/10.1007/
s00221-020-05861-4, PMID: 32617882, PMCID: PMC7438375

2340

Zeitschrift für kognitive Neurowissenschaften

Volumen 32, Nummer 12

D
Ö
w
N
l
Ö
A
D
e
D

/
J

F
/

T
T

ich
T
.

:
/
/

/
J

Ö
N

0
5

M
A
j

2
0
2
1

Cao, L., Veniero, D., Thut, G., & Brutto, J. (2017). Role of the
cerebellum in adaptation to delayed action effects. Current
Biology, 27, 2442–2451. DOI: https://doi.org/10.1016/j.
cub.2017.06.074, PMID: 28781049, PMCID: PMC5571438
Chase, R. A., Harvey, S., Standfast, S., Rapin, ICH., & Sutton, S. (1959).
Comparison of the effects of delayed auditory feedback on
speech and key tapping. Wissenschaft, 129, 903–904. DOI: https://
doi.org/10.1126/science.129.3353.903, PMID: 13635035
Chase, R. A., Rapin, ICH., Gilden, L., Sutton, S., & Guilfoyle, G.

(1961). II Sensory feedback influences on keytapping motor
tasks. Quarterly Journal of Experimental Psychology, 13,
153–167. DOI: https://doi.org/10.1080/17470216108416488
Corneil, B. D., & Munoz, D. P. (2014). Overt responses during
covert orienting. Neuron, 82, 1230–1243. DOI: https://doi.
org/10.1016/j.neuron.2014.05.040, PMID: 24945769

Corneil, B. D., Olivier, E., & Munoz, D. P. (2004). Visual responses
on neck muscles reveal selective gating that prevents express
saccades. Neuron, 42, 831–841. DOI: https://doi.org/10.1016/
S0896-6273(04)00267-3, PMID: 15182721

Elijah, R. B., Le Pelley, M. E., & Whitford, T. J. (2018). Act now, play
später: Temporal expectations regarding the onset of self-initiated
sensations can be modified with behavioral training. Zeitschrift für
Cognitive Neuroscience, 30, 1145–1156. DOI: https://doi.org/
10.1162/jocn_a_01269, PMID: 29668396

Elliott, D., Helsen, W. F., & Chua, R. (2001). A century later:

Woodworth’s (1899) two-component model of goal-directed
aiming. Psychological Bulletin, 127, 342–357. DOI: https://
doi.org/10.1037/0033-2909.127.3.342, PMID: 11393300
Flanagan, J. R., Vetter, P., Johansson, R. S., & Wolpert, D. M.

(2003). Prediction precedes control in motor learning. Current
Biology, 13, 146–150. DOI: https://doi.org/10.1016/S0960-9822
(03)00007-1, PMID: 12546789

Kalmus, H., Denes, P., & Fry, D. B. (1955). Effect of delayed acoustic
feed-back on some non-vocal activities. Natur, 175, 1078.
DOI: https://doi.org/10.1038/1751078a0, PMID: 14394110

Kunde, W., Koch, ICH., & Hoffmann, J. (2004). Anticipated action
effects affect the selection, initiation, and execution of actions.
Quarterly Journal of Experimental Psychology, Section A:
Human Experimental Psychology, 57, 87–106. DOI: https://
doi.org/10.1080/02724980343000143, PMID: 14681005

Lee, B. S. (1950). Some effects of side-tone delay. Journal of the
Acoustical Society of America, 22, 639–640. DOI: https://doi.
org/10.1121/1.1906665

Maris, E., & Oostenveld, R. (2007). Nonparametric statistical testing
of EEG- and MEG-data. Journal of Neuroscience Methods, 164,
177–190. DOI: https://doi.org/10.1016/j.jneumeth.2007.03.024,
PMID: 17517438

Neszmélyi, B., & Horváth, J. (2018). Temporal constraints in
the use of auditory action effects for motor optimization.
Journal of Experimental Psychology: Human Perception
und Leistung, 44, 1815–1829. DOI: https://doi.org/
10.1037/xhp0000571, PMID: 30091635

Pfordresher, P. Q., & Dalla Bella, S. (2011). Delayed auditory

feedback and movement. Journal of Experimental Psychology:

Human Perception and Performance, 37, 566–579. DOI:
https://doi.org/10.1037/a0021487, PMID: 21463087

Pruszynski, J. A., King, G. L., Boisse, L., Scott, S. H., Flanagan, J. R.,
& Munoz, D. P. (2010). Stimulus-locked responses on human
arm muscles reveal a rapid neural pathway linking visual input
to arm motor output. Europäisches Journal für Neurowissenschaften, 32,
1049–1057. DOI: https://doi.org/10.1111/j.1460-9568.2010.07380.x,
PMID: 20726884

Ruhm, H. B., & Cooper, W. A., Jr. (1962). Low sensation level
effects of pure-tone delayed auditory feedback. Zeitschrift für
Speech and Hearing Research, 5, 185–193. DOI: https://doi.
org/10.1044/jshr.0502.185, PMID: 14495207

Sasisekaran, J. (2012). Effects of delayed auditory feedback on
speech kinematics in fluent speakers. Perceptual and Motor
Skills, 115, 845–864. DOI: https://doi.org/10.2466/15.22.
PMS.115.6.845-864, PMID: 23409597, PMCID: PMC3718456

Sassenhagen, J., & Draschkow, D. (2019). Cluster-based
permutation tests of MEG/EEG data do not establish
significance of effect latency or location. Psychophysiology,
56, e13335. DOI: https://doi.org/10.1111/psyp.13335,
PMID: 30657176

Shadmehr, R., Schmied, M. A., & Krakauer, J. W. (2010). Error
correction, sensory prediction, and adaptation in motor
Kontrolle. Jahresrückblick auf die Neurowissenschaften, 33, 89–108. DOI:
https://doi.org/10.1146/annurev-neuro-060909-153135,
PMID: 20367317

Todorov, E. (2004). Optimality principles in sensorimotor control.

Naturneurowissenschaften, 7, 907–915. DOI: https://doi.org/
10.1038/nn1309, PMID: 15332089, PMCID: PMC1488877
Ulrich, R., Rinkenauer, G., & Müller, J. (1998). Effects of stimulus
duration and intensity on simple reaction time and response
force. Journal of Experimental Psychology: Human
Perception and Performance, 24, 915–928. DOI: https://doi.
org/10.1037/0096-1523.24.3.915, PMID: 9627425

van Vugt, F. T., & Tillmann, B. (2015). Auditory feedback in

error-based learning of motor regularity. Gehirnforschung, 1606,
54–67. DOI: https://doi.org/10.1016/j.brainres.2015.02.026,
PMID: 25721795

Wing, A. M. (1977). Perturbations of auditory feedback delay
and the timing of movement. Journal of Experimental
Psychologie: Human Perception and Performance, 3,
175–186. DOI: https://doi.org/10.1037/0096-1523.3.2.175,
PMID: 864391

Wolpert, D. M., & Flanagan, J. R. (2001). Motorvorhersage.
Aktuelle Biologie, 11, R729–R732. DOI: https://doi.org/
10.1016/S0960-9822(01)00432-8, PMID: 11566114

Wolpert, D. M., Miall, R. C., & Kawato, M. (1998). Internal models
in the cerebellum. Trends in den Kognitionswissenschaften, 2, 338–347.
DOI: https://doi.org/10.1016/S1364-6613(98)01221-2, PMID:
21227230

Woodworth, R. S. (1899). Accuracy of voluntary movement.
Psychological Review: Monograph Supplements, 3, 1–119.
DOI: https://doi.org/10.1097/00005053-189912000-00005,
https://doi.org/10.1037/h0092992

Cao, Kunde, and Haendel

2341

D
Ö
w
N
l
Ö
A
D
e
D

/
J

F
/

T
T

ich
T
.

:
/
/

/
J

Ö
N

0
5

M
A
j

2
0
2
1 Rapid and Accumulated Modulation of image

PDF Herunterladen