REPORT
Coordination and Consonance Between
Interacting, Improvising Musicians
Matthew Setzler
1 and Robert Goldstone1,2
1Program in Cognitive Science, Indiana University
2Department of Psychological and Brain Sciences, Indiana University
Keine offenen Zugänge
Tagebuch
Schlüsselwörter: joint action, distributed cognition, improvisation, time series modeling, Musik
ABSTRAKT
Joint action (JA) is ubiquitous in our cognitive lives. From basketball teams to teams of
surgeons, humans often coordinate with one another to achieve some common goal.
Idealized laboratory studies of group behavior have begun to elucidate basic JA mechanisms,
but little is understood about how these mechanisms scale up in more sophisticated and
open-ended JA that occurs in the wild. We address this gap by examining coordination in a
paragon domain for creative joint expression: improvising jazz musicians. Coordination in
jazz music subserves an aesthetic goal: the generation of a collective musical expression
comprising coherent, highly nuanced musical structure (z.B., rhythm, harmony). In our study,
dyads of professional jazz pianists improvised in a “coupled,” mutually adaptive condition,
and an “overdubbed” condition that precluded mutual adaptation, as occurs in common
studio recording practices. Using a model of musical tonality, we quantify the flow of
rhythmic and harmonic information between musicians as a function of interaction
condition. Our analyses show that mutually adapting dyads achieve greater temporal
alignment and produce more consonant harmonies. These musical signatures of coordination
were preferred by independent improvisers and naive listeners, who gave higher quality
ratings to coupled interactions despite being blind to condition. We present these results and
discuss their implications for music technology and JA research more generally.
EINFÜHRUNG
High-level cognition is often achieved by groups of interacting individuals (Knoblich et al.,
2011; Sebanz et al., 2006). Group behavior in joint action (JA) settings is less dependent
on isolated individual efforts and more on the ability to coordinate (Goldstone & Gureckis,
2009; Hasson et al., 2012). Insight into the mechanisms underlying successful coordination
has important implications for how we understand interpersonal interaction, optimize team
Leistung, and engineer humanlike artificial intelligence systems (Cooke & Hilton, 2015;
Guimera et al., 2005; Rebsamen et al., 2010; D. C. Richardson et al., 2007). This study exam-
ines coordination in collaboratively improvising jazz musicians. Coordination in jazz music
subserves an aesthetic goal: the generation of a collective musical expression, and the expertise
of professional jazz musicians lies largely in their ability to coordinate and adapt spontaneously
in real-time performance. Professional jazz ensembles thus offer a remarkably sophisticated
paragon domain to study the basic properties and limits of our capacity to coordinate with one
another.
Zitat: Setzler, M., & Goldstone, R.
(2020). Coordination and Consonance
Between Interacting, Improvising
Musicians. Open Mind: Discoveries
in Cognitive Science. 4, 88—101.
https://doi.org/10.1162/opmi_a_00036
DOI:
https://doi.org/10.1162/opmi_a_00036
Supplemental Materials:
https://doi.org/10.1162/opmi_a_00036
Erhalten: 26 Februar 2020
Akzeptiert: 26 August 2020
Konkurrierende Interessen: The authors
declare no conflict of interest.
Korrespondierender Autor:
Matthew Setzler
msetzler@iu.edu
Urheberrechte ©: © 2020
Massachusetts Institute of Technology
Veröffentlicht unter Creative Commons
Namensnennung 4.0 International
(CC BY 4.0) Lizenz
Die MIT-Presse
l
D
Ö
w
N
Ö
A
D
e
D
F
R
Ö
M
H
T
T
P
:
/
/
D
ich
R
e
C
T
.
M
ich
T
.
/
e
D
u
Ö
P
M
ich
/
l
A
R
T
ich
C
e
–
P
D
F
/
D
Ö
ich
/
ich
/
.
/
1
0
1
1
6
2
Ö
P
M
_
A
_
0
0
0
3
6
1
8
6
8
4
4
0
Ö
P
M
_
A
_
0
0
0
3
6
P
D
/
.
ich
F
B
j
G
u
e
S
T
T
Ö
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
Coordination in Interacting, Improvising Musicians
Setzler, Goldstone
Humans align their behaviors as they interact (Hasson & Frith, 2016; Pickering & Garrod,
2004, 2013). We spontaneously entrain periodic motions (z.B., postural sway, walking gait),
and such entrainment is predictive of successful interaction and performance on joint tasks
(Demos et al., 2012; Paxton & Dale, 2013; M. J. Richardson et al., 2007; Schmidt & Richardson,
2008; Shockley et al., 2003; Shockley et al., 2009; Valdesolo et al., 2010). Interlocutors tend
to mirror one another’s posture, speech prosody and align eye gaze to fixate on the same ob-
jects as they interact (Garrod & Pickering, 2009; Louwerse et al., 2012; D. C. Richardson &
Dale, 2005; D. C. Richardson et al., 2007; D. C. Richardson et al., 2009). Alignment occurs
at more abstract levels as well. Interlocutors mirror vocabulary and syntactical constructions,
and come to share common mental representations for situations under discussion (Abney
et al., 2014; Dale & Spivey, 2006; Pickering & Garrod, 2004). Past JA research demonstrates
that alignment is an important interpersonal mechanism that facilitates joint attention and pre-
dictive emulation (of a partner’s future actions), and streamlines communication by providing
a common representational scheme (Garrod & Pickering, 2009; Pickering & Garrod, 2004;
D. C. Richardson et al., 2009; Sebanz et al., 2006; Sebanz & Knoblich, 2009).
Another issue in JA research is whether group behavior is supported by mutual adapta-
tionen (bidirectional coordination) or fixed leader-follower roles (unidirectional coordination).
Clearly delineated leader-follower roles appear to support stable coordination in many nat-
uralistic JA domains (z.B., conductor of an orchestra, lead dancer in a salsa pair), and ex-
perimental studies have affirmed the utility of unidirectional coordination with respect to
particular task constraints and participant expertise levels (Curioni et al., 2019; Noy et al.,
2011; M. J. Richardson et al., 2015). Andererseits, finger-tapping studies have shown
that dyads achieve greater synchronization when mutually coupled compared to unidirectional
Bedingungen (Demos et al., 2017; Konvalinka et al., 2010). Rather than adopting leader-follower
roles, mutually coupled individuals each adapted their own tapping rates to their partner’s pre-
vious tapping rates (Konvalinka et al., 2010). A similar result has been observed in a simplified
experimental adaptation of the “mirror game,” which requires dyads to synchronize improvised
hand movements with one another. Mutually coupled dyads synchronized more fluidly and
generated more dynamic movements compared to dyads that were assigned leader-follower
roles (Noy et al., 2011).
These findings show that mutual coupling often promotes coordination by supporting
robust and flexible behavioral alignment. Jedoch, they were obtained in idealized experi-
mental paradigms using greatly simplified behaviors (z.B., synchrony of a tapped pulse), Also
it is unclear whether and how they generalize to more sophisticated coordinated behavior
found in the real world. Naturalistic JA is often open-ended, and requires not just behav-
ioral matching but also complementary coordination in service of abstract, functional goals
(z.B., operating on a patient, generating ideas in group brainstorming sessions; Hasson & Frith,
2016; Paulus et al., 2010). How does mutual coupling shape coordination in these more
Komplex, naturalistic forms of JA? Does mutual coupling support greater behavioral alignment
in underconstrained tasks, where this is no explicit goal of synchronization? Does it support
complementary coordination, in service of abstract goals?
In this study we use improvised music as a model domain to explore the effects of mu-
tual coupling in the wild. Conveniently, joint music performance is naturally mediated by
organizational structures that constrain ensemble coordination. Orchestras are hierarchically
organized with fixed leader-follower roles, whereas free improvising jazz ensembles are typ-
ically more characterized by feedback loops of mutual influence (Borgo, 2005; D’Ausilio
et al., 2012). Studio recording practices such as “overdubbing” also constrain coordination by
sequentially recording individual musical parts. Ensemble performance research has shown
OPEN MIND: Discoveries in Cognitive Science
89
l
D
Ö
w
N
Ö
A
D
e
D
F
R
Ö
M
H
T
T
P
:
/
/
D
ich
R
e
C
T
.
M
ich
T
.
/
e
D
u
Ö
P
M
ich
/
l
A
R
T
ich
C
e
–
P
D
F
/
D
Ö
ich
/
ich
/
/
.
1
0
1
1
6
2
Ö
P
M
_
A
_
0
0
0
3
6
1
8
6
8
4
4
0
Ö
P
M
_
A
_
0
0
0
3
6
P
D
/
.
ich
F
B
j
G
u
e
S
T
T
Ö
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
Coordination in Interacting, Improvising Musicians
Setzler, Goldstone
that these underlying patterns of coordination are reflected in the music and movements of
ensemble members (Hennig, 2014; Keller, 2014; Rasch, 1979), such as small temporal asyn-
chronies of co-performer note onsets (Demos et al., 2017; Goebl & Palmer, 2009; Keller &
Appel, 2010; Palmer & Ich habe es behalten, 2017), and postural sway couplings (Chang et al., 2017; Eerola
et al., 2018).
Improvised music is of particular interest, because the influence of coordination extends
beyond sensorimotor coupling and into the music’s formal architecture, which is freely evolv-
ing over time in its rhythm, melody, harmony, and texture. We might thus expect underlying
coordination patterns to constrain these structural elements, similar to how they constrain sen-
sorimotor coupling in scored music performance. Do mutually coupled improvisers engage
in bidirectional coordination at the level of notes and rhythms? Wenn ja, does this result in higher
quality music? Answering these questions will extend our understanding of JA beyond idealized
laboratory tasks and into sophisticated, open-ended coordination that occurs in elite artistic
Aufführungen. It will also yield direct implications for music technology. Results will reveal
repercussions of the popular recording technique of overdubbing, and our quantitative mea-
sures of improvised musical coordination can be incorporated into artificial interactive music
Systeme (Gillick et al., 2019; Linson et al., 2015) and benefit music pedagogy by automating
assessment of ensemble performance.
Despite a paucity of cognitive science research on collective improvisation, some no-
table efforts have begun. Previous studies have shown that improvised musical coordination is
shaped by musical context (z.B., playing with a drone versus a swing backing track), und das
experimentally manipulated social attitudes (z.B., dominant, caring) are sonically encoded in
improvised musical interactions (Aucouturier & Canonne, 2017; Walton et al., 2018). Diese
studies lay an important foundation, but they did not experimentally isolate mutual coupling
between musicians. Darüber hinaus, their analyses did not incorporate music theory, and thus the
findings are limited to temporal and acoustic coordination properties, and do not extend to
more abstract musical phenomena such as the emergence of tonal structure (d.h., harmony,
melody).
In the current study we directly manipulate interaction in co-improvising musicians, Und
examine how different underlying patterns of coordination constrain the exchange and emer-
gence of rhythmic and tonal information. Professional jazz musicians freely improvised in two
duo conditions: a coupled condition, in which both pianists improvised simultaneously, und ein
one-way condition, in which a single pianist improvised along with a recording of another pi-
anist (a “ghost partner”) from a previous coupled duet. Improvisations were completely “free”
in the sense that there was no predetermined songform, key signature, or tempo; the only in-
struction was to improvise a compelling piece of music de novo, as in an actual performance.
These duo conditions provided two naturalistic musical settings to isolate the effects of mutual
coupling in freely improvising musicians. Whereas coupled duos had the ability to mutually
adapt to one another, one-way duos were restricted to unidirectional coordination (d.h., Sei-
cause the “ghost partner” was unresponsive to the live musician), as in the common studio
recording technique of overdubbing.
Participants were recorded in isolated MIDI1 tracks as they improvised in each con-
dition. Time series of two fundamental musical features were extracted and analyzed: onset
1 Musical Instrument Digital Interface (MIDI) is a format for representing music on a computer. It symbolically
represents the pitch, volume and timing (onset and offset) of musical note sequences.
OPEN MIND: Discoveries in Cognitive Science
90
l
D
Ö
w
N
Ö
A
D
e
D
F
R
Ö
M
H
T
T
P
:
/
/
D
ich
R
e
C
T
.
M
ich
T
.
/
e
D
u
Ö
P
M
ich
/
l
A
R
T
ich
C
e
–
P
D
F
/
D
Ö
ich
/
ich
/
.
/
1
0
1
1
6
2
Ö
P
M
_
A
_
0
0
0
3
6
1
8
6
8
4
4
0
Ö
P
M
_
A
_
0
0
0
3
6
P
D
.
/
ich
F
B
j
G
u
e
S
T
T
Ö
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
Coordination in Interacting, Improvising Musicians
Setzler, Goldstone
density and tonal consonance. Onset density indexes overall rhythmic activity level, und hat
been shown to correlate with listener perception of musical tension (Farbood, 2012). Tonal
consonance refers to how different combinations of notes sound on a continuum from disso-
nant/unstable to consonant/stable (Johnson-Laird et al., 2012), and was operationalized using
a previously established model of musical tonality, the tonal spiral array (Chew, 2005, 2014;
Herremans & Chew, 2016). We find that interaction condition systematically altered the coor-
dinated musical behavior of dyads, who were more rhythmically coupled and produced more
consonant tonal structure when mutually coupled. These effects were paralleled in the subjec-
tive experiences of participants as well as nonmusician listeners, who preferred coupled duets
despite being blind to condition. These results are presented and discussed in terms of their
implications for music technology and JA research more generally.
METHODEN
Teilnehmer
Twenty-eight professional pianists (25 männlich, 3 weiblich) from the New York City jazz scene
participated in this study. Participant age ranged from 21 Zu 37. On average participants had
über 22 years’ experience playing piano (SD = 5.2) Und 15 years experience improvising
(SD = 4.6). All participants had extensive experience with free improvisation, and received
formal training in piano performance and/or jazz studies at elite conservatories. Teilnehmer
were recruited by word of mouth, and had no prior experience performing with one another.
Es gab 122 individuals that participated in the listener study. Of those, 101 war
undergraduate psychology students from Indiana University without any particular musical
background, Und 21 (19 männlich, 2 weiblich) were professional jazz musicians, each with over 10
years of experience as improvising musicians, recruited by word of mouth from the New York
City music scene. None of these listeners participated in the initial music-generation stage of
the study.
Design and Procedure
Participants played a series of short (4–7 min “free”) improvisations, with no accompanying
stimuli and no prior musical template or constraints. Other than the suggested time frame, Die
only instruction was to improvise a compelling piece of music, as in a typical performance
setting. Participants were informed of the two interaction conditions, but were not told which
condition they were playing in on any given trial (and there was no visual or audible indication
of condition, see the Supplemental Materials, Setzler & Goldstone, 2020). After each trial, Sie
responded to questionnaires indicating their subjective experience playing in the previous trial
in terms of: (1) how easy it was to coordinate with their partner, (2) how well coordinated they
were with their partner, (3) quality of the improvised piece, Und (4) degree to which they played
a leader versus a supporter role.
Each participant played at least three duets (Versuche) in each condition, with the same “live”
partner for every coupled duet and the same “ghost” partner for every one-way duet. Condi-
tions were interleaved within participant pairs and counterbalanced across pairs to control
for possible order effects. Participants were recorded in isolated MIDI tracks, and individual
recordings from coupled duets yoked one-way duets in subsequent sessions, as depicted in the
Supplemental Materials (Setzler & Goldstone, 2020). Insgesamt 50 coupled duets and 86 eins-
way duets were collected; duets had an average duration of 342 S (min = 108 S, max = 738 S,
SD = 12 S). See the Supplemental Materials for a link to the publicly hosted data set (Setzler
& Goldstone, 2020).
OPEN MIND: Discoveries in Cognitive Science
91
l
D
Ö
w
N
Ö
A
D
e
D
F
R
Ö
M
H
T
T
P
:
/
/
D
ich
R
e
C
T
.
M
ich
T
.
/
e
D
u
Ö
P
M
ich
/
l
A
R
T
ich
C
e
–
P
D
F
/
D
Ö
ich
/
ich
/
.
/
1
0
1
1
6
2
Ö
P
M
_
A
_
0
0
0
3
6
1
8
6
8
4
4
0
Ö
P
M
_
A
_
0
0
0
3
6
P
D
.
/
ich
F
B
j
G
u
e
S
T
T
Ö
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
Coordination in Interacting, Improvising Musicians
Setzler, Goldstone
Tisch 1. Consonance ratings of exemplar pitch sets.
Pitch Set
Consonance
{C,E,G} (Cmaj)
{C,Eb,G} (Cmin)
{C,B,G}
{C,E,G,F,A,C} (Cmaj + Fmaj)
{C,B}
{C,E,G,F#,A#,C#} (Cmaj + F#maj)
serial (alle 12 pitches)
.65
.65
.54
.49
.48
.13
.09
A post hoc study was conducted with populations of naive listeners and expert jazz
Musiker. Listeners heard 30-s audio clips randomly sampled from duets in both conditions
(audio from each pianist was panned to separate ears). After listening to each clip they were
asked to rate (1) their enjoyment of the music, (2) how well coordinated they perceived the
musicians to be, Und (3) which musician played more of a leader role. Listeners were also asked
to guess which condition a clip came from. Each participant heard complementary yoked sets
of coupled and one-way clips. See the Supplemental Materials for full specification of the
sequencing design, which controlled for possible order and stereo-panning effects (Setzler &
Goldstone, 2020).
Tonal Consonance Measure
Our tonal consonance measure is based on the tonal spiral array model, which has been vali-
dated against listener ratings and expert music theory analyses (Chew, 2005, 2014; Herremans
& Chew, 2016). Tisch 1 shows model ratings for exemplar pitch sets. See the Supplemental
Materials for specification of the measure (Setzler & Goldstone, 2020).
Data Analysis
Listener ratings were analyzed with Bayesian mixed-effects models for each response type,
using the brms package in R (Bürkner, 2016; Carpenter et al., 2017). Instead of predicting en-
joyment and coordination ratings directly, models predicted the difference between ratings of
coupled audio clips minus ratings of correspondingly yoked one-way clips, such that positive
intercepts indicated preference for coupled clips. Leadership ratings within one-way trials were
modeled such that positive intercepts indicated perception of “ghosts” leading, and negative
values indicated perception of live musicians leading. Accuracy of condition guesses was mod-
eled as binomial outcome: whether or not listeners guessed the correct condition, such that
positive intercepts indicated above-chance predictions. Models included a predictor for sub-
ject type (naive listener or professional jazz musician), and random intercepts per individual.
Bayesian mixed-effects models were also used to analyze time series measures of musical co-
ordination (cross-correlation of onset density and lagged consonance, see Results). Dependent
measures were predicted by a fixed-effect of interaction condition, with random intercepts for
yoked groupings at the duo and duet levels.2 Unidirectional coordination in one-way duos was
analyzed by predicting dependent measures as a function of lag direction (d.h., ghost-to-live
versus live-to-ghost), with random-effects for each duo and duet.
2 “Duo” refers to a pair of performers and “duet” refers to a particular piece produced by a duo. Each coupled
duo yoked two one-way duos, same for duets.
OPEN MIND: Discoveries in Cognitive Science
92
l
D
Ö
w
N
Ö
A
D
e
D
F
R
Ö
M
H
T
T
P
:
/
/
D
ich
R
e
C
T
.
M
ich
T
.
/
e
D
u
Ö
P
M
ich
/
l
A
R
T
ich
C
e
–
P
D
F
/
D
Ö
ich
/
ich
.
/
/
1
0
1
1
6
2
Ö
P
M
_
A
_
0
0
0
3
6
1
8
6
8
4
4
0
Ö
P
M
_
A
_
0
0
0
3
6
P
D
.
/
ich
F
B
j
G
u
e
S
T
T
Ö
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
Coordination in Interacting, Improvising Musicians
Setzler, Goldstone
ERGEBNISSE
Subjective Ratings
Despite being blind to condition, performers and naive listeners both exhibited a strong pref-
erence for coupled over one-way duets. Performers rated coupled trials as producing higher
quality music (21 out of 26 performers rated coupled higher; probability of success = 0.81; ex-
act binomial test p < .01). Coupled trials were also rated as being better coordinated (23 out of
26 performers rated coupled trials as being better coordinated; probability of success = 0.88;
binomial test p < .01), and more easily coordinated (24 out of 26 performers found it easier to
coordinate with their partner on coupled trials; probability 0.92; p < .01). Performers also rated
themselves as playing more of a supportive (versus lead) role in one-way duos, whereas lead-
ership was rated to be more evenly distributed throughout coupled duos (difference between
average ratings within participant by condition; paired t(25) = 3.16, p < .01).
Bayesian mixed-effects models predicting the difference in listener ratings between cou-
pled clips and correspondingly yoked one-way clips indicated that listeners found coupled
clips to be more enjoyable (M = .24, SD = .08, 95% CI = [.08, .40]) and better coordinated
(M = .43, SD = .11, 95% CI = [.21, .64]). Listeners also perceived unresponsive “ghost part-
ners” to lead live musicians in one-way duos (M = .14, SD = .03, 95% CI = [.08, .20]),
whereas leadership was perceived to be more evenly distributed in coupled duos (effect of
condition on deviation of leadership ratings from neutral: M = .14, SD = .03, 95% CI = [.08,
.19]). However, listeners did not guess the correct condition above chance level (M = .03,
SD = .09, 95% CI = [−.14, .21]). These results held equally for both populations of listeners,
as no effects of subject type were observed.
Mutual Coupling Promotes Synchrony
How does coupling influence musicians’ ability to synchronize with one another? Asynchronies
between “near-simultaneous” onsets (co-occurring within 100 ms) played by co-performers
were measured throughout all duets in each condition. Near-zero asynchronies indicate close
temporal alignment, while asynchronies of larger magnitude reflect less precise synchroniza-
tion. As depicted in Figure 1, asynchronies in coupled trials peak around zero (red distribution),
whereas asynchronies in one-way trials are more widely distributed throughout the +/− 100 ms
range (blue distribution) (KS.test D = 0.024, p < .01), indicating that mutually coupled mu-
sicians achieved more precise synchronization compared to musicians in the overdubbed
condition. We were also curious about leader-follower asymmetries in one-way duos, as pre-
vious studies have reported that supporting musicians lag behind lead musicians in certain
composed musical contexts (Keller & Appel, 2010) (asynchronies are mathematically symmet-
ric around zero for coupled trials because each asynchrony was computed from the perspective
of both live musicians, but asynchronies were only computed from perspective of the single
live musician in one-way trials). However, no such effects were observed here; the distribution
of asynchronies in one-way duets was not significantly asymmetric around 0 in one direction
or the other (mixed-effects model of asynchronies from one-way duos, with random-intercepts
per duo: intercept = −.54, SE = .34, t(18.63) = −1.58, p = .13; model fit with lme4 package
in R, significance assessed with lmerTest using Satterthwaite’s method used to estimate degrees
of freedom; Bates et al., 2014; Kuznetsova et al., 2017).
Activity Matching
Lagged cross-correlation of co-performers’ onset density was computed to analyze how mu-
sicians responded to one another’s rhythmic activity level. Onset density contributes to the
OPEN MIND: Discoveries in Cognitive Science
93
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
o
p
m
i
/
l
a
r
t
i
c
e
-
p
d
f
/
d
o
i
/
i
/
/
.
1
0
1
1
6
2
o
p
m
_
a
_
0
0
0
3
6
1
8
6
8
4
4
0
o
p
m
_
a
_
0
0
0
3
6
p
d
/
.
i
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
Coordination in Interacting, Improvising Musicians
Setzler, Goldstone
Figure 1. Mutual coupling facilitates precise synchronization. Distribution of asynchronies
(asyncs) between co-performers’ near-simultaneous (within 100 ms) note onsets throughout all tri-
als in each condition. Asyncs are more tightly clustered around 0 in coupled trials, indicating more
precise temporal alignment. Asyncs are mathematically symmetric around 0 for coupled trials.
perception of musical tension (Farbood, 2012). A frenzied musical passage comprising many
notes in rapid succession would yield high onset density, whereas a more sparse, mellow
passage would yield low-onset density. Onset density time series were computed for each in-
dividual note sequence using a 2-s sliding window, with a 0.2-s hop size. Figure 2 depicts
lagged cross-correlations, averaged across all duets in each condition. Cross-correlation was
positive overall (cross-correlation averaged across +/− 20 s lag range: M = .39, SD = .04,
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
o
p
m
i
/
l
a
r
t
i
c
e
-
p
d
f
/
d
o
i
/
i
/
.
/
1
0
1
1
6
2
o
p
m
_
a
_
0
0
0
3
6
1
8
6
8
4
4
0
o
p
m
_
a
_
0
0
0
3
6
p
d
/
.
i
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
Figure 2. Musicians match the activity level of their partners. Points represent mean lagged
cross-correlation across all trials within each condition. Error ribbons denote standard error of the
mean. Positive lags in one-way trials represent correlation of ghost recording onset density with
future onset density of live musicians (ghost-to-live) and vice versa for negative lags (live-to-ghost).
Cross-correlation is mathematically symmetric around 0 for coupled trials.
OPEN MIND: Discoveries in Cognitive Science
94
Coordination in Interacting, Improvising Musicians
Setzler, Goldstone
95% CI = [.31, .47]), but significantly higher in coupled duos (red curve) (fixed effect of condi-
tion: M = −.13, SD = .04, 95% CI = [−.21, −.06]). These results indicate a general tendency
for musicians to match the onset density of their partners, which was exaggerated in mutually
coupled duos.
Within one-way duos, cross-correlation was significantly higher at positive, “ghost-to-
live” lags (onset density of ghost recording correlated with future onset density of live musician)
compared to negative, “live-to-ghost” lags (effect of direction: M = .05, SD = .01, 95% CI =
[.02, .08]). This reflects the underlying asymmetry in one-way duos: live musicians were re-
sponsive to notes of ghost recordings but not the other way around. (In contrast, cross- is
mathematically symmetric around zero for coupled trials, because it was computed from the
perspective of both musicians in each duo.) As reported in the Supplemental Materials (Setzler
& Goldstone 2020), a complementary Granger Causality analysis also revealed greater ghost-
to-live versus live-to-ghost Granger causality in one-way duos. Lastly, Figure 2 reveals a dip
in cross-correlation for coupled duets at simultaneous timepoints, but this was not statistically
significant (paired t test of average cross-correlation at lag-0 versus 2-s lag across all trials for
each coupled duo; t(13) = −1.6335, p = .063).
Emergence and Directed Flow of Tonal Information
A previously established model of tonal structure (see Methods and Supplemental Materials,
Setzler & Goldstone, 2020) was adapted to provide a measure of tonal consonance, quantifying
how collections of notes sound on a continuum from unstable/dissonant to stable/consonant
(Chew, 2014; Herremans & Chew, 2016). Time series of combined consonance (conso-
nance of merged music streams from both players in a duo) were computed with a sliding
window.3 Emergent consonance (EC) was operationalized as Combined consonance minus
average consonance of each individual music stream. EC captures the consonance arising
from the interaction of pitches played by collaborating musicians. A situation in which each
pianist plays self-consonant notes that clash with one another would result in low EC (e.g.,
{C,E,G} and {F#,A#,C#} are consonant on their own but {C,E,G,F#,A#,C#} is highly dissonant),
whereas a situation in which each pianist plays dissonant notes that stabilize one another when
sounded together would result in high EC (e.g., {C,B} and {E,G} have low average consonance
but {C,E,G,B} has high consonance because it is tonicized to a Cmaj7 chord). Negative EC
values indicate that combined consonance is less consonant than the average individual conso-
nance and can be interpreted as emergent dissonance. Less negative values can be interpreted
as indexing greater EC (less emergent dissonance) compared to more negative values.
A novel lagged consonance analysis was conducted to quantify how musicians harmo-
nized with one another’s notes as a function of interaction condition. Lagged consonance was
computed by shifting individual note sequences of co-performers relative to one another, com-
puting combined and emergent consonance time series of the merged pitch collections with
a sliding window, and then averaging over time to get a single consonance value per piece
at each lag (5-s sliding window and 2-s hop size were used, although these results were ro-
bust across a range of window sizes, as documented in the Supplemental Materials, Setzler
& Goldstone 2020). This analysis captures the directed flow of tonal information, as it quanti-
fies the degree to which individuals harmonized with the preceding notes of their partner. For
example, Player A might harmonize with Player B’s past notes but not the other way around,
3 A range of window sizes (2, 5, and 10 s) were evaluated, with a hop size of 2 s. The following reported
results were robust across all window sizes.
OPEN MIND: Discoveries in Cognitive Science
95
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
o
p
m
i
/
l
a
r
t
i
c
e
-
p
d
f
/
d
o
i
/
i
.
/
/
1
0
1
1
6
2
o
p
m
_
a
_
0
0
0
3
6
1
8
6
8
4
4
0
o
p
m
_
a
_
0
0
0
3
6
p
d
.
/
i
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
Coordination in Interacting, Improvising Musicians
Setzler, Goldstone
which would be reflected in high consonance for B-to-A lags but not A-to-B lags. Lagged
consonance was computed for every trial in each condition with lags in the range of +/− 20
seconds, spaced by increments of 2 s. Positive lags in one-way duos correspond to evaluat-
ing past notes of the ghost recording with future notes of the live musician (ghost-to-live) and
vice versa for negative lags (live-to-ghost). The beginnings and endings of pieces (first and last
10%) were discarded to avoid boundary effects.
Figure 3 depicts average lagged EC by condition. Time lags are plotted on the x-axis,
and the y-axis represents average lagged EC throughout all duets in each condition. EC is
essentially symmetric around 0 seconds (simultaneous playing) for coupled trials (red curve),
but significantly higher in ghost-to-live (positive) lags compared to live-to-ghost (negative) lags
for one-way trials (blue curve) (effect of lag sign on EC averaged across negative and positive
lags: M = −2.90e-3, SD = 1.18e-3, 95% CI = [−5.22e-3, −5.35e-3]). This asymmetry in
one-way trials was also found with respect to Combined Consonance (effect of lag sign on
average CC: M = −3.45e-3, SD = 1.45e-3, 95% CI = [−6.32e-3, −6.19e-4]). These results
reflect the underlying causal entanglements of each condition. Live musicians in one-way trials
responded to ghost recordings by harmonizing with their past notes, but ghost recordings could
not respond to notes of live musicians. There was no such asymmetry in coupled trials, because
musicians were mutually responsive. As suggested by the difference in height between red
(coupled) and blue (one-way) data points in Figure 3, EC was significantly higher overall in
coupled versus one-way duos (effect of condition on simultaneous EC: M = −1.09e-2, SD =
4.91e-3, 95% CI = [−2.05e-2, −1.01e-3]), although this effect was not significant with respect
to Combined Consonance (M = −1.51e-2, SD = 9.50e-3, 95% CI = [−43.41e-2, 3.21e-3]).
In sum, coupled improvisers mutually harmonized with one another’s preceding notes, and
this dynamic supported more consonant harmonization between them.
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
o
p
m
i
/
l
a
r
t
i
c
e
-
p
d
f
/
d
o
i
/
i
/
/
.
1
0
1
1
6
2
o
p
m
_
a
_
0
0
0
3
6
1
8
6
8
4
4
0
o
p
m
_
a
_
0
0
0
3
6
p
d
.
/
i
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
Figure 3. Lagged consonance analysis reveals musicians harmonize with preceding notes of their
partner. Points denote average emergent consonance (EC) at a given lag across every piece within
each condition, error bars denote standard error of the mean. Negative lags correspond to notes of
the live musician merged with future notes of the ghost recording (live-to-ghost) and vise versa for
positive lags (ghost-to-live). Linear fits of EC by lag are shown for negative and positive lags in each
condition.
OPEN MIND: Discoveries in Cognitive Science
96
Coordination in Interacting, Improvising Musicians
Setzler, Goldstone
DISCUSSION
This study examined how music produced by collaboratively improvising musicians is shaped
by underlying patterns of coordination. Professional jazz pianists improvised in two duo con-
ditions: a coupled condition in which they improvised together simultaneously, and an “over-
dubbed” (one-way) condition which precluded mutual adaptation because improvisers were
recorded sequentially. Our analyses show that coupled duos achieved greater alignment of
their note onsets and more consonant tonal coordination. These results were paralleled in
the subjective experience of the performers and naive listeners, who preferred coupled duets
despite being blind to condition.
Performers and listeners demonstrated systematic insight into the different causal en-
tanglements of each condition. Leadership was rated as evenly distributed amongst coupled
duos, but listeners perceived “ghost partners” as leading live musicians and performers rated
themselves as playing more of a follower/supporter role in one-way duets. These listener re-
sults are remarkable in light of the fact that they were unable to guess which condition music
samples were produced in above chance-level. Listeners were thus implicitly influenced by
the presence or absence of mutual coupling, without their conscious awareness.
Coupled duos synchronized their note onsets more precisely than one-way duos, as in
previous studies which showed that bidirectional coordination promotes synchronization in
finger-tapping tasks (Konvalinka et al., 2010) and scored music performance (Demos et al.,
2017; Goebl & Palmer, 2009). Here this phenomenon is observed in freely improvising mu-
sicians, with no explicit synchronization objective. Rather, precise synchronization emerged
spontaneously, in service of the higher-level goal of collectively generating compelling music.
Previous findings have also suggested that humans have an innate predisposition to entrain
rhythms in social contexts (Kirschner & Tomasello, 2009), which could elucidate our result
insofar as pianists may have sensed a lack of live responsiveness in their partners in one-way
duets.
Mutual coupling supported note onset alignment at longer timescales as well. A cross-
correlation analysis of onset density revealed that improvisers tended to match the rhythmic
activity of their partners, and this tendency was significantly stronger in coupled duos. This re-
lates to findings in non-musical JA domains. Previous dyadic conversation studies have shown
that people spontaneously entrain their movements, and mimic one another’s facial expres-
sions, manual gestures, eye gaze, and acoustic speech characteristics when verbally interacting
with one another (Abney et al., 2014; Louwerse et al., 2012; D. C. Richardson & Dale, 2005;
Shockley et al., 2003, 2009). Behavioral alignment has been proposed to foster successful in-
teraction by signaling affiliative attitudes (Demos et al., 2012; Hove & Risen, 2009), and off-
loading predictive emulation (i.e., of a conversation partner’s future behavior) onto one’s own
behavior (Garrod & Pickering, 2009); the temporal alignment observed here may serve these
same interpersonal functions in improvised musical interactions.
Our onset density cross-correlation analysis also inferred different profiles of directional
influence for each interaction condition. Cross-correlation was symmetric between coupled
partners, but there was an asymmetry in one-way duos such that onset density of the live
musician correlated with past onset density of the ghost partner (prerecorded track) but not
vice versa. This result adds to previous demonstrations that causal influence in performing
music ensembles is reflected in the movements and music of co-performers. This has been
shown numerous times in the context of composed music (Chang et al., 2017; Demos et al.,
2017; Keller & Appel, 2010), and the work of (Aucouturier & Canonne, 2017) suggested
that leader-follower roles induced by experimentally manipulated social attitudes (e.g., caring,
OPEN MIND: Discoveries in Cognitive Science
97
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
o
p
m
i
/
l
a
r
t
i
c
e
-
p
d
f
/
d
o
i
/
i
.
/
/
1
0
1
1
6
2
o
p
m
_
a
_
0
0
0
3
6
1
8
6
8
4
4
0
o
p
m
_
a
_
0
0
0
3
6
p
d
/
.
i
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
Coordination in Interacting, Improvising Musicians
Setzler, Goldstone
dominant) are reflected in sound envelopes (loudness) of improvising musicians. However, this
latter finding was somewhat speculative because inter-musician coupling was not explicitly
manipulated. In contrast, our overdubbed interaction condition provides a ground-truth to
verify our analysis against.
Analogous findings were uncovered in the realm of abstract tonal structure. A novel
lagged consonance analysis demonstrated that musicians harmonized with past notes of their
partners. This occurred mutually in coupled duos but asymmetrically in one-way duos, where
live musicians harmonized with preceding notes of the ghost recording, but not vice versa.
Causal influence between improvisers was thus reflected not just in their rhythms, but also in
the notes they played and the directed exchange of tonal information. Additionally, simultane-
ous EC was significantly greater in coupled duos, suggesting that the ability to mutually adapt
to one another’s previous notes promoted robust tonal coordination.
Importantly, our consonance analysis detected not just alignment, but complementary
tonal coordination as well. Consonance is not only achieved when musicians play the same
pitch, but also when they play complementary sets of pitches that combine to produce conso-
nant harmonies. The tonal coupling observed here can be understood in terms of interpersonal
synergies, which have been proposed to emerge in interacting groups whose individuals co-
constrain one another in support of group-level objectives (Fusaroli & Tylén, 2016; Hasson &
Frith, 2016; Riley et al., 2011). In this case, note selection is co-constrained between collabo-
ratively improvising musicians in order to generate tonal structure. Our consonance analysis
contributes an important extension to previous analyses of naturalistic JA, which have pri-
marily operationalized coordination in terms of behavioral matching, using techniques like
cross-correlation and cross recurrence analysis (Dale & Spivey, 2006; Louwerse et al., 2012;
Paxton & Dale, 2013; D. C. Richardson & Dale, 2005; D. C. Richardson et al., 2007). Here we
demonstrate the feasability of using domain-specific measures (i.e., a tonal consonance model
informed by music theory) to assess complementary coordination in support of abstract, func-
tional properties at the group-level (i.e., emergent tonal structure). While there can be no doubt
that alignment is an important interpersonal mechanism, more work of this kind is needed to
investigate complementary coordination in naturalistic JA contexts (Hasson & Frith, 2016).
Successful coordination is difficult to operationalize in freely improvised music, because
it is not explicitly clear what the intentions of musicians are. We analyzed rhythmic alignment
and tonal consonance because they are basic musical elements, and we were able to opera-
tionalize them while imposing minimal musical assumptions (atonal music would be rated low
consonance, onset density works for pulsed and nonpulsed music). The goal of participants
was to generate compelling music, as they would strive for in a typical performance, but they
were not explicitly instructed to synchronize note onsets or produce consonant harmonies.
In fact, some level of musical tension and dissonance is typically desired. This being said,
we observed robust effects that mutual coupling promoted temporal alignment and emergent
tonal consonance overall. We also observed directional effects on these features consistent
with the ground-truth unidirectional influence from recording to musician in one-way duets.
Furthermore, these results were paralleled in the subjective experience of professional impro-
visers and naive listeners with no particular background in jazz music, who preferred mutually
coupled duos, and correctly inferred leadership roles in both conditions.
Taken together, these results suggest that coupled dyads achieved enhanced, bidirec-
tional temporal and tonal coordination, which supported the higher level goal of generating
compelling music. This extends previous investigations of mutual coupling in idealized ex-
perimental paradigms, such as finger tapping (Konvalinka et al., 2010) and the improvised
OPEN MIND: Discoveries in Cognitive Science
98
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
o
p
m
i
/
l
a
r
t
i
c
e
-
p
d
f
/
d
o
i
/
i
.
/
/
1
0
1
1
6
2
o
p
m
_
a
_
0
0
0
3
6
1
8
6
8
4
4
0
o
p
m
_
a
_
0
0
0
3
6
p
d
/
.
i
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
Coordination in Interacting, Improvising Musicians
Setzler, Goldstone
mirror game (Noy et al., 2011), into the rich, naturalistic setting of unconstrained musical im-
provisation. More specifically, our findings directly implicate the common studio recording
technique of overdubbing—which we show results in systematically different music than live,
coupled interaction. Lastly, our measures of expert musical coordination can be incorporated
into the design of generative AI music systems to make them more humanlike and more musical
(Datseris et al., 2019; Gillick et al., 2019; Hawthorne et al., 2019; Hennig, 2014; Hennig
et al., 2011; Huang et al., 2018; Huang et al., 2019).
ACKNOWLEDGMENTS
The authors thank Douglas Hofstadter for his generous support of this work. We also thank
members of the Geolab and Evolutionary and Adaptive Systems (EASy) labs at Indiana Univer-
sity, as well as Todd Gureckis’s Computational Cognitive Science lab at New York University
for invaluable feedback. And of course, we thank all of the tremendous musicians who partic-
ipated in this study.
AUTHOR CONTRIBUTIONS
MS: Conceptualization: Lead; Data curation: Lead; Formal analysis: Lead; Investigation: Lead;
Methodology: Equal; Project administration: Lead; Resources: Lead; Software: Lead; Visual-
ization: Lead; Writing - Original Draft: Lead; Writing - Review & Editing: Lead. RG: Conceptu-
alization: Supporting; Formal analysis: Supporting; Funding acquisition: Lead; Methodology:
Equal; Resources: Supporting; Supervision: Lead; Validation: Lead; Visualization: Supporting;
Writing - Review & Editing: Supporting.
REFERENCES
Abney, D. H., Paxton, A., Dale, R., & Kello, C. T. (2014). Complexity
Journal of Experimental Psy-
matching in dyadic conversation.
chology: General, 143(6), 2304–2315. DOI: https://doi.org/10
.1037/xge0000021, PMID: 25285431
Aucouturier, J.-J., & Canonne, C. (2017). Musical friends and foes:
The social cognition of affiliation and control in improvised inter-
actions. Cognition, 161, 94–108. DOI: https://doi.org/10.1016
PMCID:
/j.cognition.2017.01.019,
PMC5348120
28167396,
PMID:
Bates, D., Mächler, M., Bolker, B., & Walker, S. (2014). Fitting lin-
ear mixed-effects models using lme4. arXiv. DOI: https://arxiv
org/abs/1406.5823, https://doi.org/10.18637/jss.v067.i01
Borgo, D. (2005). Sync or swarm: Improvising music in a complex
age. A&C Black.
Bürkner, P.-C. (2016). brms: An R package for bayesian generalized
Journal of Statistical Software,
linear mixed models using stan.
80(1), 1–28.
Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B.,
Betancourt, M., Brubaker, M. A., Guo, J., Li, P., & Riddell, A.
(2017). Stan: A probabilistic programming language. Journal of
Statistical Software, 76(1).
Chang, A., Livingstone, S. R., Bosnyak, D. J., & Trainor, L. J. (2017).
Body sway reflects leadership in joint music performance.
Proceedings of the National Academy of Sciences, 114(21),
E4134–E4141. DOI: https://doi.org/10.1073/pnas.1617657114,
PMID: 28484007, PMCID: PMC5448222
Chew, E.
(2005). Regards on two regards by Messiaen: Post-tonal
music segmentation using pitch context distances in the spiral
Journal of New Music Research, 34(4), 341–354. DOI:
array.
https://doi.org/10.1080/09298210600578147
Chew, E.
(2014). Mathematical and computational modeling of
tonality: Theory and applications (Vol. 204). Springer.
Cooke, N. J., & Hilton, M.
(2015). Enhancing the effectiveness of
team science. National Academies Press.
Curioni, A., Vesper, C., Knoblich, G., & Sebanz, N. (2019). Recipro-
cal information flow and role distribution support joint action
coordination. Cognition, 187, 21–31. DOI: https://doi.org/10
.1016/j.cognition.2019.02.006, PMID: 30797991, PMCID:
PMC6446186
Dale, R., & Spivey, M. J.
(2006). Unraveling the dyad: Using re-
currence analysis to explore patterns of syntactic coordination
between children and caregivers in conversation. Language Learn-
ing, 56(3), 391–430. DOI: https://doi.org/10.1111/j.1467-9922
.2006.00372.x
Datseris, G., Ziereis, A., Albrecht, T., Hagmayer, Y., Priesemann, V.,
& Geisel, T.
(2019). Microtiming deviations and swing feel in
jazz. Scientific Reports, 9(1), 1–10. DOI: https://doi.org/10.1038
/s41598-019-55981-3, PMID: 31882842, PMCID: PMC6934603
D’Ausilio, A., Badino, L., Li, Y., Tokay, S., Craighero, L., Canto, R.,
Aloimonos, Y., & Fadiga, L.
(2012). Leadership in orchestra
emerges from the causal relationships of movement kinematics.
OPEN MIND: Discoveries in Cognitive Science
99
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
o
p
m
i
/
l
a
r
t
i
c
e
-
p
d
f
/
d
o
i
/
i
/
/
.
1
0
1
1
6
2
o
p
m
_
a
_
0
0
0
3
6
1
8
6
8
4
4
0
o
p
m
_
a
_
0
0
0
3
6
p
d
/
.
i
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
Coordination in Interacting, Improvising Musicians
Setzler, Goldstone
PLoS One, 7(5), Article e35757. DOI: https://doi.org/10.1371
/journal.pone.0035757, PMID: 22590511, PMCID: PMC3348913
Demos, A. P., Carter, D. J., Wanderley, M. M., & Palmer, C. (2017).
The unresponsive partner: Roles of social status, auditory feed-
back, and animacy in coordination of joint music performance.
Frontiers in Psychology, 8, Article 149. DOI: https://doi.org/10.3389
/fpsyg.2017.00149, PMID: 28261123, PMCID: PMC5306131
Demos, A. P., Chaffin, R., Begosh, K. T., Daniels, J. R., & Marsh, K. L.
(2012). Rocking to the beat: Effects of music and partner’s move-
ments on spontaneous interpersonal coordination. Journal of Ex-
perimental Psychology: General, 141(1), 49–53. DOI: https://doi
.org/10.1037/a0023843, PMID: 21668129
Eerola, T., Jakubowski, K., Moran, N., Keller, P. E., & Clayton, M.
(2018). Shared periodic performer movements coordinate in-
teractions in duo improvisations. Royal Society Open Science,
5(2), Article 171520. DOI: https://doi.org/10.1098/rsos.171520,
PMID: 29515867, PMCID: PMC5830756
Farbood, M. M. (2012). A parametric, temporal model of musical
tension. Music Perception: An Interdisciplinary Journal, 29(4),
387–428.
Fusaroli, R., & Tylén, K.
Investigating conversational dy-
namics: Interactive alignment, interpersonal synergy, and collec-
tive task performance. Cognitive Science, 40(1), 145–171. DOI:
https://doi.org/10.1111/cogs.12251, PMID: 25988263
(2016).
Garrod, S., & Pickering, M. J. (2009). Joint action, interactive align-
ment, and dialog. Topics in Cognitive Science, 1(2), 292–304.
DOI: https://doi.org/10.1111/j.1756-8765.2009.01020.x, PMID:
25164934
Gillick, J., Roberts, A., Engel, J., Eck, D., & Bamman, D.
(2019).
Learning to groove with inverse sequence transformations. arXiv.
https://arxiv.org/abs/1905.06118
Goebl, W., & Palmer, C.
(2009). Synchronization of timing and
motion among performing musicians. Music Perception: An Inter-
disciplinary Journal, 26(5), 427–438. DOI: https://doi.org/10
.1525/mp.2009.26.5.427
Goldstone, R. L., & Gureckis, T. M.
(2009). Collective behavior.
Topics in Cognitive Science, 1(3), 412–438. DOI: https://doi
.org/10.1111/j.1756-8765.2009.01038.x, PMID: 25164995
Guimera, R., Uzzi, B., Spiro, J., & Amaral, L. A. N. (2005). Team as-
sembly mechanisms determine collaboration network structure
Science, 308(5722), 697–702. DOI:
and team performance.
https://doi.org/10.1126/science.1106340, PMID: 15860629,
PMCID: PMC2128751
Hasson, U., & Frith, C. D. (2016). Mirroring and beyond: Coupled
dynamics as a generalized framework for modelling social in-
teractions. Philosophical Transactions of the Royal Society B:
Biological Sciences, 371(1693), Article 20150366. DOI: https://
doi.org/10.1098/rstb.2015.0366, PMID: 27069044, PMCID:
PMC4843605
Hasson, U., Ghazanfar, A. A., Galantucci, B., Garrod, S., &
Keysers, C.
(2012). Brain-to-brain coupling: A mechanism for
creating and sharing a social world. Trends in Cognitive Sciences,
16(2), 114–121. DOI: https://doi.org/10.1016/j.tics.2011.12.007,
PMID: 22221820, PMCID: PMC3269540
Hawthorne, C., Stasyuk, A., Roberts, A., Simon, I., Huang, C.-Z. A.,
Dieleman, S., Elsen, E., Engel, J., & Eck, D.
(2019). Enabling
factorized piano music modeling and generation with the MAE-
In 7th International Conference on Learning
STRO dataset.
Representations (ICLR). OpenReview.net. https://openreview
.net/forum?id=r1lYRjC9F7
Hennig, H. (2014). Synchronization in human musical rhythms and
mutually interacting complex systems. Proceedings of the Na-
tional Academy of Sciences, 111(36), 12974–12979. DOI:
https://doi.org/10.1073/pnas.1324142111, PMID: 25114228,
PMCID: PMC4246955
Hennig, H., Fleischmann, R., Fredebohm, A., Hagmayer, Y.,
Nagler, J., Witt, A., Theis, F. J., & Geisel, T. (2011). The nature and
perception of fluctuations in human musical rhythms. PloS ONE,
6(10), Article e26457. DOI: https://doi.org/10.1371/journal.pone
.0026457, PMID: 22046289, PMCID: PMC3202537
(2016). Tension ribbons: Quantifying
Herremans, D., & Chew, E.
and visualising tonal
In R. Hoadley, C. Nash, &
D. Fober (Eds.), Proceedings of the International Conference on
Technologies for Music Notation and Representation (TENOR
’16) (pp. 8–18). Anglia Ruskin University.
tension.
Hove, M. J., & Risen, J. L. (2009). It’s all in the timing: Interpersonal
synchrony increases affiliation. Social Cognition, 27(6), 949–960.
DOI: https://doi.org/10.1521/soco.2009.27.6.949
Huang, A., Hawthorne, C., Roberts, A., Dinculescu, M., Wexler, J.,
(2019). Approachable music com-
Hong, L., & Howcroft, J.
In Proceedings of the
position with machine learning at scale.
20th International Society for Music Information Retrieval Con-
ference (ISMIR) (pp. 793–800). ISMIR. http://archives.ismir.net
/ismir2019/paper/000097.pdf
Huang, C.-Z. A., Vaswani, A., Uszkoreit, J., Simon, I., Hawthorne, C.,
Shazeer, N., Dai, A. M., Hoffman, M. D., Dinculescu, M., & Eck,
D. (2018). Music transformer: Generating music with long-term
structure. In 7th International Conference on Learning Represen-
tations. (ICLR). OpenReview.net. https://openreview.net/forum
?id=rJe4ShAcF7
Johnson-Laird, P. N. (2002). How jazz musicians improvise. Music
Perception: An Interdisciplinary Journal, 19(3), 415–442. DOI:
https://doi.org/10.1525/mp.2002.19.3.415
Johnson-Laird, P. N., Kang, O. E., & Leong, Y. C. (2012). On musi-
cal dissonance. Music Perception: An Interdisciplinary Journal,
30(1), 19–35. DOI: https://doi.org/10.1525/mp.2012.30.1.19
Ju, W., & Leifer, L. (2008). The design of implicit interactions: Making
interactive systems less obnoxious. Design Issues, 24(3), 72–84.
DOI: https://doi.org/10.1162/desi.2008.24.3.72
Keller, P. E. (2014). Ensemble performance: Interpersonal alignment
of musical expression. In D. Fabian, R. Timmers, & E. Schubert
(Eds.), Expressiveness in music performance: Empirical ap-
proaches across styles and cultures (pp. 260–282). Oxford
University
https://doi.org/10.1093/acprof:oso
/9780199659647.003.0015
Press DOI:
(2010).
Keller, P. E., & Appel, M.
Individual differences, auditory
imagery, and the coordination of body movements and sounds
in musical ensembles. Music Perception: An Interdisciplinary
Journal, 28(1), 27–46. DOI: https://doi.org/10.1525/mp.2010
.28.1.27
Kirschner, S., & Tomasello, M. (2009). Joint drumming: Social con-
text facilitates synchronization in preschool children. Journal of
Experimental Child Psychology, 102(3), 299–314. DOI: https://
doi.org/10.1016/j.jecp.2008.07.005, PMID: 18789454
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
o
p
m
i
/
l
a
r
t
i
c
e
-
p
d
f
/
d
o
i
/
i
.
/
/
1
0
1
1
6
2
o
p
m
_
a
_
0
0
0
3
6
1
8
6
8
4
4
0
o
p
m
_
a
_
0
0
0
3
6
p
d
.
/
i
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
Knoblich, G., Butterfill, S., & Sebanz, N.
research on joint action: Theory and data.
(2011). Psychological
In B. Ross (Ed.),
OPEN MIND: Discoveries in Cognitive Science
100
Coordination in Interacting, Improvising Musicians
Setzler, Goldstone
Psychology of Learning and Motivation (Vol. 54, pp. 59–101).
Elsevier.
29(6), 1045–1060. DOI: https://doi.org/10.1207/s15516709cog0000
_29, PMID: 21702802
Konvalinka, I., Vuust, P., Roepstorff, A., & Frith, C. D. (2010). Follow
you, follow me: Continuous mutual prediction and adaptation in
joint tapping. Quarterly Journal of Experimental Psychology,
63(11), 2220–2230. DOI: https://doi.org/10.1080/17470218
.2010.497843, PMID: 20694920
Kuznetsova, A., Brockhoff, P. B., & Christensen, R. H. (2017). lmertest
package: Tests in linear mixed effects models. Journal of Statis-
tical Software, 82(13), 1–26. DOI: https://doi.org/10.18637/jss
.v082.i13
Linson, A., Dobbyn, C., Lewis, G. E., & Laney, R.
(2015). A sub-
sumption agent for collaborative free improvisation. Computer
Music Journal, 39(4), 96–115. DOI: https://doi.org/10.1162
/COMJ_a_00323
Louwerse, M. M., Dale, R., Bard, E. G., & Jeuniaux, P. (2012). Be-
havior matching in multimodal communication is synchronized.
Cognitive Science, 36(8), 1404–1426.
Noy, L., Dekel, E., & Alon, U. (2011). The mirror game as a paradigm
for studying the dynamics of two people improvising motion
together. Proceedings of the National Academy of Sciences,
108(52), 20947–20952. DOI: https://doi.org/10.1073/pnas
.1108155108, PMID: 22160696, PMCID: PMC3248496
Palmer, C., & Zamm, A. (2017). Interactions in ensemble music per-
formance: Empirical and mathematical accounts. In M. Lessaffre,
M. Leman, & P.-J. Maes (Eds.), The Routledge companion to em-
bodied music interaction (pp. 370–379). Routledge Press. DOI:
https://doi.org/10.4324/9781315621364-41, PMID: 28701879
Paulus, P. B., Levine, D. S., Brown, V., Minai, A. A., & Doboli, S.
(2010). Modeling ideational creativity in groups: Connecting
cognitive, neural, and computational approaches. Small Group
Research,
https://doi.org/10.1177
/1046496410369561
688–724. DOI:
41(6),
Paxton, A., & Dale, R.
(2013). Argument disrupts interpersonal
SAGE Publications. DOI: https://doi.org/10.1080
synchrony.
/17470218.2013.853089, PMID: 24303888
Pickering, M. J., & Garrod, S.
Toward a mechanistic
psychology of dialogue. Behavioral and Brain Sciences, 27(2),
169–190. DOI: https://doi.org/10.1017/S0140525X04000056,
PMID: 15595235
(2004).
Pickering, M. J., & Garrod, S.
(2013). An integrated theory of
language production and comprehension. Behavioral and Brain
Sciences, 36(04), 329–347. DOI: https://doi.org/10.1017
/S0140525X12001495, PMID: 23789620
Rasch, R. A. (1979). Synchronization in performed ensemble music.
Acta Acustica United With Acustica, 43(2), 121–131.
Rebsamen, B., Guan, C., Zhang, H., Wang, C., Teo, C., Ang, M. H.,
& Burdet, E. (2010). A brain controlled wheelchair to navigate in
familiar environments. IEEE Transactions on Neural Systems and
Rehabilitation Engineering, 18(6), 590–598. DOI: https://doi.org
/10.1109/TNSRE.2010.2049862, PMID: 20460212
Repp, B. H., & Su, Y.-H.
(2013). Sensorimotor synchronization:
A review of recent research (2006–2012). Psychonomic Bulletin
& Review, 20(3), 403–452. DOI: https://doi.org/10.3758/s13423
-012-0371-2, PMID: 23397235
Richardson, D. C., & Dale, R. (2005). Looking to understand: The
coupling between speakers’ and listeners’ eye movements and
its relationship to discourse comprehension. Cognitive Science,
Richardson, D. C., Dale, R., & Kirkham, N. Z. (2007). The art of con-
versation is coordination. Psychological Science, 18(5), 407–413.
Richardson, D. C., Dale, R., & Tomlinson, J. M. (2009). Conversa-
tion, gaze coordination, and beliefs about visual context.
Cognitive Science, 33(8), 1468–1482. DOI: https://doi.org/10
.1111/j.1551-6709.2009.01057.x, PMID: 21585512
Richardson, M. J., Harrison, S. J., Kallen, R. W., Walton, A., Eiler,
B. A., Saltzman, E., & Schmidt, R.
Self-organized
complementary joint action: Behavioral dynamics of an inter-
Journal of Experimental Psy-
personal collision-avoidance task.
chology: Human Perception and Performance, 41(3), 665–679.
DOI: https://doi.org/10.1037/xhp0000041, PMID: 25751036,
PMCID: PMC4801327
(2015).
Richardson, M. J., Marsh, K. L., Isenhower, R. W., Goodman, J. R., &
Schmidt, R. C. (2007). Rocking together: Dynamics of intentional
and unintentional interpersonal coordination. Human Move-
ment Science, 26(6), 867–891. DOI: https://doi.org/10.1016/j
.humov.2007.07.002, PMID: 17765345
Riley, M. A., Richardson, M., Shockley, K., & Ramenzoni, V. C.
Frontiers in Psychology, 2,
(2011).
Article 38. DOI: https://doi.org/10.3389/fpsyg.2011.00038,
PMID: 21716606, PMCID: PMC3110940
Interpersonal synergies.
Schmidt, R. C., & Richardson, M. J. (2008). Dynamics of interper-
sonal coordination. In A. Fuchs & V. K. Jirsa (Eds.), Coordination:
Neural, behavioral and social dynamics (pp. 281–308). Springer.
DOI: https://doi.org/10.1007/978-3-540-74479-5_14
Sebanz, N., Bekkering, H., & Knoblich, G. (2006). Joint action: Bod-
ies and minds moving together. Trends in Cognitive Sciences,
10(2), 70–76. DOI: https://doi.org/10.1016/j.tics.2005.12.009,
PMID: 16406326
Sebanz, N., & Knoblich, G. (2009). Prediction in joint action: What,
when, and where. Topics in Cognitive Science, 1(2), 353–367.
DOI: https://doi.org/10.1111/j.1756-8765.2009.01024.x, PMID:
25164938
Setzler, M., & Goldstone, R.
(2020). Supplemental material for
“Coordination and consonance between interacting, improvis-
ing musicians.” Open Mind: Discoveries in Cognitive Science.
DOI: https://doi.org/10.1162/opmi_a_00036
Shockley, K., Richardson, D. C., & Dale, R.
(2009). Conversation
and coordinative structures. Topics in Cognitive Science, 1(2),
305–319. DOI: https://doi.org/10.1111/j.1756-8765.2009.01021.x,
PMID: 25164935
Shockley, K., Santana, M.-V., & Fowler, C. A.
(2003). Mutual
interpersonal postural constraints are involved in cooperative
Journal of Experimental Psychology: Human Per-
conversation.
ception and Performance, 29(2), 326—332. DOI: https://doi
.org/10.1037/0096-1523.29.2.326, PMID: 12760618
Valdesolo, P., Ouyang, J., & DeSteno, D.
(2010). The rhythm of
joint action: Synchrony promotes cooperative ability. Journal of
Experimental Social Psychology, 46(4), 693–695. DOI: https://
doi.org/10.1016/j.jesp.2010.03.004
Walton, A. E., Washburn, A., Langland-Hassan, P., Chemero, A.,
Kloos, H., & Richardson, M. J. (2018). Creating time: Social col-
laboration in music improvisation. Topics in Cognitive Science,
10(1), 95–119. DOI: https://doi.org/10.1111/tops.12306, PMID:
29152904, PMCID: PMC5939966
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
o
p
m
i
/
l
a
r
t
i
c
e
-
p
d
f
/
d
o
i
/
i
/
/
.
1
0
1
1
6
2
o
p
m
_
a
_
0
0
0
3
6
1
8
6
8
4
4
0
o
p
m
_
a
_
0
0
0
3
6
p
d
/
.
i
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
OPEN MIND: Discoveries in Cognitive Science
101
