Respire
Virtual Reality Art with Musical Agent
Guided by Respiratory Interaction
K i V A n Ç t A t A r , M i r J A n A p r p A A N D p h i l i p p e p A S Q u i e r
T
C
A
R
T
S
B
A
Respire is an immersive art piece that brings together three components:
an immersive virtual reality (VR) Umfeld, embodied interaction (über
a breathing sensor) and a musical agent system to generate unique
experiences of augmented breathing. The breathing sensor controls
the user’s vertical elevation of the point of view under and over the
virtual ocean. The frequency and patterns of breathing data guide the
arousal of the musical agent, and the waviness of a virtual ocean in
die Umgebung. Respire proposes an intimate exploration of breathing
through an intelligent mapping of breathing data to the parameters of
visual and sonic environments.
the iMMerSiVe enVironMent of RESPIRE
Respire (2018) is built on our previous work Pulse.Breath.Wa-
ter (2016), which immerses the user in a virtual environment
depicting an ocean, and the user traverses the environment
using their breathing (Feige. 1). Th e environment, made with
Unity and presented using the HTC Vive headset, evokes a
dark, gloomy atmosphere with elements like fog and waves
that wrap around the user [1]. Ambiguous visuals and a lack
of focal objects stimulate the user to engage in the process
of making sense of the scene. Ambiguity in the design of
the interactive artifacts engages users to project their own
values and experiences in the process of making meaning
of the visual stimulus [2]. Respire also exercises a minimalist
color scheme to elicit the beholder’s share eff ect. Kandel [3]
proposes beholder’s share as a process of the user’s projection
of previous experiences in sense-making of an ambiguous
scene. By immersing the user in an ambiguous environment,
Respire aims to give the user a canvas to “paint” their own
Kıvanç Tatar (post-doctoral researcher), School of Interactive Arts and Technology,
Simon Fraser University, 250-13450 102nd Avenue, Surrey, British Columbia,
V3T 0A3, Kanada. Email: ktatar@sfu.ca. ORCID: 0000-0003-4133-8641.
Mirjana Prpa (Ph.D. candidate), School of Interactive Arts and Technology,
Simon Fraser University, 250-13450 102nd Avenue, Surrey, British Columbia,
V3T 0A3, Kanada. Email: mprpa@sfu.ca. ORCID: 0000-0003-0368-6581
Philippe Pasquier (artist, Erzieher, Forscher), School of Interactive Arts and
Technologie, Simon Fraser University, 250-13450 10 2nd Avenue, Surrey,
British Columbia, V3T 0A3, Kanada. Email: pasquier@sfu.ca.
ORCID: 0000-0001-8675-3561.
See www.mitpressjournals.org/toc/lmj/29 for supplemental fi les associated with
this issue.
fig. 1. The virtual environment visuals in Respire. (© Mirjana Prpa)
Erfahrungen. Th e immersion avoids imposing an explicit nar-
rative and the constraints of a story. Hence the environment
empowers the user to curate their experiences.
body And breAth in
iMMerSiVe VirtuAl enVironMentS
Cartesian dualism approaches a human being as a “thinking
thing” that is divorced from bodily experience. Questioning
Cartesian dualism sparks new discourses on what it means
“to be” in the environment. Th e examples of embodied in-
teraction challenge the Cartesian separation between subject
and object, and this separation translates into artistic prac-
tices as a separation between the artwork and the audience.
Embodied interaction [4] emphasizes the value of engaging
our bodies in interaction and transcends Cartesian dualism
by focusing on the body along with the mind as a united
medium for experience of the environment.
What could that link between this united medium and the
environment be? Artists explored breathing as a connector
between the body and virtual environments generated using
computational means. By positioning the body in the center
of the artwork, and employing breath in an embodied inter-
action paradigm, artists succeeded in creating that tangible
©2019 ISAST
doi:10.1162/LM J_a_01057
LEONARDO MUSIC JOURNAL, Bd. 29, S. 19–24, 2019 19
Von http heruntergeladen://direct.mit.edu/lmj/article-pdf/doi/10.1162/lmj_a_01057/1716705/lmj_a_01057.pdf by guest on 07 September 2023
yet invisible link. Zum Beispiel, Sonia Cillari’s As an artist, ICH
need to rest (2009) explores how a body can be a source of
artificial life through breathing [5]. Cillari employs breathing
data to generate a virtual environment of feathers and maps
the breathing patterns to the movement of the feathers in the
virtual environment.
Likewise, Char Davies’s pioneering piece Osmose (1995)
is an immersive virtual environment presented on a head-
mounted display [6]. The user navigates movement in the
virtual environment with breathing and body balance. Der
breathing controls the elevation whereas the body balance
changes the horizontal 2D direction. This mapping resembles
the experience of diving, Und, likewise, we are inspired by
diving phenomena in the breathing interaction of Respire.
Davies juxtaposes two ideas: the immateriality of computer-
generated worlds and the body-felt phenomena elicited by
those environments. The sense of virtual presence afforded
by VR opened a dynamic space for artistic explorations, als
demonstrated in Davies’s piece [7]. Respire maintains that
space for artistic exploration by introducing agent architec-
ture that reacts to breathing.
ArtificiAl intelligence,
Multi-Agent SySteMS And MuSicAl AgentS
All humans breathe, consciously or unconsciously. A critical
element of being alive, breathing continues even when we do
not attend to the act of breathing. Our control of breathing
shifts depending on our attention. Inspired by this mecha-
nism, Respire is intended to use advanced computational
tools for an intelligent mapping from breathing to move-
ment, sound and visuals so as to elicit attention and mindful-
ness. AI and Multi-Agent Systems (MAS) provide such tools
for computational creative applications [8].
The agent paradigm appears in many disciplines, einschließlich
social sciences, philosophy, cognitive science and computer
Wissenschaften. In computer science, an agent is an autonomous
system that initiates actions to respond to its environment
in timely fashion [9]. MAS studies the agent architectures
for computational applications. Musical agents are artificial
agents that automatize musical creative tasks [10]. Respire’s
architecture implements this intelligent mapping using a mu-
sical agent system.
Affect recognition
Affect recognition focuses on designing computational mod-
els that can estimate the affective state of a piece of content;
the content can be, Zum Beispiel, an image, a video, a human
body posture, a sound or a music piece. Two main types
of affective models appear in affective computing: discrete
and continuous [11]. Respire implements a two-dimensional
continuous affect model previously presented by Tatar and
Pasquier [12]. Dimensional affect estimation models gener-
ate a bounded, continuous output to which we apply signal
Verarbeitung, mapping and generative algorithms.
Respire computationally generates the visual and sonic
environments using two separate frameworks: a VR system
and a musical agent generating the sonic environment. Diese
frameworks utilize affective dimensions in the system archi-
tectures. The generative content and reactive behaviors of
these systems use affective dimensions as a high-level cross-
medium paradigm for intelligent mapping. This enables a
human-readable parametrization of two systems generating
visuals and audio separately.
SySteM detAilS
Respire aims to bridge the virtual presence and innate experi-
ences of the user. The artwork builds upon breath-based em-
bodied interaction and utilizes a breath controller (Thought
Technology ProComp2 with respiration harness) for the
user’s vertical position in the scene, simultaneously allow-
ing for exploration of the virtual environment and breath-
ing patterns (Feige. 2). This mapping of breathing to vertical
elevation in a resemblance to diving guides the interaction
such that the audience does not need to learn anything new
to participate in the artwork. We have compared this map-
ping with other options and found the current mapping
to be intuitive from the user’s perspective [13]. Auch, schnell
breathing creates a more eventful sonic environment and is
reflected in and ocean surface filled with waves. Less eventful
breathing (slow-paced breathing) calms the ocean surface
and generates a calmer sonic environment. The pleasantness
dimension calculated by the musical agent controls the color
of the sky in the environment (see Fig. 2), changing the sky
in a range between black (low pleasantness values) to white
(high pleasantness values).
fig. 2. The system architecture of Respire. (© Mirjana Prpa)
20
Tatar, Prpa and Pasquier, Respire
Von http heruntergeladen://direct.mit.edu/lmj/article-pdf/doi/10.1162/lmj_a_01057/1716705/lmj_a_01057.pdf by guest on 07 September 2023
for the musical agent’s memory. We focused on quartal and
quintal harmonies in piano recordings. Th e quartal and quin-
tal harmony theory uses the musical interval of fourths and
fi ft hs, thus generating ambient sounds that avoid tensions and
resolutions of the tonal harmony. In line with the visual aes-
thetics, this harmonic choice avoids imposing an explicit nar-
rative on the user. To obscure the recordings’ source material,
we suppressed the initial attacks of sounds using long fade-in
durations. Th en we applied time stretch and pitch shift ing to
increase the number of samples in the sound memory. Für
the pitch-shift ing, we applied intervals of fourths and fi ft hs
to stay within the quartal harmony. Th en we automatically
labeled each sound using an aff ect estimation algorithm for
Klang. Th e labels are vectors with two dimensions: average
pleasantness and average eventfulness of an audio sample.
Th e details of this aff ect estimation algorithm are previously
published, where the multivariate linear regression model is
trained on ground-truth data [16]. Th e negative correlation
between the eventfulness and pleasantness in Fig. 3 has been
observed in our previous studies. In sound studies, the af-
fective dimensions valence and arousal are exchanged with
eventfulness and pleasantness, because a sound doesn’t feel
an emotion; it stimulates one. Zum Beispiel, there is no con-
cept of a happy sound (excluding anthropomorphism), Aber
some sounds trigger positive emotions in humans.
fig. 3. The affective labels of audio samples in the musical agent’s
Erinnerung. Each dot represents one audio sample. (© Kıvanç Tatar)
Th e musical agent is developed in Cycling74’s Max, und das
communication between Max and Unity utilizes UDP-based
OSC. Th e breathing sensor data is passed to Max using M+M
middleware [14]. Below, we delve into the agent architecture,
which consists of fi ve main modules: Erinnerung, perception,
Ziel, action and postprocessing.
Sound Memor y
Th e agent’s memory consists of audio samples and symbolic
data on pleasantness and eventfulness of the samples (Feige. 3).
Th is type of corpus in a musical agent is known as a hybrid
corpus [15]. In line with Respire’s aesthetic choice of ambigu-
ität, we aimed for a curation of abstract and ambient sounds
Signal Processing of Breathing Data
Th e perception module recognizes the frequency of user
breathing by the wavelet transform of the breathing am-
plitude stream (Feige. 4A). Th e wavelet transform outputs the
spectrum of a signal. Th e breathing frequencies can go as low
fig. 4. The musical agent in Respire. (© Kıvanç Tatar)
Von http heruntergeladen://direct.mit.edu/lmj/article-pdf/doi/10.1162/lmj_a_01057/1716705/lmj_a_01057.pdf by guest on 07 September 2023
Tatar, Prpa and Pasquier, Respire 21
ment oscillates between a muddy, low-frequency-prominent
audio environment and a full-spectrum audio environment.
This mapping is meant to enhance the user’s feeling of sub-
mersion.
zuletzt, the agent applies an affect estimation algorithm to
estimate the eventfulness and pleasantness of the generated
audio environment (Feige. 4e). The estimation algorithm is
the online version of the estimation algorithm that is used
to label the audio in the agent’s memory. The output of the
online affect estimation is a vector with two dimensions:
eventfulness and pleasantness. These values are further used
to control the parameters of the virtual reality environment
(see the System Details section and Fig. 2).
exhibitionS
Respire is the continuation of our previous artwork Pulse.
Breath.Water. Both artworks share the same system design,
and we improved the visual environment in Respire by ex-
ploring different color schemes and virtual lights and adding
fog to improve the ambiguous aesthetics. We presented Pulse.
Breath.Water in three collective exhibitions. The first exhibi-
tion, Scores + Traces: exposing the body through computation,
took place at the One Art Space gallery in Manhattan, New York
[18], in March 2016. The theme of the exhibition was move-
ment and computation, and the exhibition brought a new
perspective on how to incorporate movement theories in
computational arts.
After the Scores + Traces exhibition, we were invited by
the Regina Miranda & Actors/Dancers Company for a col-
laboration to create and produce a piece titled P.O.E.M.A
(Percursos Organizados Entre Movimentos Aleatórios; In
English: Organized Paths among Aleatory Movements) für
the cultural program at the 2016 Rio Olympics, which was
shown for five weeks in summer 2016.
P.O.E.M.A is a choreographic installation that incorpo-
rates contemporary dance to Pulse.Breath.Water (see Figs
5–7). The piece was exhibited in a 10-×-10m room, und das
als 0.03 Hz. As the window size increases, we introduce longer
delays to the system. The wavelet transform addresses this by
using different window sizes to calculate each band, welche
provides for the detection of sudden changes in the signal
while calculating low frequencies. In our implementation,
the wavelet minimum frequency is 0.03 Hz, the maximum
frequency is 2 Hz, the carrier frequency is 0.06 Hz and there
Sind 4 bands per octave. Somit, the output of the wavelet
transform is the power of 24 bands. zuletzt, the perception
module outputs the band with the highest power.
Generative Algorithm of Musical Agent
The goal module (Feige. 4B) generates a vector with two di-
mensions, eventfulness and pleasantness, to select a sound
from the memory (Feige. 4C) to be played by the action module
(Feige. 4D). The memory module chooses the audio sample with
the closest Euclidean distance to the 2D vector (pleasantness
and eventfulness) generated by the goal module. The goal
module generates the eventfulness values by using a map-
ping between the wavelet frequency with the highest power
and the eventfulness of audio samples. The lowest and high-
est wavelet frequency bands are mapped to the lowest and
highest eventfulness values of the agent’s memory, bzw-
aktiv. Using these maximum values (Feige. 3), the frequency
bands of breathing are mapped to 24 eventfulness values with
equal distance. The agent applies a 2D random walk around
one of these 24 eventfulness values so that the 24 wavelet
bands correspond to 24 different areas in 2D affective space in
Feige. 3.
The action module incorporates two playback engines
(Feige. 4D). The first engine applies wavetable synthesis to
generate a heartbeat-like sound. The prominent frequency
of breathing data is mapped to the looping frequency of the
wavetable synthesis. Somit, the pulsation sound in the sonic
environment slows down as the user breathes more slowly.
Ähnlich, the pulsation speeds up following acceleration in
breathing. The lower-frequency bands slow down the pul-
sation sound to a point that the pulsation
morphs into an ambient, pad-like sound.
The second playback engine in the action
module includes four voices. Each voice is
active at all times. When the goal module
chooses an audio sample, this playback en-
gine plays the given sample. When a sample
finishes, the action module requests a new
sample from the goal and memory module.
We developed the postprocessing mod-
ule to further enhance the interaction be-
tween the user and the audio environment
by introducing a low-pass filter (Feige. 4e).
The amplitude of breathing controls the
cutoff frequency of this filter. The cutoff
frequency increases as the user breathes in
und umgekehrt. This mapping resembles the
relation between musical pitch and move-
ment [17]. Somit, as the user breathes in
and out, the timbre of the sonic environ-
fig. 5. A dancer in still position in P.O.E.M.A, 2016. (© Kıvanç Tatar. Photo © Adriano Fagundes.)
22
Tatar, Prpa and Pasquier, Respire
Von http heruntergeladen://direct.mit.edu/lmj/article-pdf/doi/10.1162/lmj_a_01057/1716705/lmj_a_01057.pdf by guest on 07 September 2023
Und 360 videos [19]. We heard that
Pulse.Breath.Water brought a new
perspective to the exhibition by pro-
voking the idea of exploring a virtual
presence using breathing as a way of
interacting with the immersive en-
vironment. Later, we revisited the
visuals and created Respire. We ex-
hibited this new version at the CHI
2018 VR exhibition in Montreal in
April 2018 [20] and Digital Carnival
2018 in Vancouver in August 2018
[21].
future StepS
The overall interaction design and
the system architecture of Respire
stand closer to reactivity in com-
parison to interactivity. Tatar and
Pasquier [22] clarify that interactivity of agents involves
proactivity, das ist, planning future actions and interacting
with other human, biological or artificial agents. A further
step could be to research how the introduction of proactive
behaviors would affect the ambiguity and meaning-making
of Respire’s user experience. Zum Beispiel, in an interactive
scenario, periods of prolonged rapid breathing could lead to
a calming of the water surface and less musical agent activity,
which in turn could lead to a change in the user’s breathing.
Jedoch, our previous exhibition experiences showed that
the audience interacted with Respire in two main phases: A
playful phase where the effect of the breathing and the sen-
sor was explored, followed by a calmer phase where the user
started moving less than in the initial phase. Although these
findings are speculative, they initiate a direction for further
research on reactive and interactive scenarios.
fig. 6. A user with VR headset, a dancer and spectators in the exhibition space of P.O.E.M.A, 2016.
(© Kıvanç Tatar. Photo © Adriano Fagundes.)
main challenge was creating a space that would bring the
virtual environment of Pulse.Breath.Water and the dancers
together. There would be several spectators in the space while
one audience member was in the virtual environment (Feige.
6). The view of the user in the virtual environment was pro-
jected on one wall. To expand the 2D projection of the virtual
environment to the 3D space of the dancers, we utilized a
white colored space with a light design inspired by the work
of James Turrell (Feige. 7). This light design blended the 2D
projection of the virtual environment with the 3D space of
dancers. The audio followed this approach of blending, Und
we expanded the stereo output of the sonic environment to
quadrophonics by using spectral spatialization.
Three dancers joined the project and began perform-
ing one at a time. The dancers were changed daily to allow
them rest after several hours of performance. The dancers
interacted with the virtual environ-
ment using a set of 200 choreo-
graphic cells, snippets of move-
gen. These choreographic cells
were the dancers’ vocabulary for
reaction to the emerging behaviors
initiated by the interaction between
the user and the virtual environ-
ment. During changes of the user
of the virtual environment, the vi-
suals went to black, the lights were
dimmed, the dancers remained in
a static posture and there were two
video loops of three dancers on the
side walls.
The third exhibition of Pulse.
Breath.Water, at the MUTEK Mixed
Realities VR Exhibition (November
2016), enthalten 40 virtual reality
artworks covering interactive im-
mersive environments, narrations
within immersive environments
fig. 7. A dancer in movement in P.O.E.M.A, 2016. (© Kıvanç Tatar. Photo © Adriano Fagundes.)
Von http heruntergeladen://direct.mit.edu/lmj/article-pdf/doi/10.1162/lmj_a_01057/1716705/lmj_a_01057.pdf by guest on 07 September 2023
Tatar, Prpa and Pasquier, Respire 23
Danksagungen
This research was funded by the Natural Sciences and Engineering Re-
search Council of Canada (NSERC) and the Social Sciences and Hu-
manities Research Council of Canada (SSHRC).
Referenzen und Notizen
1 K. Tatar, M. Prpa and P. Pasquier, Respire: www.kivanctatar.com
/respire (zugegriffen 26 September 2018).
2 W.W. Gaver, J. Beaver and S. Benford, “Ambiguity as a Resource for
Design,” in Proceedings of the SIGCHI Conference on Human Fac-
tors in Computing Systems (CHI ’03) (New York: ACM, 2003) S.
233–240.
3 E. Kandel, Reductionism in Art and Brain Science: Bridging the Two
Cultures (New York: Columbia Univ. Drücken Sie, 2016).
4 P. Dourish, Where the Action Is (MIT Press, 2004).
5 NIMkartchannel, Sonia Cillari: As an Artist, I Need to Rest (2009):
www.youtube.com/watch?v=GcA6Yw2QERo.
6 C. Davies, “OSMOSE: Notes on Being in Immersive Virtual Space,”
Digital Creativity 9, NEIN. 2, 65–74 (Januar 1998).
7 M. Slater, “Place Illusion and Plausibility Can Lead to Realistic Be-
haviour in Immersive Virtual Environments,” Philosophical Transac-
tions of the Royal Society of London B: Biological Sciences 364, 1535,
3549–3557 (Dezember 2009).
8 P. Pasquier et al., “An Introduction to Musical Metacreation,” Comput-
ers in Entertainment 14 (2017) S. 1–14: www.doi.org/10.1145/2930672.
9 M. Wooldridge, An Introduction to MultiAgent Systems (Chichester,
VEREINIGTES KÖNIGREICH.: Wiley, 2009).
10 K. Tatar and P. Pasquier, “Musical Agents: A Typology and State of
the Art towards Musical Metacreation,” Journal of New Music Re-
suchen 48, NEIN. 1, 1–50 (2018).
11 J. Panksepp, Affective Neuroscience: The Foundations of Human and
Animal Emotions (Oxford: Oxford Univ. Drücken Sie, 1998).
12 K. Tatar and P. Pasquier, “MASOM: A Musical Agent Architecture
Based on Self Organizing Maps, Affective Computing, and Variable
Markov Models,” in Proceedings of the 5th International Workshop on
Musical Metacreation (MUME 2017) (Atlanta, GA, USA.: MUME,
2017).
13 M. Prpa et al., “The Pulse Breath Water System: Exploring Breathing
as an Embodied Interaction for Enhancing the Affective Potential of
Virtual Reality,” in HCI International 2017, Verfahren (Vancouver:
Springer, 2017).
14 “M+M Architecture”: H+ Technology www.hplustech.com/blogs
/news/m-m-middleware (zugegriffen 31 Dezember 2018).
15 Tatar and Pasquier [10].
16 J. Fan, M. Thorogood and P. Pasquier, “Automatic Soundscape Affect
Recognition Using A Dimensional Approach,” Journal of the Audio
Engineering Society 64, NEIN. 9, 646–653 (2016): www.doi.org/10.17743
/jaes.2016.0044.
17 B. Sievers et al., “Music and Movement Share a Dynamic Structure
That Supports Universal Expressions of Emotion,” Proceedings of the
Nationale Akademie der Wissenschaften 110, NEIN. 1, 70–75 (2013).
18 “Scores+Traces: Exposing the body through computation,” One
Art Space (2016, Marsch 10): www.oneartspace.com/2016/03/10
/scorestraces-exposing-the-body-through-computation (zugegriffen
16 Juli 2019).
19 “MUTEK_IMG Mixed Realities,” MUTEK: www.mutek.org
/img/2016/artworks (zugegriffen 4 Januar 2018).
20 M. Prpa et al., “Respire: A Breath Away from the Experience in
Virtual Environment,” CHI EA ’18 Extended Abstracts of the 2018
CHI Conference on Human Factors in Computing Systems (Montreal:
ACM, 2018) S. 1–6.
21 “Digital Carnival 2018,” Cinevolution Media: www.cinevolution
media.com/digital-carnival-twines-ancestral-wisdom-and-digital
-innovation-aug-31-sep-1-2018 (zugegriffen 26 September 2018).
22 Tatar and Pasquier [10] P. 42.
Manuscript received 18 Juli 2018.
Kıvanç TaTar is “a worker in rhythms, frequencies and
intensities,” playing trumpet and electronics, composing ex-
perimental music, performing audiovisuals and researching
creative artificial intelligence for music and interactive media.
His career aims to integrate science, Technologie, engineering,
interactive arts, contemporary arts and design to research in-
terdisciplinary topics to create transdisciplinary knowledge.
Mırjana PrPa researches immersive virtual environments
that support mindfulness, embodied interaction and regulation
of breathing and affective states.
PhılıPPe Pasquıer works on creative AI and generative
Systeme. He is a scientist specialized in artificial intelligence, A
multidisciplinary artist, an educator and a community leader.
24
Tatar, Prpa and Pasquier, Respire
Von http heruntergeladen://direct.mit.edu/lmj/article-pdf/doi/10.1162/lmj_a_01057/1716705/lmj_a_01057.pdf by guest on 07 September 2023
