Mohammadhossein
Moghimi*
School of Electronic, Electrical and
Systems Engineering
University of Birmingham
Robert Stone
School of Electronic, Electrical and
Systems Engineering
University of Birmingham
Pia Rotshtein
School of Psychology
University of Birmingham
Neil Cooke
School of Electronic, Electrical and
Systems Engineering
University of Birmingham
Presence, Vol. 25, No. 2, Primavera 2016, 81–107
doi:10.1162/PRES_a_00249
Influencing Human Affective
Responses to Dynamic Virtual
Environments
Astratto
Detecting and measuring emotional responses while interacting with virtual reality
(VR), and assessing and interpreting their impacts on human engagement and
‘‘immersion,’’ are both academically and technologically challenging. While many
researchers have, in the past, focused on the affective evaluation of passive environ-
menti, such as listening to music or the observation of videos and imagery, virtual real-
ities and related interactive environments have been used in only a small number of
research studies as a mean of presenting emotional stimuli. This article reports the first
stage (focusing on participants’ subjective responses) of a range of experimental inves-
tigations supporting the evaluation of emotional responses within a virtual environ-
ment, according to a three-dimensional (Valence, Arousal, and Dominance) model of
affects, developed in the 1970s and 1980s. To populate this three-dimensional model
with participants’ emotional responses, an ‘‘affective VR,’’ capable of manipulating users’
emozioni, has been designed and subjectively evaluated. The VR takes the form of a
dynamic ‘‘speedboat’’ simulation, elements (controllable VR parameters) of which were
assessed and selected based on a 35-respondent online survey, coupled with the
implementation of an affective power approximation algorithm. A further 68 partici-
pants took part in a series of trials, interacting with a number of VR variations, while
subjectively rating their emotional responses. The experimental results provide an early
level of confidence that this particular affective VR is capable of manipulating individu-
als’ emotional experiences, through the control of its internal parameters. Inoltre,
the approximation technique proved to be fairly reliable in predicting users’ potential
emotional responses, in various affective VR settings, prior to actual experiences.
Finalmente, the analysis suggested that the emotional response of the users, with different
gender and gaming experiences, could vary, when presented with the same affective
VR situation.
1
introduzione
Virtual reality (VR), and interactive 3D environments generally, have expe-
rienced a significant ‘‘comeback’’ of recent years, courtesy of developments in
the gaming industry and the relentless demand for high-fidelity escapist experi-
ences on the part of gamers and simulation users alike. Yet, despite many interna-
tional initiatives involving the design and development of highly innovative and
affordable human–computer interaction (HCI) technologies in the quest for the
ª 2016 by the Massachusetts Institute of Technology
*Correspondence to m.moghimi@pgr.bham.ac.uk.
Moghimi et al. 81
l
D
o
w
N
o
UN
D
e
D
F
R
o
M
H
T
T
P
:
/
/
D
io
R
e
C
T
.
M
io
T
.
/
e
D
tu
P
v
UN
R
/
UN
R
T
io
C
e
–
P
D
l
F
/
/
/
/
/
2
5
2
8
1
1
8
3
6
5
7
9
P
R
e
S
_
UN
_
0
0
2
4
9
P
D
.
F
B
sì
G
tu
e
S
T
T
o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
82 PRESENCE: VOLUME 25, NUMBER 2
ultimate ‘‘immersive’’ experience,1 some believe that true
‘‘immersion’’ may only ever be achieved through the use
of advanced brain–computer interfaces (BCI) (Cairns,
Cox, Berthouze, Dhoparee, & Jennett, 2006). Tuttavia,
until that day arrives, it is important to understand how it
may be possible to measure and, Infatti, influence human
engagement and emotional connectivity with virtual
worlds using psychophysiological techniques.
The term immersion has most often been used to
describe the multisensory experience of presence by indi-
viduals, while performing a task in VR. Tuttavia, differ-
ent researchers have suggested different definitions for
this term (Brown & Cairns, 2004). As an illustration,
Cairns et al. suggested that immersion could be defined
as a feeling of being deeply engaged when people enter a
make-believe world and feel as if it is real (Cairns, Cox,
Berthouze, Dhoparee, & Jennett, 2006). In 2004,
Brown and Cairns suggested that immersion can be di-
vided into three levels: engagement (during which the
users invest time, effort, and most importantly attention),
engrossment (the time that the user’s emotions are
directly affected by the environment), and total immer-
sion (when the users are detached from reality and the
virtual world is, for them, all that matters). They claim
that engagement and engrossment could be achieved
much easier than a total level of immersion, believing
instead that it could be achieved by overcoming other
barriere. Such barriers include empathy as a ‘‘growth of
attachment’’ to the environment, and atmosphere as rep-
resenting the VR’s environmental realism. The authors
also mentioned that ‘‘total immersion can be difficult to
achieve: there are barriers to immersion from both the
human and the system perspectives’’ (Brown & Cairns).
Other researchers combine the immersive experience in
virtual realities and 3D environments with the term pres-
ence, which is defined as ‘‘the extent to which a person’s
cognitive and perceptual systems are tricked into believ-
ing they are somewhere other than their physical loca-
zione’’ (Patrick, Cosgrove, Slavkovie, Rode, Verratti, &
Chiselko, 2000). Based on the variety of definitions of
immersion evident in the literature, several discussions
have been presented on the topic of how to evaluate
immersive experiences. Many believe that true immersion
might even be impossible to achieve with the present
state of maturity in VR and gaming technologies. Others
believe it could be achieved simply by defining the term
more appropriately (Brown & Cairns).
To date, HCI systems designers have, in their
attempts to increase the sense of end user immersion,
introduced several multidimensional input/output devi-
ces, in order to provide user-friendly, intuitive techni-
ques and styles of interaction with real-time 3D worlds.
Tuttavia, the area of HCI research that strives toward
establishing direct communication between a computer
system and the human brain has, until recently, been
treated as science fiction.2,3 In the HCI domain, BCI
systems attempt to improve human–computer interac-
tion and increase the sense of immersion by interfacing
directly with the human brain and, così, removing the
artificial barriers to intuitive interaction afforded by con-
ventional input-display techniques. This new interface
channel has the potential to introduce a large number of
new communication techniques in advanced HCI sys-
tems, and may be able to improve the interaction process
considerably (per esempio., translating imaginary movements to
virtual actions, improving levels of concentration, affect-
ing emotional states, eccetera.). So far the interaction process
has been based mostly on conventional methods, in that
computer users typically use physical interaction devices
to see, hear, act, sense haptic or olfactory stimuli, and in
some cases even talk to the system. The near-term goal
of BCI systems, as an extension to these conventional
systems (as opposed to a replacement, which is a longer-
term aspiration), would be to translate human thoughts
and emotions by direct connection to the human brain
and use this information as a new modality channel for
HCI systems (Nijholt, Plass-Oude, & Reuderink, 2009).
Turning briefly to the field of VR and the relevance of
issues of affect, to date, researchers have studied the
implementation of virtual realities in many different
domini. As well as entertainment, virtual realities and
1. Witness, Per esempio, the wide range of visual displays, dati
inputs, haptic, and other forms of devices available from ‘‘crowd-
funding’’ platforms, such as Kickstarter and Indiegogo.
2. https://en.wikipedia.org/wiki/The_Matrix
3. https://en.wikipedia.org/wiki/Pacific_Rim_(film)
l
D
o
w
N
o
UN
D
e
D
F
R
o
M
H
T
T
P
:
/
/
D
io
R
e
C
T
.
M
io
T
.
/
e
D
tu
P
v
UN
R
/
UN
R
T
io
C
e
–
P
D
l
F
/
/
/
/
/
2
5
2
8
1
1
8
3
6
5
7
9
P
R
e
S
_
UN
_
0
0
2
4
9
P
D
.
F
B
sì
G
tu
e
S
T
T
o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
Moghimi et al. 83
their so-called ‘‘serious games’’ counterparts have been
used for training purposes (Ahlberg et al., 2007; Zyda,
2005; Seymour et al., 2002), pain distraction (Mahrer &
Gold, 2009; Hoffman, Doctor, Patterson, Carrougher,
& Furness, 2000; Hoffman et al., 2004), rehabilitation
(Rizzo et al., 2002; Jack et al., 2001), and disorder ther-
apy (Parsons & Rizzo, 2008; Difede et al., 2007; Rizzo
et al., 2013; Kaganoff, Bordnick, & Carter, 2012). IL
focus of all of these studies has been to engage the
human users in an interactive virtual environment, E
to increase the sense of presence and immersion within
them, thereby effectively delivering new skills, knowl-
edge, or in some cases, acting as a form of clinical dis-
traction. In 2006, Joels et al. suggested that changes in
the excitement level (depending on pleasurable or dis-
pleasurable condition) affect the learning and memory
processi. They proposed that memory performance
i cambiamenti (either improvements or impairments) are
highly dependent on the time and context of the emo-
tional experience (Joels, Pu, Wiegert, Oitzl, & Krugers,
2006). Therefore, the recognition of the users’ emotions
when exposed to virtual realities, and controlling their
affective experiences within the virtual environments
(regardless of their purpose) can be as important as the
VR’s contextual outcome.
One of the subcategories of research into BCI systems
is described as affective computing. During the process of
affective computing, psychophysiological signals from
the users are recorded to enable the BCI system to
extract data of relevance to their emotional and cognitive
stati. This new input channel could provide several fea-
tures for an advanced HCI system attempting to support
the generation of believable immersive experiences. As
an illustration, the system could use this information to
adapt itself to the user’s emotions and, by doing so,
increase his/her performance and immersion levels dur-
ing the interaction process. Recentemente, new techniques in
HCI-mediated emotional recognition have been devel-
oped using noninteractive or passive environments, come
as listening to music, or the observation of videos and
imagery (Koelstra et al., 2012; Frantzidis, Bratsas, Papa-
delis, Konstantinidis, Pappas, & Bamidis, 2010; Yazdani,
Lee, & Ebrahimi, 2009; Rizon, Murugappan, Nagara-
jan, & Yaacob, 2008; Murugappan, Rizon, Nagarajan,
Yaacob, Hazry, & Zunaidi, 2008; Katsis, Katertsidis,
Ganiatsas, & Fotiadis, 2008; Takahashi & Tsukaguchi,
2003). Others are now beginning to focus on virtual
realities and more interactive environments (per esempio., Par-
nandi, Son, & Gutierrez-Osuna, 2013; Wu et al., 2010;
Antje, Peter, Markert, Meer, & Voskamp, 2005).
To perform the affect recognition process in virtual real-
ities, first the affective features of VR environments need
to be carefully investigated. Secondo, a system has to be
designed, trained, and validated with respect to a psycho-
physiological affective database, recorded from a large
number of users exposed to a number of controlled and
known affective stimuli (considering supervised learning
algorithms; Mohri, Rostamizadeh, & Talwalkar, 2012).
To construct such a database, a number of controlled
emotional situations (evoking some specific affective states
on the part of the users4), would need to be presented to
participants in an experiment, while taking part in a physi-
ological measurement paradigm. These recordings,
tagged by the corresponding affective states, would then
be analyzed for the design, training, and validation of the
affect recognition system. Therefore, two distinct steps in
the psychophysiological affective database construction
can be considered: (UN) evoking controlled emotional expe-
riences and (B) the measurement of physiological parame-
ters. It would be important to ensure the implementation
of strict experimental designs in such a paradigm, in order
to avoid the development of an inappropriate psychophy-
siological affective database, which would invalidate the
recognition system’s training process. As an illustration, if
the users’ emotional experiences were poorly controlled
(per esempio., it was not possible to state with confidence that an-
ger had been experienced by the users during the corre-
sponding session), then the classification techniques
would be unable to train the affect recognition system
properly and the accuracy of the system would be affected
accordingly. To prevent such incidents, the emotional
stimuli must be subjectively evaluated and categorized
prior to the undertaking of physiological measurements,
in order to validate their effectiveness in evoking the
required emotional experiences on the part of all users.
4. Such as images that evoke fear and disgust in users—image num-
ber 3000 A 3266 in IAPS (Lang, Bradley, & Cuthbert, 2008).
l
D
o
w
N
o
UN
D
e
D
F
R
o
M
H
T
T
P
:
/
/
D
io
R
e
C
T
.
M
io
T
.
/
e
D
tu
P
v
UN
R
/
UN
R
T
io
C
e
–
P
D
l
F
/
/
/
/
/
2
5
2
8
1
1
8
3
6
5
7
9
P
R
e
S
_
UN
_
0
0
2
4
9
P
D
.
F
B
sì
G
tu
e
S
T
T
o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
84 PRESENCE: VOLUME 25, NUMBER 2
To date, a number of evaluated affective stimuli data-
bases using images (the International Affective Picture
System—IAPS; Lang, Bradley, & Cuthbert, 2008),
sounds (the International Affective Digital Sounds—
IADS; Bradley & Lang, 1999), and video clips (Baveye,
Bettinelli, Dellandrea, Chen, & Chamaret, 2013) Avere
been presented in the literature. These established
databases provide investigators with a variety of pre-
evaluated affective stimuli, Quale (from a subjective
outcome perspective) have been found to elicit specific
(and quite strong) emotions in recipients. Tuttavia, A
the knowledge of the authors, no validated affective
VR-based stimuli database has been presented as yet.
The availability and reliability of such a database of stim-
uli in form of a virtual reality (VR) is crucial in the design
and validation of an affective computing system, Quale
can be used in VR-based systems.
In the present article, an affective virtual reality and the
process by which it was conceptualized, designed, E
subjected to an early validation study is discussed in detail.
The affective VR is capable of eliciting multiple emotions
within the users, and manipulating their affective experi-
ences within a 3-dimensional affective space (described
Dopo), by controlling the VR’s internal parameters. UN
number of ‘‘sub-games’’ (based on the selection of multi-
ple unique VR parameters) have been selected using an
affective power estimation process, and evaluated in a
subjective experiment employing 68 participants. IL
work described in this article represents a number of early
steps in research that is working toward a more compre-
hensive psychophysiological understanding of the future
role of brain–computer interfaces in VR and so-called se-
rious games. The affective VR described herein is sup-
posed to be used in construction of an affective physiologi-
cal database, to be used in the conceptualization, progetto,
and validation of an affect recognition system.
2 Model of Affects, Self-Assessment, E
Affective Clusters
2.1 Model of Affects
One of the most important challenges in the study
of emotions is the definition one adopts. Bradley, In
2006, stated that ‘‘part of the complexity in studying
emotion is defining it: there are almost as many defini-
tions as there are investigators’’ (Bradley & Lang, 2006).
The common factor among all of these definitions is that
of physiological effects, broadly reflecting the fact that,
in emotional situations, the body reacts and performs
accordingly. In high-tempo, high-pressure contexts, for
esempio, the heart rate changes, sweating occurs, IL
muscles tense, facial expressions such as smiling and
frowning appear, and many other less overt
physiological changes take place (Bradley & Lang,
2006). The term emotion has been presented by some
researchers in the form of either quantitative (dimen-
sional) or qualitative (categorico) models, often referred
to as affective space.
In qualitative models, the affective space is presented
by using an emotion set (a number of emotional labels),
such that the user can be ‘‘categorized’’ as experiencing
either one or a mixture of these emotional labels. As an
illustration, Ekman and Friesen (2003) used a qualitative
presentation of emotions, categorizing them as surprise,
fear, disgust, anger, happiness, and sadness. Researchers
have introduced several emotion sets, although there are
some common strong emotions that are present in most
of them. These strong emotions include anger, fear, dis-
gust, excitement, happiness, sadness, and boredom
(Bradley & Lang, 2006).
In contrast to the work by Bradley and Lang, both
Russell and Mehrabian presented two similar quantita-
tive models in the 1980s and 1970s. These models
define emotions based on two or three continuous inde-
pendent parameters (dimensions or axes) (Mehrabian,
1970; Russell, 1980). Mehrabian introduced three inde-
pendent quantities: Valence, defining pleasure and dis-
pleasure; Arousal, describing the excitation level; E
Dominance, identifying the level of control within a
given situation. Russell, on the other hand, ignored
Dominance, and created a 2-dimensional Circumplex of
Affect. Mehrabian and Russell believed that representa-
tion of verbal labels of emotions within either the 2D- O
3D-model would differ between people with different
cultures, especially those with different languages (Meh-
rabian, 1970; Russell, 1980). In 1980, Russell repre-
sented some the most common English verbal labels,
within his Circumplex of Affect (shown in Figure 1).
l
D
o
w
N
o
UN
D
e
D
F
R
o
M
H
T
T
P
:
/
/
D
io
R
e
C
T
.
M
io
T
.
/
e
D
tu
P
v
UN
R
/
UN
R
T
io
C
e
–
P
D
l
F
/
/
/
/
/
2
5
2
8
1
1
8
3
6
5
7
9
P
R
e
S
_
UN
_
0
0
2
4
9
P
D
.
F
B
sì
G
tu
e
S
T
T
o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
Moghimi et al. 85
(per esempio., high heart rate tempo means high arousal status),
and the emotional status of the users is evaluated accord-
ingly (Takahashi & Tsukaguchi, 2003). Therefore, COME
self-assessment has been employed by the majority of
research studies, and also due to the lack of availability of
a psychologist or human emotion expert in the present
study, it was decided to employ self-assessment techniques
in the emotional evaluation process.
2.3 Self-Assessment
In the present study, both qualitative and quantita-
tive affective spaces were employed within the experi-
ments when performing participants’ self-assessments.
The participants were asked to evaluate their emotional
experiences and self-report them in the 3-dimensional
affective space (as conceived by Mehrabian—scaled arbi-
trarily from (cid:2)3 to þ3), while each axis was defined and
presented to the participants as discussed next. To per-
form the emotional assessment in the 3D model of
affect, an interactive version of Bradley and Lang’s Self-
Assessment Manikin (SAM) questionnaire (Bradley &
Lang, 1994), was employed to enable participants to
self-report their Valence, Arousal, and Dominance levels.
1. Valence: How pleasurable this gaming experience
era. Higher positive values mean more pleasure
(per esempio., you enjoyed it), and higher negative values
mean more displeasure (per esempio., you did not enjoy it).
2. Arousal: How arousing this gaming experience
era. Higher positive values mean more aroused
(per esempio., excited, alert, stressful, eccetera.), and higher neg-
ative values mean minimally aroused (per esempio., relaxed,
tired, bored, eccetera.).
3. Dominance: How much control you had in this
gaming experience. Higher positive values mean
higher control in the game (per esempio., proper controller
risposta, ability to perform required maneuvers,
eccetera.), and higher negative values mean lower con-
trol during game-play (per esempio., inability in performing
required maneuvers, eccetera.).
The dimensional assessment was followed by a qualita-
tive eight-label assessment (labels: Relaxed, Content,
Happy, Excited, Angry, Afraid, Sad, and Bored). These
Figura 1. Simplified Russell Circumplex of Affect for English verbal
labels of emotions (Russell, 1980).
2.2 Emotional Experience Assessment
In the affective psychophysiological database con-
struction process, each session, in which participants are
exposed to an affective stimulus, has to be tagged by an
emotional experience state, within an affective space
(qualitatively or quantitatively or both). This assessment
has to be able to reliably categorize the participants’
emotional experience.
So far, researchers have, in the main, employed either
self or expert assessments. In expert assessments, a psy-
chologist or human emotion expert would be instructed
to evaluate the participant’s affective state, and catego-
rize it within an affective space (Katsis, Katertsidis,
Ganiatsas, & Fotiadis, 2008). Tuttavia, the majority of
studies appear to employ self-assessment techniques to
evaluate participants’ emotional states within an affective
spazio (Murugappan, Rizon, Nagarajan, Yaacob, Hazry,
& Zunaidi, 2008; Rizon, Murugappan, Nagarajan, &
Yaacob, 2008; Frantzidis, Bratsas, Papadelis, Konstanti-
nidis, Pappas, & Bamidis, 2010). In this process the user
is instructed to evaluate his/her affective state according
to a particular model (either qualitatively or quantita-
tively). D'altra parte, in some cases, a pre-
emotional hypothesis was presented prior to the experi-
ment. As an illustration, a certain physiological behavior
is considered as a result of a specific emotional status
l
D
o
w
N
o
UN
D
e
D
F
R
o
M
H
T
T
P
:
/
/
D
io
R
e
C
T
.
M
io
T
.
/
e
D
tu
P
v
UN
R
/
UN
R
T
io
C
e
–
P
D
l
F
/
/
/
/
/
2
5
2
8
1
1
8
3
6
5
7
9
P
R
e
S
_
UN
_
0
0
2
4
9
P
D
.
F
B
sì
G
tu
e
S
T
T
o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
86 PRESENCE: VOLUME 25, NUMBER 2
Figura 2. Presentation of 8 English verbal labels of emotions within the 3D affective model. Dots present the location of
each label within the 3D space. The dashed boxes present the affective clusters—the clusters’ names (PVLAPD, PVHPAPD,
NVPAND, and NVNAND) have been presented within the dashed boxes. The dashed vectors are the affective clusters’
centroid vectors.
eight labels were selected as they were assumed to be rel-
evant to most VR experiences, and equally distributed
along the multidimensional space (Guarda la figura 1).
2.4 Affective Clusters—Subjective
Experiment
2.4.1 Participants and Method. To subjectively
evaluate the position of the selected eight labels within
the Circumplex of Affect (Guarda la figura 1), a questionnaire
was designed and presented to all participants (103 In
total, with a mean age of 23.23 years, and a distribution
Di 52% male and 46% gamers), who partook in all experi-
menti (see Experiments 1 E 2 in Sections 4 E 5).
The purpose behind this experiment was to assess the
placement of the eight presented labels within the Rus-
sell’s Circumplex of Affect, within the gaming and VR
experience context.
The questionnaire contained eight questions, each of
which required the participants to locate one of the emo-
tional labels (Relaxed, Content, Happy, Excited, Angry,
Afraid, Sad, and Bored) within the 3-dimensional space.
The example given next presents one of the questions,
assessing the Relaxed label. The participants were asked
to choose one of the integer scalars, (arbitrarily) between
(cid:2)3 to þ3, for each parameter.
‘‘What value of these parameters (Valence, Arousal,
and Dominance) would describe the experience of ‘Being
Relaxed’ in virtual realities?’’
2.4.2 Results. Figura 2 presents the subjective
arrangement of these labels within the 3D (Valence,
Arousal, and Dominance) affective space. The mean rat-
ings across participants, within each axis, have been used
as the subjective position of the labels within the Cir-
cumplex. As can be seen in Figure 2, the labels follow
the Russell’s Circumplex order, while the position of
Relaxed and Content are associated with higher arousal
states than expected (compare Figure 1 and the ‘‘Valence
vs. Arousal’’ plot in Figure 2). It can be considered that
this reflects the fact that the ratings were undertaken in
l
D
o
w
N
o
UN
D
e
D
F
R
o
M
H
T
T
P
:
/
/
D
io
R
e
C
T
.
M
io
T
.
/
e
D
tu
P
v
UN
R
/
UN
R
T
io
C
e
–
P
D
l
F
/
/
/
/
/
2
5
2
8
1
1
8
3
6
5
7
9
P
R
e
S
_
UN
_
0
0
2
4
9
P
D
.
F
B
sì
G
tu
e
S
T
T
o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
Moghimi et al. 87
the context of the gaming and VR experience. It can be
expected that relaxing while playing a game can be more
arousing than simply relaxing on a sofa, Per esempio.
D'altra parte, there is a high correlation between
the Valence and Dominance ratings (R (960) ¼ 0.69,
P < 0.001—this correlation can be observed in all affec-
tive ratings in Experiments 1 and 2, described in Sections
4 and 5). This correlation makes the ‘‘Valence vs.
Arousal’’ and ‘‘Dominance vs. Arousal’’ graphs almost
identical (see Figure 2). Furthermore, it makes the ‘‘Neg-
ative Valence, Positive Dominance, and Positive/Negative
Arousal’’ and ‘‘Positive Valence, Negative Dominance,
and Positive/Negative Arousal’’ octants5 (four octants
out of eight) completely empty (containing no emotion
label), while the other four contain all eight affective
labels (see ‘‘Valence vs. Dominance’’ plot in Figure 2).
To the contrary, the cluster containing the Relaxed
and Content emotional labels is occupying the entire
‘‘Positive Valence, Negative Arousal, and Positive Domi-
nance’’ and part of the ‘‘Positive Valence, Positive
Arousal, and Positive Dominance’’ octants. Thus, it can
be concluded that the verbal labels are not separated
with respect to the octants; however, the separation is
based on four affective clusters, defined as follows:
1. PVLAPD Cluster: Positive Valence, Low Positive
Arousal, Positive Dominance (PVPPAPD), and
Positive Valence, Negative Arousal, Positive Domi-
nance (PVNAPD). Therefore this cluster can be
named: Positive Valence, Low Arousal, Positive
Dominance (PVLAPD)—Containing Relaxed and
Content labels:
1) 0 (cid:3) Valence (cid:3) 3
2) (cid:2)3 (cid:3) Arousal (cid:3) 1.16
3) 0 (cid:3) Dominance (cid:3) 3
2. PVHPAPD Cluster: Positive Valence, High Posi-
tive Arousal, Positive Dominance (PVHPAPD)—
Containing Happy and Excited labels:
1) 0 (cid:3) Valence (cid:3) 3
2) 1.16 (cid:3) Arousal (cid:3) 3
3) 0 (cid:3) Dominance (cid:3) 3
5. An octant is one of the eight divisions of a Euclidean three-
dimensional coordinate system, defined based on the signs of the
coordinates.
3. NVPAND Cluster: Negative Valence, Positive
Arousal, Negative Dominance (NVPAND)—
Containing Afraid and Angry labels:
1) (cid:2)3 (cid:3) Valence (cid:3) 0
2) 0 (cid:3) Arousal (cid:3) 3
3) (cid:2)3 (cid:3) Dominance (cid:3) 0
4. NVNAND Cluster: Negative Valence, Negative
Arousal, Negative Dominance (NVNAND)—
Containing Sad and Bored labels:
1) (cid:2)3 (cid:3) Valence (cid:3) 0
2) (cid:2)3 (cid:3) Arousal (cid:3) 0
3) (cid:2)3 (cid:3) Dominance (cid:3) 0
2.4.3 Discussion. To be able to design the affect
recognition system, an affective VR, capable of manipu-
lating the users’ emotions within the entire emotional
space, needs to be designed. Considering the previous
discussions, it can be concluded that an affective virtual
reality needs to be capable of manipulating users’ emo-
tions such that they gravitate toward all four affective
clusters.
3 Affective Virtual Reality
Emotional stimuli play a vital role in the design
and performance evaluation of any affective computing
system. To date, the majority of the researchers have
used images (Frantzidis, Bratsas, Papadelis, Konstantini-
dis, Pappas, & Bamidis, 2010), video clips (Rizon, Mur-
ugappan, Nagarajan, & Yaacob, 2008; Murugappan,
Rizon, Nagarajan, Yaacob, Hazry, & Zunaidi, 2008;
Yazdani, Lee, & Ebrahimi, 2009), music (Takahashi &
Tsukaguchi, 2003; Koelstra, et al., 2012) and, occasion-
ally, real-life scenarios (Katsis, Katertsidis, Ganiatsas, &
Fotiadis, 2008) to evoke emotional experiences on the
subjects. However, virtual realities, as a potentially
powerful affective medium, have been used only in a
small number of research studies as a means of present-
ing emotional stimuli (Parnandi, Son, & Gutierrez-
Osuna, 2013). The focus of this study was to design
and subjectively evaluate an affective VR, capable of
evoking multiple emotional experiences within the
user population.
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
p
v
a
r
/
a
r
t
i
c
e
-
p
d
l
f
/
/
/
/
/
2
5
2
8
1
1
8
3
6
5
7
9
p
r
e
s
_
a
_
0
0
2
4
9
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
88 PRESENCE: VOLUME 25, NUMBER 2
3.1 Affective VR Design
Two different approaches are possible for design-
ing an affective virtual reality, capable of evoking certain
emotions:
1. Multiple VRs: A number of entirely different VRs
can be designed to evoke different emotions. The
first advantage of this approach is that every VR
can be designed in a way that would have the maxi-
mum impact on the users, in order to evoke a par-
ticular emotion. The disadvantage of this method
is the variability between the environments. Differ-
ent environments may well result in different VR
experiences, which may in turn lead to too much
variability between the recorded data. This would
leave no ground truth for any comparison between
the independent situations. Also, each VR would
take the form of a new environment for the partici-
pants and could create an element of surprise in ev-
ery attempt. This issue may decrease or even
change the expected emotional experience on the
part of the participants.
2. Single VR: A single but well-constructed virtual
reality can be designed that is capable of evoking
different emotions by changing the simulated envi-
ronment’s internal properties. The first advantage
of this approach is the minimum variability
between the emotional experiences, as the back-
ground environment or scenario (or so-called Neu-
tral Scenario of VR—the overall theme, environ-
ment, and rules of the VR) for all experiences
would be the same, and changes in the parameters
of the VR and incidents could elicit different emo-
tions. Another advantage would be the minimum
element of surprise on the participants (compared
to the multiple-VRs approach). The overall VR
environment, interaction algorithm, and other
aspects related to the background scenario would
stay the same and allow participants to concentrate
on the affective parameters rather than the changes.
The disadvantage of this approach is that the effec-
tiveness of emotional experiences may be less influ-
ential than the first method. The reason for this is
that, in a multiple VR approach, one scenario can
be designed to evoke boredom and another to
elicit excitement, each in a very powerful way; while
in this approach there is only one VR scenario,
which should be capable of evoking all emotions in
an effective manner.
In human-centered experimentation, minimum vari-
ability between VR experiences is an extremely impor-
tant matter, as any acceptable analysis dealing with either
affects or physiological databases has to be based on
changes in emotional experiences, due to different envi-
ronments (between games), rather than different personal
experiences (between participants). Multiple VRs may
create different experiences among participants, rather
than a single VR with an overriding context, due to vari-
ability between environments. Therefore, in the present
project, the Single VR approach has been adopted as the
design approach for the virtual affective medium.
A Speedboat Simulation6 game (see Figure 3) was
designed for use as the background scenario of the affec-
tive VR. As the experimental cohort was anticipated to
comprise both gamers and nongamers, it was decided to
use a driving-based simulation with a simple directional
interface style (a speedboat scenario in this case), to
reduce the amount of prior gaming experience required
for participation (i.e., when compared to the skills and
experiences typically manifested by players of first-person
combat and strategic games). Also, the creation of an
environment with a very basic contextual setting in terms
of graphical elements drove the choice of a speedboat
simulation (as opposed to automobile driving, which
typically consists of complex urban representations).
Moreover, the dynamics of the environment would, it
was felt, provide a wide range of possible parameters and
variables that could be implemented and controlled in
the environment (described in more detail in Section
3.2).
In the neutral speedboat simulation environment, the
user is able to navigate a small boat, freely, within a
coastal virtual environment originally created for VR
healthcare research (Stone & Hannigan, 2014). By
6. This simulation can be viewed at: https://www.youtube.com
/watch?v=pqn-X1Z5AoM.
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
p
v
a
r
/
a
r
t
i
c
e
-
p
d
l
f
/
/
/
/
/
2
5
2
8
1
1
8
3
6
5
7
9
p
r
e
s
_
a
_
0
0
2
4
9
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
Moghimi et al. 89
Figure 3. Speedboat simulation environment.
manipulating the VR parameters (described in more
detail in Section 3.2), a number of different variations of
the neutral environment were created. In this study,
these variations have been called sub-games.
In the majority of sub-games, there are a number of
floating ‘‘ring buoys’’ on the water that the users can col-
lect to gain higher scores. In all sub-games, the users can
either finish the game by passing the finish line at a dis-
tance (from where the game is started), or continue
exploring for as long as they require (only in the sub-
games, which do not have any time limitations). Regard-
less of the time settings for each VR variation, no sub-
game is allowed to continue beyond 5 minutes. If the
participant spends longer than 5 minutes in a particular
sub-game, it terminates automatically. Depending on
the VR settings (Section 3.2), participants can interact
with the virtual environment using either a mouse or a
force-feedback joystick. The joystick is capable of dis-
playing vibration effects according to simulated ‘‘water
turbulence’’ and, in addition, left/right forces on the
grip, simulating simple ‘‘water resistance’’ effects, cre-
ated when the boat is turning. A Samsung 32-inch flat
LCD screen was used to present the VR scenes, together
with a Sennheiser RS-170 wireless headphone to play
the environmental sound effects.
3.2 Affective Parameters (‘‘Incidents’’)
In order to evoke multiple emotions in the partici-
pants, a number of controllable affective parameters need
to be identified and implemented within the VR. Manip-
ulation of these parameters would (it was hypothesized)
evoke different emotions within the participants. The
general nature of these parameters or ‘‘incidents’’ needs
to be studied prior to any identification or implementa-
tion within the environment.
3.2.1 Categorization of Incidents.
I. VR Aspects: For the purposes of this study, the pa-
rameters or incidents presented in the speedboat simula-
tion were categorized into four major aspects:
1. Visualization: Any aspect of the game related to
visual stimulation, including lighting, textures, fi-
delity, scale of the objects, realism of any action
(such as avatar animation), and physical behaviors.
2. Auditory: All features of the game that are related
to the auditory sense of the users, including the
background music, sounds of objects, voices of
avatars, and so on.
3. Interaction: Keyboard, mouse, joysticks, voice rec-
ognition systems, gestural translators, and so on,
all fall within the interaction category.
4. Narrative: Any aspect of the game (visual, audi-
tory, and interaction) that is presented to the users
in a meaningful or contextually relevant way
through a narrative or background scenario. This
aspect can influence the user’s perception and
change his or her experience quite dramatically. As
an illustration, even a game created using extraor-
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
p
v
a
r
/
a
r
t
i
c
e
-
p
d
l
f
/
/
/
/
/
2
5
2
8
1
1
8
3
6
5
7
9
p
r
e
s
_
a
_
0
0
2
4
9
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
90 PRESENCE: VOLUME 25, NUMBER 2
Table 1. Twenty-One Incidents Categorization According to VR and Timing Aspect
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
p
v
a
r
/
a
r
t
i
c
e
-
p
d
l
f
/
/
/
/
/
2
5
2
8
1
1
8
3
6
5
7
9
p
r
e
s
_
a
_
0
0
2
4
9
p
d
.
dinary visualizations, auditory, and interaction fac-
tors can have different influences on users simply
by the way the game’s narrative has been pre-
sented. If the background presents a science fiction
scenario, for example, the user may expect to expe-
rience extreme levels of action, a high tempo, even
fear. Yet, the same game presented with a real-life
scenario as its background narrative, perhaps one
that depicts a desert island or peaceful countryside
setting, can create a completely different set of
expectations and perceptions on the part of the
user.
short aggressive attack, a short screen vibration, etc.),
or throughout the whole duration of a game (Game-
Persistence, such as a time limitation, a change in the
input device control law, etc.).
3.2.2 Incidents Identification and
Assessment. According to the speedboat simulation’s
environmental capabilities, 21 possible incidents were
identified for implementation within the affective VR.
These incidents were categorized based on their presen-
tation timing, together with the VR aspects. Table 1
shows these incidents, clustered with respect to the VR
aspect and timing classifications.
II. Timing Aspects: Each incident in the game can be
presented to the users either as a single event in the game
(In-Game Discrete Event, such as a sudden sound, a
Different combinations of these incidents could create
different sub-games. Combination of elements within
the columns can create 1444 different sub-game combi-
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
Moghimi et al. 91
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
p
v
a
r
/
a
r
t
i
c
e
-
p
d
l
f
/
/
/
/
/
2
5
2
8
1
1
8
3
6
5
7
9
p
r
e
s
_
a
_
0
0
2
4
9
p
d
.
Figure 4. Incidents presentation examples within the Affective VR. A) Start line flag for time limitation scenarios. B) Ramps in the
virtual environment. C) Mine avoidance. D) Jumping over ships. E) Driving freely outside the ring buoys lane. F) Finding hidden ring
buoys inside the bushes, by using the radar. G) Torpedo avoidance. H) Splashing water to the flying ball. I) Inverse black and white
screen in the torpedo avoidance sub-scenario. J) Black and white screen in maze sub-scenario. K) Finish line flag to terminate the
game on demand. L) Score calculation at the game termination.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
Þ (cid:4) 2 C2ð
Þ (cid:4) 2 C6ð
Þ (cid:4) 2 C7ð
Þ (cid:4) 3 C5ð
nations (Ci means ithcolumn – 5 C1ð
(cid:4) 2 C4ð
As some of these combinations are not possible (e.g., no
time limitation while the timer is faulty, etc.), the total
number of possible combinations is, as a consequence,
reduced to 792 different sub-games.
Þ (cid:4) 2 C3ð
Þ
Þ ¼ 1444).
Þ (cid:4) 2 C8ð
3.3 Affective Virtual Reality
The speedboat VR is capable of generating all
required combinations of incidents (described in Section
3.2.2). Figure 4 presents some examples of possible sub-
game combinations. Each sub-game was allocated an
92 PRESENCE: VOLUME 25, NUMBER 2
8-digit code. Each code represented the index number
within each column of Table 1. As an illustration, code
‘‘21223111’’ would set up a sub-game environment
with the following settings:
fore, a high number of participants need to be
recruited, to enable each game to be played by ‘‘n’’
participants (approach II).
2. Subjective Approximation: In this method, the
Mine Avoidance þ Time Limitation þ
Faulty Timer þ Invisible Barrier þ
Joystick with Force Feedback þ
Normal Controller þ
Shaking and Blurring the Camera þ Color Screen
The experimenter generates a list of these codes for the
VR, in order to create an automated sequential (random-
ized) experiment. In addition, an interactive SAM (Bradley
& Lang, 1994) questionnaire (scaled between (cid:2)3 to þ3
in all axes), followed by the eight-emotions list described
earlier, was automatically presented to the user, at the end
of each sub-game (Section 2.3). The rating results, fol-
lowed by the sub-game information, were saved in a text
file during the run-time of the experiment, and could be
simply extracted after the experiment.
4
Pre-Experiment Survey (Experiment 1)
4.1 Sub-Games Affective Power
Approximation
As explained in Section 3.2.2, 792 different sub-
games can be constructed using the 21 incidents. Two
different approaches were available to test the emotional
effect of each sub-game:
1. Subjective Assessment: In this approach, all sub-
games need to be played at least once by one of the
participants. It is impossible to allow each partici-
pant to play all 792 sub-games, as no one individ-
ual would be able to play all of them without expe-
riencing extreme fatigue (even in multiple sessions,
over different days). Therefore, all 792 sub-games
can be divided into ‘‘m’’ subsets, each of which
contains a number of sub-games. Then each subset
can be played (I) either by ‘‘n’’ participants, or (II)
by only one participant. To be able to perform a
meaningful affective analysis, the affective power of
sub-games cannot be assessed by subjective assess-
ment of a single participant (approach I). There-
emotional effect of each incident, rather than each
sub-game, would be evaluated, using an approxi-
mation technique. This means that each participant
would estimate the possible emotional effect of
each incident (all 21 incidents considered),
described verbally (Section 4.2). Then, by employ-
ing the approximated emotional effects of all
incidents, and an estimation technique (Section
4.3), the overall affective power of each single
sub-game, containing a number of incidents, can
be approximated.
Due to the high number of possible sub-game combi-
nations, to reduce this number to include those affective
combinations which would be most likely to manipulate
the participants’ emotional status toward all four affec-
tive clusters (described in Section 2), the Subjective
Approximation approach was employed in this study.
4.2 Participants and Method
Subsequently, a subjective survey was designed and
presented online to 35 respondents (with a mean age of
24.72 years, and a distribution of 57% males and 66%
nongamers). The study was reviewed and approved by
the University of Birmingham’s Ethical Review Commit-
tee (Ethical Reference Number: ERN_13-1157). To dis-
tinguish gamers from nongamers, the following descrip-
tion was presented to the respondents as part of the
online survey to enable them to self-assess appropriately:
‘‘If you follow games in the market regularly and have
a lot of experience playing games on PC and consoles,
you are a gamer.’’
Within the survey, a brief overarching explanation of
the VR, followed by a short video of the environment (as
referenced earlier under footnote 6), was presented to
the respondents. Then, each incident was described in
text form, such as: ‘‘imagine that you need to drive the
boat through mines scattered on the water,’’ or
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
p
v
a
r
/
a
r
t
i
c
e
-
p
d
l
f
/
/
/
/
/
2
5
2
8
1
1
8
3
6
5
7
9
p
r
e
s
_
a
_
0
0
2
4
9
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
Moghimi et al. 93
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
p
v
a
r
/
a
r
t
i
c
e
-
p
d
l
f
/
/
/
/
/
2
5
2
8
1
1
8
3
6
5
7
9
p
r
e
s
_
a
_
0
0
2
4
9
p
d
.
Figure 5. Presentation of incidents within the 3D affective space.
‘‘imagine that the controller used to control the boat is
faulty and is not responding to your actions.’’ The
respondents were then required to estimate their Va-
lence, Arousal, and Dominance levels, and to choose one
of the eight emotion labels (as presented in Section 2),
for each incident, by considering themselves within the
described affective situation.
4.3 Results
Using the mean ratings (across participants) of the
respondents for each incident, the affective power of all
VR parameters have been approximated within the 3-
dimensional affective space and are shown in Figure 5.
1. Interacting and Additive Effect: Each individual
incident can have an effect on another incident if
they are both presented within the same sub-game.
This means that incidents can have additive effects
from each other. It also means that, if several inci-
dents are presented in a sub-game, the overall emo-
tional effect of that combination can be considered
as the summation of Circumplex values of all indi-
vidual incidents.
2. Background Game as the Neutral: The back-
ground scenario can be considered as neutral, with
(0, 0, 0) as its 3D emotional effect. This means that
all possible combinations would be evaluated with
respect to the background VR scenario.
To analyze and estimate the total emotional power of
Accepting these hypotheses, then, the affective power
each sub-game, an estimation algorithm was designed
based on the following two hypotheses.
of all 792 sub-games can be estimated by adding the
approximated 3D affective values of all incidents within
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
94 PRESENCE: VOLUME 25, NUMBER 2
Figure 6. Presentation of sub-games within the 3D affective space. Dots represent the sub-games. Circled dots
represent the 22 selected sub-games.
each combination. Figure 6 presents the positioning of
all 792 sub-games within the 3-dimensional affective
space. Furthermore, to estimate the Occurrence Probabil-
ity (OP) of each categorical label for each game in the
future subjective experiment, Equation 1 was employed
in the analysis. By using this equation, the probability in
which a particular label can be selected in a specific sub-
game is approximated.
Equation 1. Categorical Label Occurrence Probability
Estimation Formula
OP ¼
P ðOccurence Frequency of all
incidents in a subgameÞ
Number of Incidences within the subgame
ð
(cid:4) Number of Participants in the experiment
Þ
ð
Þ
5
Preliminary Subjective Experiment
(Experiment 2)
5.1 Sub-Games Selection for
Preliminary Experiment
After performing the affective power estimation
process on the sub-games, the postexposure subjective
affective response of the participants to a number of them
can be assessed. This subjective affective evaluation can,
first, assess the accuracy of the approximation process and
second, identify the subjective emotional power of a num-
ber of sub-games (rather than their estimated values), for
future affective experiments. Therefore, a number of sub-
games need to be selected to be presented to a number of
participants, for subjective affective evaluation.
For this experiment, it was decided to adopt an overall
Experiment Duration (ED) of less than 2 hours (in order
to minimize participant fatigue). Considering the maxi-
mum duration of each sub-game as 5 minutes (no sub-
game was permitted to take longer than 5 minutes, as
described in Section 3.1, although they usually lasted less
than this), and the training session (Section 0) duration
of between 5 and 15 minutes (average 10 minutes), the
maximum number of sub-games for the experiment (not
to exceed 2 hours experiment duration), was calculated
as 22 (Equation 2). Therefore the 22 most affective sub-
games (those which are most likely to evoke emotional
experience within the four affective clusters) have to be
identified (using their approximated values) to be pre-
sented in the experiment.
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
p
v
a
r
/
a
r
t
i
c
e
-
p
d
l
f
/
/
/
/
/
2
5
2
8
1
1
8
3
6
5
7
9
p
r
e
s
_
a
_
0
0
2
4
9
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
Moghimi et al. 95
Table 2. Seven-Dimensional Presentation of Games’ and Clusters’ Ideal Vectors
Valence
Arousal
Dominance
PVLAPD
Occurrence
Percentage
PVHPAPD
Occurrence
Percentage
NVPAND
Occurrence
Percentage
NVNAND
Occurrence
Percentage
Valence Mean
Arousal Mean
Value
Across All
Participants
Value
Across All
Participants
Dominance
Mean
Value
Across All
Participants
Fraction of
Fraction of
Fraction of
Fraction of
Participants
Who Chose
Either
Relaxed or
Content
Participants
Who Chose
Either
Happy or
Excited
Participants
Who Chose
Either
Angry or
Afraid
Participants
Who
Chose
Either Sad
or Bored
Table 3. Clusters’ Ideal Vectors
Cluster
Valence Arousal Dominance
PVLAPD
Occurrence
Percentage
PVHPAPD
Occurrence
Percentage
NVPAND
Occurrence
Percentage
NVNAND
Occurrence
Percentage
1.5
PVLAPD
PVHPAPD
1.5
NVPAND (cid:2)1.5
NVNAND (cid:2)1.5
(cid:2)0.85
2.14
1.5
(cid:2)1.5
1.5
1.5
(cid:2)1.5
(cid:2)1.5
100%
0%
0%
0%
0%
100%
0%
0%
0%
0%
100%
0%
0%
0%
0%
100%
Equation 2. Calculation of Maximum Required Number
of Sub-Games
ED ¼ n (cid:4) 5 minutes
Þ (cid:3) 2 hours
ð
Þ
Þ þ 10 minutes
ð
ð
n ¼ number of subgames
n (cid:3) 22
5.2 Most Affective Sub-Games Selection
In order to select the most affective combinations,
each sub-game was presented as a vector, in a 7-dimen-
sional space, presented in Table 2. The Valence, Arousal,
and Dominance values were calculated through the sub-
game affective power estimation algorithm (Section
4.3). In addition, the approximated Occurrence Probabil-
ity (OP) for each sub-game was used to create the sub-
games’ affective vectors (Section 4.3).
In an ideal situation, the most affective game within
each cluster would feature that cluster’s central values
for Valence, Arousal, and Dominance (ideally, in the
dimensional model—Section 2—the clusters’ centroids);
while all participants have chosen one of the two verbal
labels within that cluster (i.e., the probability of selecting
either of the cluster’s labels is 100%—ideally in the cate-
gorical model). Therefore the clusters’ ideal vectors
could be presented as shown in Table 3.
To select the most affective sub-games, the Cosine
Similarity algorithm (Pang-Ning Tan, 2005) was
employed to find the 4 most similar sub-game affective
vectors to the clusters’ ideal vectors in each affective clus-
ter. Therefore, in each cluster the 4 most powerful com-
binations were selected to consider the 16 most affective
sub-games, which can cover the entire 3D affects space
effectively. Furthermore, 5 additional test combinations
(added manually—those which have the maximum
standard deviation and minimum level of agreement
among participants), followed by the neutral scenario
(the sub-game with background scenario settings—
‘‘12111121’’ combination) were added to create the 22-
game experiment. Figure 6 presents the 22 selected sub-
games among 792 combinations, highlighted with
circles. Table 4 presents the arrangement of the incidents
within the 22 selected sub-games.
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
p
v
a
r
/
a
r
t
i
c
e
-
p
d
l
f
/
/
/
/
/
2
5
2
8
1
1
8
3
6
5
7
9
p
r
e
s
_
a
_
0
0
2
4
9
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
96 PRESENCE: VOLUME 25, NUMBER 2
Table 4. The 22 Selected Sub-Games’ Settings
#
1
2
3
4
5
6
7
8
Narrative
Main
Scenario
Free
Environment
Exploring
Free
Environment
Exploring
Free
Environment
Exploring
Free
Environment
Exploring
Free
Environment
Exploring
Free
Environment
Exploring
Free
Environment
Exploring
Free
Environment
Exploring
Interactive
Visualization
Time
Limitation Timer
Invisible
Barrier
Controller
Type
Faulty
Controller
Camera
Screen
Color
No Time
Limitation
Normal
Timer
No Invisible
Barrier
Mouse
Normal
Controller
No Shake or
Blurring
Color
Screen
No Time
Limitation
Normal
Timer
No Invisible
Barrier
Mouse
Normal
Controller
No Shake or
Blurring
Black &
white
No Time
Limitation
Normal
Timer
No Invisible
Barrier
Mouse
Faulty
Controller
No Shake or
Blurring
Black &
white
No Time
Limitation
Normal
Timer
No Invisible
Barrier
Mouse
Faulty
Controller
No Shake or
Blurring
Inverse
Black &
white
No Time
Limitation
Normal
Timer
No Invisible
Barrier
No Time
Limitation
Normal
Timer
No Invisible
Barrier
No Time
Limitation
Normal
Timer
Invisible
Barrier
Joystick
Without
Force
Feedback
Joystick
With Force
Feedback
Mouse
Normal
Controller
No Shake or
Blurring
Black &
white
Normal
Controller
No Shake or
Blurring
Black &
white
Normal
Controller
No Shake or
Blurring
Black &
white
No Time
Limitation
Normal
Timer
Invisible
Barrier
Mouse
Faulty
Controller
No Shake or
Blurring
Black &
white
9 Mine
Avoidance
Time
Limitation
Normal
Timer
No Invisible
Barrier
10 Mine
Avoidance
Time
Limitation
Faulty
Timer
No Invisible
Barrier
Joystick
With Force
Feedback
Joystick
With Force
Feedback
Normal
Controller
Normal
Controller
Shaking and
Blurring the
Camera
Shaking and
Blurring the
Camera
Color
Screen
Color
Screen
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
p
v
a
r
/
a
r
t
i
c
e
-
p
d
l
f
/
/
/
/
/
2
5
2
8
1
1
8
3
6
5
7
9
p
r
e
s
_
a
_
0
0
2
4
9
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
Moghimi et al. 97
Screen
Color
Inverse
Black &
white
Inverse
Black &
white
Color
Screen
Color
Screen
Inverse
Black &
white
Inverse
Black &
white
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
p
v
a
r
/
a
r
t
i
c
e
-
p
d
l
f
/
/
/
/
/
2
5
2
8
1
1
8
3
6
5
7
9
p
r
e
s
_
a
_
0
0
2
4
9
p
d
.
Table 4. (Continued)
Narrative
Interactive
Visualization
Main
Scenario
#
Time
Limitation Timer
Invisible
Barrier
Controller
Type
Faulty
Controller
Camera
11 Mine
Avoidance
Time
Limitation
Faulty
Timer
Invisible
Barrier
Mouse
Faulty
Controller
No Shake or
Blurring
12 Mine
Avoidance
Time
Limitation
Faulty
Timer
Invisible
Barrier
Mouse
Faulty
Controller
13 Torpedo
Avoidance
Time
Limitation
Normal
Timer
No Invisible
Barrier
14 Torpedo
Avoidance
Time
Limitation
Faulty
Timer
No Invisible
Barrier
15 Torpedo
Avoidance
Time
Limitation
Faulty
Timer
Invisible
Barrier
Joystick
With Force
Feedback
Joystick
With Force
Feedback
Mouse
Normal
Controller
Normal
Controller
Faulty
Controller
No Shake or
Blurring
Shaking and
Blurring the
Camera
Shaking and
Blurring the
Camera
Shaking and
Blurring the
Camera
16 Torpedo
Avoidance
Time
Limitation
Faulty
Timer
Invisible
Barrier
Mouse
Faulty
Controller
Shaking and
Blurring the
Camera
17 Shooting a
Flying Ball
Time
Limitation
Normal
Timer
No Invisible
Barrier
18 Shooting a
Flying Ball
Time
Limitation
Normal
Timer
No Invisible
Barrier
19 Shooting a
Flying Ball
Time
Limitation
Normal
Timer
No Invisible
Barrier
20 Shooting a
Flying Ball
Time
Limitation
Normal
Timer
No Invisible
Barrier
Joystick
Without
Force
Feedback
Joystick
Without
Force
Feedback
Joystick
With Force
Feedback
Joystick
With Force
Feedback
Normal
Controller
No Shake or
Blurring
Color
Screen
Faulty
Controller
No Shake or
Blurring
Color
Screen
Normal
Controller
No Shake or
Blurring
Color
Screen
Faulty
Controller
No Shake or
Blurring
Color
Screen
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
98 PRESENCE: VOLUME 25, NUMBER 2
Table 4. (Continued)
Narrative
Interactive
Visualization
Main
Scenario
#
Time
Limitation Timer
Invisible
Barrier
Controller
Type
Faulty
Controller
Camera
Screen
Color
21 Maze
No Time
Limitation
Normal
Timer
No Invisible
Barrier
22 Maze
No Time
Limitation
Normal
Timer
No Invisible
Barrier
Joystick
Without
Force
Feedback
Joystick
Without
Force
Feedback
Normal
Controller
No Shake or
Blurring
Black &
white
Normal
Controller
Shaking and
Blurring the
Camera
Black &
white
5.3 Participants and Method
5.4 Results
An experiment was conducted in which the 22
selected sub-games were presented to 68 participants
(with a mean age of 24.12 years). The study was reviewed
and approved by the University of Birmingham Ethical
Review Committee (Ethical Reference Number:
ERN_13-1157). The participants consisted of four differ-
ent groups: male gamers, female gamers, male nongamers,
and female nongamers (17 participants for each group—
the gaming experience was subjectively assessed by the
participants, according to the description presented in
Section 4.2). Each experiment commenced with a training
session (see Figure 7) to prepare the participants for every
possible incident within the games. The training intro-
duced the game environment to the participants and
served to reduce any element of surprise in the games.
The sessions were performed in a quiet room. All par-
ticipants were provided with a 32-inch Samsung LCD
display, a Microsoft Wireless Mouse 5000, a Logitech
Wingman 3D force feedback joystick, and a Sennheiser
RS-170 wireless headphone. On average, participants
spent 58 minutes playing the games, and 1 hour, 46
minutes to complete the entire experiment. Therefore,
on average, participants spent 48 minutes of the experi-
ment to complete the questionnaire, or to rest between
the sub-game sessions.
5.4.1 Raw Results. Table 6 presents the esti-
mated (through Experiment 1) and subjectively reported
(through Experiment 2) Valence, Arousal, and Domi-
nance levels for each sub-game. The estimated values are
calculated by adding the incident’s (VR parameter) affec-
tive values (according to ‘‘Interactive and Additive
Effect’’ presented in Section 4.3); therefore, the scaling
is different from the sub-games’ measured ratings, which
are scaled between (cid:2)3 and þ3. Table 5 presents the esti-
mated and reported OP of each categorical label in each
sub-game.
5.4.2 Estimated Versus Reported
Correlation. Figure 8 shows the scatter plot of the esti-
mated Valence, Arousal, and Dominance levels in the Pre-
Experiment (Experiment 1), versus the subjectively
reported levels in the Preliminary Experiment (Experi-
ment 2). A comparison of the estimated values shows a
high correlation factor (Pearson technique) with the
reported values across Valence, Arousal, and Dominance
axes (see Table 7). From this, one can conclude that not
only were the participants able to accurately estimate their
emotions for each incident, before the real experience
(using the dimensional affective space), but also that the
estimation algorithm (presented in Section 4) was suffi-
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
p
v
a
r
/
a
r
t
i
c
e
-
p
d
l
f
/
/
/
/
/
2
5
2
8
1
1
8
3
6
5
7
9
p
r
e
s
_
a
_
0
0
2
4
9
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
Moghimi et al. 99
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
Figure 7. Affective VR training session. A) Practicing the maneuvering procedures. B) Explaining the radar. C) Describing the dot
colors in radar and their definitions. D) Presenting the ‘‘ring buoys’’ and the scoring procedure.
Table 5. Estimated Versus Subjectively Reported Categorical Levels for the 22 Selected Sub-Games
#
1
2
3
4
5
6
7
8
9
Relaxed Content Happy
Excited
Angry
Afraid
Sad
Bored
Estimated Percentage
Reported Percentage
Estimated Percentage
Reported Percentage
Estimated Percentage
Reported Percentage
Estimated Percentage
Reported Percentage
Estimated Percentage
Reported Percentage
Estimated Percentage
Reported Percentage
Estimated Percentage
Reported Percentage
Estimated Percentage
Reported Percentage
Estimated Percentage
Reported Percentage
18.41%
23.53%
17.14%
27.94%
13.97%
7.35%
15.24%
10.29%
15.87%
23.53%
13.97%
11.76%
15.24%
10.29%
13.97%
0.00%
10.16%
1.47%
14.60%
36.76%
13.65%
20.59%
11.43%
13.24%
12.38%
7.35%
13.65%
23.53%
11.43%
16.18%
12.38%
17.65%
11.43%
7.35%
10.48%
13.24%
10.79%
17.65%
9.52%
27.94%
6.98%
10.29%
9.21%
2.94%
12.38%
19.12%
8.57%
27.94%
8.25%
13.24%
6.98%
8.82%
9.84%
25.00%
17.46%
14.71%
15.56%
1.47%
12.70%
5.88%
13.97%
7.35%
16.51%
5.88%
22.54%
20.59%
14.29%
8.82%
12.70%
2.94%
32.38%
45.59%
5.08%
0.00%
5.71%
2.94%
20.63%
27.94%
14.92%
26.47%
5.71%
0.00%
9.52%
2.94%
13.33%
19.12%
20.63%
39.71%
9.84%
4.41%
2.54%
1.47%
2.22%
0.00%
2.86%
1.47%
3.49%
1.47%
1.90%
0.00%
3.17%
1.47%
2.86%
2.94%
2.86%
1.47%
6.03%
2.94%
0.00%
0.00%
2.22%
0.00%
3.17%
4.41%
1.90%
8.82%
2.22%
0.00%
2.54%
1.47%
2.86%
4.41%
3.17%
5.88%
0.32%
1.47%
31.11%
5.88%
33.97%
16.18%
28.25%
27.94%
28.89%
35.29%
31.75%
25.00%
28.25%
16.18%
30.79%
20.59%
28.25%
32.35%
20.95%
1.47%
/
e
d
u
p
v
a
r
/
a
r
t
i
c
e
-
p
d
l
f
/
/
/
/
/
2
5
2
8
1
1
8
3
6
5
7
9
p
r
e
s
_
a
_
0
0
2
4
9
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
100 PRESENCE: VOLUME 25, NUMBER 2
Table 5. (Continued)
#
10
11
12
13
14
15
16
17
18
19
20
21
22
Relaxed Content Happy
Excited Angry
Afraid
Sad
Bored
Estimated Percentage
Reported Percentage
Estimated Percentage
Reported Percentage
Estimated Percentage
Reported Percentage
Estimated Percentage
Reported Percentage
Estimated Percentage
Reported Percentage
Estimated Percentage
Reported Percentage
Estimated Percentage
Reported Percentage
Estimated Percentage
Reported Percentage
Estimated Percentage
Reported Percentage
Estimated Percentage
Reported Percentage
Estimated Percentage
Reported Percentage
Estimated Percentage
Reported Percentage
Estimated Percentage
Reported Percentage
8.25%
5.88%
6.35%
5.88%
4.76%
1.47%
10.16%
0.00%
8.25%
0.00%
6.35%
0.00%
4.76%
1.47%
12.38%
2.94%
11.11%
0.00%
12.06%
5.88%
10.79%
0.00%
13.97%
4.41%
12.38%
4.41%
9.52%
11.76%
8.25%
7.35%
6.67%
4.41%
10.48%
4.41%
9.52%
14.71%
8.25%
7.35%
6.67%
2.94%
12.06%
25.00%
11.11%
10.29%
11.43%
13.24%
10.48%
11.76%
12.38%
8.82%
10.79%
7.35%
9.21%
14.71%
7.30%
8.82%
6.03%
4.41%
10.16%
14.71%
9.52%
10.29%
7.62%
2.94%
6.35%
1.47%
14.92%
35.29%
13.65%
10.29%
12.38%
32.35%
11.11%
7.35%
11.75%
11.76%
10.48%
2.94%
32.70%
29.41%
20.95%
7.35%
23.17%
4.41%
31.43%
69.12%
31.75%
44.12%
20.00%
8.82%
22.22%
14.71%
23.81%
19.12%
22.22%
16.18%
27.62%
33.82%
26.03%
22.06%
16.83%
4.41%
19.05%
2.94%
15.87%
22.06%
29.52%
41.18%
33.65%
60.29%
10.48%
1.47%
16.51%
19.12%
30.16%
64.71%
34.29%
52.94%
6.98%
5.88%
14.29%
42.65%
6.67%
8.82%
13.97%
44.12%
8.57%
20.59%
12.70%
27.94%
6.03%
2.94%
6.67%
5.88%
7.62%
5.88%
6.67%
1.47%
6.67%
1.47%
7.30%
2.94%
8.25%
5.88%
5.40%
0.00%
5.40%
1.47%
5.71%
0.00%
5.71%
1.47%
2.54%
2.94%
3.49%
1.47%
0.32%
2.94%
2.54%
7.35%
2.86%
5.88%
0.32%
4.41%
0.32%
2.94%
2.54%
2.94%
2.86%
8.82%
0.32%
2.94%
0.63%
5.88%
0.32%
1.47%
0.63%
5.88%
2.54%
11.76%
2.86%
4.41%
18.10%
4.41%
18.41%
14.71%
15.24%
11.76%
20.32%
0.00%
17.46%
4.41%
17.78%
8.82%
14.60%
8.82%
24.13%
7.35%
21.59%
10.29%
23.81%
4.41%
21.27%
5.88%
31.43%
32.35%
28.25%
45.59%
Table 6. Estimated Versus Subjectively Reported Dimensional Levels for the 22 Selected Sub-Games (SE Is the Standard Error)
Valence
Arousal
Num
Estimated
1
2
3
1.74
(SE ¼ 0.26)
0.2
(SE ¼ 0.26)
(cid:2)2.37
(SE ¼ 0.26)
Subjectively
Rated
1.53
(SE ¼ 0.12)
1.23
(SE ¼ 0.15)
(cid:2)0.57
(SE ¼ 0.18)
Estimated
(cid:2)1
(SE ¼ 0.26)
(cid:2)2.43
(SE ¼ 0.26)
(cid:2)0.91
(SE ¼ 0.26)
Subjectively
Rated
0.04
(SE ¼ 0.19)
(cid:2)0.26
(SE ¼ 0.21)
(cid:2)0.01
(SE ¼ 0.19)
Dominance
Estimated
8.54
(SE ¼ 0.22)
7.43
(SE ¼ 0.23)
2.8
(SE ¼ 0.24)
Subjectively
Rated
2.09
(SE ¼ 0.12)
2.64
(SE ¼ 0.09)
(cid:2)0.39
(SE ¼ 0.19)
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
p
v
a
r
/
a
r
t
i
c
e
-
p
d
l
f
/
/
/
/
/
2
5
2
8
1
1
8
3
6
5
7
9
p
r
e
s
_
a
_
0
0
2
4
9
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
Table 6. (Continued)
Valence
Arousal
Num
Estimated
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
(cid:2)1.49
(SE ¼ 0.26)
1.34
(SE ¼ 0.26)
2.11
(SE ¼ 0.25)
(cid:2)0.94
(SE ¼ 0.26)
(cid:2)2.37
(SE ¼ 0.26)
4.63
(SE ¼ 0.24)
4
(SE ¼ 0.25)
(cid:2)2.29
(SE ¼ 0.26)
(cid:2)2
(SE ¼ 0.27)
5
(SE ¼ 0.24)
4.37
(SE ¼ 0.25)
(cid:2)1.91
(SE ¼ 0.26)
(cid:2)1.63
(SE ¼ 0.27)
3.49
(SE ¼ 0.25)
2.06
(SE ¼ 0.25)
3.97
(SE ¼ 0.24)
2.54
(SE ¼ 0.24)
0.89
(SE ¼ 0.26)
1.17
(SE ¼ 0.26)
Subjectively
Rated
(cid:2)0.6
(SE ¼ 0.16)
0.91
(SE ¼ 0.17)
1.1
(SE ¼ 0.16)
0.23
(SE ¼ 0.18)
(cid:2)0.76
(SE ¼ 0.18)
1.43
(SE ¼ 0.16)
0.61
(SE ¼ 0.2)
(cid:2)1.15
(SE ¼ 0.18)
(cid:2)1.07
(SE ¼ 0.18)
1.85
(SE ¼ 0.15)
0.8
(SE ¼ 0.19)
(cid:2)1.28
(SE ¼ 0.2)
(cid:2)1.14
(SE ¼ 0.19)
1.31
(SE ¼ 0.15)
(cid:2)0.45
(SE ¼ 0.21)
1.54
(SE ¼ 0.16)
(cid:2)0.18
(SE ¼ 0.23)
(cid:2)0.53
(SE ¼ 0.23)
(cid:2)1.11
(SE ¼ 0.22)
Estimated
(cid:2)1.43
(SE ¼ 0.26)
(cid:2)1.03
(SE ¼ 0.26)
0.57
(SE ¼ 0.25)
(cid:2)1.4
(SE ¼ 0.26)
(cid:2)0.91
(SE ¼ 0.26)
5.4
(SE ¼ 0.24)
6.74
(SE ¼ 0.25)
4.34
(SE ¼ 0.26)
5.37
(SE ¼ 0.21)
5.54
(SE ¼ 0.24)
6.89
(SE ¼ 0.25)
4.49
(SE ¼ 0.26)
5.51
(SE ¼ 0.27)
3.6
(SE ¼ 0.25)
4.09
(SE ¼ 0.25)
4.17
(SE ¼ 0.24)
4.66
(SE ¼ 0.24)
0.09
(SE ¼ 0.26)
1.11
(SE ¼ 0.26)
Subjectively
Rated
(cid:2)0.15
(SE ¼ 0.19)
(cid:2)0.24
(SE ¼ 0.22)
0.76
(SE ¼ 0.2)
0.23
(SE ¼ 0.19)
0.22
(SE ¼ 0.19)
1.77
(SE ¼ 0.11)
1.47
(SE ¼ 0.14)
0.6
(SE ¼ 0.2)
0.88
(SE ¼ 0.17)
2.31
(SE ¼ 0.1)
1.76
(SE ¼ 0.14)
1.03
(SE ¼ 0.16)
1.39
(SE ¼ 0.16)
1.24
(SE ¼ 0.14)
1.08
(SE ¼ 0.19)
1.69
(SE ¼ 0.13)
1.39
(SE ¼ 0.15)
(cid:2)0.33
(SE ¼ 0.23)
(cid:2)0.48
(SE ¼ 0.23)
Moghimi et al. 101
Subjectively
Rated
(cid:2)0.54
(SE ¼ 0.2)
2.36
(SE ¼ 0.12)
1.61
(SE ¼ 0.16)
1.79
(SE ¼ 0.18)
(cid:2)0.67
(SE ¼ 0.2)
1.42
(SE ¼ 0.16)
0.7
(SE ¼ 0.21)
(cid:2)1.12
(SE ¼ 0.2)
(cid:2)0.96
(SE ¼ 0.18)
1.35
(SE ¼ 0.17)
1.09
(SE ¼ 0.18)
(cid:2)1.55
(SE ¼ 0.19)
(cid:2)1.53
(SE ¼ 0.17)
1.81
(SE ¼ 0.16)
(cid:2)1.14
(SE ¼ 0.19)
1.53
(SE ¼ 0.15)
(cid:2)1.04
(SE ¼ 0.18)
1.7
(SE ¼ 0.17)
1.27
(SE ¼ 0.22)
Dominance
Estimated
4.23
(SE ¼ 0.23)
8.2
(SE ¼ 0.22)
6.94
(SE ¼ 0.23)
5.26
(SE ¼ 0.23)
2.8
(SE ¼ 0.24)
7.71
(SE ¼ 0.23)
6.23
(SE ¼ 0.24)
0.23
(SE ¼ 0.25)
(cid:2)0.83
(SE ¼ 0.25)
7.63
(SE ¼ 0.23)
6.14
(SE ¼ 0.24)
0.14
(SE ¼ 0.25)
(cid:2)0.91
(SE ¼ 0.26)
8.34
(SE ¼ 0.22)
5.89
(SE ¼ 0.23)
8.14
(SE ¼ 0.23)
5.69
(SE ¼ 0.23)
7.29
(SE ¼ 0.23)
6.23
(SE ¼ 0.23)
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
p
v
a
r
/
a
r
t
i
c
e
-
p
d
l
f
/
/
/
/
/
2
5
2
8
1
1
8
3
6
5
7
9
p
r
e
s
_
a
_
0
0
2
4
9
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
102 PRESENCE: VOLUME 25, NUMBER 2
Figure 8. Estimation vs. reported dimensional correlation.
Table 7. Axis and Emotion Labels Correlation Report
Label
Correlation
Axis
Correlation
Relaxed
Content
Happy
Excited
Angry
Afraid
Sad
Bored
r (22) ¼ 0.702, P ¼ 0.0003
r (22) ¼ 0.724, P ¼ 0.0001
r (22) ¼ 0.536, P ¼ 0.0100
r (22) ¼ 0.838, P < 0.0001
r (22) ¼ 0.878, P < 0.0001
r (22) ¼ 0.566, P ¼ 0.0060
r (22) ¼ 0.371, P ¼ 0.0892
r (22) ¼ 0.595, P ¼ 0.0034
Valence
r (22) ¼ 0.774, P < 0.0001
Arousal
r (22) ¼ 0.867, P < 0.0001
Dominance
r (22) ¼ 0.837, P < 0.0001
ciently accurate to accumulate the overall affective power
of each sub-game (using the dimensional affective space).
Thus, it can be concluded that the presented dimensional
affective power estimation algorithm appears to be capa-
ble of predicting the sub-games’ affective power prior to
the participants’ actual subjective experience.
A comparison of the estimated versus reported cate-
gorical emotional labels (occurrence probability) is pre-
sented in Figure 9. As can be seen in the graphs, not only
were the correlation coefficients smaller (see Table 7)
compared to the dimensional model, but also they were
slightly less significant (see Table 7). One can conclude
that the dimensional affective power estimation process
was more accurate than the categorical approximation.
5.4.3 Affective VR Effectiveness. A multivariate
analysis of variance (MANOVA) showed that the differ-
ent VR-combinations were an extremely important factor
(PVR-Combinations < 0.001,7 Valence, Arousal, and Domi-
7. Calculated using Pillai’s Trace, Wilks’ Lambda, Hotelling’s Trace,
and Roy’s Largest Root algorithms.
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
p
v
a
r
/
a
r
t
i
c
e
-
p
d
l
f
/
/
/
/
/
2
5
2
8
1
1
8
3
6
5
7
9
p
r
e
s
_
a
_
0
0
2
4
9
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
Moghimi et al. 103
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
p
v
a
r
/
a
r
t
i
c
e
-
p
d
l
f
/
/
/
/
/
2
5
2
8
1
1
8
3
6
5
7
9
p
r
e
s
_
a
_
0
0
2
4
9
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
Figure 9. Estimation vs. reported categorical correlation.
nance are considered as dependent variables and differ-
ent sub-games as independent parameter) in creating dif-
ferent emotional experiences (i.e., significantly different
Valence, Arousal, and Dominance levels). From this one
can conclude that the single controllable affective envi-
ronment designed for this study has been able to manip-
ulate the participants’ emotions significantly, by control-
ling the VR parameters.
5.4.4 Groups Comparison. Further to the previ-
ous discussion, the same (MANOVA) test supported the
conclusion that both Gender (male vs. female) and Gam-
ing Experience (gamer vs. nongamer) are extremely im-
portant factors in the participants’ emotional experiences
(PGender < 0.001, PGaming-Experience < 0.0017—Valence,
Arousal, and Dominance are considered as dependent
variables and gender and gaming-experience as inde-
pendent parameters). Thus it can be concluded that the
emotional experiences of each group (male-gamer, male-
nongamer, female-gamer, and female-nongamer) are sig-
nificantly different from each other.
One of the most important challenges of designing
any affective psychophysiological database is the minimi-
zation of variability between participants (in each indi-
vidual sub-game), while maximizing the variability
between sub-games’ experiences. This is due to the fact
that, in any human-centered experiment, minimum vari-
ability between participants’ experiences in a single VR
session is an extremely important issue. Any acceptable
analysis, dealing with either affects or physiological data-
bases, should, intuitively, be based on changes in emo-
tional experiences, due to different environments, rather
than different personal experiences.
Therefore, to reveal the similarity level between all 4
participant groups (male-gamer, male-nongamer,
female-gamer, and female-nongamer), the Cosine Simi-
larity Algorithm (Pang-Ning Tan, 2005) was once again
employed. Table 8 presents the mean similarity compari-
son levels, across games, for the four groups, in order of
the similarity level. As can be seen, the male gamers,
male nongamers, and female gamers are the most similar
groups, compared to the female nongamers (according
104 PRESENCE: VOLUME 25, NUMBER 2
Table 8. Groups Similarity Comparison Table
Group Comparison
Mean Similarity Level Across Games
Male Gamer Vs. Male Non-Gamer
Male Gamer Vs. Female Gamer
Male Non-Gamer Vs. Female Gamer
Female Gamer Vs. Female Non-Gamer
Male Non-Gamer Vs. Female Non-Gamer
Male Gamer Vs. Female Non-Gamer
94.23% (61.27%)
93% (61.54%)
90.98% (62.07%)
78.48% (68.88%)
77.22% (69.29%)
74.47% (610.82%)
to higher average and lower standard deviation in simi-
larity levels, across the games). This means that in an
affective VR situation, the emotional experience of male
gamers, male nongamers, and female gamers are very
similar, particularly when compared to the female
nongamers.
6
Discussion
The analysis presented in this study shows a num-
ber of early evaluation results based on an affective vir-
tual reality scenario. The affective VR, based on a speed-
boat simulation, is capable of evoking multiple
controllable emotional experiences within the users,
through changes of its internal parameters. The analysis
has highlighted the ability of this affective VR to manip-
ulate the users’ emotional experiences effectively, within
the affective space (examined on both dimensional and
categorical models), by combining a variety of affective
incidents (VR parameters) within a number of sub-
games. Moreover, the results suggest that the affective
power approximation algorithm (also presented in this
study) has been able to evaluate the emotional effect of
each sub-game, fairly accurately, prior to the real experi-
ence of the users. The analysis concluded that the
approximation algorithm has been able to estimate the
emotional experience of the users more accurately in the
dimensional affective space when compared to the cate-
gorical model. In addition, the analysis highlighted the
fact that gender and gaming experience are significant
factors in experiencing different emotional experience on
the part of the users. This means that the male-gamers,
male-nongamers, female-gamers, and female-nongamers
would have significantly different emotional experiences
(compared to each other), if exposed to a similar affec-
tive stimulus. However, despite the different emotional
experiences, the similarity level comparison revealed that
male gamers, male nongamers, and female gamers have a
higher similarity level in their affective responses, with
least variability, when compared to the female non-
gamers.
7
Conclusions
The human–computer interface has become one of
the most important research topics in computer science
since the introduction of the first ‘‘computers’’ (calcula-
tors) in the 17th century. Today, highly complex real-
time computer-based systems and their interfaces with
human operators are undergoing an evolution on a hith-
erto unheard-of scale, in what has become a quest to
ensure that they become synergistic, even symbiotic with
their human users—transparent, usable, intuitive, sensi-
tive, and reactive. As a key part of this evolution, psycho-
physiological interfaces generally, and Brain–Computer
Interaction (BCI) techniques specifically have intro-
duced new dimensions to the human interaction process,
by the introduction of direct human-to-computer con-
nections. Enhancing this symbiosis is, today, both tech-
nically and ethically possible. From a VR perspective,
affective computing, as one the subclasses of BCI
research endeavors, could be exploited in an attempt to
measure users’ emotions and affective experiences, and,
by incorporating them within advanced VR systems, sup-
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
p
v
a
r
/
a
r
t
i
c
e
-
p
d
l
f
/
/
/
/
/
2
5
2
8
1
1
8
3
6
5
7
9
p
r
e
s
_
a
_
0
0
2
4
9
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
Moghimi et al. 105
port endeavors to enhance participants’ sense of
‘‘immersion’’ and engagement.
This article has focused on the conceptualization,
design, and early validation of an affective virtual reality,
based on a dynamic VR environment, capable of evoking
multiple emotional experiences on the part of end users.
The literature reviewed demonstrated that much atten-
tion has, in the recent past, focused on affective comput-
ing motivated by passive human attention to music,
static image or video-based stimuli, with less apparent in-
terest being shown in the pursuit of similar interests in
interactive, dynamic virtual environments. Moreover, it
was also discovered that the construction of any affective
computing and recognition system requires an affective
psychophysiological database, recorded using reliable
emotional stimuli when interacting with particular types
of affective media. By focusing on affect recognition in
virtual realities and artificial environments, a highly con-
trollable affective VR-based simulation has been
designed and evaluated through two early experiments,
to be further used in the construction of such a database.
Indeed, the aspiration of this ongoing project is to con-
struct a reliable affective psychophysiological database to
generate techniques that will ultimately support the
design of adaptive virtual environments, not only by
measuring human emotions stimulated by virtual envi-
ronments through physiological parameters, but also by
adapting the game’s contents and events accordingly,
with the ultimate aim of increasing the ‘‘immersion fac-
tor,’’ possibly even tailored to the needs and responses of
each individual user.
Elements of the research described in this article are
currently being considered for application in other
domains, such as the role of emotions in the exploitation
of VR or serious games within healthcare contexts—for
example, during cognitive restoration therapies (e.g.,
Stone, Small, Knight, Qian, & Shingari, 2014), or in
rehabilitation and distraction therapies (e.g., Small,
Stone, Pilsbury, Bowden, & Bion, 2015). In particular,
these healthcare projects are generating early findings,
based on issues with poor sustained participation on the
part of certain patients, suggesting that the methods
described herein may well be significant in future techni-
ques of early ‘‘affective screening’’ during recruitment.
References
Ahlberg, G., Enochsson, L., Gallagher, A. G., Hedman, L., Hog-
man, C., McClusky, D. A., et al. (2007). Proficiency-based vir-
tual reality training significantly reduces the error rate for resi-
dents during their first 10 laparoscopic cholecystectomies. The
American Journal of Surgery, 193(6), 797–804.
Antje, H., Peter, C., Markert, L., Meer, E. V., & Voskamp, J.
(2005). Emotion studies in HCI—A new approach. HCI
International Conference, 1. Las Vegas, Nevada.
Baveye, Y., Bettinelli, J.-N., Dellandrea, E., Chen, L., & Cha-
maret, C. (2013). A large video data base for computational
models of induced emotion. 2013 Humaine Association
Conference on Affective Computing and Intelligent Interac-
tion, ACII 2013, 13–18.
Bradley, M. M., & Lang, P. J. (2006). Emotion and motivation.
In J. T. Cacioppo, L. G. Tassinary, & G. G. Berntson (Eds.),
Handbook of psychophysiology (2nd ed.), pp. 581–607. New
York: Cambridge University Press.
Bradley, M. M., & Lang, P. J. (1994). Measuring emotion, the
self-assessment manikin and the semantic differential. Jour-
nal of Behavior Therapy and Experimental Psychiatry, 25(1),
49–59.
Bradley, M., & Lang, P. (1999). International affective digi-
tized sounds (IADS): Stimuli, instruction manual and affec-
tive ratings. Gainesville: The Center for Research in Psycho-
physiology, University of Florida.
Brown, E., & Cairns, P. (2004). A grounded investigation of
game immersion. CHI ’04 Extended Abstracts on Human
Factors in Computing Systems, 1297–1300.
Cairns, P., Cox, A., Berthouze, N., Dhoparee, S., & Jennett,
C. (2006). Quantifying the experience of immersion in
games. Proceedings of the CogSci 2006 Workshop: Cognitive
Science of Games and Gameplay.
Difede, J., Cukor, J., Jayasinghe, N., Patt, I., Jedel, S., Spiel-
man, L., et al. (2007). Virtual reality exposure therapy for
the treatment of posttraumatic stress disorder following Sep-
tember 11, 2001. The Journal of Clinical Psychiatry, 68(11),
1639–1647.
Ekman, P., & Friesen, W. V. (2003). Unmasking the face. Los
Altos: ISHK.
Frantzidis, C. A., Bratsas, C., Papadelis, C. L., Konstantinidis,
E., Pappas, C., & Bamidis, P. D. (2010). Toward emotion
aware computing: An integrated approach using multichan-
nel neurophysiological recordings and affective visual stimuli.
IEEE Transactions on Information Technology in Biomedi-
cine, 14(3), 589–597.
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
p
v
a
r
/
a
r
t
i
c
e
-
p
d
l
f
/
/
/
/
/
2
5
2
8
1
1
8
3
6
5
7
9
p
r
e
s
_
a
_
0
0
2
4
9
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
106 PRESENCE: VOLUME 25, NUMBER 2
Hoffman, H. G., Doctor, J. N., Patterson, D. R., Carrougher,
G. J., & Furness, T. A. (2000). Virtual reality as an adjunc-
tive pain control during burn wound care in adolescent
patients. Pain, 18(2), 305–309.
Hoffman, H. G., Sharar, S. R., Coda, B., Everett, J. J., Ciol,
M., Richards, T., et al. (2004). Manipulating presence influ-
ences the magnitude of virtual reality analgesia. Pain,
11(1–2), 162–168.
Jack, D., Boian, R., Merians, A. S., Tremaine, M., Burdea, G.
C., Adamovich, S. V., et al. (2001). Virtual reality-enhanced
stroke rehabilitation. IEEE Transactions on Neural Systems
and Rehabilitation Engineering, 9(3), 308–318.
Joels, M., Pu, Z., Wiegert, O., Oitzl, M. S., & Krugers, H. J.
(2006). Learning under stress: How does it work? Trends in
Cognitive Sciences, 10(4), 152–158.
Kaganoff, E., Bordnick, P. S., & Carter, B. (2012). Feasibility of
using virtual reality to assess nicotine cue reactivity during
treatment. Research on Social Work Practice, 22(2), 159–165.
Katsis, C. D., Katertsidis, N., Ganiatsas, G., & Fotiadis, D. I.
(2008). Toward emotion recognition in car-racing drivers: A
biosignal processing approach. IEEE Transactions on Systems,
Man, and Cybernetics—Part A: Systems and Humans, 38(3),
502–512.
Koelstra, S., Mu¨hl, C., Soleymani, M., Lee, J.-S., Yazdani, A.,
Ebrahimi, T., et al. (2012). DEAP: A database for emotion
analysis using physiological signals. IEEE Transactions on
Affective Computing, 3(1), 18–31.
Lang, P., Bradley, M., & Cuthbert, B. (2008). International
affective picture system (IAPS): Affective ratings of pictures
and instruction manual. Gainesville: University of Florida.
Mahrer, N. E., & Gold, J. I. (2009). The use of virtual reality
for pain control: A review. Current Pain and Headache
Reports, 13(2), 100–109.
Mehrabian, A. (1970). A semantic space for nonverbal
behaviour. Consulting and Clinical Psychology, 35(2), 248–
257.
Mohri, M., Rostamizadeh, A., & Talwalkar, A. (2012). Foun-
dations of machine learning. Cambridge, MA: MIT Press.
Murugappan, M., Rizon, M., Nagarajan, R., Yaacob, S., Hazry,
D., & Zunaidi, I. (2008). Time-frequency analysis of EEG sig-
nals for human emotion detection. Proceedings of the Inter-
national Federation of Medical & Biological Engineering
(IFMBE), 21, 262–265.
Nijholt, A., Plass-Oude, B. D., & Reuderink, B. (2009).
Turning shortcomings into challenges: Brain–computer
interfaces for games. Entertainment Computing 1, 1(2),
85–94.
Pang-Ning Tan, M. S. (2005). Introduction to data mining.
Reading, MA: Addison-Wesley.
Parnandi, A., Son, Y., & Gutierrez-Osuna, R. (2013). A con-
trol-theoretic approach to adaptive physiological games.
2013 Humaine Association Conference on Affective Comput-
ing and Intelligent Interaction, 7–12.
Parsons, T. D., & Rizzo, A. A. (2008). Affective outcomes of
virtual reality exposure therapy for anxiety and specific pho-
bias: A meta-analysis. Journal of Behavior Therapy and Exper-
imental Psychiatry, 39(3), 250–261.
Patrick, E., Cosgrove, D., Slavkovie, A., Rode, J. A., Verratti,
T., & Chiselko, G. (2000). Using a large projection screen
as an alternative to head-mounted displays for virtual envi-
ronments. CHI Letters, 2(1), 478–485.
Rizon, M., Murugappan, M., Nagarajan, R., & Yaacob, S.
(2008). Asymmetric ratio and FCM based salient channel
selection for human emotion detection using EEG. WSEAS
Transactions on Signal Processing, 4(10), 596–603.
Rizzo, A. A., Bowerly, T., Buckwalter, J. G., Schultheis, M.,
Matheis, R., Shahabi, C., et al. (2002). Virtual environments
for the assessment of attention and memory processes: The
virtual classroom and office. Virtual Reality, 3–12.
Rizzo, A., Buckwalter, J., Forbell, E., Reist, C., Difede, J.,
Rothbaum, B. O., et al. (2013). Virtual reality applications
to address the wounds of war. Psychiatric Annals, 43,
123–138.
Russell, J. A. (1980). A circumplex model of affect. Journal of
Personality and Social Psychology, 39(6), 1161–1178.
Seymour, N. E., Gallagher, A. G., Roman, S. A., O’Brien, M.
K., Bansal, V. K., Andersen, D. K., et al. (2002). Virtual real-
ity training improves operating room performance. Annals
of Surgery, 236(4), 458–464.
Small, C., Stone, R. J., Pilsbury, J., Bowden, M., & Bion, J.
(2015). Virtual restorative environment therapy as an
adjunct to pain control during burns dressing changes. Tri-
als, 16(1), 1–7.
Stone, R. J., & Hannigan, F. P. (2014). Applications of virtual
environments: An overview. In K. S. Hale & K. M. Stanney
(Eds.), Handbook of virtual environments: Design, implemen-
tation and applications (883–957). New York: CRC Press
(Taylor & Francis).
Stone, R. J., Small, C. L., Knight, J. F., Qian, C., & Shingari, V.
(2014). Virtual environments for healthcare restoration
and rehabilitation. Virtual and Augmented Reality in
Healthcare 1. Intelligent Systems Reference Library, 68, 497–
521.
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
p
v
a
r
/
a
r
t
i
c
e
-
p
d
l
f
/
/
/
/
/
2
5
2
8
1
1
8
3
6
5
7
9
p
r
e
s
_
a
_
0
0
2
4
9
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
Moghimi et al. 107
Takahashi, K., & Tsukaguchi, A. (2003). Remarks on emotion rec-
ognition from multi-modal bio-potential signals. Proceedings
of the IEEE International Conference on Systems, Man, and
Cybernetics, 2, 1654–1659.
Wu, D., Courtney, C. G., Lance, B. J., Narayanan, S. S.,
Dawson, M. E., Oie, K. S., et al. (2010). Optimal arousal
identification and classification for affective computing
using physiological signals: Virtual reality Stroop task.
IEEE Transactions on Affective Computing, 1(2),
109–118.
Yazdani, A., Lee, J.-S., & Ebrahimi, T. (2009). Implicit emo-
tional tagging of multimedia using EEG signals and brain
computer interface. Proceedings of the First SIGMM Workshop
on Social Media, 81–88.
Zyda, M. (2005). From visual simulation to virtual reality to
games. Computer, 38(9), 25–32.
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
/
e
d
u
p
v
a
r
/
a
r
t
i
c
e
-
p
d
l
f
/
/
/
/
/
2
5
2
8
1
1
8
3
6
5
7
9
p
r
e
s
_
a
_
0
0
2
4
9
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
