g e n e r a l a r t i c l e
Linking Science and Technology
with Arts and the Next Generation
The STEAM Imaging Experimental Artist
Residency, A Case Study
B i A N k A H o f m A N N
T
C
UN
R.
T
S
B
UN
The author describes designing an artist-in-residence project within a
research institute for applied science as a dialogue-oriented form of
science communication and education. The author collaborated with
a scientist, an artist, a software architect and a sound designer to
realize the STEAM Imaging pilot project as a conceptual framework for
fostering collaborative engagement among school students, scientists
and an artist. A constant through the project is the connection of tools
for computer-assisted medicine with teaching topics from science,
technologie, engineering and mathematics (STEM). The first artist in
résidence, Yen Tzu Chang, integrated programming for sound art,
creation of plaster models and discussion on ethical topics. The aim
was to foster engagement with and ownership of future technology.
WHy STEAm imAGiNG?
As Head of Communications focusing on science commu-
nication at Fraunhofer MEVIS, I mediate the procedures de-
veloped to transform digital, integrated precision medicine
through systematic computer support as well as the required
contributions of mathematics, physics and computer science.
The Institute’s research aims to enhance decision support and
information integration for physicians, supporting them in
recognizing severe diseases such as cancer more reliably, tai-
loring treatments to each patient and making therapeutic
success measurable. Applied research in medical technology
is affected in many ways by increasingly shorter innovation
cycles. The rise of digitalization, big data, artificial intelli-
gence and biotechnology demands a prompt discussion of
social implications. Big data and artificial intelligence, dans
particular, sharpened the shift in science toward economic
output—the focus has changed from the fundamental under-
standing of basic principles of nature to approximations and
predictions based on vast amounts of data and statistics. Ce
change in focus influences ideas about our future and pre-
Bianka Hofmann (producer, science and tech communicator, project developer),
Fraunhofer MEVIS, Am Fallturm 1, 28359 Bremen, Allemagne.
E-mail: bianka.hofmann@mevis.fraunhofer.de. ORCID: 0000-0003-1151-793X.
See https://direct.mit.edu/leon/issue/54/2 for supplemental files associated
with this issue.
fig. 1. The residency’s key visual, the STEAM-brain. A fusion of anatomical
and functional MR imaging shows the brain areas that are activated while
a person listens to music, although the steam comes from a locomotive.
(© Fraunhofer MEVIS)
cipitates rethinking the pure economization of science and
knowledge to observe the societal and ecological effects of
science and technology. Computer scientists are challenged
to integrate other disciplines, such as ethics, psychologie, donc-
ciology and art.
I create formats that let experts remain in their fields of
expertise and that involve the public at an early stage, tel que
the STEAM Imaging residency (figue. 1). This helps engender
discussion among scientists about their work to gain clar-
ville, sense and composure concerning their R&D. I have led
various projects disseminating and communicating, hence
demonstrating, the value of the Institute’s R&D [1], inspiring
nonexperts to engage in sci-tech topics, par exemple, through
interactive hands-on exhibits. Starting from the interactive
moving images of the exhibits, especially the 4D moving im-
ages of the heart integrated into the interactive exhibit Image
Homme [2], I further developed with my colleague Alexander
Köhn a craft of visual storytelling. We started to produce
short films in 2D, 3D [3] and immersive experiences in 360°
[4] by processing moving images based on real medical data
to provide wider receptions for the audience. I focus on
aesthetics to remove barriers that prevent interaction with
severe health topics, thereby creating an approach to STEM
©2021 ISAST
https://doi.org/10.1162/leon_a_01792
LEONARDO, Vol. 54, Non. 2, pp. 185–190, 2021 185
Téléchargé depuis http://direct.mit.edu/leon/article-pdf/54/2/185/1910839/leon_a_01792.pdf by guest on 07 Septembre 2023
whether we could integrate art within a STEM workshop,
she was surprised, then enthusiastic, while confessing that
she had no idea where it would lead. We received funding
from the Fraunhofer Gesellschaft’s young talents marketing
department.
On behalf of MEVIS, I mediated between the participating
scientists and the artist. On the artistic side, we depended on
cooperation with experienced partners to find, select and
mentor the artists. Subsequently, Fraunhofer MEVIS began
a fruitful partnership with Ars Electronica as part of the Eu-
ropean Digital Art and Science Network [6]. The artist in
residence was Taiwanese media artist Yen Tzu Chang. In the
initial phase of the residency, she worked closely with scien-
tists at the Institute for two weeks. The scientists provided
expertise in topics such as auditory guidance in the operat-
ing room, segmentation of the heart and machine learning.
In the second phase, the artist resided at the Ars Electronica
Future Lab for several weeks to finish her artwork, a per-
formance installation titled Whose Scalpel, which addresses
the future of the relationship between humans and machines
in surgery. The team hosted an open discussion on ethical
issues with coworkers. Yen Tzu Chang has experience in de-
veloping electronic instruments inspired by the methods and
approaches of both natural and computer science. Within
the residency, she aimed to expand her artistic toolbox. À
develop her artwork, she acquired an MRI scan of her body
(figue. 2) and started to learn MeVisLab. The artistic outcome,
titled Whose Scalpel [7] (figue. 3), is a fusion of an interactive
3D model of the artist’s heart and movies based on her medi-
cal images. It invites the question: If machines can reason
even better than humans, will we lose some of our abilities
and even stop believing in ourselves? The artwork’s concept
is based on the application of sound in medical technology,
coronary artery bypass surgery and machine learning. Le
underlying story is based on the idea that, in the future, ma-
chines will give advice to surgeons. Chang states that sound
that provides joy, amazement and fun. With these movies and
exhibits, we generated a massive outreach effort with hun-
dreds of thousands of visitors and reached target groups not
familiar with STEM. Hands-on workshops with the young
generations are another format of sci-com at the Institute.
We offer a wide range of courses led by scientists such as
mathematician Sabrina Haase. These professionals place an
abundance of trust in school students, connect them with
the research activities at the Institute and impart concrete
knowledge so that they can participate actively. The scientists
make the role and importance of STEM subjects for medi-
cine tangible and offer role models for girls. In return, le
scientists explain their fields of work to nonexperts and thus
reflect on their discourse with school students in a different
way than with their colleagues.
PERfoRmiNG A-i-R AT fRAuNHofER mEViS
I started exploring how a tech-oriented institution could
integrate the arts in our sci-com to foster socially respon-
sible technology engagement and lead informed public dis-
cussion and decision-making. Existing artist-in-residence
(a-i-r) programs included outlined institutionalized plans far
removed from the everyday realities of an industry- oriented
research institution. I needed a bottom-up approach in
which researchers and developers served as a driving force.
I found allies for whom such a project would offer new op-
portunities and value, from enhancing their own workshop
experience to having a chance to participate playfully in the
Institute’s sci-tech, following paths of curiosity or pushing
the boundaries of its usage. Together with my colleagues, je
designed a concept and defined the goals of the residency
and the roles and responsibilities within the new team. I fo-
cused on the Institute’s technology, such as our 3-tesla MRI
scanner and MeVisLab, a rapid-prototype software develop-
ment platform that allows processing of medical data and
building assistance tools for clinicians [5]. I integrated the In-
stitute’s values and culture, namely the
interdisciplinary orientation governed
by transparency and cooperation. Le
employees act in a flexible network and
are not bound to a hierarchy. Sharing
information is encouraged and ex-
pected at all levels, and the STEAM
team is actively aided by using a col-
laborative social platform. En outre,
I integrated established and successful
STEM courses for young researchers,
with which Fraunhofer MEVIS already
has a lot of experience, although this
is not the Institute’s primary mission.
During these events, the scientists teach
dedicated STEM topics, nurture talents,
promote and support intrinsic motiva-
tion, and practice scientific thinking.
In the field of medical technology, inter-
disciplinary work is normal, mais quand
I asked my colleague Sabrina Haase
fig. 2. Artist Yen Tzu Chang is scanned at the Institute’s MRI scanner to acquire medical data for the blood
flow visualizations and 3D model of her heart. (© Fraunhofer MEVIS)
186 Hofmann, Linking Science and Technology with Arts and the Next Generation
Téléchargé depuis http://direct.mit.edu/leon/article-pdf/54/2/185/1910839/leon_a_01792.pdf by guest on 07 Septembre 2023
fig. 3. Yen Tzu Chang’s installation performance
Whose Scalpel is based on a 3D-printed model of
the artist’s heart equipped with lights and electronic
sound generators. (© Fraunhofer ICT Group)
in the performance not only serves to provide an atmosphere
but has been essential in diagnoses such as stethoscope ex-
aminations. Auditory guidance systems for navigated liver
surgery inspired her to apply the concept of Whose Scalpel to
this residency program [8]. Chang transformed her heart, su-
persized and 3D-printed, into a musical instrument (figue. 4).
The enlarged model of her heart is designed for interaction
by plugging in audio cables to bridge connections. Chang
equipped the inner parts with electronic components such
as an Arduino system and LEDs. Coronary bypass surgery
enables blood to follow a new path, which Chang describes as
an approach similar to a modular synthesizer, which routes
a signal from one module to another to produce sound.
During the performance, the artist plugs in the cables and
triggers sound. The artistically imagined AI is a character
in the performance who speaks offstage. The AI is in charge
of critical medical tasks and gives suggestions and advice to
the performing artist-surgeon. The surgeon’s goal is to cure
the heart, which symbolizes human faith and conscious-
ness. Cependant, no machine learning technology or deep
learning networks are used within the artwork. En outre
to Me VisLab, Chang used Blender and a 3D tool to create
the model and Mind Mapper for some videos. Chang used
PureData for the multimedia performance and Audacity for
sound editing. The performance is neither easy nor catchy
for visitors, but it features a conciliatory finale in which the
heartbeat of the heart model on which the surgery is per-
formed matches that of the surgeon.
In public forums, the idea of an AI as an intelligent entity
taking over the role of doctors is stressed, but in the clinical
reality, the process of collaboration between computers and
doctors has been already rewritten. Workflows in diagnoses,
therapy planning and therapies are changing, and disciplines,
human-computer interaction and decision-making are merg-
ing. When does a surgery begin: At the moment the surgeon
applies the scalpel? When the intervention is planned? Quand
an operation on a specific patient is simulated in advance?
Such procedures are already the reality. En particulier, tedious
tasks such as automated laboratory and image analysis will be
more often performed by the computer, which will present the
analyzed results to the clinicians for diagnosis. Predecisions
are being made without the clinicians. En outre, machine
fig. 4. Yen Tzu Chang performs Whose Scalpel at the Fraunhofer Society’s event The Art of Complexity in Berlin, 5 Juin 2018.
During the performance, Chang acts as a “surgeon” and places a bypass with the help of cables. (© Fraunhofer ICT Group)
Téléchargé depuis http://direct.mit.edu/leon/article-pdf/54/2/185/1910839/leon_a_01792.pdf by guest on 07 Septembre 2023
Hofmann, Linking Science and Technology with Arts and the Next Generation 187
perception to generate meaningful content. In addition to
gaining knowledge about anatomy, medical imaging, sound
art, plastering and soldering, the young students learned to
work with STEAM creatively and with self-empowerment.
Cependant, the evaluation did not show whether the workshop
reached school students with no prior interest in STEM. Nous
will address this issue in the future through closer coopera-
tion with schools and teachers.
ouTComE
STEAM Imaging was an experiment in dialogue-oriented
sci-com to realize a space for possibilities not predefined.
The players and content were selected, but no one knew what
the results would look like. What mattered were collabora-
tive exploring, learning in personal encounters, relating to
each other and openness to what could emerge. Cependant,
there is room for improvement. It was challenging to boost
creativity in the office environment to reach beyond work-
ing at a computer. It was difficult to open and engage the
actors in the process within the daily goal-oriented routine
of applied R&D. The team would have enjoyed having more
time for face-to-face encounters to allow playfulness, ambi-
guity and collaboration within the transdisciplinary process.
We believe this could have increased serendipitous benefits.
STEAM Imaging was an exploration of a future transdisci-
plinary educational model to strengthen the bond between
R.&D and the coming generation’s engagement with upcom-
ing technology. Applied research organizations can host
such intergenerational projects to confront the impact of
technologies at an early stage. If society truly aims to foster
the talents and resources of every child, the cooperation of
different social actors (not only teachers) is necessary within
flexible permanent learning environments. In Germany, un
child’s education level still largely reflects that of their par-
ents. It is essential to expand research teams, not only using
art, ethics and sociology, but to include, Par exemple, femmes
and people of color.
STEAM Imaging served as an incubator for all participants
to act without being measured. It allowed participants to learn
specific knowledge and to deal creatively with medical imag-
ing. En même temps, students were granted space to work
autonomously. The influence of role models for young people
cannot be stressed enough. The aim of such project-oriented
learning is not to impart a canon of knowledge but to allow
students to conduct an exemplary and deep examination of
a topic and learn how to autonomously develop new topics
and handle real-world technologies. The working methods
at Fraunhofer MEVIS range from planned and structured
to iterative and creative. The approach partially resembles
the artist’s methods, although the spectrum covered by her
solutions was much broader. This dialogue-oriented sci-com
project allows scientists to contextualize their expertise and
to create a mutual process: The scientists learn to relate their
developments to social realities and ideas, opinions and val-
ues of other actors and to reflect more upon their actions
than is often possible during everyday work. These relations
challenge the researchers at a personal level because they
fig. 5. Students work at the STEAM workshop under the guidance of
mathematician Sabrina Haase. (© Martin Hieslmair/Ars Electronica)
learning (concrete deep learning convolutional neural net-
travaux) is already capable, par exemple, of outperforming ex-
tensively trained dermatologists in detecting melanomas [9].
THE SCHooL STudENTS’ STEAm WoRkSHoP
A key element of the artist’s residency was the two-day
hands-on workshop [10] with school students that Chang
hosted with Haase, an experienced workshop leader (figue.
5). The roles of teacher and learner, giver and taker changed
dynamically between scientists and artist. The STEAM work-
shop introduced the school students—22 girls and 20 boys
in grades 7 to 9—to a problem-centered approach to learn-
ing. The fusion of theoretical contributions and explorative
teaching activities was essential in creating a holistic experi-
ence [11]. The courses in Bremen and Linz were somewhat
different. In Linz, no MRI scanner was available, although we
held a Skype call with the laboratory, and Haase and Chang
worked with one large group instead of dividing the crowd.
The feedback from the school students in Bremen was also
considered when preparing for the workshop in Linz. All
students created artistic images and short video sequences
combined with ambient noise, sounds of the MRI, or sound
recorded and processed with microphones soldered by and
sound software programmed by the students. They discussed
ethical questions and the impact of new technologies in
health care, such as patient data and privacy issues. Ils
held audiovisual performances to showcase their achieve-
ments. An external evaluator concluded in her report [12]
that the workshop allowed art and STEM subjects to be in-
terleaved. Especially in Bremen, students nevertheless felt
a division, which was also visible at the beginning with the
class in Linz. Cependant, this ended up blurring borders and
creating unexpected outcomes. The exploratory setting of
the workshop demonstrated to students that problems and
research situations are addressed and answered by combin-
ing knowledge from different subjects. Students were also
introduced to basic hardware and software technologies and
became familiar with how sound and visuals affect human
188 Hofmann, Linking Science and Technology with Arts and the Next Generation
Téléchargé depuis http://direct.mit.edu/leon/article-pdf/54/2/185/1910839/leon_a_01792.pdf by guest on 07 Septembre 2023
cannot refer to or rely on defined procedures and processes.
They also expose themselves to questions that they may not
be able to answer. An example of this is the ethical discus-
sion that took place within the framework of the residency.
The scientists pondered their responsibilities as individuals
and as part of an organization in technological development.
How do they behave and position themselves as employees
and as an organization? There are informal ethical discus-
sions of issues relevant to our topics, but the artist’s ques-
tions and artwork provided a point of crystallization to come
together in a group and reflect upon the possibilities, side
effects and risks of the Institute’s R&D [13].
A fundamental understanding of technology must al-
ways be a part of education. Cependant, because of the rate
of development, this can no longer be achieved solely by the
working generation. Intergenerational dialogue, especially
in education, is needed. Aujourd'hui, the ability to learn, changement,
adapt and be flexible and open to new skills and knowledge
are crucial for employees, companies and organizations. Comme
innovation cycles in R&D become shorter and more com-
plex, the ability to move fluidly between disciplines is an
essential competence. In the current educational system,
test- and goal- oriented development of knowledge has lim-
ited the discovery, support and, most significantly, expres-
sion of talent. Students focus on external goals instead of
experiences and growth of their abilities. The revolution in
communication technology in the last decade has proven to
be a social revolution and has contributed to this shift. Self-
restricted thinking, depending on the judgment of others,
leads to limited innovation. Pilot projects such as STEAM
Imaging experimentally explore how spaces, both physical
and metaphorical, could be conceived and established. Tel
new formats locate experts in their field of expertise, pave
the artists’ ways toward new technologies, open further pro-
fessional areas and, in the case of STEAM Imaging, include
teaching school students. The STEAM Imaging residency
and students’ workshops were hands-on experiences to help
exit the trodden paths of teaching and learning STEM. It was
an experiment in letting knowledge flow between genera-
tion. The residency served as an equalizer. The strategies of
the artist influenced the researchers’ usual approaches and
allowed them to develop and conduct a collaborative STEAM
workshop.
Yen Tzu Chang remarked that STEAM projects like this
one are an excellent opportunity for young people to be-
come acquainted with art and assess its significance. Ce,
her first STEAM workshop, increased her understanding of
how to lead a group and what school students require in
such seminars. This sparked her development of further
approaches to mediating educational encounters with tech-
nology and art. She has continued working on projects in a
social services agency to help establish hands-on workshops
and create interactive devices for children and teenagers [14].
Additional results inspired by the residency include the sci-
entists’ awareness of existing intersections with the arts. Ce
inspired the creation of medical image video performances
with school students, not only as a gimmick but as a result
in its own right, as well as the “Inside Insight” Web app. Le
software used in the students’ workshops, a light version of
the MeVisLab platform, has been developed over the years
and transferred into a Web application by software developer
Alexander Köhn. The team integrated the training materi-
als with artist Hannah Klatt and augmented them with an
explanatory voiceover. Using Inside Insight, Sabrina Haase
is now combining science and art in an online workshop to
teach medical image processing methods.
STEAM workshops could be part of scientific project dis-
semination strategies. With this project, I encourage other
industry-related institutions and companies to implement
suitable formats of artists’ residencies. These should be tai-
lored to their technologies and expertise, values and culture.
Such formats should ease making contact with other societal
actors. This could impact promotion of young talent (tel
as in STEAM Imaging). Companies must envision our next
generation no longer as solely consumers but also as future
citizens and employees. The impact can also manifest itself in
innovative processes, such as those targeted in the STARTS
calls [15] or in the development of human resources to im-
prove managers’ sensitivity for societal trends. Sound STEM
education is indispensable for becoming engaged with, creat-
ing and owning societally beneficial technologies. Involving
artists in STEM education and development of technologies
not only poses further questions for researchers and en-
hances aesthetic dimensions and design but encourages all
participants to engage creatively with STEM knowledge in a
process of a transdisciplinary invention.
Remerciements
I thank the curious and courageous colleagues from Fraunhofer MEVIS
R.&D—doers, makers, contributors—who made these explorations and
projects possible. Many thanks to David Black and Viktoriya Viktorova
for editing my English manuscript.
References and Notes
screen showing a beating heart. By touching, visitors can interac-
tively navigate through the results of different imaging methods. UN
3D-CT image serves as a spatial illustration of the heart. A sequence
of blood-flow MRI reveals the blood streaming through the ves-
sels and swirling turbulently. See a video of the Image Man moving
heart: https://youtu.be/n4kIqNFsEEw. See the 3 Juin 2015 press re-
lease: www.mevis.fraunhofer.de/en/press-and-scicom/press-release
/an-_image-man_-you-can-touch.html.
1 See examples: www.mevis.fraunhofer.de/en/press-and-scicom
/science-communication.html.
2
Image Man interactively showcases the capabilities of modern medi-
cal imaging methods. At the center of the installation is a touch
3 Encouraged by experiences with Yen Tzu Chang, we cooperated with
artists Ina Conradi and Mark Chavez to create a short movie titled
Digital Medicine, Arts, and STEAM: BEFORE US LIES ETERNERDY,
which won the Industry Award for Best Infographic at the Raw Sci-
ence Film Festival, 2019, in California, U.S.A. It offers an artistic,
Téléchargé depuis http://direct.mit.edu/leon/article-pdf/54/2/185/1910839/leon_a_01792.pdf by guest on 07 Septembre 2023
Hofmann, Linking Science and Technology with Arts and the Next Generation 189
large-scale immersive experience and was produced in 2D for the
15-×-2-m Media Art Nexus video wall at NTU, Singapore and in
stereoscopic 3D for the Deep Space 8K, Linz, Austria. It premiered in
Septembre 2018 at the Ars Electronica Festival. See a 16:9 2D preview
video with a voice-over of the stereoscopic 3D version: https://youtu
.be/oMJdszJ8PxQ.
4 For the short movie Beauty of Blood Flow Analysis, we received the
Industry Award for Best Immersive Media at the 4th Raw Science
Film Festival: The Bridge between Science and Media, held in Janu-
et 2018 in Santa Barbara, Californie, U.S.A. It was nominated at the 12th
FullDome Festival in Jena, Allemagne, May 2018. See a blog post and
a 2D preview of the 360° video at www.rawscience.tv/the-beauty-of
-blood-flow-analysis.
5 MeVisLab home page: www.mevislab.de.
6 Ars Electronica Center Linz, The Practice of Art & Science: The Euro-
pean Digital Art and Science Network (Hatje Cantz, 2017) pp. 50–55.
7 Y.T. Chang, sound sculpture, master thesis on stage, 2017, Linz.
Chang presented Whose Scalpel for the first time at the Ars Elec-
tronica Festival in Linz, Austria, in September 2017, and addition-
ally in April, June and August 2018 in Munich, Berlin and London,
respectivement.
8 C. Hansen et al., “Auditory Support for Resection Guidance in Navi-
gated Liver Surgery,” International Journal of Medical Robotics and
Computer Assisted Surgery 9, Non. 1, 36–43 (Mars 2013). See video:
www.youtube.com/watch?v=gCg5nJSI2pY.
9 H.A. Haenssle, “Man against Machine: Diagnostic Performance of
a Deep Learning Convolutional Neural Network for Dermoscopic
Melanoma Recognition in Comparison to 58 Dermatologists,” An-
nals of Oncology 29, Non. 8, 1836–1842 (Août 2018).
10 Find more details in B. Hofmann, S. Haase, D. Noir, “STEAM Im-
aging: A Pupils’ Workshop Experiment in Computer Science, Phys-
ics, and Sound Art,” SciArt Magazine 26, Special Topics: STEAM
(Août 2017): www.sciartmagazine.com/steam-imaging-a-pupils
rsquo-workshop-experiment-in-computer-science-physics-and
-sound-art.html (accessed 2 Avril 2018).
11 For the Evaluation Summary, see C. Schnugg, “STEAM Imaging:
Art Meets Medical Research: Evaluation Summary” (2017): www
.researchgate.net/publication/318572754_STEAM_Imaging_Art
_Meets_Medical_Research_Evaluation_Summary.
12 C. Schnugg, “Art Meets Medical Research, Evaluation” (2017), et-
published manuscript, accessible upon request.
13 We achieved funding (by the Federal Ministry of Education and Re-
recherche) for a research project jointly with ethicists, ELSA–AID: www
.gesundheitsforschung-bmbf.de/de/elsa-aid-kunstliche-intelligenz
-in-der-diagnostik-ethische-berufliche-und-soziale-aspekte-11030
.php.
14 Y.T. Chang, email communication with the author (25 Février
2018).
15 European Commission, “ICT and Art—the STARTS Initiative”:
www.ec.europa.eu/digital-single-market/en/ict-art-starts-platform.
Manuscript received 24 Août 2018.
Bianka Hofmann studied Comparative Religion and Biol-
ogy at the University of Bremen, focusing on communication
in highly evolved mammals, interspecies communication and
neurobiology of teaching and learning, writing her interdisci-
plinary Diploma thesis on ritualization and conducting field
studies on pilot whales in the Atlantic off the Canary Islands.
Trained as a mental coach, she has also run her own companies
and currently works in communications at the intersection of
R.&D and industry. Drawing on her studies’ practical impli-
cations, she specializes in sharing knowledge via interactive
experiences integrating science, art and humanities.
190 Hofmann, Linking Science and Technology with Arts and the Next Generation
Téléchargé depuis http://direct.mit.edu/leon/article-pdf/54/2/185/1910839/leon_a_01792.pdf by guest on 07 Septembre 2023
