RESEARCH ARTICLE - Specialized Research AI at MIT

RESEARCH ARTICLE

From indicators to indicating interdisciplinarity:
A participatory mapping methodology for
research communities in-the-making

a n o p e n a c c e s s

j o u r n a l

1Centre for Interdisciplinary Methodologies, University of Warwick, Coventry CV4 7AL, United Kingdom
2Centre for Science and Technology Studies, Leiden University

Noortje Marres1

and Sarah de Rijcke2

Citation: Marres, N., & de Rijcke, S.
(2020). From indicators to indicating
interdisciplinarity: A participatory
mapping methodology for research
communities in-the-making.
Quantitative Science Studies, 1(3),
1041–1055. https://doi.org/10.1162/
qss_a_00062

DOI:
https://doi.org/10.1162/qss_a_00062

Corresponding Author:
Noortje Marres
n.marres@warwick.ac.uk

Handling Editors:
Loet Leydesdorff, Ismael Rafols,
and Staša Milojevic´

Copyright: © 2020 Noortje Marres and
Sarah de Rijcke. Published under a
Creative Commons Attribution 4.0
International (CC BY 4.0) license.

The MIT Press

Keywords: digital cartography, interdisciplinarity, inventive methods, participatory methods,
Science and Technology Indicators, Science and Technology Studies

ABSTRACT
This article discusses a project under development called “Inventing Indicators of
Interdisciplinarity,” as an example of work in methodology development that combines
quantitative methods with interpretative approaches in social and cultural research. Key to our
project is the idea that Science and Technology Indicators not only have representative
value, enabling empirical insight into fields of research and innovation but simultaneously
have organizing capacity, as their deployment enables the curation of communities of
interpretation. We begin with a discussion of concepts and methods for the analysis of
interdisciplinarity in Science and Technology Studies (STS) and scientometrics, stressing that
both fields recognize that interdisciplinarity is contested. To make possible a constructive
exploration of interdisciplinarity as a contested—and transformative—phenomenon, we
sketch out a methodological framework for the development and deployment of “engaging
indicators.” We characterize this methodology of indicating as participatory, abductive,
interactive, and informed by design, and emphasize that the method is inherently
combinatory, as it brings together approaches from scientometrics, STS, and humanities
research. In a final section, we test the potential of our approach in a pilot study of
interdisciplinarity in AI, and offer reflections on digital mapping as a pathway towards
indicating interdisciplinarity.

INTRODUCTION

The ambition to move beyond the divide between quantitative and qualitative methods is a
familiar one in the social studies of science and technology (STS). It goes back to at least
the 1980s, when an interdisciplinary group of researchers in Paris combined anthropological
fieldwork with statistical methods of network analysis to study innovation, and developed well-
known maps of “actor-networks”: emergent associations among actors in science, engineering,
and society (Callon, Courtial, et al., 1983; Teil & Latour, 1995; for a discussion, see the
editorial in this special issue: Leydesdorff, Ràfols, & Milojevic´, 2020). This enthusiasm for mov-
ing across the quantitative-qualitative divide was revived in the late 1990s and early 2000s,
when the Worldwide Web introduced qualitatively trained scholars in STS to a new type of
research environment (Rogers & Marres, 2000; Wouters, Vann, et al., 2008). Unlike the
scientific and engineering laboratories in which STS researchers had conducted their

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
q
s
s
/
a
r
t
i
c
e
–
p
d

f
/

1
3
1
0
4
1
1
8
6
9
8
5
3
q
s
s
_
a
_
0
0
0
6
2
p
d

b
y
g
u
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

From indicators to indicating interdisciplinarity

ethnographic studies during the decades before, the data-intensive settings of the 1990s Web,
with its graphical user interface and sprawling civic and literary content, had emerged out of the
combination of quantitative and qualitative traditions—in design, engineering, computer
science, civic culture, and literary experimentation. As such, the Web rendered many of
us comfortable with the idea of mixed competencies in computational knowledge practices,
and with diversity of methods in data-intensive social research on science and technology
(cf. Hine, 2005; Thelwall, Vaughan, & Björneborn, 2005).

Today we may be entering yet another period of experimentation across the divides
between quantitative—or more widely defined, scientific—and qualitative—interpretative—
approaches. On the one hand, many of us have by now grown more aware of the irreversible,
even irreconcilable, differences that derive from different methodological backgrounds in
either primarily scientific or predominantly interpretative traditions. On the other hand, the
potential of collaboration is today recognized in most if not all fields, and many of us remain
committed to the possibilities of combining the methodological strengths of diverse ap-
proaches across the sciences and humanities. We have become only more curious about
the possible results—not just the findings, but also in terms of methods, tools, concepts, and
environments—that we might be able to realize if we learn how to do this work of “method-
ological recombination.” Furthermore, in the social studies of science and technology, we find
today new and different interlocutors at the table as compared to 10 or 20 years ago. Over the
last years, design research, for instance, has emerged as a crucial force of methodological in-
novation in this area, with creative methodologies such as prototyping enabling new forms of
exchange between diverse actors and forms of knowledge (Dantec & DiSalvo, 2013; Vertesi,
Ribes, et al., 2016).

In this article, we discuss a project currently under development called “Inventing
Indicators of Interdisciplinarity1” as an example of this type of boundary-crossing work in
methodology development, which seeks to connect quantitative methods with interpretative
approaches developed in STS and creative forms of humanities research. The project originated
from a shared interest among STS scholars, scientometricians, and humanities scholars in
digital metrics and indicators as a site of methodological innovation. Recent work in each of
these areas has proposed that such measures can be deployed to enable generative, inclusive,
and interactive ways of exploring and evaluating research (Fochler & de Rijcke, 2017; Holtrop,
2018; Lury, Fensham, et al., 2018; Marres, 2017; Ràfols, 2019). The development of such
engaging indicators, however, is likely to require interdisciplinary collaboration of its own,
and to entail changes in the (inter)disciplinary embedding of the indicators themselves.
Crucially, it means recognizing that indicators are designed entities, which in their material,
visual, and interactive forms engage and include or exclude actors—scientists, administrators,
policy-makers, stakeholders, activists, and publics—in evaluation processes. We will argue that
this recognition has methodological implications for how indicators can be used to explicate
phenomena such as interdisciplinarity.

Key to our project is the idea that indicators not only have representative value, enabling
empirical insights into fields of research and innovation but simultaneously have organizing
capacity, as their deployment enables the active curation of communities of interpretation.
When asking whether and how indicators are capable of configuring and reconfiguring

1 This pilot project started in 2018 and is a collaboration between the Centre for Interdisciplinary
Methodologies (CIM) at the University of Warwick, and the Centre for Science and Technology Studies
(CWTS) at Leiden University: https://warwick.ac.uk/fac/cross_fac/cim/research/inventing-indicators-of-inter-
disciplinarity/.

Quantitative Science Studies

1042

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
q
s
s
/
a
r
t
i
c
e
–
p
d

f
/

1
3
1
0
4
1
1
8
6
9
8
5
3
q
s
s
_
a
_
0
0
0
6
2
p
d

b
y
g
u
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

From indicators to indicating interdisciplinarity

audiences and assembling communities, this opens indicators up to issues that are specifically
addressed in the area of participatory methods, user studies, and design research (Lezaun,
Marres, & Tironi, 2016).

It is important, then, that we assess how the relevant fields—scientometrics, STS, and design
methods—could work together to develop engaging indicators. We selected Science and
Technology indicators of interdisciplinarity as the empirical focus for this work of methodolog-
ical experimentation. “Interdisciplinarity” is both a current area of indicator development in
scientometrics and a topic of special interest to STS and humanities scholars. For the latter,
interdisciplinary research is a key site where the role of interpretative forms of enquiry in
the academy and society is being negotiated today (Biagioli, 2009). As we will discuss, it is
also a topical area in which the need for, and benefits of, indicators capable of engaging
diverse stakeholders and publics is more readily apparent than in others. In a series of meetings
and workshops, we then reviewed the capacities of methods developed in diverse fields—
scientometric analysis of Web of Science, social media analysis, and playful mapping—to
indicate interdisciplinarity in a specific area of research and innovation (artificial intelligence)
and on this basis, we identified key features and requirements for what we here call engaging
indicators. On this basis, we here outline our proposal for a combinatory methodology for the
development of engaging indicators. We reflect on what is at stake in this project for us as STS
scholars working in interpretative traditions, and briefly indicate the methodological potential
of engaging indicators through a discussion of our shared efforts to indicate interdisciplinarity
in AI.

INTERDISCIPLINARITY AS A CATEGORY AT STAKE: APPROACHES IN STS

2.
AND SCIENTOMETRICS

In recent years, research evaluations using Science and Technology Indicators have been
identified as a key site where the shape and viability of interdisciplinary research are being
contested (Ràfols, Leydesdorff, et al., 2012; Wagner, Roessner, et al., 2011). Interdisciplinary
contributions to knowledge are often evaluated less positively than disciplinary research,
in peer-review panels for journals and grants, and evaluation exercises such as the United
Kingdom’s Research Excellence Framework (REF). It has been argued that performance indica-
tors such as journal impact factors exemplify and consolidate this bias against interdisciplinary
research, insofar as core disciplinary journals tend to perform better according to this popular
metric. On a more general level, performance indicators, such as those derived from citation
data, have been criticized for driving the metricization of research culture (De Rijcke, Holtrop,
et al., 2019): The de facto reliance on indicators such as the impact factor in evaluation pro-
cesses creates a situation in which scientists, assessors, and policy-makers are encouraged, and
become more inclined, to value research in terms of quantifiable markers of recognition and
significance (citation, impact), rather than in substantive terms. As Ismael Ràfols put it (2019):
“[A] compelling argument of ‘what went wrong with indicators’ (Barré, 2019) … is that,
although indicators were developed in the spirit of enlightenment as tools that would inform
decision-making, they have become ‘ignorance producing devices’, in the sense that they are
used as horse blinkers that reduce the issues taken into consideration.” At the same time, and
precisely for these reasons, the design and deployment of indicators has been identified as a
potential site for methodological innovation: Developing indicators of interdisciplinarity could
be an effective way to counter the devaluation of interdisciplinary research and enable the
articulation and valuation of interdisciplinary research agendas (Holtrop, 2018; see also Fochler
& de Rijcke, 2017).

Quantitative Science Studies

1043

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
q
s
s
/
a
r
t
i
c
e
–
p
d

f
/

1
3
1
0
4
1
1
8
6
9
8
5
3
q
s
s
_
a
_
0
0
0
6
2
p
d

b
y
g
u
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

From indicators to indicating interdisciplinarity

Before discussing indicators of interdisciplinarity in more detail, it is worth noting that in-
terdisciplinarity holds special importance in both STS and scientometrics. In STS, interdisci-
plinarity has significance on multiple levels: as a methodological approach, an object of
enquiry, a strategic site, and a normative commitment. First, STS as a field itself tends to be
defined as interdisciplinary, bringing together social science and humanities disciplines such
as sociology and anthropology with the sciences and engineering. Second, STS seeks to un-
derstand empirical transformations of science and innovation and their roles in society and
culture, and this notably includes the increased valorization of interdisciplinarity in some re-
search communities, and by policy-makers and research funders over the last decades
(Biagioli, 2009; Nowotny, Scott, & Gibbons, 2003). Third, interdisciplinarity has been wel-
comed in STS on strategic grounds: It presents a knowledge ideal that can help to render
the humanities newly relevant in a changing research and innovation landscape in which
the organization of knowledge is increasingly problem-centered (Biagioli, 2009; see also
Davidson & Goldberg, 2004). Fourth, STS has a normative investment in interdisciplinarity.
The encounter between diverse knowledge communities is valued by many in the field as a
mechanism for rendering monodisciplines more accountable, both to one another and to
society. It is understood to be aligned with the wider agenda of the democratization of knowl-
edge (Nowotny et al., 2003; Barry, Born, & Weszkalnys, 2008). In STS, finally, interdisciplin-
arity is also valued on epistemological grounds: as a way of realizing adventures in knowledge
(Savransky, 2016).

One recent contribution to the study of interdisciplinarity (Barry et al., 2008) is especially
relevant to our purposes here. Drawing on ethnographic fieldwork in interdisciplinary research
laboratories, Barry et al. identify different types of interdisciplinarity, from hierarchical forms of
collaboration, where one discipline makes up for a lacuna in another (master) discipline, to
interdisciplinary research that operates in the ontological mode. In the latter type, it becomes
the objective of interdisciplinary research “to re-conceive both the object(s) of research and
the relations between research subjects and objects” (Barry et al., 2008, p. 25). When intro-
ducing this typology, Barry et al. note that interdisciplinarity is best treated as an agonistic
category. That is, they conceive of interdisciplinarity—and disciplinarity—not just as a given
attribute of existing fields of knowledge but as a contested category: the forms that interdisci-
plinary research will take—the division of labor between fields; where key concepts are de-
rived from; the relations between data and method—are the focus of disagreement and power
struggle. Building on this proposition, we want to propose that interdisciplinarity can be pro-
ductively understood as a category at stake: the shape of interdisciplinary research is not only
contested on the level of discourse, as the notion of agonism highlights, it is called into ques-
tion as part of interdisciplinary knowledge practices themselves, and must therefore be under-
stood as at least partly an accomplishment—an outcome—of interdisciplinary research
collaborations, exchanges, and communication. We call this the transformative understanding
of interdisciplinarity2. We think that this understanding of interdisciplinarity can connect with
scientometrics in interesting ways, not least because the latter field is methodologically and
technically equipped for the analysis of knowledge dynamics, the detection of interdisciplin-
ary fields and communities in formation (Leydesdorff & Schank, 2008).

2 In their formulation of agonistic interdisciplinarity, Barry et al. rely on a political concept, the idea of agon-
ism put forward by the political theorist Chantal Mouffe. As such, their conceptualization leaves under-
specified the epistemological and ontological process through which the relations between the objects
and subjects of research, and between concepts, methods, and data, are redefined on the level of knowl-
edge practice.

Quantitative Science Studies

1044

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
q
s
s
/
a
r
t
i
c
e
–
p
d

f
/

1
3
1
0
4
1
1
8
6
9
8
5
3
q
s
s
_
a
_
0
0
0
6
2
p
d

b
y
g
u
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

From indicators to indicating interdisciplinarity

In tune with the significant interest in interdisciplinarity in science policy and in STS, scien-
tometricians have during the last 15 years or so developed a variety of ways to measure inter-
disciplinarity. In a review of this literature, Wagner et al. (2011) describe different
scientometric measures developed to assess interdisciplinarity. The most common approach
is citation analysis, which measures engagement between fields by detecting “the occurrence
of what are considered discipline-specific citations pointing to other fields” (Wagner et al.,
2011, p. 19). Such methods provide a relational approach to the detection of interdisciplinar-
ity, and this is one of the key advantages that we, as Science and Technology Studies scholars
working in interpretative traditions, find in them.

Relying on methods of network analysis such as citation analysis—and related measures
such as keyword co-occurrences—research in scientometrics has addressed substantive ques-
tions about the organization of research—such as the relative closedness and openness of
research fields—in terms of relations among authors, topics, and outputs. On a general level,
such a methodological approach seems to us well aligned with the transformative understand-
ing of interdisciplinarity discussed above, which is concerned with changing relations between
fields, between subjects and objects and research, and we add, between data, methods, and
concepts (Marres & Gerlitz, 2012). From this perspective, the key advantage of scientometrics
is that one does not have to assume a fixed ontology at the outset of research—say assuming
“disciplines” as already constituted ex ante. One can recognize that ontologies are dynamic,
so that one cannot only treat as an empirical question what the relevant entities are, which
their relations are, and what their attributes are but also recognize that these very categories
are in question in the empirical realities under study3.

Ràfols et al. (2012) proposed to use network-analytic measures to operationalize Barry
et al’s conception of interdisciplinarity, and show how such measures can help to foreground
the contributions to knowledge of actors located at the margins of or in-between fields. These con-
tributions are “seldom captured in conventional classification categories” (Ràfols et al., 2012,
p. 10). “Interdisciplinarity” is thus defined not ex ante but operationally. To this operationali-
zation of “agonistic interdisciplinarity,” we would like to add that scientometrics as a relational
methodology can also be used to elaborate the understanding of interdisciplinarity as con-
tested and transformative: Scientometrics can help to render visible, and explorable, interdis-
ciplinarity as a dynamic space, a space of possible transformation of the relations between
disciplines, and between concepts, methods, and data. Furthermore, such an analysis of trans-
formative interdisciplinarity opens up distinctive methodological opportunities for the evalua-
tion of interdisciplinarity. With the help of indicators, the empirical study of different types and
logics of interdisciplinarity can be turned into an opportunity for reconstructive engagement.
Indicators can actively enable the type of active negotiation and contestation of interdisciplin-
ary knowledge and innovation, which the notion of transformative interdisciplinarity denotes.

Reviewing indicators of interdisciplinarity, we find both alignment and divergences be-
tween, on the one hand, measures of interdisciplinarity put forward in scientometrics, and,
on the other, the concepts of agonistic and transformative interdisciplinarity developed in
STS and related fields. We have been struck by the relative emphasis on knowledge integration
in scientometric definitions of interdisciplinarity, which frames interdisciplinarity as a process
of integrating different bodies of knowledge (Porter & Ràfols, 2009; Wagner et al., 2011,
p. 16). In our view, this focus on the production of newly coherent fields of knowledge risks

3 Note that mapping does not necessarily presume flat ontology: In scientometrics, the use of cartographic
methods goes hand in hand with clear ontological assumptions, for instance, positing authors, journals, and
citations as constitutive of scientific research. ( With thanks to Ràfols, pers. communication.)

Quantitative Science Studies

1045

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
q
s
s
/
a
r
t
i
c
e
–
p
d

f
/

1
3
1
0
4
1
1
8
6
9
8
5
3
q
s
s
_
a
_
0
0
0
6
2
p
d

b
y
g
u
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

From indicators to indicating interdisciplinarity

displacing attention away from the dynamics of epistemological and ontological contestation
of concepts, methods, and object delineations between diversely composed fields and
communities. However, a number of indicators of interdisciplinarity have been proposed that
foreground diversity in the composition of fields of knowledge and connectivity across fields,
thus pointing precisely beyond the preoccupation with the unification of scientific fields. This
notably includes the compositional measure of “diversity”—which foregounds the “heteroge-
neity of the bibliographic set,” of the outputs composing a field of research and innovation
(Ràfols & Meyer, 2010; Stirling, 2007)— and the citation-based measure of interdisciplinarity
in terms of intermediation and betweenness (Leydesdorff, Wagner, & Bornmann, 2019; see
also Wang & Schneider, 2019). The latter foregrounds the location of interdisciplinarity re-
search between different fields, and the relational capacities of this research to bring diverse
thematic communities into relation. Such compositional and relational concepts and measures
hold significant promise as ways to translate transformative understandings of interdisciplinar-
ity, insofar as they direct attention to shifting connections on multiple levels: between outputs,
clusters of outputs, and potentially thematic and methodological communities. This makes
possible an appreciation of the relational dynamics through which ontological and epistemo-
logical transformations play out at the intersections of science, innovation, and society.

Scientometric research on interdisciplinarity demonstrates significant awareness of the
contested nature of this category. The measurement of interdisciplinarity is marked by a lack
of consensus (Ràfols et al., 2012) and as Wagner et al. (2011, p. 15) observe, the term
interdisciplinary is “notable for conflicting meaning.” Ràfols and Meyer (2010) noted that
a specific difficulty in developing indices of interdisciplinarity is that “research areas qual-
ifying as interdisciplinary tend to be marked by ‘multidimensional[-ity] and inherent conflict
with categorisation’” (Ràfols & Meyer, 2010, p. 263). Attempts to map interdisciplinarity
using scientometric methods, then, are not exempt from the challenges posed by the con-
tested and transformative nature of interdisciplinarity. Importantly, scientometric work has
made it clear that these challenges of defining interdisciplinarity not only play out on the
level of ideas—in how we conceptualize interdisciplinarity—they equally trouble its oper-
ationalization in indicators and metrics. This raises the question of whether and how the
process of the detection of interdisciplinarity can itself be organized so as to facilitate the
exploration of interdisciplinarity as a zone of epistemological and ontological contestation
and transformation. Rather than deploying scientometric measures to close down the de-
bate, the objective then becomes to deploy indicators so as to facilitate exploration of
knowledge in (trans-)formation.

3. FROM INDICATORS TO INDICATING INTERDISCIPLINARITY
In his paper “Indicators in the Wild,” Ràfols (2019) proposed that indicators can be purpose-
fully deployed as participatory devices, provided “the development of STI indicators should
take place not only in ‘secluded spaces’ such a scientometric labs, but with the participation of
stakeholders so as to take in consideration their contexts, both in terms of relevant social
spaces and values.” In proposing this, Ràfols offers a key articulation of indicators as a special
type of interface between measurement and participation, which has the capacity to assemble
and engage diverse actors through processes of interpretation of metrical outputs across
boundaries (sciences and the humanities, university and society). Ràfols’ proposal to treat in-
dicators as participatory devices resonates in interesting ways with recent work by Celia Lury
and colleagues, in which they seek to clarify the double role of methods in society and culture
as “means by which the social world is not only investigated, but may also be engaged” (Lury
& Wakeford, 2012, p. 6). In a recent paper, Gerlitz and Lury (2014) extend this perspective to

Quantitative Science Studies

1046

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
q
s
s
/
a
r
t
i
c
e
–
p
d

f
/

1
3
1
0
4
1
1
8
6
9
8
5
3
q
s
s
_
a
_
0
0
0
6
2
p
d

b
y
g
u
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

From indicators to indicating interdisciplinarity

social media metrics: They point out that metrics such as the Klout score—a popular, aggre-
gate measure of influence across different online platforms such as Twitter and Facebook—
may operate simultaneously in two registers: On the one hand, they perform an epistemic
operation by ordering relations between social media users through mathematical operations
of quantification (producing, among other things, influence rankings). On the other hand, they
enable participation as they assemble users into a dynamic collective, by enabling these users,
as well as third parties, to compare, contrast, and relate to one another by way of Klout scores.

Asking whether and how indicators are capable of configuring audiences and assembling
communities opens indicators to issues that are commonly addressed in fields other than STS
or scientometrics, such as user studies and design research. The question also implies a rec-
ognition that indicators are more than forms of measurement. By drawing attention to their
material, visual, and interactive form, we recognize that indicator development is likely to
require different competencies, such as those of design. And by noting that indicators not
only represent the worlds of research and innovation but are and can be deployed to
configure communities of interpretation, we signal that their specification will require taking
their context of application into account: where indicators are deployed, whom they address,
and with what purpose? Here, we will not be able to pursue all of these multiple questions,
but instead will sketch a methodological framework to render the questions tractable in
empirical research.

Key to our methodological approach to indicators is a constructive valuation of disagree-
ment. In scientific methodology, a lack of agreement can easily be interpreted in negative
terms, as preventing the stabilization of a concept or a measure, and thus, its operationaliza-
tion or application. However, in the social sciences and humanities, disagreement, or contest-
ability, must in some cases be understood as constitutive of the empirical phenomenon in
question, and here controversy has been valued as an occasion in which relations, divisions
of labor, and ideas may be undergoing transformation (Latour, 2005). This includes interdis-
ciplinary research, in particular that of the transformative type.

To capture this contestability, sociologists have put forward the notion of heterarchy
(Stark, 2011; Weber, 1946), to highlight situations in which multiple principles intersect
and can be contested. In our opinion, this term is applicable to the lack of consensus that
becomes noticeable when defining interdisciplinarity. However, we want to emphasize that
recognizing heterarchy has methodological implications for the design and deployment of
indicators of interdisciplinarity. When indicators are used, the methodological assumption
often is that the concept in question—interdisciplinarity—can be clearly and robustly de-
fined before the act of measurement takes place. In other words, the framework for indicator
usage is deductive. However, in the case of an “essentially contested” and transformative
category such as interdisciplinarity, it would not be appropriate to assume that the definition
of interdisciplinary research can be agreed a priori, and a deductive approach is thus un-
likely to work in this case.

This could be taken as a weakness of indicator-based methodologies, but it is also possible
to turn this assessment around: The measurement of interdisciplinarity provides interesting op-
portunities to redefine, reconstruct, or reinvent the use of indicators in the evaluation of re-
search and innovation. We find inspiration to develop such an approach in recent work by
De Rijcke et al. (2019) which put forward the “evaluative inquiry,” an experimental approach
to evaluation that treats it as a knowledge production process. A key step towards this frame-
work is to let go of reproducing the conventional role of analysts as detached accountants who
treat indicators as external proxies for quality. Instead, evaluative enquiry treats analysts as

Quantitative Science Studies

1047

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
q
s
s
/
a
r
t
i
c
e
–
p
d

f
/

1
3
1
0
4
1
1
8
6
9
8
5
3
q
s
s
_
a
_
0
0
0
6
2
p
d

b
y
g
u
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

From indicators to indicating interdisciplinarity

engaged experts, who take part in formulating the broader projects of ensuring accountability
and autonomy with which an evaluation is always intertwined (De Rijcke et al., 2019, p. 178).
Building on this, we claim that a participatory, abductive, interactive approach to indicator
development and deployment is promising, in light of the objective defined above, namely,
to facilitate a collective exploration of interdisciplinarity as a “zone of epistemological and
ontological contestation.” Below we define each of these terms in turn.

Participatory. Rather than assuming that the development of an indicator is strictly
separate from the process of its interpretation, we propose to approach these tasks
as interrelated. Understanding indicators as designed entities suggests that both the
context of application and the community of evaluation need to be actively curated—
or even scripted, and in turn, that indicators can be deployed to actively assemble
actors that do not already form a social group into an interpretative community.
The interpretation of indicators—in evaluative situations—may be treated dramaturgically,
as a situation in which different participants are assigned a role and brought into re-
lation according to a basic script. As a provisional name for this script we propose the
term indicating (more about which below). It means that we approach participation in
evaluation as a methodological proposition. It is not merely something to strive for
because it is considered the correct thing to do. Instead, actors with different kinds
of expertise in and relevant experiences of an area of research and innovation are ex-
plicitly invited to inform, at designated moments, the process of interpretation in
which indicators play a role.

2. Abductive. Timmermans and Tavory (2012, p. 167), drawing on the work of the prag-
matist philosopher C. S. Pierce, define abduction as “a creative inferential process
aimed at producing new hypotheses and theories based on surprising research evi-
dence.” It provides an alternative to both deductive and inductive approaches in that
it proposes an iterative framework for data analysis. It involves a moving back and
forth between empirical material and concepts that can assist in interpretation. Such
an iterative approach enables a constructive recognition of the limitations of existing
indicators and finds in these limitations the starting point for their elaboration.
Timmermans and Tavory (2012) also note that “abductive analysis arises from actors’
social and intellectual positions” but can be elaborated through data analysis (p. 173).
As such, the approach is also in line with the participatory commitment above, as it
values participants’ perspectives, and exchanges among them, as a potential resource
for the interpretation of the indicator under scrutiny.
Interactive. This last commitment complements the former two. It foregrounds that the
productive deployment of indicators in a participatory and abductive manner requires
methodological attention to how indicators facilitate interaction among different points
of view, data sources, measures, concepts, and contexts. This methodological interac-
tivity goes beyond interaction in the social sense: It does means not just exchange be-
tween actors but a process of active interarticulation between data sets, measures,
perspectives, and contexts (Marres & Gerlitz, 2012). Deploying an indicator interac-
tively, methodologically speaking, means that indicators are used not just to facilitate
an exchange between different points of view but to bring to the surface different types
of interdisciplinarity, in a shared process of data selection, analysis, and evaluation.
This methodological commitment connects with the sociotechnical requirements that
today’s digital, networked data infrastructures and tools place on the use of indicators:
The increased availability of underlying data sources and contextual information, and
the growing uptake of interactive, visual user interfaces, in today’s digital culture,

Quantitative Science Studies

1048

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
q
s
s
/
a
r
t
i
c
e
–
p
d

f
/

1
3
1
0
4
1
1
8
6
9
8
5
3
q
s
s
_
a
_
0
0
0
6
2
p
d

b
y
g
u
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

From indicators to indicating interdisciplinarity

Figure 1. Garden bugs, Kussaberg (Germany), close to Leibstadt nuclear power plant, by C. Hesse-
Honegger (Raffles, 2010).

facilitate a more exploratory approach to rendering data interpretable and analyzable
(for a discussion, see Waltman & Van Eck, 2016)4.

4. Designed. A shift from indicators to indicating offers an opportunity to open up indi-
cator development to design methodology, and approach indicating as a situated,
equipped, and/or visual process. The interactive design of indicators can be informed
by design research, where creative methods such as rapid prototyping are used in
workshop settings to explicate understandings of interdisciplinarity with diverse stake-
holders and using simple materials such as paper and ribbon. Here play with materials—
and data?—serves the methodological purpose of “surfac[ing] implicit assumptions and
values as well as to communicate and test ideas” (Lockton, Brawley, et al., 2019, p. 3). To
be sure, there are significant challenges in connecting such an approach with
scientometric methods, as the success of the latter depends on the capacities of data,
concept, and method to act as constraints on the process of interpretation.

Still, when approached as participatory, abductive, and interactive, the process of indicat-
ing can be significantly enriched by the deployment of material and visual—and not neces-
sarily metrical—proxies. This also has the benefit of opening up our understanding of the
empirical content of indicators to the nonmetric. The German STS scholar Tahani Nadim,
who works in the area of biodiversity research, for example, reminded us of the existence
of biological indicators, and told us that “stink bugs are very good indicators—their markings
change markedly and look differently when something goes wrong in their living environ-
ment” (Figure 1). In this approach, the empirical content of an indicator derives at least in part
from its indexicality: An index is valued as a “material trace” (Schuppli, 2012) rather than as a

4 Such as approach does not endorse a perspectival understanding of interdisciplinarity (“everyone under-
stands it differently”): To define the articulation of interdisciplinarity as a collective task recognizes that
different definitions of interdisciplinarity do not merely coexist but are likely to pose a challenge to one
another, insofar as the endorsement of one definition may reduce opportunities for the pursuit of another.
This is also why we insist that interdisciplinarity is at stake in AI research and innovation.

Quantitative Science Studies

1049

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
q
s
s
/
a
r
t
i
c
e
–
p
d

f
/

1
3
1
0
4
1
1
8
6
9
8
5
3
q
s
s
_
a
_
0
0
0
6
2
p
d

b
y
g
u
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

From indicators to indicating interdisciplinarity

representation of a phenomenon, and this trace can be read as an indication of something that
is latent in a particular situation (e.g., a latent danger).

Taken together, the above four commitments enable an approach to indicators that is in-
herently combinatory, in that it brings together methodological propositions from sciento-
metrics, STS, design research, and interpretative social and cultural research methodology.
We describe this approach as indicating to highlight something that each of the four terms
above have in common: They frame the development and use of indicators as a process.
This is key insofar as it enables us to understand the assembly of communities of interpretation
as an ongoing process, one that spreads out across the design and deployment of indicators. In
referring to our approach as “indicating,” we also build on the Routledge Handbook of
Interdisciplinary Research Methods (Lury et al., 2018), in which the editors proposed that
the development of interdisciplinary methods can benefit from a focus on the “doing of
methods” (p. 2). In contrast to the definition of method as formalized procedure, method is here
understood as referring to a generative process that unfolds through specific socioepistemic and
sociomaterial processes such as mapping, prototyping, and visualizing, and as such, offers a way
of working with intractable differences. We now add the term indicating to this list.

4. TEST CASE: INDICATING INTERDISCIPLINARITY IN AI

To test the possibility of combining scientometrics with other approaches in this way, we
conducted a pilot study of interdisciplinarity in artificial intelligence (AI). AI is an interesting
case, because interdisciplinarity has a contradictory status in this area of research and inno-
vation: In recent years, interdisciplinarity has been identified as both an existing strength and
a major challenge for AI research. Thus, on the one hand, the crossing of boundaries be-
tween disciplines and between the sciences and humanities is viewed as inherent to this
area of research and innovation. As a recent report by Elsevier (2018), noted: “Research
in AI is both theoretical and applied, and transcends traditional disciplinary boundaries,
bringing together experts from diverse fields of study.” On the other hand, recent efforts
to map AI research have highlighted important current barriers to interdisciplinary collabo-
ration. A report by the Nuffield Foundation, “Ethical and societal implications of algorithms,
data, and AI: a roadmap for research,” observed that efforts to advance interdisciplinarity in
AI are currently held back, not least because interdisciplinarity itself is understood in diverse
ways across different areas of AI research (Whittlestone, Nyrup, et al., 2019). We then
sought to combine scientometrics with other methods, such as evaluative enquiry in STS,
visualization, and playful methods in order to open up for investigation—and constitute
as matters of reflection (Venturini & Meunier, 2019)—the contradictory status of interdisci-
plinarity in AI.

In investigating this, we were especially interested to establish whether and how AI can be
considered as a potential site of transformative interdisciplinarity. Based on a cocitation anal-
ysis of Web of Science publications in the area of AI, Cardon, Cointet, et al. (2018) suggested
that with the rise of neural networks and deep learning a connective paradigm is now super-
seding previous mentalist and cognitive frameworks in AI, and that this new paradigm may
bring AI research in the computational sciences closer to the social and human sciences.
Castelle (2020) explores this further through fieldwork methods in his study of the uptake of
pragmatist and sociological concepts of learning and communication in current research on
adversarial neural networks. This work thus suggests to us that relations between scientific,
social-scientific, and humanities fields may currently be at stake in AI as an area of research
and innovation.

Quantitative Science Studies

1050

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
q
s
s
/
a
r
t
i
c
e
–
p
d

f
/

1
3
1
0
4
1
1
8
6
9
8
5
3
q
s
s
_
a
_
0
0
0
6
2
p
d

b
y
g
u
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

From indicators to indicating interdisciplinarity

In the international workshop organized as part of the Inventing Indicators of Interdisciplinarity
project 5 we posed the following questions: Is interdisciplinary AI principally concerned
with the application of computational methods to social and cultural data? Which other
forms of collaboration across disciplines can we detect in AI as an interdisciplinary area
of research and innovation? And what is the potential contribution that new types of disci-
plinary and interdisciplinary ensembles spanning the faculties seek to make in this area? The
workshop yielded various insights, but here we want to highlight a single one: the impor-
tance of digital mapping—or data cartography—as the methodological framework for indi-
cating interdisciplinarity in this area, one that can enable the exploration of transformative
interdisciplinarity.

On a general level, mapping is well attuned to an abductive approach to data analysis, one
where data selection, analysis, and visualization are performed recurrently, in a process of
continuous exploration. During our workshop, Ludo Waltman clarified the concrete implica-
tions of endorsing a mapping methodology for the design of the “pipeline” of bibliometric re-
search. Identifying three key steps in bibliometric mapping—field delineation, indicator
selection, and data interpretation (labeling), Waltman proposed that there are ways to adapt
the pipeline to affirming—rather than overcoming—the challenge of reaching consensus in
each of the three steps. For example, in the case of field delineation, the question is what
belongs and does not belong to AI research?

Challenges to consensus can come from various directions: For one, outputs that did not
mention the word AI may nevertheless be key to the field. Waltman also highlighted a con-
sequential step in the Elsevier study discussed above, which relied on a small group of experts
to identify the keywords used to train the machine learning algorithm guiding field delinea-
tion in this study. Based on the fields that the study eventually identified—Search and
Optimization, Fuzzy Systems, Natural Language Processing and Knowledge Representation,
Computer Vision, Machine Learning and Probabilistic Reasoning, Planning and Decision
Making, and Neural Networks—it seems likely that the keyword selection did not include con-
sideration of topics that could help to surface AI research in the social sciences and human-
ities. It appears the experts did not pay sufficient attention to AI research in the social sciences
and humanities in the training of the machine learning algorithm.

Similar challenges to consensus can be raised in relation to indicator selection and the la-
beling of clusters in the process of field delineation. Presenting a mapping of the field of AI
using VOSviewer at our workshop, Waltman identified two clusters, “AI and Society” and “Big
Data and Society,” as areas where interdisciplinary exchanges between the sciences and hu-
manities were growing (Figure 2). However, this act of labeling, too, could be interpreted as
introducing a partial perspective: Why privilege science/social sciences/humanities interac-
tion as a modality of interdisciplinarity in AI? Answering this question is likely to take us be-
yond the map into a space of deliberation.

Such investigations of “selection effects” in scientometric analysis could be taken as ex-
pressing concerns with bias, opening up the analysis to criticisms to the effect that objective
and thus shareable frameworks of reference are not being secured through scientometric anal-
ysis. However, if we recognize interdisciplinarity in AI as contested and transformative, a dif-
ferent methodological interpretation of these debates is opened up. From this perspective, the
issue of selection can be treated constructively: The moment of selection presents a participa-
tory occasion, one in which experts, stakeholders, and other participants can be invited into

5 Indicating Interdisciplinarity in AI, workshop funded by the Alan Turing Institute, University of Warwick,

February 6, 2020; https://warwick.ac.uk/fac/cross_fac/cim/events/indicating-interdisciplinarity-in-ai/.

Quantitative Science Studies

1051

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
q
s
s
/
a
r
t
i
c
e
–
p
d

f
/

1
3
1
0
4
1
1
8
6
9
8
5
3
q
s
s
_
a
_
0
0
0
6
2
p
d

b
y
g
u
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

From indicators to indicating interdisciplinarity

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
q
s
s
/
a
r
t
i
c
e
–
p
d

f
/

1
3
1
0
4
1
1
8
6
9
8
5
3
q
s
s
_
a
_
0
0
0
6
2
p
d

b
y
g
u
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Figure 2.
shop, University of Warwick, February 2020).

Interdisciplinarity in AI, Leiden field delineation, VOSviewer ( Waltman, presented at the Indicating Interdisciplinarity in AI work-

the process of identifying the boundaries and composition of the field on methodological
grounds. If these boundaries and composition are inherently contested, then the identification
of multiple keywords, multiple ways of labeling clusters, is not only appropriate; it is likely to
strengthen the process of interpretation. This is what we mean when saying that indicating
involves not only the construction of a proxy but also the curation of a script for the interplay
between data, measures, questions, and participants6.

5. CONCLUSION: MAPPING AS A PATH TOWARDS INDICATING

One important implication of the participatory approach to mapping interdisciplinarity is that
it entails a move away from the classic opposition between maps and indicators in the soci-
ology of science and innovation. In institutional sociology, indicators safeguard the realist
view; they secure the externality of social reality. From this realist perspective, to move beyond
measurement to the construction of an indicator means to construct a theory of what a variable
is a measure of. It is to posit a proxy.

Performative approaches, by contrast, traditionally rely on mapping to facilitate iterative
processes of data exploration (selection, analysis, visualization). This makes it possible to af-
firm that the specification of the object of enquiry—interdisciplinarity in AI—is accomplished
through this iterative process (Marres & Moats, 2015). In a scientometric analysis of interdis-
ciplinarity in AI, we could then recognize that the “appropriate” delineation of the research
field of AI, and the concept of interdisciplinarity appropriate to its exploration, emerges from
the interplay between bibliometric data (say all abstracts containing the word AI in a database)
and method (say, the Leiden method of field delineation through clustering). From this perfor-
mative perspective, the move from mapping to indicating involves more than the construction
of a proxy. Crafting—or curating—are needed to configure the context of application: What is

6 See Lammes and Wilmott (2018) on the map as playground and playful methods taking us one step below

the level of the “finding.”

Quantitative Science Studies

1052

From indicators to indicating interdisciplinarity

the mapping for? In other words, the move from mapping to indicating is a move from the
relatively open-ended exploration of relations between entities to scripting interactions be-
tween questions, data, measures, concepts, and participants— in such a way that the issues
at stake can be surfaced and receive a formulation.

The move from mapping to indicating also enables us to move beyond an appreciation of
scientometrics in terms of its relational approach to data exploration, which we ourselves in-
voked above. Rather than celebrating the methodological contribution of relational methods
such as keyword co-occurences and citation analysis in terms of the emergent entities these
methods render traceable, to move to indicating is to welcome the task of actively configuring
the context of evaluation, and the community of interpretation, that we produce our mapping
for (Marres, 2017). This could include identifying prevalent themes and ambitions in AI re-
search and innovation, their operationalization, the people and resources that are mobilized,
and the outputs this generates. In emphasizing process and engagement in indicating interdis-
ciplinarity in AI, indicators may play a role in characterizing forms of interdisciplinarity still in
formation. They may be scripted to enable the negotiation of interdisciplinarity among diverse
participants, data sources, and methodologies, and amidst multiple epistemic commitments,
which are not already congealed into a research community; for example, when interdisciplin-
arity involves the creation of new combinations and connections between sciences and hu-
manities. To explore different ways in which indicators can be used to enable negotiation, it
may be helpful to start with more open-ended spaces of exploration, such as network visual-
izations and forms of relational mapping. Thus, perhaps somewhat surprisingly, if our wider
aim is to move from indicators to indicating, it is good to start with mapping. In doing so, new
ways of indicating interdisciplinarity in AI research and innovation might emerge, which sup-
port and highlight exchanges across boundaries rather than suppress them.

ACKNOWLEDGMENTS
We would like to thank contributors to the “Inventing Indicators of Interdisciplinarity” project:
Anne Beaulieu, Rodrigues Costa Comesana, Thomas Franssen, Tjitkse Holtrop, Sybille
Lammes, Celia Lury, Greg McInerny, Ismael Ràfols, and Ludo Waltman. We are grateful to
the reviewers, Loet Leydesdorff, and James McNally for their comments on an earlier version
of this article. We would also like to thank participants in the Indicating Interdisciplinarity in AI
workshop that took place at the University of Warwick in February 2020 and the Alan Turing
Institute, which funded the event.

COMPETING INTERESTS

The authors have no competing interests.

FUNDING INFORMATION

This article has benefitted from a workshop grant from The Alan Turing Institute ( JES no. 64678).

REFERENCES

Barré, R. (2019). Les indicateurs sont morts, vive les indicateurs!
Towards a political economy of S&T indicators: A critical
the past 35 years. Research Evaluation, 28(1),
overview of
2–6.

Barry, A., Born, G., & Weszkalnys, G. (2008). Logics of interdisci-

plinarity. Economy and Society, 37(1), 20–49.

Biagioli, M. (2009). Postdisciplinary liaisons: Science studies and

the humanities. Critical Inquiry, 35(4), 816–833.

Quantitative Science Studies

1053

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
q
s
s
/
a
r
t
i
c
e
–
p
d

f
/

1
3
1
0
4
1
1
8
6
9
8
5
3
q
s
s
_
a
_
0
0
0
6
2
p
d

b
y
g
u
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

From indicators to indicating interdisciplinarity

Callon, M., Courtial, J. P., Turner, W. A., & Bauin, S. (1983). From
translations to problematic networks: An introduction to co-word
analysis. Information (International Social Science Council), 22(2),
191–235.

Cardon, D., Cointet, J. P., & Mazieres, A. (2018). Neurons spike
back: The invention of inductive machines and the artificial in-
telligence controversy. Réseaux, La Découverte, 211(5), 173.
https://doi.org/10.3917/res.211.0173.hal-02190026

Castelle, M. (2020). The social lives of generative adversarial net-
works. In Proceedings of the 2020 Conference on Fairness,
Accountability, and Transparency (pp. 413–413). New York:
Association for Computing Machinery.

Dantec, C. A. L., & DiSalvo, C. (2013). Infrastructuring and the for-
mation of publics in participatory design. Social Studies of
Science, 43(2), 241–264.

Davidson, C. N., & Goldberg, D. T. (2004). Engaging the humani-

ties. Profession, 2004(1), 42–62.

De Rijcke, S., Holtrop, T., Kaltenbrunner, W., Zuijderwijk, J. B.,
Beaulieu, J. A., … Wouters, P. F. (2019). Evaluative inquiry:
Engaging research evaluation analytically and strategically.
Journal for Research and Technology Policy Evaluation, 48,
176–182.

Elsevier. (2018). Artificial intelligence: How knowledge is created,
transferred, and used. Trends in China, Europe, and the United
States. Amsterdam: Elsevier.

Fochler, M., & De Rijcke, S. (2017). Implicated in the indicator
game? An experimental debate. Engaging Science, Technology,
and Society, 3, 21–40.

Gerlitz, C., & Lury, C. (2014). Social media and self-evaluating as-
semblages: On numbers, orderings and values. Distinktion:
Scandinavian Journal of Social Theory, 15(2), 174–188.

Hine, C. (2005). Virtual methods and the sociology of cyber-social-
scientific knowledge. In C. Hine (Ed.) Virtual methods: Issues in
social research on the internet (pp. 1–13). Oxford: Berg.

Holtrop, T. J. (2018). The evaluative inquiry: A new approach to
research evaluation. https:// blogs.lse.ac.uk/impactofsocials-
ciences/2018/11/29/the-evaluative-inquiry-a-new-approach-to-
research-evaluation/

Lammes, S., & Wilmott, C. (2018). The map as playground: Location-
based games as cartographical practices. Convergence, 24(6),
648–665.

Latour, B. (2005). Re-assembling the social: A guide to actor-network

theory. Oxford: Oxford University Press.

Leydesdorff, L., Ràfols, I, & Milojevic´, S. (2020). Bridging the divide
between qualitative and quantitative science studies. Quantitative
Studies of Science, 1(3), 918–926.

Leydesdorff, L., Wagner, C. S., & Bornmann, L.

(2019).
Interdisciplinarity as diversity in citation patterns among journals:
Rao-Stirling diversity, relative variety, and the Gini coefficient.
Journal of Informetrics, 13(1), 255–264.

Leydesdorff, L., & Schank, T. (2008). Dynamic animations of jour-
nal maps: Indicators of structural change and interdisciplinary
developments. Journal of the American Society for Information
Science and Technology, 59(11), 1810–1818.

Lezaun, N., Marres, M., & Tironi, M. (2016). Experiments in partic-
ipation. In C. Miller, L. Smith-Doerr, U. Felt, & R. Fouche (Eds.) The
handbook of science and technology studies (4th ed., pp. 195–221).
Cambridge, MA: MIT Press.

Lockton, D., Brawley, L., Aguirre Ulloa, M., Prindible, M.,
Forlano, L., … Nissen, B. (2019). Tangible thinking: Materialising
how we imagine and understand systems, experiences, and
relationships. Proceedings of Relating Systems Thinking and Design
RSD8 Symposium, Chicago: Systemic Design Association.

Lury, C., & Wakeford, N. (Eds.). (2012). Inventive methods: The

happening of the social. Routledge.

Lury, C., Fensham, R., Heller-Nicholas, A., Lammes, S., Last, A., …
Uprichard, E. (2018). Routledge handbook of interdisciplinary
research methods. London: Routledge.

Marres, N. (2017). Digital sociology: The reinvention of social re-

search. Cambridge: Polity.

Marres, N., & Moats, D. (2015). Mapping controversies with social
media: The case for symmetry. Social Media + Society, 1(2).
https://doi.org/10.1177/2056305115604176

Marres, N. and C. Gerlitz (2012). Interface methods: Renegotiating
relations between digital social research, STS and sociology.
Sociological Review, 1(64), 21–46.

Nowotny, H., Scott, P., & Gibbons, M. (2003). Introduction: ‘Mode
2’ revisited: The new production of knowledge. Minerva, 41(3),
179–194.

Porter, A., & Ràfols, I. (2009). Is science becoming more interdisci-
plinary? Measuring and mapping six research fields over time.
Scientometrics, 81(3), 719–745.

Raffles, H. (2010) Insectopedia. New York: Pantheon Books.
Ràfols, I., Meyer, M. (2010). Diversity and network coherence as
indicators of interdisciplinarity: Case studies in bionanoscience.
Scientometrics, 82, 263–287. https://doi.org/10.1007/s11192-
009-0041-y

Ràfols, I., Leydesdorff, L., O’Hare, A., Nightingale, P., & Stirling, A.
(2012). How journal rankings can suppress interdisciplinary re-
search: A comparison between innovation studies and business
& management. Research Policy, 41(7), 1262–1282.

Ràfols, I. (2019). S&T indicators in the wild: Contextualization and
participation for responsible metrics. Research Evaluation, 28(1),
7–22.

Rogers, R., & Marres, N. (2000). Landscaping climate change: A
mapping technique for understanding science and technology
debates on the World Wide Web. Public Understanding of
Science, 9(2), 141–163.

Savransky, M. (2016). The adventure of relevance: An ethics of so-

cial inquiry. Basingstoke: Palgrave.

Schuppli, S. (2012). Material malfeasance: Trace evidence of vio-

lence in three image-acts. Photoworks, 17, 28–33.

Stark, D. (2011). The sense of dissonance: Accounts of worth in

economic life. Princeton: Princeton University Press.

Stirling, A. (2007). A general framework for analysing diversity in
science, technology and society. Journal of the Royal Society
Interface, 4(15), 707–719.

Teil, G., & Latour, B. (1995). The Hume Machine: can associations
networks do more than formal rules? Stanford Humanities
Review, 2(4), 47–66.

Thelwall, M., Vaughan, L., & Björneborn, L. (2005). Webometrics.
Annual Review of Information Science and Technology, 39(1),
81–135.

Timmermans, S., & Tavory, I. (2012). Theory construction in qualitative
research: From grounded theory to abductive analysis. Sociological
Theory, 30(3), 167–186. https://doi.org/10.1177/0735275112457914
Venturini, T., & Meunier, A. (2019). Drafting an atlas of artificial
intelligence’s matters of reflection. Exhibit, Global Forum on
Artificial Intelligence for Humanity, Paris, October 28–30.
Vertesi, J., Ribes, D., Forlano, L., Loukissas, Y., & Leavitt Cohn, M.
(2016). Engaging, designing, and making digital systems. In C.
Miller, L. Smith-Doerr, U. Felt, & R. Fouche (Eds.) The handbook
of science and technology studies (4th ed., pp. 169–194).
Cambridge, MA: MIT Press.

Wagner, C. S., Roessner, J. D., Bobb, K., Thompson Klein, J., Boyack,
K. W., … Börner, K. (2011). Approaches to understanding and

Quantitative Science Studies

1054

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
q
s
s
/
a
r
t
i
c
e
–
p
d

f
/

1
3
1
0
4
1
1
8
6
9
8
5
3
q
s
s
_
a
_
0
0
0
6
2
p
d

b
y
g
u
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

From indicators to indicating interdisciplinarity

measuring interdisciplinary scientific research (IDR): A review of
the literature. Journal of Informetrics, 1(5), 14–26.

Wouters, P., Vann, K., Scharnhorst, A., Ratto, M., Hellsten,

I.,
… Beaulieu, A. (2008). Messy shapes of knowledge—STS ex-
plores informatization, new media, and academic work. The
virtual knowledge studio. In E. J. Hackett, O. Amsterdamska, M.
Lynch, & J. Wajcman (Eds.) The handbook of science and
technology studies (pp. 319–353). Cambridge, MA: MIT
Press.

Waltman, L., & Van Eck, N. J. (2016). The need for contextualized
scientometric analysis: An opinion paper. In Proceedings of the

21st International Conference on Science and Technology
Indicators—STI 2016.

Wang, Q., & Schneider, J. W. (2019). Consistency and validity of inter-
disciplinarity measures. Quantitative Science Studies, 1(1), 239–263.
Weber, M. (1946). Science as a vocation. In A. I. Tauber (Ed.)
Science and the Quest for Reality (pp. 382–394). Palgrave
Macmillan, London.

Whittlestone, J., Nyrup, R., Alexandrova, A., Dihal, K., & Cave, S.
(2019). Ethical and societal implications of algorithms, data, and
artificial intelligence: A roadmap for research. London: Nuffield
Foundation.

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
q
s
s
/
a
r
t
i
c
e
–
p
d