ARTÍCULO DE INVESTIGACIÓN
Consistency and validity of
interdisciplinarity measures
Qi Wang1,2,3
and Jesper Wiborg Schneider1
1The Danish Centre for Studies in Research and Research Policy (CFA), Aarhus University, Dinamarca
2KTH Library, KTH-Royal Institute of Technology, Suecia
3Department of Philosophy and History, KTH-Royal Institute of Technology, Suecia
Palabras clave: interdisciplinary research, interdisciplinarity, measures, consistencia, validity
ABSTRACTO
Measuring interdisciplinarity is a pertinent but challenging issue in quantitative studies of
ciencia. There seems to be a consensus in the literature that the concept of interdisciplinarity is
multifaceted and ambiguous. Como era de esperar, various different measures of interdisciplinarity
have been proposed. Sin embargo, few studies have thoroughly examined the validity and
relations between these measures. en este estudio, we present a systematic review of these
interdisciplinarity measures and explore their inherent relations. We examine these measures
in relation to the Web of Science journal subject categories. Our results corroborate recent
claims that the current measurements of interdisciplinarity in science studies are both
confusing and unsatisfying. We find surprisingly deviant results when comparing measures that
supposedly should capture similar features or dimensions of the concept of interdisciplinarity.
We therefore argue that the current measurements of interdisciplinarity should be interpreted
with much caution in science and evaluation studies, or in relation to science policies. Nosotros
also question the validity of current measures and argue that we do not need more of the same,
but rather something different in order to be able to measure the multidimensional and complex
construct of interdisciplinarity.
1.
INTRODUCCIÓN
Studies that examine interdisciplinarity quantitatively tend to bemoan the measurement situ-
ación. Criticisms are rife and there is no consensus in relation to the definition and operatio-
nalization of interdisciplinary research (IDR; p.ej., Rafols, Leydesdorff, O’Hare, Nightingale, &
Stirling, 2012; Wagner et al., 2011). Como consecuencia, numerous indicators or metrics purport
to measure the concept or aspects of it.
The concept of interdisciplinarity is tied to notions of academic disciplines. Esencialmente, el
concept is often envisioned as a synthesis of theories or methodological activities from differ-
ent disciplines resulting in an emergent interdisciplinary activity. Sin embargo, there is consider-
able ambiguity with the discipline concept itself, its delineation, and empirical manifestations
(Sugimoto & Weingart, 2015). Históricamente, disciplines have been linked to the organization of
teaching at universities. Sin embargo, there is more to disciplines than the fact that something is a
subject taught in an academic setting. The concept has evolved to become a more general
term encompassing the organization of learning, but also the systematic production of new
conocimiento (Abbott, 2001). Focusing on knowledge production, Sugimoto and Weingart
(2015) suggest that academic disciplines can be examined empirically from three perspectives,
un acceso abierto
diario
Citación: Wang, P., & Schneider, j. W..
(2020). Consistency and validity of
interdisciplinarity measures.
Estudios de ciencias cuantitativas, 1(1),
239–263. https://doi.org/10.1162/
qss_a_00011
DOI:
https://doi.org/10.1162/qss_a_00011
Recibió: 13 Junio 2019
Aceptado: 29 Agosto 2019
Autor correspondiente:
Qi Wang
qiwang@kth.se
Editor de manejo:
Juego Waltman
Derechos de autor: © 2019 Qi Wang and Jesper
Wiborg Schneider. Published under a
Creative Commons Attribution 4.0
Internacional (CC POR 4.0) licencia.
La prensa del MIT
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
1
1
2
3
9
1
7
6
0
8
5
8
q
s
s
_
a
_
0
0
0
1
1
pag
d
.
/
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Consistency and validity of interdisciplinarity measures
what they term “publications,” “people,” and “ideas.” In essence, all three perspectives rely on
data from (journal) publications in multidisciplinary bibliographic databases. What Sugimoto
and Weingart (2015) refer to as “publications” are disciplinary delineations based on index-
ing and classification of publications and/or their parent journals. “People” uses authors,
mentors, and affiliations as delineations, whereas “ideas” refer to cognitive attributes such
as language use, temas, and methodology.
Most measures of interdisciplinarity are rooted in scientometric operationalizations of dis-
ciplinary structures, mainly based on the “publication” perspective suggested by Sugimoto and
Weingart (2015). The scientometric approaches utilize the vast quantities of publications in
bibliographic databases to structure the literature according to established categories or by
applying various different indexing techniques. Depending on the purpose, such structures
are then perceived as “disciplinary structures.” However, with no conceptual or operational
consensus and plenty of attributes and researcher degrees of freedom, it is no surprise that IDR
based on scientometric techniques has been scrutinized and interpreted in many different
maneras. Numerous indicators, measures or metrics have been proposed for measuring interdis-
ciplinarity, but only a few studies have actually examined the relation between such measures,
their validity, and their consistency. Although Rafols and Meyer (2007) initially concluded that
interdisciplinarity measures based on publications and their citation relations can provide a
comparatively accurate description of cross-boundary knowledge creation, Leydesdorff and
Rafols (2011, pag. 98) later concluded that “different indicators may capture different understand-
ings of such a multi-faceted concept as interdisciplinarity.” Recently, a report by Digital Science
(2016, pag. 2) concluded that the choice of data sets and methodologies produces “inconsistent
and sometimes contradictory” results.
The aim of the present study is to further examine the relations between a number of seem-
ingly similar interdisciplinarity measures. We examine empirically to what extent findings
based upon them are consistent, in order to be able to shed more light on their validity and
potential use for science policy. We limit our empirical review to proposed interdisciplinarity
measures based on the “publication” perspective (es decir., publications and citation relations).
The article is structured as follows: En la sección 2 we briefly introduce the data used for the
empirical analyses. En la sección 3 we summarize definitions and measures of interdisciplinarity
examined in the study. Después, we present the results in Section 4. Discussion and con-
clusions follow in Sections 5 y 6.
2. UNIT OF ANALYSIS
En el presente estudio, we examine a number of known interdisciplinarity measures empirically
by comparing their outcomes when applied to the Web of Science (WoS) journal subject cat-
egories (SC). As data, we use all publications of the document type article published in 2010
from the in-house WoS database at the Centre for Science and Technology Studies (CWTS) en
Universidad de Leiden. As the validity and effectiveness of using a journal citation database is ques-
tionable for research fields where journals are not the main scientific communication medium,
we exclude SCs from the Arts & Índice de citas de humanidades, Resultando en 224 WoS SCs included
in the analysis.
Classification systems are essential when quantifying interdisciplinarity using bibliometric
methods. The groupings are presumed to represent “disciplines” and their potential mutual
relations become the determining factor when measuring “interdisciplinarity.” Numerous clas-
sification systems are available. en este estudio, we use the WoS journal classification system as
disciplines or categories. The WoS classification is by no means a “ground truth”; sobre el
Estudios de ciencias cuantitativas
240
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
1
1
2
3
9
1
7
6
0
8
5
8
q
s
s
_
a
_
0
0
0
1
1
pag
d
/
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Consistency and validity of interdisciplinarity measures
contrary, it is arbitrary in its details (Wang & waltman, 2016). En efecto, no classification system
can be seen as the “truth,” as different systems may serve different purposes, y, in that sense,
the choice of a system should be seen in relation to the purpose of a study. Por ejemplo, el
lower level of the Organisation for Economic Co-operation and Development (OECD) classi-
fication system has around 40 categories, whereas the WoS classification system consists of
250 SC. Following the above logic, a unit of analysis may show a higher degree of inter-
disciplinarity when using the WoS system compared to the OECD system simply due to their
different granularities. We use the WoS classification system because it is generally the most
frequently used system in scientometric studies, including interdisciplinarity research based on
bibliometric methods.
Before we introduce measures of interdisciplinarity, two issues need to be stressed. Primero,
references from, and citations to, scientific publications can both be used to operationalize
interdisciplinarity. According to Levitt, Thelwall, and Oppenheim (2011, pag. 1121), “there does
not seem to be clear evidence that one is preferable to the other.” However, others stress that
citations and references have different implications (p.ej., Portero & Chubin, 1985). Creemos
that references given in a publication are a reflection of the “knowledge base” upon which the
work is built. Compared to citations, reference lists in publications better reflect the potential
integration of knowledge from different research fields, a focal issue in relation to measuring
interdisciplinarity. Como consecuencia, we focus on the use of references in this study.
Segundo, most interdisciplinarity measures can be applied at the level of individual publi-
cations, but also at higher levels of aggregation. Por ejemplo, for a set of publications belong-
ing to a unit of analysis, one can apply the Rao-Stirling index (RS) to measure the degree of
interdisciplinarity for each of the publications individually and subsequently compute the
mean or median and use it as the degree of interdisciplinarity for the unit. Alternativamente,
one could instead view all publications from the unit of analysis as one entity under which
all references are subsumed. Entonces, the proportion of these references over different WoS SCs
would be used as the input for the RS index. We use the latter approach in this study.
Sin embargo, the RS index will also be used to demonstrate differences in degrees of interdisci-
plinarity when measured at different levels. We elaborate on this in the following section.
3. OVERVIEW OF INTERDISCIPLINARITY MEASURES
En esta sección, we first review and discuss the definitions of IDR used in bibliometric studies.
Después, we summarize the proposed interdisciplinarity measures. We briefly discuss the
reviewed measures at the end of this section.
3.1. Related Work on Definitions and Operationalizations of IDR
We have identified a seed set of publications studying interdisciplinarity in bibliometric studies
based on our prior knowledge. The set was expanded by including pertinent publications ac-
cording to the references given in the seed publications. The definitions of IDR were extracted
de 15 publications that are outlined chronologically in Table A1 in the Appendix.
Although the overall concept of IDR is seen by some as ambiguous or uncertain (p.ej., Rafols
et al., 2012; Wagner et al., 2011), the impression one gets from reading the definitions sum-
marized in Table A1 is actually one of small nuances between them. They are quite similar in
their conceptualization, seeing IDR as a kind of knowledge integration. It seems that especially
two key attributes are frequently emphasized: “diversity,” which describes the differences in
the bodies of knowledge that are integrated, and “coherence,” which describes the intensities
of the relations between these bodies of knowledge. Diversity is presumed to reflect to what
Estudios de ciencias cuantitativas
241
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
1
1
2
3
9
1
7
6
0
8
5
8
q
s
s
_
a
_
0
0
0
1
1
pag
d
/
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Consistency and validity of interdisciplinarity measures
extent a body of knowledge or a unit of analysis comprises knowledge rooted in two or more
different research fields. It seems to be the most important attribute of IDR, as almost all studies
reviewed discuss it.
According to Rafols et al. (2012), the “integration” of research is perceived as the process of
establishing connections between cognitively distant or separate bodies of knowledge. Alguno
argue that integration (es decir., coherencia) is a necessary complement to diversity in order to iden-
tify IDR (p.ej., Rafols & Meyer, 2010). Although a high degree of diversity implies that a body of
knowledge draws upon knowledge from several fields, it does not indicate to what extent, if at
todo, such knowledge is mutually integrated. In this view, IDR is seen as a combination of a high
degree of diversity and coherence (es decir., knowledge integration; Rafols & Meyer, 2010; Rafols
et al., 2012).
One can argue that the definition and operationalization of IDR by Leydesdorff (2007) dif-
fers from the other listed studies, as the degree of interdisciplinarity depends on a unit’s posi-
tion in a citation network. Rafols et al. (2012) use the notion of “intermediation” to refer to this
network perspective of interdisciplinarity. Sin embargo, in our view, the operationalization of
intermediation is essentially related to diversity. Por ejemplo, a journal’s degree of interdisci-
plinarity is based upon its relations to adjacent journals in a citation network. More citation
links for a journal implies a higher degree of interdisciplinarity. En ese sentido, we think that
intermediation depicts diversity externally to a body of knowledge, and not internally, as ini-
tially intended with the diversity definition state above, where the focus is upon knowledge
heterogeneity within a body of knowledge.
Finalmente, some terminological inconsistencies exist. Many near-synonyms are used to de-
scribe attributes of interdisciplinarity or its opposite features: “specialization,” “concentration,"
“unevenness,” “information richness,” “information abundance,” etc. As they are poorly
delineated and most likely redundant in relation to the abovementioned main features, nosotros
exclude these variant terms from our analyses.
In the following subsections, we classify the interdisciplinarity measures proposed from pre-
vious studies into four groups based on the resemblance of their strategies. This grouping is
subjective and serves to improve the reading flow of this section. Other groupings can be con-
ceived of. Además, given the limited space, it is not possible to make exhaustive descrip-
tions of each measure. For more detailed descriptions we refer to the previous studies.
3.2. Group 1: Interdisciplinarity Measures Depending on a Multi-Classification System
The first group of measures are characterized by depending to a large extent on the WoS jour-
nal classification system for their calculation. As the WoS SC indexing is not exclusive, jour-
nals can be assigned to multiple categories. The multiple indexing strategy is utilized by some
to measure interdisciplinarity. The relevant measures are briefly introduced below.
(cid:129) Percentage of multiassigned journals (p_multi). Let i denote a WoS SC. P_multi is the
percentage of journals in i that have been assigned to more than one SC (Morillo,
Bordones, & Gómez, 2001, 2003).
(cid:129) Percentage of journals outside the area (p_outside). The percentage of journals in i that
have been assigned to more than one research area. Research areas are higher aggre-
gation levels consisting of several SCs. Notice, such levels are not part of the WoS sys-
tem and need to be constructed (Morillo et al., 2001, 2003). This study applies the
CWTS high aggregation of SCs as research areas.
Estudios de ciencias cuantitativas
242
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
1
1
2
3
9
1
7
6
0
8
5
8
q
s
s
_
a
_
0
0
0
1
1
pag
d
.
/
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Consistency and validity of interdisciplinarity measures
(cid:129) Percentage of references outside the category (pro). The percentage of journal references
that publications in SC i cited outside i (Morillo et al., 2001; Portero & Chubin, 1985).
(cid:129) Diversity of links (d_links). The number of distinct journal pairs generated from journals
belonging to SC i, in which the pair of journals should belong to different SCs. To reduce
size effects, it is normalized by the total number of journals in SC i (Morillo et al., 2003).
(cid:129) Pratt index. The Pratt index was initially proposed to measure concentration that allows
comparison of SCs and journals in different research fields (Pratt, 1977). Respectivamente,
the higher the proportion of references from different SCs that publications in SC i cited,
the more interdisciplinary SC i is considered. Because this index has a negative relation
with interdisciplinarity, we use 1 − Pratt instead.
(cid:129) Specialization index (Spec). Porter and colleagues suggested that the exploration of spe-
cialization could provide insight into IDR (Portero, cohen, Roessner, & Perreault, 2007).
They proposed the specialization index to measure the spread of references that publi-
cations in SC i cited over all other SCs. The specialization index is similar to the Pratt
índice, but perhaps more intuitive. Like the Pratt index, the specialization index is also
inversely related to interdisciplinarity, and hence we use 1 − Spec instead.
3.3. Group 2: Interdisciplinarity Measures Borrowed from Other Fields
Indices originating in other fields such as economics and biology have also been suggested as
measures of interdisciplinarity. These indices were originally constructed to measure different
constructs such as biodiversity, income inequality, and information uncertainty to name three
well-known examples. They are briefly summarized below.
(cid:129) Simpson diversity index. Simpson diversity index measures the probability that two en-
tities randomly sampled from a population will not belong to the same category.
(Simpson, 1949; see also Zhang, Rousseau, & Glänzel, 2016).
(cid:129) Shannon entropy. Shannon entropy was proposed to measure “information uncertainty”
(shannon, 1948, 2001). Some argue that “information uncertainty” is linked to the con-
cept of diversity, because entropy can quantify the distribution of references over SCs.
This is to some extent similar to the design of 1 − Pratt and 1 − Spec. Por ejemplo, si
publications in a category only cited other publications in this category, the diversity of
the reference distribution would be maximal and the uncertainty would be minimal
(Leydesdorff & Rafols, 2011).
(cid:129) Brillouin diversity index. Brillouin index is a modification of Shannon entropy, and also
aims to measure the uncertainty of information (Brillouin, 1956). Steele and Stier (2000)
argued that “the Brillouin index is a proper indicator of interdisciplinarity since it con-
siders the number of observations and the distribution of observations among catego-
ries” (Huang & Chang, 2012, pag. 793).
(cid:129) Gini coefficient. The Gini coefficient was proposed as a measure of income inequality.
When assessing interdisciplinarity, it considers the distribution of references over SCs for
a group of publications (p.ej., Leydesdorff & Rafols, 2011; Wang, Thijs, & Glänzel,
2015). This is also to some extent similar to the design of 1 − Pratt and 1 − Spec. Él
should be noted that the Gini coefficient has a negative relation with interdisciplinarity,
and thus we use 1 − Gini.
3.4. Group 3: Interdisciplinarity Measures That Consider the Similarity of Research Fields
The measures introduced so far focus on the overlap of publications or references in SCs and/
or the distribution of references over SCs. Such measures are criticized for not considering the
Estudios de ciencias cuantitativas
243
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
1
1
2
3
9
1
7
6
0
8
5
8
q
s
s
_
a
_
0
0
0
1
1
pag
d
/
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Consistency and validity of interdisciplinarity measures
similarity of SCs. Como consecuencia, interdisciplinarity measures that include a similarity index
have been proposed, including the three measures below.
(cid:129) RS index. The RS index has been widely used to measure diversity and more generally,
interdisciplinarity (p.ej., Porter et al., 2007; Portero & Rafols, 2009; Wang y cols., 2015). Es
assumed that the index incorporates essential attributes of diversity (es decir., variety, bal-
ance, and similarity; Rafols & Meyer, 2010). We will not discuss the attributes of diversity
further in this study. Sin embargo, because the construction of similarity matrices and the
influence it has upon the RS are often neglected in interdisciplinarity studies, we briefly
discuss this issue at the end of this section.
(cid:129) Hill-type measure. Recientemente, Zhang et al. (2016) claimed that the RS index produces results with
low discriminative power. They stated, “[a]s seen in the study by Zhou, Rousseau, Cual, yue,
and Yang (2012), the Rao-Stirling measure showed only low discriminatory power since values,
at least in their work, sometimes differ only by the third decimal” (pag. 1257). The low discrimi-
native power among the interdisciplinarity values may therefore cause problems in practical
applications (see also Zhou et al., 2012). To overcome the presumed limitations of RS, el
Hill-type measure was proposed (Colina, 1973; Leinster & Cobbold, 2012; Zhang et al., 2016).
(cid:129) Coherence measure. This measure emphasizes the knowledge integration between dif-
ferent research fields. It was used, por ejemplo, by Soós and Kampis (2012) and Wang
(2016). We name it a coherence measure. En el presente estudio, the coherence of SC i
was examined based on the references of publications belonging to i. The more inten-
sive the citation links of the references belonging to different SCs, the higher the knowl-
edge integration between research fields and the larger the degree of interdisciplinarity
for SC i is considered. We do not distinguish between the directions of citation links of
the references here. This is unnecessary because we are only interested in the degree of
integration of references from different SCs. Finalmente, some claim that this measure actu-
ally combines diversity and coherence, and hence argue that it can be used exclusively
for measuring interdisciplinarity (Soós & Kampis, 2012; Wang, 2016).
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
1
1
2
3
9
1
7
6
0
8
5
8
q
s
s
_
a
_
0
0
0
1
1
pag
d
/
.
3.5. Group 4: Interdisciplinarity Measures That Rely on Networks
The measures we introduced above can be seen as “within-based” measures because they rely
upon publications and reference relations within a set of publications. The global SC network
(es decir., the bibliometric relations between the SC under investigation and the external SCs) es
rarely included in the discussion of interdisciplinarity. As discussed in Section 2, some argue
that the location of a category in a global network can indicate its degree of interdisciplinarity
(Leydesdorff, 2007; Rafols et al., 2012). More specifically, it is assumed that when a group of
publications are located in an intermediate position in a network, it is an indication of IDR
(Rafols et al., 2012). Such types of interdisciplinarity measures are listed below.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
(cid:129) Betweenness-centrality (BC). Leydesdorff (2007) proposed to use the BC index (Hombre libre,
1977) to measure the degree of interdisciplinarity of journals. Betweenness measures the
degree of centrality that a node (entidad) is located on the shortest path between two other
nodes in a network (Hombre libre, 1997). Además, if a journal or a SC is at the interme-
diate position between other journals or SCs, then the journal or the SC can be consid-
ered as interdisciplinary, and its publications function as a communication channel for
other journals or SCs (Leydesdorff, 2007; silva, rodrigues, Oliveira, & Costa, 2013).
(cid:129) Cluster coefficient (CC). This measure was introduced by Rafols et al. (2012). For a given
CAROLINA DEL SUR, it first identifies the proportion of observed references between this category and other
Estudios de ciencias cuantitativas
244
Consistency and validity of interdisciplinarity measures
SCs over the expected maximum number of references. The proportion is then weighted
by the percentage of publications that this SC has over the total number of publications.
The CC of this category is the sum of these weighted proportions to other different SCs.
(cid:129) Average similarity (AS). This was also introduced by Rafols et al. (2012). For a given
CAROLINA DEL SUR, it simply measures the AS of this SC to all other SCs, and weights by the percent-
age of publications that this category has over the total number of publications. El
AS of this category is the sum of these weighted similarities.
3.6. Summary of Interdisciplinarity Measures
We have outlined 16 interdisciplinarity measures that use publications and reference relations.
Mesa 1 lists the notation used in this study. Mesa 2 presents the measures and their formula.
Despite our attempt to cover all interdisciplinarity measures in bibliometric studies, not all of these
are included in the present work for several reasons. Por ejemplo, Mugabushaka, Kyriakou, y
Papazoglou (2016) examined the use of different threshold values for the parameter in the Hill-
type measure and concluded that the differences are in fact small. Por eso, we will not do a further
test on other Hill-type measures in the present work.
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
3.7. Further Discussion on Similarity and Dissimilarity Matrices
It is necessary to elaborate on different approaches to generate similarity and dissimilarity
matrices. Although the so-called Salton’s cosine similarity index (Salton & McGill, 1983) is fre-
quently applied in bibliometric analyses, it actually has several different variants, and conse-
frecuentemente, different solutions and results can be expected (Schneider & Borlund, 2007a, b). Aquí,
Mesa 1. Notation in this study
Notation
ai
Descripción
The number of publications in category i
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
1
1
2
3
9
1
7
6
0
8
5
8
q
s
s
_
a
_
0
0
0
1
1
pag
d
/
.
cik
rjik
rjk
pik
qik
Pi
sij
ci
jk
norte
The number of references from publications in category i that cited
publications in category k
The number of the shortest paths from categories j to k that pass through
category i
The number of shortest paths between categories j and k
The proportion of references from publications in category i that cited
publications in category k. This can be expressed as pik = cik/(cid:1)
j cij
The proportion of references from a publication in category i that cited
publications in category k. qik is measured at the individual
publication level
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
The proportion of publications in category i over the total number of
publications of all SCs. This can be expressed as Pi = ai/(cid:1)
j aj
The similarity between categories i and j
For publications in SC i, the number of citation links between their cited
references in categories j and k
The number of SCs that the references of the publications in SC i
belong to
Estudios de ciencias cuantitativas
245
Consistency and validity of interdisciplinarity measures
Measure
p_multi
p_outside
pro
d_links
Pratt index
Spec
Simpson index
Shannon entropy
Brillouin index
Gini coefficient
RS
Hill-type measure
Coherence
BC1
CC
AS
Mesa 2. Interdisciplinarity measures reviewed in this study
The percentage of multi-assigned journals
Formula
The percentage of journals that are classified in
more than one research area
(cid:1)
k≠i cik/(cid:1)
j cij
The number of links between different SCs established
by journals in a given category
gpik
k
Þ
, where g is the index obtained by ranking
d
2 nþ1
d
Þ=
PAG
−
2
n−1
(cid:1)
k cik)2
pik in decreasing order
k c2
ik/((cid:1)
1 − (cid:1)k p2
I
−(cid:1)
k pik lnpik
k cik
k log cik!)/(cid:1)
k cik)! − (cid:1)
Þcik
(registro((cid:1)
PAG
2h−n−1
d
PAG
k
norte
according to cik in increasing order
j.k(1 − sjk)pijpik
(cid:1)
cik
k
, where h is the index attained by sorting SCs
1/(cid:1)
j.k sjkpijpik
jk(1 − sjk)
(cid:1)j,k ci
rjik
rjk
(cid:1)
j,k
(cid:1)
(cid:1)
cij
j Pj
aiaj
i Pi ( 1
norte
(cid:1)
j sij), where N is the number of all other SCs
1Leydesdorff (2007) used a symmetrical cosine matrix instead of a citation matrix to measure the interdisciplinarity of
journals using BC. He claimed that the size of journals was “controlled” in this way. Sin embargo, some researchers still
use a citation matrix to measure BC (for instance Silva et al., 2013). En nuestra opinión, it is more intuitive to use a citation
matrix as input for BC, as it better demonstrates the concept of intermediation. Por eso, we use a citation matrix in the
estudio actual. The shortest path indicates the path with the lowest total edge weight. Sin embargo, in our case, resultados
obtained by BC can be scaled with size (es decir. if SC i has a large number of publications, it is likely to have a high
degree of interdisciplinarity when BC is used). BC was calculated using the R package SNA: Tools for social network
análisis. The input citation matrix was generated using SQL from the in-house WoS database at CWTS. Otro
measures were calculated using SQL or a combination of SQL and R.
we first discuss two variants of the cosine formula. Suppose we aim to construct a symmetric sim-
ilarity matrix of SCs [sij] based on their mutual citation relations. The first step is to construct a
transaction matrix of citation relations between SCs [cij]. Tenga en cuenta que [cij] is an asymmetric matrix
with self-citations in the diagonal, whereas the similarity matrix, [sij], is a symmetric matrix.
One application of Salton’s cosine index can be illustrated as follows. Two SCs are consid-
ered to be strongly related if they commonly cite the same SCs (es decir., their vector profiles are
similar). Por eso, the similarity of two SCs i and j is given by
PAG
SC i; jð
Þ ¼
Estudios de ciencias cuantitativas
kcikcjk
ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
r
(cid:6)
(cid:5)
(cid:3)
(cid:4) PAG
PAG
kc2
I
kc2
jk
(1)
246
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
1
1
2
3
9
1
7
6
0
8
5
8
q
s
s
_
a
_
0
0
0
1
1
pag
d
.
/
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Consistency and validity of interdisciplinarity measures
Mesa 3. RS combinations
RS index
RS_P[1 − SC]
RS_P[1 − SO]
RS_P[1/CAROLINA DEL SUR]
RS_P[1/SO]
RS_G[1 − SC]
RS_G[1 − SO]
RS_G[1/CAROLINA DEL SUR]
RS_G[1/SO]
Formula
(1/ai)(cid:1)i((cid:1)j,k(1 − SC( j,k))qijqik)
j,k(1 − SO( j,k))qijqik)
j,k(1/CAROLINA DEL SUR( j,k))qijqik)
j,k(1/So( j,k))qijqik)
i((cid:1)
i((cid:1)
i((cid:1)
(1/ai)(cid:1)
(1/ai)(cid:1)
(1/ai)(cid:1)
(cid:1)
(cid:1)
(cid:1)
j,k(1 − SC( j,k))pijpik
j,k(1 − SO( j,k))pijpik
j,k(1/CAROLINA DEL SUR( j,k))pijpik
(cid:1)j,k(1/SO( j,k))pijpik
This vector application is closely aligned to the original application suggested by Salton and
McGill (1983) in relation to the Vector Space Model used in information retrieval.1 This
approach is used in some bibliometric studies (p.ej., Leydesdorff & Rafols, 2011). Otro
application of the cosine index is based on binary or scalar values (also known as the
Ochiai index):
SO i; jð
Þ ¼
cij þ cji
q
ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
(cid:3)
(cid:4)
(cid:3)
(cid:4) PAG
PAG
kckj
kcik
kcjk
kcki
PAG
PAG
þ
þ
;
i ≠j
(2)
Aquí, cij + cji is equal to the total number of citations between SCs i and j. Note that SO(i,i) is set
a 1. This way of calculating similarity was used in the study of Zhang et al. (2016).
Además, several strategies can transform a similarity matrix into a dissimilarity matrix.
The frequently applied solution is to use 1 − [sij] to obtain a dissimilarity matrix. There are also
studies using 1/[sij] (Jensen & Lutkouskaya, 2014). en este estudio, RS is used as an example to
demonstrate the potential empirical differences in interdisciplinarity at different levels of ag-
gregation that may result from choosing different dissimilarity matrices. Several combinations
of RS are summarized in Table 3, in which RS_P indicates the average RS interdisciplinarity of
publications in an SC. En cambio, RS_G views publications in an SC as one entity and uses the
proportion of all its references over different SCs as the input for the RS index.
4. RESULTADOS
Primero, we present the results regarding the relations between the interdisciplinarity measures
examined; then we outline their distributions over the WoS SCs. Finalmente, we present an in-
depth analysis of five selected WoS SCs.
4.1. Relations of Interdisciplinarity Measures
Primero, we examine the consistency of the four dissimilarity measures when applied to the 224
WoS SCs. This is shown in Table 4 using Pearson’s correlation coefficients. We find inconsis-
tencies in the dissimilarity matrices that use different versions of the cosine formulas.
1 Salton and McGill’s approach, sin embargo, is more in line with traditional matrix algebra, where an asymmetric
data matrix of publications and terms, n × m, is transformed into a symmetric similarity matrix, n × n.
Estudios de ciencias cuantitativas
247
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
1
1
2
3
9
1
7
6
0
8
5
8
q
s
s
_
a
_
0
0
0
1
1
pag
d
.
/
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Consistency and validity of interdisciplinarity measures
Mesa 4. Pearson’s correlation coefficients for four dissimilarity measures
1 − Sc
1/Sc
1 − So
1/So
1 − Sc
1
0.35
0.54
0.12
1/Sc
1
0.13
0.3
1 − So
1/So
1
0.04
1
Curiosamente, we also find inconsistencies among the ones using the same cosine formulas. Como
dissimilarity is an essential element for interdisciplinarity measures such as RS, variations in
these matrices obviously influence the resulting interdisciplinarity values (Schneider &
Borlund, 2007a, 2007b).
Cifra 1 shows the distributions of pairs of SCs over dissimilarity values. All distributions are
skewed. The matrices based on So as the input are extremely skewed. Por ejemplo, el dis-
similarity values yielded by 1 − So are largely between 0.95 y 1, which in this case implies
that RS would be close to the Simpson index.
As all reviewed measures are supposed to capture interdisciplinarity, we expected that their
mutual correlations would be high. We use Pearson’s and Spearman’s correlation coefficients
to examine the correlations between these measures when applied to the 224 WoS SCs. El
results we obtained by using both methods are in good agreement. Por lo tanto, we will mainly
explain the Pearson’s results shown in Table 5. The Spearman’s results can be found in
Table A2 in the Appendix. Primero, we examine the measures in the first two groups, which do
not rely on dissimilarity indices or global networks. According to the correlation coefficients, estos
measures can be roughly put into two clusters: (a) p_multi, p_outside, d_links, y 1 − Spec,
which are moderately correlated among each other; y (b) pro, 1 − Pratt, Simpson measure,
Shannon entropy, Brillouin index, y 1 − Gini, which are likewise moderately correlated among
entre sí. As discussed above, the Pratt index is to some extent similar to the Spec measure and
Cifra 1. Distributions of pairs of WoS SCs over dissimilarity values.
Estudios de ciencias cuantitativas
248
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
1
1
2
3
9
1
7
6
0
8
5
8
q
s
s
_
a
_
0
0
0
1
1
pag
d
.
/
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
q
tu
a
norte
t
i
t
a
i
t
i
v
mi
S
C
mi
norte
C
mi
S
tu
d
mi
s
t
i
1. p_multi
1
1.00
Mesa 5. Pearson’s correlation coefficients of interdisciplinarity measures
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21 22 23
2. p_outside
0.79 1.00
3. pro
0.44 0.45 1.00
4. d_links
0.63 0.65 0.56 1.00
5. 1 − Pratt
−0.23 −0.02 0.33 0.14 1.00
6. 1 − Spec
0.85 0.74 0.42 0.59 −0.29 1.00
7. Simpson
0.29 0.40 0.83 0.47 0.42 0.40 1.00
8. shannon
0.19 0.37 0.64 0.43 0.55 0.33 0.86 1.00
9. Brillouin
0.22 0.39 0.64 0.44 0.49 0.37 0.86 1.00 1.00
10. 1 − Gini
0.09 0.28 0.52 0.43 0.67 0.13 0.60 0.80 0.79 1.00
11. RS_P[1 − Sc] 0.14 0.31 0.09 0.21 0.36 0.13 0.15 0.25 0.23 0.32 1.00
12. RS_G[1 − Sc] 0.13 0.27 0.00 0.16 0.35 0.09 0.01 0.17 0.15 0.32 0.91 1.00
13. RS_P[1/Sc]
0.13 0.22 0.03 0.13 0.25 0.09 −0,02 0.09 0.07 0.21 0.82 0.86 1.00
C
oh
norte
s
i
s
t
mi
norte
C
y
a
norte
d
v
a
yo
i
d
i
t
y
oh
F
i
norte
t
mi
r
d
i
s
C
i
pag
yo
i
norte
a
r
i
t
y
metro
mi
a
s
tu
r
mi
s
0.26 0.38 0.20 0.28 0.32 0.22 0.19 0.31 0.30 0.43 0.69 0.78 0.78 1.00
14. RS_G[1/Sc]
15. RS_P[1 − So] 0.00 0.17 0.40 0.27 0.36 0.07 0.60 0.59 0.59 0.56 0.18 −0.05 −0.14 0.01 1.00
16. RS_G[1 − So] 0.03 0.22 0.39 0.28 0.43 0.08 0.60 0.65 0.64 0.68 0.29 0.15 0.00 0.22 0.93 1.00
17. RS_P[1/So]
−0,04 0.12 0.18 0.14 0.33 −0,03 0.15 0.21 0.19 0.38 0.55 0.59 0.65 0.41 0.20 0.27 1.00
18. RS_G[1/So]
0.15 0.32 0.29 0.33 0.48 0.14 0.36 0.55 0.54 0.67 0.52 0.59 0.51 0.76 0.22 0.38 0.43 1.00
19. Hill type
0.13 0.27 0.01 0.17 0.35 0.11 0.04 0.19 0.18 0.35 0.87 0.96 0.83 0.78 −0.01 0.18 0.58 0.60 1.00
20. coherencia
0.23 0.39 0.41 0.37 0.50 0.20 0.44 0.46 0.44 0.49 0.82 0.77 0.64 0.64 0.26 0.40 0.53 0.56 0.74 1.00
21. BC
22. CC
23. AS
2
4
9
−0,02 0.08 −0.15 −0.25 0.00 0.14 0.07 0.30 0.32 0.13 0.05 0.12 0.08 0.08 −0.04 −0.01 0.10 0.21 0.14 −0,03 1.00
0.14 0.11 −0.06 −0.16 −0.18 0.23 0.08 0.10 0.12 −0.08 −0.30 −0.30 −0.30 −0.19 −0.11 −0.15 −0.39 −0.08 −0.28 −0.36 0.38 1.00
−0,02 0.10 0.21 0.16 0.18 0.13 0.44 0.62 0.64 0.46 −0.29 −0.38 −0.41 −0.21 0.52 0.47 −0.19 0.15 −0.31 −0.23 0.31 0.33 1.00
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
1
1
2
3
9
1
7
6
0
8
5
8
q
s
s
_
a
_
0
0
0
1
1
pag
d
.
/
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Consistency and validity of interdisciplinarity measures
the Gini coefficient. De este modo, a high positive correlation among them should be expected. El
empirical results differ from our expectations, mostrando que 1 − Pratt has weak correlations
con 1 − Spec or 1 − Gini. Sin embargo, measures of Shannon and Brillouin are perfectly linearly
correlacionado.
Próximo, we examine the different combinations of RS. Por un lado, measures using the
same dissimilarity indices tend to have higher correlations also when calculated at different
levels of aggregation (es decir., individual vs. aggregated). Por otro lado, measures based on
different dissimilarity matrices show inconsistent results even at the same level of analysis. Para
instancia, measures RS_P[1 − Sc] and RS_P[1 − So] both take the average RS value of individual
publications as the degree of interdisciplinarity for the SCs. Sin embargo, due to the differences in
the dissimilarity matrices, their mutual correlation coefficient is only 0.18. As expected, differ-
ent dissimilarity matrices influence RS outcomes significantly. This is in line with the conclu-
sion of Leydesdorff and Rafols (2011). Además, it was also expected that RS_R[1 − So] y
RS_G[1 − So] show correlations with the Simpson diversity measure because [1 − So] is highly
left skewed.
The Hill-type and coherence measures also take the dissimilarity of SCs into consideration.
Because the two measures apply the dissimilarity matrix 1 − Sc, they are strongly linearly
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
1
1
2
3
9
1
7
6
0
8
5
8
q
s
s
_
a
_
0
0
0
1
1
pag
d
/
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Cifra 2. Cluster dendrogram of interdisciplinarity measures.
Estudios de ciencias cuantitativas
250
Consistency and validity of interdisciplinarity measures
correlated with the other measures using the same matrix. Note that interdisciplinarity mea-
sures relying on dissimilarities are somehow correlated with each other (except the ones
usando 1 − So as a dissimilarity matrix), but poorly correlated with most other measures.
It is difficult to interpret the correlations between the interdisciplinarity measures in the
fourth group (BC, CC, and AS) and the other examined measures. There are several negative
coefficients. One possible explanation could be that the measures focus upon within-category
parameters which differ from the network measures. Además, these network measures
also do not show strong mutual correlations with each other.
To provide more insight into the associations between the interdisciplinarity measures ex-
amined, a cluster solution based on the correlation coefficients is presented in Figure 2. De
the dendrogram, it is clear that these measures cluster in two groups depending on whether or
not a dissimilarity matrix is used. Más, among the ones excluding dissimilarity, the mea-
sures depending largely on the overlap of SCs (except pro and 1 − Pratt) and the network mea-
sures are clustered together. Those coming from other fields are also clustered together. Este
conclusion can also be observed in the heatmap shown in Figure A1 in the Appendix.
4.2. Distribution of WoS SCs over Interdisciplinarity
Cifra 3 shows the distributions of WoS SCs over interdisciplinarity values. For each histogram
in the panel, the x-axis shows the degree of interdisciplinarity and the y-axis shows the fre-
quency of SCs. Some of the histograms reveal considerable differences in the distributions.
Por ejemplo, RS_P[1 − So] and RS_G[1 − So] are left-skewed, whereas RS_P[1/So] y
RS_G[1/So] are highly right skewed. Además, the interdisciplinarity values for some of the
measures are concentrated within specific ranges. Taking RS_G[1 − So] as an example, es
values are highly concentrated between 0.9 y 1. Además, it should be noted that some
measures are not bounded (p.ej., Shannon entropy, Brillouin index, and the Hill-type measure).
One may argue that these distributions are not important because we can transform values
into more “suitable” distributions. Sin embargo, we do believe that they are useful. Primero, suppose
that we measured interdisciplinarity for an SC using, por ejemplo, RS_G[1 − So] and obtained
the value of 0.95. Sin embargo, it does not seem justified to conclude that this SC is highly inter-
disciplinary, as RS_G[1 − So] values are almost always close to 1.
Más, it is widely acknowledged that quantitatively determining the validity of an inter-
disciplinarity measure is challenging, because no benchmarks are available. Sin embargo, uno
may for instance expect the distribution of an interdisciplinarity measure to be approximately
normal or, alternatively, to be right skewed. Using the distributions presented in Figure 3, re-
searchers can compare them to these prior expectations. Por eso, we believe that studies on
interdisciplinarity measures should explicitly state the typical range of interdisciplinarity values
and present the empirical distributions.
4.3.
In-Depth Analysis of Several WoS SCs
A specific examination of the degree of interdisciplinarity for the WoS SCs based on the var-
ious measures provides a more direct impression regarding the effectiveness of these measures.
Five WoS SCs are selected; these are NANOSCIENCE & NANOTECHNOLOGY (NANO),
BIOCHEMISTRY & MOLECULAR BIOLOGY (BIOM), which previous studies often consider
to be highly interdisciplinary (p.ej., Aboelela et al., 2007; Portero & Youtie, 2009; Portero,
Roessner, cohen, & Perreault, 2006), LAW, MATHEMATICS (MATH), which are presumed
to show a low degree of interdisciplinarity, and INFORMATION SCIENCE & LIBRARY
Estudios de ciencias cuantitativas
251
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
1
1
2
3
9
1
7
6
0
8
5
8
q
s
s
_
a
_
0
0
0
1
1
pag
d
.
/
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Consistency and validity of interdisciplinarity measures
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
1
1
2
3
9
1
7
6
0
8
5
8
q
s
s
_
a
_
0
0
0
1
1
pag
d
.
/
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Cifra 3. Distributions of WoS SCs over interdisciplinarity values
SCIENCE (LIS). LIS is chosen because most papers investigating interdisciplinarity measures are
published in journals attached to this category.
Mesa 6 presents the different rankings according to the reviewed measures of the five se-
lected SCs among the 224. Instead of reporting the actual interdisciplinarity values obtained
from the measures, we provide the ranking obtained after sorting all 224 SCs according to their
Estudios de ciencias cuantitativas
252
Consistency and validity of interdisciplinarity measures
interdisciplinarity values in decreasing order. Some of the rankings are not in line with our
expectations. Por ejemplo, NANO is per se considered to be an interdisciplinary SC, but its
rankings in Table 6 based on the measures that include dissimilarities are generally quite low.
Además, measures having mutually strong Spearman’s correlations (see Table A2) alguno-
times lead to conflicting rankings for one specific category. Por ejemplo, RS_P[1 − Sc] y
RS_G[1 − Sc] have a strong Spearman’s correlation coefficient (0.91); sin embargo, MATH was
ranked as 221 y 79 respectively by these measures among 224 SC. Como consecuencia, given
the lack of agreement between the different interdisciplinarity measures, it is difficult to deter-
mine the degree of interdisciplinarity for a particular WoS SC. Measures that are supposed to
be similar or reflect similar aspects of IDR can produce different results.
Mesa 6. Interdisciplinarity rankings of the five SCs
Interdisciplinarity measures
p_multi
NANO
6
BIOM
60
p_outside
Pro
d_links
1 − Pratt
1 − Spec
Simpson index
Shannon entropy
Brillouin index
1 − Gini
RS_P[1 − Sc]
RS_G[1 − Sc]
RS_P[1/Sc]
RS_G[1/Sc]
RS_P[1 − So]
RS_G[1 − So]
RS_P[1/So]
RS_G[1/So]
Hill-type measure
Coherence
BC
CC
AS
32
21
41
206
3
101
170
168
201
192
203
181
175
189
207
203
197
203
209
123
12
88
104
166
165
106
71
112
74
71
93
217
213
210
180
150
170
202
149
213
214
17
3
8
LIS
185
140
140
173
88
137
121
83
81
73
3
9
4
12
109
95
37
37
9
19
30
139
138
LAW
177
MATH
186
133
213
169
133
182
203
141
141
97
80
34
39
52
100
39
4
43
34
152
29
100
89
214
223
221
224
201
223
224
224
222
221
79
42
124
224
223
180
216
79
224
68
105
223
253
Estudios de ciencias cuantitativas
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
1
1
2
3
9
1
7
6
0
8
5
8
q
s
s
_
a
_
0
0
0
1
1
pag
d
.
/
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Consistency and validity of interdisciplinarity measures
Más, the distributions in Figure 3 should also be taken into consideration when we com-
pare the rankings of the five SCs. Some measures have dense distributions in an extremely
short interval. Most rankings based on such measures are therefore not robust, as they depend
on the third or even fourth decimal and may thus be hard to explain.
5. DISCUSIÓN
Based on our analyses, three issues are worth further discussion. Primero, we discuss the defini-
tions and attributes of IDR, then we focus on interdisciplinarity measures and their operatio-
nalization, and finally we consider their policy implications.
5.1. Attributes of IDR
As already indicated, previous studies have argued that the conception of IDR is ambiguous
(p.ej., Rafols et al., 2012). We claim that the same holds for its definitions in scientometric
estudios. Based on our review, we found that diversity has been widely seen as the essential
and necessary attribute for measuring interdisciplinarity. The attribute of coherence is mainly
seen as supplementary and are often ignored in practice. Por lo tanto, most interdisciplinarity
measures in scientometric studies use the diversity attribute and most often this attribute
solo. Por ejemplo, Steele and Stier (2000) state that “[i]n effect, we treat diversity as a proxy
measure of interdisciplinarity” (pag. 477). This raises the important question of whether diver-
sity in itself is sufficient to capture the concept of interdisciplinarity. We are skeptical.
Generally, we are concerned about the definitions and simplistic indicators used to mea-
sure the multidimensional concept of IDR. We especially question to the extent to which di-
versity is an appropriate attribute that in itself can encompass and reflect the concept. During
our review, it became clear that the definition of IDR by the US Committee on Facilitating
Interdisciplinary Research and Committee on Science was frequently referred to. According
lo (National Academies, 2004, pag. 2):
[i]nterdisciplinary research (IDR) is a mode of research by teams or individuals that inte-
grates information, datos, técnicas, herramientas, perspectives, conceptos, and/or theories from
two or more disciplines or bodies of specialized knowledge to advance fundamental un-
derstanding or to solve problems whose solutions are beyond the scope of a single disci-
pline or area of research practice.
The definition simply means that IDR would require the integration of knowledge from two
or more disciplines. Por lo tanto, we argue that high diversity does not seem to be either a nec-
essary or sufficient attribute for measuring interdisciplinarity.
En nuestra opinión, much more work should be done in order to define the concept, identify key
atributos, and subsequently empirically examine the construct validity of the measures or
composite measures developed to assess interdisciplinarity. Some researchers have indeed ar-
gued that the concept of IDR is multidimensional and hence its attributes should be portrayed
using various measures (p.ej., Leydesdorff & Rafols, 2011; Rafols & Meyer, 2010; Sugimoto &
Weingart, 2014). The recent report from Digital Science states that “no single indicator can
unequivocally identify and monitor IDR activity and no present proxy is a demonstrably
satisfactory management tool on its own” (pag. 9). Our results support these claims inasmuch
as we demonstrate that seemingly similar measures produce different results and are sensitive
to levels of analysis. This is a serious breach of construct validity.
Estudios de ciencias cuantitativas
254
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
1
1
2
3
9
1
7
6
0
8
5
8
q
s
s
_
a
_
0
0
0
1
1
pag
d
.
/
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Consistency and validity of interdisciplinarity measures
Few studies have examined theoretical frameworks around interdisciplinarity and linked it
to measurement. The main contribution comes from Rafols and colleagues (p.ej., Rafols, 2014;
Rafols & Meyer, 2010; Rafols et al., 2012). A theoretical framework is needed in order to out-
line the dimensions of interdisciplinarity and relations between attributes. Desafortunadamente, semejante
an important endeavor has not received much attention. En cambio, “novel” measures, mostly
based on diversity, are being proposed continuously. Their relevance, validity, and resem-
blance to other measures are most often overlooked. In this context, we believe that the dis-
cussion on which attributes are essential for depicting the nature of interdisciplinarity, también
as the similarity of measures, is necessary and essential.
5.2.
Interdisciplinarity Measures and Operationalization
Based on our analyses, we found that even measures with a supposedly similar focus can pro-
duce contradictory results when measuring interdisciplinarity using WoS SCs. Inconsistency in
our results implies that some measures are problematic for the purpose of describing interdis-
ciplinarity, as they do not capture their target attribute. Por ejemplo, 1 − Pratt and 1 − Spec are
expected to be consistent. But our results show that this is not the case when applied to WoS
SC. We do not imply that these measures are mathematically wrong, but their validity as mea-
sures of interdisciplinarity is doubtful and should be carefully considered.
We also found that the justification for the use of a measure is not always convincing. por ejemplo-
amplio, the Brillouin index is an entropy-based indicator. When it was introduced as an interdisci-
plinarity measure (p.ej., Steele & Stier, 2000; Chang & Huang, 2012; Huang & Chang, 2012), es
relation to Shannon entropy was not discussed. Shannon entropy was already used to measure
interdisciplinarity, so the supposed merits of the Brillouin index compared to Shannon’s entropy
debería, por supuesto, have been explained in these studies. Our results obtained from Shannon and
Brillouin are almost perfectly correlated, which suggests that at least one of them is superfluous. Nosotros
therefore argue that the introduction and creation of new measures should aim to improve the va-
lidity and accuracy of measurement, instead of constantly introducing new and perhaps nearly
identical measures. Following the suggestions given by Waltman (2016) on citation impact indica-
tores, we reckon that given the large number of interdisciplinarity measures that already exists, es
not necessary to provide more indicators, especially not indicators relying on the diversity attribute,
unless a novel measure has some convincing new merits in relation to validity and accuracy.
The operationalization of interdisciplinarity measures in scientometric studies is relatively
chaotic. The report by Digital Science (2016) shows that the degree of interdisciplinarity is
influenced by the choice of data sources and classification systems. The present study further
demonstrates the tangled and unsustainable situation of measuring interdisciplinarity, con
inconsistent outcomes generated by seemingly similar measures.
To be more specific, the present study examines various combinations of RS, demonstrating
that they lead to quite different results. Desafortunadamente, we see that important details have per-
sistently been overlooked in previous studies, por ejemplo, explanations for the choice of co-
sine formulas (p.ej., Portero & Rafols, 2009). Because substantial differences may result from
such choices, we suggest that researchers should provide sufficient details on the operationa-
lization of their interdisciplinarity measures and preferably perform sensitivity and robustness
analiza.
También, measures with extremely narrow distributions or without boundary have little practi-
cal use. As shown, interdisciplinarity measures such as Shannon entropy and the Hill-type
indicator do not have an obvious domain of values. Como consecuencia, it is difficult to evaluate
to what degree a unit is interdisciplinary according to such measures.
Estudios de ciencias cuantitativas
255
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
1
1
2
3
9
1
7
6
0
8
5
8
q
s
s
_
a
_
0
0
0
1
1
pag
d
.
/
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Consistency and validity of interdisciplinarity measures
5.3.
Interdisciplinarity and Policy Implication
The importance of IDR has been widely acknowledged. Many studies argue that it could solve
complex problems and promote scientific developments and innovations (Gibbons et al., 1994;
see also Rafols et al., 2012; Hollingsworth & Hollingsworth, 2000; Lowe & Phillipson, 2006). Como un
consequence, “funding agencies in many developed countries are considering enhancing IDR as
a topic of priority (Bordones, Morillo, & Gómez, 2004; Rinia, 2007). Por ejemplo, research-funding
agencies such as NSF, Research Councils UK (RCUK), NSFC, and Swedish Research Council (VR)
take the promotion of interdisciplinary research an essential task” (Wang, 2016, pag. 21).
Por un lado, we observe the enthusiasm of research-funding agencies to encourage and
finance IDR. Por otro lado, we found that the interdisciplinarity measures in scientometric
studies are confusing because they lack validity. The degree of interdisciplinarity for a unit of
analysis most likely varies with the choice of measure, data source, and classification system. Para
a set of units, different measures will most likely produce different rankings between the units.
Obviamente, this is untenable and of great concern in science policy and research evaluation. Es
simply too easy to intentionally influence the outcomes of interdisciplinarity measures. Hay
too many researcher degrees of freedom (Wicherts et al., 2016).
Interdisciplinarity measures in scientometric studies use publications and citation relations
as the data source to identify IDR. En otras palabras, we understand interdisciplinarity from a
bibliometric perspective. Sin embargo, we are indifferent to how other stakeholders, like policy-
fabricantes, understand IDR. Interdisciplinarity measures in scientometric studies may be able to
deal with some aspects of interdisciplinarity but not others. Por eso, it is important and neces-
sary to thoroughly state which aspects (atributos) of interdisciplinarity they actually depict
when reporting studies of interdisciplinarity. Además, it is also important not to be blinded
by measures relying on bibliometric methods. They tend to produce a “tunnel vision” where
this is the only way to measure interdisciplinarity.
6. CONCLUSIONS
The present article aims to systematically examine the consistency and relation between inter-
disciplinarity measures based on bibliometric methods. We first examined these measures fo-
cusing on the WoS SCs. Based on correlations and clustering, we found that the 23 reviewed
measures can be roughly classified into two groups depending on whether or not a dissimilar-
ity matrix is used. Measures in the same cluster tend to have fairly strong mutual correlations,
but are weakly correlated with the measures in other groups. Sin embargo, although some mea-
sures are supposed to measure similar aspects, they nevertheless turn out to be inconsistent
(p.ej., 1 − Pratt and 1 − Spec). Histograms showing the distribution of interdisciplinarity
values over different WoS SCs reveal tight distributions for some measures, as their values
are concentrated in limited intervals (p.ej., CC and AS). These measures may be problematic
when used in practice. We therefore conclude that the degree of interdisciplinarity for a
unit of analysis is strongly dependent on the choice of measures.
The findings in our study complement the conclusions in the report from Digital Science
(2016): “choice of data, methodology and indicators can produce seriously inconsistent results
despite a common set of disciplines and countries” (pag. 2). Our results further demonstrate that
inconsistent and even conflicting findings can come out of analyses based on the same data
source and the same classification. The current state of interdisciplinarity measurement is con-
fusing and unsustainable. Interdisciplinarity is a multidimensional concept and measures
should reflect these dimensions through various different attributes either as single or
Estudios de ciencias cuantitativas
256
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
1
1
2
3
9
1
7
6
0
8
5
8
q
s
s
_
a
_
0
0
0
1
1
pag
d
/
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Consistency and validity of interdisciplinarity measures
composite indicators. Sin embargo, we find that the definitions of interdisciplinarity are quite sim-
ilar and hardly multidimensional. This fact makes it even more complicated to interpret the
inconsistent values we obtained from these presumably similar measures.
The validity and robustness of interdisciplinarity studies using bibliometric methods should be
questioned. As it is, measures and their values are inconsistent and non-robust. This can lead to an
untenable situation where the choice of (arbitrary) measures determines the degree of interdisci-
plinarity, but not the underlying nature of research which they are supposed to characterize. Nosotros
therefore suggest that future studies on interdisciplinarity focus more upon the theoretical and
measurement frameworks, and put more effort into examining the validity and relations between
the definition and the use of measures. Además, we recommend that we simply stop using the
current interdisciplinarity measures in policy studies, as they have no warrant.
EXPRESIONES DE GRATITUD
The authors would like to thank Ismael Rafols, Juego Waltman, Gaël Dubus, and reviewers for
their valuable comments and suggestions.
CONTRIBUCIONES DE AUTOR
Qi Wang: conceptualization, data curation, formal analysis, investigación, methodology, soft-
mercancía, validation, visualization, writing—original draft. Jesper Wiborg Schneider: conceptual-
ización, methodology, resources, supervision, writing—review & edición.
CONFLICTO DE INTERESES
Los autores no tienen intereses en competencia.
INFORMACIÓN DE FINANCIACIÓN
No funding has been received.
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
1
1
2
3
9
1
7
6
0
8
5
8
q
s
s
_
a
_
0
0
0
1
1
pag
d
/
.
DISPONIBILIDAD DE DATOS
The raw bibliometric data were obtained from Clarivate Analytics’ Web of Science database. A
Web of Science license is required to access the data. The values of the different interdisciplinarity
measures at the level of Web of Science subject categories and the table of research areas used to
measure p_outside are available at https://kth.box.com/s/zbcfmvpuhmhwvl8ql1u522snyht34y50.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
REFERENCIAS
Abbott, A. (2001). Chaos of Disciplines. chicago, IL: Universidad de
Chicago Press.
Aboelela, S. w., larson, MI., Bakken, S., Carrasquillo, o., Formicola,
A., Glied, S. A., …, Gebbie, k. METRO. (2007). Defining interdisciplin-
ary research: Conclusions from a critical review of the literature.
Health Services Research, 42(1p1), 329–346.
Bordones, METRO., Morillo, F., & Gómez, I. (2004). Analysis of cross-disciplinary
research through bibliometric tools. In Moed H. F., Glänzel W., &
Schmoch U., (Editores.) Handbook of Quantitative Science and Tech-
nology Research. (páginas. 437–456). Saltador, Dordrecht.
Brillouin, l. (1956). Science and Information Theory. Nueva York:
Prensa académica.
Chang, Y. w., & Huang, METRO. h. (2012). A study of the evolution of
interdisciplinarity in library and information science: Using three
bibliometric methods. Journal of the Association for Information
Science and Technology, 63(1), 22–33.
Digital Science. (2016). Interdisciplinary research: Methodologies for
identification and assessment. Available at https://www.mrc.ac.uk/
documents/pdf/assessment-of-interdisciplinary-research/
Hombre libre, l. C. (1977). A set of measures of centrality based on be-
tweenness. Sociometry, 1(40), 35–41.
Gibbons, METRO., Limoges, C., Nowotny, h., Schwartzman, S., Scott,
PAG., & Trow, METRO. (1994). The New Production of Knowledge: El
Dynamics of Science and Research in Contemporary Societies.
Londres: Sage.
Hollingsworth, r., & Hollingsworth, E.J., (2000). Major discoveries and
biomedical research organizations: Perspectives on interdisciplinarity,
nurturing leadership, and integrated structure and cultures. In P.
Estudios de ciencias cuantitativas
257
Consistency and validity of interdisciplinarity measures
Weingart & norte. Stehr (Editores.), Practising Interdisciplinarity. toronto:
University of Toronto Press, páginas. 215–244.
Colina, METRO. oh. (1973). Diversity and evenness: A unifying notation and
its consequences. Ecology, 54(2), 427–432.
Huang, METRO. h., & Chang, Y. W.. (2012). A comparative study of in-
terdisciplinary changes between information science and library
ciencia. cienciometria, 91(3), 789–803.
Huutoniemi, K., Klein, j. T., Bruun, h., & Hukkinen, j. (2010).
Analyzing interdisciplinarity: Typology and indicators. Investigación
Política, 39, 79–88.
Jensen, PAG., & Lutkouskaya, k. (2014). The many dimensions of lab-
oratories’ interdisciplinarity. cienciometria, 98(1), 619–631.
Leinster, T., & Cobbold, C. A. (2012). Measuring diversity: The im-
portance of species similarity. Ecology, 93(3), 477–489.
Levitt, j. METRO., Thelwall, METRO., & Oppenheim, C. (2011). Variations be-
tween subjects in the extent to which the social sciences have
become more interdisciplinary. Journal of the American Society
for Information Science and Technology, 62(6), 1118–1129.
Leydesdorff, l.. (2007). Betweenness centrality as an indicator of the in-
terdisciplinarity of scientific journals. Journal of the American Society
for Information Science and Technology, 58(9), 1303–1319.
Leydesdorff, l., & Probst, C. (2009). The delineation of an interdis-
ciplinary specialty in terms of a journal set: The case of commu-
nication studies. Journal of the American Society for Information
Science and Technology, 60(8), 1709–1718.
Leydesdorff, l., & Rafols, I. (2011). Indicators of the interdisciplin-
arity of journals: Diversity, centrality, and citations. Diario de
Informetrics, 5(1), 87–100.
Lowe, PAG., & Phillipson, j. (2006). Reflexive interdisciplinary research:
The making of a research programme on the rural economy and
land use. Journal of Agricultural Economics, 57(2), 165–184.
Morillo, F., Bordones, METRO., & Gómez, I. (2001). An approach to in-
terdisciplinarity through bibliometric indicators. cienciometria,
51(1), 203–222.
Morillo, F., Bordones, METRO., & Gómez, I. (2003). Interdisciplinarity in
ciencia: A tentative typology of disciplines and research areas.
Journal of the American Society for Information Science and Tech-
nología, 54(13), 1237–1249.
Mugabushaka, A. METRO., Kyriakou, A., & Papazoglou, t. (2016).
Bibliometric indicators of interdisciplinarity: The potential of
the Leinster–Cobbold diversity indices to study disciplinary di-
versity. cienciometria, 107(2), 593–607.
National Academies. (2004). Committee on Science, Ingeniería,
and Public Policy (COSEPUP) Committee on Facilitating Inter-
disciplinary Research. Washington, corriente continua: National Academies
Prensa.
Pierce, S. j. (1999). Boundary crossing in research literatures as a means
of interdisciplinary information transfer. Journal of the American
Society for Information Science, 50(3), 271–279.
Portero, A., & Rafols, I. (2009). Is science becoming more interdisci-
plinary? Measuring and mapping six research fields over time.
cienciometria, 81(3), 719–745.
Portero, A. l., Roessner, j. D., cohen, A. S., & Perreault, METRO. (2006).
Interdisciplinary research: Significado, metrics and nurture. Investigación
Evaluation, 15(3), 187–195.
Portero, A. l., cohen, A. S., Roessner, j. D., & Perreault, METRO. (2007).
Measuring researcher interdisciplinarity. cienciometria, 72(1),
117–147.
Portero, A. l., Roessner, D. J., & Heberger, A. mi. (2008). How in-
terdisciplinary is a given body of research? Research Evaluation,
17(4), 273–282.
Portero, A. l., & Youtie, j. (2009). How interdisciplinary is nanotechnology?
Revista de investigación de nanopartículas, 11(5), 1023–1041.
Portero, A. l. D. MI., & Chubin, D. (1985). An indicator of cross-
disciplinary research. cienciometria, 8(3–4), 161–176.
Pratt, A. D. (1977). A measure of class concentration in biblio-
métrica. Journal of the American Society for Information Science,
28(5), 285–292.
Rafols, I. (2014). Knowledge integration and diffusion: Measures and
mapping of diversity and coherence. In Y. Ding, R. Rousseau, &
D. Wolfram (Editores.). Measuring Scholarly Impact: Methods and
Practice. Dordrecht: Saltador.
Rafols, I., Leydesdorff, l., O’Hare, A., Nightingale, PAG., & Stirling, A.
(2012). How journal rankings can suppress interdisciplinary re-
buscar: A comparison between innovation studies and business
& management. Política de investigación, 41(7), 1262–1282.
Rafols, I., & Meyer, METRO. (2007). How cross-disciplinary is bionano-
tecnología? Explorations in the specialty of molecular motors.
cienciometria, 70(3), 633–650.
Rafols, I., & Meyer, METRO. (2009). Diversity and network coherence as
indicators of interdisciplinarity: Case studies in bionanoscience.
cienciometria, 82(2), 263–287.
Rafols, I., & Meyer, METRO. (2010). Diversity and network coherence as
indicators of interdisciplinarity: Case studies in bionanoscience.
cienciometria, 82(2), 263–287.
Rodríguez, j. METRO. (2017). Disciplinarity and interdisciplinarity in cita-
tion and reference dimensions: Knowledge importation and expor-
tation taxonomy of journals. cienciometria, 110(2), 617–642.
Rinia, mi. j. (2007). Measurement and Evaluation of Interdisciplinary
Research and Knowledge transfer. Doctoral dissertation, Center
for Science and Technology Studies (CWTS), Faculty of Social
and Behavioral Sciences, Universidad de Leiden.
Rosenfield, PAG. l. (1992). The potential of transdisciplinary research for
sustaining and extending linkages between the health and social
sciences. Social Science and Medicine, 35(11), 1343–1357.
Salton, GRAMO., & McGill, METRO. j. (1983). Introduction to Modern Infor-
mation Retrieval. Nueva York: McGraw-Hill, Cª.
Schneider, j. w., & Borlund, PAG. (2007a). Matrix comparison, Parte 1:
Motivation and important issues for measuring the resemblance be-
tween proximity measures or ordination results. Journal of the
American Society for Information Science and Technology, 58(11),
1586–1595.
Schneider, j. w., & Borlund, PAG. (2007b). Matrix comparison, Parte 2:
Measuring the resemblance between proximity measures or ordi-
nation results by use of the Mantel and Procrustes statistics.
Journal of the American Society for Information Science and
Tecnología, 58(11), 1596–1609.
shannon, C. mi. (1948). A mathematical theory of communication.
Bell System Technical Journal, 27(3), 379–423.
shannon, C. mi. (2001). A mathematical theory of communication. ACM
SIGMOBILE Mobile Computing and Communications Review, 5(1),
3–55.
silva, F. NORTE., rodrigues, F. A., Oliveira, oh. NORTE., & Costa, l. D. F.
(2013). Quantifying the interdisciplinarity of scientific journals
and fields. Journal of Informetrics, 7(2), 469–477.
Simpson, mi. h. (1949). Measurement of diversity. Naturaleza, 163,
688.
Soós, S., & Kampis, GRAMO. (2012). Beyond the basemap of science:
Mapping multiple structures in research portfolios: Evidencia de
Hungary. cienciometria, 93(3), 869–891.
Steele, t. w., & Stier, j. C. (2000). The impact of interdisciplinary
research in the environmental sciences: A forestry case study.
Journal of the American Society for Information Science, 51(5),
476–484.
Sugimoto, C. r., & Weingart, S. (2015). The kaleidoscope of disci-
plinarity. Journal of Documentation, 71(4), 775–794.
Estudios de ciencias cuantitativas
258
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
1
1
2
3
9
1
7
6
0
8
5
8
q
s
s
_
a
_
0
0
0
1
1
pag
d
/
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Consistency and validity of interdisciplinarity measures
Tijssen, R. j. W.. (1992). A quantitative assessment of interdisciplinary
structures in science and technology: Co-classification analysis of
energy research. Política de investigación, 21(1), 27–34.
Van den Besselaar, PAG., & Leydesdorff, l. (1996). Mapping change in
scientific specialties: A scientometric reconstruction of the devel-
opment of artificial intelligence. Journal of the American Society
for Information Science, 47(6), 415–436.
Wagner, C. S., Roessner, j. D., Bobb, K., Klein, j. T., Boyack, k. w.,
Keyton, J., … Börner, k. (2011). Approaches to understanding
and measuring interdisciplinary scientific research (IDR): A re-
view of the literature. Journal of Informetrics, 5(1), 14–26.
waltman, l. (2016). A review of the literature on citation impact
indicators. Journal of Informetrics, 10(2), 365–391.
Wang, J., Thijs, B., & Glänzel, W.. (2015). Interdisciplinarity and
impacto: Distinct effects of variety, balance, and disparity. PloS
Uno, 10(5), e0127298.
Wang, q. (2016). Studies in the dynamics of science: Exploring
emergence, clasificación, and interdisciplinarity. (Doctoral dis-
sertación, KTH Royal Institute of Technology). Retrieved from
http://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-184724
Wang, P., & waltman, l. (2016). Large-scale analysis of the accu-
racy of the journal classification systems of Web of Science and
Scopus. Journal of Informetrics, 10(2), 347–364.
Wicherts, j. METRO., Veldkamp, C. l. S., Augusteijn, h. mi. METRO., Bakker,
METRO., van Aert, R. C. METRO., & van Assen, METRO. A. l. METRO. (2016). Degrees
of freedom in planning, running, analyzing, and reporting psy-
chological studies: A checklist to avoid p-hacking. Fronteras en
Psicología, 7, 1832.
zhang, l., Rousseau, r., & Glänzel, W.. (2016). Diversity of references
as an indicator for interdisciplinarity of journals: Taking similarity be-
tween subject fields into account. Journal of the American Society
for Information Science and Technology, 67(5), 1257–1265.
zhou, q. J., Rousseau, r., Cual, l. y., yue, T., & Cual, GRAMO. l. (2012).
A general framework for describing diversity within systems and
similarity between systems with applications in informetrics.
cienciometria, 93(3), 787–812.
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
1
1
2
3
9
1
7
6
0
8
5
8
q
s
s
_
a
_
0
0
0
1
1
pag
d
/
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Estudiar
Steele and Stier (2000)
Morillo et al. (2001)
Morillo et al. (2003)
Leydesdorff (2007)
APPENDIX
Table A1. Definitions of IDR from a bibliometric perspective
Definición
“In effect, we treat diversity as a proxy measure of interdisciplinarity. Theoretical support
for this approach is provided by Gibbons, Limoges, Nowotny, Schwartzman, Scott,
& Trow (1994), who asserted that, by definition, interdisciplinarity involves heterogeneity,
specifically a diversity of individuals, habilidades, experiencias, institutions, linkages, y
locations.”
(pag. 477)
“Strictly speaking, we consider “multidisciplinarity” as a basic situation in which
elements from different disciplines are present, whilst “interdisciplinarity” is a
more advanced stage of the relationship between disciplines in which integration
between them is attained.” (p.204)
“Interdisciplinary research leads to the creation of a theoretical, conceptual, y
methodological identity, so more coherent and integrated results are obtained.”
(pag. 1237)
“… it occurred to me that the interdisciplinarity of journals corresponds with their
visible position in the vector space” (pag. 1305)
Porter et al. (2007);
“We apply the following definition, based on a National Academies report: Interdisciplinary
Porter et al. (2008); Wang (2016);
Zhang et al. (2016)
investigación (IDR) is a mode of research by teams or individuals that integrates
(cid:129) perspectives/concepts/theories and/or
(cid:129) tools/techniques and/or
(cid:129) information/data
from two or more bodies of specialized knowledge or research practice.”
(Porter et al., 2007, pag. 119)
Estudios de ciencias cuantitativas
259
Consistency and validity of interdisciplinarity measures
Table A1. (continued )
Estudiar
Definición
Rafols and Meyer (2009)
“Thus, the process of integrating different bodies of knowledge rather than
transgression of disciplinary boundaries per se, has been identified as the key
aspect of so-called ‘interdisciplinary research.’ (National Academies, 2005)."
(pag. 264)
Portero & Rafols (2009)
“This report operationally defined interdisciplinary research as:
a mode of research by teams or individuals that integrates perspectives/
concepts/theories and/or
tools/techniques and/or
information/data from two or more bodies of knowledge or research practice.” (pag. 720)
“Understood as knowledge integration, interdisciplinarity is not the opposite of
specialization. … Our investigation here does not concern the degree of topic
specialization of research but the degree that it relies on distinct.” (pag. 720)
Leydesdorff & Rafols (2011)
“Furthermore, interdisciplinarity may be a transient phenomenon. As a new
specialty emerges, it may draw heavily on its mother disciplines/specialties, pero
as it matures a set of potentially new journals can be expected to cite one another
cada vez más, and thus to develop a type of closure that is typical of “disciplinarity”
(Van den Besselaar & Leydesdorff, 1996). Interdisciplinarity, sin embargo, may mean
something different at the top of the journal hierarchy (as in the case of Science
and Nature) than at the bottom, where one has to draw on different bodies of
knowledge for the sake of the application (p.ej., in engineering). Similarmente, en el
clinic one may be more inclined to integrate knowledge from different specialties
at the bedside than a laboratory where the focus is on specialization and
refinement.” (pag. 88)
“A common feature of interdisciplinarity as it is manifested in a variety of research
activities is the transfer of information across disciplines (Portero, Roessner, cohen,
& Perreault, 2006). Pierce (1999) grouped interdisciplinary information transfer into
three types: the citation of references from different disciplines, the co-authoring
of articles by researchers from different disciplines, and the publishing of works
within other disciplines. Such transfer implies that the degree of interdisciplinarity
of a specific discipline can be determined by analyzing the discipline distribution
of references and co-authors in publications.” (pag. 22)
“The concept of interdisciplinarity has been discussed by many researchers (Huutoniemi, Klein,
Bruun, & Hukkinen, 2010; Leydesdorff and Probst, 2009; Rosenfield, 1992; Tijssen, 1992),
and can be defined as the use of knowledge, methods, técnicas, and devices as a result
of scientific activities from other fields (Tijssen, 1992)." (pag. 790)
“We propose to investigate interdisciplinarity from two perspectives, each of which
we claim has more general applicability. The first is by means of the widely used
conceptualisation of interdisciplinarity as knowledge integration (National
Academies, 2004; Porter et al., 2006), which is perceived as crucial for innovation
or solving social problems. The second is by conceptualising interdisciplinarity as
a form of research that lies outside or in between established practices, es decir. en
terms of intermediation (Leydesdorff, 2007)." (p.1265)
“In a sense, this movement brought science closer to the paradigm adopted by Greek
philosophers who treated Nature as a landscape of knowledge glued together in an
indivisible discipline. No es sorprendente, in recent years new areas have been established
Chang & Huang (2012)
Huang & Chang (2012)
Rafols et al. (2012)
Silva et al. (2013)
Estudios de ciencias cuantitativas
260
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
1
1
2
3
9
1
7
6
0
8
5
8
q
s
s
_
a
_
0
0
0
1
1
pag
d
.
/
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Consistency and validity of interdisciplinarity measures
Table A1. (continued )
Estudiar
Soós & Kampis (2012)
Wang et al. (2015)
Rodríguez (2017)
Definición
with this interdisciplinary character, as is the case of nanoscience and nanotechnology,
in addition to new disciplines arising from the merging of two or more areas, como
computational biology and biomolecular physics.” (pag. 469)
“… diversity measures are clearly associated with the degree multidisciplinarity. Sobre el
other hand, the notion of interdisciplinary research (IDR) is decomposed into two
differing perspectives: on one account, IDR is conceived as knowledge integration,
an indicator of which is the degree of overall interrelatedness of the units of analysis
constituting a portfolio.” (pag. 871).
“On the other account, sin embargo, interdisciplinarity is viewed as intermediation between
knowledge domains, embodied in publication sets positioned between more established
clusters of journals or fields.” (pag. 871)
“For interdisciplinary research, integrating knowledge from more disciplines contributes
to potential more broadly useful outcomes.” (pag. 11)
“Finally, interdisciplinarity entails the integration of ‘separate disciplinary data, methods,
herramientas, concepts and theories in order to create a holistic view or common understanding
of a complex issue, question or problem’.” (pag. 619)
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
1
1
2
3
9
1
7
6
0
8
5
8
q
s
s
_
a
_
0
0
0
1
1
pag
d
/
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Estudios de ciencias cuantitativas
261
q
tu
a
norte
t
i
t
a
i
t
i
v
mi
S
C
mi
norte
C
mi
S
tu
d
mi
s
t
i
1. p_multi
1
1.00
Table A2 Spearman’s correlation coefficients of interdisciplinarity measures
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22 23
2. p_outside
0.78 1.00
3. pro
0.45 0.44 1.00
4. d_links
0.64 0.66 0.59 1.00
5. 1 − Pratt
−0.07 0.14 0.44 0.27 1.00
6. 1 − Spec
0.82 0.78 0.47 0.63 0.05 1.00
7. Simpson
0.26 0.40 0.78 0.54 0.69 0.43 1.00
8. shannon
0.17 0.35 0.59 0.43 0.78 0.36 0.92 1.00
9. Brillouin
0.19 0.36 0.59 0.44 0.75 0.38 0.92 1.00 1.00
10. 1 − Gini
0.09 0.27 0.54 0.40 0.85 0.20 0.77 0.84 0.83 1.00
11. RS_P[1 − Sc] 0.15 0.31 0.09 0.21 0.41 0.18 0.18 0.26 0.24 0.37 1.00
12. RS_G[1 − Sc] 0.16 0.27 0.04 0.14 0.37 0.17 0.11 0.21 0.19 0.33 0.91 1.00
13. RS_P[1/Sc]
0.15 0.22 0.08 0.11 0.28 0.15 0.07 0.13 0.12 0.25 0.83 0.90 1.00
C
oh
norte
s
i
s
t
mi
norte
C
y
a
norte
d
v
a
yo
i
d
i
t
y
oh
F
i
norte
t
mi
r
d
i
s
C
i
pag
yo
i
norte
a
r
i
t
y
metro
mi
a
s
tu
r
mi
s
0.26 0.35 0.24 0.25 0.42 0.31 0.29 0.34 0.33 0.47 0.77 0.87 0.88 1.00
14. RS_G[1/Sc]
15. RS_P[1 − So] −0,10 0.08 0.28 0.30 0.56 −0,02 0.51 0.54 0.53 0.68 0.13 −0.03 −0.11 −0.05 1.00
16. RS_G[1 − So] −0,02 0.18 0.35 0.36 0.73 0.09 0.62 0.69 0.68 0.84 0.41 0.33 0.19 0.31 0.87 1.00
17. RS_P[1/So]
−0,02 0.13 0.24 0.15 0.43 −0,03 0.23 0.25 0.24 0.42 0.65 0.65 0.75 0.62 0.28 0.42 1.00
18. RS_G[1/So]
0.16 0.34 0.36 0.29 0.62 0.28 0.51 0.58 0.58 0.68 0.66 0.71 0.68 0.84 0.25 0.56 0.67 1.00
19. Hill type
0.16 0.27 0.04 0.14 0.37 0.17 0.11 0.21 0.19 0.33 0.91 1.00 0.90 0.87 −0,03 0.33 0.65 0.71 1.00
20. coherencia
0.25 0.39 0.38 0.36 0.55 0.25 0.42 0.44 0.42 0.49 0.81 0.77 0.66 0.69 0.19 0.47 0.60 0.67 0.77 1.00
0.00 0.11 −0.17 −0.26 0.06 0.14 0.10 0.30 0.32 0.10 0.03 0.09 0.06 0.10 −0.09 −0.02 −0.03 0.19 0.09 −0,06 1.00
0.21 0.21 −0.10 −0.04 −0.28 0.38 0.09 0.14 0.17 −0.13 −0.30 −0.30 −0.33 −0.19 −0.25 −0.26 −0.53 −0.13 −0.30 −0.39 0.55 1.00
−0,04 0.09 0.19 0.17 0.31 0.13 0.49 0.58 0.59 0.44 −0.29 −0.38 −0.47 −0.31 0.58 0.46 −0.22 0.02 −0.38 −0.24 0.33 0.40 1.00
21. BC
22. CC
23. AS
2
6
2
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
1
1
2
3
9
1
7
6
0
8
5
8
q
s
s
_
a
_
0
0
0
1
1
pag
d
.
/
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Consistency and validity of interdisciplinarity measures
Figure A1. Heatmap of interdisciplinarity measures.
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
1
1
2
3
9
1
7
6
0
8
5
8
q
s
s
_
a
_
0
0
0
1
1
pag
d
.
/
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Estudios de ciencias cuantitativas
263