RESEARCH ARTICLE - IA de Investigación especializada en el MIT

ARTÍCULO DE INVESTIGACIÓN

Productivity and interdisciplinary impacts
of Organized Research Units

Daniel J. Hicks1,2

1Data Science Initiative, UC Davis
2Cognitive and Information Sciences, UC Merced

Palabras clave: interdisciplinarity, Organized Research Unit, productivity impacts, research evaluation,
topic models

ABSTRACTO

Organized Research Units (ORUs) are nondepartmental units utilized by U.S. investigación
universities to support interdisciplinary research initiatives, among other goals. This study
examined the impacts of ORUs at one large public research university, la Universidad de
California, davis (UC Davis), using a large corpus of journal article metadata and abstracts
for both faculty affiliated with UCD ORUs and a comparison set of other faculty. Usando
regression analysis, I find that ORUs appeared to increase the number of coauthors of affiliated
faculty, but did not appear to directly affect publication or citation counts. Próximo, I frame
interdisciplinarity in terms of a notion of discursive space, and use a topic model approach to
situate researchers within this discursive space. The evidence generally indicates that ORUs
promoted multidisciplinarity rather than interdisciplinarity. In the conclusion, drawing on
work in philosophy of science on inter- and multidisciplinarity, I argue that multidisciplinarity
is not necessarily inferior to interdisciplinarity.

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

INTRODUCCIÓN

While there is no formal definition, Organized Research Units (ORUs) are nondepartmental
organizational units utilized by U.S. research universities to support clusters of researchers
working on related topics. ORUs are typically organized internally—many researchers are uni-
versity faculty; other researchers and staff are university employees—but funded externally.
Examples might include museums, observatories, research stations, some large physical science
labs (Geiger, 1990, pag. 6); the numerous small “centers” and “labs” containing just one or two
faculty and limited external funding are usually not counted as ORUs (Stahler & Tash, 1994,
pag. 542). As the examples suggest, some ORUs support research by providing an institutional
home for highly capital-intensive research activities such as specimen collections or the
maintenance of large or complex instruments. ORUs can also serve as focal points for recruiting
external funding, either by demonstrating to funders that the university is actively engaged in a
particular area of research (Geiger, 1990, pag. 9) or by providing, Por ejemplo, dedicated support
staff for the grant-writing and -administering cycle.

Sin embargo, at least since Geiger (1990), research policy scholars have theorized ORUs as
key sites for bridging barriers between disciplines (interdisciplinarity) and between academic
and social interests (extradisciplinarity) (Etzkowitz & Kemelgor, 1998; Geiger, 1990; Sá &

un acceso abierto

diario

Citación: Hicks, D. j. (2021).
Productivity and interdisciplinary
impacts of Organized Research Units.
Estudios de ciencias cuantitativas, 2(3),
990–1022. https://doi.org/10.1162/qss_a
_00150

DOI:
https://doi.org/10.1162/qss_a_00150

Revisión por pares:
https://publons.com/publon/10.1162
/qss_a_00150

Supporting Information:
https://doi.org/10.1162/qss_a_00150

Recibió: 22 Febrero 2021
Aceptado: 5 Agosto 2021

Autor correspondiente:
Daniel J. Hicks
hicks.daniel.j@gmail.com

Editor de manejo:
Juego Waltman

Derechos de autor: © 2021 Daniel J. Hicks.
Publicado bajo Creative Commons
Atribución 4.0 Internacional
(CC POR 4.0) licencia.

La prensa del MIT

Interdisciplinary impacts of Organized Research Units

Oleksiyenko, 2011; verano, 1994). Eso es, it is thought that ORUs support research not just
materially (with resources and support staff ) but also culturally (creating a certain kind of
research community).

The aim of the current project is to examine the impact of ORUs at one large public
research university—the University of California, davis (UC Davis)—in terms of both traditional
bibliometric notions of productivity (papers written, citations received) as well as interdisciplin-
arity. En otras palabras, have the ORUs at UC Davis promoted research productivity? And have
they promoted interdisciplinarity?

To answer these questions, I link rosters of faculty affiliated with ORUs to publication meta-
data retrieved from Scopus. En tono rimbombante, I include not only faculty affiliated with ORUs, pero
also a comparison set of researchers who are affiliated with the same departments but are not
affiliated with any ORU. I use regression models to control for variables, such as career length
y género, and a directed acyclic graph (DAG) and a sequence of models to examine the
mechanisms by which ORUs might increase productivity.

To examine disciplinarity and interdisciplinarity, I introduce the conceptual framework of
“discursive space” and situate researchers in this space by applying topic modeling—a text
mining technique—to a large corpus of journal articles abstracts by both ORU-affiliated faculty
and comparison faculty.

In brief, I find that the UC Davis ORUs have had productivity impacts, but likely did so by
enabling researchers to work with more coauthors1. The analysis of “discursive space” sug-
gests that the UC Davis ORUs have promoted multidisciplinarity rather than interdisciplinarity.
In the conclusion, drawing on work in philosophy of science on inter- and multidisciplinarity,
I argue that multidisciplinarity is not necessarily inferior to interdisciplinarity.

Tenga en cuenta que, because I examine ORUs at a single institution, I do not claim that my results

generalize to ORUs at any other institution.

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

1.1. Organized Research Units at UC Davis

At the time data collection began for this study (caer 2018), UC Davis had eight ORUs, cada
of which describes itself as engaged in interdisciplinary research or education. See Tables 1
y 2.

Four of these ORUs are dedicated to environmental topics, broadly constructed to include
ecology, conservation biology, environmental science, and environmental policy: the Air
Quality Research Center (AQRC); the Bodega Marine Lab/Coastal and Marine Science
Instituto (BML/CMSI); the Institute of Transportation Studies (ITS); and the John Muir Institute
of the Environment ( JMIE). Of these four ORUs, JMIE is the most heterogeneous, with distinct
initiatives in climate change, data science in environmental science, energy systems, polar
ciencia, and water science and policy.

Three other ORUs are dedicated to biomedical topics: the Comprehensive Cancer Center
(CCC); the Center for Healthcare Policy and Research (CHPR); and the Program in
International and Community Nutrition (PICN). CCC has both research and clinical aspects,

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

1 An anonymous reviewer suggests the possibility that these impacts are highly contingent on funding; if fund-
ing were to disappear, then these impacts might disappear as well. I agree that this is possible; but as none of
the UC Davis ORUs experienced dramatic funding changes during the study period, it is outside of the scope
of the current paper.

Estudios de ciencias cuantitativas

991

Interdisciplinary impacts of Organized Research Units

Abbreviation
AQRC

BML/CMSI

CNPRC

CCC

CHPR

ITS

JMIE

PICN

Mesa 1. UC Davis Organized Research Units [ORUs] examined in this study

Air Quality Research Center

Full name

Founded in

2005

Bodega Marine Laboratory & Coastal Marine Science Institute

1960 (BML)/2013 (CMSI)

California National Primate Research Center

Comprehensive Cancer Center

Center for Healthcare Policy & Investigación

Institute for Transportation Studies

John Muir Institute of the Environment

Program in International & Community Nutrition

1962

2002

1994

1991

1997

1987

and as the name indicates CHPR supports both academic research and policy analysis. PICN
has a strong global focus, with active research projects in Laos; Haiti; Cameroon and Ethiopia;
Burkina Faso, Ghana, and Malawi; the Gambia; Niger; Bangladesh; India; and Kenya.

The eighth ORU, CNPRC, is organized into four “units,” devoted to infectious diseases,
neuroscience and behavioral research, reproductive science and regenerative medicine,
and respiratory diseases. As these labels suggest, CNPRC supports a mix of behavioral and
investigación biomédica.

As Table 1 indicates, the ages of these ORUs vary substantially. BML is the oldest of the
current ORUs, founded in 1960. CMSI, the youngest of the current ORUs, was formed in 2013
to coordinate research activities between BML (a single laboratory on the Pacific Ocean north
of San Francisco) and other water research (such as the Tahoe Environmental Research Center
in Incline Village, Nevada, on the north shore of Lake Tahoe in the Sierra Nevada mountains).
BML/CMSI are treated as a single unit for the purposes of this study.

This study only considered ORU affiliations as of fall 2018; researchers who might have
been affiliated with an ORU previously, but were no longer affiliated as of fall 2018, and were
still actively publishing with a UC Davis affiliation during the period 2015–2017, would be
considered not affiliated with any ORU.

1.2. Productivity and Discursive Impacts

1.2.1. Productivity impacts

Research evaluation often focuses on measures such as publication counts, citation counts,
and perhaps patents or other indicators of economic impact (Hicks & Melkers, 2012). En el
context of evaluating the effects of a particular (set of ) programmatic interventions—namely,
recruiting faculty to an ORU—I refer to these familiar kinds of outputs as productivity impacts
of the intervention. Research evaluation might also consider productivity inputs, such as grant
application success rate or quantity of external research funds received.

en este estudio, I examine three productivity impacts of the UC Davis ORUs. Publication and
citation counts are familiar measures of research productivity. The third, coauthor count, is not
usually used as a primary measure of productivity. Sin embargo, it is highly plausible that in-
creased collaboration—and so an increased number of coauthors—leads at least to increased

Estudios de ciencias cuantitativas

992

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

I
norte
t
mi
r
d
i
s
C
i
pag
yo
i
norte
a
r
y

metro
pag
a
C
t
s

oh
F

oh
r
gramo
a
norte
i
z
mi
d
R
mi
s
mi
a
r
C
h
Ud.
norte
i
t
s

Mesa 2.

Each ORU describes itself as engaged in interdisciplinary research or education. All quotations were retrieved on 28 Puede 2020

ORU
AQRC

BML/CMSI

CCC

CNPRC

Interdisciplinary self-description
“The AQRC provides support for teams of collaborative researchers to conduct scientific, engineering, salud, social and economic research

that educates and informs planning and regulations for air quality and climate change.” (https://aqrc.ucdavis.edu/about)

“CMSI’s 100+ affiliated faculty and staff and over 120 graduate students and postdoctoral scholars are internationally recognized for their
expertise across the full spectrum of modern marine science, including ecology, evolutionary biology, conservation biology, microbiology,
coastal oceanography, environmental toxicology, geochemistry, political science, natural resource management, economics, law,
corporate sustainability, and marine wildlife health.” (https://marinescience.ucdavis.edu/about)

“Center members are organized into programs, shared resources and innovation groups to transcend departmental affiliations and to be

transdisciplinary, translational and transformative.” (https://health.ucdavis.edu/cancer/research/membership/index.html)

“The California National Primate Research Center (CNPRC) is improving health and advancing science with interdisciplinary programs in

biomedical research on significant human medical conditions throughout the lifespan.” (https://cnprc.ucdavis.edu/about-us/)

CHPR

“Since our inception in 1994, the UC Davis Center for Healthcare Policy and Research (CHPR) has conducted interdisciplinary and

collaborative research and research synthesis to improve health outcomes and services, educated the next generation of health services
investigadores, and assist policymakers in formulating effective health policies.” (https://medicine.ucdavis.edu/chpr/)

ITS

JMIE

PICN

“We have a strong commitment not just to research, but interdisciplinary education and engagement with government, industria, and non-

governmental organizations.” (https://its.ucdavis.edu/about/)

“The founding of the John Muir Institute of the Environment in 1997 was a monumental achievement for visionary academics. It was the
culmination of over 30 years of strong leadership in environmental research, interdisciplinary collaboration, and campus and system-wide
endorsement.” (https://johnmuir.ucdavis.edu/about)

“The educational curriculum provides students with the necessary knowledge and skills to master their chosen primary disciplines, mientras
simultaneously exposing them to interdisciplinary research methods. The training program therefore combines courses in basic biological
sciences, behavioral sciences, and social sciences, as well as interdisciplinary seminars in the planning, implementación, and evaluation of
nutrition programs at the community and national levels.” (https://globalnutrition.ucdavis.edu/about)

q
tu
a
norte

t
i
t

t
i
v
mi
S
C
mi
norte
C
mi
S
tu
d
mi
s

9
9
3

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Interdisciplinary impacts of Organized Research Units

publicación. Even if increased collaboration does not itself count as increased productivity, es
one potential mechanism by which ORUs might increase productivity. Eso es, along with pro-
viding (or facilitating the provision of ) research funds and other material resources, ORUs
might serve an important network function, encouraging researchers to work together more
than they would have otherwise. I therefore include coauthor count as a potentially significant
productivity impact.

1.2.2. Discursive impacts

Evaluating the productivity impacts of an interdisciplinary research program or organizational
unit is, methodologically and conceptually, essentially the same as evaluating a disciplinary
program or organizational unit: The same kinds of data will be collected and analyzed in the
de la misma manera. Además, productivity impacts abstract from the content of research. Counting
publications doesn’t consider what those publications are about.

But interdisciplinary research is typically justified in terms of distinctive pragmatic goals.
Por ejemplo, an unsigned editorial in Nature argues that “tackl[En g] society’s challenges
through research requires the engagement of multiple disciplines” (Naturaleza, 2016). Following
the distinctions made by Vanevar Bush and James Conant in the early Cold War period, Geiger
contrasts disciplinary research with “programmatic research” (Geiger, 1990, pag. 8). Geiger ar-
gues that the norms of disciplinary research are enforced by academic departments, haciendo
them “inherently [epistemically] conservative institutions.” By contrast, ORUs (broadly under-
stood to include museums, observatories, and extension offices) “exist to do what departments
cannot do: to operate in interdisciplinary, aplicado, or capital-intensive areas in response to
social demands for new knowledge” (Geiger, 1990, pag. 17; see also Sá, 2008).

Interdisciplinary research may also have epistemic goals. Huutoniemi, Klein et al. (2010)
distinguish epistemologically oriented and instrumentally oriented interdisciplinary research
(and sometimes use a third category of “mixed orientation” interdisciplinary research) (see also
Bruun, Hukkinen et al., 2005, páginas. 29–30, 90–91). In a qualitative analysis of interdisciplinary
research funded by the Academy of Finland, they find that the majority of interdisciplinary
funding is directed towards (purely) epistemologically oriented projects (Bruun et al., 2005,
pag. 104), y eso, weighted by funding, epistemologically oriented research is more likely
to be deeply integrative (rather than merely multidisciplinary research) than instrumentally
oriented research (Bruun et al., 2005, pag. 106).

Even when interdisciplinary research has purely epistemic goals, we would expect these
goals to be distinctive from those of disciplinary research. “Integration of various disciplinary
perspectives is expected to lead to a more profound scientific understanding or more compre-
hensive explanations of the phenomena under study” (Huutoniemi et al., 2010, pag. 85).
Different disciplines are assumed to offer complementary perspectives on the phenomenon
or subject. Entonces, bringing these complementary perspectives together is expected to produce
qualitatively better knowledge than each could have produced on its own.

Sin embargo, different disciplinary perspectives are not necessarily complementary. Different
disciplines—or even lines of research within a given discipline—may depend on different
metafísico, epistemological, and methodological background assumptions (Cartwright,
1999; Eigenbrode, O’Rourke et al., 2007; Holbrook, 2013; Kuhn, 1996; Longino, 2013;
Potochnik, 2017 ch. 7). These sets of background assumptions may be deeply incompatible
with each other, and attempts to integrate them might be frustrating and unproductive. En otra
palabras, en este caso, interdisciplinary research might be less, rather than more, than the sum of
its disciplinary parts.

Estudios de ciencias cuantitativas

994

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Interdisciplinary impacts of Organized Research Units

Insofar as interdisciplinary integration has been successful in a particular case—that is, en-
sofar as a body of research has been interdisciplinary rather than multidisciplinary—a variety
of theoretical perspectives predict that researchers will have to have produced a collection of
material and linguistic affordances spanning the divide. In a material mode, Star and
Griesemer (1989) examine “boundary objects” that circulate across disciplinary communities,
serving as both shared objects of inquiry and shared sources of evidence. Work on “trading
zones” has drawn on concepts from linguistics, such as pidgins and creoles, to analyze lin-
guistic innovation in successful cross-disciplinary interactions (collins, evans, & Gorman,
2007; Galison, 1997). Por ejemplo, andersen (2012) analyzes a successful collaboration be-
tween chemists and physicists, stressing the need for “interlocking mental models” such that
“the same concepts may form part of multiple lexica concerned with, Por ejemplo, diferente
aspects of a phenomenon” (andersen, 2012, pag. 281ff ).

Citation data are often used in quantitative studies of interdisciplinarity (Wagner, Roessner
et al., 2011). Cited works are classified into disciplines, often at the journal level (decir, cualquier
paper published in Cell counts as a biology paper), and empirical distributions of citations
across disciplines are used in various metrics of variety, balance, and/or disparity/similarity
(Rafols & Meyer, 2010).

While citation data can help us assess the extent to which researchers draw on work across
disciplines (Youtie, kay, & Melkers, 2013), these data have limited ability to tell us to what ex-
tent researchers engage in different goals, pursue different research topics, or adopt different
mental models. Accessing these features of research in large-scale quantitative studies likely
requires textual data and methods from text mining and natural language processing (NLP).
Por ejemplo, Hicks, Stahmer, and Smith (2018) suggest that text mining methods might be useful
for developing measures relating to what they call “outward-facing goals,” defined as “the value
of research for other [extra-academic] social practices.” They focus specifically on nouns ex-
tracted from journal article abstracts, and show how clusters of these nouns can be matched
to an existing taxonomy of basic human needs and values. Hofstra, Kulkarni et al. (2020) ana-
lyze abstracts of PhD dissertations from 1977 a 2015 to identify novel word associations, cual
they interpret as measures of innovation. They combine this analysis with name-based auto-
mated gender and race attributions and author-level bibliometric data from Web of Science to
examine how the relationship between innovation and career success varies across demo-
graphic groups. Sin embargo, neither of these papers examined interdisciplinary research as such.

en este documento, I propose that interdisciplinary research, as contrasted with disciplinary and
multidisciplinary research, will have distinctive linguistic traces that can be detected using text
mining methods.

Conceptually, I begin with the idea of discursive space, the space of research topics and
conceptual schemes as they manifest in language. Cifra 1 suggests how disciplinary and in-
terdisciplinary researchers might be configured in this discursive space. There are two groups
of disciplinary researchers, “red” and “blue.” These researchers have simple primary colors
and are clustered close together, indicating that they work on similar research topics, employ
similar conceptual schemes, and more generally use similar language. The circles representing
these researchers are small, indicating that they work on a relatively small set of topics. Y
the clusters are in distinct areas of discursive space, indicating that they differ substantially in
their research topics and conceptual schemes. The clusters are internally homogeneous but
externally heterogeneous.

Cifra 1 also includes two interdisciplinary researchers. These researchers are shades of
purple and are located in the space between the red and blue clusters, indicating that they

Estudios de ciencias cuantitativas

995

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Interdisciplinary impacts of Organized Research Units

Cifra 1. Conceptual model of “discursive space.” Disciplinary researchers (red and blue dots)
tend to be located close to members of the same discipline but far from members of other disci-
plines. Interdisciplinary researchers (purple ellipses) are located in between these disciplinary
grupos.

use a mix of research topics, conceptual schemes, and language more generally from the two
disciplines. The ellipses representing the interdisciplinary researchers are larger, Indicando que
they work on a relatively large set of topics. The shading and position of the researchers sug-
gests that they have home departments or disciplines: One is a bluish purple, and is closer to
the blues; the other is a redish purple, and is closer to the reds. But these interdisciplinary
researchers are closer to each other than they are to their home disciplinary clusters.

Más precisamente, I suggest three hypotheses based on Figure 1 and the conceptual framework
of “discursive space”: compared to their departmental peers, interdisciplinary researchers will:

H1. have greater discursive breadth, eso es, work on a wider variety of issues or use a wider

variety of methods, and so have more linguistic diversity;

H2. be further from the discursive central tendency of their home departments, eso es, el

center of departmental clusters in discursive space; y

H3. be closer to their interdisciplinary peers than their departmental peers.

In a context where we expect an intervention to promote interdisciplinary research—for
ejemplo, recruiting a faculty member to an ORU—I refer to these three hypotheses as the
expected discursive impacts of the intervention. This concept of discursive impacts provides
a framework for evaluating ORUs and other interdisciplinary research initiatives. Insofar as my
hypotheses are correct and the UC Davis ORUs have effectively promoted interdisciplinary
investigación, ORU-affiliated researchers should exhibit discursive impacts.

Bibliometricians and researchers in related areas of quantitative science studies have devel-
oped various similarity measures that might be interpreted as situating researchers (or other
units of analysis, such as documents or journals) relative to each other in “space” (Boyack,
Klavans, & Börner, 2005; Leydesdorff & Rafols, 2011; Wagner et al., 2011). Several common
metrics are based on citations (Boyack et al., 2005, pag. 355; Lu & Wolfram, 2012, pag. 1974).
Bibliographic coupling operationalizes similarity in terms of outgoing citations: Two units of
análisis (documentos, autores, journals) are similar insofar as they cite the same sources.
Cocitation analysis works in the opposite direction, operationalizing similarity in terms of

Estudios de ciencias cuantitativas

996

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Interdisciplinary impacts of Organized Research Units

incoming citations: Two units are similar insofar as they are cited by the same sources. A third
citation-based metric is intercitation analysis, on which two units are similar insofar as they cite
entre sí. (Calculations of intercitation similarity usually combine citations from x to y with
citations from y to x to create a symmetric statistic; Boyack et al., 2005, pag. 356.) De nuevo, mientras
citation data carry certain kinds of information, they do not seem to tell us much about the
objetivos, research topics, or mental models used by researchers.

Another common similarity metric, coword analysis is based on word usage: Two units are
similar insofar as they use the same terms. Coword data is usually combined with cosine sim-
ilarity, which automatically adjusts for differences in total word count (Por ejemplo, a senior
researcher is likely to have a much greater total word count than a junior researcher;
Leydesdorff & Rafols, 2011, pag. 88).

In this study I use topic models to assess similarity. Topic models begin with document-term
frequency data, as in coword analysis; Por ejemplo, the term “researcher” appears in a certain
document five times. The models then interpolate probability distributions of topics conditional
on documents and terms conditional on topics. Por ejemplo, a given document may “contain”
50% topic 1, 25% topic 2, 10% topic 3, etcétera. Each topic, Sucesivamente, has a probability
distribution over terms. Por ejemplo, topic 1 might have “researcher” with probability 1%,
“starfish” with probability 0.5%, “cancer” with probability 0.0001%, etcétera. Más formalmente,
topic models begin with observed conditional probability distributions over terms, pag(termw|
documenti), and fit two probability distributions β
t,i = p(topict|
documenti).

w,t = p(termw|topict) and γ

Examining the quantitative science studies literature, I was unable to find any systematic
reviews that compare topic models with other approaches to measuring similarity, como
coword or cocitation analysis. Lu and Wolfram (2012) found modest correlations (0.40–
0.48, Kendall’s τ-b) among coword, cocitation, and topic model-based measures of similarity;
sin embargo, they used a small sample of only “the 50 most prolific authors” in a data set and
selected the particular value of k used in their topic model “because it produced the most
reasonable outcome by our judgment,” with no further explanation or justification. Yan and
Ding (2012) compare similarity networks constructed using bibliographic coupling, cocitation,
coword, and topic model similarities. They find that the topic model networks are highly
similar to all of the other networks (0.93–0.99, cosine similarity). Sin embargo, this finding is
difficult to interpret. Most of the other networks are pairwise quite dissimilar to each other
(Por ejemplo, the cocitation networks’ similarities for most other network types range from
0.01 a 0.65); and Yan and Ding (2012) dichotomize the topic network by connecting nodes
if and only if their topic similarities (measured using cosine similarity) are greater than 0.82.

In quantitative science studies, topic models are often used diachronically, to examine the
ways research foci have changed over time (Han, 2020; Malaterre, Lareau et al. 2020; Rehs,

2 The term similarity is also used in information retrieval (IR), where it is used for scoring schemes that find the
documents in the corpus that are most relevant or related to a given document (or search query). Boyack,
Newman et al. (2011) compare several different clustering methods based on different similarity scores, en
this sense, including one derived from topic modeling. In their application—several years of papers indexed
by PubMed—the coherence of the topic model-based clusters is generally comparable to the two best
methods, BM25 (a widely used IR similarity score) and PRMA (the IR similarity score used by PubMed).
The only cases where BM25 and PRMA clusters were substantially more coherent than those of the topic
model-based method were a few very large clusters. Note that IR similarity scores are generally not symmet-
ric: For a score s and documents d1, d2, it can be the case that s(d1, d2) ≠ s(d2, d1). For this reason these
similarity scores do not correspond to metrics or distance functions. All of the similarity measures discussed
in the main text do correspond to metrics.

Estudios de ciencias cuantitativas

997

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Interdisciplinary impacts of Organized Research Units

2020). Nichols (2014) used a (previously fitted) topic model synchronically, to examine traces of
interdisciplinary research in projects funded by the U.S. Fundación Nacional de Ciencia. In all of
these examples, a single topic model was used for analysis, and topics were interpreted as
disciplines, campos, or research areas, depending on the scope of the corpus. Por ejemplo,
Nichols (2014) assigned almost all topics from a previously fitted 1000-topic model to NSF
directorates (which roughly correspond to high-level scientific fields, such as biology vs. com-
puter science vs. social and behavioral science), then calculated interdisciplinarity scores based
on whether these topics indicated interdisciplinarity within or between directorates. Malaterre
et al. (2020) used a corpus comprising articles from eight journals within a single field, philos-
ophy of science, and interpreted the 25-topic model in terms of research areas.

As the examples in the last paragraph suggest, the topics in topic models can be interpreted as
disciplines (or subdisciplinary units of analysis such as subfields). Because the topics are clusters
of terms (o, strictly speaking, probability distributions over terms), “disciplinary topics” corre-
spond immediately to research areas, methods, and other features that readily appear in the lan-
guage used within disciplines, rather than to features of disciplines as social organizations, semejante
as power hierarchies or patterns of funding or training. Sin embargo, it is highly plausible that these
two kinds of features are related, as the social organization enforces norms that regulate research
areas, methods, technical terminology, and other linguistic features. Por ejemplo, morgan,
Economou et al. (2018) combined academic hiring data with the occurrence of keywords in
article titles to examine the effect of prestige on the spread of research areas across a discipline.

As they are usually applied, topic models are known to have two significant researcher degrees
of freedom. Primero, the number of topics, k, is a free parameter, and there is no consensus on the
appropriate way to choose a value for k. As far as I am aware, the current state of the art in topic
model development requires fitting multiple models across a range of values for k, then calculating
various goodness-of-fit statistics on each model, such as semantic coherence, exclusivity (Roberts,
Stewart et al., 2014, páginas. 6–7), the log likehood of a holdout subset of terms (es decir., not used to fit the
modelo), the standard deviation of the residuals (which should converge to 1 at the “true” value of k;
Taddy, 2012), and the number of iterations required by the algorithm to fit the model. Sin embargo,
these statistics all have known limitations. Semantic coherence favors a small number of “large”
temas; exclusivity favors a large number of “small” topics. The log likelihood and residual methods
both assume that there is a “true” correct number of topics; but we would expect different numbers
of topics at different levels of conceptual granularity (Por ejemplo, at a coarse level of granularity
“biology” might be a single topic, while at a finer level “ecology” and “molcular biology” might be
distinct topics). This last conceptual mismatch is directly related to what Carole Lee has called the
“reference class problem for credit valuation in science” (Sotavento, 2020). So in general these goodness-
of-fit statistics will not agree on a single “best” model to use in further analysis.

Topic interpretation introduces a second major research degree of freedom. Topics are usu-
ally interpreted by extracting short term lists—usually the five or 10 highest-probability terms
from each topic—and manually assigning a label to each topic based on these term lists.
Sometimes interpretation also involves a review of the documents with the highest probabilities
for each topic. Topic labels are almost always assigned by the authors of the topic model study—
who may or may not have subject matter expertise in the areas covered by the corpus—and
reports often provide little or no detail about how labels were validated (Por ejemplo, to what
extent there were substantive disagreements between labelers about how to interpret a given
topic and how such disagreements were reconciled).

To mitigate these concerns about researcher degrees of freedom, the current project does
not select a single model for analysis, and does not lean on topic interpretation. En cambio, todo

Estudios de ciencias cuantitativas

998

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Interdisciplinary impacts of Organized Research Units

·,i and γ

fitted topic models are analyzed purely quantitatively to compare and situate authors relative
to each other in discursive space. Eso es, treating authors as “documents” in the topic model,
for authors i and j we can locate them relative to each other in discursive space by comparing
the distributions γ
·,j. Entonces, insofar as the UC Davis ORUs have promoted interdisci-
plinary research, I expect this space to have the features predicted by the three hypotheses
arriba. Here topic models function as a technique of dimensionality reduction, moving from
the high-dimensional space of all terms to the relatively low-dimensional space of topics.
Comparisons of these quantitative analyses across all fitted topic models allow us to assess
the robustness of results.

2. DATA AND METHODS

en este estudio, my unit of analysis is individual researchers or authors, as individuated by the
Scopus Author Identifier system3, except for a few analytical moments in which I compare
individual researchers to organizational entities (departments or ORUs). My unit of observa-
tion is publications—paradigmatically, journal articles—retrieved from Scopus and aggregated
as either author-level totals or concatenated blocks of text (específicamente, the abstracts of an
author’s published work, treated as a single block of text).

Unless otherwise noted, all data used in this project was retrieved from Scopus, using either
the web interface or application programming interface (API), between November 2018 y
Junio 2019. Due to intellectual property restrictions, the data cannot be made publicly avail-
capaz. Some downstream analysis files may be provided upon request.

All data collection and analysis was conducted in R (R Core Team, 2018). The RCurl pack-
age was used for API access (CRAN Team & Temple Lang, 2020); the spaCy Python library
was used for tokenizing, lemmatizing, and tagging abstract texts with parts of speech (spaCy,
2018); the spacyr package was used as an interface between R and spaCy; the stm package
was used to fit topic models (Roberts et al., 2014); and numerous tools in the tidyverse suite
were used to wrangle data (Wickham & RStudio, 2017). Because work on this project was
interrupted for a period of approximately 18 meses, software versions were not consistent
across the lifespan of this project.

All code used in data collection and analysis is available at https://github.com/dhicks/orus.

2.1. Author Identification

En noviembre 2018, the UC Davis Office of Research provided me with then-current rosters for
each ORU. These rosters included faculty (tenured/tenure-track faculty), “other academics” (pri-
marily staff scientists), and “collaborators” (other researchers, who may or may not be affiliated
with UC Davis and who generally did not receive funding from the ORU). I extracted the names
and ORU affiliation for all 134 affiliated faculty. In the remainder of this paper, I refer to these
ORU-affiliated faculty interchangeably as “ORU faculty” and “ORU researchers.”

In January 2019, I conducted searches using the Scopus web interface for all papers pub-
lished with UC Davis affiliations in 2016, 2017, y 2018. These searches returned 7,460,
7,771, y 8,066 resultados, respectivamente, totaling 23,297 publicaciones. The metadata retrieved
for these papers included names, affiliation strings, and Scopus author IDs for each author.

3 Ver https://service.elsevier.com/app/answers/detail/a_id/11212/supporthub/scopus/ for current details on

this system.

Estudios de ciencias cuantitativas

999

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Interdisciplinary impacts of Organized Research Units

Using a combination of automated and manual matching, I identified author IDs for ORU-
affiliated faculty, matching 125 out of 134 affiliated faculty. I next searched the affiliation
strings (from the publication metadata) for “Department of” to identify departmental affiliations
for these ORU faculty.

To identify a comparison set of researchers, I first identified all authors in the 2016–2018
Scopus results with the same departmental affiliations; eso es, an author was included in this
stage if they shared at least one departmental affiliation string with an ORU researcher. Este
resulted in 5,645 “candidate” authors for the comparison set. Sin embargo, many of these candi-
dates were likely graduate students and postdoctoral researchers. Because ORU researchers
are generally tenured faculty, including students and postdoctoral researchers would confound
the analysis of differences between ORU and non-ORU researchers. Por ejemplo, estudiantes
and postdoctoral researchers have much stronger incentives than tenured faculty to engage
in narrowly disciplinary research.

I therefore used the Scopus API (application programming interface) to retrieve author-level
metadata for both the ORU faculty and the candidate comparison researchers. Específicamente, I
examined the number of publications and the year of first publication. After exploratory data
análisis, I filtered both ORU faculty and comparison researchers, including them for further anal-
ysis only if they met two conditions: 15 or more total publications (as a proxy for student/postdoc
estado), and first publication after 1970. The first condition removed 60% of the candidate
comparison authors, and the second was used to exclude a small number of researchers with very
early first publication years (p.ej., 1955) that were plausibly due to data errors.

Tenga en cuenta que, in the analysis below, departmental affiliations for all authors are based on the

2016–2018 Scopus results, not entire publication careers.

After applying these filters, 2,298 researchers had been selected for analysis, incluido 116
ORU-affiliated researchers and 2,182 “codepartmental” comparison researchers. Cifra 2
shows the number of researchers in the analysis data set for each ORU and the comparison
set4, and Figures 3 and S1 show the structure of organizational relationships in the data.
(Tenga en cuenta que, because some researchers are affiliated with multiple ORUs, the ORU counts are
greater than 116.)

A substantial body of work in social studies of science finds consistent evidence that
female academics have lower publication rates and typically receive fewer citations per
publication than male academics (Beaudry & Larivière, 2016; Cameron, Blanco, & Gray,
2016; Chauvin, Mulsant et al., 2019; Ghiasi, Larivière, & Sugimoto, 2015; Larivière, Ni et al.,
2013; Symonds, Gemmell et al., 2006; van Arensbergen, van der Weijden, & van den Besselaar,
2012). (I was unable to find any literature that reported findings for nonbinary, genderqueer,
or transgender identities. Chauvin et al. (2019) note that they “planned to include faculty
identifying as nonbinary or genderqueer in a separate group,” but “were unable to identify
any such faculty from publicly available data sources.”) To control for these gender effects,
the online tool genderize.io was used to attribute gender to all authors based on their
first or given name. This tool resulted in a trinary gender attribution: woman, hombre, y
“missing” when the tool was not sufficiently confident in either primary gender attribution.
While extremely limited, this tool allows us to account for a known confounder given the

4 At the time of data collection, the AQRC roser also included nine “other academics,” all of whom had non-
faculty titles such as “Researcher,” “Research Professor,” or “Operations Manager.” For completeness, AQRC
is included in all analyses, except those comparing researchers within a given ORU, such as distance to
members of the same ORU.

Estudios de ciencias cuantitativas

1000

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Interdisciplinary impacts of Organized Research Units

Cifra 2. Number of researchers in the analysis data set for each ORU and the comparison set. Note that the y-axis uses a square-root scale.

limited time available for this project. Cifra 4 shows the distribution of attributed gender for
ORU- and non-ORU-affiliated researchers and each ORU separately. All together, ORUs
appear to be slightly more male-dominated than the comparison group. Sin embargo, hay
substantial variation across ORUs; the single AQRC-affiliated faculty member is attributed as
a man, more than half of PICN-affiliated faculty are attributed as women, and most ORUs have

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Cifra 3. Organizational relationships in the data. Nodes are either ORUs or departments at UC Davis. Edges connect an ORU to a depart-
ment if the two organizations share a common faculty member. Edge width and color indicate the number of such individuals.

Estudios de ciencias cuantitativas

1001

Interdisciplinary impacts of Organized Research Units

Cifra 4. Distribution of attributed gender for ORUs and comparison researchers. A: All ORU-affiliated researchers grouped together. B:
Gender distribution within each ORU. Gender attributions were made using an automated online tool based only on first or given name.

20–40% faculty attributed as women. In the regression analyses below, hombres (attributed gender)
are used as the reference level for estimated gender effects.

2.2. Productivity Impacts

To investigate the productivity impacts of ORUs, I used author-level metadata from the Scopus
API. Específicamente, all-career publication counts and (incoming) citation counts are both reported
in the Scopus author retrieval API, and so these data were retrieved prior to the filtering step
arriba. Coauthor counts (total number of unique coauthors) were calculated from the article-
level metadata retrieved for the text analysis steps discussed below. Because coauthor counts,
publication counts, and citation counts all varied over multiple orders of magnitude, I used the
registro (base 10) value for these variables in all analyses.

I fit regression models for each of these three dependent variables, using ORU affiliation as
the primary independent variable of interest and incorporating controls for gender, first year of
publicación (centered at 1997, which is the rounded mean first year in the analysis data set),
and dummy variables for departmental affiliation.

Because of the log transformation of the dependent variables, the regression model coeffi-
cients can be exponentiated and interpreted as multiplicative associations. Por ejemplo, a
coefficient of 0.5 can be interpreted as an association with a 100.5 ≈ 3.16-fold or 3.16 ×
100% − 100% = 216% increase in the dependent variable.

To account for relationships between the three dependent variables, I use the simplified
DAG shown in Figure 5. According to this model, the number of coauthors influences the
number of publications, which influences the number of citations. The number of publications
thus mediates between coauthors and citations, and coauthors mediates between the indepen-
dent variables and publications; I also allow that coauthors might directly influence citations.

Estudios de ciencias cuantitativas

1002

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Interdisciplinary impacts of Organized Research Units

Cifra 5. A simplified directed acyclic graph (DAG) used to account for relationships between the
productivity dependent variables.

Both ORU affiliation and all of the included control variables (first year of publication, género,
department affiliation) might directly influence all three dependent variables.

2.3. Discursive Impacts

I use topic models and related text analysis methods to examine the discursive impacts of
ORUs.

2.3.1. Topic modeling

Específicamente, I first used the Scopus API to retrieve paper-level metadata for all authors in the
analysis data set. I aimed to collect complete author histories—metadata for every publication
each author had written in their entire career. Metadata were retrieved for 128,778 distinto
papers in June 2021, of which 114,461 had abstract text5.

Abstract texts were aggregated within individual authors, treating each individual author as a
single “document.” For example, suppose researcher A was an author on documents 1 y 2,
and researcher B was an author on documents 2 y 3. Researcher A, as a single “document,"
would be represented for text analysis by adding together the term counts of abstracts 1 y 2;
while researcher B would be represented by adding together the term counts of abstracts 2 y 3.

Vocabulary selection began by using part-of-speech tagging to identify noun phrases in each
paper abstract. Nouns are more likely than other parts of speech to carry substantive informa-
tion about research topics and methods. Noun phrases, such as “random effects model” or
“unintended pregnancy,” are more specific and informative than nouns alone. 414,423 distinto

5 In review, changes to the inclusion criteria used to select comparators required me to reretrieve these meta-
datos. Only papers published through 2019 were included at this stage. One Scopus author identifier no
longer existed when the metadata were reretrieved. This identifier corresponded to one comparison author,
who was excluded from analysis.

Estudios de ciencias cuantitativas

1003

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Interdisciplinary impacts of Organized Research Units

noun phrases were extracted from the corpus. I then counted occurrences for each noun phrase
for each author. (In the remainder of this paper, I generally use terms to refer to noun phrases.) I
used these author-term counts to calculate an entropy-based statistic for each term, keeping the
arriba 11,490 terms to achieve a 5:1 ratio between authors (“documents”) and terms. Tenga en cuenta que
stopwords were not explicitly excluded, though typical lists of English stopwords do not include
many nouns.

To score the “importance” of each term, consider the task of identifying an author (o, más
generally, a “document”), selected uniformly at random from the analysis data set. For N
autores, the unconditional probability of drawing any given author is pN = 1
norte; this distribution
has entropy

H pNð

Þ ¼

norte

− pN log2pN

log2

1
norte

¼ −N

1
norte
¼ log2N

Now suppose we are given a term w (“word”) drawn from the token distribution of the
unknown author. Because the uniform distribution has maximal entropy, the conditional author
distribution given the term, pw = p(authorj|termw), has a lower entropy Hw = H( pw) ≤ log2 N. Dejar
ΔHw = log2 N − Hw. ΔHw measures the information about the identity of the author gained when
we are given the term w. (This formula derives from the Kullback-Leibler divergence from the
uniform distribution pN to pw.) A high-information term dramatically narrow downs the range of
possible authors. Eso es, terms have higher information insofar as they are specific to a smaller
group of authors.

Sin embargo, típicamente, the most high-information terms will be unique to a single author, semejante
as typos or idiosyncratic terms. To account for this, I also calculate the order of magnitude of
the occurrence of a term across the entire corpus, log10 nw. We then take the product log10
ΔHw, which I represent in the code as ndH, and select the top terms according to this log10
nw
nwΔHw statistic. Table S1 shows the top 50 terms selected for the analysis vocabulary. As the
term list suggests, this statistic is effective at identifying terms that are clearly distinctive (en esto
caso, to different disciplines and research fields), meaningful, and frequent. The term list also
ΔHw balances information gain with word occurrence. Some terms,
illustrates how log10 nw
such as “cuttlefish,” have extremely high information gain (very low H = 0.17) but are com-
mon enough (occurring 118 times across the corpus) that they are not typos or idiosyncratic to
a single author. Other terms, such as schizophrenia, have more modest information gain (H =
4.59) but are extremely common (occurring 2,374 veces).

As discussed above, topic models require setting the number of topics k before fitting the
modelo, and there is no consensus on the appropriate way to choose a value for k. Exploratory
analysis of the author-term distributions using principal components found that 50% del
variance could be covered by 24 principal components, 80% required 167 principal compo-
nents, y 90% required more than 300 principal components. I also speculated that small-k
topic models might capture coarse disciplinary distinctions, but would also be less stable.
Given these considerations, I fit models with 5, 10, 15, 20, 25, y luego 50, 75, 100, 125,
y 150 temas. I calculated five goodness-of-fit statistics for each of these models: semantic
coherencia, exclusivity, the log likehood of a holdout subset of terms (es decir., not used to fit the
modelo), the standard deviation of the residuals, and the number of iterations required by the
algorithm to fit the model. As expected, these statistics did not indicate a uniformly “best”

Estudios de ciencias cuantitativas

1004

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Interdisciplinary impacts of Organized Research Units

topic model; though k = 50 minimized both the number of iterations and the residuals, y
had a greater coherence and approximately the same exclusivity as the larger models.

Rather than selecting a single “best” topic model, in the analysis below I either (a) conduct
and report analyses using all of the topic models, highlighting k = 50, o (b) conduct analyses for
k = 5, 25, 50, 100, reporting all four equally. Acercarse (b) is generally used when the analysis
involved a complex visualization component, to keep the number of plots manageable.

2.3.2. Analyses

·,i) = (cid:1)

I focused my analysis on the topic distribution γ
t,i = p(topict|authori) for each topic model. Recordar
the three hypotheses for discursive impacts, introduced in Section 1.2.2. For H1, I calculated
“discursive breadth” for author i as the entropy of the topic distribution Hi = H(γ
− γ
t,i. In information theory, entropy is understood as a measure of the “width” or “breadth”
log2
of a distribution (McElreath, 2016, pag. 267). Rafols and Meyer (2010) examine the use of diversity
concepts in studies of interdisciplinarity, and analyze them into the “attributes” or “categories”
of variety, balance, and disparity/similarity (Rafols & Meyer, 2010, pag. 266). They note that en-
tropy combines variety and balance (Rafols & Meyer, 2010, pag. 268). Rosen-Zvi, Griffiths et al.
(2004) use the entropy of author-topic distributions (from an author-document-topic model) “to
assess the extent to which authors tend to address a single topic in their work, or cover multiple
topics” (Rosen-Zvi et al., 2004, pag. 8). At the journal level, Leydesdorff and Rafols (2011) consider
the entropy of citation distributions as a measure of interdisciplinarity. In a factor analysis, ellos
find it is related to a Rao–Stirling diversity measure, and conclude that “Shannon entropy qual-
ifies as a vector-based measure of interdisciplinarity” (Leydesdorff & Rafols, 2011, pag. 96).

t,i

H2 and H3 both appeal to a measure of “discursive distance.” I operationalize this distance
2 as Hellinger distance (“Hellinger Distance,"

1, γ

between any pair of topic distributions γ
2021):

h γ
d

; γ

ﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ
X
(cid:4)
(cid:3)
2

pag

−

ﬃﬃﬃﬃﬃﬃﬃ
γ
t;1

Þ ¼ 1ﬃﬃﬃ
pag
2
r
ﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ
pag
ﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ
1 −
γ
t;2

t
X

t;1 _

ﬃﬃﬃﬃﬃﬃﬃ
γ
t;2

(cid:5)
pag

ﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ
(cid:6)
ﬃﬃﬃ
γ
1 −
1

ﬃﬃﬃ
γ
2

pag

;

Hellinger distances range from 0 a 1, dónde 0 indicates that two distributions are the same
y 1 indicates that the two distributions have completely different support. Hellinger distance
can be understood as a scaled version of the Euclidean distance between the square root
vectores
2; o, because the square root vectors are all unit length, as a distance measure
corresponding to the cosine similarity x;yh
xj j yj j between the square root vectors. Cosine similarity is
widely used in bibliometrics (Mingers & Leydesdorff, 2015).

ﬃﬃﬃ
γ
1,

ﬃﬃﬃ
γ

pag

H2 requires constructing department-level topic distributions. The stm package provides
functions that, given a fitted topic model and an observed term distribution for a “document,"
estimate a conditional topic distribution γ for that “document” (somewhat like using a fitted re-
gression model to predict outcomes for new observations). One simple way to construct a
department-level “document” would be to aggregate the work (take the sum of term counts)
of all of the authors associated with that department. Sin embargo, for the purposes of investigating
H2, this construction would lead to various problems. Primero, papers by multiple authors in the
department would contribute to the department-level distribution multiple times. Segundo,
insofar as ORU faculty are distant from the other members of their department, their contribu-
tions to the department distribution will act as outliers, and the resulting distance measures will

Estudios de ciencias cuantitativas

1005

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Interdisciplinary impacts of Organized Research Units

be biased towards the ORU faculty, leading to underestimates of the effect for H2. En el otro
mano, if all and only non-ORU faculty members contribute to the department-level distribution,
then their work would be counted twice: First they would be used to construct the department-
level distribution, and then second we would calculate their distances from this distribution. En
este caso, the distance measures will be biased towards the non-ORU faculty.

To avoid these problems, I constructed department-level distributions as follows. I first borrowed
an approach from machine learning (James, Witten et al., 2013, pag. 176ff ), and randomly separated
non-ORU authors into two discrete subsets. The first subset—referred to as the “training” set in
machine learning—was used to construct the department-level topic distributions. El segundo
subset—the “testing” subset—as used to make the distance comparisons, using Hellinger distance.
50% of non-ORU authors by departmental affiliation were allocated to the training set, selected
uniformly at random, and the remaining non-ORU authors were assigned to the testing set. (Este
means that a non-ORU author affiliated with multiple departments had the same role—testing or
training—across all of their affiliations.) Entonces, for each department, I aggregated all of the papers
eso (a) had at least one training set author and (b) did not have an ORU-affiliated author.

After allocating authors to these subsets and constructing department-level topic distributions, I
calculated “departmental distance” using Hellinger distance for all ORU-affiliated faculty and all
comparison authors in the testing subset. I used these departmental distance values as dependent
variables in a series of regression models, one for each value of k, including first publication year,
género, log number of documents and coauthors, and department dummies as controls.

For H3, I made both “individual” and “organizational” comparisons for each ORU-affiliated
faculty member. At the individual level, I calculated the Hellinger distance between the ORU-
affiliated researcher and other individuals, (a) in the same ORU and (b) in the same departments,
and then took the minimum for both (a) y (b). At the organizational level, I calculated the
Hellinger distance between the ORU-affiliated researcher and (a) an ORU-level topic distribution,
constructed by aggregating the papers authored by affiliates, y (b) the department topic distribu-
tion described above. (Note that this means ORU-affiliated authors contribute to the ORU topic
distribución, but not the department distribution. So this construction might tend to bias distance
estimates towards ORUs and away from departments.) At both levels, (a) gives us a measure of
distance within the researcher’s ORU and (b) gives us a measure within the researcher’s depart-
mentos. If the ORU distance is less than the departmental distance, this indicates that the ORU faculty
member is closer to their ORU than to their home department, consistent with H3. Using both “in-
dividual” and “organizational”-level comparisons accounts for the possibility that an ORU-
affiliated researcher may be quite close (in discursive space) to one or a few non-ORU-affiliated
departmental colleagues but still relatively far from the “core” or “mainstream” of their department.

3. RESULTADOS

3.1. Productivity Impacts

Regression analyses indicate that ORU affiliation is associated with a substantial increase in
the order-of-magnitude number of coauthors, 1.5–2.1-fold (1.8-fold)6. See Figure 6. ORU

6 Statistical estimates are reported as 95% confidence intervals followed by the maximum likelihood point
estimate in parentheses. No statistical hypothesis testing was done in this paper, so no p-values are calcu-
lated or reported. Confidence intervals should be interpreted as a range of values that are highly compatible
with the observed data, given the modeling assumptions (Amrhein, Trafimow, & Greenland, 2018). A con-
fidence interval that contains zero can still be evidence of a substantial (nonzero, decir, positivo) relationship
insofar as the bulk of the interval is greater than zero.

Estudios de ciencias cuantitativas

1006

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Interdisciplinary impacts of Organized Research Units

Cifra 6. Regression estimates for number of coauthors.

affiliation had a much weaker direct association with the number of publications, 1.0–1.2-fold
(1.1-fold), while an order-of-magnitude increase in number of coauthors had a much stronger
association, 2.9–3.3-fold (3.1-fold). See Figure 7.

The estimates for citations are similar. Order-of-magnitude increases in number of publica-
tions and number of coauthors are both associated with substantial increases in the number of
citations a researcher has received to date: 4.0–5.3-fold (4.6-fold) for publications, 2.4–
3.0-fold (2.7-fold) for coauthors. When controlling for these midstream dependent variables,
the association between ORU affiliation and citations received is small or perhaps even neg-
ative, 0.9–1.2-fold change (1.0-fold). See Figure 8.

All together, in causal terms, these regression results suggest that ORU affiliation has a
substantial direct effect only on the number of coauthors. This increase in coauthors in turn
leads to increased publications and increased citations; but ORU affiliation has a much smaller
direct effect on these two downstream productivity measures. On this interpretation of the
findings, ORU affiliation makes faculty more productive primarily by connecting them with
collaborators.

Cifra 7. Regression estimates for number of publications.

Estudios de ciencias cuantitativas

1007

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Interdisciplinary impacts of Organized Research Units

Cifra 8. Regression estimates for number of citations.

Sin embargo, the evidence for this causal interpretation is limited, because we do not have the
data to compare a researchers’s number of coauthors before and after they join the ORU7. El
available data are consistent with a pattern where some unobserved variable is a common
cause of both ORU affiliation and coauthor count. Por ejemplo, highly gregarious and extro-
verted faculty members might tend to have more coauthors and also be more likely to be in-
vited to join an ORU. O, an interdisciplinary group of researchers might have already been
working together, then formed an ORU to provide institutional support for their collaboration.
Por ejemplo, the AQRC “About Us” page states that “The Air Quality Research Center was
established in the summer of 2005, although our faculty, staff and student affiliates had been
working together for many years prior” (https://airquality.ucdavis.edu/about)8.

3.2. Discursive Space

En esta sección, I report the results of an exploratory analysis of the topic model results. I focus
primarily on the department- and ORU-level distributions, and evaluating the suitability of the

7 Some readers might object that, without this data, this study cannot make any causal claims. This objection
involves a common mistake about causal inference, confusing a sufficient condition for strong evidence for
causal claims (satisfying the assumptions of causal inference theory) with a necessary condition for (possibly
weaker) evidence for causal claims. Any correlation between two variables provides a reason to believe—
eso es, evidence—that there is a causal relationship between them (compare Reichenbach’s common cause
principle; Hitchcock & Rédei, 2021). The reason we conduct observational studies is that correlations pro-
vide evidence of causation. But the inference from correlation to causation is not very reliable, and thus mere
correlation provides only rather weak, highly defeasible evidence for causation. The point of causal infer-
ence theory is to give sufficient conditions for more reliable inferences and thus better evidence.
Desafortunadamente, the data needed to satisfy these conditions for this study—the dates when researchers first
became affiliated with ORUs—do not exist. Still, even weak, defeasible evidence is evidence, and so I some-
times put forward—defeasible, qualified—causal claims in this paper. Readers may, at their discretion, reject
the causal claims even if they accept the correlational results.

8 Thanks to an anonymous reviewer for pointing out this example confounder.

Estudios de ciencias cuantitativas

1008

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Interdisciplinary impacts of Organized Research Units

topic model results for analyzing interdisciplinarity using the “discursive space” conceptual
estructura.

A list of the five highest-probability terms from each topic in the k = 50 model is provided in
Table S2. As discussed above, I do not label these topics, and my analysis does not depend on
the content of the term lists.

Cifra 9 visualizes “discursive space” based on the pairwise Hellinger distance for the topic
distributions of authors, departments, and ORUs. In the visualization, these pairwise distances
are represented in two-dimensional space using the t-SNE algorithm (van der Maaten & Hinton,
2008). This algorithm uses an information-theoretic approach to represent high-dimensional
relaciones (aquí, the Hellinger distances) in two dimensions. The algorithm is widely used
in fields such as computational biology. But it is designed to emphasize local topology rather
than global geometry. This means that a t-SNE visualization indicates nearness, but that
distances in the visualization do not necessarily correspond to distances in the original

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Cifra 9. Visualization of “discursive space.” Panels correspond to different values of k (number
of topics). Circles are authors; non-ORU-affiliated authors are indicated with translucent yellow
puntos. Squares and diamonds represent ORU- and department-level topic distributions, respectivamente.
Point positions are calculated using t-SNE on the pairwise Hellinger distance between author-topic
distributions. Ellipses indicate bounds on the researchers affiliated with each ORU (based on the
convex hull).

Estudios de ciencias cuantitativas

1009

Interdisciplinary impacts of Organized Research Units

high-dimensional space. En particular, t-SNE tends to organize points into visual clusters. Two
clusters might be visually far apart but relatively close in the original high-dimensional space.

The t-SNE visualizations of “discursive space” suggest complex archipelagoes of re-
searchers. Some ORUs, such as PICN and ITS, have all of their affiliates in one or a few clusters
(recall that AQRC has a single faculty affiliate in these data). For others, such as BML/CMSI and
CHPR, most of their affiliates are clustered near the ORU-level topic distribution (indicated by
the square), with a few further-flung affiliates. De nuevo, note that the t-SNE algorithm can place
two clusters visually far apart even when they are close in the original high-dimensional space.

Cifra 10 breaks out the visualization of “discursive space” by department for large depart-
mentos (50 or more authors in the data set). While a few departments are tightly clustered into a
single island, most are somewhat scattered, with authors distributed across the visualization.

Figures S2 and S3 show the department- and ORU-level topic distributions, y figura 11
shows the entropy of these distributions. These figures indicate that, except for the highest
values of k, most departments and ORUs have entropies of 1–3 bits, roughly corresponding

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Cifra 10. Visualization of “discursive space” for departments with 50 or more authors in the data set, k = 50 topic model. Circles are
autores; non-ORU-affiliated authors are indicated with translucent yellow points. Diamonds represent the department-level topic distribution.
Positions within each panel are the same as those calculated in Figure 9.

Estudios de ciencias cuantitativas

1010

Interdisciplinary impacts of Organized Research Units

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Cifra 11.
Entropy of the ORU and department-level topic distributions. Thin grey lines corre-
spond to departments; colored lines correspond to ORUs. Each point corresponds to the entropy
of the distribution shown in Figure S2 or S3. The solid black curve corresponds to the maximum
possible entropy at that value of k (a uniform distribution across all topics).

to 2–8 topics. Por ejemplo, BML/CMSI has an entropy of about 2 at k = 100, and four colored
bars are visible in the corresponding panel in Figure S3.

Cifra 12 visualizes the similarity network among the departments and ORUs based on the
Hellinger distance between their topic distributions for the k = 50 topic model; Figure S4
shows networks across four values of k. (All pairs of departments and ORUs are used in this
network analysis; eso es, no edges with “low” values are trimmed to zero. This eliminates a
common researcher degree of freedom in this kind of network analysis.) For k > 5, borde
weights/similarity scores are uniformly low, indicating that these units are generally far from
each other in “discursive space.” Connections between ORUs and related departments are
among the strongest in each network, although even these are still only moderate in absolute
terms (Por ejemplo, < 0.50 for k = 100). The similarity network visualizations also include clusters produced using the Louvain community detection algorithm (Blondel, Guillaume et al., 2008; while this algorithm is widely used in network analysis, Hicks [2016] reviews fundamental problems with Louvain and other modularity-based community detection algorithms), using Hellinger similarity scores as edge weights. The cluster results suggest that the topic models are effectively encoding higher- level relationships between disciplines, with higher values of k enabling the detection of more fine-grained relationships. For example, Human Ecology, Psychology, Psychology and Behavioral Sciences, and CNPR are consistently clustered together. Similarly, another consis- tent cluster is Evolution and Ecology; Wildlife, Fish, and Conservation Biology; Environmental Science and Policy; and BML/CMSI. These figures enable us to align the topic models and the conceptual framework of “discur- sive space” with the institutional account of disciplines, namely, departments as the primary Quantitative Science Studies 1011 Interdisciplinary impacts of Organized Research Units l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / e d u q s s / a r t i c e - p d l f / / / / 2 3 9 9 0 1 9 7 0 7 0 4 q s s _ a _ 0 0 1 5 0 p d / . f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 Figure 12. Similarity network for departments and ORUs, based on Hellinger distance between department- and ORU-level topic distribu- tions (Figures S2 and S3) for the k = 50 topic model. Nodes (labels) correspond to departments/ORUs, with ORUs indicated by slightly larger labels. Edge widths and shading correspond to similarity values, and edges for all pairs are used in each network (no values are trimmed or rounded to 0). Node placement uses the sparse stress majorization algorithm (Ortmann, Klimenta, & Brandes, 2016), with similarity scores used as edge weights. Label color corresponds to results of Louvain community detection (Blondel et al., 2008) with similarity scores used as edge weights. sites where disciplinary standards are enforced and codified, and thus the sites where disci- plines are brought into being (Geiger, 1990; Hicks & Stapleford, 2016; Pence & Hicks, n.d.). Departments specialize in just a few topics, and at higher values of k these topics separate departments from each other (with high Hellinger distance/low Hellinger similarity). That is, topics are distinctive and characteristic of departments. At the same time, higher-level disci- plinary relationships between departments can be recovered using clustering methods. Thus the topic models distinguish, for example, behavioral science or conservation science and pol- icy. For the ORUs, most appear to specialize in distinctive combinations of topics that are well represented in departments. That is, the ORUs do not seem to work on ORU-specific topics, but instead combine disciplinary topics (in either interdisciplinary or multidisciplinary ways). These combinations of topics may correspond to an ORU’s distinctive research aims or linguis- tic affordances that have been developed to facilitate interdisciplinary research. The one ap- parent exception to this pattern is PICN; here the topic models do seem to have identified a distinctive topic for this ORU, locating it on the margins of the similarity networks. 3.3. Discursive Impacts 3.3.1. Discursive breadth H1 states that ORU interdisciplinarity may lead to increased “discursive breadth,” operation- alized as the entropy of the topic distribution. Figure 13 visualizes the entropy for each indi- vidual researcher in the data set, by ORU affiliation status and across selected values of k. Quantitative Science Studies 1012 Interdisciplinary impacts of Organized Research Units l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . Figure 13. Researcher entropies, by ORU affiliation status and across selected values of k (number of topics). Violin plots include 25th, 50th (median), and 75th percentiles. Color fills are points for individual researchers, staggered horizontally to correspond to the violin plots. Panels correspond to values of k. Gray horizontal lines at the top of each plot indicate the maximum possible entropy (uniform distribution across all topics) for that value of k. / e d u q s s / a r t i c e - p d l f / / / / 2 3 9 9 0 1 9 7 0 7 0 4 q s s _ a _ 0 0 1 5 0 p d / . f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 Figure 14. Coefficient estimates for association between topic entropy and ORU affiliation across values of k (number of topics). Whiskers are 95% confidence intervals. Regression models include as controls attributed gender, first year of publication, number of publications, number of coauthors, and departmental dummies. k = 50 is highlighted because it was judged “best” by some but not all goodness-of-fit statistics. Quantitative Science Studies 1013 Interdisciplinary impacts of Organized Research Units (Figure S5 shows entropies across all values of k.) Across values of k, the distributions for ORU- affiliated and nonaffiliated researchers are similar: For k > 5 the median researcher for both has
entropy H ≈ 1, roughly corresponding to two topics, 75% of researchers have entropy less than
H ≈ 2, roughly corresponding to four topics, and the modal researcher has H ≈ 0, apenas
corresponding to a single topic. En otras palabras, most researchers work in only a handful of
temas, whether they are affiliated with an ORU or not. This pattern is consistent even for topic
models with high values of k, though the right tail (the “neck” of the violin) is longer as k
aumenta (especially for non-ORU-affiliated researchers), meaning there are a few researchers
in the data set who work on a very wide range of topics.

Cifra 14 shows the regression coefficient estimates for the association between topic en-
tropy and ORU affiliation across all topic models, controlling for first year of publication, generación-
der, department affiliation, and logged number of documents and coauthors. Across all
modelos, confidence intervals generally cover from −0.15 to 0.2 bits, with point estimates in
the range −0.1 to 0.1 bits. Usando 0.5 bits (“half of a coin flip”) as a threshold for substantive
diferencia (eso es, treating any value between −0.5 and 0.5 as too small to be interesting), el
models uniformly indicate that the difference in discursive breadth between ORU authors and
their peers is trivial.

H1 does not seem to be supported by these results. This hypothesis posits a connection
between interdisciplinary, the notion of “discursive breadth,” and topic distribution entropy
as an operationalization of this notion of breadth. The apparent failure of H1 might be due
to failures at any of these three points. Primero, the ORUs at UC Davis might not have effectively
promoted interdisciplinarity. Segundo, “discursive breadth” might be an inapt way of charac-
terizing interdisciplinarity. And third, entropy might be an inapt operationalization of

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Cifra 15. Coefficient estimates for association between distance to departmental mean distribution and ORU affiliation across values of k
(number of topics). Whiskers are 95% confidence intervals. Regression models include as controls attributed gender, first year of publication,
number of publications, number of coauthors, and departmental dummies. k = 50 is highlighted because it was judged “best” by some but not
all goodness-of-fit statistics.

Estudios de ciencias cuantitativas

1014

Interdisciplinary impacts of Organized Research Units

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Cifra 16. Minimal distance to ORU peers vs. departmental peers. Both x- and y-axes are on the Hellinger distance scale (0–1). The dashed
line in each panel indicates y = x. Points above this line are closer to ORU peers than departmental peers, supporting H3. Note that com-
parisons to the dashed line should be made vertically or horizontally.

“discursive breadth.” For example, if the topic model included an ORU-specific “interdisci-
plinary topic,” then an interdisciplinary researcher might have a narrower distribution than
their disciplinary peers9. Sin embargo, the examination of department- and ORU-level topic dis-
tributions in Figures 11, 12, S2, S3, and S4 indicated that, except for PICN, the topic models
did not include ORU-specific “interdisciplinary topics,” suggesting that the third link is not the
problema. If the other analyses of discursive impacts give indications that the UC Davis ORUs
have effectively promoted interdisciplinarity, then this would suggest that the problem with H1
is in the second or third link. So this would imply that the problem is the second link. Sobre el
other hand, if these other analyses indicate that the ORUs have not effectively promoted in-
terdisciplinarity, this would indicate that the problem is with the first link, and consequently,
“discursive breadth,” operationalized by the entropy of topic distributions, might still be apt for
measuring interdisciplinarity.

3.3.2. Departmental distance

H2 states that ORU interdisciplinarity may lead to increased departmental distance, eso es,
increased Hellinger distance from the department-level topic distribution. Cifra 15 shows co-
efficient estimates for the association between departmental distance and ORU affiliation
across values of k (number of topics), junto con 95% confidence intervals. Here the estimates
may appear to support H2, as they are generally positive. Sin embargo, most confidence intervals

9 Thanks to an anonymous reviewer for suggesting this potential problem.

Estudios de ciencias cuantitativas

1015

Interdisciplinary impacts of Organized Research Units

end well below 0.06, y (except for k = 5) point estimates are all in the the range 0.02–0.04.
Recall that Hellinger distance is on a 0–1 scale. On this scale, distances less than 0.05 would
seem to be trivial. Eso es, there does not seem to be a meaningful difference between ORU
faculty and the mean of their departmental peers, and so H2 does not appear to be supported
either.

It might be suspected that departmental distance effects could vary across ORUs. Figure S6
reports coefficient estimates for ORU dummy variables, rather than the binary yes/no ORU
affiliation used above; “no ORU affiliation” is used as the contrast value for the ORU
dummies.

Figure S6 does indeed suggest that the potential association between ORU affiliation and
departmental distance does vary across ORUs, albeit still to a limited extent. Except for CHPR
and JMIE, for at least some values of k the point estimate is greater than 0.10. There does seem
to be some evidence of a nontrivial association for CNPRC, PICN, and AQRC (though recall
that this last had only a single faculty affiliate during the analysis period). So H2 might be true
for these three ORUs, but not the others.

3.3.3. ORU-department relative distance

H3 proposes that ORU interdisciplinarity leads researchers to be closer to their ORU peers
than their (non-ORU-affiliated) departmental peers in discursive space. Cifra 16 shows scat-
terplots for minimal distances to both kinds of peers, for each ORU and four values of k. En
these scatterplots, the dashed line indicates y = x. Points above this line are closer to ORU
peers than departmental peers, so these points would be compatible with H3.

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Cifra 17. Comparison of minimal ORU and departmental distances, across ORUs and values of k. Positive x-axis values indicate that de-
partmental distance is greater than ORU distance, supporting H3. Dashed vertical line indicates 0; solid lines within densities indicate median
valores; and small vertical dashes indicated individual values.

Estudios de ciencias cuantitativas

1016

Interdisciplinary impacts of Organized Research Units

For most ORUs, across values of k, most researchers are located near or somewhat below
la línea discontinua. This means that researchers are typically equidistant from or closer to their
closest departmental peers than their closest ORU peers.

Because distance comparisons in scatterplots can be misleading (comparing vertical dis-
tance to the dashed line, not Euclidean distance), Cifra 17 shows the distribution of these
comparisons. In this figure, positive x-axis values would indicate that departmental distance
is greater than ORU distance, which would support H3. In this figure, modal and median
values are all negative or near 0. While there are a few exceptional individuals, ORU faculty
are generally equidistant from or closer to their closest departmental peers. As with the other
two hypotheses, these findings conflict with H3.

Figures 18 y 19 show distance comparisons to the ORU- and department-level distribu-
ciones. For most ORUs and most values of k, the median and modal differences are positive,
indicating that generally ORU researchers are closer to the ORU than their departments,
which is consistent with H3. This tendency is most obvious for PICN, with a median > 0.50
on the Hellinger scale for k > 5. The one exception is JMIE, which has a negative median
across all values of k.

All together, the distance comparisons suggest the following. ORU faculty are often quite
close to some individual non-ORU researchers in their home departments, but still relatively
far from the “center” of the department as a whole. This suggests that ORU faculty are not the
only members of their departments engaged in cross-disciplinary research, but they are still

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Cifra 18. Distance to ORU- and department-level topic distributions. Both x- and y-axes are on the Hellinger distance scale (0–1). El
dashed line in each panel indicates y = x. Points above this line are closer to ORU peers than departmental peers, supporting H3. Tenga en cuenta que
comparisons to the dashed line should be made vertically or horizontally.

Estudios de ciencias cuantitativas

1017

Interdisciplinary impacts of Organized Research Units

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Cifra 19. Comparison of distance to ORU- and department-level topic distributions, across ORUs and values of k. Positive x-axis values
indicate that departmental distance is greater than ORU distance, supporting H3. Dashed vertical line indicates 0; solid lines within densities
indicate median values; and small vertical dashes indicated individual values.

somewhat outside of the disciplinary mainstream. As originally posed, H3 is ambiguous be-
tween individual-level and organizational-level comparisons. The individual reading is not
supported here, but there is support for the organizational-level reading.

4. DISCUSIÓN

The analysis of productivity impacts provides some evidence that ORUs have increased the
productivity of affiliated faculty at UC Davis. Específicamente, the sequence of regression models
suggests that ORUs have increased the number of coauthors that affiliated faculty have; eso
this increased collaboration leads to increased publications and citations; but that ORUs have
had at most a small direct effect on publications and citations.

Due to the limitations of the study design and data, we cannot be sure whether this rela-
tionship is indeed a causal effect of ORUs on productivity. The data are also compatible with
an effect of productivity on ORUs (eso es, perhaps ORUs have tended to recruit faculty who
already tend to be more productive), an unmeasured common cause (Por ejemplo, tal vez
more extroverted faculty have tended both to be more productive and to be recruited to
ORUs), or indeed a combination of multiple causal relationships. A more complex possibility,
in line with the Matthew Effect (DiPrete & Eirich, 2006; Merton, 1968), is that ORUs may have
enhanced pre-existing trends, by directing more resources towards faculty who already tended
to be more productive and have more coauthors. To sort out these plausible causal relation-
buques, we would need data on when researchers began their affiliation with ORUs.
Desafortunadamente, the UC Davis Office of Research does not keep such data.

Estudios de ciencias cuantitativas

1018

Interdisciplinary impacts of Organized Research Units

Keeping these limitations of any causal interpretation in mind, the productivity findings of
this study suggest that ORUs—and similar infrastructure for interdisciplinary research—may
have a key social network formation role. Hicks and Simmons (2019) and Hicks, Coil et al.
(2019) also analyzed specific interdisciplinary research funding programs, finding evidence
that these programs appeared to be effective at supporting novel collaborations and stimu-
lating the formation of a new research community, respectivamente.

Turning to discursive impacts, the data do not appear to be compatible with hypotheses 1
o 2, discursive breadth or departmental distance; nor with hypothesis 3, relative distance from
ORU vs. departamento, when read individualistically. Aquí, the expected effects of interdisci-
plinarity (rather than multidisciplinarity) do not appear. These results suggest that, if ORUs
at UC Davis have any effect on researchers’ location and distribution in “discursive space,"
they have fostered multidisciplinarity rather than interdisciplinarity.

There is some support for hypothesis 3, relative distance from ORU vs. departamento, if this
comparison is interpreted at an organizational rather than individual level. Estos hallazgos
indicate that ORU affiliated researchers are located “away” from the mainstream of their
home disciplines, and are closer to ORU-distinctive research questions and methods. Pero,
thinking of the results for hypothesis 2, non-ORU-affiliated researchers are not homogeneous,
and many of them are located just as far from the disciplinary “center” as ORU-affiliated
investigadores. Eso es, both ORU-affiliated and non-ORU-affiliated faculty are distributed around
the disciplinary “center,” with comparable magnitudes; but ORU-affiliated faculty have a
distinct direction.

JMIE is an important exception to the pattern observed for hypothesis 3. The median JMIE
researcher is closer to their home department than to the JMIE topic distribution. JMIE also has
the widest topic distribution. Where ORUs such as PICN and BML/CMSI appear to be tightly
focused on a small set of issues (represented by a single topic), JMIE appears to be much more
heterogeneous (spread across several topics). The evidence from JMIE consistently points to
multidisciplinarity rather than interdisciplinarity.

Multidisciplinarity is often seen as inferior to interdisciplinarity; Por ejemplo, Holbrook
defines multidisciplinarity as “the (mere) juxtaposition of two or more academic disciplines
focused on a single problem” (Holbrook, 2013, pag. 1867, parentheses in original, my emphasis).
Sin embargo, interdisciplinary research faces serious challenges that multidisciplinary research
might avoid. Primero, genuine integration across interdisciplinary lines is extremely difficult, no
merely for pragmatic/logistical reasons but for deep conceptual reasons as well. Eigenbrode
et al. (2007) argue that research disciplines are divided not only by subject matter, pero también
by conceptual frameworks, research methods, standards of evidence, emphasis on the social
(rather than narrowly epistemic) value of research, and metaphysical assumptions about the
nature of the world. Drawing on philosophy of language and philosophy of science, ellos
develop a workshop-style intervention designed to make these divides explicit in cross-
disciplinary teams (O’Rourke & Crowley, 2012); although clarifying the problem is not the same
as solving it. Holbrook (2013) takes this kind of insight further, drawing on the work of Kuhn and
MacIntyre to argue that interdisciplinary collaboration is conceptually incoherent because it
would require crossing the boundaries of incommensurable conceptual schemes. Even if we
do not follow Holbrook (2013) to this radical conclusion, we can recognize the point that inter-
disciplinary research will tend to face deep communicative-conceptual challenges.

Además, Brister (2016) notes that cross-disciplinary collaborations can exemplify the
same status and power hierarchies as academia more generally, leading to a phenomenon that
she calls “disciplinary capture.” For example, as a natural science, biology generally has

Estudios de ciencias cuantitativas

1019

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Interdisciplinary impacts of Organized Research Units

higher status than anthropology; this hierarchy appeared in a collaboration between biologists
and anthropologists, with the result that “Both groups of scientists … perceived that conserva-
tion activities are dominated by biological research” (Brister, 2016, pag. 86). Fernández Pinto
(2016) makes similar points in terms of “scientific imperialism”.

Because of these challenges, interdisciplinary research may be difficult, to the point of being
highly impractical, without specific interventions or institutional designs. Even if successful in-
terdisciplinary research can bring the epistemic and social benefits that it is supposed to, estos
may not be worth the costs required in particular cases. So, all things considered, En particular
cases multidisciplinarity might be preferable to interdisciplinarity.

EXPRESIONES DE GRATITUD

Thanks to Duncan Temple Lang and Jane Carlen for advice on the analytical approaches used
in this study.

COMPETING INTERESTS AND FUNDING
The author’s postdoctoral fellowship at UC Davis was funded by a gift to the university from
Elsevier. The funder had no influence on the design, data collection, análisis, or interpretation
of this study.

DISPONIBILIDAD DE DATOS

Unless otherwise noted, all data used in this project was retrieved from Scopus, using either
the web interface or application programming interface (API), between November 2018 y
Junio 2019. Due to intellectual property restrictions the data cannot be made publicly avail-
capaz. Some downstream analysis files may be provided upon request. All code used in data
collection and analysis is available at https://github.com/dhicks/orus.

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

REFERENCIAS

Amrhein, v., Trafimow, D., & Greenland, S. (2018). Inferential
statistics as descriptive statistics: There is no replication crisis if
we don’t expect replication. The American Statistician, 73(sup1),
262–270. https://doi.org/10.1080/00031305.2018.1543137
andersen, h. (2012). Conceptual development in interdisciplinary
investigación. In U. Feest & F. Steinle (Editores.), Scientific concepts and
investigative practice (páginas. 3–271). De Gruyter. https://doi.org/10
.1515/9783110253610.271

Beaudry, C., & Larivière, V. (2016). Which gender gap? Factors
affecting researchers’ scientific impact in science and medicine.
Política de investigación, 45(9), 1790–1817. https://doi.org/10.1016/j
.respol.2016.05.009

rubia, V. D., Guillaume, J.-L., Lambiotte, r., & Lefebvre, mi. (2008).
Fast unfolding of communities in large networks. Diario de
Statistical Mechanics: Theory and Experiment, 2008(10), P10008.
https://doi.org/10.1088/1742-5468/2008/10/P10008

Boyack, k. w., Klavans, r., & Börner, k. (2005). Mapping the back-
bone of science. cienciometria, 64(3), 351–374. https://doi.org
/10.1007/s11192-005-0255-6

Boyack, k. w., Hombre nuevo, D., Duhon, R. J., Klavans, r., Patek, METRO.,
… Börner, k. (2011). Clustering more than two million biomed-
ical publications: Comparing the accuracies of nine text-based
similarity approaches. MÁS UNO, 6(3), e18029. https://doi.org
/10.1371/diario.pone.0018029, PubMed: 21437291

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Brister, mi. (2016). Disciplinary capture and epistemological obsta-
cles to interdisciplinary research: Lessons from central African
conservation disputes. Studies in History and Philosophy of
Science Part C: Studies in History and Philosophy of Biological
and Biomedical Sciences, 56, 82–91. https://doi.org/10.1016/j
.shpsc.2015.11.001, PubMed: 26651422

Bruun, h., Hukkinen, J., Huutoniemi, K., & Klein, j. t. (2005).
Promoting interdisciplinary research: The case of the Academy
of Finland (pag. 206). Helsinki, Finland: Academy of Finland.
Cameron, mi. Z., Blanco, A. METRO., & Gray, METRO. mi. (2016). Solving the
productivity and impact puzzle: Do men outperform women,
or are metrics biased? BioScience, 66(3), 245–252. https://doi
.org/10.1093/biosci/biv173

Cartwright, norte. (1999). The dappled world. Retrieved from https://
b o o k s . g o o g l e . c o m / b o o k s ? i d = t O F v _ i 9 o i A g C & p r i n t s e c
=frontcover. https://doi.org/10.1017/CBO9781139167093

Chauvin, S., Mulsant, B. h., Sockalingam, S., Stergiopoulos, v.,
taylor, V. h., & Vigod, S. norte. (2019). Gender differences in
research productivity among academic psychiatrists in Canada.
The Canadian Journal of Psychiatry, 64(6), 415–422. https://doi
.org/10.1177/0706743718802798, PubMed: 30616409

collins, h., evans, r., & Gorman, METRO. (2007). Trading zones and interac-
tional expertise. Studies in History and Philosophy of Science Part A,
38(4), 657–666. https://doi.org/10.1016/j.shpsa.2007.09.003

Estudios de ciencias cuantitativas

1020

Interdisciplinary impacts of Organized Research Units

CRAN Team & Temple Lang, D. (2020). RCurl: General Network
(HTTP/ FTP/…) Client Interface for R ( Versión 1.98-1.2).
Retrieved from https://CRAN.R-project.org/package=RCurl

DiPrete, t. A., & Eirich, GRAMO. METRO. (2006). Cumulative advantage as a
mechanism for inequality: A review of theoretical and empirical
developments. Annual Review of Sociology, 32(1), 271–297.
https://doi.org/10.1146/annurev.soc.32.061604.123127

Eigenbrode, S. D., O’Rourke, METRO., Wulfhorst, j. D., Althoff, D. METRO.,
Goldberg, C. S., … Bosque-Pérez, norte. A. (2007). Employing phil-
osophical dialogue in collaborative science. BioScience, 57(1),
55. https://doi.org/10.1641/B570109

Etzkowitz, h., & Kemelgor, C. (1998). The role of research centres
in the collectivisation of academic science. Minerva, 36(3),
271–288. https://doi.org/10.1023/A:1004348123030

Fernández Pinto, METRO. (2016). Economics imperialism in social episte-
mology: A critical assessment. Philosophy of the Social Sciences,
46(5), 443–472. https://doi.org/10.1177/0048393115625325
Galison, PAG. (1997). Image and logic: A material culture of micro-

física. University of Chicago Press.

Geiger, R. l. (1990). Organized Research Units—Their role in the
development of university research. The Journal of Higher
Educación, 61(1), 1–19. https://doi.org/10.1080/00221546.1990
.11775088

Ghiasi, GRAMO., Larivière, v., & Sugimoto, C. R. (2015). On the compli-
ance of women engineers with a gendered scientific system.
MÁS UNO, 10(12), e0145931. https://doi.org/10.1371/journal
.pone.0145931, PubMed: 26716831

Han, X. (2020). Evolution of research topics in LIS between 1996
y 2019: An analysis based on latent Dirichlet allocation topic
modelo. cienciometria, 125(3), 2561–2595. https://doi.org/10
.1007/s11192-020-03721-0

Hellinger Distance. (2021). In Wikipedia. Retrieved from https://en
.wikipedia.org/w/index.php?title=Hellinger_distance&oldid
=1001077827

Hicks, D. j. (2016). Bibliometrics for social validation. MÁS UNO,
11(12), e0168597. https://doi.org/10.1371/journal.pone
.0168597, PubMed: 28005974

Hicks, D. J., Coil, D. A., Stahmer, C. GRAMO., & Eisen, j. A. (2019).
Network analysis to evaluate the impact of research funding on
research community consolidation. MÁS UNO, 14(6), e0218273.
https://doi.org/10.1371/journal.pone.0218273, PubMed:
31211808

Hicks, D., & Melkers, j. (2012). Bibliometrics as a tool for research
evaluación. In A. Link & norte. Vornatas (Editores.), Handbook on the
theory and practice of program evaluation (páginas. 323–349).
Cheltenham, Reino Unido, and Northampton, MAMÁ: Edward Elgar.
Retrieved from https://works.bepress.com/diana_hicks/31/

Hicks, D., & Simmons, R. (2019). The National Robotics Initiative:
A five-year retrospective. IEEE Robotics Automation Magazine,
26(3), 2–9. https://doi.org/10.1109/MRA.2019.2912860

Hicks, D. J., Stahmer, C., & Herrero, METRO. (2018). Impacting capabili-
corbatas: A conceptual framework for the social value of research.
Frontiers in Research Metrics and Analytics, 3, 24. https://doi
.org/10.3389/frma.2018.00024

Hicks, D. J., & Stapleford, t. A. (2016). The virtues of scientific
práctica: MacIntyre, virtue ethics, and the historiography of
ciencia. Isis, 107(3), 449–472. https://doi.org/10.1086/688346,
PubMed: 28707850

Hitchcock, C., & Rédei, METRO. (2021). Reichenbach’s common cause
principle. In E. norte. Zalta (Ed.), The Stanford encyclopedia of phi-
losophy (Verano 2021). Metaphysics Research Lab, stanford
Universidad. Retrieved from https://plato.stanford.edu/archives
/sum2021/entries/physics-Rpcc/

Hofstra, B., Kulkarni, V. v., Munoz-Najar Galvez, S., Él, B., Jurafsky,
D., & McFarland, D. A. (2020). The diversity–innovation paradox
in science. procedimientos de la Academia Nacional de Ciencias,
201915378. https://doi.org/10.1073/pnas.1915378117,
PubMed: 32291335

Holbrook, j. B. (2013). What is interdisciplinary communication?
Reflections on the very idea of disciplinary integration. Synthese,
190(11), 1865–1879. https://doi.org/10.1007/s11229-012-0179-7
Huutoniemi, K., Klein, j. T., Bruun, h., & Hukkinen, j. (2010).
Analyzing interdisciplinarity: Typology and indicators. Investigación
Política, 39(1), 79–88. https://doi.org/10.1016/j.respol.2009.09.011
James, GRAMO., Witten, D., Hastie, T., & Tibshirani, R. (2013). Un
introduction to statistical learning. Nueva York, Nueva York: Saltador
Nueva York. Retrieved from https://link.springer.com/10.1007/978
-1-4614-7138-7

Kuhn, t. S. (1996). The structure of scientific revolutions (3tercera ed.).
chicago, IL: University of Chicago Press. https://doi.org/10.7208
/chicago/9780226458106.001.0001

Larivière, v., En, C., Gingras, y., Cronin, B., & Sugimoto, C. R. (2013).
Bibliometrics: Global gender disparities in science. Noticias de la naturaleza,
504(7479), 211–213. https://doi.org/10.1038/504211a, PubMed:
24350369

Sotavento, C. j. (2020). The reference class problem for credit valuation in
ciencia. Philosophy of Science, 87(5), 1026–1036. https://doi.org
/10.1086/710615

Leydesdorff, l., & Rafols, I. (2011). Indicators of the interdisciplin-
arity of journals: Diversity, centrality, and citations. Diario de
Informetrics, 5(1), 87–100. https://doi.org/10.1016/j.joi.2010.09
.002

Longino, h. mi. (2013). Studying human behavior: How scientists
investigate aggression and sexuality. University of Chicago Press.
https://doi.org/10.7208/chicago/9780226921822.001.0001

Lu, K., & Wolfram, D. (2012). Measuring author research related-
ness: A comparison of word-based, topic-based, and author coci-
tation approaches. Journal of the American Society for Information
Science and Technology, 63(10), 1973–1986. https://doi.org/10
.1002/asi.22628

Malaterre, C., Lareau, F., Pulizzotto, D., & St-Onge, j. (2020). Eight
journals over eight decades: A computational topic-modeling
approach to contemporary philosophy of science. Synthese.
https://doi.org/10.1007/s11229-020-02915-6

McElreath, R. (2016). Statistical rethinking: A Bayesian course with

examples in R and Stan. CRC Press.

Merton, R. k. (1968). The Matthew Effect in science: The reward and
communication systems of science are considered. Ciencia,
159(3810), 56–63. https://doi.org/10.1126/science.159.3810
.56, PubMed: 5634379

Mingers, J., & Leydesdorff, l. (2015). A review of theory and prac-
tice in scientometrics. European Journal of Operational Research,
246(1), 1–19. https://doi.org/10.1016/j.ejor.2015.04.002

morgan, A. C., Economou, D. J., Way, S. F., & cláusula, A. (2018).
Prestige drives epistemic inequality in the diffusion of scientific
ideas. EPJ Data Science, 7(1), 40. https://doi.org/10.1140/epjds
/s13688-018-0166-4

Naturaleza. (2016). Meet the challenge of interdisciplinary science.
Naturaleza, 534(7609), 589–590. https://doi.org/10.1038/534589b,
PubMed: 27357757

Nichols, l. GRAMO. (2014). A topic model approach to measuring interdis-
ciplinarity at the National Science Foundation. cienciometria,
100(3), 741–754. https://doi.org/10.1007/s11192-014-1319-2
O’Rourke, METRO., & Crowley, S. (2012). Philosophical intervention and
cross-disciplinary science: The story of the Toolbox Project.
Synthese, 1–18. https://doi.org/10.1007/s11229-012-0175-y

Estudios de ciencias cuantitativas

1021

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
3
9
9
0
1
9
7
0
7
0
4
q
s
s
_
a
_
0
0
1
5
0
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Interdisciplinary impacts of Organized Research Units

Ortmann, METRO., Klimenta, METRO., & Brandes, Ud.. (2016). A sparse stress
modelo. arXiv:1608.08909 [Cs]. Retrieved from https://arxiv.org
/abs/1608.08909

Pence, C., & Hicks, D. (n.d.). Challenges for ‘community’ in science

and values: Cases from robotics research.

Potochnik, A. (2017). Idealization and the aims of science. Universidad
of Chicago Press. https://doi.org/10.7208/chicago/9780226507194
.001.0001

R Core Team. (2018). R: A language and environment for statistical
computing. R Foundation for Statistical Computing. Retrieved
from https://www.r-project.org/

Rafols, I., & Meyer, METRO. (2010). Diversity and network coherence as
indicators of interdisciplinarity: Case studies in bionanoscience.
cienciometria, 82(2), 263–287. https://doi.org/10.1007/s11192
-009-0041-y

Rehs, A. (2020). A structural topic model approach to scientific
reorientation of economics and chemistry after German reunifi-
catión. cienciometria, 125(2), 1229–1251. https://doi.org/10
.1007/s11192-020-03640-0

Roberts, METRO. MI., Stewart, B. METRO., Tingley, D., lucas, C., Leder-Luis,
J., … Rand, D. GRAMO. (2014). Structural topic models for open-
ended survey responses. American Journal of Political Science,
58(4), 1064–1082. https://doi.org/10.1111/ajps.12103

Rosen-Zvi, METRO., Griffiths, T., Steyvers, METRO., & Smyth, PAG. (2004). El
author-topic model for authors and documents. Actas de
the 20th Conference on Uncertainty in Artificial Intelligence
(páginas. 487–494). AUAI Press.

Sá, C. METRO. (2008). ‘Interdisciplinary strategies’ in U.S. research univer-
ciudades. Higher Education, 55(5), 537–552. https://doi.org/10.1007
/s10734-007-9073-5

Sá, C. METRO., & Oleksiyenko, A. (2011). Between the local and the
global: Organized research units and international collaborations
in the health sciences. Higher Education, 62(3), 367–382. https://
doi.org/10.1007/s10734-010-9393-8

verano, R. (1994). Serving two masters: Center for Consumer
Research at the University of California, davis, 1976–1992.
Journal of Consumer Affairs, 28(1), 170–186. https://doi.org/10
.1111/j.1745-6606.1994.tb00821.x

spaCy. (2018). Industrial-strength Natural Language Processing
(NLP) with Python and Cython ( Versión 2.0.3) [Python].
Explosion AI. Retrieved from https://github.com/explosion/spaCy

Stahler, GRAMO. J., & Tash, W.. R. (1994). Centers and institutes in the
research university: Asuntos, problemas, and prospects. The Journal
of Higher Education, 65(5), 540–554. https://doi.org/10.1080
/00221546.1994.11778519

Star, S. l., & Griesemer, j. R. (1989). Institutional ecology, ‘translations’
and boundary objects: Amateurs and professionals in Berkeley’s
Museum of Vertebrate Zoology, 1907–39. Social Studies of Science,
19(3), 387–420. https://doi.org/10.1177/030631289019003001
Symonds, METRO. R. MI., Gemmell, norte. J., Braisher, t. l., Gorringe, k. l., &
Elgar, METRO. A. (2006). Gender differences in publication output:
Towards an unbiased metric of research performance. PLOS
ONE, 1(1), e127. https://doi.org/10.1371/journal.pone.0000127,
PubMed: 17205131

Taddy, METRO. A. (2012). On estimation and selection for topic models.
Proceedings of the 15th International Conference on Artificial Intel
ligence and Statistics (AISTATS), 10. La Palma, Canary Islands.
van Arensbergen, PAG., van der Weijden, I., & van den Besselaar, PAG.
(2012). Gender differences in scientific productivity: A persisting
fenómeno? cienciometria, 93(3), 857–868. https://doi.org/10
.1007/s11192-012-0712-y, PubMed: 23162173

van der Maaten, l., & Hinton, GRAMO. (2008). Visualizing data using
t-SNE. Journal of Machine Learning Research, 9(86), 2579–2605.
Retrieved from https://jmlr.org/papers/v9/vandermaaten08a.html
Wagner, C. S., Roessner, j. D., Bobb, K., Klein, j. T., Boyack, k. w., …
Börner, k. (2011). Approaches to understanding and measuring
interdisciplinary scientific research (IDR): A review of the literature.
Journal of Informetrics, 5(1), 14–26. https://doi.org/10.1016/j.joi
.2010.06.004

Wickham, h., & RStudio. (2017). tidyverse: Easily install and load
the ‘Tidyverse’ ( Versión 1.2.1). Retrieved from https://CRAN.R
-project.org/package=tidyverse

yan, MI., & Ding, Y. (2012). Scholarly network similarities: Cómo
bibliographic coupling networks, citation networks, cocitation
redes, topical networks, coauthorship networks, and coword
networks relate to each other. Journal of the American Society for
Information Science and Technology, 63(7), 1313–1326. https://
doi.org/10.1002/asi.22680

Youtie, J., kay, l., & Melkers, j. (2013). Bibliographic coupling and
network analysis to assess knowledge coalescence in a research
center environment. Research Evaluation, 22(3), 145–156.
https://doi.org/10.1093/reseval/rvt002

Estudios de ciencias cuantitativas

1022

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d