RESEARCH ARTICLE - IA de Investigación especializada en el MIT

ARTÍCULO DE INVESTIGACIÓN

Can altmetrics reflect societal impact
considerations?: Exploring the potential of
altmetrics in the context of a sustainability science
research center

un acceso abierto

diario

Omar Kassab1, Lutz Bornmann2

, and Robin Haunschild3

1Swiss Federal Institute of Technology (ETH Zurich), Professorship for Social Psychology and Research on Higher Education,
Andreasstrasse 15, 8050 Zurich, Suiza
2Administrative Headquarters of the Max Planck Society, Division for Science and Innovation Studies, Hofgartenstrasse 8,
80539 Munich, Alemania
3Max Planck Institute for Solid State Research, Heisenbergstrasse 1, 70569 Stuttgart, Alemania

Palabras clave: altmetrics, bibliometría, MHq indicator, research center, societal impact, sustainability
ciencia

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
a
_
0
0
0
3
2
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

ABSTRACTO

Societal impact considerations play an increasingly important role in research evaluation. En
particular, in the context of publicly funded research, proposal templates commonly include
sections to outline strategies for achieving broader impact. Both the assessment of the
strategies and the later evaluation of their success are associated with challenges in their own
bien. Ever since their introduction, altmetrics have been discussed as a remedy for
assessing the societal impact of research output. On the basis of data from a research center in
Suiza, this study explores their potential for this purpose. The study is based on the
documentos (and the corresponding metrics) published by about 200 either accepted or rejected
applicants for funding by the Competence Center Environment and Sustainability (CCES). El
results of the study seem to indicate that altmetrics are not suitable for reflecting the societal
impact of research that was considered: The metrics do not correlate with the ex ante
considerations of an expert panel.

INTRODUCCIÓN

Many studies dealing with the societal impact of research begin by describing a paradigmatic
transformation in research policy that has presumably led to an increased accountability of
publicly funded research. Researchers and universities, according to this narrative, would in-
creasingly have to justify their work to the tax-paying public. This almost confrontational por-
trayal of the relationship could make the reader believe that the public is concerned with a
petty cost–benefit calculation to tease out their return on investment. Sin embargo, this simplified
view undermines the potentially genuine interest of societal actors to inform and educate
themselves on the basis of scientific facts. Especially in times of rapid technological develop-
mentos, the interaction between science and society is easier than ever.

The very emergence of social media, Por ejemplo, has heralded a new age for the public
dissemination of scientific knowledge. It therefore comes as no surprise that “altmetrics,” an
endeavor to quantitatively represent mentions and interactions on social media platforms such

Citación: Kassab, o., Bornmann, l., &
Haunschild, R. (2020). Can altmetrics
reflect societal impact considerations?:
Exploring the potential of altmetrics in
the context of a sustainability science
research center. Quantitative Science
Estudios, 1(2), 792–809. https://doi.org/
10.1162/qss_a_00032

DOI:
https://doi.org/10.1162/qss_a_00032

Recibió: 29 Agosto 2019
Aceptado: 08 Febrero 2020

Autor correspondiente:
Omar Kassab
omar.kassab@gess.ethz.ch

Editor de manejo:
Juego Waltman

Derechos de autor: © 2020 Omar Kassab, Lutz
Bornmann, and Robin Haunschild.
Publicado bajo Creative Commons
Atribución 4.0 Internacional (CC POR 4.0)
licencia.

La prensa del MIT

Can altmetrics reflect societal impact considerations?

as Twitter or Facebook, have been proposed as a means to evaluate the societal impact of
research ex post (see the literature overview by Bornmann, 2014). Yet despite the consensus
over their potential for impact assessment, the jury is still out as to what kind of impact alt-
metrics scores actually reflect. Addressing this puzzle, Bornmann, Haunschild, and Adams
(2019) compared peer assessments of societal impact of research with altmetrics scores for
the corresponding publications. Their results reveal that altmetrics seem to measure public
“discussions” around research rather than societal impact, further qualifying that the latter
may more likely be assessed by experts in a specific field. Sin embargo, there are also other
empirical findings suggesting a contrary conclusion. Wooldridge and King (2019), for exam-
por ejemplo, used the same data set as Bornmann et al. (2019) but other methods, and concluded that
“the work presented in this study provides direct evidence, por primera vez, of a correlation
between expert peer review of the societal impact of research and altmetric data from the
publications defining the underpinning research” (pag. 281). Against the backdrop of these
contradicting results, it is necessary to advance further empirical investigations about the cor-
relation between assessments of societal impact of research and altmetrics scores.

Taking up the question in the context of a research center, the CCES in Switzerland, el
study examines the altmetrics scores of journal articles published by researchers either ac-
cepted or rejected for funding by CCES. As a research field “defined by the problems it
addresses rather than by the disciplines it employs” (clark, 2007), sustainability science rep-
resents a prime case for solution-oriented research of high societal relevance (Yarime,
Trencher, et al., 2012; Brandt, Ernst, et al., 2013; Wiek, Talwar, et al., 2014; Kassab,
2019). De este modo, whether the research was funded or rejected depended not solely on the as-
sessment of the scientific quality, but initially on whether the prospect of societal impact
was explicitly outlined in the proposal or not (CCES, personal communication). We explore
in this study whether this latter criterion is reflected in later altmetrics scores: Do papers of
researchers funded by CCES receive higher altmetrics scores than papers from rejected re-
searchers? Or in other words, using another data set than Bornmann et al. (2019) y
Wooldridge and King (2019), this study targets the question of whether or not altmetrics
scores are consistent with ex ante assessments of societal impact considerations.

The remainder of the article is structured as follows: Sección 2 introduces the case and de-
scribes the hypothesized relationship between societal impact assessments and altmetrics
puntuaciones. Sección 3 then gives an overview of the data and the methods used for the investiga-
ción. Sección 4 presents the results of the study, y sección 5 discusses them to draw conclu-
siones. Finalmente, sección 6 outlines the limitations of the study while giving indications for further
research and recommendations.

2. CAN ALTMETRICS REFLECT SOCIETAL IMPACT CONSIDERATIONS?

2.1. Case Description: A Sustainability Science Research Center in Switzerland

The CCES was founded in 2006 for a period of 10 años (until 2016) to foster inter- and trans-
disciplinarity within and between the six institutions that constitute the ETH Domain, a union
of Swiss Federal universities and research institutes. Strategically managed by the ETH Board,
the ETH Domain comprises the two Federal Institutes of Technology in Zurich (ETH Zurich)
and Lausanne (EPFL), as well as four research institutes: the Paul Scherrer Institute (PSI), el
Swiss Federal Institute for Forest, Snow and Landscape Research ( WSL), the Swiss Federal
Laboratories for Materials Science and Technology (Empa), and the Swiss Federal Institute
of Aquatic Science and Technology (Eawag).

Estudios de ciencias cuantitativas

793

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
a
_
0
0
0
3
2
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Can altmetrics reflect societal impact considerations?

CCES was established with the mission to “identify the relevant questions and the appro-
priate answers to foster the sustainable development of a future society while minimizing the
impact on the environment” (CCES, 2005). To comprehensively achieve this mission, CCES
operated in three areas of activity: investigación, capacity-building, and public outreach. Objetivos
have been set for each of the three areas, with a total of five goals. In the area of “research,"
three goals were defined: (a) foster major inter- and transdisciplinary research advancements in
the areas of environment and sustainability, (b) establish the CCES partner institutions as na-
tional and international focal points for the areas of environment and sustainability, y (C)
achieve a long-term structuring effect and a coherent strategy for the areas of environment
and sustainability. In the area of “capacity-building,” the goal was (d) to establish a strong
and wide-ranging education program for the areas of environment and sustainability. And lastly,
the goal set in the area of “public outreach” was (mi) to achieve a visible societal impact with a
focus on socioeconomic implementation.

Activities at CCES were clustered in 26 projects along five thematic areas of environment
and sustainability science: (a) Climate and Environmental Change, (b) Sustainable Land Use,
(C) Food, Environment, and Health, (d) Natural Resources, y (mi) Natural Hazards and Risks.
Some exemplary projects included OPTIWARES, in which researchers worked on optimizing
the use of wood as a renewable energy source, TRAMM, which aimed at developing early
warning systems for rapid mass movements in steep terrain, and the ADAPT project, cual
studied social and environmental constraints for large-scale dams and water resource manage-
mento (Kassab, Schwarzenbach, & Gotsch, 2018).

2.2. Societal Impact Considerations in the Evaluation Procedure

The few aforementioned synopses demonstrate exemplarily that projects funded by the re-
search center were characterized by a strong practice orientation. This property is based on
the notion that sustainability science concentrates on the most pressing challenges facing
human society and the development of concrete solutions (Yarime et al., 2012; Kajikawa,
Tacoa, & Yamaguchi, 2014; SDSN, 2017). In order to find these solutions, sin embargo, es
necessary not only to overcome disciplinary boundaries, through interdisciplinarity, pero también
transcend the university ecosystem and engage other stakeholders from society, negocio, y
política, through transdisciplinarity approaches (Pohl, 2010; Lang, Wiek, et al., 2012). In terms
of the underlying research mode, sustainability science thus differs considerably from basic
investigación (clark, 2007; Mobjörk, 2010; Kates, 2011; Molinero, 2013).

The special attention given to inter- and transdisciplinarity as well as the objective to devel-
op applied solutions was explicitly reflected in the CCES evaluation procedure. For the purpose
of assessing the project proposals, an ad hoc Research Council (RC) was established. Consisting
de 17 researchers from the ETH Domain institutions, the RC was responsible for reviewing pro-
posals with respect to their overall suitability for CCES (see goals above). En particular, it was the
task of the RC to evaluate the added value of the project for CCES, stressing (a) societal rele-
vance, either as a goal to be achieved during the project duration or with an identified follow-
up implementation phase, (b) the importance of the project for long-term sustainability and for a
durable structuring effect, y (C) the relevance in the international context, and in particular,
the potential for applications in developing countries (CCES, 2006). As this focus suggests, el
assessments of the RC were primarily based on the prospect of societal impact, reflecting the
three aforementioned dimensions, and did not include an evaluation of the scientific quality. De hecho,
only if the projects passed the initial assessment were they forwarded to the next stage, cual estafa-
sisted of a classical peer review procedure coordinated by the ETH Zurich Research Commission.

Estudios de ciencias cuantitativas

794

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
a
_
0
0
0
3
2
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Can altmetrics reflect societal impact considerations?

Given the still inconclusive debate about the validity of altmetrics for reflecting the soci-
etal impact of research, the question that lies at the heart of this study is whether or not there
is a relationship between ex ante assessments of societal impact and altmetrics scores. Nosotros
approach the answer to this question indirectly: According to the CCES evaluation proce-
dure outlined above, special emphasis was attributed to the prospect of societal impact.
Under the premise that research funded through CCES would yield more societal impact
than the research of rejected applicants, and assuming that altmetrics scores are capable
of reflecting this impact, the hypothesis arises that the researchers funded by CCES achieve
higher impact in terms of altmetrics scores with their research than those who were not
funded. Should the findings of this study corroborate the hypothesis, this would lead to
the conclusion that altmetrics are indeed capable of reflecting the ex ante societal impact
considerations of the RC. Sin embargo, should the results not confirm the hypothesis, this does
not automatically imply the opposite. Bastante, this would raise the question of what else alt-
metric scores are indicative of. De hecho, a refutation of the hypothesis could also be inter-
impacto
preted in a way that
considerations in the assessments (even though this was explicitly demanded), but rather
focused on other aspects. In what follows, we describe the data and the methods we use
to test the hypothesis empirically.

take sufficient account of societal

the RC did not

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
a
_
0
0
0
3
2
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

3. DATA AND METHODS

3.1. Description of Altmetrics

We acknowledge that altmetrics are heterogeneous in many ways and specifically with regard
to which aspect of societal impact they actually reflect (if any). We considered six different
altmetric sources in this study, including Twitter, Wikipedia, policy-related documents, blogs,
Facebook, and news. They differ strongly with regard to the effort and the process preceding
the actual mention, contenido, and substance of the information that is communicated, and also
the readership. While a tweet or a Facebook post is shared at the touch of a button, the thresh-
old for Wikipedia entries, blog posts, or mentions in news outlets is much higher. También, el
demographic background of the readership of policy-related documents as opposed to
Facebook posts is much more specific. Sin embargo, we chose those types since they have
been frequently used and investigated in previous altmetrics studies (see Bornmann et al.,
2019), qualifying them as “standard” sources.

Twitter (https://www.twitter.com) is a popular microblogging platform. Tweets may refer to
the content of scientific publications, but it seems that they do not correlate with traditional
citas (Bornmann, 2015). En cambio, they may reflect discussion around these publications
(Haustein, Peters, et al., 2014), possibly by public users (Haustein, Larivière, et al., 2014;
Yu, 2017), but this is not entirely clear, as outlined by Sugimoto, Work, et al. (2016). The re-
sults by Andersen and Haustein (2015) suggest that tweets reflect the attractiveness of papers
for a broader audience. Sin embargo, contradictory results are also available: “A multi-year cam-
paign has sought to convince us that counting the number of tweets about papers has value.
Todavía, reading tweets about dental journal articles suggested the opposite. This analysis found:
obsessive single issue tweeting, duplicate tweeting from many accounts presumably under
centralized professional management, bots, and much presumably human tweeting duplica-
tivo, almost entirely mechanical and devoid of original thought” (Robinson-Garcia, costas,
et al., 2017). In the study at hand, the number of tweets (and retweets) including references
to scientific papers in our data set is counted.

Estudios de ciencias cuantitativas

795

Can altmetrics reflect societal impact considerations?

Wikipedia (https://www.wikipedia.org) is a free encyclopedia platform which includes ed-
itable content (Mas-Bleda & Thelwall, 2016). Although contributors to this platform include
scholarly references, most of them do not refer to research papers (Principal, 2014). If scientific
papers are cited, open access (OA) papers seem to be preferred (Teplitskiy, Lu, & Duede,
2017; Dehdarirad, Didegah, & Sotudeh, 2018). Guglielmi (2018) reports on Wikipedia’s most
frequently mentioned papers. Sin embargo, this list does not correspond with lists based on tra-
ditional citations: Study results suggest that Wikipedia mentions do not correlate with citations
(Samoilenko & Yasseri, 2014). A Wikipedia case study with papers on wind power showed
that less than 1% of relevant papers have been cited on Wikipedia, “implying that the direct
societal impact through the Wikipedia is extremely small for Wind Power research” (serrano-
López, Ingwersen, & Sanz-Casado, 2017, pag. 1471). Kousha and Thelwall (2017) found that
solo 5% of papers had any citation from Wikipedia—based on a significantly larger sample
of papers than considered by Serrano-López et al. (2017). en este estudio, the number of
Wikipedia articles with reference to papers in our data set is counted.

Policy-related documents are an important source of altmetrics, since one is interested in
the impact of science on the policy realm (OPENing UP, 2016; Vilkins & Grant, 2017).
Mentions in these documents are searched using text mining databases of, por ejemplo, el
World Health Organization or European Food Safety Authority (Bornmann, Haunschild, &
Marx, 2016; Haunschild & Bornmann, 2017). Haunschild and Bornmann (2017) reported that
the company Altmetric tracks more than 100 policy sources (en 2015). Tattersall and Carroll
(2018) analyzed nearly 100 papers published by authors from the University of Sheffield: El
“research topics with the greatest policy impact are medicine, dentistry, and health, seguido
by social science and pure science.” Papers published OA seem to have an advantage to be
cited in policy-related documents (Vilkins & Grant, 2017). Sin embargo, the impact of papers (OA
or not) on these documents is usually very low, as the results of Haunschild and Bornmann
(2017) reveal: “Less than 0.5% of the papers published in different subject categories are men-
tioned at least once in policy-related documents” (pag. 1209). The study of Bornmann et al.
(2016) shows that “only 1.2 % (norte = 2,341) have at least one policy mention” (pag. 1477). El
authors analyzed a large set of 191,276 publications from the field of climate change, cual es
policy relevant. en este estudio, the number of policy-related documents with references to pa-
pers in our data set is counted.

Blogs are written about scientific papers, including formal or informal citations of papers
(Shema, Bar-Ilan, & Thelwall, 2014a). These citations can be counted—with the limitation
that informal citations lead to uncertainty (Principal & Hemminger, 2010; Luzón, 2013; Shema,
Bar-Ilan, & Thelwall, 2014b). Since blogs allow extended informal discussions about research,
they are an interesting altmetrics source (Fausto, Machado, et al., 2012; Shema, Bar-Ilan, &
Thelwall, 2012a). Blogging may be a bridge between the general public and the research area
(Bonetta, 2007; Bar-Ilan, Shema, & Thelwall, 2014), whereby bloggers seem to have prefer-
ences for papers from high-impact journals and research in the life and behavioral sciences
(Shema, Bar-Ilan, & Thelwall, 2012b). Sin embargo, a study revealed that bridging public and re-
search “was one of the less popular motivations for academics to blog” (Mewburn & Thomson,
2013, pag. 1113). The literature overview published by Sugimoto et al. (2016) shows that the cov-
erage of papers in blog mentions is low, as is the correlation between blog mentions and tra-
ditional citations. en este estudio, the number of blog posts with references to the papers in our
data set is counted.

Facebook is a popular social networking and social media platform (Bik & Goldstein,
2013). Since users share papers among themselves, mentions of papers in posts or
Facebook likes can be counted. Ringelhan, Wollersheim, and Welpe (2015) investigated

Estudios de ciencias cuantitativas

796

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
a
_
0
0
0
3
2
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Can altmetrics reflect societal impact considerations?

whether Facebook “likes” are an indicator of scientific impact. Their results show “an inter-
disciplinary difference in the predictive value of Facebook likes, según la cual
Facebook likes only predict citations in the psychological area but not in the nonpsycholog-
ical area of business or in the field of life sciences.” In this study, the number of Facebook
posts with references to scientific papers in our data set are counted (note that we did not
include likes).

News attention relates to scientific papers mentioned in news reports (via direct links or
unique identifiers in, Por ejemplo, the New York Times). On the basis of these paper mentions,
public attention can be counted. The overview of altmetrics studies published by Sugimoto
et al. (2016) reveals that the correlation between mentions of papers in news reports and tra-
ditional citations is between low and medium. According to Altmetric.com1, más que 2,000
different news sources are analyzed for news mentions. en este estudio, the number of news ar-
ticles with mentions of scientific papers in our data set is counted.

3.2. Data Set Used

We used the Web of Science ( WoS, Clarivate Analytics) custom data of our in-house database
and the database from the Competence Centre for Bibliometrics (CCB: http://www.bibliome-
trie.info). Both are derived from the Science Citation Index Expanded (SCI-E), Social Sciences
Citation Index (SSCI), and Arts and Humanities Citation Index (AHCI) produced by Clarivate
Analytics (Filadelfia, EE.UU). All publications published between 2011 y 2015 with a DOI
were exported with the following information: DOI, WoS UT (unique accession number from
WoS), WoS subject categories, publication year, citation counts with a three-year citation
window starting after the publication year, and Hazen percentiles. Percentiles are field-
and time-normalized impact scores that are between 0 (low citation impact) y 100 (alto
citation impact) (Bornmann, Leydesdorff, & Mutz, 2013). Raw citation data were taken from the
database maintained by the CCB. Both databases (our in-house database and the database
maintained by the CCB) were last updated at the end of April 2019. We kept only those pub-
lications that fulfilled the following two criteria: (a) the publication belongs to a field (superposición-
ping WoS category; Rons, 2012, 2014) to which at least one research center publication
belongs; (b) a requirement of at least 10 publications per field and publication year combination
has been set.

Altmetrics data were sourced from a locally maintained database using data shared with us
by Altmetric (https://www.altmetric.com) and dumped on October 8, 2019. For research pro-
jects, the company shares the data for free. The data include altmetric counts from sources
such as social networking, blogging, microblogging, wikis, and policy-relevant usage. We ap-
pended a mention count to each DOI using the following altmetrics sources: Twitter,
Facebook, blogs, noticias, policy documents, and Wikipedia (see above). One DOI not known
to the altmetrics database was recorded as “not mentioned.” Altmetrics data and information
about their unit status (applied for research center funding which was accepted or not) eran
appended to the publications via their DOI. Cifra 1 provides a schematic overview of how
the respective units were constructed: unit 0 contains all WoS papers that do not belong to
units 1 o 2. Unidad 1 contains the publications of 28 participants who had submitted project
proposals for CCES but were not funded. Unidad 2, Sucesivamente, contains the publications of 170 par-
ticipants that were affiliated with CCES as principal investigators and project partners. Unidad 2 es

1 https://www.altmetric.com/about-our-data/our-sources/.

Estudios de ciencias cuantitativas

797

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
a
_
0
0
0
3
2
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Can altmetrics reflect societal impact considerations?

Cifra 1. Schematic overview of the units an number of papers per unit.

further subdivided into units 3 y 4. Unidad 4 contains the papers that were published in the
research center context, while unit 3 contains papers that accepted applicants published be-
yond their project at the research center. The numbers of mentioned and not mentioned pub-
lications in the different altmetrics sources broken down by unit status and publication year are
mostrado en la tabla 1.

We acknowledge that the subdivision into units and the comparison between units is a
simplification of reality, especially with regard to the hypothesis to be tested. Mientras que la
CCES evaluation procedure took place on the project level at a specific moment in time,
the units here are constructed on the level of the entire publication output of researchers that
were funded or not funded by CCES. Además, we focus in this study on scientific publi-
cations as the main research output of the research center. While it would have been bene-
ficial to consider other outputs as well, such as those emanating from public outreach activities
(Kassab, 2019), we are constrained by the fact that altmetrics data are only available for out-
puts that have a unique and persistent identifier, such as a DOI. Sin embargo, besides altmetrics
datos, we also considered citation data (a) to compare the results with those based on altmetrics
data and (b) to investigate whether societal impact assessments correspond with traditional
impact scores.

3.3. Mantel-Haenszel Quotient (MHq)2

en este estudio, we compare the impact of papers published by various units (p.ej., papers pub-
lished by rejected or accepted applicants; ver figura 1). Since altmetrics data concern field-
specific differences (like citation data), field-normalized indicators should be used instead of
raw data for group comparisons. Sin embargo, it is a critical drawback of altmetrics data that they
are inflated by zeros: En el estudio actual, 5,586,077 documentos (71%) have no impact in any
altmetrics source. For zero-inflated data, it is not possible to use methods for field normal-
ization that are usually applied in bibliometrics (methods based on mean citations or cita-
tion percentiles; Bornmann et al., 2013). Since Bornmann and Haunschild (2018) y

2 The explanation of the MHq indicator has been mainly adopted from Bornmann and Haunschild (2018) y

Haunschild and Bornmann (2018).

Estudios de ciencias cuantitativas

798

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
a
_
0
0
0
3
2
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

C
a
norte

a
yo
t

metro
mi
t
r
i
C
s

r
mi
F
yo
mi
C
t

s
oh
C
i
mi
t
a
yo

metro
pag
a
C
t

C
oh
norte
s
i
d
mi
r
a
t
i
oh
norte
s
?

Mesa 1. Number of mentioned and not mentioned (and cited and not cited) documentos, respectivamente, broken down by data source, publication year, and funded or not-
funded groups

Unit Publication year

Mentioned

mentioned Mentioned

No
mentioned

Cited

No
cited

Twitter

Facebook

Blogs

Noticias

Policy documents

Wikipedia

2011

2012

2013

2014

2015

2011

2012

2013

2014

2015

2011

2012

2013

2014

2015

2011

2012

2013

2014

2015

2011

2012

2013

2014

2015

139,766

1,233,952

34,842

1,338,876

35,790

1,337,928

20,922

1,352,796

29,028

1,344,690

28,036

1,345,682 1,057,600 316,118

312,073

1,181,879

77,809

1,416,143

43,989

1,449,963

29,490

1,464,462

28,391

1,465,561

28,024

1,465,928 1,149,376 344,576

398,950

1,194,767

115,234

1,478,483

52,390

1,541,327

47,867

1,545,850

28,422

1,565,295

27,288

1,566,429 1,245,496 348,221

510,575

1,144,363

105,419

1,549,519

56,832

1,598,106

59,324

1,595,614

24,927

1,630,011

25,303

1,629,635 1,305,474 349,464

627,508

1,123,571

17,5196

1,575,883

60,213

1,690,866

76,049

1,675,030

20,868

1,730,211

24,118

1,726,961 1,394,085 356,994

179

285

301

165

173

271

294

164

113

690

650

611

511

180

627

618

560

489

172

118

127

151

140

750

793

822

755

294

685

755

761

727

285

111

129

152

134

726

777

821

753

321

665

744

763

725

312

122

143

154

129

750

813

843

761

315

685

777

784

732

306

109

131

159

132

719

787

856

787

337

660

752

800

760

328

121

147

165

143

742

808

869

800

338

677

770

805

771

329

120

139

163

146

734

788

862

782

336

668

750

797

755

328

Nota: WoS papers (unit 0; neither accepted, nor rejected for funding), papers published by rejected applicants (unit 1), and papers published by accepted applicants (unit 2). The papers from accepted applicants are further divided into
papers from funded projects (unit 4) and papers published in other contexts (unit 3).

q
tu
a
norte

t
i
t

t
i
v
mi
S
C
mi
norte
C
mi
S
tu
d
mi
s

7
9
9

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
a
_
0
0
0
3
2
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Can altmetrics reflect societal impact considerations?

Haunschild and Bornmann (2018) proposed the MHq indicator, which is specifically de-
signed for dealing with zero-inflated data in field normalization, we used the indicator in
el estudio actual.

For pooling data from multiple 2 × 2 cross tables based on such subgroups (which are part
of the larger population, including all papers in the considered time period), MH analysis is a
popular method (Mantel & Haenszel, 1959; Hollander & lobo, 1999; Sheskin, 2007).
According to Fleiss, Levin, and Paik (2003), the method “permits one to estimate the assumed
common odds ratio and to test whether the overall degree of association is significant.
Curiously, it is not the odds ratio itself but another measure of association that directly under-
lies the test for overall association … The fact that the methods use simple, closed-form for-
mulas has much to recommend it” (pag. 250). The results by Radhakrishna (1965) demonstrate
that the MH approach seems to be valid.

The MH analysis results in a summary odds ratio for multiple 2 × 2 cross tables, cual
Bornmann and Haunschild (2018) and Haunschild and Bornmann (2018) name MHq. Para
the comparison of the papers published by the applicants with reference sets in view of im-
pact, el 2 × 2 cross tables (which are pooled) consist of the number of papers mentioned and
not mentioned in subject category and publication year combinations f. En el 2 × 2 sujeto-
specific cross table (ver tabla 2), the cells af, bf, cf, and df, are defined as follows:

–

–
–

af is the number of mentioned papers published by unit g (p.ej., rejected applicants) en
subject category and publication year f,
bf is the number of not mentioned papers published by unit g in subject category and

publication year f,
cf is the number of mentioned papers in subject category and publication year f, y
df is the number of not mentioned papers published in subject category and publication

year f. Note that the papers of group g are also part of the papers in the world.

The following dummy variables are needed for the MH analysis:

¼ af df
nf

and R ¼

¼ bf cf
nf

and S ¼

;

f ¼1

;

Pf ¼ af

þ df
nf

and Qf ¼ 1−Pf

MHq ¼ R
S

where nf = af + bf + cf + df.

MHq is simply

The CIs for MHq are calculated following Fleiss et al. (2003). The variance of ln MHq is
estimated by

C
Var ln MHq

Þ ¼

(

PAG

1
2

F
f ¼1 Pf Rf
R2

PAG

d
F
f ¼1 Pf Sf
RS

þ Qf Rf

)

PAG
F
f ¼1 Qf Sf
S2

(5)

800

Estudios de ciencias cuantitativas

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
a
_
0
0
0
3
2
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

(1)

(2)

(3)

(4)

Can altmetrics reflect societal impact considerations?

Mesa 2. 2 × 2 subject-specific cross table

Group g

Mundo

Number of mentioned papers
af

Number of not mentioned papers
bf

The CI for the MHq can be constructed with

(cid:3)

MHqL ¼ exp ln MHq

Þ − 1:96

(cid:3)

MHqU ¼ exp ln MHq

Þ þ 1:96

(cid:4)
ﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ
C
(cid:2)
Þ
d
Var ln MHq

(6)

(7)

We used the data in Table 3 to produce a small world example for explaining the MHq: El
world consists of papers in four subject categories. The papers of two units (publication set A
y B) determine the world. For each unit, the numbers of mentioned and not mentioned pa-
pers as well as the corresponding proportion of mentioned papers are listed. Por ejemplo, el
unit named as publication set B has published 26 mentioned and seven not mentioned papers
in subject category 1. The proportion of the papers mentioned is 0.27. It is an advantage of the
MHq that the world average has a value of 1: This value indicates that there is no difference
between the chances of a focal publication set and the reference sets (es decir., el mundo) of being
mentioned (p.ej., on Wikipedia). An MHq value less than 1.0 indicates lower chances for the
publications in the set of being mentioned compared with the reference sets. The MHq values
en mesa 3 can be interpreted as follows: The chances of the papers in publication set A of
being mentioned are 0.81 times as large as the world’s papers’ chances. The chances of the
papers in set B of being mentioned are 1.3 times greater than the world’s papers’ chances. Es
an advantage of the MHq that the result can be expressed as a percentage, which is relative to
the world average. Expressed as percentages, por lo tanto, the difference between publication set
B and the world is

d
100 * 1:3−1:0

Þ ¼ 30%

(8)

De este modo, the publications in set B have a 30% higher chance of being mentioned than the
world’s publications. We also added CIs to the MHqs in Table 3. Since the CIs of both pub-
lication sets (A and B) overlap substantially among themselves and with 1.0 (the world MHq),
they do not differ statistically significantly from one another and the world average.

4. RESULTADOS

Cifra 2 displays the MHq values (based on six altmetrics sources) for all WoS papers in the
given years (unit 0: red points; neither accepted nor rejected); papers published by rejected
applicants (unit 1: green squares); and papers published by accepted applicants (unit 2: azul
diamonds). The papers from accepted applicants are further differentiated into papers written
in the context of projects funded by the research center (unit 4: orange diamonds) and papers
published in other contexts (unit 3: yellow diamonds). For all MHq values, CIs are indicated.
Since the paper numbers from funded projects for some publication years are too low, ellos
could not be presented in the figure.

Estudios de ciencias cuantitativas

801

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
a
_
0
0
0
3
2
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Can altmetrics reflect societal impact considerations?

Mesa 3. Small world example for the Mantel–Haenszel quotient (MHq)

Mundo (reference
conjuntos)
Subject category 1

Subject category 2

Subject category 3

Subject category 4

Total

Publication set A

Subject category 1

Subject category 2

Subject category 3

Subject category 4

Total

Publication set B

Subject category 1

Subject category 2

Subject category 3

Subject category 4

Total

Paper is
mentioned
44

Paper is not
mentioned
20

Número de
documentos
64

MHq

1.00 [0.61, 1.64]

0.81 [0.46, 1.44]

1.30 [0.66, 2.53]

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
a
_
0
0
0
3
2
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

The results as summarized in Figure 2 do not support the hypothesis that funded researchers
achieve higher altmetrics scores with their research than those who were not funded by the
research center. Por ejemplo, the MHq values based on Twitter data for the papers published
by rejected applicants (green squares) are consistently higher than the papers published by
accepted applicants (blue diamonds). The differences between both groups are statistically
significant in 2011 y 2012, but not in 2013 a 2015 (aquí, the CIs mostly overlap). Quite
strikingly, the figure also reveals that papers published by accepted applicants in the context of
the funded research center projects (orange diamonds) even receive lower Twitter scores than
the papers they published outside of the research center project (yellow diamonds). The results
for the other altmetric scores mainly concur with the Twitter results. Only the findings for the
policy-related documents show a different picture: Research-center-based papers published
entre 2012 y 2014 (orange diamonds) received higher altmetric scores than papers by
the same researchers that do not emanate from research center projects (yellow diamonds).
Sin embargo, the results are not statistically significant and are not confirmed by the results for
2011 (results for 2015 are not available).

We further analyzed whether the ex ante societal impact considerations are reflected in
citation scores. The results are shown in Figure 3. The figure reveals that the results are more

Estudios de ciencias cuantitativas

802

Can altmetrics reflect societal impact considerations?

or less in agreement with the altmetrics results (with papers published by rejected applicants
performing similarly to or better than those of funded applicants). If we inspect the aggregated
MHq results based on the papers from all years, papers published by accepted applicants
(MHq = 3.31) have a higher citation impact than papers published by rejected applicants
(MHq = 2.87). Since the CIs of both groups overlap, sin embargo, the results are not statistically
significativo. We obtained similar results (missing substantial differences between the groups),
when we compared median citations (accepted applicants = 9, rejected applicants = 9) y
percentile citation scores (accepted applicants = 73, rejected applicants = 70) of both groups.

5. DISCUSSION AND CONCLUSION

Universities and researchers are increasingly under pressure to disclose how their research
contributes to the welfare of society to garner political support and funding (Puschmann,
2014; Thune, Reymert, et al., 2016; Bornmann & Haunschild, 2017). In light of this develop-
mento, assessing the societal impact of research is a critically debated issue among evaluation
scholars and research policy experts. Because of their widespread use, social media have been
at the heart of methodological discussions over the past years, including both their potential
(p.ej., velocidad, broadness) and their shortcomings (p.ej., data quality, zero-inflated data).
Sin embargo, the critical question of whether social media (or altmetrics for that matter) are able
to reflect societal impact is so far not answered due to conflicting empirical evidence. Against
this background, the aim of this study was to contribute to solving this puzzle. For this purpose,
the present paper compared ex ante assessments on the societal relevance of research with
altmetrics scores that the respective research received in the years after. A research center from
the field of sustainability science (CCES) and the societal impact assessments made by the
members of an ad hoc RC served as the case study for this investigation.

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
a
_
0
0
0
3
2
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Cifra 2. MHq values based on six altmetrics sources for all WoS papers (unit 0: red points; neither accepted, nor rejected); papers published
by rejected applicants (unit 1: green squares); and papers published by accepted applicants (unit 2: blue diamonds). The papers from accepted
applicants are further divided into papers from funded projects (unit 4: orange diamonds) and papers published in other contexts (unit 3:
yellow diamonds). Durante algunos años, the values of unit 4 are missing because the numbers of mentioned papers are too low.

Estudios de ciencias cuantitativas

803

Can altmetrics reflect societal impact considerations?

Cifra 3. MHq values based on citation counts for all WoS papers (unit 0: red points; neither ac-
cepted nor rejected for funding), papers published by rejected applicants (unit 1: green squares),
papers published by accepted applicants (unit 2: blue diamonds), and papers published in other
contexts from accepted applicants (unit 3: yellow diamonds). Papers published from funded pro-
jects (unit 4) are not shown, because the numbers of uncited papers are too low.

En conclusión, the proposed hypothesis that researchers funded by CCES achieve higher
impact in terms of altmetrics scores with their research than those who were not funded could
not be confirmed based on the results. We found no correlation between the RC’s assessment
and the corresponding altmetric scores. With a few exceptions, this finding seems to be con-
firmed in the case of all six types of altmetrics. For comparison with altmetrics, we investigated
the relationship with citation scores as well. The results are similar to those based on alt-
métrica: The correlation is not in the expected direction.

Our results might be interpreted in such a way that altmetrics are not entirely suitable for
reflecting the societal impact of research. Sin embargo, since we investigated only one case-
specific evaluation procedure and the results are not homogeneous throughout the different
types of altmetric scores, this conclusion cannot be drawn with certainty. We conclude therefore
that more research is needed to better understand what altmetrics are reflecting, particularmente en
light of their heterogeneity. Further research should clarify whether altmetric scores rather cap-
ture “unknown attention or unstructured noise produced by published research” (Moed, 2017;
Bornmann et al., 2019), some sort of “public discussion” (Haunschild, Leydesdorff, et al.,
2019) or anything else altogether.

Nuestros resultados, at the same time, could be interpreted with a critical view of the RC’s assess-
mentos. Did the members of the RC select the “right” projects in the first place or how should
the missing correlation between the ex ante assessments and the received citations be
interpreted?

Another question is whether the members of the RC were qualified to judge the societal
impact of proposed research. In most cases, expert panels are composed only of researchers
rather than of representatives of other sectors of society, which was also true for in the case of
CCES. This circumstance may have led to the fact that the potential societal impact could not
be accurately judged or that the aspect of societal impact was not given enough importance in

Estudios de ciencias cuantitativas

804

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
a
_
0
0
0
3
2
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Can altmetrics reflect societal impact considerations?

the evaluation procedure. En general, we note that our findings can take the discussion forward,
but also that they should be interpreted with caution.

6. LIMITATIONS, FURTHER RESEARCH, AND RECOMMENDATIONS

The study revealed that the ex ante assessments considering societal impact of research and
the altmetric scores of the same research do not correlate. We could conclude the debate at
this point and throw altmetrics overboard as a potential measure of societal impact. Pero, de
curso, this study has several limitations that need to be discussed.

One key aspect is related to the fact that altmetrics are still in their infancy in many ways.
Por ejemplo, are altmetrics really a good proxy for societal impact? Are social media mentions
in themselves societal impact? Does a mention or interaction on social media automatically
imply that there has been a cognitive engagement with the content of the research, y eso
societal impact has occurred? Or is it perhaps a buzzword-laden title, zeitgeist, or fame-related
reason why some research output scores higher on altmetrics than others (Sala, 2014)? Y
what can we say about all the research that does not have any mentions on Twitter or
Wikipedia? It would be highly questionable to conclude that no societal impact has been
achieved in all these null observation cases. Además, our study does not differentiate be-
tween “self-mentions” or “in-house users” (the own department or the university’s communi-
cations team) and mentions and interactions by other (more independent) individuals and
entidades. Despite being somewhat complex, further research should account for these aspects,
as well as for the actual content of the tweets or Facebook posts. This latter strategy could
allow for a better understanding of the intentions and meanings of social-media-based inter-
actions with research. By looking at the content or the timing of the mentions in more detail,
we could possibly identify different strategies in using social media, which could help us for-
mulate new hypotheses.

The results of this work have again shown that the true value added of altmetrics is not yet
entirely clear, but rather ranges on a scale between societal impact and unstructured noise.
This fundamental problem concerns all six types of altmetrics that have been considered in this
estudiar (to a greater or lesser extent). With regard to the inability of tweets to measure the so-
cietal impact of research, the results of the present study are consistent with those of Haustein
et al. (2014) and Andersen and Haustein (2015). From our point of view, off-the-cuff retweets
in particular are simply too inflationary to imply a serious engagement with the content of the
trabajar. Mentions on Wikipedia also do not seem to reflect the societal relevance of research
(Kousha & Thelwall, 2017). Más, it does not yet seem to be common practice to incorporate
scientific research into policy and policy-related documents, either in the field of climate
cambiar, as Bornmann et al. (2016) found, or in the likewise societally relevant field of sustain-
ability science, as the present study showed. This finding also underlines that the dialog and
knowledge transfer between science and policy is far from established (Hessels, Van Lente, &
Smits, 2009). Al mismo tiempo, it must be given fair consideration that, as with classical cita-
ciones, it can take up to several years for the results of scientific studies to become relevant and
cited in policy documents. The time window of the present study was simply too small.
Finalmente, the valid assessment of societal impacts by means of blogging, Facebook, or in news
outlets largely suffers from the fact that there is a bias toward publications from renowned
journals (Shema et al., 2012a, 2012b) or specific fields of interest (Ringelhan et al., 2015).
Although our findings seem to lend additional support in favor of the argument that altmetrics
are not capable of reflecting the societal impact of research, much more research will need to
be done before we can actually have a clear picture of what altmetrics are capable of.

Estudios de ciencias cuantitativas

805

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
a
_
0
0
0
3
2
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Can altmetrics reflect societal impact considerations?

Another limitation of this study is related to the evaluation process itself, and less to the
shortcomings of altmetrics. Even though the prospect of societal impact was a key criterion
for the RC, the assessments were not based on standardized rating scales along individual cri-
teria, but rather on a holistic rating, we can only assume that societal impact considerations
played a role in the evaluation process rather than having clearly traceable evidence for the
specific weighting that this critical aspect ultimately had. One remedy for future evaluations
could therefore be to assess societal impact as a single dimension using a standardized scale.

A related limitation of the study is associated with the heterogeneity of the societal impact.
Societal impact can manifest itself in different ways, such as in the form of policy impact, en-
vironmental impact, health impact, or educational impact. Due to the holistic rating in the
assessment process, it is not clear what kind of societal impact was the focus of the experts’
atención. This heterogeneity is also an issue for altmetrics. A Twitter mention compared to a
mention in a policy document, Por ejemplo, has not only been made a different way but prob-
ably has a different kind of impact as well.

With regard to the societal impact of research, this study focused exclusively on the pub-
lished journal papers and the corresponding altmetrics scores they received. It could certainly
bring added value if other outputs were taken into account as well, such as outputs that re-
searchers produce within the framework of public outreach activities. Specifically designed to
catalyze the societal impact of research, Por ejemplo, stakeholder publications or teaching
material and their respective altmetrics scores could much more accurately reflect the societal
impact of research. Sin embargo, in order for these alternative types of outputs to receive an alt-
metrics score, they would have to be assigned a unique and persistent identifier, such as a DOI
in the future (see https://www.doi.org/).

EXPRESIONES DE GRATITUD

The bibliometric data used in this study are from two sources: (a) the online version of the Web
of Science database provided by Clarivate Analytics (Filadelfia, Pensilvania, EE.UU) y (b)
the bibliometric in-house databases of the Max Planck Society (MPG) and the Competence
Centre for Bibliometrics (CCB: http://www.bibliometrie.info/). The MPG’s database is devel-
oped and maintained in cooperation with the Max Planck Digital Library (MPDL, Munich),
and the CCB’s database is developed and maintained by the cooperation of various
German research organizations. Both databases are derived from the Science Citation Index
Expanded (SCI-E), Social Sciences Citation Index (SSCI), and the Arts and Humanities Citation
Index (AHCI) prepared by Clarivate Analytics, formerly the IP & Science business of Thomson
Reuters (Filadelfia, Pensilvania, EE.UU). The authors thank the management of the CCES for
their kind support in conducting this study. The altmetrics data were retrieved from our locally
maintained database with data shared with us and dumped by the company altmetric.com on
Octubre 8, 2019.

CONTRIBUCIONES DE AUTOR

Omar Kassab: Conceptualización, data curation, formal analysis, investigación, methodology,
project administration, resources, writing—original draft, writing—review & edición. Lutz
Bornmann: Conceptualización, data curation, formal analysis, funding acquisition, investi-
gation, methodology, project administration, resources, software, validation, visualization,
writing—original draft, writing—review & edición. Robin Haunschild: Conceptualización,
data curation, formal analysis, investigación, methodology, project administration, re-
sources, software, validation, writing—original draft, writing—review & edición.

Estudios de ciencias cuantitativas

806

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
a
_
0
0
0
3
2
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Can altmetrics reflect societal impact considerations?

CONFLICTO DE INTERESES

Los autores no tienen intereses en competencia. Between 2013 y 2015, OK worked as an execu-
tive assistant to the CCES management. Afterward he joined the Professorship for Social
Psychology and Research on Higher Education at ETH Zurich, where he conducted this study
in collaboration with coauthors LB and RH.

INFORMACIÓN DE FINANCIACIÓN

No funding has been received for this research.

DISPONIBILIDAD DE DATOS

For the data sources, see Acknowledgments section. The authors do not have permission to
share the data. Information specific to the research center can be drawn from the performance
reporting of CCES, disponible en https://cces.ethz.ch/about/downloads.html.

REFERENCIAS

andersen, j. PAG., & Haustein, S. (2015). Influence of study type on twit-
ter activity for medical research papers. In A. A. Salah, Y. Tonta, A.
A. A. Salah, C. Sugimoto, & Ud.. Alabama (Editores.), The 15th Conference of
the International Society for Scientometrics and Informetrics,
páginas. 26–36. Istanbul, Pavo: ISSI, Bogazici University Printhouse.
Bar-Ilan, J., Shema, h., & Thelwall, METRO. (2014). Bibliographic refer-
ences in Web 2.0. In B. Cronin & C. R. Sugimoto (Editores.), Beyond
Bibliometrics: Harnessing Multi-Dimensional Indicators of
Actuación, páginas. 307–325. Cambridge, MAMÁ: CON prensa.

Bik, h. METRO., & Goldstein, METRO. C. (2013). An introduction to social
media for scientists. PLOS Biology, 11(4), e1001535. https://
doi.org/10.1371/journal.pbio.1001535

Bonetta, l. (2007). Scientists enter the blogosphere. Cell, 129(3),

443–445.

Bornmann, l. (2014). Do altmetrics point to the broader impact of
investigación? An overview of benefits and disadvantages of alt-
métrica. Journal of Informetrics, 8(4), 895–903. https://doi.org/
10.1016/j.joi.2014.09.005

Bornmann, l. (2015). Alternative metrics in scientometrics: un meta-
analysis of research into three altmetrics. cienciometria, 103(3),
1123–1144.

Bornmann, l., & Haunschild, R. (2017). Does evaluative sciento-
metrics lose its main focus on scientific quality by the new
orientation towards societal impact? cienciometria, 110(2),
937–943.

Bornmann, l., & Haunschild, R. (2018). Normalization of zero-
inflated data: An empirical analysis of a new indicator family
and its use with altmetrics data. Journal of Informetrics, 12(3),
998–1011. https://doi.org/10.1016/j.joi.2018.01.010

Bornmann, l., Haunschild, r., & Adams, j. (2019). Do altmetrics
assess societal impact in a comparable way to case studies? Un
empirical test of the convergent validity of altmetrics based on
data from the UK research excellence framework. Diario de
Informetrics, 13(1), 325–340. https://doi.org/10.1016/j.
joi.2019.01.008

Bornmann, l., Haunschild, r., & Marx, W.. (2016). Policy docu-
ments as sources for measuring societal impact: How often is cli-
mate change research mentioned in policy-related documents?
cienciometria, 109(3), 1477–1495. https://doi.org/10.1007/
s11192-016-2115-y

Bornmann, l., Leydesdorff, l., & Mutz, R. (2013). The use of per-
centiles and percentile rank classes in the analysis of biblio-
metric data: Opportunities and limits. Journal of Informetrics,
7(1), 158–165. https://doi.org/10.1016/j.joi.2012.10.001

Brandt, PAG., Ernst, A., Gralla, F., Luederitz, C., Lang, D. J., … von
Hehrden, h. (2013). A review of transdisciplinary research in
sustainability science. Ecological Economics, 92, 1-15.

CCES. (2005). Competence Center Environmental and Sustainability
(CCES) of the ETH Domain: Business Plan. Retrieved from: https://
ethz.ch/content/dam/ethz/special-interest/dual/cces-dam/docu-
ments/CCES_BP_2005_11_22_final.pdf

clark, W.. C. (2007). Sustainability science: A room of its own.
procedimientos de la Academia Nacional de Ciencias, 104(6),
1737–1738.

Dehdarirad, T., Didegah, F., & Sotudeh, h. (2018). Which type of
research is cited more often in Wikipedia? Un estudio de caso de
PubMed research. In P. Wouters (Ed.), Actas de la
Science and Technology Indicators Conference 2018 Leiden
“Science, Technology and Innovation Indicators in Transition.”
Leiden, Países Bajos: University of Leiden.

Fausto, S., Machado, F. A., Bento, l. F. J., Iamarino, A., Nahas, t. r.,
& Munger, D. S. (2012). Research blogging: Indexing and register-
ing the change in science 2.0. MÁS UNO, 7(12), e50109. https://
doi.org/10.1371/journal.pone.0050109

Fleiss, J., Levin, B., & Paik, METRO. C. (2003). Statistical methods for

rates and proportions, 3rd Ed. Hoboken, Nueva Jersey: wiley.

Guglielmi, GRAMO. (2018). Wikipedia’s top-cited scholarly articles –

revealed. Naturaleza, 557(7705), 291–292.

Sala, norte. (2014). The Kardashian index: A measure of discrepant so-
cial media profile for scientists. Genome Biology, 15(7), 424.
https://doi.org/10.1186/s13059-014-0424-0

Haunschild, r., & Bornmann, l. (2017). How many scientific pa-
pers are mentioned in policy-related documents? An empirical
investigation using Web of Science and Altmetric data.
cienciometria, 110(3), 1209–1216. https://doi.org/10.1007/
s11192-016-2237-2

Haunschild, r., & Bornmann, l. (2018). Field- and time-normalization
of data with many zeros: An empirical analysis using citation and
Twitter data. cienciometria, 116(2), 997–1012. https://doi.org/
10.1007/s11192-018-2771-1

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
a
_
0
0
0
3
2
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Estudios de ciencias cuantitativas

807

Can altmetrics reflect societal impact considerations?

Haunschild, r., Leydesdorff, l., Bornmann, l., Hellsten, I., & Marx,
W.. (2019). Does the public discuss other topics on climate
change than researchers? A comparison of explorative networks
based on author keywords and hashtags. Journal of Informetrics,
13(2), 695–707.

Haustein, S., Larivière, v., Thelwall, METRO., Amyot, D., & Peters, I. (2014).
Tweets vs. Mendeley readers: How do these two social media met-
rics differ? IT – Information Technology 2014, 56(5), 207–215.
Haustein, S., Peters, I., Bar-Ilan, J., Principal, J., Shema, h., &
Terliesner, j. (2014). Coverage and adoption of altmetrics
sources in the bibliometric community. cienciometria, 101(2),
1145–1163. https://doi.org/10.1007/s11192-013-1221-3

Hessels, l. K., Van Lente, h., & Smits, R. (2009). In search of rele-
vance: The changing contract between science and society.
Science and Public Policy, 36(5), 387–401.

Hollander, METRO., & lobo, D. A. (1999). Nonparametric statistical

methods. Nueva York, Nueva York: wiley.

Kajikawa, y., Tacoa, F., & Yamaguchi, k. (2014). Sustainability sci-
ence: The changing landscape of sustainability research.
Sustainability Science, 9(4), 431–438.

Kassab, oh. (2019). Does public outreach impede research perfor-
mance? Exploring the “researcher’s dilemma” in a sustainability
research center. Science and Public Policy, 46(5), 710–720.
https://doi.org/10.1093/scipol/scz024

Kassab, o., Schwarzenbach, R. PAG., & Gotsch, norte. (2018). Assessing ten
years of inter- and transdisciplinary research, education, and out-
reach: The Competence Center Environment and Sustainability
(CCES) of the ETH Domain. GAIA – Ecological Perspectives for
Science and Society, 27(2), 226–234.

Kates, R. W.. (2011). What kind of a science is sustainability science?
procedimientos de la Academia Nacional de Ciencias, 108(49),
19449–19450.

Kousha, K., & Thelwall, METRO. (2017). Are Wikipedia citations impor-
tant evidence of the impact of scholarly articles and books?
Journal of the Association for Information Science and
Tecnología, 68(3), 762–779. https://doi.org/10.1002/asi.23694
Lang, D. J., Wiek, A., Bergmann, METRO., Stauffacher, METRO., Martens, PAG.,
… Thomas, C. j. (2012). Transdisciplinary research in sustainabil-
ity science: Practice, principios, and challenges. Sustainability
Ciencia, 7(1), 25–43.

Luzón, METRO. j. (2013). Public communication of science in blogs:
Recontextualizing scientific discourse for a diversified audi-
ence. Written Communication. https://doi.org/10.1177/
0741088313493610

Mantel, NORTE., & Haenszel, W.. (1959). Statistical aspects of the anal-
ysis of data from retrospective studies of disease. Journal of the
National Cancer Institute, 22(4), 719–748.

Mas-Bleda, A., & Thelwall, METRO. (2016). Can alternative indicators
overcome language biases in citation counts? A comparison of
Spanish and UK research. cienciometria, 109(3), 2007–2030.
https://doi.org/10.1007/s11192-016-2118-8

Mewburn, I., & Thomson, PAG. (2013). Why do academics blog? Un
analysis of audiences, purposes and challenges. Studies in
Higher Education, 38(8), 1105–1119. https://doi.org/10.1080/
03075079.2013.835624

Molinero, t. R. (2013). Constructing sustainability science: Emerging
perspectives and research trajectories. Sustainability Science, 8(2),
279–293.

Mobjörk, METRO. (2010). Consulting versus participatory transdiscipli-
narity: A refined classification of transdisciplinary research.
Futures, 42(8), 866–873.

Moed, h. F. (2017). Applied evaluative informetrics. cham:

Saltador.

OPENing UP. (2016). Deliverable D5.1—Altmetrics Status Quo.
OPENing UP new methods, indicators and tools for peer review,
impact measurement and dissemination of research. https://ec.
europa.eu/research/participants/documents/downloadPublic?
documentIds=080166e5ae8a151c&appId=PPGMS Project
acronym: OpenUP. Retrieved from Brussels, Bélgica.
Pohl, C. (2010). From transdisciplinarity to transdisciplinary research.

Transdisciplinary Journal of Engineering Science, 1, 65–73.

Principal, j. (2014). Altmetrics. In B. Cronin & C. R. Sugimoto (Editores.),
Beyond Bibliometrics: Harnessing Multi-Dimensional Indicators
of Performance, páginas. 263–288. Cambridge, MAMÁ: CON prensa.

Principal, J., & Hemminger, B. METRO. (2010). cienciometria 2.0: Toward
new metrics of scholarly impact on the social Web. Primero
Monday, 15(7).

Puschmann, C. (2014). (Micro) blogging science? Notes on poten-
tials and constraints of new forms of scholarly communication.
In S. Bartling & S. Friesike (Editores.), Opening Science: The Evolving
Guide on How the Internet is Changing Research, Collaboration
and Scholarly Publishing, páginas. 89–106. cham: Saltador.

Radhakrishna, S. (1965). Combination of results from several 2 × 2

contingency tables. Biometrics, 21, 86–98.

Ringelhan, S., Wollersheim, J., & Welpe, I. METRO. (2015). I like, I cite?
Do Facebook likes predict the impact of scientific work? PLOS
ONE, 10(8), e0134389. https://doi.org/10.1371/journal.
pone.0134389

Robinson-Garcia, NORTE., costas, r., Isett, K., Melkers, J., & Hicks, D.
(2017). The unbearable emptiness of tweeting—About journal
artículos. MÁS UNO, 12(8), e0183551. https://doi.org/10.1371/
diario.pone.0183551

Rons, norte. (2012). Partition-based field normalization: An approach
to highly specialized publication records. Journal of Informetrics,
6(1), 1–10. https://doi.org/10.1016/j.joi.2011.09.008

Rons, norte. (2014). Investigation of Partition Cells as a Structural Basis
Suitable for Assessments of Individual Scientists. In P. Wouters
(Ed.), Proceedings of the Science and Technology Indicators
Conferencia 2014 Leiden “Context Counts: Pathways to Master
Big and Little Data”, páginas. 463–472. Leiden, Países Bajos:
University of Leiden.

Samoilenko, A., & Yasseri, t. (2014). The distorted mirror of
Wikipedia: A quantitative analysis of Wikipedia coverage of ac-
ademics. EPJ Data Science, 3(1), 1.

SDSN Australia/Paciﬁc. (2017). Getting started with the SDGs in
universidades: A guide for universities, higher education institu-
ciones, and the academic sector. Australia, New Zealand and
Paciﬁc Edition. Sustainable Development Solutions Network—
Australia/Paciﬁc, Melbourne.

Serrano-López, A. MI., Ingwersen, PAG., & Sanz-Casado, mi. (2017).
Wind power research in Wikipedia: Does Wikipedia demonstrate
direct influence of research publications and can it be used as ad-
equate source in research evaluation? cienciometria, 112(3),
1471–1488. https://doi.org/10.1007/s11192-017-2447-2

Shema, h., Bar-Ilan, J., & Thelwall, METRO. (2012a). Research blogs and
the discussion of scholarly information. MÁS UNO, 7(5),
e35869. https://doi.org/10.1371/journal.pone.0035869

Shema, h., Bar-Ilan, J., & Thelwall, METRO. (2012b). Self-citation of blog-
gers in the science blogosphere. In A. Tokar, METRO. Beurskens,
S. Keuneke, METRO. Mahrt, I. Peters, C. Puschmann, k. Weller, &
t. van Treeck (Editores.), Science and the Internet, páginas. 183–192.
Düsseldorf, Alemania: Düsseldorf University Press.

Shema, h., Bar-Ilan, J., & Thelwall, METRO. (2014a). Scholarly blogs are

a promising altmetric source. Research Trends, 37, 11–13.

Shema, h., Bar-Ilan, J., & Thelwall, METRO. (2014b). Do blog citations
correlate with a higher number of future citations? Investigación

Estudios de ciencias cuantitativas

808

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
a
_
0
0
0
3
2
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Can altmetrics reflect societal impact considerations?

blogs as a potential source for alternative metrics. Journal of the
Association for Information Science and Technology, 65(5),
1018–1027. https://doi.org/10.1002/asi.23037

Sheskin, D. (2007). Handbook of parametric and nonparametric
statistical procedures (4th Ed.). Boca Raton, Florida: Chapman &
Hall/CRC.

Sugimoto, C. r., Work, S., Larivière, v., & Haustein, S. (2016).
Scholarly use of social media and altmetrics: A review of the
literature. Retrieved from https://arxiv.org/abs/1608.08112

Tattersall, A., & Carroll, C. (2018). What can Altmetric.com tell us
about policy citations of research? An analysis of Altmetric.com
data for research articles from the University of Sheffield.
Frontiers in Research Metrics and Analytics, 2(9). https://doi.
org/10.3389/frma.2017.00009

Teplitskiy, METRO., Lu, GRAMO., & Duede, mi. (2017). Amplifying the impact of open
access: Wikipedia and the diffusion of science. Journal of the
Association for Information Science and Technology, 68, 2116–2127.
https://doi.org/10.1002/asi.23687

Thune, T., Reymert, I., Gulbrandsen, METRO., & Olaf Aamodt, PAG. (2016).
Universities and external engagement activities: Particular pro-

files for particular universities? Science and Public Policy, 43(6),
774–786. https://doi:10.1002/asi.23687

Vilkins, S., & Grant, W.. j. (2017). Types of evidence cited in
Australian Government publications. cienciometria, 113(3),
1681–1695. https://doi.org/10.1007/s11192-017-2544-2

Wiek, A., Talwar, S., O’Shea, METRO., & robinson, j. (2014). Toward a
methodological scheme for capturing societal effects of participa-
tory sustainability research. Research Evaluation, 23(2), 117–132.
lanaridge, J., & Rey, METRO. B. (2019). Altmetric scores: An early in-
dicator of research impact. Journal of the Association for
Information Science and Technology, 70(3), 271–282. https://
doi.org/10.1002/asi.24122

Yarime, METRO., Trencher, GRAMO., Mino, T., Scholz, R. w., Olsson, l., Ness,
B., … Rotmans, j. (2012). Establishing sustainability science in
higher education institutions: Towards an integration of academ-
ic development, institutionalization, and stakeholder collabora-
ciones. Sustainability Science, 7(1), 101–113.

Yu, h. (2017). Context of altmetrics data matters: An investigation of
count type and user category. cienciometria, 111(1), 267–283.
https://doi.org/10.1007/s11192-017-2251-z

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d