RESEARCH ARTICLE

Can altmetrics reflect societal impact
considerations?: Exploring the potential of
altmetrics in the context of a sustainability science
research center

开放访问

杂志

Omar Kassab1, Lutz Bornmann2

, and Robin Haunschild3

1Swiss Federal Institute of Technology (ETH Zurich), Professorship for Social Psychology and Research on Higher Education,
Andreasstrasse 15, 8050 Zurich, 瑞士
2Administrative Headquarters of the Max Planck Society, Division for Science and Innovation Studies, Hofgartenstrasse 8,
80539 慕尼黑, 德国
3Max Planck Institute for Solid State Research, Heisenbergstrasse 1, 70569 Stuttgart, 德国

关键词: altmetrics, bibliometrics, MHq indicator, research center, societal impact, sustainability
科学

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你
q
s
s
/
A
r
t
我
C
e
–
p
d

我

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
A
_
0
0
0
3
2
p
d

乙
y
G
你
e
s
t

哦
n
0
7
S
e
p
e
米
乙
e
r
2
0
2
3

抽象的

Societal impact considerations play an increasingly important role in research evaluation. 在
特别的, in the context of publicly funded research, proposal templates commonly include
sections to outline strategies for achieving broader impact. Both the assessment of the
strategies and the later evaluation of their success are associated with challenges in their own
正确的. Ever since their introduction, altmetrics have been discussed as a remedy for
assessing the societal impact of research output. On the basis of data from a research center in
瑞士, this study explores their potential for this purpose. The study is based on the
文件 (and the corresponding metrics) published by about 200 either accepted or rejected
applicants for funding by the Competence Center Environment and Sustainability (CCES). 这
results of the study seem to indicate that altmetrics are not suitable for reflecting the societal
impact of research that was considered: The metrics do not correlate with the ex ante
considerations of an expert panel.

介绍

Many studies dealing with the societal impact of research begin by describing a paradigmatic
transformation in research policy that has presumably led to an increased accountability of
publicly funded research. Researchers and universities, according to this narrative, would in-
creasingly have to justify their work to the tax-paying public. This almost confrontational por-
trayal of the relationship could make the reader believe that the public is concerned with a
petty cost–benefit calculation to tease out their return on investment. 然而, this simplified
view undermines the potentially genuine interest of societal actors to inform and educate
themselves on the basis of scientific facts. Especially in times of rapid technological develop-
评论, the interaction between science and society is easier than ever.

The very emergence of social media, 例如, has heralded a new age for the public
dissemination of scientific knowledge. It therefore comes as no surprise that “altmetrics,” an
endeavor to quantitatively represent mentions and interactions on social media platforms such

引文: Kassab, 奥。, Bornmann, L。, &
Haunschild, 右. (2020). Can altmetrics
reflect societal impact considerations?:
Exploring the potential of altmetrics in
the context of a sustainability science
research center. Quantitative Science
学习, 1(2), 792–809. https://doi.org/
10.1162/qss_a_00032

DOI:
https://doi.org/10.1162/qss_a_00032

已收到: 29 八月 2019
公认: 08 二月 2020

通讯作者:
Omar Kassab
omar.kassab@gess.ethz.ch

处理编辑器:
Ludo Waltman

版权: © 2020 Omar Kassab, Lutz
Bornmann, and Robin Haunschild.
在知识共享下发布
归因 4.0 国际的 (抄送 4.0)
执照.

麻省理工学院出版社

Can altmetrics reflect societal impact considerations?

as Twitter or Facebook, have been proposed as a means to evaluate the societal impact of
research ex post (see the literature overview by Bornmann, 2014). Yet despite the consensus
over their potential for impact assessment, the jury is still out as to what kind of impact alt-
metrics scores actually reflect. Addressing this puzzle, Bornmann, Haunschild, and Adams
(2019) compared peer assessments of societal impact of research with altmetrics scores for
the corresponding publications. Their results reveal that altmetrics seem to measure public
“discussions” around research rather than societal impact, further qualifying that the latter
may more likely be assessed by experts in a specific field. 然而, there are also other
empirical findings suggesting a contrary conclusion. Wooldridge and King (2019), 为了考试-
普莱, used the same data set as Bornmann et al. (2019) but other methods, and concluded that
“the work presented in this study provides direct evidence, 首次, of a correlation
between expert peer review of the societal impact of research and altmetric data from the
publications defining the underpinning research” (p. 281). Against the backdrop of these
contradicting results, it is necessary to advance further empirical investigations about the cor-
relation between assessments of societal impact of research and altmetrics scores.

Taking up the question in the context of a research center, the CCES in Switzerland, 这
study examines the altmetrics scores of journal articles published by researchers either ac-
cepted or rejected for funding by CCES. As a research field “defined by the problems it
addresses rather than by the disciplines it employs” (克拉克, 2007), sustainability science rep-
resents a prime case for solution-oriented research of high societal relevance (Yarime,
Trencher, 等人。, 2012; Brandt, Ernst, 等人。, 2013; Wiek, Talwar, 等人。, 2014; Kassab,
2019). 因此, whether the research was funded or rejected depended not solely on the as-
sessment of the scientific quality, but initially on whether the prospect of societal impact
was explicitly outlined in the proposal or not (CCES, personal communication). We explore
in this study whether this latter criterion is reflected in later altmetrics scores: Do papers of
researchers funded by CCES receive higher altmetrics scores than papers from rejected re-
searchers? Or in other words, using another data set than Bornmann et al. (2019) 和
Wooldridge and King (2019), this study targets the question of whether or not altmetrics
scores are consistent with ex ante assessments of societal impact considerations.

The remainder of the article is structured as follows: 部分 2 introduces the case and de-
scribes the hypothesized relationship between societal impact assessments and altmetrics
scores. 部分 3 then gives an overview of the data and the methods used for the investiga-
的. 部分 4 presents the results of the study, and section 5 discusses them to draw conclu-
西翁. 最后, 部分 6 outlines the limitations of the study while giving indications for further
research and recommendations.

2. CAN ALTMETRICS REFLECT SOCIETAL IMPACT CONSIDERATIONS?

2.1. Case Description: A Sustainability Science Research Center in Switzerland

The CCES was founded in 2006 for a period of 10 年 (直到 2016) to foster inter- and trans-
disciplinarity within and between the six institutions that constitute the ETH Domain, a union
of Swiss Federal universities and research institutes. Strategically managed by the ETH Board,
the ETH Domain comprises the two Federal Institutes of Technology in Zurich (ETH Zurich)
and Lausanne (EPFL), as well as four research institutes: the Paul Scherrer Institute (PSI), 这
Swiss Federal Institute for Forest, Snow and Landscape Research ( WSL), the Swiss Federal
Laboratories for Materials Science and Technology (Empa), and the Swiss Federal Institute
of Aquatic Science and Technology (Eawag).

Quantitative Science Studies

793

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你
q
s
s
/
A
r
t
我
C
e
–
p
d

我

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
A
_
0
0
0
3
2
p
d

乙
y
G
你
e
s
t

哦
n
0
7
S
e
p
e
米
乙
e
r
2
0
2
3

Can altmetrics reflect societal impact considerations?

CCES was established with the mission to “identify the relevant questions and the appro-
priate answers to foster the sustainable development of a future society while minimizing the
impact on the environment” (CCES, 2005). To comprehensively achieve this mission, CCES
operated in three areas of activity: 研究, capacity-building, and public outreach. Goals
have been set for each of the three areas, with a total of five goals. In the area of “research,”
three goals were defined: (A) foster major inter- and transdisciplinary research advancements in
the areas of environment and sustainability, (乙) establish the CCES partner institutions as na-
tional and international focal points for the areas of environment and sustainability, 和 (C)
achieve a long-term structuring effect and a coherent strategy for the areas of environment
和可持续性. In the area of “capacity-building,” the goal was (d) to establish a strong
and wide-ranging education program for the areas of environment and sustainability. And lastly,
the goal set in the area of “public outreach” was (e) to achieve a visible societal impact with a
focus on socioeconomic implementation.

Activities at CCES were clustered in 26 projects along five thematic areas of environment
and sustainability science: (A) Climate and Environmental Change, (乙) Sustainable Land Use,
(C) Food, 环境, 和健康, (d) Natural Resources, 和 (e) Natural Hazards and Risks.
Some exemplary projects included OPTIWARES, in which researchers worked on optimizing
the use of wood as a renewable energy source, TRAMM, which aimed at developing early
warning systems for rapid mass movements in steep terrain, and the ADAPT project, 哪个
studied social and environmental constraints for large-scale dams and water resource manage-
蒙特 (Kassab, Schwarzenbach, & Gotsch, 2018).

2.2. Societal Impact Considerations in the Evaluation Procedure

The few aforementioned synopses demonstrate exemplarily that projects funded by the re-
search center were characterized by a strong practice orientation. This property is based on
the notion that sustainability science concentrates on the most pressing challenges facing
human society and the development of concrete solutions (Yarime et al., 2012; Kajikawa,
Tacoa, & Yamaguchi, 2014; SDSN, 2017). In order to find these solutions, 然而, 这是
necessary not only to overcome disciplinary boundaries, through interdisciplinarity, 但是也
transcend the university ecosystem and engage other stakeholders from society, 商业, 和
政治, through transdisciplinarity approaches (Pohl, 2010; Lang, Wiek, 等人。, 2012). 就条款而言
of the underlying research mode, sustainability science thus differs considerably from basic
研究 (克拉克, 2007; Mobjörk, 2010; Kates, 2011; 磨坊主, 2013).

The special attention given to inter- and transdisciplinarity as well as the objective to devel-
op applied solutions was explicitly reflected in the CCES evaluation procedure. For the purpose
of assessing the project proposals, an ad hoc Research Council (RC) was established. Consisting
的 17 researchers from the ETH Domain institutions, the RC was responsible for reviewing pro-
posals with respect to their overall suitability for CCES (see goals above). 尤其, it was the
task of the RC to evaluate the added value of the project for CCES, stressing (A) societal rele-
vance, either as a goal to be achieved during the project duration or with an identified follow-
up implementation phase, (乙) the importance of the project for long-term sustainability and for a
durable structuring effect, 和 (C) the relevance in the international context, and in particular,
the potential for applications in developing countries (CCES, 2006). As this focus suggests, 这
assessments of the RC were primarily based on the prospect of societal impact, reflecting the
three aforementioned dimensions, and did not include an evaluation of the scientific quality. 实际上,
only if the projects passed the initial assessment were they forwarded to the next stage, which con-
sisted of a classical peer review procedure coordinated by the ETH Zurich Research Commission.

Quantitative Science Studies

794

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你
q
s
s
/
A
r
t
我
C
e
–
p
d

我

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
A
_
0
0
0
3
2
p
d

乙
y
G
你
e
s
t

哦
n
0
7
S
e
p
e
米
乙
e
r
2
0
2
3

Can altmetrics reflect societal impact considerations?

Given the still inconclusive debate about the validity of altmetrics for reflecting the soci-
etal impact of research, the question that lies at the heart of this study is whether or not there
is a relationship between ex ante assessments of societal impact and altmetrics scores. 我们
approach the answer to this question indirectly: According to the CCES evaluation proce-
dure outlined above, special emphasis was attributed to the prospect of societal impact.
Under the premise that research funded through CCES would yield more societal impact
than the research of rejected applicants, and assuming that altmetrics scores are capable
of reflecting this impact, the hypothesis arises that the researchers funded by CCES achieve
higher impact in terms of altmetrics scores with their research than those who were not
funded. Should the findings of this study corroborate the hypothesis, this would lead to
the conclusion that altmetrics are indeed capable of reflecting the ex ante societal impact
considerations of the RC. 然而, should the results not confirm the hypothesis, 这确实
not automatically imply the opposite. 相当, this would raise the question of what else alt-
metric scores are indicative of. 实际上, a refutation of the hypothesis could also be inter-
impact
preted in a way that
considerations in the assessments (even though this was explicitly demanded), 反而
focused on other aspects. 下文中, we describe the data and the methods we use
to test the hypothesis empirically.

take sufficient account of societal

the RC did not

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你
q
s
s
/
A
r
t
我
C
e
–
p
d

我

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
A
_
0
0
0
3
2
p
d

乙
y
G
你
e
s
t

哦
n
0
7
S
e
p
e
米
乙
e
r
2
0
2
3

3. DATA AND METHODS

3.1. Description of Altmetrics

We acknowledge that altmetrics are heterogeneous in many ways and specifically with regard
to which aspect of societal impact they actually reflect (如果有的话). We considered six different
altmetric sources in this study, including Twitter, 维基百科, policy-related documents, 博客,
Facebook, and news. They differ strongly with regard to the effort and the process preceding
the actual mention, 内容, and substance of the information that is communicated, and also
the readership. While a tweet or a Facebook post is shared at the touch of a button, the thresh-
old for Wikipedia entries, blog posts, or mentions in news outlets is much higher. 还, 这
demographic background of the readership of policy-related documents as opposed to
Facebook posts is much more specific. 尽管如此, we chose those types since they have
been frequently used and investigated in previous altmetrics studies (see Bornmann et al.,
2019), qualifying them as “standard” sources.

推特 (https://www.twitter.com) is a popular microblogging platform. Tweets may refer to
the content of scientific publications, but it seems that they do not correlate with traditional
citations (Bornmann, 2015). 反而, they may reflect discussion around these publications
(Haustein, Peters, 等人。, 2014), possibly by public users (Haustein, Larivière, 等人。, 2014;
于, 2017), but this is not entirely clear, as outlined by Sugimoto, 工作, 等人. (2016). 那里-
sults by Andersen and Haustein (2015) suggest that tweets reflect the attractiveness of papers
for a broader audience. 然而, contradictory results are also available: “A multi-year cam-
paign has sought to convince us that counting the number of tweets about papers has value.
然而, reading tweets about dental journal articles suggested the opposite. This analysis found:
obsessive single issue tweeting, duplicate tweeting from many accounts presumably under
centralized professional management, bots, and much presumably human tweeting duplica-
主动的, almost entirely mechanical and devoid of original thought” (Robinson-Garcia, Costas,
等人。, 2017). In the study at hand, the number of tweets (and retweets) including references
to scientific papers in our data set is counted.

Quantitative Science Studies

795

Can altmetrics reflect societal impact considerations?

维基百科 (https://www.wikipedia.org) is a free encyclopedia platform which includes ed-
itable content (Mas-Bleda & Thelwall, 2016). Although contributors to this platform include
scholarly references, most of them do not refer to research papers (Priem, 2014). If scientific
papers are cited, open access (OA) papers seem to be preferred (Teplitskiy, 鲁, & Duede,
2017; Dehdarirad, Didegah, & Sotudeh, 2018). Guglielmi (2018) reports on Wikipedia’s most
frequently mentioned papers. 然而, this list does not correspond with lists based on tra-
ditional citations: Study results suggest that Wikipedia mentions do not correlate with citations
(Samoilenko & Yasseri, 2014). A Wikipedia case study with papers on wind power showed
that less than 1% of relevant papers have been cited on Wikipedia, “implying that the direct
societal impact through the Wikipedia is extremely small for Wind Power research” (Serrano-
López, Ingwersen, & Sanz-Casado, 2017, p. 1471). Kousha and Thelwall (2017) found that
仅有的 5% of papers had any citation from Wikipedia—based on a significantly larger sample
of papers than considered by Serrano-López et al. (2017). In this study, 的数量
Wikipedia articles with reference to papers in our data set is counted.

Policy-related documents are an important source of altmetrics, since one is interested in
the impact of science on the policy realm (OPENing UP, 2016; Vilkins & 授予, 2017).
Mentions in these documents are searched using text mining databases of, 例如, 这
World Health Organization or European Food Safety Authority (Bornmann, Haunschild, &
马克思, 2016; Haunschild & Bornmann, 2017). Haunschild and Bornmann (2017) reported that
the company Altmetric tracks more than 100 policy sources (在 2015). Tattersall and Carroll
(2018) analyzed nearly 100 papers published by authors from the University of Sheffield: 这
“research topics with the greatest policy impact are medicine, dentistry, 和健康, followed
by social science and pure science.” Papers published OA seem to have an advantage to be
cited in policy-related documents (Vilkins & 授予, 2017). 然而, the impact of papers (OA
或不) on these documents is usually very low, as the results of Haunschild and Bornmann
(2017) reveal: “Less than 0.5% of the papers published in different subject categories are men-
tioned at least once in policy-related documents” (p. 1209). The study of Bornmann et al.
(2016) shows that “only 1.2 % (n = 2,341) have at least one policy mention” (p. 1477). 这
authors analyzed a large set of 191,276 publications from the field of climate change, 这是
policy relevant. In this study, the number of policy-related documents with references to pa-
pers in our data set is counted.

Blogs are written about scientific papers, including formal or informal citations of papers
(Shema, Bar-Ilan, & Thelwall, 2014A). These citations can be counted—with the limitation
that informal citations lead to uncertainty (Priem & Hemminger, 2010; Luzón, 2013; Shema,
Bar-Ilan, & Thelwall, 2014乙). Since blogs allow extended informal discussions about research,
they are an interesting altmetrics source (Fausto, Machado, 等人。, 2012; Shema, Bar-Ilan, &
Thelwall, 2012A). Blogging may be a bridge between the general public and the research area
(Bonetta, 2007; Bar-Ilan, Shema, & Thelwall, 2014), whereby bloggers seem to have prefer-
ences for papers from high-impact journals and research in the life and behavioral sciences
(Shema, Bar-Ilan, & Thelwall, 2012乙). 然而, a study revealed that bridging public and re-
search “was one of the less popular motivations for academics to blog” (Mewburn & Thomson,
2013, p. 1113). The literature overview published by Sugimoto et al. (2016) shows that the cov-
erage of papers in blog mentions is low, as is the correlation between blog mentions and tra-
ditional citations. In this study, the number of blog posts with references to the papers in our
data set is counted.

Facebook is a popular social networking and social media platform (Bik & 戈德斯坦,
2013). Since users share papers among themselves, mentions of papers in posts or
Facebook likes can be counted. Ringelhan, Wollersheim, and Welpe (2015) investigated

Quantitative Science Studies

796

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你
q
s
s
/
A
r
t
我
C
e
–
p
d

我

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
A
_
0
0
0
3
2
p
d

乙
y
G
你
e
s
t

哦
n
0
7
S
e
p
e
米
乙
e
r
2
0
2
3

Can altmetrics reflect societal impact considerations?

whether Facebook “likes” are an indicator of scientific impact. Their results show “an inter-
disciplinary difference in the predictive value of Facebook likes, according to which
Facebook likes only predict citations in the psychological area but not in the nonpsycholog-
ical area of business or in the field of life sciences.” In this study, the number of Facebook
posts with references to scientific papers in our data set are counted (note that we did not
include likes).

News attention relates to scientific papers mentioned in news reports (via direct links or
unique identifiers in, 例如, the New York Times). On the basis of these paper mentions,
public attention can be counted. The overview of altmetrics studies published by Sugimoto
等人. (2016) reveals that the correlation between mentions of papers in news reports and tra-
ditional citations is between low and medium. According to Altmetric.com1, 多于 2,000
different news sources are analyzed for news mentions. In this study, the number of news ar-
ticles with mentions of scientific papers in our data set is counted.

3.2. Data Set Used

We used the Web of Science ( WoS, Clarivate Analytics) custom data of our in-house database
and the database from the Competence Centre for Bibliometrics (CCB: http://www.bibliome-
trie.info). Both are derived from the Science Citation Index Expanded (SCI-E), 社会科学
Citation Index (SSCI), and Arts and Humanities Citation Index (AHCI) produced by Clarivate
Analytics (费城, 美国). All publications published between 2011 和 2015 with a DOI
were exported with the following information: DOI, WoS UT (unique accession number from
WoS), WoS subject categories, publication year, citation counts with a three-year citation
window starting after the publication year, and Hazen percentiles. Percentiles are field-
and time-normalized impact scores that are between 0 (low citation impact) 和 100 (高的
citation impact) (Bornmann, 莱德斯多夫, & Mutz, 2013). Raw citation data were taken from the
database maintained by the CCB. Both databases (our in-house database and the database
maintained by the CCB) were last updated at the end of April 2019. We kept only those pub-
lications that fulfilled the following two criteria: (A) the publication belongs to a field (重叠-
ping WoS category; Rons, 2012, 2014) to which at least one research center publication
belongs; (乙) a requirement of at least 10 publications per field and publication year combination
has been set.

Altmetrics data were sourced from a locally maintained database using data shared with us
by Altmetric (https://www.altmetric.com) and dumped on October 8, 2019. For research pro-
项目, the company shares the data for free. The data include altmetric counts from sources
such as social networking, blogging, microblogging, wikis, and policy-relevant usage. We ap-
pended a mention count to each DOI using the following altmetrics sources: 推特,
Facebook, 博客, 消息, policy documents, 和维基百科 (see above). One DOI not known
to the altmetrics database was recorded as “not mentioned.” Altmetrics data and information
about their unit status (applied for research center funding which was accepted or not) 是
appended to the publications via their DOI. 数字 1 provides a schematic overview of how
the respective units were constructed: 单元 0 contains all WoS papers that do not belong to
units 1 或者 2. Unit 1 contains the publications of 28 participants who had submitted project
proposals for CCES but were not funded. Unit 2, 反过来, contains the publications of 170 par-
ticipants that were affiliated with CCES as principal investigators and project partners. Unit 2 是

1 https://www.altmetric.com/about-our-data/our-sources/.

Quantitative Science Studies

797

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你
q
s
s
/
A
r
t
我
C
e
–
p
d

我

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
A
_
0
0
0
3
2
p
d

乙
y
G
你
e
s
t

哦
n
0
7
S
e
p
e
米
乙
e
r
2
0
2
3

Can altmetrics reflect societal impact considerations?

数字 1. Schematic overview of the units an number of papers per unit.

further subdivided into units 3 和 4. Unit 4 contains the papers that were published in the
research center context, while unit 3 contains papers that accepted applicants published be-
yond their project at the research center. The numbers of mentioned and not mentioned pub-
lications in the different altmetrics sources broken down by unit status and publication year are
如表所示 1.

We acknowledge that the subdivision into units and the comparison between units is a
simplification of reality, especially with regard to the hypothesis to be tested. While the
CCES evaluation procedure took place on the project level at a specific moment in time,
the units here are constructed on the level of the entire publication output of researchers that
were funded or not funded by CCES. 此外, we focus in this study on scientific publi-
cations as the main research output of the research center. While it would have been bene-
ficial to consider other outputs as well, such as those emanating from public outreach activities
(Kassab, 2019), we are constrained by the fact that altmetrics data are only available for out-
puts that have a unique and persistent identifier, such as a DOI. 然而, besides altmetrics
数据, we also considered citation data (A) to compare the results with those based on altmetrics
data and (乙) to investigate whether societal impact assessments correspond with traditional
impact scores.

3.3. Mantel-Haenszel Quotient (MHq)2

In this study, we compare the impact of papers published by various units (例如, papers pub-
lished by rejected or accepted applicants; 见图 1). Since altmetrics data concern field-
specific differences (like citation data), field-normalized indicators should be used instead of
raw data for group comparisons. 然而, it is a critical drawback of altmetrics data that they
are inflated by zeros: In the current study, 5,586,077 文件 (71%) have no impact in any
altmetrics source. For zero-inflated data, it is not possible to use methods for field normal-
ization that are usually applied in bibliometrics (methods based on mean citations or cita-
tion percentiles; Bornmann et al., 2013). Since Bornmann and Haunschild (2018) 和

2 The explanation of the MHq indicator has been mainly adopted from Bornmann and Haunschild (2018) 和

Haunschild and Bornmann (2018).

Quantitative Science Studies

798

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你
q
s
s
/
A
r
t
我
C
e
–
p
d

我

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
A
_
0
0
0
3
2
p
d

乙
y
G
你
e
s
t

哦
n
0
7
S
e
p
e
米
乙
e
r
2
0
2
3

C
A
n

A
我
t

米
e
t
r
我
C
s

r
e
F
我
e
C
t

s
哦
C
我
e
t
A
我

我

米
p
A
C
t

C
哦
n
s
我
d
e
r
A
t
我
哦
n
s
?

桌子 1. Number of mentioned and not mentioned (and cited and not cited) 文件, 分别, broken down by data source, publication year, and funded or not-
funded groups

Unit Publication year

Mentioned

mentioned Mentioned

不是

不是
提及

Cited

不是
引用

推特

Facebook

Blogs

消息

Policy documents

维基百科

2011

2012

2013

2014

2015

2011

2012

2013

2014

2015

2011

2012

2013

2014

2015

2011

2012

2013

2014

2015

2011

2012

2013

2014

2015

139,766

1,233,952

34,842

1,338,876

35,790

1,337,928

20,922

1,352,796

29,028

1,344,690

28,036

1,345,682 1,057,600 316,118

312,073

1,181,879

77,809

1,416,143

43,989

1,449,963

29,490

1,464,462

28,391

1,465,561

28,024

1,465,928 1,149,376 344,576

398,950

1,194,767

115,234

1,478,483

52,390

1,541,327

47,867

1,545,850

28,422

1,565,295

27,288

1,566,429 1,245,496 348,221

510,575

1,144,363

105,419

1,549,519

56,832

1,598,106

59,324

1,595,614

24,927

1,630,011

25,303

1,629,635 1,305,474 349,464

627,508

1,123,571

17,5196

1,575,883

60,213

1,690,866

76,049

1,675,030

20,868

1,730,211

24,118

1,726,961 1,394,085 356,994

179

285

301

165

173

271

294

164

113

690

650

611

511

180

627

618

560

489

172

118

127

151

140

750

793

822

755

294

685

755

761

727

285

111

129

152

134

726

777

821

753

321

665

744

763

725

312

122

143

154

129

750

813

843

761

315

685

777

784

732

306

109

131

159

132

719

787

856

787

337

660

752

800

760

328

121

147

165

143

742

808

869

800

338

677

770

805

771

329

120

139

163

146

734

788

862

782

336

668

750

797

755

328

笔记: WoS papers (单元 0; neither accepted, nor rejected for funding), papers published by rejected applicants (单元 1), and papers published by accepted applicants (单元 2). The papers from accepted applicants are further divided into
papers from funded projects (单元 4) and papers published in other contexts (单元 3).

问
你
A
n

t
我
t

我

t
我
v
e
S
C
e
n
C
e
S
你
d
e
s

我

7
9
9

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你
q
s
s
/
A
r
t
我
C
e
–
p
d

我

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
A
_
0
0
0
3
2
p
d

乙
y
G
你
e
s
t

哦
n
0
7
S
e
p
e
米
乙
e
r
2
0
2
3

Can altmetrics reflect societal impact considerations?

Haunschild and Bornmann (2018) proposed the MHq indicator, which is specifically de-
signed for dealing with zero-inflated data in field normalization, we used the indicator in
the current study.

For pooling data from multiple 2 × 2 cross tables based on such subgroups (which are part
of the larger population, including all papers in the considered time period), MH analysis is a
popular method (Mantel & Haenszel, 1959; Hollander & 沃尔夫, 1999; Sheskin, 2007).
According to Fleiss, 莱文, and Paik (2003), the method “permits one to estimate the assumed
common odds ratio and to test whether the overall degree of association is significant.
奇怪的是, it is not the odds ratio itself but another measure of association that directly under-
lies the test for overall association … The fact that the methods use simple, closed-form for-
mulas has much to recommend it” (p. 250). The results by Radhakrishna (1965) demonstrate
that the MH approach seems to be valid.

The MH analysis results in a summary odds ratio for multiple 2 × 2 cross tables, 哪个
Bornmann and Haunschild (2018) and Haunschild and Bornmann (2018) name MHq. 为了
the comparison of the papers published by the applicants with reference sets in view of im-
协议, 这 2 × 2 cross tables (which are pooled) consist of the number of papers mentioned and
not mentioned in subject category and publication year combinations f. 在里面 2 × 2 主题-
specific cross table (见表 2), the cells af, bf, 比照, and df, are defined as follows:

–

–
–

af is the number of mentioned papers published by unit g (例如, rejected applicants) 在
subject category and publication year f,
bf is the number of not mentioned papers published by unit g in subject category and

publication year f,
cf is the number of mentioned papers in subject category and publication year f, 和
df is the number of not mentioned papers published in subject category and publication

year f. Note that the papers of group g are also part of the papers in the world.

The following dummy variables are needed for the MH analysis:

¼ af df
nf

and R ¼

¼ bf cf
nf

and S ¼

;

f ¼1

;

Pf ¼ af

þ df
nf

and Qf ¼ 1−Pf

MHq ¼ R
S

where nf = af + bf + 比照 + df.

MHq is simply

The CIs for MHq are calculated following Fleiss et al. (2003). The variance of ln MHq is
estimated by

C
Var ln MHq

Þ ¼

(

磷

1
2

F
f ¼1 Pf Rf
R2

磷

ð
F
f ¼1 Pf Sf
RS

þ Qf Rf

)

磷
F
f ¼1 Qf Sf
S2

(5)

800

Quantitative Science Studies

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你
q
s
s
/
A
r
t
我
C
e
–
p
d

我

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
A
_
0
0
0
3
2
p
d

乙
y
G
你
e
s
t

哦
n
0
7
S
e
p
e
米
乙
e
r
2
0
2
3

(1)

(2)

(3)

(4)

Can altmetrics reflect societal impact considerations?

桌子 2. 2 × 2 subject-specific cross table

Group g

世界

Number of mentioned papers
af

Number of not mentioned papers
bf

比照

The CI for the MHq can be constructed with

(西德:3)

MHqL ¼ exp ln MHq

Þ − 1:96

(西德:3)

MHqU ¼ exp ln MHq

Þ þ 1:96

(西德:4)
ﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ
C
(西德:2)
Þ
ð
Var ln MHq

(6)

(7)

We used the data in Table 3 to produce a small world example for explaining the MHq: 这
world consists of papers in four subject categories. The papers of two units (publication set A
和乙) determine the world. For each unit, the numbers of mentioned and not mentioned pa-
pers as well as the corresponding proportion of mentioned papers are listed. 例如, 这
unit named as publication set B has published 26 mentioned and seven not mentioned papers
in subject category 1. The proportion of the papers mentioned is 0.27. It is an advantage of the
MHq that the world average has a value of 1: This value indicates that there is no difference
between the chances of a focal publication set and the reference sets (IE。, 世界) of being
提及 (例如, on Wikipedia). An MHq value less than 1.0 indicates lower chances for the
publications in the set of being mentioned compared with the reference sets. The MHq values
表中 3 can be interpreted as follows: The chances of the papers in publication set A of
being mentioned are 0.81 times as large as the world’s papers’ chances. The chances of the
papers in set B of being mentioned are 1.3 times greater than the world’s papers’ chances. 这是
an advantage of the MHq that the result can be expressed as a percentage, which is relative to
the world average. Expressed as percentages, 所以, the difference between publication set
B and the world is

ð
100 * 1:3−1:0

Þ ¼ 30%

(8)

因此, the publications in set B have a 30% higher chance of being mentioned than the
world’s publications. We also added CIs to the MHqs in Table 3. Since the CIs of both pub-
lication sets (A and B) overlap substantially among themselves and with 1.0 (the world MHq),
they do not differ statistically significantly from one another and the world average.

4. 结果

数字 2 displays the MHq values (based on six altmetrics sources) for all WoS papers in the
given years (单元 0: red points; neither accepted nor rejected); papers published by rejected
applicants (单元 1: green squares); and papers published by accepted applicants (单元 2: 蓝色的
diamonds). The papers from accepted applicants are further differentiated into papers written
in the context of projects funded by the research center (单元 4: orange diamonds) and papers
published in other contexts (单元 3: yellow diamonds). For all MHq values, CIs are indicated.
Since the paper numbers from funded projects for some publication years are too low, 他们
could not be presented in the figure.

Quantitative Science Studies

801

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你
q
s
s
/
A
r
t
我
C
e
–
p
d

我

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
A
_
0
0
0
3
2
p
d

乙
y
G
你
e
s
t

哦
n
0
7
S
e
p
e
米
乙
e
r
2
0
2
3

Can altmetrics reflect societal impact considerations?

桌子 3. Small world example for the Mantel–Haenszel quotient (MHq)

世界 (reference
套)
Subject category 1

Subject category 2

Subject category 3

Subject category 4

全部的

Publication set A

Subject category 1

Subject category 2

Subject category 3

Subject category 4

全部的

Publication set B

Subject category 1

Subject category 2

Subject category 3

Subject category 4

全部的

Paper is
提及
44

Paper is not
提及
20

Number of
文件
64

MHq

1.00 [0.61, 1.64]

0.81 [0.46, 1.44]

1.30 [0.66, 2.53]

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你
q
s
s
/
A
r
t
我
C
e
–
p
d

我

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
A
_
0
0
0
3
2
p
d

乙
y
G
你
e
s
t

哦
n
0
7
S
e
p
e
米
乙
e
r
2
0
2
3

The results as summarized in Figure 2 do not support the hypothesis that funded researchers
achieve higher altmetrics scores with their research than those who were not funded by the
research center. 例如, the MHq values based on Twitter data for the papers published
by rejected applicants (green squares) are consistently higher than the papers published by
accepted applicants (blue diamonds). The differences between both groups are statistically
significant in 2011 和 2012, 但不在 2013 到 2015 (这里, the CIs mostly overlap). Quite
strikingly, the figure also reveals that papers published by accepted applicants in the context of
the funded research center projects (orange diamonds) even receive lower Twitter scores than
the papers they published outside of the research center project (yellow diamonds). The results
for the other altmetric scores mainly concur with the Twitter results. Only the findings for the
policy-related documents show a different picture: Research-center-based papers published
之间 2012 和 2014 (orange diamonds) received higher altmetric scores than papers by
the same researchers that do not emanate from research center projects (yellow diamonds).
然而, the results are not statistically significant and are not confirmed by the results for
2011 (results for 2015 are not available).

We further analyzed whether the ex ante societal impact considerations are reflected in
citation scores. The results are shown in Figure 3. The figure reveals that the results are more

Quantitative Science Studies

802

Can altmetrics reflect societal impact considerations?

or less in agreement with the altmetrics results (with papers published by rejected applicants
performing similarly to or better than those of funded applicants). If we inspect the aggregated
MHq results based on the papers from all years, papers published by accepted applicants
(MHq = 3.31) have a higher citation impact than papers published by rejected applicants
(MHq = 2.87). Since the CIs of both groups overlap, 然而, the results are not statistically
重要的. We obtained similar results (missing substantial differences between the groups),
when we compared median citations (accepted applicants = 9, rejected applicants = 9) 和
percentile citation scores (accepted applicants = 73, rejected applicants = 70) of both groups.

5. DISCUSSION AND CONCLUSION

Universities and researchers are increasingly under pressure to disclose how their research
contributes to the welfare of society to garner political support and funding (Puschmann,
2014; Thune, Reymert, 等人。, 2016; Bornmann & Haunschild, 2017). In light of this develop-
蒙特, assessing the societal impact of research is a critically debated issue among evaluation
scholars and research policy experts. Because of their widespread use, social media have been
at the heart of methodological discussions over the past years, including both their potential
(例如, 速度, broadness) and their shortcomings (例如, data quality, zero-inflated data).
然而, the critical question of whether social media (or altmetrics for that matter) are able
to reflect societal impact is so far not answered due to conflicting empirical evidence. Against
this background, the aim of this study was to contribute to solving this puzzle. For this purpose,
the present paper compared ex ante assessments on the societal relevance of research with
altmetrics scores that the respective research received in the years after. A research center from
the field of sustainability science (CCES) and the societal impact assessments made by the
members of an ad hoc RC served as the case study for this investigation.

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你
q
s
s
/
A
r
t
我
C
e
–
p
d

我

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
A
_
0
0
0
3
2
p
d

乙
y
G
你
e
s
t

哦
n
0
7
S
e
p
e
米
乙
e
r
2
0
2
3

数字 2. MHq values based on six altmetrics sources for all WoS papers (单元 0: red points; neither accepted, nor rejected); papers published
by rejected applicants (单元 1: green squares); and papers published by accepted applicants (单元 2: blue diamonds). The papers from accepted
applicants are further divided into papers from funded projects (单元 4: orange diamonds) and papers published in other contexts (单元 3:
yellow diamonds). For some years, the values of unit 4 are missing because the numbers of mentioned papers are too low.

Quantitative Science Studies

803

Can altmetrics reflect societal impact considerations?

数字 3. MHq values based on citation counts for all WoS papers (单元 0: red points; neither ac-
cepted nor rejected for funding), papers published by rejected applicants (单元 1: green squares),
papers published by accepted applicants (单元 2: blue diamonds), and papers published in other
contexts from accepted applicants (单元 3: yellow diamonds). Papers published from funded pro-
项目 (单元 4) are not shown, because the numbers of uncited papers are too low.

综上所述, the proposed hypothesis that researchers funded by CCES achieve higher
impact in terms of altmetrics scores with their research than those who were not funded could
not be confirmed based on the results. We found no correlation between the RC’s assessment
and the corresponding altmetric scores. With a few exceptions, this finding seems to be con-
firmed in the case of all six types of altmetrics. For comparison with altmetrics, we investigated
the relationship with citation scores as well. The results are similar to those based on alt-
指标: The correlation is not in the expected direction.

Our results might be interpreted in such a way that altmetrics are not entirely suitable for
reflecting the societal impact of research. 然而, since we investigated only one case-
specific evaluation procedure and the results are not homogeneous throughout the different
types of altmetric scores, this conclusion cannot be drawn with certainty. We conclude therefore
that more research is needed to better understand what altmetrics are reflecting, particularly in
light of their heterogeneity. Further research should clarify whether altmetric scores rather cap-
ture “unknown attention or unstructured noise produced by published research” (Moed, 2017;
Bornmann et al., 2019), some sort of “public discussion” (Haunschild, 莱德斯多夫, 等人。,
2019) or anything else altogether.

Our results, 同时, could be interpreted with a critical view of the RC’s assess-
评论. Did the members of the RC select the “right” projects in the first place or how should
the missing correlation between the ex ante assessments and the received citations be
interpreted?

Another question is whether the members of the RC were qualified to judge the societal
impact of proposed research. 在多数情况下, expert panels are composed only of researchers
rather than of representatives of other sectors of society, which was also true for in the case of
CCES. This circumstance may have led to the fact that the potential societal impact could not
be accurately judged or that the aspect of societal impact was not given enough importance in

Quantitative Science Studies

804

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你
q
s
s
/
A
r
t
我
C
e
–
p
d

我

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
A
_
0
0
0
3
2
p
d

乙
y
G
你
e
s
t

哦
n
0
7
S
e
p
e
米
乙
e
r
2
0
2
3

Can altmetrics reflect societal impact considerations?

the evaluation procedure. 全面的, we note that our findings can take the discussion forward,
but also that they should be interpreted with caution.

6. LIMITATIONS, FURTHER RESEARCH, AND RECOMMENDATIONS

The study revealed that the ex ante assessments considering societal impact of research and
the altmetric scores of the same research do not correlate. We could conclude the debate at
this point and throw altmetrics overboard as a potential measure of societal impact. 但, 的
课程, this study has several limitations that need to be discussed.

One key aspect is related to the fact that altmetrics are still in their infancy in many ways.
例如, are altmetrics really a good proxy for societal impact? Are social media mentions
in themselves societal impact? Does a mention or interaction on social media automatically
imply that there has been a cognitive engagement with the content of the research, 然后
societal impact has occurred? Or is it perhaps a buzzword-laden title, zeitgeist, or fame-related
reason why some research output scores higher on altmetrics than others (大厅, 2014)? 和
what can we say about all the research that does not have any mentions on Twitter or
维基百科? It would be highly questionable to conclude that no societal impact has been
achieved in all these null observation cases. 此外, our study does not differentiate be-
tween “self-mentions” or “in-house users” (the own department or the university’s communi-
cations team) and mentions and interactions by other (more independent) individuals and
实体. Despite being somewhat complex, further research should account for these aspects,
as well as for the actual content of the tweets or Facebook posts. This latter strategy could
allow for a better understanding of the intentions and meanings of social-media-based inter-
actions with research. By looking at the content or the timing of the mentions in more detail,
we could possibly identify different strategies in using social media, which could help us for-
mulate new hypotheses.

The results of this work have again shown that the true value added of altmetrics is not yet
entirely clear, but rather ranges on a scale between societal impact and unstructured noise.
This fundamental problem concerns all six types of altmetrics that have been considered in this
学习 (to a greater or lesser extent). With regard to the inability of tweets to measure the so-
cietal impact of research, the results of the present study are consistent with those of Haustein
等人. (2014) and Andersen and Haustein (2015). From our point of view, off-the-cuff retweets
in particular are simply too inflationary to imply a serious engagement with the content of the
工作. Mentions on Wikipedia also do not seem to reflect the societal relevance of research
(Kousha & Thelwall, 2017). 更远, it does not yet seem to be common practice to incorporate
scientific research into policy and policy-related documents, either in the field of climate
改变, as Bornmann et al. (2016) 成立, or in the likewise societally relevant field of sustain-
ability science, as the present study showed. This finding also underlines that the dialog and
knowledge transfer between science and policy is far from established (Hessels, Van Lente, &
Smits, 2009). 同时, it must be given fair consideration that, as with classical cita-
系统蒸发散, it can take up to several years for the results of scientific studies to become relevant and
cited in policy documents. The time window of the present study was simply too small.
最后, the valid assessment of societal impacts by means of blogging, Facebook, or in news
outlets largely suffers from the fact that there is a bias toward publications from renowned
journals (Shema et al., 2012A, 2012乙) or specific fields of interest (Ringelhan et al., 2015).
Although our findings seem to lend additional support in favor of the argument that altmetrics
are not capable of reflecting the societal impact of research, much more research will need to
be done before we can actually have a clear picture of what altmetrics are capable of.

Quantitative Science Studies

805

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你
q
s
s
/
A
r
t
我
C
e
–
p
d

我

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
A
_
0
0
0
3
2
p
d

乙
y
G
你
e
s
t

哦
n
0
7
S
e
p
e
米
乙
e
r
2
0
2
3

Can altmetrics reflect societal impact considerations?

Another limitation of this study is related to the evaluation process itself, and less to the
shortcomings of altmetrics. Even though the prospect of societal impact was a key criterion
for the RC, the assessments were not based on standardized rating scales along individual cri-
teria, but rather on a holistic rating, we can only assume that societal impact considerations
played a role in the evaluation process rather than having clearly traceable evidence for the
specific weighting that this critical aspect ultimately had. One remedy for future evaluations
could therefore be to assess societal impact as a single dimension using a standardized scale.

A related limitation of the study is associated with the heterogeneity of the societal impact.
Societal impact can manifest itself in different ways, such as in the form of policy impact, 在-
vironmental impact, health impact, or educational impact. Due to the holistic rating in the
assessment process, it is not clear what kind of societal impact was the focus of the experts’
注意力. This heterogeneity is also an issue for altmetrics. A Twitter mention compared to a
mention in a policy document, 例如, has not only been made a different way but prob-
ably has a different kind of impact as well.

With regard to the societal impact of research, this study focused exclusively on the pub-
lished journal papers and the corresponding altmetrics scores they received. It could certainly
bring added value if other outputs were taken into account as well, such as outputs that re-
searchers produce within the framework of public outreach activities. Specifically designed to
catalyze the societal impact of research, 例如, stakeholder publications or teaching
material and their respective altmetrics scores could much more accurately reflect the societal
impact of research. 然而, in order for these alternative types of outputs to receive an alt-
metrics score, they would have to be assigned a unique and persistent identifier, such as a DOI
in the future (see https://www.doi.org/).

致谢

The bibliometric data used in this study are from two sources: (A) the online version of the Web
of Science database provided by Clarivate Analytics (费城, 宾夕法尼亚州, 美国) 和 (乙)
the bibliometric in-house databases of the Max Planck Society (MPG) and the Competence
Centre for Bibliometrics (CCB: http://www.bibliometrie.info/). The MPG’s database is devel-
oped and maintained in cooperation with the Max Planck Digital Library (MPDL, 慕尼黑),
and the CCB’s database is developed and maintained by the cooperation of various
German research organizations. Both databases are derived from the Science Citation Index
Expanded (SCI-E), Social Sciences Citation Index (SSCI), and the Arts and Humanities Citation
Index (AHCI) prepared by Clarivate Analytics, formerly the IP & Science business of Thomson
Reuters (费城, 宾夕法尼亚州, 美国). The authors thank the management of the CCES for
their kind support in conducting this study. The altmetrics data were retrieved from our locally
maintained database with data shared with us and dumped by the company altmetric.com on
十月 8, 2019.

作者贡献

Omar Kassab: 概念化, data curation, formal analysis, 调查, 方法论,
project administration, 资源, writing—original draft, writing—review & 编辑. Lutz
Bornmann: 概念化, data curation, formal analysis, funding acquisition, investi-
gation, 方法论, project administration, 资源, 软件, 验证, 可视化,
writing—original draft, writing—review & 编辑. Robin Haunschild: 概念化,
data curation, formal analysis, 调查, 方法论, project administration, 关于-
来源, 软件, 验证, writing—original draft, writing—review & 编辑.

Quantitative Science Studies

806

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你
q
s
s
/
A
r
t
我
C
e
–
p
d

我

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
A
_
0
0
0
3
2
p
d

乙
y
G
你
e
s
t

哦
n
0
7
S
e
p
e
米
乙
e
r
2
0
2
3

Can altmetrics reflect societal impact considerations?

COMPETING INTERESTS

The authors have no competing interests. 之间 2013 和 2015, OK worked as an execu-
tive assistant to the CCES management. Afterward he joined the Professorship for Social
Psychology and Research on Higher Education at ETH Zurich, where he conducted this study
in collaboration with coauthors LB and RH.

资金信息

No funding has been received for this research.

DATA AVAILABILITY

For the data sources, see Acknowledgments section. The authors do not have permission to
share the data. Information specific to the research center can be drawn from the performance
reporting of CCES, available at https://cces.ethz.ch/about/downloads.html.

参考

安徒生, J. P。, & Haustein, S. (2015). Influence of study type on twit-
ter activity for medical research papers. 在一个. A. Salah, 是. Tonta, A.
A. A. Salah, C. Sugimoto, & U. 铝 (编辑。), The 15th Conference of
the International Society for Scientometrics and Informetrics,
PP. 26–36. Istanbul, Turkey: ISSI, Bogazici University Printhouse.
Bar-Ilan, J。, Shema, H。, & Thelwall, 中号. (2014). Bibliographic refer-
ences in Web 2.0. 在乙. 克朗的 & C. 右. Sugimoto (编辑。), 超过
Bibliometrics: Harnessing Multi-Dimensional Indicators of
Performance, PP. 307–325. 剑桥, 嘛: 与新闻界.

Bik, H. M。, & 戈德斯坦, 中号. C. (2013). An introduction to social
media for scientists. PLOS Biology, 11(4), e1001535. https://
doi.org/10.1371/journal.pbio.1001535

Bonetta, L. (2007). Scientists enter the blogosphere. 细胞, 129(3),

443–445.

Bornmann, L. (2014). Do altmetrics point to the broader impact of
研究? An overview of benefits and disadvantages of alt-
指标. Journal of Informetrics, 8(4), 895–903. https://doi.org/
10.1016/j.joi.2014.09.005

Bornmann, L. (2015). Alternative metrics in scientometrics: A meta-
analysis of research into three altmetrics. Scientometrics, 103(3),
1123–1144.

Bornmann, L。, & Haunschild, 右. (2017). Does evaluative sciento-
metrics lose its main focus on scientific quality by the new
orientation towards societal impact? Scientometrics, 110(2),
937–943.

Bornmann, L。, & Haunschild, 右. (2018). Normalization of zero-
inflated data: An empirical analysis of a new indicator family
and its use with altmetrics data. Journal of Informetrics, 12(3),
998–1011. https://doi.org/10.1016/j.joi.2018.01.010

Bornmann, L。, Haunschild, R。, & Adams, J. (2019). Do altmetrics
assess societal impact in a comparable way to case studies? 一个
empirical test of the convergent validity of altmetrics based on
data from the UK research excellence framework. 杂志
Informetrics, 13(1), 325–340. https://doi.org/10.1016/j.
joi.2019.01.008

Bornmann, L。, Haunschild, R。, & 马克思, 瓦. (2016). Policy docu-
ments as sources for measuring societal impact: How often is cli-
mate change research mentioned in policy-related documents?
Scientometrics, 109(3), 1477–1495. https://doi.org/10.1007/
s11192-016-2115-y

Bornmann, L。, 莱德斯多夫, L。, & Mutz, 右. (2013). The use of per-
centiles and percentile rank classes in the analysis of biblio-
metric data: Opportunities and limits. Journal of Informetrics,
7(1), 158–165. https://doi.org/10.1016/j.joi.2012.10.001

Brandt, P。, Ernst, A。, Gralla, F。, Luederitz, C。, Lang, D. J。, … von
Hehrden, H. (2013). A review of transdisciplinary research in
sustainability science. Ecological Economics, 92, 1–15.

CCES. (2005). Competence Center Environmental and Sustainability
(CCES) of the ETH Domain: Business Plan. Retrieved from: https://
ethz.ch/content/dam/ethz/special-interest/dual/cces-dam/docu-
ments/CCES_BP_2005_11_22_final.pdf

克拉克, 瓦. C. (2007). Sustainability science: A room of its own.
美国国家科学院院刊, 104(6),
1737–1738.

Dehdarirad, T。, Didegah, F。, & Sotudeh, H. (2018). Which type of
research is cited more often in Wikipedia? A case study of
PubMed research. 在P. Wouters (埃德。), 诉讼程序
Science and Technology Indicators Conference 2018 Leiden
“Science, Technology and Innovation Indicators in Transition.”
Leiden, 荷兰: University of Leiden.

Fausto, S。, Machado, F. A。, Bento, L. F. J。, Iamarino, A。, Nahas, 时间. R。,
& Munger, D. S. (2012). Research blogging: Indexing and register-
ing the change in science 2.0. PLOS ONE, 7(12), e50109. https://
doi.org/10.1371/journal.pone.0050109

弗莱斯, J。, 莱文, B., & Paik, 中号. C. (2003). Statistical methods for

rates and proportions, 3rd Ed. 霍博肯, 新泽西州: 威利.

Guglielmi, G. (2018). Wikipedia’s top-cited scholarly articles –

透露. 自然, 557(7705), 291–292.

大厅, 氮. (2014). The Kardashian index: A measure of discrepant so-
cial media profile for scientists. Genome Biology, 15(7), 424.
https://doi.org/10.1186/s13059-014-0424-0

Haunschild, R。, & Bornmann, L. (2017). How many scientific pa-
pers are mentioned in policy-related documents? An empirical
investigation using Web of Science and Altmetric data.
Scientometrics, 110(3), 1209–1216. https://doi.org/10.1007/
s11192-016-2237-2

Haunschild, R。, & Bornmann, L. (2018). Field- and time-normalization
of data with many zeros: An empirical analysis using citation and
Twitter data. Scientometrics, 116(2), 997–1012. https://doi.org/
10.1007/s11192-018-2771-1

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你
q
s
s
/
A
r
t
我
C
e
–
p
d

我

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
A
_
0
0
0
3
2
p
d

乙
y
G
你
e
s
t

哦
n
0
7
S
e
p
e
米
乙
e
r
2
0
2
3

Quantitative Science Studies

807

Can altmetrics reflect societal impact considerations?

Haunschild, R。, 莱德斯多夫, L。, Bornmann, L。, Hellsten, 我。, & 马克思,
瓦. (2019). Does the public discuss other topics on climate
change than researchers? A comparison of explorative networks
based on author keywords and hashtags. Journal of Informetrics,
13(2), 695–707.

Haustein, S。, Larivière, 五、, Thelwall, M。, Amyot, D ., & Peters, 我. (2014).
Tweets vs. Mendeley readers: How do these two social media met-
rics differ? IT – Information Technology 2014, 56(5), 207–215.
Haustein, S。, Peters, 我。, Bar-Ilan, J。, Priem, J。, Shema, H。, &
Terliesner, J. (2014). Coverage and adoption of altmetrics
sources in the bibliometric community. Scientometrics, 101(2),
1145–1163. https://doi.org/10.1007/s11192-013-1221-3

Hessels, L. K., Van Lente, H。, & Smits, 右. (2009). In search of rele-
vance: The changing contract between science and society.
Science and Public Policy, 36(5), 387–401.

Hollander, M。, & 沃尔夫, D. A. (1999). Nonparametric statistical

方法. 纽约, 纽约: 威利.

Kajikawa, Y。, Tacoa, F。, & Yamaguchi, K. (2014). Sustainability sci-
恩斯: The changing landscape of sustainability research.
Sustainability Science, 9(4), 431–438.

Kassab, 氧. (2019). Does public outreach impede research perfor-
曼斯? Exploring the “researcher’s dilemma” in a sustainability
research center. Science and Public Policy, 46(5), 710–720.
https://doi.org/10.1093/scipol/scz024

Kassab, 奥。, Schwarzenbach, 右. P。, & Gotsch, 氮. (2018). Assessing ten
years of inter- and transdisciplinary research, 教育, 和出来-
抵达: The Competence Center Environment and Sustainability
(CCES) of the ETH Domain. GAIA – Ecological Perspectives for
Science and Society, 27(2), 226–234.

Kates, 右. 瓦. (2011). What kind of a science is sustainability science?
美国国家科学院院刊, 108(49),
19449–19450.

Kousha, K., & Thelwall, 中号. (2017). Are Wikipedia citations impor-
tant evidence of the impact of scholarly articles and books?
Journal of the Association for Information Science and
技术, 68(3), 762–779. https://doi.org/10.1002/asi.23694
Lang, D. J。, Wiek, A。, Bergmann, M。, Stauffacher, M。, Martens, P。,
… Thomas, C. J. (2012). Transdisciplinary research in sustainabil-
ity science: 实践, 原则, and challenges. Sustainability
科学, 7(1), 25–43.

Luzón, 中号. J. (2013). Public communication of science in blogs:
Recontextualizing scientific discourse for a diversified audi-
恩斯. Written Communication. https://doi.org/10.1177/
0741088313493610

Mantel, N。, & Haenszel, 瓦. (1959). Statistical aspects of the anal-
ysis of data from retrospective studies of disease. Journal of the
National Cancer Institute, 22(4), 719–748.

Mas-Bleda, A。, & Thelwall, 中号. (2016). Can alternative indicators
overcome language biases in citation counts? A comparison of
Spanish and UK research. Scientometrics, 109(3), 2007–2030.
https://doi.org/10.1007/s11192-016-2118-8

Mewburn, 我。, & Thomson, 磷. (2013). Why do academics blog? 一个
analysis of audiences, purposes and challenges. Studies in
Higher Education, 38(8), 1105–1119. https://doi.org/10.1080/
03075079.2013.835624

磨坊主, 时间. 右. (2013). Constructing sustainability science: Emerging
perspectives and research trajectories. Sustainability Science, 8(2),
279–293.

Mobjörk, 中号. (2010). Consulting versus participatory transdiscipli-
narity: A refined classification of transdisciplinary research.
Futures, 42(8), 866–873.

Moed, H. F. (2017). Applied evaluative informetrics. 占婆:

施普林格.

OPENing UP. (2016). Deliverable D5.1—Altmetrics Status Quo.
OPENing UP new methods, indicators and tools for peer review,
impact measurement and dissemination of research. https://ec.
europa.eu/research/participants/documents/downloadPublic?
documentIds=080166e5ae8a151c&appId=PPGMS Project
acronym: OpenUP. Retrieved from Brussels, 比利时.
Pohl, C. (2010). From transdisciplinarity to transdisciplinary research.

Transdisciplinary Journal of Engineering Science, 1, 65–73.

Priem, J. (2014). Altmetrics. 在乙. 克朗的 & C. 右. Sugimoto (编辑。),
Beyond Bibliometrics: Harnessing Multi-Dimensional Indicators
of Performance, PP. 263–288. 剑桥, 嘛: 与新闻界.

Priem, J。, & Hemminger, 乙. 中号. (2010). Scientometrics 2.0: 走向
new metrics of scholarly impact on the social Web. 第一的
Monday, 15(7).

Puschmann, C. (2014). (Micro) blogging science? Notes on poten-
tials and constraints of new forms of scholarly communication.
在S. Bartling & S. Friesike (编辑。), Opening Science: The Evolving
Guide on How the Internet is Changing Research, Collaboration
and Scholarly Publishing, PP. 89–106. 占婆: 施普林格.

Radhakrishna, S. (1965). Combination of results from several 2 × 2

contingency tables. Biometrics, 21, 86–98.

Ringelhan, S。, Wollersheim, J。, & Welpe, 我. 中号. (2015). I like, I cite?
Do Facebook likes predict the impact of scientific work? PLOS
ONE, 10(8), e0134389. https://doi.org/10.1371/journal.
pone.0134389

Robinson-Garcia, N。, Costas, R。, Isett, K., Melkers, J。, & 希克斯, D.
(2017). The unbearable emptiness of tweeting—About journal
文章. PLOS ONE, 12(8), e0183551. https://doi.org/10.1371/
journal.pone.0183551

Rons, 氮. (2012). Partition-based field normalization: An approach
to highly specialized publication records. Journal of Informetrics,
6(1), 1–10. https://doi.org/10.1016/j.joi.2011.09.008

Rons, 氮. (2014). Investigation of Partition Cells as a Structural Basis
Suitable for Assessments of Individual Scientists. 在P. Wouters
(埃德。), Proceedings of the Science and Technology Indicators
会议 2014 Leiden “Context Counts: Pathways to Master
Big and Little Data”, PP. 463–472. Leiden, 荷兰:
University of Leiden.

Samoilenko, A。, & Yasseri, 时间. (2014). The distorted mirror of
维基百科: A quantitative analysis of Wikipedia coverage of ac-
ademics. EPJ Data Science, 3(1), 1.

SDSN Australia/Paciﬁc. (2017). Getting started with the SDGs in
大学: A guide for universities, higher education institu-
系统蒸发散, and the academic sector. 澳大利亚, New Zealand and
Paciﬁc Edition. Sustainable Development Solutions Network—
Australia/Paciﬁc, 墨尔本.

Serrano-López, A. E., Ingwersen, P。, & Sanz-Casado, 乙. (2017).
Wind power research in Wikipedia: Does Wikipedia demonstrate
direct influence of research publications and can it be used as ad-
equate source in research evaluation? Scientometrics, 112(3),
1471–1488. https://doi.org/10.1007/s11192-017-2447-2

Shema, H。, Bar-Ilan, J。, & Thelwall, 中号. (2012A). Research blogs and
the discussion of scholarly information. PLOS ONE, 7(5),
e35869. https://doi.org/10.1371/journal.pone.0035869

Shema, H。, Bar-Ilan, J。, & Thelwall, 中号. (2012乙). Self-citation of blog-
gers in the science blogosphere. 在一个. Tokar, 中号. Beurskens,
S. Keuneke, 中号. Mahrt, 我. Peters, C. Puschmann, K. Weller, &
时间. van Treeck (编辑。), Science and the Internet, PP. 183–192.
Düsseldorf, 德国: Düsseldorf University Press.

Shema, H。, Bar-Ilan, J。, & Thelwall, 中号. (2014A). Scholarly blogs are

a promising altmetric source. Research Trends, 37, 11–13.

Shema, H。, Bar-Ilan, J。, & Thelwall, 中号. (2014乙). Do blog citations
correlate with a higher number of future citations? 研究

Quantitative Science Studies

808

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你
q
s
s
/
A
r
t
我
C
e
–
p
d

我

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
A
_
0
0
0
3
2
p
d

乙
y
G
你
e
s
t

哦
n
0
7
S
e
p
e
米
乙
e
r
2
0
2
3

Can altmetrics reflect societal impact considerations?

blogs as a potential source for alternative metrics. Journal of the
Association for Information Science and Technology, 65(5),
1018–1027. https://doi.org/10.1002/asi.23037

Sheskin, D. (2007). Handbook of parametric and nonparametric
statistical procedures (4th Ed.). Boca Raton, FL: Chapman &
Hall/CRC.

Sugimoto, C. R。, 工作, S。, Larivière, 五、, & Haustein, S. (2016).
Scholarly use of social media and altmetrics: A review of the
文学. Retrieved from https://arxiv.org/abs/1608.08112

Tattersall, A。, & 卡罗尔, C. (2018). What can Altmetric.com tell us
about policy citations of research? An analysis of Altmetric.com
data for research articles from the University of Sheffield.
Frontiers in Research Metrics and Analytics, 2(9). https://土井.
org/10.3389/frma.2017.00009

Teplitskiy, M。, 鲁, G。, & Duede, 乙. (2017). Amplifying the impact of open
使用权: Wikipedia and the diffusion of science. Journal of the
Association for Information Science and Technology, 68, 2116–2127.
https://doi.org/10.1002/asi.23687

Thune, T。, Reymert, 我。, Gulbrandsen, M。, & Olaf Aamodt, 磷. (2016).
Universities and external engagement activities: Particular pro-

files for particular universities? Science and Public Policy, 43(6),
774–786. https://土井:10.1002/asi.23687

Vilkins, S。, & 授予, 瓦. J. (2017). Types of evidence cited in
Australian Government publications. Scientometrics, 113(3),
1681–1695. https://doi.org/10.1007/s11192-017-2544-2

Wiek, A。, Talwar, S。, O’Shea, M。, & 罗宾逊, J. (2014). Toward a
methodological scheme for capturing societal effects of participa-
tory sustainability research. Research Evaluation, 23(2), 117–132.
Wooldridge, J。, & 国王, 中号. 乙. (2019). Altmetric scores: An early in-
dicator of research impact. Journal of the Association for
Information Science and Technology, 70(3), 271–282. https://
doi.org/10.1002/asi.24122

Yarime, M。, Trencher, G。, Mino, T。, Scholz, 右. W., 奥尔森, L。, Ness,
B., … Rotmans, J. (2012). Establishing sustainability science in
higher education institutions: Towards an integration of academ-
ic development, institutionalization, and stakeholder collabora-
系统蒸发散. Sustainability Science, 7(1), 101–113.

于, H. (2017). Context of altmetrics data matters: An investigation of
count type and user category. Scientometrics, 111(1), 267–283.
https://doi.org/10.1007/s11192-017-2251-z

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你
q
s
s
/
A
r
t
我
C
e
–
p
d

我

F
/

1
2
7
9
2
1
8
8
5
7
5
0
q
s
s
_
A
_
0
0
0
3
2
p
d