RESEARCH ARTICLE

RESEARCH ARTICLE

Which types of online evidence show
the nonacademic benefits of research?
Websites cited in UK impact case studies

开放访问

杂志

Statistical Cybermetrics Research Group, University of Wolverhampton, Wolverhampton, 英国

Kayvan Kousha

, Mike Thelwall

, and Mahshid Abdoli

关键词: impact case studies, nonacademic impact, Research Assessment Exercise (REF), 社会的
impact

抽象的

While funders increasingly request evidence of the societal benefits of research, all academics
in the UK must periodically provide this information to gain part of their block funding within
the Research Excellence Framework (REF). The impact case studies produced in the UK are
public and can therefore be used to gain insights into the types of sources used to justify
societal impact claims. This study focuses on the URLs cited as evidence in the last public REF
to help researchers and resource providers to understand what types can be used and the
disciplinary differences in their uptake. Based on a new semiautomatic method to classify the
URLs cited in impact case studies, the results show that there are a few key online types of
source for most broad fields, but these sources differ substantially between subject areas. 为了
例子, news websites are more important in some fields than others, and YouTube is
sometimes used for multimedia evidence in the arts and humanities. Knowledge of the
common sources selected independently by thousands of researchers may help others to
identify suitable sources for the complex task of evidencing societal impacts.

D

w
n

A
d
e
d

F
r


H

t
t

p

:
/
/

d

r
e
C
t
.


t
.

/

e
d

q
s
s
/
A
r
t

C
e

p
d

F
/

/

/

/

2
3
8
6
4
1
9
7
0
7
7
2
q
s
s
_
A
_
0
0
1
4
5
p
d

.

/

F


y
G

e
s
t

t


n
0
7
S
e
p
e


e
r
2
0
2
3

1.

介绍

Although knowledge-building is a core goal of much scholarship, it is important to assess the
impacts of research outside academia when evaluators or funders need evidence of its societal
影响 (Dinsmore, 艾伦, & Dolby, 2014; Thelwall, Kousha et al., 2015). 这是因为
funders consider research findings to have added value when they benefit society, 例如
by influencing policy (Oliver, Innvar et al., 2014). Assessing these nonacademic impacts is
difficult because there are many types and no systematic record of them. 相比之下, academic
impacts are partly trackable by citation indexes. To illustrate the variety of potential non-
academic impacts, 27 categories of impact within four broad areas (research-related, 政策,
服务, 社会的) have been suggested to help health researchers describe the benefits of their
research when writing impact narratives (Kuruvilla, Mays et al., 2006). At a finer-grained level,
100 indicators have been suggested for the policy, 健康, 经济的, 教学, and career devel-
opment impacts of biomedical research alone (格思里, Krapels et al., 2017). General recommen-
dations have also been provided for interpreting nonacademic indicators of research impacts
(Wilsdon, Allen et al., 2015).

The UK Research Excellence Framework (REF) is an exercise that runs every 6–7 years, 作为-
sessing scholarly and nonscholarly research achievements to allocate block grant research

引文: Kousha, K., Thelwall, M。, &
Abdoli, 中号. (2021). Which types of online
evidence show the nonacademic
benefits of research? Websites cited in
UK impact case studies. Quantitative
Science Studies, 2(3), 864–881. https://
doi.org/10.1162/qss_a_00145

DOI:
https://doi.org/10.1162/qss_a_00145

Peer Review:
https://publons.com/publon/10.1162
/qss_a_00145

支持信息:
https://doi.org/10.1162/qss_a_00145

已收到: 20 四月 2021
公认: 16 六月 2021

通讯作者:
Kayvan Kousha
k.kousha@wlv.ac.uk

处理编辑器:
Ludo Waltman

版权: © 2021 Kayvan Kousha,
Mike Thelwall, and Mahshid Abdoli.
在知识共享下发布
归因 4.0 国际的
(抄送 4.0) 执照.

麻省理工学院出版社

Which types of online evidence show the nonacademic benefits of research?

资金. It groups UK academic research into four broad disciplinary panels (A, 乙, C, 和D),
containing 36 field-based Units of Assessment (UoAs, see Supplementary Information Tables
S1–S4 for a list) 在里面 2014 iteration. The REF assesses the nonacademic impacts of research
primarily through impact case studies, which are structured evidence-based narrative claims of
nonacademic impacts written by the groups of researchers evaluated. In the REF context, 关于-
search impact has been defined as “an effect on, change or benefit to the economy, 社会,
文化, public policy or services, 健康, the environment or quality of life, beyond academia”
(Research Excellence Framework, 2014, p. 26). The weighting of the case studies for funding
purposes has been increased from 20% in REF 2014 到 25% in REF 2021 (Research Excellence
Framework, 2019). Impact case study mandatory “Sources to corroborate the impact” sections
contain citations to the evidence underpinning the narratives. These must be found by the re-
searchers themselves, typically with the support of university impact support officers and digital
资源, such as Altmetric.com (nonacademic citations to academic publications) 和
Overton.io (policy documents mentioning researchers).

A range of sources may be used as evidence of nonacademic impacts. 这些包括
government publications, 法规, legislation, policy documents, parliamentary reports, sta-
tistics, white papers, medical treatment information sheets, clinical guidelines, patents, 标准,
book reviews, and news stories. 例如, an independent review of the role of metrics in
research assessment in REF 2014 and future exercises has suggested that “citations from online
‘grey’ literature seem to be an additional useful source of evidence of the wider impact of research,
but there do not seem to be any systematic studies of these” (Wilsdon et al., 2015, p. 38). 这些
sources of nonacademic impact evidence cannot be easily captured through scientific databases
and may need extensive searches on the web to locate, if they are online at all. Although there
have been attempts to propose methods to capture different types of nonacademic impacts based
on web citation searches (Kousha, 2019) and social media websites (Thelwall, Haustein et al.,
2013), these have tended to focus on assessing the availability of information rather than its utility
for evidencing nonacademic impacts. It is therefore important to identify sources commonly used
by academics to corroborate their claims of nonscholarly impacts in different subject areas. 这
may help researchers and university impact support officers to build their cases and may help
altmetrics providers or others to index the necessary sources.

Most previous studies of REF case studies have used text mining (例如, King’s College London
and Digital Science, 2015; Parks, Ioppolo et al., 2018) or content analysis (例如, Brook, 2018;
Wilkinson, 2019) to identify the types of impacts claimed by researchers, rather than the types of
evidence cited. 相比之下, 一 (not peer-reviewed) study has listed the 40 websites most cited in
impact case studies, broken down into four broad disciplinary groups (Digital Science, 2016) 但
did not analyze the cited URLs further. There seems to have been no large-scale assessment of
the types of URLs cited in “Sources to corroborate the impact” evidence sections. The current
study addresses this gap with a hybrid automatic and manual method to extract and classify the
most cited of these URLs for 6,637 downloadable REF 2014 impact case studies across all 36
UoAs. The same method can also be used for the systematic classification of URLs cited in future
impact case studies (例如, REF 2021) or other similar large-scale exercises with URL citations
outside the UK to understand their characteristics and disciplinary differences.

2. 背景

2.1. Text Mining Analyses of REF 2014 Impact Cases Studies

Several text mining studies have assessed the narrative sections of impact case studies. A large-
scale topic modeling of the REF 2014 impact case studies found subject differences in the

Quantitative Science Studies

865

D

w
n

A
d
e
d

F
r


H

t
t

p

:
/
/

d

r
e
C
t
.


t
.

/

e
d

q
s
s
/
A
r
t

C
e

p
d

F
/

/

/

/

2
3
8
6
4
1
9
7
0
7
7
2
q
s
s
_
A
_
0
0
1
4
5
p
d

/

.

F


y
G

e
s
t

t


n
0
7
S
e
p
e


e
r
2
0
2
3

Which types of online evidence show the nonacademic benefits of research?

types of impact reflected in them. 例如, in medical and biological sciences (Panel A),
关于 20% of the case studies related to Clinical guidance, whereas in the arts and humanities
(Panel D) the most common topic was “Media” (26%) (King’s College London and Digital
科学, 2015). Another text mining analysis of REF 2014 impact case studies used seven cat-
egories (人们, 经济的, Reach, Significance, Prestige, 健康, and Environment) 识别
quantitative indicators of impact, finding that sentences matching the categories People (35%)
and Economy (30%) were the most common (Parks et al., 2018). A further study classified the
words in two sections (“Summary of impact” and “Details of impact”) of the impact case studies
into six categories: 教育 (22.8%), Public engagement (17%), Environmental and energy
solutions (17.7%), 企业 (11.8%), 政策 (17.1%) and Clinical uses (13.7%). Differences be-
tween broad disciplines in types of impact were identified. 例如, in the Social Sciences
(Panel C) over one-third (34%) of the identified impact types were classified as Museums and
cultural heritage, whereas in the Life Sciences (Panel A) about half of the impact types were
categorized as Public health policy (Terämä, Smallman et al., 2016).

2.2. Content Analyses of REF 2014 Impact Cases Studies

Several content analyses have used human coders to classify aspects of the REF 2014 impact
case studies. In terms of the types of documents cited, most case studies corroborate impact
through at least one of Testimonial (80%) or Project report (78%) compared to Websites (30%)
or Media (26%) (休斯, Webber, & O’Regan, 2019).

The types of narrative impact claims found have differed greatly between disciplines. 一个
analysis of the REF impact case studies from one university faculty in Health and Applied
科学 (n = 18) found impacts on Policy (例如, policy reports and guidelines), Specific infor-
mation and advice (例如, online materials or toolkits), Research field (例如, clinical trial proce-
dures), and Patient interventions, protocols or standards of care (Wilkinson, 2019). 为了 162
impact case studies submitted to the Public Health, Health Services and Primary Care sub-
panel, three quarters (75%) had impacts on New or revised clinical guidelines and more than
half influenced International, national or local policy (54%) or changed Clinical or public
health practice (52%) (Greenhalgh & Fahy, 2015). 为了 194 REF 2014 impact case studies in
Business and Management, impact claims mentioned “Specific actions by practitioners or
policy-makers” (93%), “Specific and quantifiable results” (43%), “Indirect influence on the
public” (31%), or “Direct influence on the public” (4%). One study used a different approach
to select case studies to examine. Using selected Leadership, Governance, and Management
keywords, 1,309 relevant impact case studies were identified. Their most common impacts
were related to Government policy (52%), Training (47%), Impact on understanding (例如,
意识, attitude, or behaviors) (39%), and Strategy (例如, knowledge transfer, organizational
发展, or performance) (37%) (Morrow, Goreham, & Ross, 2017).

Different types of evidence can be presented to justify impact claims. Most of the 63 艺术
REF impact case studies contained evidence of the number of people who attended an event
(73%). Other common types of evidence were implementing policy or influencing policy-
创客, 行业, or other activities (60%), media coverage (52%), the number of events in
a festival or other relevant cultural program (52%), and benefit to artists, curators, 和文化
机构 (51%). The study argued that it is particularly challenging to provide evidence for
artistic impacts in the REF because it requires looking at the opinions or behaviors of a wide
range of audiences (Brook, 2018).

Some content analyses have examined the sources of evidence cited. 为了 46 cancer
trials impact case studies, 最多 (93%) of the supporting evidence was from either clinical

Quantitative Science Studies

866

D

w
n

A
d
e
d

F
r


H

t
t

p

:
/
/

d

r
e
C
t
.


t
.

/

e
d

q
s
s
/
A
r
t

C
e

p
d

F
/

/

/

/

2
3
8
6
4
1
9
7
0
7
7
2
q
s
s
_
A
_
0
0
1
4
5
p
d

/

.

F


y
G

e
s
t

t


n
0
7
S
e
p
e


e
r
2
0
2
3

Which types of online evidence show the nonacademic benefits of research?

guidelines (例如, National Institute for Health and Care Excellence, National Comprehensive
Cancer Network, or European Society for Medical Oncology) or trial research published by
medical journals (例如, The Lancet, Journal of Clinical Oncology, New England Journal of
药品) (Hanna, Gatting et al., 2020). Another analysis of 25 Library and Information
科学 (LIS) case studies found that the most frequent types of impact evidence identified
were about Cultural and heritage preservation, Historical archives, and Informing government
政策. The categories Workers, Policymakers, Companies/businesses, and Governments
were most frequently mentioned as research beneficiaries (Marcella, Lockerbie, & Bloice,
2016).

2.3. Alternative Sources for Assessing Wider Impacts of REF Case Studies

Alternative indicators might help to evidence the societal impacts of publications submitted as
research outputs or referenced in impact case studies within REF 2014. One study identified
mentions of social media platforms in REF 2014 impact case studies (all sections) 通过 42
条款, finding that blogs (52%), podcasts (21%), and YouTube (25.6%) were more commonly
mentioned in Panel D case studies (Arts and Humanities) than other main panels. 然而, 在
Panel A (药品, 健康, and Life Sciences) about a quarter (23.7%) of social media men-
tions were for YouTube, whereas Google Scholar (46%) was commonly referenced in Panel B
(Physical and Mathematical Sciences), despite being a primarily academic source (它是
sometimes used to evidence the credentials of the researcher or the scholarly uptake of the
research despite this not being assessed, 例如, https://impact.ref.ac.uk/casestudies/CaseStudy
.ASPX?Id=20952, https://impact.ref.ac.uk/casestudies/CaseStudy.aspx?Id=938). In Panel C
(社会科学), 博客 (关于 40%) were most common (约旦 & Carrigan, 2018).
Another investigation gathered six altmetric indicators (推特, 维基百科, Facebook, 博客,
消息, and policy-related documents) for publications (with DOIs) submitted either as REF
2014 research output or publications cited in impact case studies to support the underpinning
研究, finding that the publications referenced in impact case studies tended to be men-
tioned more commonly in altmetrics sources than were publications submitted as REF research
outputs (Bornmann, Haunschild, & Adams, 2019). Although an early study of REF 2014 案件
studies found no obvious association between altmetric scores and REF impact scores
(Ravenscroft, Liakata et al., 2017), a later investigation found a significant correlation between
altmetric scores and expert peer review ratings of nonacademic impacts for publications (和
DOIs) cited in the “Underpinning Research” sections of 1,469 REF 2014 impact case studies
submitted under main panel B (Wooldridge & 国王, 2019).

It seems that only one study has assessed the frequency of URL citations from all impact
case studies, reporting the 40 most cited websites (Digital Science, 2016, p. 30, Annex 4). 这
study did not classify URL types and did not use manual checking to exclude URLs mentioned
for other reasons (例如, archived copies of submitted REF impact case studies from https://万维网
.wiki.ed.ac.uk/ and https://apps.lse.ac.uk/). It also did not merge all relevant types of cited
URLs under one category (例如, URL citations from all newspapers or news agencies and
sources under the category “News and media”). 因此, this study has not given an overall pic-
ture of the types of URL cited in REF case studies.

3. RESEARCH QUESTIONS

The objective is to identify the main types of websites cited in REF 2014 impact case studies.
This will shed light on how academics in all fields use online sources differently to reflect

Quantitative Science Studies

867

D

w
n

A
d
e
d

F
r


H

t
t

p

:
/
/

d

r
e
C
t
.


t
.

/

e
d

q
s
s
/
A
r
t

C
e

p
d

F
/

/

/

/

2
3
8
6
4
1
9
7
0
7
7
2
q
s
s
_
A
_
0
0
1
4
5
p
d

/

.

F


y
G

e
s
t

t


n
0
7
S
e
p
e


e
r
2
0
2
3

Which types of online evidence show the nonacademic benefits of research?

the nonacademic impacts of their research. The following questions address different aspects
of this.

1. Which types of website (例如, news and media, governmernal, clinical guideline or so-
cial media) are cited in UK REF impact case studies to evidence research impacts?
2. Which websites (例如, BBC, UK Parliament, or NHS) are most frequently cited in the

impact case studies in all broad fields and all 36 Units of Assessment?
3. Are there disciplinary differences in the answers to the above questions?

4. 方法

4.1. The Data Set of REF 2014 Impact Case Study URL Citations

The metadata and full text of all 6,637 REF 2014 case studies1 were downloaded from the
main REF website2 in Excel format. Note that of the 6,975 of impact case studies submitted
to the REF2014, 6,637 case studies were downloadable from the REF database due to reuse
and licensing arrangements (https://impact.ref.ac.uk/casestudies/FAQ.aspx). A program was
designed and added to the free Webometric Analyst software (see https://lexiurl.wlv.ac.uk/,
“Extract URLs from Impact Case Studies” option under “Citations”) to automatically identify
and extract URL citations from these impact case studies. The term “URL citation” in this article
refers to mentions of URLs in the “Sources to corroborate the impact” section of impact case
学习 (见图 1). Only this section of case studies was used for analyses because researchers
“should list sufficient sources that could corroborate key claims made about the impact of the
unit’s research” in it, such as “reports, reviews, web links or other documented sources of infor-
mation in the public domain” (Research Excellence Framework, 2014, p. 54). The official case
study template had recommended an indicative maximum of 10 references in this section
(Research Excellence Framework, 2014). The software extracted 32,196 raw URLs from all
impact case studies based on the mentions of https://, https://, or www. anywhere in the ref-
erences to corroborate the impact.

4.1.1. Data cleaning

An initial check of the 32,196 extracted URLs showed that 1,929 were from the link shortening
websites tinyurl.com (1,055) or bit.ly (874). 因此, a program in Webometric Analyst was
used to identify the redirected URLs (see “Get redirected URLs” under the “Service” menu),
finding 1,871 (97%) of the ultimately cited URLs, which were used for analysis. 然而,
manual checks of URLs containing the terms “REF,” “impact,” or “case study” revealed that
1,059 of the extracted URLs (mostly from a few universities) were either archived copies of
submitted REF impact case studies (例如, https://ref2014.inf.ed.ac.uk/impact/) or other up-
loaded files or relevant information about the submitted impact case studies that were inac-
cessible (https://apps.lse.ac.uk/impact/download/file/1194) and hence were excluded from the
学习. To have more unique and reliable cited URLs for analysis, duplicate URLs in case stud-
ies were excluded (例如, see https://impact.ref.ac.uk/casestudies/CaseStudy.aspx?Id=38782),
giving a final total of 29,830 URLs from all 36 UoAs (data is available via https://doi.org/10
.6084/m9.figshare.14447295.v1).

1 The REF 2021 case studies will be online at some stage after 2021.
2 https://impact.ref.ac.uk/casestudies/

Quantitative Science Studies

868

D

w
n

A
d
e
d

F
r


H

t
t

p

:
/
/

d

r
e
C
t
.


t
.

/

e
d

q
s
s
/
A
r
t

C
e

p
d

F
/

/

/

/

2
3
8
6
4
1
9
7
0
7
7
2
q
s
s
_
A
_
0
0
1
4
5
p
d

/

.

F


y
G

e
s
t

t


n
0
7
S
e
p
e


e
r
2
0
2
3

Which types of online evidence show the nonacademic benefits of research?

数字 1.

Examples of URLs cited in an impact case study from the section “Sources to corroborate the impact.”

4.2. Semiautomatic Classification of the Websites of the Cited URLs

An initial URL classification scheme was developed by checking the most cited websites (IE。,
domain name or domain name ending) of the URL citations from all UoAs. 例如, 男人-
ual checks showed that many cited URLs from impact case studies in the arts, 人文学科, 和
social sciences were from news and media (例如, BBC News, the Guardian, and the Daily
Telegraph) or governmental websites (例如, UK government and UK parliament). In Clinical
and Applied Medicine, health care organizations (例如, the National Health Service) and clin-
ical guidelines (例如, NICE clinical guidelines) commonly documented research impacts. 在
Science and Engineering subject areas, commercial or business websites (例如, Rolls-Royce
or Apple) also frequently evidenced societal impacts. 尽管如此, the initial categories were
subsequently modified to include new types of websites identified during the classification
过程. 例如, only one general category was first assigned for social media websites,
but due to many cited URLs to online videos, it was split into two: Social Media and Blogs
(例如, 推特, Facebook, WordPress) and Video and Photo Sharing websites (例如, YouTube,
Vimeo, Flickr). 而且, in the arts and music a new category was added for artistic-related
websites that could not be classified elsewhere (例如, 音乐, film, 电视, galleries, and mu-
seums). The URLs cited by impact case studies were eventually classified into 18 类别
and eight broad areas, as shown below.

4.2.1.

Initial automatic classification of cited URL websites

Because it was not practical to manually classify the websites of all 29,830 cited URLs extracted
from the impact case studies, a systematic method was developed to automatically match the
domains of the cited URLs (例如, https://www.bbc.co.uk/news/health-18366437) against a man-
ually curated list of relevant websites in predefined categories (例如, bbc.co.uk in the category
News and media). The relevant URLs for each category were identified and extracted from dif-
ferent sources such as DMOZ—The Directory of the Web (https://dmoz-odp.org/), 维基百科
lists of websites (例如, https://en.wikipedia.org/wiki/List_of_intergovernmental_organizations),
and top visited websites listed by alexa.com in different categories (例如, https://www.alexa
.com/topsites/category/Top/Reference/Encyclopedias/). Additional searches were carried out
to identify reliable lists of websites for each category, such as the Webometrics Ranking of
World Universities (https://www.webometrics.info/) for university websites worldwide, a list

Quantitative Science Studies

869

D

w
n

A
d
e
d

F
r


H

t
t

p

:
/
/

d

r
e
C
t
.


t
.

/

e
d

q
s
s
/
A
r
t

C
e

p
d

F
/

/

/

/

2
3
8
6
4
1
9
7
0
7
7
2
q
s
s
_
A
_
0
0
1
4
5
p
d

.

/

F


y
G

e
s
t

t


n
0
7
S
e
p
e


e
r
2
0
2
3

Which types of online evidence show the nonacademic benefits of research?

of UK healthcare organizations published by the NHS (https://www.england.nhs.uk/tis/our
-members/), Ulrich’s Periodicals Directory (https://www.ulrichsweb.com/) for academic journal
websites, the Directory of Intellectual Property Offices (https://www.wipo.int/directory/en/urls
.jsp) for URL citations to patents, or National and International Clinical Guidelines
组织机构 (https://www.openclinical.org/guidelines.html/) for clinical guidelines. A pro-
gram was written and added to Webometric Analyst to match lists of domain names in one cat-
egory against the URLs from the impact case studies (see “Copy all URLs from long results files
that match list of domain names” option under “Utilities”).

The systematic classification of the cited URLs may be useful to assess how academics are
documenting research impacts in terms of the types of online sources but does not provide
contextual evidence about how the cited sources have been used—this needs manual content
分析.

1. 艺术: This broad category includes URLs of art-related websites, such as museums (例如,
the British Museum), galleries (Tate Modern or National Portrait Gallery), film and tele-
想象 (例如, the British Film Institute or British Academy of Film and Television Arts),
剧院 (例如, Royal National Theatre), 音乐 (例如, ukmusic.org), dance (例如, 这
National Dance CATs), or other relevant websites, such as the Royal Academy of
艺术 (royalacademy.org.uk), Arts Council England (artscouncil.org.uk), 互联网
Movie Database (imdb.com), or The Stage magazine (thestage.co.uk).

2. Governmental websites: URLs of governmental and parliamentary websites were clas-

sified in this broad category.
2A. UK government: This subcategory contains URLs of the main UK government
websites, such as the main GOV.UK website (www.gov.uk) 和英国
Government Web Archive (nationalarchives.gov.uk) and ministerial departments,
such as the Department of Health & Social Care (dh.gov.uk), 部门
教育 (education.gov.uk)3, and other local authorities, such as Birmingham
City Council (birmingham.gov.uk).

2乙. UK Parliament: URLs from impact case studies mentioning UK parliament and
other relevant parliamentary sources such as parliamentlive.tv were classified in
this category.

2C. Non-UK governments or parliaments: This includes any other URLs from non-UK
governmental or parliamentary websites, such as the U.S. Environmental
Protection Agency (epa.gov), the Parliament of Canada (parl.gc.ca) 和美国.
State Department (state.gov).

3. Organizational websites (其他): This includes URLs from organizations not classified

别处.
3A.

International organizations (including EU): This subclassification includes URLs
of international organizations such as the World Health Organization (who.int),
the United Nations (un.org), and the World Bank (worldbank.org). Citations to
the European Union website (europa.eu) have also been classified under this cat-
egory due to its intergovernmental structure for 27 European member states.

3 The internet addresses of most UK ministerial departments have not changed and are included under the

main gov.uk website.

Quantitative Science Studies

870

D

w
n

A
d
e
d

F
r


H

t
t

p

:
/
/

d

r
e
C
t
.


t
.

/

e
d

q
s
s
/
A
r
t

C
e

p
d

F
/

/

/

/

2
3
8
6
4
1
9
7
0
7
7
2
q
s
s
_
A
_
0
0
1
4
5
p
d

.

/

F


y
G

e
s
t

t


n
0
7
S
e
p
e


e
r
2
0
2
3

Which types of online evidence show the nonacademic benefits of research?

3乙. UK healthcare organizations: URLs of UK medical and healthcare organizations,
charities or nonprofit organizations have been classified under this category, 在-
cluding the UK National Health Service (国民医疗服务体系) (nhs.uk), Cancer Research UK
(cancerresearchuk.org) and the British Diabetic Association (diabetes.org.uk).
例如, 558 URL citations targeted NHS websites.

3C. UK organizations (nonhealthcare): This contains URLs of other UK organizations,
charities or nonprofit organizations, such as the Royal Society (royalsociety.org),
English Heritage (english-heritage.org.uk) or the British Council (britishcouncil.org).
3d. Other organizations: This contains URLs of other non-UK organizational websites
not classified above (3A, 3乙, and 3c) or in other classes (1,2, 4–8), 例如
Organization for Security and Co-operation in Europe (osce.org) 和
American Library Association (ala.org).

4. News and media: This includes the URLs of newspapers (例如, guardian.co.uk, telegraph
.co.uk), news agencies (例如, bbc.co.uk, reuters.com) and other news sources (例如,
businesswire.com, channel4.com).

5. Commercial and business: URLs of commercial, 产品, or technology websites were
classified into this category, including apple.com, rolls-royce.com, oracle.com and
tripadvisor.co.uk.

6. Scholarly publications: This category reflects URLs in scholarly or research communi-

cation systems.
6A.

Journals and conferences: This includes URLs of academic journals or conference
proceedings from the main academic publishers (例如, nature.com, onlinelibrary
.wiley.com, sciencedirect.com, oxfordjournals.org, and ieeexplore.ieee.org) 或者
databases and digital libraries of publications (例如, biomedcentral.com, ncbi
.nlm.nih.gov/pubmed, arxiv.org, scholar.google.com).
Books and encyclopedia: This reflects URL citations to books (例如, amazon.co.uk,
books.google.co.uk, goodreads.com, books.nap.edu, routledge.com) and ency-
clopedias (例如, wikipedia.org, britannica.com, surgeryencyclopedia.com).
Patents and standards: This includes patent, 标准, and intellectual property da-
tabases and websites such as Google Patents (patents.google.com), 世界
Intellectual Property Organization (www.wipo.int), 国际
Organization for Standardization (www.iso.org), the UK Intellectual Property
Office (ipo.gov.uk), 和美国. Patent and Trademark Office (www.uspto.gov).

6乙.

6C.

6d. Clinical guidelines: URL citations to clinical practice guidelines were classified in
this category, such as the National Institute for Health and Care Excellence (NICE)
(www.nice.org.uk), ClinicalTrials.gov (clinicaltrials.gov), 和美国. Food and
Drug Administration (www.fda.gov).

7. Universities: URLs of university websites not classified elsewhere are included in this

类别.
7A. UK universities: This includes URL citations for UK university websites, 例如
the University of Birmingham (birmingham.ac.uk). This may include research pro-
ject URLs (例如, https://projects.cs.kent.ac.uk/projects/kroc/trac/), research centers
(例如, https://www.ucl.ac.uk/centre-for-advanced-biomedical-imaging/), and press
releases (例如, https://www.bristol.ac.uk/news/2013/9200.html).

7乙. Non-UK universities: This includes URLs of universities and academic institutions
outside the UK, such as Harvard University (harvard.edu) and the University of
多伦多 (utoronto.ca).

Quantitative Science Studies

871

D

w
n

A
d
e
d

F
r


H

t
t

p

:
/
/

d

r
e
C
t
.


t
.

/

e
d

q
s
s
/
A
r
t

C
e

p
d

F
/

/

/

/

2
3
8
6
4
1
9
7
0
7
7
2
q
s
s
_
A
_
0
0
1
4
5
p
d

.

/

F


y
G

e
s
t

t


n
0
7
S
e
p
e


e
r
2
0
2
3

Which types of online evidence show the nonacademic benefits of research?

8. Social networking websites: This contains URL citations from impact case studies to

blogs and social networking websites.
8A.

Social media and blogs: This includes URLs of social media websites (例如,
Facebook, 推特, Tumblr, LinkedIn) and blogs (例如, WordPress and Blogspot).
Video and photo sharing websites: This includes URLs of video or photo sharing
websites.

8乙.

4.2.2. Manual checking of automatically classified URL websites

To make sure that the URL citations were reasonably classified into the predefined categories
by the above automatic domain name–based method, 这 20 most cited URLs in each of the
36 UoAs from the initial systematic classification were manually checked and reclassified if
必要的 (20 × 36 = 720 URLs). 例如, URLs for the National Audit Office (nao.org.uk)
were first classified as UK organization URL citations, but the manual checks revealed that this
organization is part of the UK government sector. The manual checking was based on visiting
the websites and reading relevant sections, including “about us,” “our mission,” or “contact
我们,” if necessary. 尽管如此, 关于 12% (3,545 在......之外 29,830) of the URLs cited in the case
studies were not classified even after the manual checking phase, although not classified cited
URLs were more common in UoA 12—Aeronautical, Mechanical, Chemical and
Manufacturing Engineering (21.4%), UoA 15—General Engineering (19.5%), UoA
11—Computer Science and Informatics (19.2%), and UoA 13—Electrical and Electronic
Engineering (17.9%) than UoA 2—Public Health, Health Services and Primary (5.6%), UoA
22—Social Work and Social Policy (5.7%), UoA 18—Economics and Econometrics (6.0%),
and UoA 20—Law (6.2%). This is because in the engineering fields researchers may use a
range of different industries, 企业, or manufacturing companies as evidence of nonaca-
demic impacts. 这 3,545 not classified URLs were from 3,028 different websites, suggesting
that they were less frequently cited in impact case studies. 例如, the most common not

桌子 1.

Examples of reclassified websites based on manual checks of the top 20 most cited URLs in the impact case studies

姓名
Public Health England
British Medical Association

People’s Trust for

Endangered Species

European Society of
人类遗传学

URL domain name

www.hpa.org.uk
www.bma.org.uk

www.ptes.org

Initial classification
UK healthcare organization
UK organization

(nonhealthcare)
Other organizations

Reclassification

UK Government websites
UK healthcare organization

UK organization (nonhealthcare)

www.eshg.org

Other organizations

International organization

D

w
n

A
d
e
d

F
r


H

t
t

p

:
/
/

d

r
e
C
t
.


t
.

/

e
d

q
s
s
/
A
r
t

C
e

p
d

F
/

/

/

/

2
3
8
6
4
1
9
7
0
7
7
2
q
s
s
_
A
_
0
0
1
4
5
p
d

/

.

F


y
G

e
s
t

t


n
0
7
S
e
p
e


e
r
2
0
2
3

Royal Exchange Theatre

www.royalexchange.co.uk

Commercial, 工业的

Farmers Weekly

www.fwi.co.uk

The Royal Ballet School

www.royalballetschool.org.uk

The Nuffield Council

www.nuffieldbioethics.org

on Bioethics

The Business of Photonics
Royal Society of Arts
Internet Engineering

Task Force

optics.org
www.thersa.org
www.ietf.org

和商业

Commercial, 工业的

和商业
UK organization

(nonhealthcare)
Other organizations

Other organizations
Other organizations
Other organizations

(including EU)
Artistic (例如, 音乐,
dance and film)
News and media

Artistic (例如, 音乐,
dance, 和电影)

UK healthcare organization

News and media
UK organization (nonhealthcare)
International organization

(including EU)

Quantitative Science Studies

872

Which types of online evidence show the nonacademic benefits of research?

classified URLs were docs.google.com (cited nine times), thefreelibrary.com (cited seven
次), and scribd.com (cited six times). 桌子 1 gives examples of website reclassifications
from this stage.

Broad subject
艺术

桌子 2.

Broad subject groupings for the 36 REF UoAs

Art and Design History, 实践, and Theory
音乐, Drama, Dance, and Performing Arts

Unit of assessment

Biological and Agricultural Sciences

农业, Veterinary, and Food Science
Biological Sciences

Engineering and Computer Science

Hard Sciences

人文学科

Medical Sciences and Healthcare

社会科学

Aeronautical, Mechanical, Chemical, and Manufacturing Engineering
Civil and Construction Engineering
Computer Science and Informatics
Electrical and Electronic Engineering, Metallurgy, and Materials
General Engineering

Chemistry
Earth Systems and Environmental Sciences
Physics
Mathematical Sciences

Anthropology and Development Studies
Area Studies
Classics
沟通, Cultural and Media Studies, Library and Information Management
English Language and Literature
历史
Law
Modern Languages and Linguistics
Philosophy
Theology and Religious Studies

Allied Health Professions, Dentistry, Nursing, and Pharmacy
Clinical Medicine
心理学, Psychiatry, and Neuroscience
公共卫生, Health Services and Primary Care

Architecture, Built Environment, and Planning
Business and Management Studies
Economics and Econometrics
教育
Geography, Environmental Studies, and Archaeology
Politics and International Studies
Social Work and Social Policy
Sociology
Sport and Exercise Sciences, Leisure, and Tourism

Quantitative Science Studies

873

D

w
n

A
d
e
d

F
r


H

t
t

p

:
/
/

d

r
e
C
t
.


t
.

/

e
d

q
s
s
/
A
r
t

C
e

p
d

F
/

/

/

/

2
3
8
6
4
1
9
7
0
7
7
2
q
s
s
_
A
_
0
0
1
4
5
p
d

.

/

F


y
G

e
s
t

t


n
0
7
S
e
p
e


e
r
2
0
2
3

Which types of online evidence show the nonacademic benefits of research?

4.2.3. Broad subject classifications

这 36 REF UoAs were combined into seven broad subjects for disciplinary analyses (桌子 2).
The REF classification of Units of Assessments across four main panels (A–D) was modified to
represent results within more uniform broad subject areas. 例如, all artistic fields (在-
cluding Art and Design: 历史, Practice and Theory and Music, Drama, Dance, 和
Performing Arts) and related humanities (例如, English Language and Literature, 历史,
Philosophy, and Law) in Panel D were combined to form the broad subject categories Arts
and Humanities, 分别. 相似地, all relevant engineering, hard science subjects and
medical relevant fields were merged to represent Engineering and Computer Science, Hard
科学, and Medical Sciences and Healthcare.

5. 结果

Considering the indicative maximum of 10 references to support wider impacts of research in
the REF 2014 template (Research Excellence Framework, 2014), it is unsurprising that the av-
erage number of URL citations is less than seven for all UoAs (数字 2). 尽管如此, in pub-
lic health and other allied health professions, impact case studies tended to cite on average
more online sources (6.0 到 6.1) than in other fields, such as most engineering subjects (2.6 到
3.8). This may reflect health information being increasingly public and online, in contrast to
engineering research innovation documentation.

D

w
n

A
d
e
d

F
r


H

t
t

p

:
/
/

d

r
e
C
t
.


t
.

/

e
d

q
s
s
/
A
r
t

C
e

p
d

F
/

/

/

/

2
3
8
6
4
1
9
7
0
7
7
2
q
s
s
_
A
_
0
0
1
4
5
p
d

/

.

F


y
G

e
s
t

t


n
0
7
S
e
p
e


e
r
2
0
2
3

数字 2. Average number of unique cited URLs in the UK REF impact case studies in 2014 across
36 UoAs, after data cleaning.

Quantitative Science Studies

874

Which types of online evidence show the nonacademic benefits of research?

数字 3. Percentage shares of the cited URLs in the impact case studies based on eight broad categories across seven fields.

D

w
n

A
d
e
d

F
r


H

t
t

p

:
/
/

d

r
e
C
t
.


t
.

/

e
d

q
s
s
/
A
r
t

C
e

p
d

F
/

/

/

/

2
3
8
6
4
1
9
7
0
7
7
2
q
s
s
_
A
_
0
0
1
4
5
p
d

.

/

F


y
G

e
s
t

t


n
0
7
S
e
p
e


e
r
2
0
2
3

数字 4. Percentage shares of specific types of cited websites in the impact case studies based on 18 categories across seven fields.

Quantitative Science Studies

875

Which types of online evidence show the nonacademic benefits of research?

About a third of the cited URLs in the impact case studies were for other organizational
websites (30%), with many others directed to news and media (19%) and government
(17%) websites. 尽管如此, there are clear disciplinary differences (数字 3). 例如,
in Medical Sciences and Biological and Agricultural Sciences, URL citations of organizational
websites were more numerous (40% 和 32%, 分别), whereas in the Arts and
人文学科, citations to news and media (28% 和 26% 分别) were more common.
In Engineering and Computer Science, more citations to commercial and business websites
(26%) were identified. This confirms that broad fields tend to cite different types of online
sources to evidence the impact of their research.

数字 4 gives more fine-grained details about the types of websites cited in the impact case
学习. 例如, artistic contents (音乐, Drama, Dance, and Performing Arts) were com-
monly cited in the Arts impact case studies (14.2%) and citations to the UK government and
the UK parliament were most common in Social Science impact case studies (20.3% 和
6.5%). UK healthcare organizations (例如, 国民医疗服务体系) and clinical guidelines (例如, NICE) were more
cited in Medical Sciences and Healthcare impact case studies (15% 和 12% 分别)
than in other fields. 也许令人惊讶, citations to social media websites were relatively
common in Humanities (7%) and Arts (5%) impact case studies.

这 10 websites with the most citations from all impact case studies also vary in prevalence
between broad subjects (数字 5). In the Arts and Humanities, 6% 和 5% of citations from

D

w
n

A
d
e
d

F
r


H

t
t

p

:
/
/

d

r
e
C
t
.


t
.

/

e
d

q
s
s
/
A
r
t

C
e

p
d

F
/

/

/

/

2
3
8
6
4
1
9
7
0
7
7
2
q
s
s
_
A
_
0
0
1
4
5
p
d

.

/

F


y
G

e
s
t

t


n
0
7
S
e
p
e


e
r
2
0
2
3

数字 5. Percentage shares of the top 10 cited websites in all impact case studies across seven fields.

Quantitative Science Studies

876

Which types of online evidence show the nonacademic benefits of research?

the impact case studies were to the BBC website and 3% 和 4% to the Guardian. YouTube
videos were also more cited in the Arts (3.5%) and Humanities (2.8%) than in other subjects. 在
社会科学, citations to UK parliament (5%) and in Medical Sciences and Healthcare
citations to the UK NHS (6.7%) and the National Institute for Health and Care Excellence
(5.4%) were more common. This suggests that disciplinary-relevant online sources can be
used to reflect the societal impacts of research. Tables S1–S4 in the Supplementary
Information report similar information for the 36 UoAs.

6. 讨论

The results show, 首次, the main types of website cited in all REF impact case stud-
是的. The method can be used to identify the most common online sources provided for societal
impact claims in the new REF 2021, informing evaluators about the norms of societal benefits
of research across their own field judgements. This might be more useful in the arts, 人类-
实体, 和社会科学, where academics may use nonstandard online sources such as news
来源, multimedia information, and social media for evidencing research impact. 因为
many online sources used as evidence of nonacademic impacts have not been covered by
altmetric platforms, future tools may capture and analyze societal impacts from wider online
sources of impact. The results show the existence of substantial disciplinary differences in the
websites cited, as might have been suspected from prior evidence of disciplinary differences in
the types of impact claim made (例如, Hanna et al., 2020; Marcella et al., 2016). 有
many ways in which academics may cite online evidence of the societal impacts of their re-
search in REF impact case studies. 在这个部分, different examples of the most frequently
cited URLs are given in across subjects to provide richer insights into the main quantitative
findings above.

In main Panel A (药品, health and life sciences), the most common types of claimed
evidence were from clinical guidelines or trials (nice.org.uk or clinicaltrials.gov) 其次是
the World Health Organization (who.int), the UK NHS (nhs.uk), and the UK government
(defra.gov.uk, gov.uk). 例如, in Public Health, Health Services, and Primary Care
about a third (32.9%), in Clinical Medicine over a quarter (26.4%) and in Allied Health
Professions, Dentistry, Nursing, and Pharmacy” less than a fifth (18.1%) of the claimed on-
line evidence about benefits or wider impact of submitted impact case studies were from the
above web sources. In Clinical Medicine, a wide range of clinical documents were used to
demonstrate the benefits of medical research, such as changes in drug labels and guidelines
(fda.gov/drugs/postmarket-drug-safety-information-patients-and-providers/information
-abacavir-marketed-ziagen-and-abacavir-containing-medications), use in NICE clinical
guidelines as treatment evidence (https://www.nice.org.uk/guidance/cg71) or cited by
WHO guidelines for health policy (who.int/nutrition/publications/guidelines/potassium
_intake_printversion.pdf).

In Agriculture, Veterinary, and Food Science, 8.5% of the URLs were from European Union
websites, including the European Food Safety Authority (efsa.europa.eu), the European
Medicines Agency (ema.europa.eu), and other relevant EU sections of food, farming, and fish-
eries websites. In Biological Sciences, some cited URLs (4%) were from ClinicalTrials.gov,
where many mentioned clinical trials for the safety or efficacy of new drugs or treatments
(例如, clinicaltrials.gov/ct2/show/nct01844986 and clinicaltrials.gov/ct2/show/nct01712074).

In main Panel B (Physical sciences, engineering and mathematics), a combination of news,
online videos, and specialized websites were frequently used as impact evidence in case stud-
是的. 例如, in Earth Systems and Environmental Sciences 5% were from environment,

Quantitative Science Studies

877

D

w
n

A
d
e
d

F
r


H

t
t

p

:
/
/

d

r
e
C
t
.


t
.

/

e
d

q
s
s
/
A
r
t

C
e

p
d

F
/

/

/

/

2
3
8
6
4
1
9
7
0
7
7
2
q
s
s
_
A
_
0
0
1
4
5
p
d

/

.

F


y
G

e
s
t

t


n
0
7
S
e
p
e


e
r
2
0
2
3

Which types of online evidence show the nonacademic benefits of research?

marine, 食物, or fishery sections of the European Union website (europa.eu). 相似地, in Civil
and Construction Engineering, and Chemistry about 3% and in General Engineering about 2%
of the cited URLs were from the Office of Rail and Road, lika Technologies (a pioneer in solid
state battery technology) and Rolls-Royce, 分别. Mathematics (3.5%) and Physics (3%)
cited URLs were of YouTube videos, such as a 3D print of a mathematical sculpture (youtube
.com/watch?v=MyUfAs30yZk), the origin of the Handbook of Mathematical Functions
(youtube.com/watch?v=Exf02R1FnXY), a TEDx talk about the universe (youtube.com/watch
?v=oCaR1uE3OV8), and a video about a new type of LCD display screen (youtube.com
/手表?v=DdhYPL87LZQ). In Electrical and Electronic Engineering, Metallurgy, 和
Materials (5%) and in Computer Science and Informatics (4%) BBC news URLs were cited,
such as for the development of Europe’s Galileo satellite-navigation system (bbc.co.uk/news
/science-environment-17755205), “musical prescriptions” for patients (bbc.co.uk/news/uk
-scotland-glasgow-west-11233452), and new software to help children with communication
problems speak better (news.bbc.co.uk/1/ hi/ health/8084422.stm). In Aeronautical,
Mechanical, Chemical, and Manufacturing Engineering more than a third (38%) of cited
URLs in case studies were to technological companies, 标准, or patents.

In Main Panel C (Social sciences), the most commonly cited URLs in the impact case stud-
ies were for the UK parliament (including the Scottish, Welsh, and Northern Ireland versions)
and UK government websites. 例如, in Politics and International Studies 12%, 姻亲
10.2%, and in Business and Management Studies 6.7% of the URLs were for UK parliament
websites. In Education, Sociology, and Economics and Econometrics, reports by the House of
Commons about “Transforming education outside the classroom” (publications.parliament.uk
/pa/cm200910/cmselect/cmchilsch/418/418.pdf), “Domestic violence, forced marriage and
‘honour’-based violence” (publications.parliament.uk/pa/cm200708/cmselect/cmhaff/263
/263i.pdf), and “Principles of tax policy” (publications.parliament.uk/pa/cm201011/cmselect
/cmtreasy/753/753.pdf) were cited as influences on policy-making. 而且, UK government
websites (ending with domain gov.uk) were also highly cited in most social science fields,
such as in Architecture, Built Environment, and Planning (26.6%), Economics and
Econometrics (24.9%) and Social Work and Social Policy (22.7%). The cited URLs in the impact
case studies from UK government websites may include a range of different contents, 例如
press releases, 报告, 法规, 统计数据, 政策, guidelines, 分析, white papers, par-
liamentary transcripts, or other publications.

In Main Panel D (Arts and humanities), the most frequently cited sources were news
故事, online videos, and social media and blogs. 例如, in History, Theology and
Religious Studies, and English Language and Literature, about a quarter of the cited URLs were
for news and media, including the BBC (25.9%), Guardian (24.5%) and Daily Telegraph
(22.7%) and in other subjects ranging between 14.4% in Classics to 20% in Art and Design
历史, 实践, and Theory. 例如, in English Language and Literature and History,
many book reviews were mentioned in the impact studies, mostly published by the Guardian
(theguardian.com/books/2011/dec/07/britains-empire-richard-gott-review), Daily Telegraph
(telegraph.co.uk/culture/books/books-life/7087391/The-Long-Song-by-Andrea-Levy-review
.html), and Independent (independent.co.uk/arts-entertainment/books/reviews/the-arabs-and
-the-holocaust-the-arabisraeli-war-of-narratives-by-gilbert-achcar-2305801.html). The most
cited URLs of YouTube videos across all REF subjects were from Classics (7.7%), 音乐,
Drama, Dance, and Performing Arts (5.6%), Area Studies (5.3%) and Modern Languages
and Linguistics (4.9%), indicating that in these areas multimedia information is important for
evidencing impacts. Examples of the importance of multimedia include a short film about
Greek Comedy in Classics (youtube.com/watch?v=H-BvMbfkxcc), an audio lecture in

Quantitative Science Studies

878

D

w
n

A
d
e
d

F
r


H

t
t

p

:
/
/

d

r
e
C
t
.


t
.

/

e
d

q
s
s
/
A
r
t

C
e

p
d

F
/

/

/

/

2
3
8
6
4
1
9
7
0
7
7
2
q
s
s
_
A
_
0
0
1
4
5
p
d

.

/

F


y
G

e
s
t

t


n
0
7
S
e
p
e


e
r
2
0
2
3

Which types of online evidence show the nonacademic benefits of research?

Philosophy (bbc.co.uk/programmes/b00xnxl4), and a picture of international festival partici-
pants (festivalpoesianicaragua.com/wp-content/uploads/poesiagranada-319.jpg), all evidenc-
ing humanities research impacts. In Modern Languages and Linguistics (7.7%) and English
Language and Literature (7.2%), social media sites ( WordPress, Blogspot, Facebook and
推特) were cited relatively more often.

In Art and Design History, 实践, and Theory many impact case studies cited information
from galleries or museums, such as information an exhibition (tate.org.uk/whats-on/tate-britain
/exhibition/turner-and-masters) or an artistic object (tate.org.uk/art/research-publications
/gaudier-brzeska-wrestlers) in the Tate Modern. Other relevant artistic information was also
引用, such as a review of a painting exhibition (youtube.com/watch?v=mlsn4Za5-as), and spe-
cific galleries or exhibitions (例如, whitechapelgallery.org/exhibitions/john-latham-anarchive/).
In Music, Drama, Dance, and Performing Arts, 6% of the cited URLs were for online videos,
such as a theatre play preview (youtube.com/watch?v=br9tafybBXM), music performances at a
festival (youtube.com/watch?v=-Z6H8jpd1fU), an interview with a Professor of Music
(youtube.com/watch?v=D1EUurZ4s98), a commercial racing game soundtrack, Need for
Speed Shift 2: Unleashed (youtube.com/watch?v=mB6X3LGIT30), and a computer-generated
light and sound music performance (youtube.com/watch?v=cysjxHzCoh0).

D

w
n

A
d
e
d

F
r


H

t
t

p

:
/
/

d

r
e
C
t
.


t
.

/

e
d

q
s
s
/
A
r
t

C
e

p
d

F
/

/

/

/

2
3
8
6
4
1
9
7
0
7
7
2
q
s
s
_
A
_
0
0
1
4
5
p
d

/

.

F


y
G

e
s
t

t


n
0
7
S
e
p
e


e
r
2
0
2
3

6.1. Limitations

This study has several limitations. The cited URLs studied here only include online sources to
corroborate impact. This ignores all cited offline or unpublished sources (例如, 字母, emails,
报告, statements) that may give different insights into the types of evidence used. 因为
there is no practical way to classify a large number of URLs, an ad hoc method was used to
categorize the broad type of the cited URLs. 虽然 720 most cited URLs from the initial
systematic classification were manually checked and reclassified when necessary, 关于 12%
of the URLs cited in the case studies were not classified. This was particularly common in
Engineering and Computer Science, where a range of different commercial websites could
be claimed as evidence of nonacademic benefits of engineering research. We could not find
a practical method to classify the websites cited in the impact case studies, and these less
common URLs may well give a different perspective. 而且, this study did not assess
the contexts or motivations for citing URLs and hence it is not clear how the online sources
were used by the impact case studies. 例如, a news story cited by a clinical medicine
case study might reflect a publicity claim (IE。, the news story is the impact) or may evidence
uptake of an invention by a company (IE。, the news story reports the impact). 最后, the clas-
sification of the 20 most cited URLs in each of the 36 UoAs was not crosschecked by a second
classifier and hence there could be disagreement about the characteristics of some websites
such as the National Audit Office (nao.org.uk) which is an independent public spending
watchdog scrutinizing public spending for parliament.

7. 结论

The results show that a wide range of nonacademic online sources have been used to corrob-
orate the benefits of research, including news stories, online videos, government publications,
议会记录, and social media websites, although there were disciplinary differ-
恩塞斯. In answer to the first research question, in Medical and Health Sciences, clinical guide-
lines and UK healthcare organizations were most frequently cited, whereas in the Arts and
人文学科, news and media, and in Social Sciences, government and parliamentary

Quantitative Science Studies

879

Which types of online evidence show the nonacademic benefits of research?

publications were more commonly mentioned in impact case studies. In answer to the second
and third research questions, there are large disciplinary differences in the websites most com-
monly cited in impact case studies across the 36 REF UoAs (Supplementary Information, Tables
S1–S4). 例如, in Clinical Medicine the NICE clinical guidelines (8.3%), in Public Health,
Health Services, and Primary Care WHO (10.5%), in Agriculture, Veterinary, and Food Science
(8.5%) and Economics and Econometrics (5.9%) European Union websites were more frequent-
ly mentioned in case studies. 相似地, in History many URLs were from BBC News (9.7%) 和
the Guardian (4%), whereas in Law the UK Parliament (10.2%) and Ministry of Justice (5.4%)
were more frequently mentioned. In Chemistry, Physics, and Mathematical Sciences about 3%
of the URL citations were to YouTube, indicating widespread disciplinary differences in the on-
line sources used to evidence the wider impacts of research.

In terms of practical implications for people compiling impact case studies or making impact
claims for funding applications, promotions, grant summary reports, or other governmental pur-
姿势, identifying the common sources that academics use to evidence their nonacademic im-
pacts may help direct them to possible solutions. 相似地, the results may help
scientometricians, research impact officers and librarians to develop strategies and tools for
capturing and analyzing societal impacts from known sources, including those that are not cov-
ered in the current altmetrics platforms (例如, Altmetric.com, Plum Analytics, Dimensions,
Overton). 例如, this analysis of URLs cited in REF 2014 impact studies showed that trials
and guidelines were the most common online sources used in medical sciences to corroborate
wider research impacts of research. 因此, future altmetric tools and methods can be devel-
oped to identify mentions of research in the references of all major national and international
online clinical trials or medical guideline sources as well as trial research published by medical
journals (例如, the journals Trials, The American Journal of Clinical Oncology Cancer Clinical
试炼, Contemporary Clinical Trials, or Controlled Clinical Trials). 相似地, developing tools
to extract mentions of research (including DOIs) from news sources, such as BBC News, 哪个
was the most common source for arts and humanities impact evidence, could be useful to iden-
tify research results that could be of interest to the public.

作者贡献
Kayvan Kousha: 概念化; 调查; 方法; Writing—original draft;
Writing—review & 编辑. Mike Thelwall: 软件; Writing—review & 编辑. Mahshid
Abdoli: 调查; Writing—review & 编辑.

COMPETING INTERESTS

The authors have no competing interests.

资金信息

No funding was provided for this research.

DATA AVAILABILITY

The shared data provides categorization of 29,830 cited URLs in the REF 2014 impact case
studies across 36 UoAs and is available via https://doi.org/10.6084/m9.figshare.14447295.v1.
The classification scheme of cited URLs might be used for analyzing the REF 2021 impact case
studies when they are available online.

Quantitative Science Studies

880

D

w
n

A
d
e
d

F
r


H

t
t

p

:
/
/

d

r
e
C
t
.


t
.

/

e
d

q
s
s
/
A
r
t

C
e

p
d

F
/

/

/

/

2
3
8
6
4
1
9
7
0
7
7
2
q
s
s
_
A
_
0
0
1
4
5
p
d

.

/

F


y
G

e
s
t

t


n
0
7
S
e
p
e


e
r
2
0
2
3

Which types of online evidence show the nonacademic benefits of research?

参考

Bornmann, L。, Haunschild, R。, & Adams, J. (2019). Do altmetrics
assess societal impact in a comparable way to case studies? 一个
empirical test of the convergent validity of altmetrics based on
data from the UK Research Excellence Framework (REF ).
Journal of Informetrics, 13(1), 325–340. https://doi.org/10.1016
/j.joi.2019.01.008

Brook, L. (2018). Evidencing impact from art research: Analysis of
impact case studies from the REF 2014. Journal of Arts
管理, Law, 与社会, 48(1), 57–69. https://doi.org/10
.1080/10632921.2017.1386148

Digital Science. (2016). Publication patterns in research underpin-
ning impact in REF2014: A report to HEFCE by Digital Science.
伦敦, 英国: Digital Science.

Dinsmore, A。, 艾伦, L。, & Dolby, K. (2014). Alternative perspectives
on impact: The potential of ALMs and altmetrics to inform funders
about research impact. PLOS Biology, 12(11), e1002003. https://
doi.org/10.1371/journal.pbio.1002003, 考研: 25423184
Greenhalgh, T。, & Fahy, 氮. (2015). Research impact in the
community-based health sciences: An analysis of 162 case studies
从 2014 UK Research Excellence Framework. BMC
药品, 13(1), 232. https://doi.org/10.1186/s12916-015-0467-4,
考研: 26391108

格思里, S。, Krapels, J。, Lichten, C. A。, & Wooding, S. (2017). 100
metrics to assess and communicate the value of biomedical re-
搜索: An ideas book. Rand Health Quarterly, 6(4), 14. https://
doi.org/10.7249/RR1606, 考研: 28983437

Hanna, C. R。, Gatting, L. P。, Boyd, K. A。, Robb, K. A。, & 琼斯, 右. J.
(2020). Evidencing the impact of cancer trials: Insights from the
2014 UK Research Excellence Framework. 试炼, 21, 1–13.
https://doi.org/10.1186/s13063-020-04425-9

休斯, T。, Webber, D ., & O’Regan, 氮. (2019). Achieving wider
impact in business and management: Analysing the case studies
from REF 2014. Studies in Higher Education, 44(4), 628–642.
https://doi.org/10.1080/03075079.2017.1393059

约旦, K., & Carrigan, 中号. (2018). How was social media cited in
2014 REF Impact Case Studies? Impact of Social Sciences blog.
https://eprints.lse.ac.uk/90751/1/Jordan_How-was-social-media
_Author.pdf

King’s College London and Digital Science. (2015). The nature, 规模
and beneficiaries of research impact: An initial analysis of Research
Excellence Framework (REF) 2014 impact case studies. HEFCE.
https://www.kcl.ac.uk/policy-institute/assets/ref-impact.pdf

Kousha, K. (2019). Web citation indicators for wider impact assess-
ment of articles. In W. Glänzel, H. F. Moed, U. Schmoch, & 中号.
Thelwall (编辑。), Springer handbook of science and technology in-
dicators. 占婆: 施普林格. https://doi.org/10.1007/978-3-030
-02511-3_31

Kuruvilla, S。, Mays, N。, Pleasant, A。, & 沃尔特, G. (2006). Describing
the impact of health research: A Research Impact Framework.
BMC Health Services Research, 6(1), 1–18. https://doi.org/10
.1186/1472-6963-6-134, 考研: 17049092

Marcella, R。, Lockerbie, H。, & Bloice, L. (2016). Beyond REF 2014:
The impact of impact assessment on the future of information
研究. Journal of Information Science, 42(3), 369–385. https://
doi.org/10.1177/0165551516636291

Morrow, 乙. M。, Goreham, H。, & Ross, F. (2017). Exploring research
impact in the assessment of leadership, governance and manage-
ment research. 评估, 23(4), 407–431. https://doi.org/10
.1177/1356389017730726

Oliver, K., Innvar, S。, Lorenc, T。, Woodman, J。, & 托马斯, J. (2014).
A systematic review of barriers to and facilitators of the use of
evidence by policymakers. BMC Health Services Research, 14(1),
1–12. https://doi.org/10.1186/1472-6963-14-2, 考研:
24383766

Parks, S。, Ioppolo, B., Stepanek, M。, & Gunashekar, S. (2018).
Guidance for standardising quantitative indicators of impact
within REF case studies. Santa Monica and Cambridge: RAND
欧洲. https://www.rand.org/content/dam/rand/pubs/research
_reports/RR2400/RR2463/RAND_RR2463.pdf. https://doi.org/10
.7249/RR2463

Ravenscroft, J。, Liakata, M。, Clare, A。, & Duma, D. (2017).
Measuring scientific impact beyond academia: An assessment
of existing impact metrics and proposed improvements. PLOS
ONE, 12(3), e0173152. https://doi.org/10.1371/journal.pone
.0173152, 考研: 28278243

Resea r ch Excellence F rame work.

(2014). R E F 20 14 :
framework and guidance on submissions. https://
Assessment
w w w . r e f . a c . u k / 2 0 1 4 / m e d i a / r e f / c o n t e n t / p u b
/assessmentframeworkandguidanceonsubmissions/GOS
%20including%20addendum.pdf

Research Excellence Framework. (2019). REF Guidance, 2019.
Guidance on submissions to REF 2021. https://www.ref.ac.uk
/publications/guidance-on-submissions-201901

Terämä, E., Smallman, M。, Lock, S. J。, 约翰逊, C。, & Austwick,
中号. Z. (2016). Beyond academia—Interrogating research impact
in the research excellence framework. PLOS ONE, 11(12),
e0168533. https://doi.org/10.1371/journal.pone.0168533,
考研: 27997599

Thelwall, M。, Haustein, S。, Larivière, 五、, & Sugimoto, C. 右. (2013).
Do altmetrics work? Twitter and ten other social web services.
PLOS ONE, 8(5), e64841. https://doi.org/10.1371/journal.pone
.0064841, 考研: 23724101

Thelwall, M。, Kousha, K., Dinsmore, A。, & Dolby, K. (2015).
Alternative metric indicators for funding scheme evaluations.
Aslib Journal of Information Management, 68(1), 2–18. https://
doi.org/10.1108/AJIM-09-2015-0146

Wilkinson, C. (2019). Evidencing impact: A case study of UK
academic perspectives on evidencing research impact. 学习
高等教育学士, 44(1), 72–85. https://doi.org/10.1080
/03075079.2017.1339028

Wilsdon, J。, 艾伦, L。, Belfiore, E., 坎贝尔, P。, Curry, S。, … Johnson,
乙. (2015). The metric tide: Report of the independent review
of the role of metrics in research assessment and management.
布里斯托尔: Higher Education Funding Council
for England
(HEFCE). https:// kar.kent.ac.uk/81123/1/ Metric_Tide_main
_report.pdf. https://doi.org/10.4135/9781473978782

Wooldridge, J。, & 国王, 中号. 乙. (2019). Altmetric scores: An early in-
dicator of research impact. Journal of the Association for
Information Science and Technology, 70(3), 271–282. https://
doi.org/10.1002/asi.24122

Quantitative Science Studies

881

D

w
n

A
d
e
d

F
r


H

t
t

p

:
/
/

d

r
e
C
t
.


t
.

/

e
d

q
s
s
/
A
r
t

C
e

p
d

F
/

/

/

/

2
3
8
6
4
1
9
7
0
7
7
2
q
s
s
_
A
_
0
0
1
4
5
p
d

/

.

F


y
G

e
s
t

t


n
0
7
S
e
p
e


e
r
2
0
2
3RESEARCH ARTICLE image
RESEARCH ARTICLE image
RESEARCH ARTICLE image
RESEARCH ARTICLE image
RESEARCH ARTICLE image
RESEARCH ARTICLE image
RESEARCH ARTICLE image

下载pdf