ARTICLE DE RECHERCHE - Recherche en IA spécialisée au MIT

ARTICLE DE RECHERCHE

Is research funding always beneficial?
A cross-disciplinary analysis of
ROYAUME-UNI. research 2014–20

Mike Thelwall1

, Kayvan Kousha1
Cristina I. Font-Julián2

, Mahshid Abdoli1
, Paul Wilson1

, Emma Stuart1

, Meiko Makita1

, and Jonathan Levitt1

1Statistical Cybermetrics and Research Evaluation Group, University of Wolverhampton, Wolverhampton, ROYAUME-UNI
2Department of Audiovisual Communication, Documentation and History of Art,
Universitat Politècnica de València, València, Espagne

Mots clés: academic careers, research funding, research grants, research quality, scientometrics

ABSTRAIT

Although funding is essential for some types of research and beneficial for others, it may
constrain academic choice and creativity. Ainsi, it is important to check whether it ever seems
unnecessary. Here we investigate whether funded U.K. research tends to be higher quality in
all fields and for all major research funders. Based on peer review quality scores for 113,877
articles from all fields in the U.K.’s Research Excellence Framework (REF) 2021, we estimate
that there are substantial disciplinary differences in the proportion of funded journal articles,
from Theology and Religious Studies (16%+) to Biological Sciences (91%+). The results
suggest that funded research is likely to be of higher quality overall, for all the largest research
funders, et pour 30 out of 34 REF Units of Assessment (disciplines or sets of disciplines), même
after factoring out research team size. There are differences between funders in the average
quality of the research supported, cependant. Funding seems particularly associated with higher
research quality in health-related fields. The results do not show cause and effect and do not
take into account the amount of funding received but are consistent with funding either
improving research quality or being won by high-quality researchers or projects.

INTRODUCTION

Writing and managing grants occupies a substantial amount of academic time, but it is not
clear whether the benefits outweigh the costs in all fields. In some cases, researchers may
be unable to experiment without funding, but scholars not needing new equipment, ressources,
or time buyout may be able to work equally well without financing. Nevertheless, little is
known about the proportion of academic time spent on grant writing and administering, donc
it is difficult to weigh the benefits of funding against its costs. Although many studies report
the proportion of time spent by academics on research, teaching, and administration (Bentley
& Kyvik, 2012), they rarely ask about grant writing as a separate activity. One exception esti-
mated that each Australian National Health and Medical Research Council grant proposal
took 38 working days (nearly two months), ou 28 for a resubmitted proposal. Dans 79% of cases,
this effort was unrewarded (Herbert, Barnett et al., 2013), so Australian researchers collectively
spent between half a year and a full year writing grant proposals for each one funded. A survey
de 12 European countries found that between 51% et 84% of academics (71% aux États-Unis
Royaume) wrote grant proposals or otherwise responded to calls for proposals each year

un accès ouvert

journal

Citation: Thelwall, M., Kousha, K.,
Abdoli, M., Stuart, E., Makita, M.,
Font-Julián, C. JE., Wilson, P., & Levitt, J..
(2023). Is research funding always
beneficial? A cross-disciplinary
analysis of U.K. research 2014–20.
Études scientifiques quantitatives, 4(2),
501–534. https://est ce que je.org/10.1162
/qss_a_00254

EST CE QUE JE:
https://doi.org/10.1162/qss_a_00254

Peer Review:
https://www.webofscience.com/api
/gateway/wos/peer-review/10.1162
/qss_a_00254

Reçu: 11 Décembre 2022
Accepté: 25 Mars 2023

Auteur correspondant:
Mike Thelwall
m.thelwall@wlv.ac.uk

Éditeur de manipulation:
Ludo Waltman

droits d'auteur: © 2023 Mike Thelwall,
Kayvan Kousha, Mahshid Abdoli,
Emma Stuart, Meiko Makita, Cristina I.
Font-Julián, Paul Wilson, and Jonathan
Levitt. Published under a Creative
Commons Attribution 4.0 International
(CC PAR 4.0) Licence.

La presse du MIT

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Is research funding always beneficial?

(Drennan, Clarke et al., 2013). This work is highly stressful, affecting work–life balance in
almost all applicants, but deemed necessary partly due to institutional pressure and expecta-
tions from colleagues (Herbert, Coveney et al., 2014). There have also been claims that the
constant pressure to win grants undermines the quality of research, particularly in situations
where ongoing employment is funding dependent (Fumasoli, Goastellec, & Kehm, 2015), et
perhaps through research time lost to grant writing. Ainsi, it is important to assess whether the
benefits of funding always outweigh the costs. This article focuses on the narrower issue of
whether funding is always beneficial, at least in the sense of being associated with higher
research quality outputs.

Although the time and equipment for early scientific research was self-financed or informally
supported by benefactors, the system of competitively awarding national grants for future
research emerged from the prize system (for previous achievements) in France before the First
World War (Crosland & Galvez, 1989). Over the last half century, university funding in many
countries has changed from being awarded unconditionally for the benefit of science, albeit
with a focus on government priority areas such as defense, to being mainly accountable and
harnessed for societal benefits (Banchoff, 2002; Demeritt, 2000; Lepori, van den Besselaar et al.,
2007; Mustar & Larédo, 2002), such as medical priority areas (Xu, Zhang et al., 2014). Resource-
seeking behaviors (“academic capitalism”; Slaughter & Leslie, 2001) are long-established norms
in several major research countries (Johnson & Hirt, 2011; Metcalfe, 2010). Research funding is
now primarily awarded for achievements (c'est à dire., performance-based funding; Hicks, 2012) ou
future promise, through competitive grants (OECD, 2014). This is supplemented by incentives
to seek finances from industry and other nonacademic sources to fund research for nonaca-
demic benefits (Laudel, 2005). External funding is considered valuable despite the huge amount
of lost time spent by experts writing grant proposals (Polster, 2007) and the potential to skew
science (Thyer, 2011). National grant awards may aim to generally support promising research
or researchers, or support research with societal benefits (par exemple., Takemura, 2021).

The effect of funding seems likely to depend on the researcher, with field-related variations.
Par exemple, in specialties needing funding for any kind of research (par exemple., areas of medicine or
genetics), such as to employ enough assistants or to access equipment or consumables, un
researcher without funding cannot research, so what do they do instead? If their role is not
changed to teaching only (Edwards, 2022; Nyamapfene, 2018) or professional (par exemple., clinical
doctor), they might use any research time allowance to write grant proposals, accept consul-
tancy or advisory roles, read academic articles, develop aspects of their skills, develop their
research methods or theory, or devote more time to teaching, administration, or other roles.
They may also support others’ funded research projects in a minor role. In some cases, ils
may also write short papers about aspects of their research process, such as ethics or minor
methods details. In contrast, other researchers may easily be able to conduct unfunded
recherche, even though funding might improve their work with better equipment or larger
teams. From a U.K. Research Excellence Framework (REF) perspective, funding might improve
research productivity by supporting larger teams (allowing divisions of labor) and give the
funded researcher a larger pool of publications from which to choose from for REF evaluations.
Ainsi, any comparison between funded and unfunded research presupposes that it is possible
to do research in a field without funding and necessarily excludes researchers that can only
conduct funded research but did not receive funding in the period examined. The situation is
complicated in the United Kingdom because university teaching budgets subsidize 13% de
recherche (Olive, 2017) in a way that may not be recorded. Researchers with the choice
may prefer unfunded research because it gives them autonomy from funder goals and require-
ments (Edwards, 2022), particularly in the era of challenge-led research (Olive, 2017).

Études scientifiques quantitatives

502

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Is research funding always beneficial?

Few studies have explicitly examined the relationship between research quality and fund-
ing, except with bibliometric proxies for quality, such as citation rates. One exception found
industry-funded submissions to a 2006 scoliosis conference not to have statistically signifi-
cantly different peer review quality scores (Roach, Skaggs et al., 2008). Another showed that
urogynecology randomized controlled trials had higher quality methods when they were
funded (Kim, Chung et al., 2018). Although, as reviewed below, many studies have shown
in different contexts that funded research tends to be more cited than unfunded research, it
is not known whether this is true in all fields and if funded research also tends to be higher
qualité. Ainsi, the research questions below collectively address issues about the relationship
between funding and research quality that have not previously been directly investigated (par-
ticularly RQ2). A descriptive first question is included for background to the main questions.
The focus is on funders rather than funding because of substantial differences between funders
in practice, and differences between funders are investigated in RQ3. Team size is also
included (RQ4) because many funders mandate or otherwise encourage larger
authorship/grant teams, so it is important to differentiate authorship team size effects (lequel
are associated with more highly cited research) from the funding itself. A question about cita-
tion counts is included (RQ5) to assess their value as a tool because of their use in most pre-
vious investigations of research funding. The questions of this exploratory study are mostly
addressed with a combination of simple bivariate statistical tests or correlations on a data
set of U.K. articles with research quality scores and funding information (par exemple., average quality
of research supported by funders or unfunded) but regression is also used for RQ4.

(cid:129) RQ1: How prevalent is research funding for U.K. REF journal articles and are there dis-

ciplinary differences in the answer?

(cid:129) RQ2: Is funded research (ROYAUME-UNI. REF journal articles) higher quality for all major research

funders?

(cid:129) RQ3: Do research funders support different quality research (ROYAUME-UNI. REF journal articles)?
(cid:129) RQ4: Is funded research (ROYAUME-UNI. REF journal articles) of higher quality irrespective of

authorship team size?

(cid:129) RQ5: Are average citation counts effective proxies for average quality for externally

funded research (ROYAUME-UNI. REF journal articles)?

2. BACKGROUND: RESEARCH FUNDING TYPES AND BENEFITS

This section mainly reviews research findings. No theory of the disciplinary organization of
science has yet shed light on the relationship between funding and research quality. A partial
exception is that grant review outcomes might be expected to be more unpredictable in fields
with low level of agreement on the objects and methods of research (task uncertainty: Whitley,
2000), probably including most of the social sciences and humanities.

2.1. Types of Funding

There are many types of grant funding, in addition to recurring block grants. Each has its own
unique characteristics, including goal specificity, competitiveness, funding a project or person,
applicant restrictions, duration, cost scope, and embedded evaluation criteria. These factors
may affect whether a grant is beneficial to a researcher’s output, so it is unsurprising that the
performance of funded researchers varies between funding schemes, even for public funders
(Wang, Wang et al., 2020). It is impossible to differentiate between funding types in practice
for any comprehensive investigation into the impact of all types of funding on research.

Études scientifiques quantitatives

503

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Is research funding always beneficial?

Nevertheless, relevant evidence has emerged from prior studies for two dimensions, as sum-
marized below.

(cid:129) Size: The average size of individual grants has increased over recent decades, for exam-
ple with long-term funding for large centers of excellence at the expense of sets of indi-
vidual grants (Bloch & Sørensen, 2015; OECD, 2014). In the United States, block-funded
National Science Foundation centers do not seem to improve the journal outputs of
members, although they do improve commercial partnerships (Gaughan & Bozeman,
2002). Smaller grants seem to help productivity more than larger grants for research cen-
ters (Bloch, Schneider, & Sinkjær, 2016). Smaller research awards lead to more citations
overall for biological science research (Gallo, Carpenter et al., 2014). In medicine, un
small amount of funding from public and private research contracts and consultancies
reduces research impact but a large amount increases it (Muscio, Ramaciotti, & Rizzo,
2017). The latter may reflect the large-scale funding needed for effective medical studies
in many cases, with underfunded research also being underpowered.

(cid:129) Rationale/source: The source of funding received by a research group influences their
research agenda (Currie-Alder, 2015; Hottenrott & Lawson, 2017; Tellmann, 2022). Dans
terms of quality, spinal research harnessed weaker types of evidence (par exemple., case series)
when it had industry funding but was more likely to report positive outcomes (Amiri,
Kanesalingam et al., 2014).

2.2. Effectiveness of Grant Proposal Peer Review and Characteristics of Recipients

For any analysis of the influence of funding on research, it is difficult to distinguish between
cause and effect in terms of funders finding the best research/researchers or the funding
improving/allowing research/researchers. Although some grant selection processes focus on
applicant characteristics, most concentrate on the proposal, checking its rationale, evaluating
its validity, et (souvent) match with funding criteria (Chubin, 1994; Franssen, Scholten et al.,
2018; van Arensbergen & van den Besselaar, 2012).

There is limited overall evidence of the effectiveness of peer review for grant proposals
(Liaw, Freedman et al., 2017), based on evaluations typically using citation indicators as a
proxy for research quality or achievements. Dans certains contextes, higher scores or success in win-
ning awards have been shown to associate with more citations (Bornmann & Daniel, 2006;
Gallo et al., 2014) or more outputs (Fang, Bowen, & Casadevall, 2016; Győrffy, Herman, &
Szabó, 2020). In contrast, for economic and social sciences research council grants in the
Netherlands, while weak researchers tended to be rejected, awardees performed substantially
worse in bibliometric terms than rejected researchers with similar scores (van den Besselaar &
Leydesdorff, 2009). This suggests that the research council process selected above-average
researchers but not the very highest performing (at least bibliometrically), or that the funding
was detrimental.

Many studies have found disparities in review outcomes that are suggestive of bias, si
deliberate or accidental, or systemic effects. These biases include gender (Cruz-Castro,
Ginther, & Sanz-Menendez, 2022; Tricco, Thomas et al., 2017), âge (Levitt & Levitt, 2017),
ethnicity (Cruz-Castro et al., 2022; Hayden, 2015), interdisciplinarity (Seeber, Vlegels, &
Cattaneo, 2022), and institutional prestige (Ali, Bhattacharyya, & Olejniczak, 2010; Enger &
Castellacci, 2016; Horta, Huisman, & Heitor, 2008; Jappe & Heinze, 2023). All biases seem
likely to reduce the effectiveness of the grant allocation process and hence, presumably, le
overall benefits of funding.

Études scientifiques quantitatives

504

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Is research funding always beneficial?

2.3. The Impact of Grants on Research Productivity and Impact

Funding could be expected to increase the productivity or impact of the funded researchers.
The benefits of research funding are impossible to fully quantify, and it is difficult to generate
meaningful statistics because of the lack of effective control groups in most cases, and partic-
ularly the ability of unfunded groups to receive funding from sources other than the one exam-
ined (Neufeld, 2016; Schneider & van Leeuwen, 2014). Most previous studies have analyzed
individual funding sources and assumed that the papers acknowledging them were primarily
caused by the funding, whereas journal articles often draw upon a range of different long-term
and short-term funding for equipment and different team members as well as specific project-
based grants, at least for biomedical research (Rigby, 2011). De plus, many studies do not
distinguish between selection effects and funding effects (Neufeld, 2016): Are funded
researchers more productive because of the money or because better researchers/proposals
were selected, or both? De plus, all studies so far have had limited scope: There are different
types of funding and disciplinary differences in funding uses and procedures so there is
unlikely to be a simple relationship between funding and impacts. Par exemple, larger funded
studies may find it easier to get ethical approval to research in clinical settings (Jonker, Cox, &
Maréchal, 2011), reducing the number of unfunded studies.

Funding usually associates with (c'est à dire., correlates with but does not necessarily cause)
increased research productivity, as measured by journal articles, often even after the end of
the funding period (Chudnovsky, López et al., 2008; Godin, 2003; Defazio, Lockett, & Wright,
2009; Ebadi & Schiffauerova, 2016; El-Sawi, Sharp, & Gruppen, 2009; Hussinger & Carvalho,
2022; Saygitov, 2018; Shimada, Tsukada, & Suzuki, 2017) but commercial funding can slow
academic publishing because of the need to write patents or produce other outcomes
(Hottenrott & Thorwarth, 2011). A systematic attempt to track down all funding sources for
research from one university suggested that funding increased productivity but not citation
impact, although it would be difficult to disentangle disciplinary differences in funding value
with this data (Sandström, 2009).

Funding also usually associates with higher citation impact (par exemple., Álvarez-Bornstein, Díaz-
Faes, & Bordons, 2019; Berman, Borkowski et al., 1995; Gush, Jaffe et al., 2018; Heyard &
Hottenrott, 2021; Jowkar, Didegah, & Gazni, 2011; Levitt, 2011; Lewison & Dawson, 1998;
Neufeld, 2016; Peritz, 1990; Rigby, 2011; Roshani, Bagherylooieh et al., 2021; Thelwall,
Kousha et al., 2016; Yan, Wu, & Song, 2018) but there are exceptions (Alkhawtani, Kwee,
& Kwee, 2020; Jowkar et al., 2011; Langfeldt, Bloch, & Sivertsen, 2015; Neufeld, 2016;
Sandström, 2009). In support of the latter, 89% of the most cited rhinoplasty articles published
par 2015 were unfunded (Sinha, Iqbal et al., 2016) et 30% of key papers for physics,
chemistry, and medicine Nobel Prize winners 2000–2008 declared no funding (Tatsioni,
Vavva, & Ioannidis, 2010). Unfunded research might sometimes be highly cited because it
has more scope to be innovative, at least in fields such as library and information science not
needing expensive resources (Zhao, 2010). Grants may constrain academic freedom, which is
a particular threat to the role of social science research in challenging authority and in being able
to interpret results free from external pressures (Kayrooz, Åkerlind, & Tight, 2007).

2.4. Levels and Types of Unfunded Research

Most research in the previous century was unfunded, at least as reported in journals. An early
study of 900 journal articles in three medical journals from 1987, 1989, et 1991 found high
levels of unfunded research (at least without declared funding sources): internal medicine
(60%), pathology (62%), and surgery (74%) (Berman et al., 1995). De la même manière, dans 1987, 1989,

Études scientifiques quantitatives

505

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Is research funding always beneficial?

et 1991, 84% of journal articles by pathologists were unfunded (Borkowski, Berman, &
Moore, 1992) et 63% of emergency medicine articles were unfunded in 1994 (Ernst, Houry,
& Blanc, 1997). Dans 1992, cependant, only 23% of internal medicine and neurology journal
articles were unfunded (Stein, Rubenstein, & Wachtel, 1993). Partly funded research is also
common in medicine (Peut, Agan et al., 2013).

Early unfunded research was often different from funded research (Bodensteiner, 1995;
Silberman & Snyderman, 1997; Stein et al., 1993) and a few studies have compared funded
with unfunded research types this century. For Spanish virology, cardiology, and cardiovascu-
lar scholars, unfunded research was hospital based and clinical, suggesting that it had been
internally supported by hospital resources (Álvarez-Bornstein et al., 2019). Unfunded investi-
gations may tend to be desk research or other cheaper types, including secondary data
analyse (Vaduganathan, Nagarur et al., 2018), guidelines (Goddard, James et al., 2011),
review articles (par exemple., Imran, Aamer et al., 2020), retrospective records-based analyses (par exemple.,
Brookes, Farr et al., 2021; Sedney, Daffner et al., 2016), small case studies (par exemple., Qi & Wei,
2021), or analytical/theoretical/opinion papers without primary data (Underhill, Dols et al.,
2020). In nursing, evidence-based practice research may often be unfunded because the data
analyzed may come mainly from investigators’ daily work roles (Higgins, Downes et al.,
2019). Researching may be a compulsory part of some higher-level courses, such as for radi-
ology, and this may result in many small-scale unfunded studies by educators and learners
(Johnson, Mathews, & Artemakis, 2002). In medicine, unfunded research may be dispropor-
tionately from general practitioners compared to hospital doctors because they lack the infra-
structure to obtain and maintain large grants (van Driel, Deckx et al., 2017).

3. MÉTHODES

3.1. Données

For this study, the U.K. Research and Innovation (UKRI) national science and research funding
government agency gave us the preliminary scores from March 2022 of all 148,977 journal
articles submitted to the U.K. REF 2021, excluding those from the University of Wolverhampton
(the project host institution, for confidentiality reasons). The REF (REF, 2022) is a periodic (en haut
to seven-year gaps) exercise to evaluate U.K. academic research to, among other things, allo-
cate the U.K. block funding research grants known as “Mainstream QR” and worth over
£2 billion per year for up to 7 années. The REF includes postpublication expert review of
selected outputs (1–5 per researcher), from which we were given the journal articles. Aca-
demics submit only their best outputs over the period and teaching staff do not need to
submit anything, so the articles analyzed are likely to represent predominantly the best
research produced by U.K. researchers 2014–20. Each article had been given a “quality”
score by at least two out of over 1,000 expert assessors (usually full professors), avec le
grades being 1* (nationally recognized), 2* (internationally recognized), 3* (internationally
excellent), et 4* (world leading). The grades reflect originality, significance, and rigor, avec
different and detailed guidelines for these from each of four overseeing Main Panels (REF,
2020, pp. 34–51). There was careful norm referencing between assessors within each of the
34 Units of Assessment (UoAs) to which they had been assigned to ensure that the scores
by different pairs of assessors were comparable. There was also overall norm referencing for
the entire REF. The peer review process is carefully managed because of its multi-billion
pound financial value (about £50,000 per individual score, on average), although the
reviewers are not experts in all areas that they need to assess. Each UoA covers what might
be called a broad field (very broad in some cases, like UoA 34: Communication, Cultural and

Études scientifiques quantitatives

506

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Is research funding always beneficial?

Media Studies, Library and Information Management) and is either a recognizable discipline
(par exemple., UoA 18 Loi) or a set of related disciplines (par exemple., UoA 8 Agriculture, Food and Veterinary
les sciences). The four Main Panels group together cognitively related UoAs for administrative
and norm referencing purposes.

The REF articles were matched against Scopus records by DOI comparisons (n = 133,218).
REF articles without a DOI in Scopus were matched instead by title, with a manual check to
accept or reject all potential matches (n = 997). The Scopus record was used for funding and
citation information. Scopus cross-references information in articles with funding information
gained from other sources populating its funding database (McCullough, 2021). Scopus reports
a single funder for each paper, at least through its Applications Programming Interface (API), comme
used to gather the data, although some studies have multiple funders. Ainsi, the funder-level
results reported here are based on incomplete data.

Some of the articles were given multiple grades from the same or different UoAs. This is
possible because each author is entitled to submit between one and five outputs for which
they are a coauthor, and coauthors from different institutions may choose the same article(s).
For analysis, duplicate articles were removed within the grouping analyzed (UoA, Main
Panel, or all). When an article had received different scores, it was given the median or a
random median when there were two.

Scopus was used for funding information because of its wider coverage than the Web of
Science (Martín-Martín, Thelwall et al., 2021) and because Google Scholar does not extract
relevant information. Scopus started systematically indexing funding in 2011 (Rigby, 2011) donc
its data should be mature for the REF period 2014–20. Funding data in academic articles
might be in a separate “Funding sources” section, in the acknowledgments, or as a footnote
alongside author information. The acknowledgment section was a common place for funding
information (Paul-Hus, Díaz-Faes et al., 2017) before the rise of the dedicated funding
section.

Some article funders were universities, suggesting that the authors had been allocated inter-
nal university money for their research or that it was unfunded but recorded as university-
funded to reflect employers allowing research time for the scholars involved, or for university
policy reasons. As it was not possible to distinguish between the two, for the regression anal-
ysis, research was classed as unfunded if the funder was a university, irrespective of country.
For this, we checked the 4,042 funders and classified 1,317 of them as internal university or
research institute funding (par exemple., Weizmann Institute of Science) et 2,725 as external funding
(par exemple., American Mathematical Society). After this stage, research was classified as externally
funded if it declared a funder in Scopus and the funder name was not one of the 1,317
universities found. When funding information was present (par exemple., a grant number) but no name
for the funder was given, it was assumed to be externally funded.

3.2. Data Quality Checks

To check whether the Scopus API funding information was accurate, for six UoAs chosen to
represent different field types, we read samples of articles for details of research support. Pour
each UoA, 100 unfunded articles, 100 university-funded articles, et 100 nonuniversity
funded articles were selected with a random number generator for checking (ou 100%,
when fewer than 100). The checks were performed by two people, the first author for all
and either ES, MM, or MA. Publisher versions of articles were checked for funding informa-
tion except when the preprint was online with funding information. In one case (Theology,

Études scientifiques quantitatives

507

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Is research funding always beneficial?

unfunded) we were unable to obtain the article through any method (including interlibrary
loans) and it was substituted with the next article selected by the random number generator.
A study was counted as university funded if the only funding source mentioned was
university based. It was recorded as externally funded if any nonuniversity funding source
was mentioned.

Funding could be mentioned in multiple places, although a “Funding” section or an
“Acknowledgments” section at the end of the article were common, and a “Disclosure of Inter-
ests” end section sometimes also included funding information. Other places included first
page footnotes, last page footnotes (rare), a notes section at the end of the article, the first par-
agraph of the article (rare), and the last paragraph of the conclusions (common in Physics, un
example in Theology). Articles sometimes declared that the research was unfunded, usually
within a funding section, and sometimes in a disclosure of interests section. In one case, un
funding section declared that the research was unfunded but the acknowledgments section
thanked a funder, so this was coded as funded. Some articles included author biographies that
might have mentioned funding sources but never did.

Funding statements varied in length from short declarations of the funder name and grant
number to several paragraphs of thanks. In some fields it was common to thank departments
hosting a visit or seminar and current and former employers. An article was classed as funded
if this was stated directly (par exemple., “funded by”) or if it was suggested by the context, such as by
naming a research funding organization or thanking one for an unspecified type of “support.”
Acknowledgments of support from universities were not counted as funding if these seemed to
be minor and routine, such as hosting a visit or supporting a seminar. University support was
counted as funding for the purposes of these checks if the term “funding” was mentioned or
“grant” or it was obvious from the context that a financial transaction had occurred, as in the
case of a PhD studentship. In a few cases, support in kind was provided, such as through
access to equipment, but this was not counted as funding. Research described as part-funded
was recorded as funded.

Although in some cases the article appeared to be the primary outcome of a grant, dans
most cases the relationship between the funding and the output was less clear. Par exemple,
the article could be one of the outputs of a PhD studentship or Leverhulme Trust fellowship.
Many articles had authors with differing funding sources, suggesting that the study itself had
not been funded but had been made possible by funding given to the participants. Tel
studies were counted as funded. In Medicine and Physics, Par exemple, long paragraphs
often recorded the financial support given to all participants as well as the equipment used
and the study itself.

The information found manually is unreliable. A funded article may have no declaration
within the text if the author forgot or the journal style or field norms discouraged it. Checks
were made of cases where Scopus recorded a funder but the article didn’t mention one. These
checks found examples where Scopus was correct because the article was listed on a funding
website as an output of the grant. Although Scopus has reported that its funding information is
imported from the acknowledgment sections of articles (Beatty, 2017), it seems likely that it
now automatically links articles to funding records from elsewhere and might also perform
wider searches of article text. In other cases the funding was plausible because the scholar
thanked the same funder on a different output at a similar time or listed the funder on their
online CV. Scopus also seemed to have listed incorrect funders in at least two cases: the wrong
funder altogether in one case, and a university in another case where the author had included
an acknowledgment that an earlier version had been presented at a seminar at that university.

Études scientifiques quantitatives

508

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Is research funding always beneficial?

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Chiffre 1. The results of manual checks of random samples of REF2021 articles recorded by Scopus as funded (listing a university or funder) ou
unfunded for six UoAs.

These were not altered in our data because the checks were for quality assessment rather than
correction.

Comparing the Scopus API information with manual checks, the Scopus API results were
always imperfect and substantially misleading in some cases (Chiffre 1). Almost all Clinical
Medicine and Physics articles were externally funded (c'est à dire., at least one nonuniversity funder)
even if the Scopus API listed none. In these cases, Scopus had presumably not found where the
funding was listed in the article. Physics article funding statements were often in the last par-
agraph of the conclusions, where they may have been missed. For all six fields, most articles
classed as university funded (c'est à dire., the Scopus API funder was apparently a university) étaient
externally funded. This typically occurred because the Scopus API reports only one external
funder, and the manual checks classed an article as externally funded if any of the funders
were not universities. In four cases, most Scopus API results were correct for unfunded and
externally funded articles, cependant. This information should be taken into consideration when
interpreting the results.

Études scientifiques quantitatives

509

Is research funding always beneficial?

3.3. Analyses

For RQ1, the proportion of articles declaring research funding was calculated for each UoA
and Main Panel.

For RQ2 and RQ3, the average quality of the articles from each funder was calculated and
compared to the average quality of unfunded research. The grade point average (GPA) était
used for this, which is the arithmetic mean of the quality scores. Although widely used in the
United Kingdom in rankings of institutions, the GPA is a convenience and not theoretically
informed because there is no reason to believe that a 4* article is four times as good as a 1*
article. Nevertheless, it at least gives a straightforward and easily understandable indicator of
average quality scores for funded journal articles. Le 30 largest funders (including unfunded
and unknown funder) were reported. The choice of 30 is relatively arbitrary. The GPA for
small funders with a few articles would be imprecise estimates of the funder’s average
research quality, et 30 is a common statistical choice for the minimum size to identify a
pattern. This calculation ignores funders not reported by the Scopus API, which particularly
affects articles with multiple funders. The RQ2 test involves making multiple comparisons
using confidence intervals and this increases the chance of getting at least one statistically
positive result by accident, the problem of familywise error rates. We have reported
confidence intervals without error rates because the individual funders are of interest, mais
use Bonferroni corrections (see Perneger, 1998) to discuss the results as a group. These
increase the probability threshold for a difference to be deemed statistically significant in a
way that protects the chance of making at least one false positive (c'est à dire., a Type I error) à
le 0.05 level.

We used ordinal regression (the polr function in the R MASS package with the Hess =
TRUE option) to answer RQ4, with research quality as the dependent variable and research
funding (binary) and the log of the number of authors as independent variables. A similar
approach has been used with citations as a proxy for research quality as the dependent
variable (Ebadi & Schiffauerova, 2016). We excluded 23 articles with no authors listed from
the regressions. We ran a separate regression for each UoA and Main Panel (combining
similar UoAs) and for all the data. Ordinal regression only assumes that the four quality
scores are in ascending order but does not assume that they are equidistant, so it is better
than types of regression requiring a scalar output (Gutiérrez, Perez-Ortiz et al., 2015). Par
including both authors and funding as independent variables, the regression output can
show whether one of the two is redundant in any area. The log of the number of authors
was used instead of the number of authors because the relationship between author num-
bers and log-transformed citation counts is approximately logarithmic (Thelwall & Maflahi,
2020), and the shape is similar for the relationship between REF scores and author numbers
(Thelwall, Kousha et al., 2022un).

For RQ5, we calculated a field-normalized citation score for every REF2021 article to allow
fair comparisons between articles from different fields. For this, we first log normalized each
citation count with ln(1 + X) to reduce skewing in the data set caused by very highly cited
articles (Thelwall & Fairclough, 2017). Alors, we calculated the average of the logged citations
for each of the 330 Scopus narrow fields and each year 2014–18 (c'est à dire., 5 × 330 averages) et
divided each article’s logged citation count by the average for its narrow field and year. Arti-
cles in multiple fields were instead divided by the average of the relevant field averages. Ce
gives a Normalized Log-transformed Citation Score (NLCS) (Thelwall, 2017) for each journal
article. These can be compared between fields and years because, by design, a score of 1
always reflects an average number of citations for the field and year of an article. Averaging

Études scientifiques quantitatives

510

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Is research funding always beneficial?

the NLCS of all articles associated with a funder gives the funder’s Mean NLCS (MNLCS),
which is a measure of the normalized average citation rate for the journal articles it funded.
Encore, an MNLCS above 1 always reflects a funder that tends to fund articles that are more
cited than average for their fields and years. The most recent 2 years were excluded from the
calculation to give a citation window of at least 2 années, reducing the influence of short citation
windows. Although a 3-year citation window is better (Wang, 2013), it would reduce the
amount of data and the log transformation in the NLCS formula reduces the statistical variabil-
ity caused by short time windows.

In all analyses, we did not take into account any dependencies in the data caused by up to
five outputs being submitted by a single researcher, and this is a limitation. En moyenne, chaque
full-time equivalent (FTE) researcher submitted 2.5 outputs. A minimum of one and a maxi-
mum of five outputs could be submitted by a single academic (whether full time or part time).
Accounting for some nonarticle outputs and an unknown number of part timers, chercheurs
(whether full time or part time) probably submitted about two articles each, on average. If
every researcher produced uniform quality solo work, then this would reduce the effective
sample sizes in all the analyses by 50%. Nevertheless, uniform quality work for all researchers
is unrealistic and most work was coauthored, so the effective sample size reduction due to
dependency (c'est à dire., two articles are more likely to have the same REF score if they have at least
one author in common) is unknown. Because of this, the widths of the confidence intervals in
all the graphs should be treated with caution.

4. RÉSULTATS

4.1. RQ1: Prevalence of Research Funding

Just under two-thirds (63%) of journal articles submitted to REF2021 had funding informa-
tion recorded by the Scopus API, with substantial disciplinary differences (Tableau 1,
Chiffre 2). This figure excludes funded journal articles where the funder was not recorded
by the author, the journal did not allow a funding declaration, or a technical issue pre-
vented Scopus from finding the declaration (voir la figure 1). This also includes research that
was internally funded, whether nominally (part of the scholar’s job to research) ou plus
substantially, such as with money for equipment or research assistants. Some universities
(par exemple., University of Wolverhampton, not in the data set) now require scholars to record
their employer as the funder within the internal research information management system
for articles not externally funded, and this may encourage them to report the same within
their articles.

Funding is the norm for Main Panels A (80%) et B (76%), but half as prevalent in Main
Panels C (40%) and D (32%). The difference is presumably due to the need for equipment
and large teams in the health, vie, and physical sciences (except for purely theoretical
contributions), whereas expensive or perishable equipment is probably rarer in the social
sciences, arts, and humanities, except for long-term purchases (par exemple., musical instruments).
De plus, there may be more social sciences, arts, and humanities topics that can be
researched in small teams or alone. The three UoAs with the highest proportions of funded
papers are Biological Sciences (91%), Physics (91%), Clinical Medicine (88%), et
Chemistry (87%). All these subjects have subfields that do not need expensive equipment:
theoretical physics, theoretical chemistry, biostatistics (related to medicine), and systems
biology. Ainsi, the result may reflect “cheaper” specialties being rare in the United Kingdom
or globally.

Études scientifiques quantitatives

511

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Is research funding always beneficial?

Tableau 1. Number of articles, unfunded articles, and university funded articles. Number of funders per UoA, main panel or all

Set
UN

UoA or Main Panel

1: Clinical Medicine

Articles
9,916

Unfunded
articles
1,173

University-funded
articles
262

2: Public Health, Health Services &

3,890

745

Primary Care

3: Allied Health Prof., Dentistry,

9,675

2,885

Nursing & Pharm

4: Psychologie, Psychiatry &

8,173

2,271

Neurosciences

5: Sciences biologiques

6: Agriculture, Food & Veterinary

les sciences

6,376

3,147

7: Earth Systems & Environnemental

3,724

les sciences

8: Chemistry

9: Physics

10: Mathematical Sciences

11: Computer Science & Informatics

12: Engineering

3,274

4,499

5,111

4,646

16,335

13: Architecture, Built Environment &

2,582

Planning

14: Geography & Environnemental

Études

15: Archaeology

16: Economics & Econometrics

3,439

545

1,762

17: Business & Management Studies

11,853

18: Loi

19: Politique & International Studies

20: Social Work & Social Policy

21: Sociology

22: Anthropology & Développement

Études

23: Éducation

24: Sport & Exercise Sciences, Leisure

& Tourism

1,864

2,502

3,295

1,498

977

3,337

2,812

576

653

541

426

396

1,402

1,565

4,395

1,225

947

156

856

8,210

1,442

1,610

1,779

727

443

2,028

1,753

25: Area Studies

524

329

130

701

390

244

178

198

172

245

250

1,000

207

266

132

759

150

209

186

205

Funders
844

401

1,045

672

592

446

456

321

272

424

438

1,095

334

467

123

216

810

153

245

334

175

160

308

327

Funders with
5+ articles
134

166

111

195

117

Études scientifiques quantitatives

512

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Is research funding always beneficial?

Set
D

UoA or Main Panel

26: Modern Languages & Linguistics

27: English Language and Literature

28: Histoire

29: Classics

30: Philosophy

31: Theology & Religious Studies

Articles
962

768

1,082

111

806

185

32: Art and Design: Histoire, Practice

1,117

& Theory

33: Music, Drama, Dance, Perform.

544

Arts, Film

34: Comm. Cultural & Media Stud.

1,020

592

769

559

155

693

380

724

Lib & Info Man

Main Panel A (UoAs 1–6)

Main Panel B (UoAs 7–12)

Main Panel C (UoAs 13–24)

Main Panel D (UoAs 25–34)

All

All (UoAs 1–34)

39,248

36,614

7,925

8,610

35,634

20,819

7,071

4,842

113,877

41,649

Tableau 1.

(a continué )

Unfunded
articles
583

University-funded
articles
49

Funders
111

Funders with
5+ articles
10

1,804

1,925

2,361

345

6,297

145

4,107

1,825

1,858

456

4,107

438

335

328

882

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

4.2. RQ2: Is Funded Research Higher Quality for All Major Research Funders?

The GPA of the REF2021 scores of funded journal articles tends to be higher than the
unfunded article GPA for most large research funders in Main Panels A–D (Figures 3–6).
In the few cases where the funded GPA is lower than the unfunded GPA, the confidence
interval for the former almost always includes the latter. The sole minor exception is the
European Commission funding in Main Panel C (Chiffre 5). Nevertheless, this exception
could be a side effect of the large number of tests (29 × 4), and with a Bonferroni correc-
tion, the difference between European Commission-funded research and unfunded research
in Main Panel C is not statistically significant. Ainsi, at the Main Panel level, the results are
broadly consistent with research funding being an advantage for all major funders, albeit
marginal in some cases.

For Main Panel A (Chiffre 3), all research funder GPAs are above the unfunded GPA and
none of the research funder confidence intervals contain the unfunded GPA. Ainsi, funding
from a major funder is an advantage in Main Panel A. The same is broadly true for Main Panel
B (Chiffre 3) except that four of the funder GPA confidence intervals contain the unfunded
score.

The pattern is mixed for Main Panel C (Chiffre 5), perhaps because of smaller sample
sizes giving less accurate mean estimates and wider confidence intervals. Although there

Études scientifiques quantitatives

513

Is research funding always beneficial?

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Chiffre 2. The percentage of U.K. REF2021 journal articles with a declared source of funding in
Scopus.

are three funders with GPAs below the unfunded GPA, there are many funders with GPAs
substantially above it and with narrow confidence intervals. Ainsi, there is still a general
trend for major funder money to be advantageous in Main Panel C. For Main Panel D,
most funders have a GPA above the unfunded GPA, and a few have substantially higher
GPAs with narrow confidence intervals, suggesting that major funder money is also an
advantage here.

Major funders also tend to support higher quality research when the data are aggregated
to the level of individual UoAs, although there are some exceptions. Some illustrative

Études scientifiques quantitatives

514

Is research funding always beneficial?

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

Chiffre 3. The average quality score of REF2021 journal articles by research funder for Main Panel
UN (mainly health and life sciences) for the 30 research funders with the most articles. Error bars
indicate 95% intervalles de confiance.

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

examples are discussed here, focusing on larger UoAs for which the patterns are clearest.
For Clinical Medicine (UoA 1, Chiffre 7), Engineering and Physical Sciences Research
Council (EPSRC) funded research surprisingly generated lower quality scores than unfunded
recherche. The reason for this may be that UoA 1 assessors did not value research with
substantial inputs from nonmedical fields in the context of their UoA (par exemple., because of more
rigid quality criteria: Whitley, 2000). There is no similar problem for UoAs 2 (Chiffre 8) et
3 (Chiffre 9).

The presence of pharmaceutical companies as funders for health and medical research is
clear in UoAs 1–3 (Figures 7–9). The research that they fund tends to have a substantially
higher GPA than unfunded research, suggesting that the commercial income enhances rather

Études scientifiques quantitatives

515

Is research funding always beneficial?

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Chiffre 4. The average quality score of REF2021 journal articles by research funder for Main Panel
B (mainly engineering, physical sciences and mathematics) for the 30 research funders with the
most articles. Error bars indicate 95% intervalles de confiance.

than compromises academic quality, or that a commercial funder boosts the significance
component of quality for REF assessors.

The second largest UoA, Engineering (Chiffre 10) illustrates the general advantage of
major research funders for quality in this field. Although most of the funders are govern-
mental research funding bodies, military funding clearly produces above-average quality
recherche.

Funding seems to be a marginal advantage for the largest UoA, Business and Manage-
ment, as eight of the top 28 funders have a below average GPA (Chiffre 11). De plus,
the core funder, the Economic and Social Research Council (ESRC), confers the relatively
minor advantage of a 0.1 higher average GPA. The European Research Council was

Études scientifiques quantitatives

516

Is research funding always beneficial?

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Chiffre 5. The average quality score of REF2021 journal articles by research funder for Main Panel
C (mainly social sciences) for the 30 research funders with the most articles. Error bars indicate 95%
intervalles de confiance.

(pre-Brexit) particularly effective at funding high-quality research, but this is a logical side
effect of its strategy of selecting “top researchers” through very competitive grants.

4.3. RQ3: Do Research Funders Support Different Quality Research?

As the graphs above illustrate, there are statistically significant differences in the average
quality of research supported by different funders. Par exemple, in UoA 1 Clinical Medicine
(Chiffre 7), the average GPA of the main three research funders is different, with their confi-
dence intervals not overlapping. En particulier, the National Institutes of Health (NIH) funded
particularly high-quality research, followed by the Wellcome Trust (ROYAUME-UNI. charity) et le
Medical Research Council (MRC), all of which have large budgets and general funding

Études scientifiques quantitatives

517

Is research funding always beneficial?

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Chiffre 6. The average quality score of REF2021 journal articles by research funder for Main Panel
D (mainly arts and humanities) for the 30 research funders with the most articles. Error bars indicate
95% intervalles de confiance.

remits. The NIH advantages may be that its research funded with U.K. partners would usually
be international, because it is based in the United States, and its funding is backed by the
greater financial resources of the United States.

4.4. RQ4: Does Authorship Team Size Moderate the Effect of Funding on Research Quality?

Research has shown that articles with more authors tend to be more cited, and funding
seems to attract large team sizes, so it is possible that the advantage of funding is some-
times primarily in bringing together many authors. In our data, for all UoAs and Main
funder GPA correlates positively with the average (geometric mean) number of
Panels,
In other
authors on papers associated with the funder (Chiffre 12, GPA vs. authors).
the higher
words,

the larger the average authorship team size supported by a funder,

Études scientifiques quantitatives

518

Is research funding always beneficial?

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Chiffre 7. The average quality score of REF2021 journal articles by research funder for UoA 1
Clinical Medicine for the 30 research funders with the most articles. Error bars indicate 95% con-
fidence intervals.

the average quality of the research it funds. The correlations tend to be strong in Main
Panels A and B.

For UoAs with at least 30 funders associated with at least five papers each, the weakest
correlation between GPA and authors is for Business and Management Studies (0.06). Ainsi,
for Business and Management Studies research funders, there is almost no relationship between
average funded authorship team size and average research quality. This may be due to rela-
tively little variation in GPA between funders and typically small research teams (average
three authors per paper for all major funders, varying between 1.7 et 4.3). In contrast,
the highest correlation is for Agriculture, Food and Veterinary Sciences (0.84), partly due
to medical funders (MRC, NIH, Wellcome) and the Gordon and Betty Moore Foundation
supporting large team research with high GPAs.

Études scientifiques quantitatives

519

Is research funding always beneficial?

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Chiffre 8. The average quality score of REF2021 journal articles by research funder for UoA 2
Public Health, Health Services and Primary Care for the 30 research funders with the most
articles. Error bars indicate 95% intervalles de confiance.

Ordinal regressions for each UoA, Main Panel, and overall (39 regressions) allow the
effects of funding and author numbers to be analyzed separately. As a conservative step
(voir la figure 1), university-funded research was classed as unfunded, so the focus is on exter-
nal funding for research. Because of the incompleteness of the funding data, the results will
tend to underestimate any differences that exist. In the regressions, an exponentiated coef-
ficient of 1 indicates that the independent variable (logged number of authors or external
funding) has no effect on the odds ratios for quality scores (1, 2, 3, ou 4). Values greater
que 1 indicate that the variable increases the odds ratio for a higher quality score and
values less than 1 the opposite. Every increase of 1 in the exponentiated regression

Études scientifiques quantitatives

520

Is research funding always beneficial?

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Chiffre 9. The average quality score of REF2021 journal articles by research funder for UoA 3
Allied Health Prof., Dentistry, Nursing & Pharmacy for the 30 research funders with the most arti-
clés. Error bars indicate 95% intervalles de confiance.

coefficient for funding increases the odds ratio for higher quality research by 1 for funded
research compared to unfunded research. De la même manière, every increase of 1 in the exponen-
tiated regression coefficient for logged authors increases the odds ratio for higher quality
research by 1 for research with e = 2.718 times more authors.

The results show that, when considered independently from the number of authors,
funding associates with improved odds of higher quality research in 33 out of 34 UoAs
and all four Main Panels (Chiffre 13). The confidence intervals exclude the null value 1
pour 30 out of 34 (et 30 out of all 39) regressions. These calculations do not include
familywise error rate corrections for 39 separate tests, so may include some false positives;

Études scientifiques quantitatives

521

Is research funding always beneficial?

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Chiffre 10. The average quality score of REF2021 journal articles by research funder for UoA
12 Engineering for the 30 research funders with the most articles. Error bars indicate 95% con-
fidence intervals.

hence the true number of UoAs where funding is important may be less than 30. Con-
versely, on the basis of 30 out of 39 positives and the possibility that the remainder could
have a funding advantage because values greater than 1 are comfortably in all 95% con-
fidence intervals, it is also plausible that, after factoring out the number of authors, funding
always associates with an improved chance of higher quality journal articles. This is
because the few exceptions could be due to normal levels of chance. Nevertheless, alors que
funding has the most substantial association with quality in Main Panels A and B, its asso-
ciation is marginal in some UoAs from Main Panels C and D. Ainsi, overall, there is evi-
dence that funding has weak or moderate value, even after factoring out authorship team
size, in associating with higher quality research in the social sciences, arts, and humanities,
but there is strong evidence that it has a considerable value in medicine, life and physical
sciences, and engineering.

Études scientifiques quantitatives

522

Is research funding always beneficial?

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Chiffre 11. The average quality score of REF2021 journal articles by research funder for UoA 17
Business and Management Studies for the 30 research funders with the most articles. Error bars
indicate 95% intervalles de confiance.

Although less important here, increased author numbers usually, but not always, associ-
ate with increased odds of higher quality journal articles, even after factoring out research
funding. The exceptions are mainly in the arts and humanities.

4.5. RQ5: Are Average Citation Counts Effective Proxies for Average Quality for Externally

Funded Research?

Research funders often have their own evaluation teams to assess the effectiveness of their
subventions. For this, the main quantitative evidence is likely to be citation data, perhaps with pro-
ject grades from end-of-grant reviewers in some cases. If they make like-for-like comparisons
against similar funders, then the only quantitative data that they would have for both would be

Études scientifiques quantitatives

523

Is research funding always beneficial?

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Chiffre 12. Pearson correlations between funder MNLCS, GPA, and geometric mean authors by
UoA or Main Panel. MNLCS correlations only cover research published 2014–18. UoAs are
included only when they have at least 30 funders associated with at least five papers each.

citation counts. Ainsi, it is useful to check whether the average citation impact of funders is an
effective proxy for the average quality of the research that they fund.

Correlations between funder citation rates (MNLCS) and average quality (GPA) are strong
(> 0.5) in all Main Panels (Chiffre 12), suggesting that citation impact is a reasonable proxy
for research quality overall. The correlations also tend to be moderate or strong in the UoAs
of Main Panels A and B (Chiffre 12), but are variable in the UoAs of Main Panel C. Dans
particular, the correlations are close to 0 (positive or negative) in UoAs 17 (Business and

Études scientifiques quantitatives

524

Is research funding always beneficial?

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Exponentiated ordinal regression coefficients for quality score against external funding
Chiffre 13.
(binary) and the logged number of authors for REF2021 articles 2014–20. Error bars show 95%
intervalles de confiance. University-funded research and research without declared funding is classified
as unfunded. Each pair of coefficients shown above is from a separate model.

Management Studies), 20 (Social Work and Social Policy), et 23 (Éducation) and weak
(0.2) in UoA 13 (Architecture, Built Environment and Planning). Ainsi, citation rates are
inappropriate proxies for funder quality in these areas. By extension, and due to a lack of
evidence, it seems that citation rates should not be used as proxies for funder research
quality throughout the social sciences, arts, and humanities, except for Geography and Envi-
ronmental Studies.

Études scientifiques quantitatives

525

Is research funding always beneficial?

5. DISCUSSION
The results are limited to journal articles from the United Kingdom, and to the best 1–5 journal
articles written by U.K. academics 2014–20, so are not representative of typical U.K. recherche
(especially books). De plus, while the scores given to the articles by the REF assessors are rel-
atively carefully allocated, usually by two senior field experts following written guidelines (REF,
2022) and norm referenced within each UoA and broadly between UoAs, they are imperfect. Dans
particular, an unknown but nontrivial number of articles will have been assessed by people
without the knowledge to understand them, so guesswork was needed for these. De plus,
research quality is subjective and other assessors may well have given different scores to the
same outputs; the assessors may also have taken into account funding when allocating scores
(especially nonacademic funding as an indicator of significance). Nevertheless, the scores seem
likely to be broadly reasonable, with unreasonable scores or errors being noise in the data. Ce
hypothesis is sufficient for the above results to make sense, although noise in the data would
tend to reduce the magnitude of any differences found. As a caveat, cependant, there are different
ways of conceiving research quality and although the REF definition is relatively universal (com-
bining originality, significance, and rigor: REF, 2022), there are others (Langfeldt, Nedeva et al.,
2020). Related to this, researchers may not submit their most creative unfunded articles to the
REF because of the significance and rigor criteria, and this may influence the results.

Another limitation is that the results only consider the funder reported by the Scopus API,
ignoring any that Scopus could not find and all funders except one in the case of multiple-
funded articles. This is a substantial limitation, as discussed in the evaluation at the end of
Section 3. En particulier, the extent of funding is underestimated in the data here. This does
not invalidate the findings because funded research is still more likely to be recorded as such
in the API (Chiffre 1), so the funded and unfunded groups are statistically distinct. This limita-
tion nevertheless indicates that differences found between funded and unfunded research are
larger than shown in the data (because the unfunded subsets are “polluted” with funded arti-
clés). The errors in the Scopus API data would also tend to reduce the difference between
funded and unfunded research for the same reason. This reduction is likely to be largest when
the Scopus API has the most missing information (probably lower-numbered UoAs).

The findings ignore the value of each grant, whether the funding was partial, what the
money was spent on, how many publications were produced from it, and whether journal
articles were the primary outcome of the project or a side effect. They also ignore disciplinary
differences in the need to record funding sources, with biomedical fields apparently most
affected due to a need to register any potential conflicts of interest. They also ignore the
purpose of the funding, which may not be to conduct high-quality research but to develop
a technology for industry, to train a PhD student, to develop a junior postdoc, to build research
réseaux, or to support researcher mobility. The results do not differentiate between projects
awarded explicit funding by a university and projects without explicit funding but presumably
consuming university resources and time: Both are classed as unfunded for the regression and
are otherwise recorded as university funded only if this is stated in the funding information.
Plus généralement, the results do not take into account the time taken to write funding proposals
for either successful or unsuccessful bids. Enfin, funding here is tied to publications, bien que
a team may be partly funded and draw on different sources (Aagaard, Mongeon et al., 2021).

5.1. Comparison with Prior Work

The findings mostly have little directly comparable prior work. For RQ1, the prevalence of
research funding for any country is reported apparently for the first time, albeit with partial

Études scientifiques quantitatives

526

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Is research funding always beneficial?

data. The existence of disciplinary differences in funding rates is unsurprising but does not
seem to have been previously investigated for all academic fields. The prevalence of funding
is much higher than previously reported (Berman et al., 1995; Borkowski et al., 1992; Ernst
et coll., 1997; Jowkar et al., 2011; Lim, Yoon et al., 2012; Shandhi, Goldsack et al., 2021; Stein
et coll., 1993), with a few exceptions (Godin, 2003), probably at least partly due to more
systematic funding reporting now, and the U.K. sample (par exemple., excluding publishing
practitioners/professionals).

The higher quality rates for major funders (RQ2) are a new finding but echo many previous
studies of individual funders that have shown funded articles or researchers to be more cited
than a comparable group (unfunded articles, unsuccessful applicants, or researchers before the
funding) (Álvarez-Bornstein et al., 2019; Berman et al., 1995; Gush et al., 2018; Heyard &
Hottenrott, 2021; Levitt, 2011; Lewison & Dawson, 1998; Peritz, 1990; Rigby, 2011; Roshani
et coll., 2021; Yan et al., 2018), and conflicts with the few studies not showing this or showing
the reverse in specific fields or contexts (Jowkar et al., 2011; Muscio et al., 2017; Neufeld,
2016). The discrepancies include two fields where citations are reasonably reliable indicators
of quality—Biology/Biochemistry and Environment/Ecology in Iran (Jowkar et al., 2011)—so it
is possible that there are international differences in the value of research funding.

The unsurprising finding that funders can support different quality research (RQ3) aligns
with prior findings that research funders can support research with different average citation
impacts (Thelwall et al., 2016), and that the amount of research funding influences the citation
impact of the research (Muscio et al., 2017).

The finding that funded research is higher quality than unfunded research even after fac-
toring out team size (RQ4) is not directly comparable to prior studies. It contradicts claims that
the current managerial approach to research in higher education reduces the quality of
research in the social sciences by restricting the autonomy of researchers (Horta & Santos,
2020), although it is not clear whether academics with more autonomy but the same amount
of funding would produce better work. The evidence of fields in which average citation counts
are effective proxies for average quality (as conceived in the REF) for externally funded
recherche (RQ5) is also not directly comparable to prior studies.

5.2. Alternative Causes of Funded Research Being Higher Quality

The higher quality of funded research has multiple possible causes, all of which may be true to
some extent. Although it seems self-evident that funding improves research, it is not always
true (Jowkar et al., 2011; Muscio et al., 2017; Neufeld, 2016). There are many pathways that
could explain the usually positive relationship.

5.2.1.

Funders select more successful researchers to fund

Research, albeit with limited scope, suggests that funding councils may be good at exclud-
ing weak researchers but not good at identifying the very best, at least if citations are
accepted as a proxy for research quality (van den Besselaar & Leydesdorff, 2009). Assuming
that the first group, together with researchers that were unable to submit funding bids,
formed a majority or were substantially weaker than the other two groups, this would likely
translate into a statistical association between funding and researcher quality. There may
also be a REF selection effect that would strengthen the results, with stronger researchers
differentially submitting their funded research and weaker researchers often not having
funded research.

Études scientifiques quantitatives

527

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Is research funding always beneficial?

5.2.2.

Funding improves existing research

At the simplest level, funding may allow some researchers to conduct better versions of the
research that they had already intended to pursue. Par exemple, the funding might support a
larger scale survey, newer equipment, expert collaborators, or additional supporting analyses.
It seems unlikely that a project given extra funding would often become worse, Par exemple
because new equipment was bought but did not work well, or an expanded survey incorpo-
rated lower quality data collection methods in the additional areas.

5.2.3.

Funding changes the research carried out, replacing weaker (or no) with stronger work

Funding might allow a study that would be impossible for the applicant(s) without external
funding (Bloch, Graversen, & Pedersen, 2014). If the funding was for expensive equipment
or other processes (par exemple., large-scale in-person interviews) then the work seems likely to be
more original than average, assuming that few researchers in a field would have access to
funding for investigations with a similar purpose. Par exemple, perhaps an Alzheimer’s
researcher gets funding to run a large-scale genetic screening test and produces one of the
few studies on this topic. Originality is one of the three components of research quality
(Langfeldt et al., 2020), so increasing this would be enough to improve the overall quality
grade for an article. Bien sûr, funded types of research could also sometimes tend to be
weaker than unfunded research in some fields or contexts. Par exemple, funding commonly
supports PhD projects (Ates & Brechelmacher, 2013), and PhD research could be better or
worse than average, depending on the field.

5.2.4.

Funding-led research goals are more valued

Research projects that align with funders’ strategic priorities may be highly valued if assessors
accept these priorities. Although there are open call grants, some may pursue unfunded
research because of the freedom to choose their own priorities (Behrens & Gray, 2001; Cheek,
2008), so strategic goals seem likely to be more common in funded research. Funding also
generates an implicit hierarchy of research value, with even unfunded goals aligning with
societal needs potentially being undervalued (Frickel, Gibbon et al., 2010).

5.2.5.

Funding is regarded as a good in itself

Given high levels of competition for research funding, a funding declaration may be seen as an
important achievement, especially as the evaluators are mainly from a U.K. higher education
environment in which funding is encouraged and rewarded. Inversement, in funding-rich areas,
articles lacking funding may be treated with extra suspicion.

5.2.6.

Funding entails impact requirements

Although industry funding typically has commercial value as a goal, research council grants
have societal impact requirements and give resources to achieve these through dissemina-
tion activities. Ainsi, funded research may be more impactful through multiple pathways
related to the funding sources.

6. CONCLUSIONS

In the United Kingdom, there are substantial disciplinary differences in the proportions of
funded research and the extent to which funded research tends to be of higher quality than

Études scientifiques quantitatives

528

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Is research funding always beneficial?

unfunded research. Although this was only evaluated in a limited U.K. REF context, the results
suggérer, but do not prove, that there are few (and perhaps no) broad fields of research in which
funding does not help academics to produce higher quality research. The main exceptions are
a few individual funders in some contexts, and the evidence is weak for the arts and humanities
and some social sciences. De plus, as the results could be equally explained by better
researchers being more successful at attracting funding or funding improving the researchers’
outputs, no cause-and-effect relationship can be claimed. The results are not due to funded
research tending to involve larger teams because the regressions showed a residual funding
advantage after taking into account team size. Dans l'ensemble, cependant, because the results are at least
consistent with research funding adding value nearly universally across disciplines, avoiding
grants seems like a risk for all researchers, unless they have good reasons to believe that their
research is an exception.

This study does not take into account productivity and the time taken writing successful and
unsuccessful bids, so the results cannot be used for a cost–benefit analysis of funding. More
detailed research that considers the amount of funding available for each study and the role of
the funding (par exemple., improving existing research, allowing expensive studies) would be needed to
make a reasonable cost–benefit analysis to give useful information about the disciplinary
differences in the effectiveness of funding, but this seems unlikely to be possible with current
public data.

A secondary finding is that citations are not always effective proxies for average funder
qualité, especially in the social sciences, arts, and humanities. Funders and studies that use
citations as proxies for quality to assess the impact of funding should only do so for the fields
identified above where appropriately field-normalized citation counts correlate at least
moderately with quality.

CONTRIBUTIONS DES AUTEURS
Mike Thelwall: Analyse formelle, Méthodologie, Writing—original draft. Kayvan Kousha:
Writing—review & édition. Mahshid Abdoli: Analyse formelle, Writing—review & édition.
Emma Stuart: Analyse formelle, Writing—review & édition. Meiko Makita: Analyse formelle,
Writing—review & édition. Cristina I. Font-Julián: Writing–review & édition. Paul Wilson:
Writing—review & édition. Jonathan Levitt: Méthodologie, Writing—review & édition.

COMPETING INTERESTS

The authors have no competing interests.

INFORMATIONS SUR LE FINANCEMENT

This study was funded by Research England, Scottish Funding Council, Higher Education
Funding Council for Wales, and Department for the Economy, Northern Ireland as part of
the Future Research Assessment Programme (https://www.jisc.ac.uk/future-research
-assessment-programme). The content is solely the responsibility of the authors and does
not necessarily represent the official views of the funders.

DATA AVAILABILITY

The raw data were deleted before submission to follow UKRI policy for REF2021. More data
information is available in an associated report (Thelwall, Kousha et al., 2022b; with extra
information here: https://cybermetrics.wlv.ac.uk/ai/).

Études scientifiques quantitatives

529

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Is research funding always beneficial?

RÉFÉRENCES

Aagaard, K., Mongeon, P., Ramos-Vielba, JE., & Thomas, D. UN.
(2021). Getting to the bottom of research funding: Acknowl-
edging the complexity of funding dynamics. PLOS ONE, 16(5),
e0251488. https://doi.org/10.1371/journal.pone.0251488,
PubMed: 33979400

Ali, M.. M., Bhattacharyya, P., & Olejniczak, UN. J.. (2010). The effects
of scholarly productivity and institutional characteristics on the
distribution of federal research grants. Journal of Higher Educa-
tion, 81(2), 164–178. https://doi.org/10.1080/00221546.2010
.11779047

Alkhawtani, R.. H. M., Kwee, T. C., & Kwee, R.. M.. (2020). Funding
of radiology research: Frequency and association with citation
rate. American Journal of Roentgenology, 215(5), 1286–1289.
https://doi.org/10.2214/AJR.20.22786, PubMed: 32877249

Álvarez-Bornstein, B., Díaz-Faes, UN. UN., & Bordons, M.. (2019).
What characterises funded biomedical research? Evidence from
a basic and a clinical domain. Scientometrics, 119(2), 805–825.
https://doi.org/10.1007/s11192-019-03066-3

Amiri, UN. R., Kanesalingam, K., Cro, S., & Casey, UN. T. H. (2014).
Does source of funding and conflict of interest influence the
outcome and quality of spinal research? Spine Journal, 14(2),
308–314. https://doi.org/10.1016/j.spinee.2013.10.047,
PubMed: 24231776

Ates, G., & Brechelmacher, UN. (2013). Academic career paths. In U.
Teichler & E. Höhle (Éd.), The work situation of the academic
profession in Europe: Findings of a survey in twelve countries
(pp. 13–35). Springer. https://doi.org/10.1007/978-94-007-5977
-0_2

Banchoff, T. (2002). Institutions, inertia and European Union
research policy. Journal of Common Market Studies, 40(1),
1–21. https://doi.org/10.1111/1468-5965.00341

Beatty, S. (2017). New on Scopus: Link to datasets, search funding
acknowledgements and find more CiteScore transparency.
Scopus Blog. https://web.archive.org/web/20170920090909
/ https:// blog.scopus.com/posts/new-on-scopus-link-to-datasets
-search-funding-acknowledgements-and-find-more-citescore
Behrens, T. R., & Gray, D. Ô. (2001). Unintended consequences of
cooperative research: Impact of industry sponsorship on climate
for academic freedom and other graduate student outcome.
Research Policy, 30(2), 179–199. https://doi.org/10.1016/S0048
-7333(99)00112-2

Bentley, P.. J., & Kyvik, S. (2012). Academic work from a compara-
tive perspective: A survey of faculty working time across 13
des pays. Higher Education, 63(4), 529–547. https://est ce que je.org/10
.1007/s10734-011-9457-4

Berman, J.. J., Borkowski, UN., Rachocka, H., & Moore, G. W.
(1995). Impact of unfunded research in medicine, pathology,
Journal, 88(3), 295–299.
and surgery. Southern Medical
https://doi.org/10.1097/00007611-199503000-00009, PubMed:
7886525

Bloch, C., Graversen, E. K., & Pedersen, H. S. (2014). Competitive
research grants and their impact on career performance.
Minerva, 52(1), 77–96. https://doi.org/10.1007/s11024-014
-9247-0

Bloch, C., Schneider, J.. W., & Sinkjær, T. (2016). Size, accumulation
and performance for research grants: Examining the role of size
for centres of excellence. PLOS ONE, 11(2), e0147726. https://
doi.org/10.1371/journal.pone.0147726, PubMed: 26862907
Bloch, C., & Sørensen, M.. P.. (2015). The size of research funding:
Trends and implications. Science and Public Policy, 42(1),
30–43. https://doi.org/10.1093/scipol/scu019

Bodensteiner, J.. B. (1995). The saga of the septum pellucidum: A tale
of unfunded clinical investigations. Journal of Child Neurology,
10(3), 227–231. https://doi.org/10.1177/088307389501000313,
PubMed: 7642894

Borkowski, UN., Berman, J.. J., & Moore, G. W. (1992). Research by
pathologists not funded by external grant agencies: A success
story. Modern Pathology, 5(5), 577–579. PubMed: 1344824
Bornmann, L., & Daniel, H. D. (2006). Selecting scientific excel-
lence through committee peer review—A citation analysis of
publications previously published to approval or rejection of
post-doctoral research fellowship applicants. Scientometrics,
68(3), 427–440. https://doi.org/10.1007/s11192-006-0121-1
Brookes, M.. J., Farr, UN., Phillips, C. J., & Trudgill, N. J.. (2021).
Management of iron deficiency anaemia in secondary care
across England between 2012 et 2018: A real-world analysis
of hospital episode statistics. Frontline Gastroenterology, 12(5),
363–369. https://doi.org/10.1136/flgastro-2020-101506,
PubMed: 35401966

Cheek, J.. (2008). The practice and politics of funded qualitative
recherche. In N. K. Denzin & Oui. S. Lincoln (Éd.), Strategies of qual-
itative inquiry (pp. 45–74). Sage.

Chubin, D. E. (1994). Grants peer review in theory and practice.
Evaluation Review, 18(1), 20–30. https://est ce que je.org/10.1177
/0193841X9401800103

Chudnovsky, D., López, UN., Rossi, M.. UN., & Ubfal, D. (2008).
Money for science? The impact of research grants on academic
output. Fiscal Studies, 29(1), 75–87. https://doi.org/10.1111/j
.1475-5890.2008.00069.X

Crosland, M., & Galvez, UN. (1989). The emergence of research
grants within the prize system of the French Academy of Sci-
ences, 1795–1914. Social Studies of Science, 19(1), 71–100.
https://doi.org/10.1177/030631289019001002

Cruz-Castro, L., Ginther, D. K., & Sanz-Menendez, L. (2022).
Gender and underrepresented minority differences in research
funding. https://www.nber.org/system/files/working_papers
/w30107/w30107.pdf. https://doi.org/10.3386/w30107

Currie-Alder, B. (2015). Research for the developing world: Public
funding from Australia, Canada, and the UK. Oxford: Oxford
U n i v e r s i t y P r e s s . h t t p s : / / d o i . o r g / 1 0 . 1 0 9 3 / a c p r o f : o s o
/9780198742937.001.0001

Defazio, D., Lockett, UN., & Wright, M.. (2009). Funding incentives,
collaborative dynamics and scientific productivity: Evidence
from the EU framework program. Research Policy, 38(2),
293–305. https://doi.org/10.1016/j.respol.2008.11.008

Demeritt, D. (2000). The new social contract for science: Account-
ability, relevance, and value in US and UK science and research
politique. Antipode, 32(3), 308–329. https://doi.org/10.1111/1467
-8330.00137

Drennan, J., Clarke, M., Hyde, UN., & Politis, Oui. (2013). The research
function of the academic profession in Europe. In U. Teichler &
E. Höhle (Éd.), The work situation of the academic profession in
Europe: Findings of a survey in twelve countries (pp. 109–136).
Springer. https://doi.org/10.1007/978-94-007-5977-0_6

Ebadi, UN., & Schiffauerova, UN. (2016). How to boost scientific pro-
duction? A statistical analysis of research funding and other
influencing factors. Scientometrics, 106(3), 1093–1116. https://
doi.org/10.1007/s11192-015-1825-x

Edwards, R.. (2022). Why do academics do unfunded research?
Resistance, compliance and identity in the UK neo-liberal uni-
versity. Studies in Higher Education, 47(4), 904–914. https://est ce que je
.org/10.1080/03075079.2020.1817891

Études scientifiques quantitatives

530

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Is research funding always beneficial?

El-Sawi, N. JE., Sharp, G. F., & Gruppen, L. D. (2009). A small grants
program improves medical education research productivity.
Academic Medicine, 84(Supplément. 10), S105–S108. https://doi.org
/10.1097/ACM.0b013e3181b3707d, PubMed: 19907368

Enger, S. G., & Castellacci, F. (2016). Who gets Horizon 2020
research grants? Propensity to apply and probability to succeed
in a two-step analysis. Scientometrics, 109(3), 1611–1638.
https://doi.org/10.1007/s11192-016-2145-5

Ernst, UN. UN., Houry, D., & Blanc, S. J.. (1997). Research funding in
the four major emergency medicine journals. American Journal
of Emergency Medicine, 15(3), 268–270. https://est ce que je.org/10
.1016/S0735-6757(97)90011-5, PubMed: 9148983

Fang, F. C., Bowen, UN., & Casadevall, UN. (2016). NIH peer review
percentile scores are poorly predictive of grant productivity.
eLife, 5, e13323. https://doi.org/10.7554/eLife.13323, PubMed:
26880623

Franssen, T., Scholten, W., Hessels, L. K., & de Rijcke, S. (2018).
The drawbacks of project funding for epistemic innovation:
Comparing institutional affordances and constraints of different
types of research funding. Minerva, 56(1), 11–33. https://doi.org
/10.1007/s11024-017-9338-9, PubMed: 29449745

Frickel, S., Gibbon, S., Howard, J., Kempner, J., Ottinger, G., &
Hess, D. J.. (2010). Undone science: Charting social movement
and civil society challenges to research agenda setting. Science,
Technologie, & Human Values, 35(4), 444–473. https://est ce que je.org/10
.1177/0162243909345836, PubMed: 32099268

Fumasoli, T., Goastellec, G., & Kehm, B. M.. (2015). Academic
careers and work in Europe: Trends, challenges, perspectives.
In T. Fumasoli (Ed.), Academic work and careers in Europe:
Trends, challenges, perspectives (pp. 201–214). Springer.
https://doi.org/10.1007/978-3-319-10720-2_10

Gallo, S. UN., Carpenter, UN. S., Irwin, D., McPartland, C. D., Travis,
J., … Glisson, S. R.. (2014). The validation of peer review through
research impact measures and the implications for funding
strategies. PLOS ONE, 9(9), e106474. https://doi.org/10.1371
/journal.pone.0106474, PubMed: 25184367

Gaughan, M., & Bozeman, B. (2002). Using curriculum vitae to
compare some impacts of NSF research grants with research cen-
ter funding. Research Evaluation, 11(1), 17–26. https://est ce que je.org/10
.3152/147154402781776952

Goddard, UN. F., James, M.. W., McIntyre, UN. S., & Scott, B. B. (2011).
Guidelines for the management of iron deficiency anaemia. Gut,
60(10), 1309–1316. https://doi.org/10.1136/gut.2010.228874,
PubMed: 21561874

Godin, B. (2003). The impact of research grants on the productivity
and quality of scientific research. Ottawa: INRS Working Paper,
2003.

Gush, J., Jaffe, UN., Larsen, V., & Laws, UN. (2018). The effect of public
funding on research output: The New Zealand Marsden Fund.
New Zealand Economic Papers, 52(2), 227–248. https://doi.org
/10.1080/00779954.2017.1325921

Gutiérrez, P.. UN., Perez-Ortiz, M., Sanchez-Monedero, J.,
Fernandez-Navarro, F., & Hervas-Martinez, C. (2015). Ordinal
regression methods: Survey and experimental study. IEEE Trans-
actions on Knowledge and Data Engineering, 28(1), 127–146.
https://doi.org/10.1109/TKDE.2015.2457911

Győrffy, B., Herman, P., & Szabó, je. (2020). Research funding: Passé
performance is a stronger predictor of future scientific output
than reviewer scores. Journal of Informetrics, 14(3), 101050.
https://doi.org/10.1016/j.joi.2020.101050

Hayden, E. C. (2015). Racial bias continues to haunt NIH grants.
Nature, 527(7578), 286–287. https://doi.org/10.1038/527286a,
PubMed: 26581270

Herbert, D. L., Barnett, UN. G., Clarke, P., & Graves, N. (2013).
On the time spent preparing grant proposals: An observational
study of Australian researchers. BMJ Open, 3(5), e002800.
https://doi.org/10.1136/ bmjopen-2013-002800, PubMed:
23793700

Herbert, D. L., Coveney, J., Clarke, P., Graves, N., & Barnett, UN. G.
(2014). The impact of funding deadlines on personal workloads,
stress and family relationships: A qualitative study of Australian
chercheurs. BMJ Open, 4(3), e004462. https://doi.org/10.1136
/bmjopen-2013-004462, PubMed: 24682577

Heyard, R., & Hottenrott, H.

(2021). The value of research
funding for knowledge creation and dissemination: A study of
SNSF Research Grants. Humanities and Social Sciences Com-
munications, 8(1), 1–16. https://doi.org/10.1057/s41599-021
-00891-X

Hicks, D. (2012). Performance-based university research funding
systèmes. Research Policy, 41(2), 251–261. https://est ce que je.org/10
.1016/j.respol.2011.09.007

Higgins, UN., Downes, C., Varley, J., Doherty, C. P., Bégley, C., &
Elliott, N. (2019). Evidence-based practice among epilepsy spe-
cialist nurses in the Republic of Ireland: Findings from the SENsE
étude. Journal of Nursing Management, 27(4), 840–847. https://
doi.org/10.1111/jonm.12747, PubMed: 30615228

Horta, H., Huisman, J., & Heitor, M.. (2008). Does competitive
research funding encourage diversity in higher education? Sci-
ence and Public Policy, 35(3), 146–158. https://doi.org/10.3152
/030234208X299044

Horta, H., & Santos, J.. M.. (2020). Organisational factors and aca-
demic research agendas: An analysis of academics in the social
sciences. Studies in Higher Education, 45(12), 2382–2397.
https://doi.org/10.1080/03075079.2019.1612351

Hottenrott, H., & Lawson, C. (2017). Fishing for complementarities:
Research grants and research productivity. International Journal
of Industrial Organization, 51(1), 1–38. https://est ce que je.org/10.1016/j
.ijindorg.2016.12.004

Hottenrott, H., & Thorwarth, S. (2011). Industry funding of univer-
sity research and scientific productivity. Kyklos, 64(4), 534–555.
https://doi.org/10.1111/j.1467-6435.2011.00519.x

Hussinger, K., & Carvalho, J.. N. (2022). The long-term effect of
research grants on the scientific output of university professors.
Industry and Innovation, 29(4), 463–487. https://est ce que je.org/10
.1080/13662716.2021.1990023

Imran, N., Aamer, JE., Sharif, M.. JE., Bodla, Z. H., & Naveed, S.
(2020). Psychological burden of quarantine in children and
adolescents: A rapid systematic review and proposed solutions.
Pakistan Journal of Medical Sciences, 36(5), 1106–1116. https://
doi.org/10.12669/pjms.36.5.3088, PubMed: 32704298

Jappe, UN., & Heinze, T. (2023). Research funding in the context of
high institutional stratification. Policy scenarios for Europe
based on insights from the United States. In B. Lepori, B.
Jongbloed, & D. Hicks (Éd.), Handbook of public research
funding (pp. 203–220). Edward Elgar. https://doi.org/10.4337
/9781800883086.00020

Johnson, UN. J., Mathews, V. P., & Artemakis, UN. (2002). Américain
Society of Neuroradiology research survey 2001. Academic
Radiology, 9(7), 810–814. https://doi.org/10.1016/S1076
-6332(03)80353-7, PubMed: 12139097

Johnson, UN. T., & Hirt, J.. B. (2011). Reshaping academic capitalism
to meet development priorities: The case of public universities in
Kenya. Higher Education, 61(4), 483–499. https://est ce que je.org/10
.1007/s10734-010-9342-6

Jonker, L., Cox, D., & Maréchal, G. (2011). Considerations, clues
and challenges: Gaining ethical and trust research approval

Études scientifiques quantitatives

531

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Is research funding always beneficial?

when using the NHS as a research setting. Radiography, 17(3),
260–264. https://doi.org/10.1016/j.radi.2010.12.004

Jowkar, UN., Didegah, F., & Gazni, UN. (2011). The effect of funding
on academic research impact: A case study of Iranian publica-
tion. Aslib Proceedings, 63(6), 593–602. https://est ce que je.org/10
.1108/00012531111187243

Kayrooz, C., Åkerlind, G. S., & Tight, M.. (Éd.) (2007). Autonomy in
social science research, volume 4: The view from United Kingdom
and Australian universities. Bradford, ROYAUME-UNI: Emerald Group Publish-
ing Limited. https://doi.org/10.1016/S1479-3628(2007)4

Kim, K. S., Chung, J.. H., Jo, J.. K., Kim, J.. H., Kim, S., … Lee, S. W.
(2018). Quality of randomized controlled trials published in the
International Urogynecology Journal 2007–2016. International
Urogynecology Journal, 29(7), 1011–1017. https://est ce que je.org/10
.1007/s00192-017-3465-6, PubMed: 28884346

Langfeldt, L., Bloch, C. W., & Sivertsen, G. (2015). Options and
limitations in measuring the impact of research grants—
Evidence from Denmark and Norway. Research Evaluation,
24(3), 256–270. https://doi.org/10.1093/reseval/rvv012

Langfeldt, L., Nedeva, M., Sörlin, S., & Thomas, D. UN. (2020).
Co-existing notions of research quality: A framework to study
context-specific understandings of good research. Minerva,
58(1), 115–137. https://doi.org/10.1007/s11024-019-09385-2
Laudel, G. (2005). Is external research funding a valid indicator for
research performance? Research Evaluation, 14(1), 27–34.
https://doi.org/10.3152/147154405781776300

Lepori, B., van den Besselaar, P., Dinges, M., Potì, B., Reale, E.,
… van der Meulen, B.
(2007). Comparing the evolution of
national research policies: What patterns of change? Science
and Public Policy, 34(6), 372–388. https://doi.org/10.3152
/030234207X234578

Levitt, J.. M.. (2011). Are funded articles more highly cited than
unfunded articles? A preliminary investigation. In E. Noyons, P..
Ngulube, & J.. Leta (Éd.), Proceedings of ISSI 2011 (pp. 1013–1015).
ISSI Press.

Levitt, M., & Levitt, J.. M.. (2017). Future of fundamental discovery in
US biomedical research. Proceedings of the National Academy of
les sciences, 114(25), 6498–6503. https://doi.org/10.1073/pnas
.1609996114, PubMed: 28584129

Lewison, G., & Dawson, G. (1998). The effect of funding on the
outputs of biomedical research. Scientometrics, 41(1–2), 17–27.
https://doi.org/10.1007/BF02457963

Liaw, L., Freedman, J.. E., Becker, L. B., Mehta, N. N., & Liscum, L.
(2017). Peer review practices for evaluating biomedical research
subventions: A scientific statement from the American Heart Associa-
tion. Circulation Research, 121(4), e9–e19. https://est ce que je.org/10
.1161/RES.0000000000000158, PubMed: 28684631

Lim, K. J., Yoon, D. Y., Yun, E. J., Seo, Oui. L., Baek, S., … Kim, S. S.
(2012). Characteristics and trends of radiology research: A survey
of original articles published in AJR and Radiology between 2001
et 2010. Radiology, 264(3), 796–802. https://doi.org/10.1148
/radiol.12111976, PubMed: 22919040

Peut, T. V., Agan, D. L., Clopton, P., Collins, G., & DeMaria, UN. N.
(2013). The magnitude and nature of unfunded published cardio-
vascular research. Journal of the American College of Cardiology,
61(3), 275–281. https://doi.org/10.1016/j.jacc.2012.10.013,
PubMed: 23219303

Martín-Martín, UN., Thelwall, M., Orduna-Malea, E., & Delgado
López-Cózar, E. (2021). Google Scholar, Microsoft Academic,
Scopus, Dimensions, Web de la Science, and OpenCitations’ COCI:
A multidisciplinary comparison of coverage via citations. Scien-
tometrics, 126(1), 871–906. https://doi.org/10.1007/s11192-020
-03690-4, PubMed: 32981987

McCullough, R.. (2021). Improvements to funding data in Scopus:
Now 16.5M articles with funding information and easier to iden-
tify funded research. Scopus Blog. https://web.archive.org/web
/20220630145119/https://blog.scopus.com/posts/improvements
-to-funding-data-in-scopus-now-165m-articles-with-funding
-information-and-easier

Metcalfe, UN. S. (2010). Revisiting academic capitalism in Canada:
No longer the exception. Journal of Higher Education, 81(4),
489–514. https://doi.org/10.1080/00221546.2010.11779062
Muscio, UN., Ramaciotti, L., & Rizzo, U. (2017). The complex rela-
tionship between academic engagement and research output:
Evidence from Italy. Science and Public Policy, 44(2), 235–245.
https://doi.org/10.1093/scipol/scw057

Mustar, P., & Larédo, P.. (2002). Innovation and research policy in
France (1980–2000) or the disappearance of the Colbertist state.
Research Policy, 31(1), 55–72. https://doi.org/10.1016/S0048
-7333(01)00107-X

Neufeld, J.. (2016). Determining effects of individual research
grants on publication output and impact: The case of the
Emmy Noether Programme (German Research Foundation).
Research Evaluation, 25(1), 50–61. https://est ce que je.org/10.1093
/reseval/rvv029

Nyamapfene, UN. Z. (2018). Teaching-only academics in a research
intensive university: From an undesirable to a desirable academic
identité. Exeter, ROYAUME-UNI: University of Exeter.

OECD. (2014). Promoting research excellence: New approaches to
funding. OECD. https://www.oecd-ilibrary.org/science-and
-technology/promoting-research-excellence_9789264207462
-dans. https://doi.org/10.1787/9789264207462-en

Olive, V. (2017). How much is too much? Cross-subsidies from
teaching to research in British universities. Oxford: Higher Educa-
tion Policy Institute.

Paul-Hus, UN., Díaz-Faes, UN. UN., Sainte-Marie, M., Desrochers, N.,
Costas, R., & Larivière, V. (2017). Beyond funding: Acknowledge-
ment patterns in biomedical, natural and social sciences. PLOS
ONE, 12(10), e0185578. https://doi.org/10.1371/journal.pone
.0185578, PubMed: 28976996

Peritz, B. (1990). The citation impact of funded and unfunded
research in economics. Scientometrics, 19(3–4), 199–206.
https://doi.org/10.1007/BF02095347

Perneger, T. V. (1998). What’s wrong with Bonferroni adjustments.
British Medical Journal, 316(7139), 1236–1238. https://doi.org
/10.1136/bmj.316.7139.1236, PubMed: 9553006

Polster, C. (2007). The nature and implications of the growing
importance of research grants to Canadian universities and aca-
demics. Higher Education, 53(5), 599–622. https://est ce que je.org/10
.1007/s10734-005-1118-z

Qi, J., & Wei, C. (2021). Performance evaluation of climate-
adaptive natural ventilation design: A case study of semi-open
public cultural building. Indoor and Built Environment, 30(10),
1714–1724. https://doi.org/10.1177/1420326X20961495

REF. (2020). Panel criteria and working methods. https://www.ref.ac
.uk/media/1450/ref-2019_02-panel-criteria-and-working
-methods.pdf

REF. (2022). Guidance on the REF 2021 résultats. https://ref.ac.uk

/guidance-on-results/guidance-on-ref-2021-results/

Rigby, J.. (2011). Systematic grant and funding body acknowl-
edgement data for publications: New dimensions and new
controversies for research policy and evaluation. Research
E v a l u a t i on, 20 ( 5) , 3 6 5 –3 7 5 . h t t ps : / / d oi . o rg / 10 . 31 5 2
/095820211X13164389670392

Roach, J.. W., Skaggs, D. L., Sponseller, P.. D., & MacLeod, L. M..
(2008). Is research presented at the scoliosis research society

Études scientifiques quantitatives

532

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Is research funding always beneficial?

annual meeting influenced by industry funding? Spine, 33(20),
2208–2212. https://doi.org/10.1097/ BRS.0b013e31817bd8f8,
PubMed: 18725871

Roshani, S., Bagherylooieh, M.. R., Mosleh, M., & Coccia, M..
(2021). What is the relationship between research funding and
citation-based performance? A comparative analysis between
critical disciplines. Scientometrics, 126(9), 7859–7874. https://
doi.org/10.1007/s11192-021-04077-9

Sandström, U. (2009). Research quality and diversity of funding: UN
model for relating research money to output of research. Scien-
tometrics, 79(2), 341–349. https://doi.org/10.1007/s11192-009
-0422-2

Saygitov, R.. T. (2018). The impact of grant funding on the publica-
tion activity of awarded applicants: A systematic review of com-
parative studies and meta-analytical estimates. bioRxiv. https://
doi.org/10.1101/354662

Schneider, J.. W., & van Leeuwen, T. N. (2014). Analysing robust-
ness and uncertainty levels of bibliometric performance statistics
supporting science policy. A case study evaluating Danish post-
doctoral funding. Research Evaluation, 23(4), 285–297. https://
doi.org/10.1093/reseval/rvu016

Sedney, C. L., Daffner, S. D., Stefanko, J.. J., Abdelfattah, H., Emery,
S. E., & France, J.. C. (2016). Fracture of fusion mass after hard-
ware removal in patients with high sagittal imbalance. Journal de
Neurochirurgie: Spine, 24(4), 639–643. https://doi.org/10.3171
/2015.7.SPINE15153, PubMed: 26682604

Seeber, M., Vlegels, J., & Cattaneo, M.. (2022). Conditions that do or
do not disadvantage interdisciplinary research proposals in pro-
ject evaluation. Journal of the Association for Information Science
and Technology, 73(8), 1106–1126. https://doi.org/10.1002/asi
.24617

Shandhi, M.. M.. H., Goldsack, J.. C., Ryan, K., Bennion, UN., Kotla,
UN. V., … Dunn, J.. (2021). Recent academic research on clinically
relevant digital measures: Systematic review. Journal of Medical
Internet Research, 23(9), e29875. https://doi.org/10.2196/29875,
PubMed: 34524089

Shimada, Oui. UN., Tsukada, N., & Suzuki, J.. (2017). Promoting diver-
sity in science in Japan through mission-oriented research grants.
Scientometrics, 110(3), 1415–1435. https://doi.org/10.1007
/s11192-016-2224-7

Silberman, E. K., & Snyderman, D. UN. (1997). Research without
external funding in North American psychiatry. American Journal
of Psychiatry, 154(8), 1159–1160. https://doi.org/10.1176/ajp
.154.8.1159, PubMed: 9247409

Sinha, Y., Iqbal, F. M., Spence, J.. N., & Richard, B. (2016). A bib-
liometric analysis of the 100 most-cited articles in rhinoplasty.
Plastic and Reconstructive Surgery Global Open, 4(7), e820.
https://doi.org/10.1097/GOX.0000000000000834, PubMed:
27536499

Slaughter, S., & Leslie, L. L. (2001). Expanding and elaborating the
concept of academic capitalism. Organization, 8(2), 154–161.
https://doi.org/10.1177/1350508401082003

Stein, M.. D., Rubenstein, L., & Wachtel, T. J.. (1993). Who pays for
published research? JAMA, 269(6), 781–782. https://est ce que je.org/10
.1001/jama.1993.03500060081035, PubMed: 8423661

Takemura, S. (2021). Health research policy and systems in Japan:
A review focused on the Health, Labour and Welfare Sciences
Research Grants. Journal of the National Institute of Public
Health, 70(1), 2–12.

Tatsioni, UN., Vavva, E., & Ioannidis, J.. P.. UN. (2010). Sources of
funding for Nobel Prize-winning work: Public or private? FASEB
Journal, 24(5), 1335–1339. https://doi.org/10.1096/fj.09-148239,
PubMed: 20056712

Tellmann, S. M.. (2022). The societal territory of academic disciplines:
How disciplines matter to society. Minerva, 60(2), 159–179.
https://doi.org/10.1007/s11024-022-09460-1

Thelwall, M.. (2017). Three practical field normalised alternative indi-
cator formulae for research evaluation. Journal of Informetrics,
11(1), 128–151. https://doi.org/10.1016/j.joi.2016.12.002

Thelwall, M., & Fairclough, R..

(2017). The accuracy of confi-
dence intervals for field normalised indicators.
Journal de
Informetrics, 11(2), 530–540. https://doi.org/10.1016/j.joi.2017
.03.004

Thelwall, M., Kousha, K., Abdoli, M., Stuart, E., Makita, M., …
Levitt, J.. (2022un). Why are co-authored academic articles more
cited: Higher quality or larger audience? arXiv, arXiv:2212.06571.
https://doi.org/10.48550/arXiv.2212.06571

Thelwall, M., Kousha, K., Abdoli, M., Stuart, E., Makita, M., …
Levitt, J.. (2022b). Can REF output quality scores be assigned by
AI? Experimental evidence. arXiv, arXiv:2212.08041. https://est ce que je
.org/10.48550/arXiv.2212.08041

Thelwall, M., Kousha, K., Dinsmore, UN., & Dolby, K. (2016). Alter-
native metric indicators for funding scheme evaluations. Aslib
Journal of Information Management, 68(1), 2–18. https://doi.org
/10.1108/AJIM-09-2015-0146

Thelwall, M., & Maflahi, N. (2020). Academic collaboration
rates and citation associations vary substantially between coun-
tries and fields. Journal of the Association for Information Sci-
ence and Technology, 71(8), 968–978. https://doi.org/10.1002
/asi.24315

Thyer, B. UN. (2011). Harmful effects of federal research grants.
Social Work Research, 35(1), 3–7. https://doi.org/10.1093/swr
/35.1.3

Tricco, UN. C., Thomas, S. M., Antony, J., Rios, P., Robson, R., …
Straus, S. E. (2017). Strategies to prevent or reduce gender bias
in peer review of research grants: A rapid scoping review. PLOS
ONE, 12(1), e0169718. https://doi.org/10.1371/journal.pone
.0169718, PubMed: 28061509

Underhill, L. J., Dols, W. S., Lee, S. K., Fabian, M.. P., & Levy, J.. je.
(2020). Quantifying the impact of housing interventions on
indoor air quality and energy consumption using coupled simu-
lation models. Journal of Exposure Science & Environnemental
Epidemiology, 30(3), 436–447. https://doi.org/10.1038/s41370
-019-0197-3, PubMed: 31959901

Vaduganathan, M., Nagarur, UN., Qamar, UN., Patel, R.. B., Navar,
UN. M., … Butler, J.. (2018). Availability and use of shared data from
cardiometabolic clinical trials. Circulation, 137(9), 938–947.
https://doi.org/10.1161/CIRCULATIONAHA.117.031883,
PubMed: 29133600

van Arensbergen, P., & van den Besselaar, P.. (2012). The selection
of scientific talent in the allocation of research grants. Higher
Education Policy, 25(3), 381–405. https://doi.org/10.1057/hep
.2012.15

van den Besselaar, P., & Leydesdorff, L. (2009). Past performance,
peer review and project selection: A case study in the social and
behavioral sciences. Research Evaluation, 18(4), 273–288.
https://doi.org/10.3152/095820209X475360

van Driel, M., Deckx, L., Cooke, G., Pirotta, M., Gill, G. F., &
Winzenberg, T. (2017). Growing and retaining general practice
research leaders in Australia: How can we do better? Australian
Family Physician, 46(10), 757–762. PubMed: 29036777

Wang, J.. (2013). Citation time window choice for research impact
evaluation. Scientometrics, 94(3), 851–872. https://est ce que je.org/10
.1007/s11192-012-0775-9

Wang, L., Wang, X., Piro, F. N., & Philipsen, N. J.. (2020). The effect
of competitive public funding on scientific output: A comparison

Études scientifiques quantitatives

533

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

4
2
5
0
1
2
1
3
6
4
5
7
q
s
s
_
un
_
0
0
2
5
4
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Is research funding always beneficial?

between China and the EU. Research Evaluation, 29(4),
418–429. https://doi.org/10.1093/reseval/rvaa023

Whitley, R.. (2000). The intellectual and social organization of the

sciences. Presse universitaire d'Oxford.

Xu, G., Zhang, Z., Lv, Q., Li, Y., Ye, R., … Liu, X. (2014). NSFC
health research funding and burden of disease in China. PLOS
ONE, 9(11), e111458. https://doi.org/10.1371/journal.pone
.0111458, PubMed: 25369330

Yan, E., Wu, C., & Song, M.. (2018). The funding factor: UN
cross-disciplinary examination of the association between
research funding and citation impact. Scientometrics, 115(1),
369–384. https://doi.org/10.1007/s11192-017-2583-8

Zhao, D. (2010). Characteristics and impact of grant-funded
recherche: A case study of the library and information science
field. Scientometrics, 84(2), 293–306. https://doi.org/10.1007
/s11192-010-0191-y

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d