ARTICLE DE RECHERCHE - Recherche en IA spécialisée au MIT

ARTICLE DE RECHERCHE

Mapping scholarly publications related to the
Sustainable Development Goals: Do independent
bibliometric approaches get the same results?

un accès ouvert

journal

University of Bergen Library, University of Bergen, Bergen, Norway

Caroline S. Armitage

, Marta Lorenz

, and Susanne Mikki

Mots clés: bibliométrie, SDG, scientometrics, sustainability, Sustainable Development Goals

ABSTRAIT

Many research and higher education institutions are interested in their contribution to
achieving the United Nation’s Sustainable Development Goals (SDG). Commercial services
from Elsevier and Times Higher Education are addressing this by developing bibliometric
queries for measuring SDG-related publications and SDG university rankings. Cependant, tel
services should be evaluated carefully before use due to the challenging nature of interpreting
the SDGs, delimiting relevance, and building queries. The aim of this bibliometric study was to
build independent queries to find scholarly publications related to SDG 1, SDG 2, SDG 3,
SDG 7, SDG 13, and SDG 14 using a consistent method based on SDG targets and indicators
(the Bergen approach), and compare sets of publications retrieved by the Bergen and Elsevier
approaches. Our results show that approach made a large difference, with little overlap
in publications retrieved by the two approaches. We further demonstrate that different
approaches can alter resulting country rankings. Choice of search terms, how they are
combined, and query structure play a role, related to differing interpretations of the SDGs
and viewpoints on relevance. Our results suggest that currently available SDG rankings and
tools should be used with caution at their current stage of development.

INTRODUCTION

Dans 2015, the Sustainable Development Goals (SDGs) were adopted by the United Nations
General Assembly as part of the 2030 Agenda for Sustainable Development. While their prede-
cessors, the Millennium Development Goals, had success in tackling problems such as poverty,
gender inequality, and disease, the SDGs were created with the reflection that more needed to
be done “to integrate the economic, social and environmental aspects of sustainable develop-
ment” (United Nations, 2015un, 2015b). Le 17 SDGs therefore cover several broad themes and
contain interrelated targets, aiming to stimulate action for people, planet, prosperity, peace, et
partnership (United Nations, 2015b; Chiffre 1).

While the SDGs require political action, research and development of technology are also
essential for achieving the goals. Research and technology are specifically mentioned in some of
the targets (United Nations, 2015b) and are implicit in many others: Par exemple, materials
science is necessary for the development of clean, efficient energy systems (Chu, Cui, & Liu,
2017), relevant for SDG 7. Further selected examples include medical research for achieving
targets under SDG 3, biological research for sustainable management and conservation of living
ressources (SDGs 14 et 15), and social sciences research to inform the governance aspects of all

Citation: Armitage, C. S., Lorenz, M., &
Mikki, S. (2020). Mapping scholarly
publications related to the Sustainable
Development Goals: Do independent
bibliometric approaches get the same
résultats? Études scientifiques quantitatives,
1(3), 1092–1108. https://est ce que je.org/10.1162/
qss_a_00071

EST CE QUE JE:
https://doi.org/10.1162/qss_a_00071

Informations complémentaires:
https://www.mitpressjournals.org/doi/
suppl/10.1162/qss_a_00071

Reçu: 20 Janvier 2020
Accepté: 29 Avril 2020

Auteur correspondant:
Caroline S. Armitage
caroline.s.armitage@gmail.com

Éditeur de manipulation:
Ludo Waltman

droits d'auteur: © 2020 Caroline S.
Armitage, Marta Lorenz, and Susanne
Mikki. Published under a Creative
Commons Attribution 4.0 International
(CC PAR 4.0) Licence.

La presse du MIT

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

1
3
1
0
9
2
1
8
6
9
9
3
0
q
s
s
_
un
_
0
0
0
7
1
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Mapping scholarly publications related to the Sustainable Development Goals

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

1
3
1
0
9
2
1
8
6
9
9
3
0
q
s
s
_
un
_
0
0
0
7
1
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Chiffre 1. The UN’s Sustainable Development Goals. Each goal consists of targets, giving 169 targets in total (the SDG icons and logo are
property of the United Nations, reprinted for informational purposes in accordance with United Nations Department of Global
Communications (2019); all rights reserved).

SDGs. Research collaboration and inter- and multidisciplinary research are also highly relevant
and necessary for progress in sustainability (Leal Filho, Azeiteiro, et coll., 2018; Reid, Bréchignac,
& Tseh Lee, 2009). And although the process has its difficulties, scientific research provides
objective input for political action via the science–policy interface (Gluckman, 2016).
Research and higher education institutions can thus contribute to achieving the SDGs through
research and other engagement (also see discussion in Körfgen, Förster, et coll., 2018). The United
Nations recognizes this via the United Nations Academic Impact initiative, stating that “The
work of these institutions is vital to achieving the Sustainable Development Goals as they serve
as incubators of new ideas, inventions and solutions to the many global challenges we face”
(United Nations, 2019un).

Once research institutions commit to supporting the SDGs, they likely have an interest in
examining their publishing output related to them. This may be exploratory; Par exemple, lequel
SDG-related research areas they focus on, how much they collaborate with specific partners, ou
to find relevant publications to make a policy brief. They may also wish to use the results to
highlight their contribution to tackling societal and environmental problems, something that
can help in building reputation and publicity. Some institutions are also interested in evaluation;
the ability to present a high number in SDG-contribution rankings or otherwise evaluate how the
institution compares to others (benchmarking). Interest in such questions is indicated by the
development of commercial services for SDG rankings (for example the Times Higher
Éducation [THE] University SDG Rankings) and services for finding and quantifying SDG-related
publications, such as the SDG “research areas” in SciVal by Elsevier.

We believe that any services that measure or map SDG contributions should be evaluated
carefully, because tools for ranking and quantification of research can have wide impacts.
University rankings can influence strategic planning, institutional reorganization, and higher
education priorities (Hazelkorn, 2009), and SciVal is advertised as allowing one to “visualize

Études scientifiques quantitatives

1093

Mapping scholarly publications related to the Sustainable Development Goals

your research performance, benchmark relative to peers, develop strategic partnerships, iden-
tify and analyse new, emerging research trends […]» (Elsevier, 2019). These services thus have
the potential to influence the evaluations, strategy, politique, and reputation of institutions, et
thus have far-reaching impacts on research and society.

Both of the commercial services mentioned are based either entirely (SciVal) or partially
(THE) on Boolean search queries containing SDG-related terms, which are then run against pub-
lication databases to find SDG-related scholarly publications. Similar approaches have been
used in research papers by Jetten, Veldhuizen, et autres. (2019), who identified academic publica-
tions concerning targets 2.1–2.4 of SDG 2, and Körfgen et al. (2018) who quantified research
contributions to the SDGs from Austrian universities. Cependant, there are four steps in the devel-
opment of such queries that could vary greatly, and thus potentially have a large impact on any
resulting insights: (un) interpretation of the themes and concepts of the SDG, (b) decisions around
how publications must discuss these concepts to be considered as a “contribution” to the chosen
interpretation of the SDG, (c) translation of concepts into a search query that will find contrib-
uting publications, et (d) data source.

The first step, interpretation, can be challenging, as the SDGs are discussed in different fora by
different stakeholders and from different angles. They also have multiple titles (short and long
versions) and have been translated into different languages, which can result in slightly different
emphasis (Par exemple, the English “Climate Action” vs. the Norwegian “Stopp Klimaendringer”
[Stop climate changes]). What should be used as their basis when defining which themes and con-
cepts are relevant to the SDG? In the second step, there is a further potentially subjective decision
about how a publication must discuss these themes to be considered relevant to an SDG. Many of
the targets concern an action (par exemple., “end hunger”) as well as topics (par exemple., “hunger”). Is a concrete,
direct contribution to this action necessary (c'est à dire., publications discussing ending hunger)? Or should
indirect contributions be counted also (par exemple., publications concerning crop technology)? Are some
topics only relevant when discussed in conjunction with other topics? This also affects the third
step: translation of these interpretations into search terms that will find relevant publications.
Where should search terms originate from? The research community or policy documents?
How should searches be structured to reflect the decisions made in the second step, and how
should recall and precision be balanced? Enfin, in the fourth step, there is the basis of the data
used to measure research contribution. It is well known that publication databases vary in cover-
age of subject fields and languages (Aksnes & Sivertsen, 2019; Mongeon & Paul-Hus, 2016; Vera-
Baceta, Thelwall, & Kousha, 2019), which will further affect any rankings and evaluations.

For these reasons, we believe it is essential to have an independent, transparent bibliometric
method for finding publications related to the SDGs. This will allow institutions to compare dif-
ferent approaches, better understand where rankings come from, and evaluate how well a spe-
cific tool might work in their case. The aim of this study was therefore to build independent
bibliometric search queries to find SDG-related scholarly publications, using a consistent and
defined method that can be reused and built upon. Six SDGs were chosen for examination:
SDG 1 (No poverty), SDG 2 (Zero hunger), SDG 3 (Good health and well-being), SDG 7
(Affordable and clean energy), SDG 13 (Climate action), and SDG 14 (Life below water). We then
compared the results of our approach to the approach currently implemented in SciVal by Elsevier.

2. MÉTHODES

Interpretation of the Themes of the SDGs and Which Publications “Contribute”

2.1.
Each of the SDGs has targets (“Outcomes” and “Means of implementation”) and indicators.
While the titles of some of the SDGs are relatively broad and open to subjective interpretation

Études scientifiques quantitatives

1094

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

1
3
1
0
9
2
1
8
6
9
9
3
0
q
s
s
_
un
_
0
0
0
7
1
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Mapping scholarly publications related to the Sustainable Development Goals

(par exemple., “Climate action”), the targets and indicators are much more specific about what should be
achieved under the goal: They mention specific actions (par exemple., “reduce”) and topics (par exemple., “hun-
ger,” “resilience,” and “tourism”; note that we use “topic” in a broad sense to also include
states, characteristics, or activities). Using the SDG targets and indicators as the theoretical basis
for interpretation thus allows some degree of objectivity, as one is working from a defined set of
topics and actions. We therefore defined the themes of the SDGs based on targets and indica-
tors. The list of SDG targets and indicators in Annex III of the IAEG-SDG 2017 Report to the UN
Statistical Commission (Inter-agency Expert Group on SDG Indicators, 2017) was used, as this
was the most recent complete list available at time of query development.

In terms of defining which research contributes to this interpretation of the SDG themes, nous
made a distinction between “direct contributions” and “indirect contributions.” We considered
“direct contributions” to be publications that refer to target concepts—specific topics or actions
in the targets or indicators. We considered “indirect contributions” to be publications that may
be related to an SDG via related concepts—topics, actions, or research areas that are related to
the target concepts. Related concepts are much more difficult to define and to limit than target
concepts, as the latter can be defined from the targets and indicators themselves, while the for-
mer lacks an objective standard to base their inclusion upon—their inclusion would be much
more based on a particular understanding of the concept and its relatedness. For this reason, nous
chose to build search queries aiming to find “direct contributions” only (Tableau 1, Exemple 1;
Chiffre 2). This is not a judgment about the value of “indirect contributions” to achieving the
SDGs but a practical approach, as defining them objectively and consistently across the
SDGs is, in our opinion, extremely challenging. En outre, the “direct contribution” approach
is well aligned with our method of interpreting the SDG themes. As the targets tend to be action
oriented, this direct interpretation is unlikely to find basic/blue skies research where no appli-
cation is discussed (Tableau 1, Exemple 2).

2.2. Translation of Interpretation into Search Queries

We took a Boolean search approach. Although this method of retrieval has been criticized, it
maintains control and transparency (Hjørland, 2015), which was important both during query
construction and for the current purpose of comparison. Some of the issues that may be crit-
icized can also be avoided by careful construction of the search query (Hjørland, 2015), comme
outlined below. A Boolean approach has recently been used to map climate change, a broad
interdisciplinary topic (Haunschild, Bornmann, & Marx, 2016).

The SDG targets mostly concern specific aspects of a topic or have a directional focus. Dans
order to reflect this focus, the queries were often constructed with combinations of topic terms
(par exemple., “resilience” and “disaster”) ensemble, or combinations of topic terms with action terms (par exemple.,
“reduce,” “increase,” “establish,” “sharing”) (Tableau 1, Examples 3 et 4; Chiffre 2). Some terms
were difficult to place into these types of terms (par exemple., “policy,” which is both a way to bring about
change and a topic), but they provided a rough framework for the construction of the queries.
Action terms were not included on occasions when most works mentioning the topic(s) étaient
considered likely to be directly relevant for the target, or when they would exclude much nec-
essary research (Tableau 1, Exemple 5). A vocabulary reference list was used when adding action
terms and commonly used topic terms to improve consistency across the SDGs.

The syntax of the queries was built with the aim of allowing maximum flexibility in phrasing
and language use in publications while retaining specificity to the targets. Search terms were
often truncated with wildcards, and combined with Boolean operators (AND, OR, very rarely
NOT) in addition to the proximity operator NEAR (where NEAR/x allows x words between two

Études scientifiques quantitatives

1095

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

1
3
1
0
9
2
1
8
6
9
9
3
0
q
s
s
_
un
_
0
0
0
7
1
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Mapping scholarly publications related to the Sustainable Development Goals

Tableau 1. Examples that illustrate our approaches to interpretation, definition of research contribution, or search query construction

Exemple
Exemple 1

Exemple 2

Exemple 3

Exemple 4

Exemple 5

Exemple 6

The FAO considers work on reducing food loss and waste to contribute to SDG2 (“Tackling food loss and waste
is a defined target within the internationally agreed Sustainable Development Goals (SDG Target 12.3, lequel
also contributes directly to SDG Target 12.5 and SDG Goal 2) and a key component of the Zero Hunger
Challenge.” (Food and Agriculture Organization, 2019)). Cependant, for our method of interpretation this link
is indirect, as no “food waste”-related concepts are included in the targets or indicators of SDG 2. It was
therefore not included in the search query; publications concerning food waste would only be found if they
link this to topic’s concepts (par exemple., ending hunger).

Implementation of target 14.5 (“[…] conserve at least 10% of coastal and marine areas […]») will be supported
and enhanced by basic research on marine biodiversity. Cependant, according to our interpretation, this is an
indirect contribution. Publications about marine conservation area establishment or management are more
direct contributions.

Target 14.3 (“minimize and address the impacts of ocean acidification […]») is not about ocean acidification
generally; it is about the impacts. Our search query therefore requires that publications contain terms for
“impacts” as well as “ocean acidification,” rather than just the latter.

Target 2.5 concerns the maintenance of genetic diversity of agricultural resources and mentions “seed and
plant banks.” The topic terms in the query were therefore expanded with “germplasm banks” and “gene
banks,” as these have a similar function. These were combined with agricultural topic terms to limit the
retrieved publications to those relevant to food production, rather than general conservation of genetic
diversity. In the Bergen topic-approach, action terms such as “maintaining,” “preserving,” and “conserving”
were removed, but the combination with agricultural terms was retained.

Target 1.4 concerns access to economic resources and services, and “microfinance” is mentioned specifically.
Since “microfinance” is a tool for providing access to financial services for low-income groups, it was not
deemed necessary to combine this term with any action terms—the action relevant to the target is already
inherent in the concept.

Target 3.1 concerns reductions in child mortality (among other things). In the Bergen action-approach,
the topic terms “mortality” and “child” were combined with action terms for “reduce”. In the Bergen
topic-approach, the action terms were dropped; thus publications discussing child mortality in any context
would be retrieved, regardless of whether they are about reducing it.

search terms). When combining related terms, use of the NEAR operator was preferred over
combining terms in fixed phrases (e.g. “sustainab*” NEAR/x “aquaculture” vs. “sustainable
aquaculture”) to allow for language flexibility. Phrases were used for multiword concepts
(par exemple., “common fisheries policy”), or when a very close link between terms was required to
exclude other subject areas.

One of the limitations of searching using terms in the text is that one must account for var-
iations in the language use of authors. Although we attempted to be as flexible as possible in
query construction, a danger with our interpretation and consequent use of action terms is that
it may exclude publications that use unusual turns of phrase. Due to this, we developed two
versions of the queries: one including all terms, hereafter referred to as the Bergen action-
approche (BAA), and one where most of the action terms were removed and some of the NEAR
terms combining topic terms loosened, hereafter the Bergen topic-approach (BTA).
Combinations of topic terms were retained (Tableau 1, Examples 4 et 6). While the action-
approach attempts to find literature that could directly contribute to achieving the SDG targets,
the topic-approach finds literature related to the target concepts generally. Comparison of these
two approaches allows examination of the effect of action terms and query structure.

Études scientifiques quantitatives

1096

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

1
3
1
0
9
2
1
8
6
9
9
3
0
q
s
s
_
un
_
0
0
0
7
1
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Mapping scholarly publications related to the Sustainable Development Goals

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

1
3
1
0
9
2
1
8
6
9
9
3
0
q
s
s
_
un
_
0
0
0
7
1
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Chiffre 2. A diagram illustrating our interpretation of target 2.1 and the first steps in converting this into search terms. The target concepts
section contains topic and action terms that are mentioned in the target. These are expanded with synonyms and subordinate topics (selected
examples displayed). Some targets are aimed specifically at one particular group (par exemple., the poor or small island developing states; gray
highlighting); in these cases, these would be included in the query. Under related concepts, the elements contain research areas (par exemple., pest
control; arrows) that are not mentioned in the targets or indicators, but may contribute to a technology, connaissance, or state (par exemple., increased
food; boxes) that could potentially contribute to the target. These were not included explicitly in our queries.

Although the targets and indicators formed the basis of our interpretation of the SDGs, we did
not limit our search queries to the terminology used in the targets. Terms were expanded con-
siderably with synonyms and directly related/subordinate concepts (e.g. Tableau 1, Exemple 4;
Chiffre 2). This allows retrieval of relevant publications even when authors do not explicitly relate
their work to the SDGs. In addition to our own subject knowledge, we anchored our queries in
terminology used by intergovernmental organizations and subject vocabularies, such as back-
ground notes from High-Level Political Forums on Sustainable Development (HLPF) (Uni
Nations (ECESA plus), 2017un, 2017b, 2017c, 2017d; United Nations, 2018, 2019b) et
resources from the United Nations, the World Health Organization ( WHO), and the Food
and Agriculture Organization (FAO). Controlled vocabulary thesauri were also examined to
gather search terms, with Emtree® (Elsevier) used for SDG 2 and MeSH® (U.S. National
Library of Medicine) for SDG 2 and SDG 3. Some of the targets needed particular attention
because they concern categories (par exemple., noncommunicable diseases, agriculture, LDCs), alors que
specialist research publications are likely to refer to specific category members (par exemple., melanoma,
poultry, Angola). In these cases, the aforementioned resources were used as the basis for
including category members as search terms. Details about the resources used in query devel-
opment are included in the data files (Armitage, Lorenz, & Mikki, 2020).

The queries were developed to search in the title, abstract, and keywords of publications in
a multidisciplinary database. Some of the topic terms in our queries are used across multiple
academic fields; thus, they were used in combinations to limit the retrieved publications to the
correct field. Par exemple, for SDG 7 the term energy was used in specific combinations to

Études scientifiques quantitatives

1097

Mapping scholarly publications related to the Sustainable Development Goals

avoid results from biology or theoretical physics. For SDGs 2 et 3, terms also had to be used
to limit some parts of the search to human-related studies. For SDG 14, where relevant termi-
nology may apply to both marine and terrestrial environments, searching in the publication
channel name ( journal title) was used in addition to marine-related words to help limit the
results to marine publications. The queries and notes on their development are included in
the data files (Armitage et al., 2020).

2.3. Data Source

Web of Science Core Collection (Clarivate Analytics) (hereafter WoS) was used as the main
database for testing and retrieving bibliographic information in the present study, as it allows
advanced search functions and provides access to comprehensive article citation data for many
different academic disciplines. Our searches were carried out using the search field topic, lequel
searches in title, abstract, author keywords, and Keywords Plus® (plus publication channel
name for part of SDG 14). WoS does not, cependant, cover all academic fields equally; naturel
sciences, technologie, medicine, and health are well indexed, while the social sciences and
particularly the arts and humanities are less well covered, in part due to its focus on articles
(Aksnes & Sivertsen, 2019; Mongeon & Paul-Hus, 2016). We ran two analyses to estimate
how well Nordic or Norwegian publications relevant to SDG 1 and SDG 3 would be covered,
which suggested a rate of ≥93% for SDG 3 and ≥79% for SDG 1 (Supplementary Material 1). Ce
should, cependant, be considered an upper estimate, given the location and high rate of English
language publications in Norway and the Nordic countries.

To build the final corpus of literature, searching was limited to documents published for the
années 2015 à 2018, as the SDGs were agreed upon in 2015 and implemented in January 2016
(United Nations, 2015b). The results were not restricted by language or document type;
around 95–98% of publications were articles, while the rest were proceedings papers, correc-
tion, and editorials.

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

1
3
1
0
9
2
1
8
6
9
9
3
0
q
s
s
_
un
_
0
0
0
7
1
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

2.4. Analysis

An evaluation of our BAA queries was done in two ways—for each SDG, we first visualized
common keywords in 500 résultats, and second examined the titles and abstracts of 30 publica-
tion. Le 500 publications for each SDG were chosen semirandomly by selecting 10 random
search-result pages (each with 50 résultats) in WoS using a random number generator. Le
30 publications were chosen from these 500 by assigning each publication a random number.
The relevance of each publication was assessed by the three authors independently, and then
discussed to get a consensus opinion on whether the publication was relevant, borderline
relevant, or irrelevant. Keywords were visualized via network analysis in VOSViewer (Van Eck
& Waltman, 2010) (full counting, keywords that occurred four or more times). Keywords were
standardized by removing hyphens and using a thesaurus to combine forms of the same word.

To compare our results with the Elsevier queries, we downloaded the most recent Elsevier
query version available at the time of analysis (Octobre 10, 2019; Jayabalasingham, Boverhof,
et coll., 2019). This was translated into WoS syntax and run in the WoS database with the same
search settings as for the Bergen queries. The number of publications found for each set of
queries was compared on a worldwide basis. Overlaps between the different sets of results
were assessed, and benchmarking of contributions from selected countries was done to
see the effect of approach on country rankings. A comparison of keywords of publications
unique to each approach was undertaken via network analysis in VOSViewer (Van Eck &

Études scientifiques quantitatives

1098

Mapping scholarly publications related to the Sustainable Development Goals

Waltman, 2010) (full counting, keywords that occurred five or more times in the 500 top
cited publications).

3. RÉSULTATS

The keyword analysis of the publications retrieved by the Bergen action-approach queries in-
dicated that they are SDG-related (Supplementary Figures 1–6), as did the examination of 30
random publications for each SDG. The number of irrelevant publications retrieved (out of 30)
was zero for SDG 3, one for SDGs 13 et 14, two for SDG 7 and three for SDGs 1 et 2. Le
number of borderline relevant publications retrieved was zero for SDG 7, one for SDGs 1 et
3, two for SDG 14, and three for SDGs 2 et 13 (list available from Armitage et al., 2020). Le
percentages of relevant publications (not including borderline cases) were therefore 97% (SDG
3), 93% (SDG 7), 90% (SDG 14), 87% (SDGs 1 et 13), et 80% (SDG 2). When including
borderline cases, the lowest relevance rate was 90%, indicating high precision of our queries.

The queries from the different approaches varied greatly in the number of publications
retrieved. The Bergen action-approach found many more publications than the Elsevier
approach for SDG 1, but the Elsevier approach found many more publications for SDGs 3, 7,
13, et 14 (Chiffre 3). When the BTA was used, this picture reversed somewhat; the topic-
approach found more publications than the Elsevier approach for SDGs 1, 2, 3, et 7, but fewer
for SDGs 13 et 14 (Chiffre 4).

The degree of overlap in the publication sets found by the Bergen action-approach and the
Elsevier approach was very low; sous 25% of the total publications found for each SDG were
found by both approaches ( Jaccard similarity index expressed as a percentage; Chiffre 3). Ce
was not solely due to the larger of the publication sets encompassing the smaller publication
ensemble: A considerable proportion of publications found by the smaller sets was not present in the
larger sets (Chiffre 3). The overlap in publication sets for SDG 2 was particularly striking because
although the number of retrieved publications was of the same order of magnitude for both
approaches, overlap was still low.

When the results from the BTA and Elsevier approach were compared, the degree of overlap
was mostly larger than for the action-approach comparison. Nevertheless, it remained low
(Chiffre 4). En particulier, the SDG 1 overlap became even smaller due to the small number of
Elsevier publications for this SDG. SDG 3 overlap rose to 56% (1,188,098 publications, le
highest overlap of all the SDGs and comparisons); cependant, this still means that almost a
million publications were only found by one of the two approaches (491,778 publications
unique to the BTA and 425,090 unique to the Elsevier approach).

An analysis of keywords in the publication sets uniquely retrieved by either the BTA or the
Elsevier approach was done for SDGs with the least overlap (SDGs 1, 2, et 14). While this
cannot tell us whether the publications can be considered direct contributions to the SDGs
(regarding our interpretation in Section 2.1), it can indicate whether they include target
concepts or not. Because the focus here was on keywords and thus relatively wide, the BTA
was used for the comparison. For SDG 1, both approaches had relatively clear clusters and
mostly relevant concepts (although detailed examination of how keywords are combined
would be needed to check relevance for some, par exemple., “health”; Supplementary Figure 7). Pour
SDGs 2 et 14, the Elsevier approach produced a less clear clustering pattern and had some
themes that may not be relevant, according to our interpretation (although again, usage context
would need to be examined). For SDG 2, the main themes in the BTA-unique set included food
safety, obesity & weight, and agriculture & climate change, and biodiversity; the main themes

Études scientifiques quantitatives

1099

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

1
3
1
0
9
2
1
8
6
9
9
3
0
q
s
s
_
un
_
0
0
0
7
1
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Mapping scholarly publications related to the Sustainable Development Goals

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

1
3
1
0
9
2
1
8
6
9
9
3
0
q
s
s
_
un
_
0
0
0
7
1
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Chiffre 3. Venn diagrams showing the overlap in publications found by the Bergen action-approach
queries and the Elsevier queries (number of publications found only by the Bergen or Elsevier queries
is shown below the respective label, together with percentage ( Jaccard similarity index) and number
of shared publications in the middle). Publications retrieved from Web of Science Core Collection,
publication years 2015–2018, all document types and languages.

in the Elsevier-approach-unique set included agriculture & climate change, heavy metals, et
fertility & nutrients (Supplementary Figure 8). For SDG 14, the main themes in the BTA-unique
set included marine pollution, climate change & ecosystem services, and fisheries & manage-
ment & impacts; the main themes of the Elsevier-approach-unique set included climate change,
temperature, evolution & diversity, and marine sediments & communautés (Supplementary
Chiffre 9).

The different approaches also made a difference to the country rankings. While the major
contributors were often similarly ranked between the Bergen action-approach and Elsevier
approche, this was not always the case, with large research nations such as China, the UK,
and Australia swapping positions for SDGs 2, 13, et 14 (Chiffre 5). There was no SDG where

Études scientifiques quantitatives

1100

Mapping scholarly publications related to the Sustainable Development Goals

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

1
3
1
0
9
2
1
8
6
9
9
3
0
q
s
s
_
un
_
0
0
0
7
1
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Chiffre 4. Venn diagrams showing the overlap in publications found by the Bergen topic-approach
queries and the Elsevier queries (number of publications found only by the Bergen or Elsevier
queries is shown below the respective label, together with percentage ( Jaccard similarity index)
and number of shared publications in the middle). Publications retrieved from Web of Science
Collection de base, publication years 2015–2018, all document types and languages.

the countries remained in the same ranking order for both approaches; the smallest change
was for SDG 3 (three countries changing position, less than 2% difference) and the largest
for SDG 14 (nine countries changing position, four cases of over 2% difference; Chiffre 5).
The top 10 countries for each SDG remained the same except for SDGs 7 et 14 (SDG 7:
Japan 15th vs. 9ème, Canada 10th vs. 11ème; SDG 14: Japan 16th vs. 10ème, Brazil 10th vs. 11ème).
The Bergen action-approach also found a larger contribution from LDCs to the total SDG-
related publication set than the Elsevier approach. Across the six SDGs, Elsevier’s approach
found a lower percentage contribution from the top-ranked LDCs than the Bergen action-
approach for 53 of the 60 ranking places (Supplementary Figure 10). This difference was
particularly large for SDGs 7, 13, et 14 (combined percentage contribution from the top
10 LDCs: SDG 7: BAA = 1%, Elsevier = 0.4%; SDG 13: BAA = 2.4%, Elsevier = 1.6%; SDG 14:
BAA = 1.6%, Elsevier = 0.6%).

Études scientifiques quantitatives

1101

Mapping scholarly publications related to the Sustainable Development Goals

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

1
3
1
0
9
2
1
8
6
9
9
3
0
q
s
s
_
un
_
0
0
0
7
1
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Chiffre 5. Comparison of percentage contribution of countries to the SDGs between the Bergen action-approach (BAA) and Elsevier queries.
The top 10 countries in order of contribution (highest to lowest) are listed for each approach, with dots indicating a difference in rank (black =
into the top 10). The percentages are the percentage contribution according to BAA (SDG-related publications from that country as a percent-
age of the total number of SDG-related publications), while the charts show the difference in each country’s (Elsevier top 10) percentage
contribution when comparing the Elsevier percentage to the BAA percentage (par exemple., for SDG 1, the USA’s percentage contribution was around
5% lower in the Elsevier results). Publications retrieved from Web of Science Core Collection, publication years 2015–2018, all document
types and languages.

4. DISCUSSION

This work indicates that methodological approach makes a large difference to the publication
set retrieved, and by extension can affect subsequent rankings. This demonstrates the impor-
tance of independent evaluation and open methodology, and has implications for the use of
commercial services currently offering SDG-related rankings or bibliometric data. Elsevier
itself is relatively open about its queries being under development, but this is not clear in
the SciVal platform itself (as of November 2019). We could also not find any caveats about
the THE queries (Times Higher Education, 2019). The danger with this is that their services’
results could be used uncritically, as they provide easy access to rankings and data for leaders,
administrators, and research managers. University rankings are used in important decisions
despite their demonstrated shortcomings (Hazelkorn, 2009; and discussion in Schmoch,
2015), and they have additional issues when it comes to the SDGs (Torabian, 2019).
Rankings and bibliometric information based on unreliable data foundations have even more
potential for adverse consequences.

4.1. Comparison of Approaches

In the development of the current version of the Elsevier queries, Jayabalasingham et al. state
that they compared their original search queries to the targets of the SDGs to assess relevance
(Jayabalasingham et al., 2019; Mu & James, 2019). Given that the Bergen approach is also
based on the targets, it is surprising that there was so little overlap in the results of these ap-
proaches. The differences seem to reflect different choices made in the first three steps of the

Études scientifiques quantitatives

1102

Mapping scholarly publications related to the Sustainable Development Goals

process—interpretation of the themes of the SDG, decisions about how a theme must be dis-
cussed in a publication to be considered “relevant,” and how these themes are translated into
search terms. Autrement dit, it is not just the practical differences in query construction, mais
also differing perspectives on which concepts, and which combinations of concepts, are “rel-
evant” for a target. Deeper examination of the Bergen and Elsevier queries revealed that there
are large differences between the approaches in query structure and the way that terms are
combined, the use of action terms, and differences in which topic terms are included.

The Elsevier queries combine topics relatively rarely, making them much broader than even
the BTA queries (an effect which is further magnified when action terms are combined with
topic terms in the Bergen action-approach). A particularly striking example is SDG 14, où
the topic “marine” is used in two parts of the Elsevier query (combined with AND), avec le
result being that a publication only has to use the word “marine” to count as SDG 14-related.
For our interpretation of SDG 14 relevance, this is far too broad, as SDG 14 is not concerned
with everything to do with the marine environment. The effect of its inclusion is indicated by
the keyword analysis, where keyword clusters not obviously closely related to the targets
of SDG 14 appear (par exemple., marine viruses/genomics/evolution, marine sediments/molecular
phylogeny; Supplementary Figure 9). De la même manière, using the phrase climate change is sufficient
to be counted as related to SDG 13 in the Elsevier approach, whereas the Bergen approaches
required this term to be used in combination with terms with which it is combined in the
targets (par exemple., climate change adaptation, mitigation). Another example from SDG 2 is the topic
“agricultural production.” For the Elsevier approach, this phrase alone is enough to result in
inclusion. For the Bergen approaches, terms for this topic had to be used with terms for
small-scale food production or sustainability, because increased agricultural production is re-
ferred to in these specific contexts in the targets (targets 2.3 et 2.4).

Combinations of topic terms were not the only difference between the approaches; in many
cases there were differences in which topic terms were included at all. The Elsevier approach
includes some topic terms that were not considered part of the SDGs by the Bergen ap-
proaches; Par exemple, “ocean circulation modelling” for SDG 14 and “fertiliser” for SDG 2.
This likely explains the presence of keyword clusters in the Elsevier-approach-exclusive results
related to temperature variability and modeling for SDG 14 and fertilizer/nutrients for SDG 2
(Supplementary Figures 8 et 9). De la même manière, the Bergen approaches include many topic terms
that are not included in the Elsevier queries. En particulier, when developing the Bergen ap-
proaches we tried to include category members as terms when categories were used in the
targets, as we are aware that researchers are often publishing for a specialised audience
and thus use specific terminology. Par exemple, the SDG targets might refer to “neglected trop-
ical diseases” or “least developed countries,” but a publication might talk about “dengue” or
“Ghana.” By only using the category as a search term, such publications are likely to be ex-
cluded unless broad keywords have been added.

The use of action terms clearly also makes a large difference to the number of publications
retrieved. The Bergen action-approach retrieved a much smaller number of publications than
the BTA, despite the same topic terms being used. These action terms make a very large
difference in some cases; Par exemple, searching for “poverty” without action terms (par exemple.,
“reduc*” OR “decreas*”) may find many publications where poverty is a causative factor
(par exemple., poverty-associated diseases) but do not directly discuss combating it.

Enfin, the use of operators is also likely to be responsible for some of the differences in re-
trieved publications between the Elsevier and Bergen approaches. To combine terms, Elsevier
used the operator AND (and sometimes NOT), while the Bergen approaches used NEAR much

Études scientifiques quantitatives

1103

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

1
3
1
0
9
2
1
8
6
9
9
3
0
q
s
s
_
un
_
0
0
0
7
1
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Mapping scholarly publications related to the Sustainable Development Goals

more often. Regardless of any differences in topic terms, this simultaneously broadens and limits
the results of Elsevier approach: Combining terms with AND instead of NEAR makes the search
wider, but using phrases instead of NEAR may result in missing relevant publications. Pour
example, the use of “extreme poverty” OR “poverty alleviation” OR “poverty eradication” OR
“poverty reduction” excludes any works discussing “ending poverty,” “eradicating poverty,” or
“eliminating child poverty.” On the other hand, combinations used in the Bergen approaches
which allow flexibility can permit some nonrelevant results (par exemple., “economic” NEAR/10 “control”
AND…, part of the Bergen queries for SDG 1, results in some publications from cancer studies due
to “control” and “economic” used in two unrelated sentences). This shows that it is not just the
topic terms that must be agreed upon for a bibliometric tool for measuring SDG-related output, mais
also how topic terms are combined, and whether action terms are also necessary for increasing
precision and the inclusion of relevant publications.

4.2. Other Approaches

In this study we have demonstrated the differences in results produced by our approaches and
Elsevier’s approach; cependant, there have also been other approaches to the same problem.
The Sustainable Development Solutions Network (SDSN; Australia, Nouvelle-Zélande, et
Pacific), the AURORA network (AURORA Universities Network, n.d.), and SIRIS Academic
(Duran-Silva, Fuster, et coll., 2019) have also been working on keyword-based bibliometric
queries to find literature related to the SDGs. A comparison of the SDSN and Elsevier queries
was shown during a webinar on Elsevier’s Research Intelligence channel (BrightTalk) and indi-
cated that the SDSN queries take a much broader approach (par exemple., “cities AND land” for SDG 11)
and find many more publications than the Elsevier queries (except for SDG 3). In that compar-
ison, the Elsevier queries were characterized as a very focused approach aiming to minimize
false positives by using very precise keywords (Mu & James, 2019). Cependant, this characteri-
sation is not reflected in the results of the present study.

We were not granted access to the detailed methodology for the Times Higher Education
(THE) approche. While the THE approach is developed in partnership with Elsevier (et
Vertigo Ventures; Times Higher Education, 2019), the queries being used by THE (at present)
are not the same as those from Elsevier currently being used in SciVal, and appear instead to
be based on an earlier version (Mu & James, 2019). We were not able to ascertain if the original
Elsevier queries have been further modified by THE. We could therefore not do a comparison
with their approach. Cependant, the original version of the Elsevier queries, according to Elsevier’s
own testing and documentation, finds many publications that are not relevant to the SDG targets
(Jayabalasingham et al., 2019; Mu & James, 2019). This suggests that if THE is using these orig-
inal queries, then their results might diverge from our approach even more so than Elsevier’s. Dans
addition, the specifics of how THE integrate their data on SDG-related scholarly publications
with their other criteria for assessment is relatively unclear from their public methodology
(Times Higher Education, 2019). University rankings can have wide reaching impacts
(Hazelkorn, 2009), et, in THE’s own words, “The data we compile to produce these rankings
are trusted by governments and universities and are a vital resource for students when they are
making decisions about where to study” (Times Higher Education, 2018). It is therefore unfor-
tunate that we could not confirm their method of evaluation well enough to compare results.

Boolean searching is not the only approach taken to examining SDG-related publication
output. Machine learning has been suggested as an approach (Mu & James, 2019), but to our
knowledge has not been applied yet. Citation analysis has also been used, where a body of SDG-
related literature is built from publications who cite publications that use the phrase Sustainable

Études scientifiques quantitatives

1104

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

1
3
1
0
9
2
1
8
6
9
9
3
0
q
s
s
_
un
_
0
0
0
7
1
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Mapping scholarly publications related to the Sustainable Development Goals

Development Goal(s) (Nakamura, Pendlebury, et coll., 2019). A potential danger with this is that
one risks missing works from researchers who are not engaged with the SDGs despite working
on relevant topics. Cependant, it does avoid some of the issues with interpretation of SDG themes
and relevance by shifting the responsibility for this to the authors of the publications referring to
the SDGs.

4.3. Défis

We carried out our comparisons within the same database using the same settings; these should
therefore be valid. Cependant, the number of publications and country rankings presented here
will be affected by the coverage of the WoS database. Our analyses indicated that we may be
missing around 7% of the relevant medical literature and 20% of the social sciences publications
(Supplementary Material 1). These percentages are likely an underestimate when considering
global publications, given the underrepresentation of non-English-language publications in
large international databases. En outre, the WoS database focuses strongly on articles.
Measuring other publication types (par exemple., theses, reports) would require a different data source.

To make our searches both flexible and specific, we used proximity operators and a complex
structure. This means that to run these searches as they are, one is limited to databases which
support advanced search structure (par exemple., WoS, Scopus). Cependant, the queries do not rely on
controlled vocabularies, thesauri, or subject indexing, and are designed to prevent fuzzy
searching (lemmatization); donc, the queries should function the same in any database
(provided proximity operators are supported and queries are translated into the appropriate
syntax). The queries can also function as a resource for gathering SDG-related keywords.

Interpreting the SDGs from the targets was not without its challenges. The SDG targets vary
in formulation and degree of specificity, with a report by the International Council for Science
assessing only 29% of the targets as “well-developed” (ICSU & ISSC, 2015). En outre, le
targets vary in how directly they can be linked to research activities, and interpretation may
be impeded by differences in the title of the goal, HLPF documents, and targets. An example
of these last two points is SDG 13. Here there is a focus on combating climate change in both
the long title of the goal (“Take urgent action to combat climate change and its impacts”) et
the HLPF background note (which states that addressing climate change impacts “require a
two-pronged approach—reduction in the greenhouse gas emissions, and adaption planning”
[United Nations, 2019b]). This implies that research and technological development for
reducing greenhouse gases (par exemple., carbon capture and storage) would be relevant. Cependant,
the targets are mostly focused on the political perspective (funding and implementing of
policies for climate change measures [13.UN, 13.2] or educating and improving capacity for
climate change adaption, mitigation, etc.. [13.3]). Reduction of greenhouse gases is mentioned
only in relation to national policies or plans (indicator 13.2.1) and in terms of improving
human and institutional capacity for mitigation (13.3). Interpreting the relevance of research
to target 13.3 therefore becomes a question of what contributes to “human and institutional
capacity,” which can be relatively subjective (the UN definition of capacity is “[…] the ability
of people, organizations and society as a whole to manage their affairs successfully” [Uni
Nations Development Group, 2017]). En outre, the use of terms such as “impact reduction”
makes it relatively unclear whether impacts on nonhuman entities (ecosystems, plants, etc.)
are relevant; we interpreted it in the broadest sense.

Another challenge concerning relevance comes from the use of search terms. As publica-
tions are retrieved based on their keywords, abstract, and title, they may be retrieved or not
retrieved based on author awareness of relating their work to wider issues. Par exemple, it is

Études scientifiques quantitatives

1105

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

1
3
1
0
9
2
1
8
6
9
9
3
0
q
s
s
_
un
_
0
0
0
7
1
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Mapping scholarly publications related to the Sustainable Development Goals

not uncommon for publications about renewable energy (SDG 7) to have a sentence in the
abstract relating their work to climate mitigation, and thus may be retrieved by the SDG 13
recherche. This means that differences in writing style and strategic positioning of buzzwords may
affect representation in the retrieved publication set. In the same way, the results of this
particular study will be affected by the WoS process of applying Keywords Plus to publications,
because if our search found a publication relevant based on Keywords Plus, this has been
mediated by WoS’s interpretation of that concept and whether it was appropriate for the
publication.

Enfin, we emphasize that the approach outlined here is limited to measuring scholarly
édition. Assessing institutions’ wider engagement with the SDGs requires an integrated
approach which takes into account a variety of institutional activities (par exemple., courses offered in
sustainability, measures to reduce inequality, climate-friendly policies), not least because the
pursuit of certain indicators or rankings (related to publishing) may encourage practices that are
in conflict with the SDGs themselves (see Torabian, 2019).

4.4. Summary and the Way Forward

Interpretation of the themes of the SDGs, making decisions about what counts as a “contribu-
tion,” and translating this into functioning search queries are not simple tasks. This study has
shown that two independent approaches can deliver two widely different sets of results.
Differences in the terms included and how they are combined makes large differences to
the final result. The results suggest that it would be premature to trust commercial SDG anal-
yses for anything other than exploratory purposes at this stage in their development.

The open methodology of the Elsevier approach facilitated this comparison and demon-
strates how open science can facilitate independent testing and potentially stimulate the
advancement of methods and tools. According to Mu and James (2019), the next stage of the
Elsevier approach is to develop a website where people can give feedback on whether a doc-
ument addresses the SDG or not, et, if yes, which of the targets it refers to. This type of crowd-
sourcing may further development, but due to the wide variety of perspectives involved,
crowdsourcing may not be ideal for developing a consensus on interpretation of SDG themes
and making decisions about which research “contributes.” We suggest that developing multiple
approaches to reflect different perspectives and uses (par exemple., our action- and topic-approaches),
perhaps with an additional even-wider approach to cover indirect contributions, could be
beneficial. This would allow for multiple interpretations of what it means to be “SDG-related
recherche,” allow researchers and managers to use the appropriate tool for their needs at that
temps, and allow bibliometric measurements to cover the diversity in SDG-related research.

REMERCIEMENTS

We thank the reviewers for their comments, and our University of Bergen colleagues Regina
Küfner Lein for help developing our SDG 3 query and Katja Enberg for input on our SDG 14
query.

CONTRIBUTIONS DES AUTEURS

Caroline Armitage: Conceptualisation, Conservation des données, Analyse formelle, Enquête,
Méthodologie, Validation, Visualisation, Writing—original draft, Writing—review & édition.
Marta Lorenz: Conceptualisation, Analyse formelle, Enquête, Méthodologie, Validation,

Études scientifiques quantitatives

1106

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

1
3
1
0
9
2
1
8
6
9
9
3
0
q
s
s
_
un
_
0
0
0
7
1
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Mapping scholarly publications related to the Sustainable Development Goals

Writing—review & édition. Susanne Mikki: Conceptualisation, Méthodologie, Project admin-
istration, Validation, Writing—review & édition.

COMPETING INTERESTS

The authors have no competing interests.

INFORMATIONS SUR LE FINANCEMENT

No funding has been received for this research.

DATA AVAILABILITY

The queries developed in this study, and the data for the testing of 30 random publications, sont
available in DataverseNO (UiB Open Research Data) at https://doi.org/10.18710/98CMDR.
Metadata directly from databases (used to create keyword analyses and compare coverage)
are not available.

RÉFÉRENCES

Aksnes, D. W., & Sivertsen, G. (2019). A criteria-based assessment
of the coverage of Scopus and Web of Science. Journal of Data
and Information Science, 4(1), 1–21. https://doi.org/10.2478/jdis-
2019-0001

Armitage, C. S., Lorenz, M., & Mikki, S. (2020). Replication data
pour: Mapping scholarly publications related to the Sustainable
Development Goals: Do independent bibliometric approaches
get the same results? [Data set]. DataverseNO. https://est ce que je.org/
10.18710/98CMDR

AURORA Universities Network. (n.d.). SDG analysis: Bibliometrics of
relevance. Retrieved March 30, 2020, from https://aurora-network.
global/project/sdg-analysis-bibliometrics-relevance/

Chu, S., Cui, Y., & Liu, N. (2017). The path towards sustainable
energy. Nature Materials, 16(1), 16–22. https://doi.org/10.1038/
nmat4834

Duran-Silva, N., Fuster, E., Massucci, F. UN., & Quinquillà, UN.
(2019). A controlled vocabulary defining the semantic perimeter
of Sustainable Development Goals ( Version 1.2) [Data set].
Zenodo. https://doi.org/10.5281/zenodo.3567769

Elsevier. (2019). SciVal. Retrieved December 9, 2019, http://www.

elsevier.com/solutions/scival

Food and Agriculture Organization. (2019). Policy Support and
Governance; Food Loss and Food Waste. Retrieved October
20, 2019, from http://www.fao.org/policy-support/policy-
themes/food-loss-food-waste/en/

Gluckman, P.. (2016). The science-policy interface. Science, 353(6303),

969. https://doi.org/10.1126/science.aai8837

Haunschild, R., Bornmann, L., & Marx, W. (2016). Changement climatique
Research in View of Bibliometrics. PLOS ONE, 11(7), e0160393.
https://doi.org/10.1371/journal.pone.0160393

Hazelkorn, E. (2009). Rankings and the battle for world-class excel-
lence. Higher Education Management and Policy, 21(1), 1–22.
https://doi.org/10.1787/hemp-v21-art4-en

Hjørland, B. (2015). Classical databases and knowledge organiza-
tion: A case for boolean retrieval and human decision-making
during searches. Journal of the Association for Information
Science and Technology, 66(8), 1559–1575. https://est ce que je.org/
10.1002/asi.23250

ICSU & ISSC. (2015). Review of Targets for the Sustainable
Development Goals: The Science Perspective. Retrieved from

http://council.science/publications/review-of-targets-for-the-
sustainable-development-goals-the-science-perspective-
2015

Inter-agency Expert Group on SDG Indicators. (2017). Report of the
Inter-agency and Expert Group on Sustainable Development
Goal Indicators (E/CN.3/2017.2). United Nations. http://unstats.
un.org/unsd/statcom/48th-session/documents/2017-2-IAEG-
SDGs-E.pdf

Jayabalasingham, B., Boverhof, R., Agnew, K., & Klein, L. (2019).
Identifying research supporting the United Nations Sustainable
Development Goals. Mendeley Data, v1 (2019-10-10). [Données
ensemble]. https://doi.org/10.17632/87txkw7khs.1

Jetten, T. H., Veldhuizen, L. J.. L., Siebert, M., & Darroch, P.. je.
(2019). An explorative study on a university’s outreach in the
field of UN Sustainable Development Goal 2 [Research report].
Retrieved from Wageningen University & Research: https://est ce que je.
org/10.18174/476199

Körfgen, UN., Förster, K., Glatz, JE., Maier, S., Becsi, B., Meyer, UN.,
Kromp-Kolb, H., & Stötter, J.. (2018). It’s a Hit! Cartographie
Austrian Research Contributions to the Sustainable Development
Goals. Durabilité, 10(9), 3295. https://doi.org/10.3390/
su10093295

Leal Filho, W., Azeiteiro, U., Alves, F., Pace, P., Mifsud, M.,
Brandli, L., Caeiro, S. S., & Disterheft, UN. (2018). Reinvigorating
the sustainable development research agenda: The role of the
sustainable development goals (SDG). Revue internationale de
Sustainable Development & World Ecology, 25(2), 131–142.
https://doi.org/10.1080/13504509.2017.1342103

Mongeon, P., & Paul-Hus, UN. (2016). The journal coverage of Web
of Science and Scopus: A comparative analysis. Scientometrics,
106(1), 213–228. https://doi.org/10.1007/s11192-015-1765-5
Mu, J., & James, C. (2019). Analyzing UN SDG research with
SciVal—The case of University of Auckland [Webinar].
Retrieved from BrightTALK: https://www.brighttalk.com/webcast/
13819/375997

Nakamura, M., Pendlebury, D., Rapide, J., & Szomszor, M.. (2019).
Navigating the Structure of Research on Sustainable Development
Goals. Retrieved from the Institute for Scientific Information: https://
clarivate.com/webofsciencegroup/campaigns/sustainable-
development-goals/

Études scientifiques quantitatives

1107

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d

F
/

1
3
1
0
9
2
1
8
6
9
9
3
0
q
s
s
_
un
_
0
0
0
7
1
p
d

b
oui
g
toi
e
s
t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Mapping scholarly publications related to the Sustainable Development Goals

Reid, W. V., Bréchignac, C., & Tseh Lee, Oui. (2009). Earth system research
priorities. Science, 325(5938), 245. https://doi.org/10.1126/science.
1178591

Schmoch, U. (2015). The informative value of international university
rankings: Some methodological remarks. In Incentives and
Performance: Governance of Research Organizations (pp. 141–154).
Cham: Springer International Publishing. https://doi.org/10.1007/978-
3-319-09785-5_9

Times Higher Education. (2018). How to participate in Times
Higher Education rankings. Retrieved December 12, 2019, depuis
https://www.timeshighereducation.com/ how-participate-times-
higher-education-rankings

Times Higher Education. (2019). THE University Impact Rankings
2019: Méthodologie. Retrieved December 12, 2019, depuis
https://www.timeshighereducation.com/world-university-rankings/
methodology-impact-rankings-2019

Torabian, J.. (2019). Revisiting Global University Rankings and
Their Indicators in the Age of Sustainable Development.
Durabilité: The Journal of Record, 12(3), 167–172. https://
doi.org/10.1089/sus.2018.0037

United Nations. (2015un). The Millennium Development Goals
Report 2015 Résumé. https://www.un.org/millenniumgoals/
2015_MDG_Report/pdf/MDG 2015 Summary web_english.pdf
United Nations. (2015b). Transforming our world: Le 2030
Agenda for Sustainable Development. Retrieved December 2,
2019, from https://sustainabledevelopment.un.org/post2015/
transformingourworld

United Nations. (2018). 2018 HLPF Review of SDG implementation:
SDG 7- Ensure access to affordable, reliable, sustainable and mod-
ern energy for all. https://sustainabledevelopment.un.org/content/
documents/195532018_background_notes_SDG_7Final1.pdf
United Nations. (2019un). Academic Impact; About UNAI.
Retrieved October 14, 2019, from http://academicimpact.un.
org/content/about-unai

United Nations. (2019b). Review of SDG implementation and inter-
relations among goals. Discussion on SDG 13—Climate action.

https://sustainabledevelopment.un.org/content/documents/
23627BN_SDG13.pdf

United Nations (ECESA plus). (2017un). 2017 HLPF Thematic
Review of SDG 1: End poverty in all its forms everywhere.
https://sustainabledevelopment.un.org/content/documents/
14379SDG1format-final_OD.pdf

United Nations (ECESA plus). (2017b). 2017 HLPF Thematic review
of SDG2: End hunger, achieve food security and improved nutri-
tion, and promote sustainable agriculture. https://sustainablede-
velopment.un.org/content/documents/14371SDG2_format.
revised_FINAL_28_04.pdf

United Nations (ECESA plus). (2017c). 2017 HLPF Thematic
Review of SDG3: Ensure healthy lives and promote well-being
for all at all ages. https://sustainabledevelopment.un.org/con-
tent/documents/14367SDG3format-rev_MD_OD.pdf

United Nations (ECESA plus). (2017d). 2017 HLFP Thematic
Review of SDG 14: Conserve and sustainably use the oceans,
seas and marine resources for sustainable development. http://
s u s t a i n a b l e d e v e l o p m e n t . u n . o r g / c o n t e n t / d o c u m e n t s /
14375SDG14format-revOD.pdf

United Nations Department of Global Communications. (2019).
Sustainable Development Goals: Guidelines for the use of the
SDG logo including the colour wheel, et 17 icons. Retrieved from
https://www.un.org/sustainabledevelopment/news/communications-
material/

Unite d Nations Development Group.

(2017). Ca pa cit y
Développement, UNDAF companion guidance. Retrieved from
https://unsdg.un.org/resources/capacity-development-undaf-
companion-guidance

Van Eck, N.

J., & Waltman, L.

(2010). Software survey:
VOSviewer, a computer program for bibliometric mapping.
Scientometrics, 84(2), 523–538. https://doi.org/10.1007/
s11192-009-0146-3

Vera-Baceta, M.. UN., Thelwall, M., & Kousha, K. (2019). Internet de
Science and Scopus language coverage. Scientometrics, 121(3),
1803–1813. https://doi.org/10.1007/s11192-019-03264-z

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
q
s
s
/
un
r
t
je
c
e
–
p
d