RESEARCH ARTICLE - IA de Investigación especializada en el MIT

ARTÍCULO DE INVESTIGACIÓN

How can citation impact in bibliometrics be
normalized? A new approach combining
citing-side normalization and
citation percentiles

un acceso abierto

diario

Lutz Bornmann

Division for Science and Innovation Studies, Administrative Headquarters of the Max Planck Society,
Hofgartenstr. 8, 80539 Munich, Alemania

Palabras clave: bibliometría, citation analysis, citation percentiles, citing-side normalization

ABSTRACTO

Since the 1980s, many different methods have been proposed to field-normalize citations. En esto
estudiar, an approach is introduced that combines two previously introduced methods: citing-side
normalization and citation percentiles. The advantage of combining two methods is that their
advantages can be integrated in one solution. Based on citing-side normalization, each citation
is field weighted and, por lo tanto, contextualized in its field. The most important advantage of
citing-side normalization is that it is not necessary to work with a specific field categorization
scheme for the normalization procedure. The disadvantages of citing-side normalization—the
calculation is complex and the numbers are elusive—can be compensated for by calculating
percentiles based on weighted citations that result from citing-side normalization. En el uno
mano, percentiles are easy to understand: They are the percentage of papers published in the
same year with a lower citation impact. Por otro lado, weighted citation distributions are
skewed distributions with outliers. Percentiles are well suited to assigning the position of a focal
paper in such distributions of comparable papers. The new approach of calculating percentiles
based on weighted citations is demonstrated in this study on the basis of a citation impact
comparison between several countries.

INTRODUCCIÓN

Research systematically investigates what is (still) not known. In order to demonstrate the lag in
current knowledge and the shoulders on which the exploration of the lag by new studies stand,
authors of papers (ideally) cite all relevant previous publications (Kostoff, Murday, et al., 2006).
On the basis of this norm in science to cite the relevant past literature, citations have been estab-
lished as a proxy for scientific quality—measuring science “impact” as an important component
of quality (Aksnes, Langfeldt, & Wouters, 2019). Narin (1976) proposed the term evaluative
bibliometrics for methods using citation-based metrics for measuring cognitive influence
(Moed, 2017; van Raan, 2019). Bornmann and Marewski (2019) introduced the bibliometrics-
based heuristics (BBHs) concept concretizing the evaluative use of bibliometrics: “BBHs charac-
terize decision strategies in research evaluations based on bibliometrics data (publications and
citas). Other data (indicators) besides bibliometrics are not considered” (Bornmann, 2020).

According to Moed and Halevi (2015), research assessment (based on bibliometrics) is an
integral part of any scientific activity these days: “it is an ongoing process aimed at improving

Citación: Bornmann, l. (2020). How can
citation impact in bibliometrics be
normalized? A new approach
combining citing-side normalization
and citation percentiles. Quantitative
Science Studies, 1(4), 1553–1569.
https://doi.org/10.1162/qss_a_00089

DOI:
https://doi.org/10.1162/qss_a_00089

Recibió: 8 Puede 2020
Aceptado: 30 Julio 2020

Autor correspondiente:
Lutz Bornmann
bornmann@gv.mpg.de

Editor de manejo:
Juego Waltman

Derechos de autor: © 2020 Lutz Bornmann.
Publicado bajo Creative Commons
Atribución 4.0 Internacional (CC POR 4.0)
licencia.

La prensa del MIT

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
4
1
5
5
3
1
8
7
1
0
0
0
q
s
s
_
a
_
0
0
0
8
9
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

How can citation impact in bibliometrics be normalized?

the quality of scientific/scholarly research. It includes evaluation of research quality and measure-
ments of research inputs, outputs, and impacts, and embraces both qualitative and quantitative
methodologies, including the application of bibliometric indicators and peer review” (pag. 1988).
Current research evaluation processes concern single researchers (Bornmann & Marx, 2014),
research groups, institutions, organizaciones (Bornmann, Bowman, et al., 2014), and countries
(Leydesdorff, Wagner, & Bornmann, 2014). Since the beginning of the 20th century, annually
produced international university rankings have become more and more popular (Vernon,
Balas, & Momani, 2018).

The analysis of citations is at the core of bibliometrics: “citation impact is an important indi-
cator of scientific contribution because it is valid, relatively objective, y, with existing data-
bases and search tools, straightforward to compute” (Nosek, graham, et al., 2010, pag. 1292).
The problem of citation analysis is, sin embargo, that fields differ in their publication, citation, y
authorship practices (waltman & van Eck, 2013b). Crespo, li, and Ruiz-Castillo (2013) estimated
eso 14% of overall citation inequality can be attributed to field-specific differences in citation
practicas. These and similar findings from bibliometrics research make clear that the results of
citation analyses from different fields cannot be compared. Whereas single publications and
researchers can be compared within one field, this is not possible with universities and many
research-focused institutions. For citation analyses in which cross-field comparisons are neces-
sary, field-normalized citation impact indicators have been developed. Field normalization aims
to remove the noise that traces back to the fields while maintaining the signal that reflects (true)
performance differences (waltman & van Eck, 2019). It is an indication of advanced bibliometrics
to use “reliable statistics, p.ej., corrections for differences in publication and citation practices
between scientific disciplines” (van Raan, 2019, pag. 244).

Since the 1980s, many approaches have been developed in the scientometrics field to field-
normalize citations. Although some approaches (p.ej., the number of publications published by an
institution that belongs to the 10% most frequently cited publications in the corresponding fields)
could reach the status of quasistandards, each approach has its specific disadvantages. En esto
paper, an approach is introduced combining the advantages of two published approaches and
smoothing their specific disadvantages. The first approach is citing-side normalization, por lo cual
each single citation of a paper is field-normalized. The second approach is the citation percentile,
which is the percentage of papers in a given set of papers with lower citation impact than the
focal paper.

2. LITERATURE OVERVIEW: A SHORT HISTORY OF FIELD NORMALIZATION

Field normalization has a long tradition in bibliometrics. Literature overviews on the developments
in the field can be found in Mingers and Leydesdorff (2015), Bornmann and Marx (2015), y
waltman (2016). Field normalizations start from the basic premise that “not all citations are equal.
Por lo tanto, normalization can be seen as a process of benchmarking that is needed to enhance
comparability across diverse scientists, campos, documentos, time periods, and so forth” (Ioannidis,
Boyack, & Wouters, 2016). Many studies on field normalization either deal with technical issues
(p.ej., the development of improved indicator variants) or with the way fields should be defined for
use in normalization (p.ej., by using journal sets or human-based assignments; see Wilsdon, allen,
et al., 2015). One of the earliest attempts in bibliometrics to field-normalize citations was made by
Schubert and Braun (1986) and Vinkler (1986). They proposed to calculate the average citation
rate for a journal or field and to use this reference score to field-normalize (single) papers published
in the journal or field (by dividing the citation counts of every single papers by the reference score).
The resulting metric was named the relative citation rate (RCR) by Schubert and Braun (1986).

Estudios de ciencias cuantitativas

1554

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
4
1
5
5
3
1
8
7
1
0
0
0
q
s
s
_
a
_
0
0
0
8
9
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

How can citation impact in bibliometrics be normalized?

Since its introduction, the RCR has been criticized for its use of the arithmetic average in the
normalization. The arithmetic average should not be used as a measure of central tendency for
skewed citation data. According to Glänzel and Moed (2013), “the mean should certainly not be
used if the underlying distribution is very skewed, and has a long tail” (pag. 383). The fact that arith-
metic averages of citation data and, de este modo, field normalized citation scores are sensitive to outliers
has been named by van Raan (2019) as the Göttingen effect: "En 2008, a paper published by a
researcher of the University of Göttingen became extremely highly cited, many thousands of times
a year, within a very short time … As a result, for several years after this publication, Göttingen
achieved a very high position in … [university] rankings” (pag. 260).

To deal with the problem of skewed distributions in field normalization, McAllister, Narin, y
Corrigan (1983, pag. 207) already proposed in the 1980s that percentiles should be used for citation
datos:

the pth percentile of a distribution is defined as the number of citations Xp the percent of papers
receiving Xp such that or fewer citations is equal to p. Since citation distributions are discrete, el
pth percentile is defined only for certain p that occur in the particular distribution of interest. De este modo
we would say that a 1974 physiology paper receiving one citation falls in the 18th percentile of
the distribution. This means that 82 por ciento (100 − 18) de todo 1974 A NOSOTROS. physiology papers
received more than one citation. For any paper in the 18th percentile of any subject area citation
distribución, 18 percent of the papers performed at a level less than or equal to the particular
paper, y 82 percent of the papers in the subject area outperformed the particular paper.

For Schreiber (2013) “percentiles … have become a standard instrument in bibliometrics”
(pag. 822) in current bibliometrics. Percentiles are recommended in the Leiden manifesto which
includes 10 principles to guide research evaluation (Hicks, Wouters, et al., 2015). The most recent
field-normalizing percentile approach has been published by Bornmann and Williams (2020).

One of the biggest challenges in field normalizing citations is the selection of the system cate-
gorizing papers to fields. The overview by Sugimoto and Weingart (2015) shows that existing sys-
tems emphasize cognitive, social, or institutional orientations of fields to a different extent. Varios
field categorization schemes are in use to normalize citations and there exists no standard use in
bibliometría. The most frequently used schemes are multidisciplinary schemes that span all fields
(Sugimoto & Weingart, 2015; Wang & waltman, 2016). These schemes are typically based on
journal sets: the Web of Science (WoS) subject categories of Clarivate Analytics and the Scopus
subject areas of Elsevier. The use of journal sets can be justified quite well: according to Milojevic(cid:1)
(2020, pag. 184) “journals often serve as anchors for individual research communities, and new jour-
nals may signify the formations of disciplines.” Each journal is a well-crafted folder sustained by
editores, reviewers, and authors who usually know and use that outlet. Authors typically direct their
manuscripts in an informed way to reach the appropriate audience for the content and argument.

There are two problems with these schemes, sin embargo, which is why Waltman and van Eck
(2012) proposed a new method for algorithmically constructing classification systems (see also
Boyack & Klavans, 2010): (a) Because journals publish many different papers, journals are usually
assigned to more than one category; y (b) journal sets represent broad fields which is why
papers from specific fields might be misclassified (see Strotmann & zhao, 2010). The results by
Shu, Julien, et al. (2019) reveal that about half of the papers published in a journal are not from the
field to which the journal has been assigned.

The system proposed by Waltman and van Eck (2012) is based on citation relations between
single publications. The advantages of the system are that (a) it assigns single publications (and not

Estudios de ciencias cuantitativas

1555

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
4
1
5
5
3
1
8
7
1
0
0
0
q
s
s
_
a
_
0
0
0
8
9
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

How can citation impact in bibliometrics be normalized?

journals) to fields and (b) it provides a fine-grained categorization scheme of publications. Ruiz-
Castillo and Waltman (2015) demonstrate the use of the system for field normalization. El sistema,
sin embargo, has not remained without criticism: porque

“fields” are algorithmic artifacts, they cannot easily be named (as against numbered), y
therefore cannot be validated. Además, a paper has to be cited or contain references in
order to be classified, since the approach is based on direct citation relations … However,
algorithmically generated classifications of journals have characteristics very different from
content-based (eso es, semantically meaningful) classifications … The new Leiden system is
not only difficult to validate, it also cannot be accessed or replicated from outside its context of
production in Leiden (Leydesdorff & Milojevic(cid:1), 2015, pag. 201).

As the recent results by Sjögårde, Ahlgren, and Waltman (2020) espectáculo, at least the labeling

problem of the fields can be solved.

Another critical point is that the field assignments based on citation relations change with new
citas. The approach does not lead to stable results, and it is elusive why the field assignment of
a paper should change. Further critical remarks can be found in Haunschild, Schier, et al. (2018).
Based on the critique of the system proposed by Waltman and van Eck (2012), Colliander and
Ahlgren (2019) introduced an item-oriented approach that avoids clustering, but uses publication-
level features to estimate subject similarities. The empirical comparison of this approach with
standard approaches in bibliometrics by the authors revealed promising results. Future indepen-
dent studies will demonstrate whether these first positive results can be confirmed.

As an alternative to multidisciplinary schemes, monodisciplinary schemes have been proposed
for field normalization. The advantages of these schemes are that papers are usually assigned to a
single research field and human indexers (field experts or authors of papers) assign the relevant field
to a paper intellectually (Bornmann, Marx, & Barth, 2013). En años recientes, studies have used dif-
ferent monodisciplinary schemes to field-normalize citations in certain fields: Bornmann and
Wohlrabe (2019) used Journal of Economic Literature classification ( JEL) codes in economics,
Bornmann, Schier, et al. (2011) and Bornmann and Daniel (2008) used Chemical Abstracts (California)
sections in chemistry and related areas, Radicchi and Castellano (2011) used Physics and
Astronomy Classification Scheme (PACS) codes in physics and related areas, and Smolinsky and
Lercher (2012) used the MathSciNet’s Mathematics Subject Classification (MSC) system in math-
ematics. The disadvantages of monodisciplinary schemes are that they are restricted to single fields
and the assignments by the indexers may be affected by subjective biases.

One problem that affects many field classification systems (mono- and multidisciplinary) es
that they exhibit different aggregation levels, and it is not clear which level should be used to
field-normalize citations (waltman & van Eck, 2019; Wouters, Thelwall, et al., 2015). In biblio-
métrica, different results and opinions have been published as to whether an aggregation level
change has any (significativo) influence on the field-normalized scores: Zitt, Ramanana-Rahary,
and Bassecoulard (2005) report a lack of stability of these scores; Colliander and Ahlgren (2011)
arrive at another conclusion. Wang (2013) holds the opinion that “normalization at finer level is
still unable to achieve its goal of improving homogeneity for a fairer comparison” (pag. 867).

3. CITING-SIDE NORMALIZATION

The literature overview in section 2 has shown that there are many problems with field normal-
ization in bibliometrics and it has not yet been possible to establish a standard. One can expect
that some problems will remain unsolved without finding a perfect solution. Por ejemplo, it will

Estudios de ciencias cuantitativas

1556

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
4
1
5
5
3
1
8
7
1
0
0
0
q
s
s
_
a
_
0
0
0
8
9
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

How can citation impact in bibliometrics be normalized?

remain a normative decision as to which field categorization scheme is used (and on what level).
Independently of the system that is used, fields are not isolated and research based on between-
field collaborations is common (Ioannidis et al., 2016). “With the population of researchers,
scientific literature and knowledge ever growing, the scientific endeavour increasingly integrates
across boundaries” (Gates, Ke, et al., 2019, pag. 34). According to Waltman and van Eck (2013a),
“the idea of science being subdivided into a number of clearly delineated fields is artificial. En
reality, boundaries between fields tend to be rather fuzzy” (pag. 700).

A possible solution to these problems might be to avoid the use of field categorization schemes
(Bornmann, Marx, et al., 2013), clustering (waltman & van Eck, 2012), and similarity approaches
(Colliander & Ahlgren, 2019), and for each focal paper (that is assessed) to manually search some
papers for comparison that are thematically similar (Kostoff, 2002; waltman, 2016). This solution
corresponds to the judgement by Hou, Cacerola, et al. (2019) that field normalization cannot be solved
by statistical techniques. The manual collection of papers for the comparison with a focal paper
might be possible in the evaluation of small sets of papers; sin embargo, it is not practicable for large
conjuntos (p.ej., all papers published by a university over several years). Además, one needs experts
from the fields to find the papers for comparison.

Another solution that can be applied to large sets of papers is not to normalize citation impact
based on expected citations from the reference sets, but to normalize single citations directly.
So-called citing-side field normalizing approaches have been proposed in recent years that
normalize each single citation of a focal paper. van Raan (2014) sees these “field-independent
normalization procedures” (pag. 22) as an important and topical issue in bibliometrics. The simplest
procedure is to divide each citation by the number of cited references of the citing paper. The use
of the number of cited references is intended to reflect the disciplinary context of the citing paper
and to standardize the citation field specifically. It is a decisive advantage of citing-side normal-
ization that it “does not require a field classification system” (waltman & van Eck, 2013a, pag. 700).
Citing-side normalization, de este modo, solves the problem with the selection of a field-categorization
scheme by refraining from it.

Citing-side normalization might be a reasonable approach for citation analysis, as the goal of
field normalization is the normalization of citation impact (see Waltman, van Eck, et al., 2013).
Given the different directions of the two basic field normalization approaches, citing-side
approaches are more focused on the aim of field normalization than approaches that are based
on reference sets on the cited side: Citing-side approaches normalize each single citation of a
focal paper. Bornmann and Marx (2015) demonstrated the problem of field normalization based
on cited-side normalization by using the well-known paper by Hirsch (2005) on the h-index as an
ejemplo. This paper is a typical bibliometrics paper (it introduces a new indicator based on
publication and citation data), but receives citations from many fields (not only from the biblio-
metrics field). If a focal paper is attractive for authors publishing in other fields with high citation
density, it has an advantage over another focal paper that is not as attractive for these fields.
Although both focal papers might belong to the same field (viewed from the cited-side perspec-
tivo), they have different chances of being cited.

The paper by Hirsch (2005) is concerned with another “problem” (for field normalization): Él
was published in the Proceedings of the National Academy of Sciences of the United States of
America. This is a multidisciplinary journal and is assigned to another journal set than most of the
papers published in bibliometrics (which are assigned to library and information science). De este modo,
by using journal sets as a field categorization scheme, the paper would not be compared with its
“true” reference papers, but with various papers from many different fields, which are usually
published in multidisciplinary journals. An appropriate reference set for this paper would be

Estudios de ciencias cuantitativas

1557

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
4
1
5
5
3
1
8
7
1
0
0
0
q
s
s
_
a
_
0
0
0
8
9
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

How can citation impact in bibliometrics be normalized?

all papers published in journals in the library and information science set. If one decides to
manually collect the reference papers for comparison (see above), the ideal reference set for the
paper by Hirsch (2005) would consist of all papers publishing a variant of the h-index or all papers
having introduced an indicator combining the number of publications and the number of citations
in a single number.

The idea of citing-side normalization has been introduced by Zitt and Small (2008). They pro-
posed a modification of the Journal Impact Factor ( JIF) by fractional citation weighting. The JIF is a
popular journal metric that is published in the Journal Citation Reports by Clarivate Analytics. El
indicator measures the average citation rate of papers published in a journal within 1 año. Citing-
side normalization is also named as source normalization, fractional counting of citations, o
a priori normalization (waltman, 2016; waltman & van Eck, 2013a). The method focuses on
the citation environment of single citations and weights each citation depending on its cita-
tion environment: A citation from a field with high citation density (on average, authors in
these fields include many cited references in their papers) receives a lower weight than a ci-
tation from a field with low citation density (on average, authors in these fields include only a
few cited references in their papers). The basic idea of the method is as follows: Each citation
is adjusted for the number of references in the citing publication or in the citing journal (as a
representative for the entire field). In the recent decade, some variants of citing-side indicators
have been published (waltman, 2016; waltman & van Eck, 2013a). These variants are pre-
sented in the following based on the explanations by Bornmann and Marx (2015).

SNCS1 ¼

i¼1

(1)

The first variant has been named SNCS1 (Source Normalized Citation Score 1). In the formula,
ai is the average number of linked references in those publications that appeared in the same
journal and in the same publication year as the citing publication i. Linked references are the part
of cited references that refers to papers from journals covered by the citation index (p.ej., WoS or
Scopus). The limitation to linked references (instead of all references) is intended to prevent a
situation in which fields that frequently cite publications are not indexed in WoS are disadvan-
taged (see Marx & Bornmann, 2015). The calculation of the average number of linked references
in SNCS1 is restricted to certain referenced publication years. Imagine a focal paper published in
2008 with a citation window covering a period of 4 años (2008 a 2011). En este caso, every ci-
tation of the focal paper is divided by the average number of linked references to the four previous
años. En otras palabras, a citation from 2010 is divided by the linked cited references from the
período 2007 a 2010. This restriction to recent publication years is designed to prevent fields
that cite rather older literature from being disadvantaged in the normalization (waltman &
van Eck, 2013b).

SNCS2 ¼

i¼1

(2)

SNCS2 is the second variant of citing-side indicators. Aquí, each citation is divided by the
number of linked cited references in the citing publication. Por lo tanto, the journal perspective
is not considered in this variant. The selection of the reference publication years is analogous
to SNCS1.

SNCS3 ¼

i¼1

1
piri

(3)

1558

Estudios de ciencias cuantitativas

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
4
1
5
5
3
1
8
7
1
0
0
0
q
s
s
_
a
_
0
0
0
8
9
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

How can citation impact in bibliometrics be normalized?

SNCS3 is a combination of SNCS1 and SNCS2. ri is equally defined as in SNCS2. pi is the share
of papers that contain at least one linked cited reference among the following papers: desde el
same journal and publication year as the citing paper i. The selection of the referenced publica-
tion years is analogous to SNCS1 and SNC2.

Whereas Leydesdorff, Radicchi, et al. (2013) concluded that cited-side normalization outper-
forms citing-side normalization, the empirical results of Waltman and van Eck (2013a) y
Bornmann and Marx (2015) demonstrated that citing-side normalization is more successful in
field-normalizing citation impact than cited-side normalization. Por lo tanto, it seems reasonable
for reaching the goal of field normalization to weight each citation “based on the referencing
behavior of the citing publication or the citing journal” (waltman & van Eck, 2013a, pag. 703).
The comparison of the three citing-side approaches by Waltman and van Eck (2013b, pag. 842)
revealed that

SNCS(2) should not be used. Además, the SNCS(3) approach appears to be preferable
over the SNCS(1) acercarse. The excellent performance of the SNCS(3) approach in the case
of classification system C … suggests that this approach may be especially well suited for
fine-grained analyses aimed for instance at comparing researchers or research groups active
in closely related areas of research.

The results by Bornmann and Marx (2015), sin embargo, did not reveal these large differences

between the three indicator variants.

Cited-side normalization is frequently confronted with the problem that the used field catego-
rization scheme assigns papers to more than one field. De este modo, it is necessary to consider these
multiple assignments in the calculation of field-normalized indicators (see Waltman, van Eck,
et al., 2011). As multiple assignments are not possible with citing-side normalization, this prob-
lem is no longer existent—a further decisive advantage of the approach.

4. PURPOSE OF THE STUDY—THE COMBINATION OF CITING-SIDE NORMALIZATION AND
CITATION PERCENTILES

En la sección 3, the advantages of field normalization using citing-side approaches have been
demonstrated based on the previous literature. Although these advantages have been reported
in several papers over many years, these approaches have not been established as standard indi-
cators in (aplicado) bibliometría. Por ejemplo, the Leiden Ranking (see https://www.leidenranking
.com) does not consider citing-side indicators, but percentile-based cited-side indicators. Uno
important reason for the avoidance of citing-side indicators might be that these indicators are more
complicated to understand (and explain) than many cited-side indicators and indicators that are
not based on field normalization. The results by Hammarfelt and Rushforth (2017) muestra esa
“simple and well-established indicators, like the JIF and the h-index, are preferred” (páginas. 177–178)
when indicators are used in practice. Jappe, Pithan, and Heinze (2018) similarly wrote that “the
journal impact factor (JIF) … and the Hirsch Index (h-index or HI) … have spread widely among
research administrators and funding agencies over the last decade.” According to the University of
Waterloo Working Group on Bibliometrics (2016), “there is often a demand for simple measures
because they are easier to use and can facilitate comparisons” (pag. 2).

This study is intended to propose a field normalization approach that combines citing-side
normalization and citation percentiles. The advantage of the combination lies in the abandonment
of a field classification system (by using citing-side normalization) and the realization of field
normalized scores (percentiles) that are relatively simple to understand and being applied in

Estudios de ciencias cuantitativas

1559

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
4
1
5
5
3
1
8
7
1
0
0
0
q
s
s
_
a
_
0
0
0
8
9
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

How can citation impact in bibliometrics be normalized?

research evaluation. In the first step of the approach, weighted citation counts are calculated
based on the formula (see above) presented by Waltman and van Eck (2013a). en este estudio,
the SNCS3 is used, as Waltman and van Eck (2013b) recommended its use (based on their em-
pirical results). Sin embargo, the approach is not bound to this SNCS variant. In the second step, el
percentile approach proposed by Bornmann and Williams (2020) is used to calculate citation
percentiles based on SNCS3. In this step, también, it is possible to use another percentile approach
such as those proposed by Bornmann, Leydesdorff, and Mutz (2013) or Bornmann and Mutz
(2014). This study prefers the approach by Bornmann and Williams (2020), because the authors
point out the advantages of their approach over previous approaches.

Bornmann and Williams (2020) calculated cumulative frequencies in percentages (CPs) como
demonstrated in Table 1 based on the size-frequency distribution (Egghe, 2005) to receive
citation percentiles. The table shows the citation counts and SNCS3 for 24 fictitious papers.
Por ejemplo, there are five papers in the set with 12 citations and a weighted citation impact
de 0.45 cada. Note that not all papers with five citations have an SNCS3 score of 0.45 and vice
versa. For the indicator CP-EXWC (the subscript WC stands for weighted citations), la primera
porcentaje (for papers with 1 citation) is set at 0. The calculation of the cumulative percentage
starts in the second row with the percentage of the lowest citation count (16.67%). By setting the
first row to zero, CP-EXWC measures exactly the percentage of papers with lower citation impact
in the set of papers. Por ejemplo, CP-EXWC = 95.83 means that exactly 95.83% de los papeles
in the set of 24 papers received a citation impact—measured by SNCS3—that is below the
weighted citation impact of 4.51. 16.67% of the papers received less impact than the weighted
citations of 0.20.

CP-EXWC can be calculated for all papers in a database (p.ej., all WoS papers) with SNCS3
scores included (or the scores based on another variant). Porque (weighted) citation impact
depends on the length of the citation window, CP-EXWC should be calculated based on all papers
en 1 año (es decir., separated by publication years). With CP-EXWC calculated using SNCS3, one re-
ceives a field-normalized indicator that is simple to understand—because the scores are

Mesa 1.

Cumulative percentages (CP-EXWC) Residencia en 24 fictitious papers

Citations
1

SNCS3
(rounded)
0.00

Número
of papers
4

Percentage
16.67

Cumulative percentage
(CP-EXWC)
0

0.20

0.37

0.45

0.48

0.67

1.16

1.63

2.17

4.51

12.50

4.17

20.83

8.33

12.50

4.17

8.33

4.17

Total

100.00

16.67

29.17

33.33

54.17

62.50

70.83

83.33

87.50

95.83

Estudios de ciencias cuantitativas

1560

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
4
1
5
5
3
1
8
7
1
0
0
0
q
s
s
_
a
_
0
0
0
8
9
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

How can citation impact in bibliometrics be normalized?

cumulative percentages—and it is based on an advantageous method of field normalization (ver
arriba). The definition of CP-EXWC for a focal paper is that x% of papers published in the same
year received a lower weighted citation impact than the focal paper. Weighted citation impact
means that each citation of the focal paper is weighted by the citation behavior in its field. Este
definition is simple to understand, not only by bibliometric experts but also by laypersons.

As citation impact is dependent not only on the publication year but also on the document
type of the cited publication (ver, p.ej., Lundberg, 2007), the CP-EXWC calculation should not
only be separated by publication year, but also by document type. en este estudio, it was not neces-
sary to consider the document type in the calculation, because only articles were included.

5. MÉTODOS

The bibliometric data used in this paper are from an in-house version of the WoS used at the Max
Planck Society (Munich, Alemania). en este estudio, all papers are included from this database with
the document type “article” and published between 2011 y 2015. The data set contains n =
7,908,584 documentos; for n = 914,472 papers no SNCS3 values are available in the in-house data-
base. De este modo, the study is based on n = 6,994,112 documentos. The SNCS3 scores and CP-EXWC values
have been calculated as explained in the sections above. In the calculation of the SNCS3 indi-
cator, we followed the procedure as explained by Waltman and van Eck (2013b). Whereas
Waltman and van Eck (2013b), sin embargo, only included selected core journals from the WoS
database in the SNCS3 calculation, the SNCS3 scores for the present study were calculated
based on all journals in the WoS database.

6. RESULTADOS

Cifra 1 shows the distribution of SNCS3 scores for 6 years using frequency distributions. Es
clearly visible that the SNCS3 distributions are very skewed and characterized by outliers (artículos
with very high weighted citation impact).

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
4
1
5
5
3
1
8
7
1
0
0
0
q
s
s
_
a
_
0
0
0
8
9
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Cifra 1. Quantile plots (Cox, 2005) of SNCS3 scores for papers published between 2011 y 2015.

Estudios de ciencias cuantitativas

1561

How can citation impact in bibliometrics be normalized?

Against the backdrop of these skewed distributions (despite citation weighting by citing-side
normalization), it sounds reasonable (more than ever) to calculate percentiles based on SNCS3
puntuaciones. According to Seglen (1992), skewed citation distributions “should probably be regarded
as the basic probability distribution of citations, reflecting both the wide range of citedness values
potentially attainable and the low probability of achieving a high citation rate” (pag. 632). Este
basic probability distribution does not appear to be valid only for citation distributions, pero también
weighted citation distributions (based on SNCS3). Similar to citations, the SNCS3 indicator
appears to follow the so-called “bibliometric laws” (de Bellis, 2009, pag. xxiv). This is a set of
regularities working behind citation processes according to which a certain number of citations
is related to the authors generating them (in their papers). The common feature of these processes
(and similar processes based on the number of publications or text words) is an “amazingly steady
tendency to the concentration of items on a relatively small stratum of sources” (de Bellis, 2009,
pag. xxiv).

One of these regularities leading to skewed citation distributions might be (más grande) quality
differences between the research published in the papers (Aksnes et al., 2019). A second regu-
larity might be the type of contribution made by the paper: Por ejemplo, one can expect many
more citations for methods papers than for papers contributing empirical results (Bornmann,
2015; van Noorden, Maher, & Nuzzo, 2014). A third regularity might be a cumulative advantage
effect by which “already frequently cited [publicaciones] have a higher probability of receiving
even more citations” (van Raan, 2019, pag. 239). According to Ruocco, Daraio, et al. (2017),
“Price’s [Derek J. de Solla Price] assumption was that the papers to be cited are chosen at random
with a probability that is proportional to the number of citations those same papers already have.
De este modo, highly cited papers are likely to gain additional citations, giving rise to the rich get richer
cumulative effect.”

Cifra 2 shows the distribution of CP-EXWC values for papers published between 2010 y
2015. Comparison of Figure 2 with Figure 1 reveals that the scores are no longer skewed.

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
4
1
5
5
3
1
8
7
1
0
0
0
q
s
s
_
a
_
0
0
0
8
9
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Cifra 2. Distribution of CP-EXWC values for papers published between 2010 y 2015.

Estudios de ciencias cuantitativas

1562

How can citation impact in bibliometrics be normalized?

Papers with low citation impact (es decir., low CP-EXWC scores) are prevalent, but the distributions
approximate a uniform distribution.

en este estudio, the proposed indicator CP-EXWC has been exemplarily applied to publication
and citation data of some countries: Suiza, Reino Unido, United States, Alemania,
Porcelana, y japon. The results are shown in Figure 3. The upper graph in the figure is based on
full counting of the countries’ papers. De este modo, each paper contributes to the citation impact of a
country with a weight of 1—independent of the additional number of countries involved. El
score for a country shown in Figure 3 is its CP-EXWC median value. The dotted line in the graph
marks the worldwide average. The score for Switzerland in the upper graph is above that line and
medio, Por ejemplo, that on average, 60.85% of the papers worldwide have a weighted citation
impact that is below the weighted citation impact of papers with a Swiss address.

The results in the upper graph correspond to results based on other (field-normalized) citation-
based indicators (p.ej., Bornmann & Leydesdorff, 2013; Bornmann, Wagner, & Leydesdorff, 2018;
esteban, Stahlschmidt, & Hinze, 2020). When citation impact is measured size independently,
certain small countries such as Switzerland show an excellent performance (the Netherlands is
another example, although it is not considered here). It follows the United Kingdom in the upper
graph of Figure 3, which has exceeded the United States in citation impact in recent years. Porcelana
and Japan are at the bottom of the country list. Although these results come as no real surprise,
differences from previous results are also observable. One difference refers to the performance
differences between the countries that do not appear to be very large. Por ejemplo, the differ-
ences between Switzerland, the United Kingdom, and the United States exceed no more than
four percentage points. Another difference from previous studies concerns the performance level.
In previous studies, countries such as Switzerland show an excellent performance far away
from midlevel performance. If we assume that the dotted line in Figure 3 represents a midlevel

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
4
1
5
5
3
1
8
7
1
0
0
0
q
s
s
_
a
_
0
0
0
8
9
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Cifra 3. CP-EXWC for papers published between 2010 y 2015 by six countries: Suiza (norte = 138,947), Reino Unido (norte = 540,287),
United States (norte = 1,949,391), Alemania (norte = 510,207), Porcelana (norte = 1,096,608), y japon (norte = 394,328). The national numbers of papers are based
on full counting.

Estudios de ciencias cuantitativas

1563

How can citation impact in bibliometrics be normalized?

actuación (50% of the papers worldwide exhibit a lower performance), the best countries
(and also the worst) are not far away from 50%. De término medio, Por ejemplo, papers from
Switzerland are (solo) alrededor 10 percentage points above the midlevel performance.

The lower graph in Figure 3 is based on fractional counting. De este modo, it has been considered
that many papers were published by more than one country. en este estudio (which is based on
the SNCS3 impact indicator), the CP-EXWC score for a paper has been weighted by the number
of countries given on a paper (Bornmann & williams, 2020).

The following formula leads to a fractionally counted mean CP-EXWC score for a country:

(cid:4)

d
CPEX1 (cid:2) FR1

mCPEX Fð Þ ¼

Þ þ CPEX2 (cid:2) FR2
PAG
y
i¼1 FRi

(cid:2)

Þ þ … þ CPEXy (cid:2) FRy

(cid:3)

(cid:5)

(4)

where CPEX1 to CPEXy are weighted by the number of countries given on a paper. Por ejemplo, si
a paper was published by authors from four countries, the paper is weighted by 0.25. The frac-
tional assignment (weighting) is included by the notation FRi for paper i = 1 to paper y. The sums of
the CP-EXWC scores for paper 1 to paper y published by the unit are divided by the sums of the
weightings for paper 1 to paper y.

By applying fractional counting, citation impact benefits arising from collaborations are
adjusted. As the results in the lower graph in Figure 3 espectáculo, fractional impact counting changes
the national results differently: Whereas larger differences are visible for Switzerland, los unidos
Kingdom, y alemania, the differences are smaller for Japan and China. Compared with the
upper graph in Figure 3, China and Japan do not really profit from controlling international
collaborations in the lower graph: The CP-EXWC scores only change from 46.80% a 46.49%
(Porcelana) y 46.62% a 46.07% ( Japón). In contrast to China, Switzerland appears to profit signif-
icantly in terms of citation impact from international collaboration: Its CP-EXWC decreases from
60.85% (upper graph) a 55.5% (lower graph). The other two countries that also appear to profit
from international collaboration are the United Kingdom and Germany (around four percentage
puntos).

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
4
1
5
5
3
1
8
7
1
0
0
0
q
s
s
_
a
_
0
0
0
8
9
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

7. DISCUSIÓN

Because only experts from the same field can properly assess the research of their colleagues, el
peer review process is the dominant research evaluation method. Since around the 1980s, the use
of indicators in research evaluation has become increasingly popular. One reason might be that
“direct assessment of research activity needs expert judgment, which is costly and onerous, entonces
proxy indicators based on metadata around research inputs and outputs are widely used”
(Adams, Loach, & Szomszor, 2016, pag. 2). For Lamont (2012), another reason is that “governments
have turned to new public management tools to ensure greater efficacy, with the result that quan-
titative measures of performance and benchmarking are diffusing rapidly” (pag. 202). Sin embargo,
peer review and the use of indicators do not have to be incompatible approaches; it is seen as
the “ideal way” in research evaluation to combine both methods in the so-called informed peer
review process. According to Waltman and van Eck (2016, pag. 542), “scientometric indicators can
… be used by the peer review committee to complement the results of in-depth peer review with
quantitative information, especially for scientific outputs that have not been evaluated in detail by
the committee.” In the confrontation of peer review and bibliometrics, one should consider
that both methods are related: “citations provide a built-in form of peer review” (McAllister
et al., 1983, pag. 205).

Citation analysis is one of the most important methods in bibliometrics, as the method appears
to measure quality issues: “at high frequency, citations are good indicators of utility, significance,

Estudios de ciencias cuantitativas

1564

How can citation impact in bibliometrics be normalized?

even the notion of impact. The late sociologist of science, Robert Merton likened citations to
repayments of intellectual debts. The normative process in science requires authors to acknowl-
edge relevant previous contributions” (Panchal & Pendlebury, 2012, pag. 1144). One of the major
challenges of citation analyses is the field dependency of citations. If larger units in science are
evaluated that are working in many fields, it is necessary to consider these differences in the
statistical analyses (Bornmann, 2020). According to Kostoff (2002), “citation counts depend
strongly on the specific technical discipline, or sub-discipline, being examined … The documen-
tation and citation culture can vary strongly by sub-discipline. Since citation counts can vary
sharply across sub-disciplines, absolute counts have little meaning, especially in the absence
of absolute citation count performance standards” (pag. 53; see also Fok & Franses, 2007).

One solution to the problem of field-specific differences in citation counts is to contextualize the
results of citation analyses “case by case, considering all the relevant information” (D’Agostino,
Dardanoni, & Ricci, 2017, pag. 826). According to Waltman and van Eck (2019), one can “use
straightforward non-normalized indicators and to contextualize these indicators with additional
information that enables evaluators to take into account the effect of field differences” (pag. 295).
This might be the best solution if smaller research groups or institutions working in clearly definable
fields are evaluated. For this solution, sin embargo, it is necessary to involve not only a bibliometric
expert in the evaluation but also an expert from the evaluated field to contextualize these indica-
tores. Por ejemplo, for the identification of research groups working in the same field as the focal
grupo, it is necessary for an expert to identify these groups that can be used for comparison of the
focal group. This solution of contextualizing the number of times when research is cited is stretched
to its limits when large units such as organizations or countries are addressed in evaluations. Estos
units are multidisciplinary by nature.

Since the 1980s, many different methods have been proposed to field-normalize citations. Él
has not been possible to establish a standard method until now. en este estudio, an approach is pro-
posed that combines two previously introduced methods: citing-side normalization and percen-
tiles. The advantage of combining two methods is that their advantages can be integrated into a
single solution. Based on citing-side normalization, each citation is field weighted and, por lo tanto,
contextualized in its field. The most important advantage of citing-side normalization is that it is
not necessary to work with a specific field categorization scheme. The disadvantages of citing-
side normalization—the calculation is complex and the values elusive—can be compensated by
calculating percentiles based on the field-weighted citations. Por un lado, percentiles are
well understandable: It is the percentage of papers published in the same year with lower citation
impacto. Por otro lado, weighted citation distributions are skewed distributions including
outliers. Percentiles are well suited to assigning the position of a focal paper in such skewed
distributions including a field-specific set of papers.

Many different approaches of percentile calculation exist (Bornmann, Leydesdorff, et al.,
2013). According to Schreiber (2013, pag. 829) “all the discussed methods have advantages and
disadvantages. Further investigations are needed to clarify what the optimal solution to the prob-
lem of calculating percentiles and assigning papers to PRCs [percentile rank classes] might be,
especially for large numbers of tied papers.” Bornmann and Williams (2020) appear to have
found a percentile solution with comparably good properties. en este estudio, their percentile ap-
proach based on weighted citations (CP-EXWC) has been applied to the analysis of several coun-
intentos. The country results are similar to many other published results. This correspondence in the
results can be interpreted as a good sign for the new approach: It appears to measure field-
normalized citation impact in a similar way to other indicators. Sin embargo, the approach also
reveals the importance of measuring citation impact based on fractional counting. Several coun-
tries are strongly internationally oriented, which has a larger influence on the results.

Estudios de ciencias cuantitativas

1565

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
4
1
5
5
3
1
8
7
1
0
0
0
q
s
s
_
a
_
0
0
0
8
9
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

How can citation impact in bibliometrics be normalized?

Further studies are necessary to investigate the new approach introduced here. These studies
could also focus on other units than those considered in this study (p.ej., institutions and research
grupos). Además, it would be interesting to know how the new approach can be understood
by people who are not bibliometric experts: Is it as easy to understand as expected, or are there
difficulties in understanding it?

EXPRESIONES DE GRATITUD

The bibliometric data used in this paper are from an in-house database developed and maintained
by the Max Planck Digital Library (MPDL, Munich) and derived from the Science Citation Index
Expanded (SCI-E), Social Sciences Citation Index (SSCI), and Arts and Humanities Citation Index
(AHCI) prepared by Clarivate Analytics, formerly the IP & Science business of Thomson Reuters
(Filadelfia, Pensilvania, EE.UU).

CONFLICTO DE INTERESES

The author has no competing interests.

INFORMACIÓN DE FINANCIACIÓN

No funding has been received for this research.

DISPONIBILIDAD DE DATOS

The data cannot be made available in a data repository because the provider of the data
(Clarivate Analytics) does not allow this.

REFERENCIAS

Adams, J., Loach, T., & Szomszor, METRO. (2016). Interdisciplinary research:
Methodologies for identification and assessment. Do we know what
we are measuring? Londres, Reino Unido: Digital Science.

Aksnes, D. w., Langfeldt, l., & Wouters, PAG. (2019). Citations, citation
indicators, and research quality: An overview of basic concepts
and theories. Sage Open, 9(1). DOI: https://doi.org/10.1177
/2158244019829575

Bornmann, l. (2015). Nature’s top 100 revisited. Journal of the
Association For Information Science and Technology, 66(10),
2166. DOI: https://doi.org/10.1002/asi.23554

Bornmann, l. (2020). Bibliometrics-based decision trees (BBDTs)
based on bibliometrics-based heuristics (BBHs): Visualized
guidelines for the use of bibliometrics in research evaluation.
Estudios de ciencias cuantitativas, 1(1), 171–182. DOI: https://doi
.org/10.1162/qss_a_00012

Bornmann, l., Bowman, B. F., Bauer, J., Marx, w., Schier, h., &
Palzenberger, METRO. (2014). Bibliometric standards for evaluating
research institutes in the natural sciences. In B. Cronin & C.
Sugimoto (Editores.), Beyond bibliometrics: Harnessing multidimen-
sional indicators of scholarly impact (páginas. 201–223). Cambridge,
MAMÁ: CON prensa.

Bornmann, l., & Daniel, H.-D. (2008). Selecting manuscripts for a
high impact journal through peer review: A citation analysis of
communications that were accepted by Angewandte Chemie
International Edition, or rejected but published elsewhere. Diario
of the American Society for Information Science and Technology,
59(11), 1841–1852. DOI: https://doi.org/10.1002/asi.20901

Bornmann, l., & Leydesdorff, l. (2013). Macro-indicators of citation
impacts of six prolific countries: InCites data and the statistical sig-
nificance of trends. MÁS UNO, 8(2), e56768. DOI: https://doi.org
/10.1371/diario.pone.0056768, PMID: 23418600, PMCID:
PMC3572076

Bornmann, l., Leydesdorff, l., & Mutz, R. (2013). The use of percen-
tiles and percentile rank classes in the analysis of bibliometric data:
Opportunities and limits. Journal of Informetrics, 7(1), 158–165.
DOI: https://doi.org/10.1016/j.joi.2012.10.001

Bornmann, l., & Marewski, j. norte. (2019). Heuristics as conceptual
lens for understanding and studying the usage of bibliometrics in
research evaluation. cienciometria, 120(2), 419–459. DOI:
https://doi.org/10.1007/s11192-019-03018-x

Bornmann, l., & Marx, W.. (2014). How to evaluate individual re-
searchers working in the natural and life sciences meaningfully?
A proposal of methods based on percentiles of citations.
cienciometria, 98(1), 487–509. DOI: https://doi.org/10.1007
/s11192-013-1161-y

Bornmann, l., & Marx, W.. (2015). Methods for the generation of
normalized citation impact scores in bibliometrics: Which method
best reflects the judgements of experts? Journal of Informetrics, 9(2),
408–418. DOI: https://doi.org/10.1016/j.joi.2015.01.006

Bornmann, l., Marx, w., & Barth, A. (2013). The normalization of
citation counts based on classification systems. Publications, 1(2),
78–86. DOI: https://doi.org/10.3390/publications1020078

Bornmann, l., & Mutz, R. (2014). From P100 to P100’: A new citation-
rank approach. Journal of the Association For Information Science

Estudios de ciencias cuantitativas

1566

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
4
1
5
5
3
1
8
7
1
0
0
0
q
s
s
_
a
_
0
0
0
8
9
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

How can citation impact in bibliometrics be normalized?

and Technology, 65(9), 1939–1943. DOI: https://doi.org/10.1002
/asi.23152

Bornmann, l., Schier, h., Marx, w., & Daniel, H.-D. (2011). Is inter-
active open access publishing able to identify high-impact sub-
missions? A study on the predictive validity of Atmospheric
Chemistry and Physics by using percentile rank classes. Diario de
the American Society for Information Science and Technology,
62(1), 61–71. DOI: https://doi.org/10.1002/asi.21418

Bornmann, l., Wagner, C., & Leydesdorff, l. (2018). The geography
of references in elite articles: What countries contribute to the
archives of knowledge. MÁS UNO, 13(3), e0194805. DOI:
https://doi.org/10.1371/journal.pone.0194805, PMID: 29579088,
PMCID: PMC5868817

Bornmann, l., & williams, R. (2020). An evaluation of percentile
measures of citation impact, and a proposal for making them better.
cienciometria, 124, 1457–1478. DOI: https://doi.org/10.1007
/s11192-020-03512-7

Bornmann, l., & Wohlrabe, k. (2019). Normalisation of citation
impact in economics. cienciometria, 120(2), 841–884. DOI:
https://doi.org/10.1007/s11192-019-03140-w

Boyack, k. w., & Klavans, R. (2010). Co-citation analysis, biblio-
graphic coupling, and direct citation: Which citation approach
represents the research front most accurately? Journal of the
American Society for Information Science and Technology, 61(12),
2389–2404. DOI: https://doi.org/10.1002/asi.21419

Colliander, C., & Ahlgren, PAG. (2011). The effects and their stability of
field normalization baseline on relative performance with respect
to citation impact: Un estudio de caso de 20 natural science departments.
Journal of Informetrics, 5(1), 101–113. DOI: https://doi.org/10
.1016/j.joi.2010.09.003

Colliander, C., & Ahlgren, PAG. (2019). Comparison of publication-
level approaches to ex-post citation normalization. cienciometria,
120(1), 283–300. DOI: https://doi.org/10.1007/s11192-019
-03121-z

Cox, norte. j. (2005). Speaking Stata: The protean quantile plot.
Stata Journal, 5(3), 442–460. DOI: https://doi.org/10.1177
/1536867X0500500312

Crespo, j. A., li, Y. r., & Ruiz-Castillo, j. (2013). The measurement of
the effect on citation inequality of differences in citation practices
across scientific fields. MÁS UNO, 8(3). DOI: https://doi.org/10.1371
/diario.pone.0058727, PMID: 23516542, PMCID: PMC3597729
D’Agostino, METRO., Dardanoni, v., & Ricci, R. GRAMO. (2017). How to stan-
dardize (if you must). cienciometria, 113(2), 825–843. DOI:
https://doi.org/10.1007/s11192-017-2495-7

de Bellis, norte. (2009). Bibliometrics and citation analysis: From the Science

Citation Index to cybermetrics. Lanham, Maryland: Scarecrow Press.

Egghe, l. (2005). Power laws in the information production process:
Lotkaian informetrics. Kidlington, Reino Unido: Emerald Group Publishing
Limited. DOI: https://doi.org/10.1108/S1876-0562(2005)05

Fok, D., & Franses, PAG. h. (2007). Modeling the diffusion of scientific
publicaciones. Journal of Econometrics, 139(2), 376–390. DOI:
https://doi.org/10.1016/j.jeconom.2006.10.021

Gates, A. J., Ke, P., Varol, o., & Barabasi, A. l. (2019). Nature’s
reach: Narrow work has broad impact. Naturaleza, 575(7781), 32–34.
DOI: https://doi.org/10.1038/d41586-019-03308-7, PMID:
31695218

Glänzel, w., & Moed, h. (2013). Opinion paper: Thoughts and
facts on bibliometric indicators. cienciometria, 96(1), 381–394.
DOI: https://doi.org/10.1007/s11192-012-0898-z

Hammarfelt, B., & Rushforth, A. D. (2017). Indicators as judgment de-
vicios: An empirical study of citizen bibliometrics in research
evaluación. Research Evaluation, 26(3), 169–180. DOI: https://
doi.org/10.1093/reseval/rvx018

Haunschild, r., Schier, h., Marx, w., & Bornmann, l. (2018).
Algorithmically generated subject categories based on citation
relaciones: An empirical micro study using papers on overall water
splitting. Journal of Informetrics, 12(2), 436–447. DOI: https://doi
.org/10.1016/j.joi.2018.03.004

Hicks, D., Wouters, PAG., waltman, l., de Rijcke, S., & Rafols, I.
(2015). Bibliometrics: The Leiden Manifesto for research metrics.
Naturaleza, 520(7548), 429–431. DOI: https://doi.org/10.1038/520429a,
PMID: 25903611

Hirsch, j. mi. (2005). An index to quantify an individual’s scientific
research output. procedimientos de la Academia Nacional de Ciencias
of the United States of America, 102(46), 16569–16572. DOI:
https://doi.org/10.1073/pnas.0507655102, PMID: 16275915,
PMCID: PMC1283832

Hou, J., Cacerola, h. X., guo, T., Sotavento, I., kong, X. J., & Xia, F. (2019).
Prediction methods and applications in the science of science:
A survey. Computer Science Review, 34. DOI: https://doi.org
/10.1016/j.cosrev.2019.100197

Ioannidis, j. PAG. A., Boyack, K., & Wouters, PAG. F. (2016). Citation met-
rics: A primer on how (no) to normalize. PLOS Biology, 14(9),
e1002542. DOI: https://doi.org/10.1371/journal.pbio.1002542,
PMID: 27599158, PMCID: PMC5012555

Jappe, A., Pithan, D., & Heinze, t. (2018). Does bibliometric research
confer legitimacy to research assessment practice? A sociological
study of reputational control, 1972–2016. MÁS UNO, 13(6),
e0199031. DOI: https://doi.org/10.1371/journal.pone.0199031,
PMID: 29902239, PMCID: PMC6002049

Kostoff, R. norte. (2002). Citation analysis of research performer quality.
cienciometria, 53(1), 49–71. DOI: https://doi.org/10.1023/A
:1014831920172

Kostoff, R. NORTE., Murday, j. S., Lau, C. GRAMO. y., & Tolles, W.. METRO. (2006).
The seminal literature of nanotechnology research. Diario de
Nanoparticle Research, 8(2), 193–213. DOI: https://doi.org/10
.1007/s11051-005-9034-9

Lamont, METRO. (2012). Toward a comparative sociology of valuation
and evaluation. Annual Review of Sociology, 38(1), 201–221.
DOI: https://doi.org/10.1146/annurev-soc-070308-120022

Leydesdorff, l., & Milojevic(cid:1), S. (2015). The citation impact of German
sociology journals: Some problems with the use of scientometric
indicators in journal and research evaluations. Soziale Welt, 66(2),
193–204. DOI: https://doi.org/10.5771/0038-6073-2015-2-193
Leydesdorff, l., Radicchi, F., Bornmann, l., Castellano, C., & de
Nooy, W.. (2013). Field-normalized impact factors (IFs): A com-
parison of rescaling and fractionally counted IFs. Journal of the
American Society for Information Science and Technology, 64(11),
2299–2309. DOI: https://doi.org/10.1002/asi.22911

Leydesdorff, l., Wagner, C. S., & Bornmann, l. (2014). The European
Union, Porcelana, and the United States in the top-1% and top-10%
layers of most-frequently cited publications: Competition and
collaborations. Journal of Informetrics, 8(3), 606–617. DOI:
https://doi.org/10.1016/j.joi.2014.05.002

Lundberg, j. (2007). Lifting the crown—citation z-score. Diario de
Informetrics, 1(2), 145–154. DOI: https://doi.org/10.1016/j.joi
.2006.09.007

Marx, w., & Bornmann, l. (2015). On the causes of subject-specific
citation rates in Web of Science. cienciometria, 102(2), 1823–1827.
DOI: https://doi.org/10.1007/s11192-014-1499-9

McAllister, PAG. r., Narin, F., & Corrigan, j. GRAMO. (1983). Programmatic
evaluation and comparison based on standardized citation scores.
IEEE Transactions on Engineering Management, 30(4), 205–211.
DOI: https://doi.org/10.1109/TEM.1983.6448622

Milojevic(cid:1), S. (2020). Practical method to reclassify Web of Science
articles into unique subject categories and broad disciplines.

Estudios de ciencias cuantitativas

1567

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
4
1
5
5
3
1
8
7
1
0
0
0
q
s
s
_
a
_
0
0
0
8
9
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

How can citation impact in bibliometrics be normalized?

Estudios de ciencias cuantitativas, 1(1), 183–206. DOI: https://doi.org
/10.1162/qss_a_00014

Mingers, J., & Leydesdorff, l. (2015). A review of theory and prac-
tice in scientometrics. European Journal of Operational Research,
246(1), 1–19. DOI: https://doi.org/10.1016/j.ejor.2015.04.002
Moed, h. F. (2017). Applied evaluative informetrics. Heidelberg,
Alemania: Saltador. DOI: https://doi.org/10.1007/978-3-319
-60522-7

Moed, h. F., & Halevi, GRAMO. (2015). The multidimensional assessment
of scholarly research impact. Journal of the American Society for
Information Science and Technology, 66(10), 1988–2002. DOI:
https://doi.org/10.1002/asi.23314

Narin, F. (1976). Evaluative bibliometrics: The use of publication
and citation analysis in the evaluation of scientific activity.
Cherry Hill, Nueva Jersey: Computer Horizons.

Nosek, B. A., graham, J., Lindner, norte. METRO., Kesebir, S., Hawkins, C. B.,
Hahn, C., … Tenney, mi. R. (2010). Cumulative and career-stage
citation impact of social-personality psychology programs and their
miembros. Personality and Social Psychology Bulletin, 36(10),
1283–1300. DOI: https://doi.org/10.1177/0146167210378111,
PMID: 20668215

Panchal, h., & Pendlebury, D. A. (2012). David A. Pendlebury.

Current Science, 103(10), 1144–1145.

Radicchi, F., & Castellano, C. (2011). Rescaling citations of publi-
cations in physics. Physical Review E, 83(4). DOI: https://doi.org
/10.1103/PhysRevE.83.046116, PMID: 21599249

Ruiz-Castillo, J., & waltman, l. (2015). Field-normalized citation
impact indicators using algorithmically constructed classification
systems of science. Journal of Informetrics, 9(1), 102–117. DOI:
https://doi.org/10.1016/j.joi.2014.11.010

Ruocco, GRAMO., Daraio, C., Folli, v., & Leonetti, METRO. (2017). Bibliometric
indicators: The origin of their log-normal distribution and why
they are not a reliable proxy for an individual scholar’s talent.
Palgrave Communications, 3, 17064. DOI: https://doi.org/10
.1057/palcomms.2017.64

Schreiber, METRO. (2013). How much do different ways of calculating
percentiles influence the derived performance indicators? A
case study. cienciometria, 97(3), 821–829. DOI: https://doi.org
/10.1007/s11192-013-0984-x

Schubert, A., & Braun, t. (1986). Relative indicators and relational
charts for comparative assessment of publication output and
citation impact. cienciometria, 9(5–6), 281–291. DOI: https://doi
.org/10.1007/BF02017249

Seglen, PAG. oh. (1992). The skewness of science. Journal of the American
Society for Information Science, 43(9), 628–638. DOI: https://doi
.org/10.1002/(CIENCIA)1097-4571(199210)43:9<628::AID-ASI5>3.0
.CO;2-0

Shu, F., Julien, C.-A., zhang, l., Qiu, J., zhang, J., & Larivière, V. (2019).
Comparing journal and paper level classifications of science.
Journal of Informetrics, 13(1), 202–225. DOI: https://doi.org
/10.1016/j.joi.2018.12.005

Sjögårde, PAG., Ahlgren, PAG., & waltman, l. (2020). Algorithmic labeling
in hierarchical classifications of publications: Evaluation of biblio-
graphic fields and term weighting approaches. Retrieved July 29,
2020, from https://arxiv.org/abs/2004.08090

Smolinsky, l., & Lercher, A. (2012). Citation rates in mathematics: A
study of variation by subdiscipline. cienciometria, 91(3), 911–924.
DOI: https://doi.org/10.1007/s11192-012-0647-3

esteban, D., Stahlschmidt, S., & Hinze, S. (2020). Performance and
structures of the German science system 2020. Studies on the
German innovation system No. 5-2020. Berlina, Alemania: Alemán
Centre for Higher Education Research and Science Studies
(DZHW).

Strotmann, A., & zhao, D. (2010). Combining commercial citation
indexes and open-access bibliographic databases to delimit
highly interdisciplinary research fields for citation analysis. Diario
of Informetrics, 4(2), 194–200. DOI: https://doi.org/10.1016/j.joi
.2009.12.001

Sugimoto, C. r., & Weingart, S. (2015). The kaleidoscope of disci-
plinarity. Journal of Documentation, 71(4), 775–794. DOI:
https://doi.org/10.1108/Jd-06-2014-0082

University of Waterloo Working Group on Bibliometrics. (2016).
White paper on bibliometrics, measuring research outputs through
bibliometría. Waterloo, Canada.

van Noorden, r., Maher, B., & Nuzzo, R. (2014). The top 100 documentos.
Naturaleza, 514(7524), 550–553. DOI: https://doi.org/10.1038
/514550a, PMID: 25355343

van Raan, A. F. j. (2014). Advances in bibliometric analysis: Investigación
performance assessment and science mapping. In W. Blockmans,
l. Engwall, & D. Weaire (Editores.), Bibliometrics: Use and Abuse in
the Review of Research Performance (páginas. 17–28). Londres, Reino Unido:
Portland Press.

van Raan, A. F. j. (2019). Measuring science: Basic principles and
application of advanced bibliometrics. In W. Glänzel, h. F.
Moed, Ud.. Schmoch, & METRO. Thelwall (Editores.), Springer Handbook
of Science and Technology Indicators (páginas. 237–280). cham,
Suiza: Springer International Publishing. DOI: https://doi
.org/10.1007/978-3-030-02511-3_10

Vernon, METRO. METRO., Balas, mi. A., & Momani, S. (2018). Are university
rankings useful to improve research? A systematic review. PLOS
ONE, 13(3). DOI: https://doi.org/10.1371/journal.pone.0193762,
PMID: 29513762, PMCID: PMC5841788

Vinkler, PAG. (1986). Evaluation of some methods for the relative assess-
ment of scientific publications. cienciometria, 10(3–4), 157–177.
DOI: https://doi.org/10.1007/BF02026039

waltman, l. (2016). A review of the literature on citation impact
indicators. Journal of Informetrics, 10(2), 365–391. DOI: https://doi
.org/10.1016/j.joi.2016.02.007

waltman, l., & van Eck, norte. j. (2012). A new methodology for construct-
ing a publication-level classification system of science. Journal of the
American Society for Information Science and Technology, 63(12),
2378–2392. DOI: https://doi.org/10.1002/asi.22748

waltman, l., & van Eck, norte. j. (2013a). Source normalized indica-
tors of citation impact: An overview of different approaches and
an empirical comparison. cienciometria, 96(3), 699–716. DOI:
https://doi.org/10.1007/s11192-012-0913-4

waltman, l., & van Eck, norte. j. (2013b). A systematic empirical com-
parison of different approaches for normalizing citation impact
indicators. Journal of Informetrics, 7(4), 833–849. DOI: https://doi
.org/10.1016/j.joi.2013.08.002

waltman, l., & van Eck, norte. j. (2016). The need for contextualized scien-
tometric analysis: An opinion paper. In I. Ràfols, j. Molas-Gallart, mi.
Castro-Martínez, & R. Woolley (Editores.), Proceedings of the 21st
International Conference on Science and Technology Indicator
(páginas. 541–549). València, España: Universitat Politècnica de València.
waltman, l., & van Eck, norte. j. (2019). Field normalization of scien-
tometric indicators. In W. Glänzel, h. F. Moed, Ud.. Schmoch, &
METRO. Thelwall (Editores.), Springer Handbook of Science and Technology
Indicators (páginas. 281–300). Heidelberg, Alemania: Saltador. DOI:
https://doi.org/10.1007/978-3-030-02511-3_11

waltman, l., van Eck, norte. J., van Leeuwen, t. NORTE., & Visser, METRO. S.
(2013). Some modifications to the SNIP journal impact indica-
colina. Journal of Informetrics, 7(2), 272–285. DOI: https://doi.org
/10.1016/j.joi.2012.11.011

waltman, l., van Eck, norte. J., van Leeuwen, t. NORTE., Visser, METRO. S., &
van Raan, A. F. j. (2011). Towards a new crown indicator: Alguno

Estudios de ciencias cuantitativas

1568

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

1
4
1
5
5
3
1
8
7
1
0
0
0
q
s
s
_
a
_
0
0
0
8
9
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

How can citation impact in bibliometrics be normalized?

theoretical considerations. Journal of Informetrics, 5(1), 37–47.
DOI: https://doi.org/10.1016/j.joi.2010.08.001

Wang, j. (2013). Citation time window choice for research impact
evaluación. cienciometria, 94(3), 851–872. DOI: https://doi.org
/10.1007/s11192-012-0775-9

Wang, P., & waltman, l. (2016). Large-scale analysis of the accuracy
of the journal classification systems of Web of Science and Scopus.
ournal of Informetrics, 10(2), 347–364. DOI: https://doi.org
/10.1016/j.joi.2016.02.003

Wilsdon, J., allen, l., Belfiore, MI., Campbell, PAG., Curry, S., Colina, S., …
Johnson, B. (2015). The metric tide: Report of the independent
review of the role of metrics in research assessment and manage-
mento. Bristol, Reino Unido: Higher Education Funding Council for England
(HEFCE). DOI: https://doi.org/10.4135/9781473978782

Wouters, PAG., Thelwall, METRO., Kousha, K., waltman, l., de Rijcke, S.,
Rushforth, A., & Franssen, t. (2015). The metric tide: Literature re-
vista (supplementary report I to the independent review of the
role of metrics in research assessment and management).
Londres, Reino Unido: Higher Education Funding Council for England
(HEFCE).

Zitt, METRO., Ramanana-Rahary, S., & Bassecoulard, mi. (2005). Relativity of
citation performance and excellence measures: From cross-field to
cross-scale effects of field-normalisation. cienciometria, 63(2),
373–401. DOI: https://doi.org/10.1007/s11192-005-0218-y

Zitt, METRO., & Pequeño, h. (2008). Modifying the journal impact factor by
fractional citation weighting: The audience factor. Diario de
the American Society for Information Science and Technology,
59(11), 1856–1860. DOI: https://doi.org/10.1002/asi.20880

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d