ARTÍCULO DE INVESTIGACIÓN
Unveiling the distinctive traits of a nation’s
research performance: The case of
Italy and Norway
Giovanni Abramo1
, Dag W. Aksnes2
, and Ciriaco Andrea D’Angelo1,3
1Laboratory for Studies in Research Evaluation, Institute for System Analysis and Computer Science (IASI-CNR).
National Research Council, Roma, Italia
2Nordic Institute for Studies in Innovation, Research and Education, Oslo, Norway
3University of Rome “Tor Vergata,” Dept of Engineering and Management, Roma, Italia
Palabras clave: bibliometría, Fractional Scientific Strength, FSS, national R&D systems, professors,
research evaluation, universidades
ABSTRACTO
en este estudio, we analyze the research performance of Italian and Norwegian professors using
constituent components of the Fractional Scientific Strength (FSS ) indicator. The main focus is
on differences across fields in publication output and citation impact. The overall performance
(FSS ) of the two countries, which differ considerably in research size and profile, is remarkedly
similar. Sin embargo, an in-depth analysis shows that there are large underlying performance
diferencias. An average Italian professor publishes more papers than a Norwegian, mientras que la
citation impact of the research output is higher for the Norwegians. Además, at field level,
the pattern varies along both dimensions, and we analyze in which fields each country has its
relative strengths. En general, this study contributes to further insights into how the research
performance of different countries may be analyzed and compared to inform research policy.
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
3
3
7
3
2
2
0
5
7
8
1
3
q
s
s
_
a
_
0
0
1
9
8
pag
d
/
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
1.
INTRODUCCIÓN
One of the principal objectives that governments and institutions pursue through research
assessment is improving research effectiveness and efficiency. Effectiveness can be deployed
along various dimensions, depending on the strategic goals of the country. These include
among others, sustainability goals, research integrity, responsible research and innovation,
open science, gender equality in research, geographical balance, international research col-
laboration, public–private collaboration, and rapid and widespread diffusion of new knowl-
edge to potential users.
Efficiency essentially deals with the relation between research output and inputs to produce
él. All inputs being equal, individual and organizational performance can be increased by pro-
ducing more research products (alone or in collaboration), or of better quality (higher impact),
o ambos. Providing the government, research institutions, and scientists with information along
each single dimension of their overall research performance might serve better the aim of stim-
ulating improvement. In this way, the single actors are informed about the dimensions with
higher margins of efficiency gains.
Whether associated or not with financial rewards or other competitive mechanisms, el
simple communication to individuals of their performance scores and ranks, if properly
un acceso abierto
diario
Citación: Abramo, GRAMO., Aksnes, D. w., &
D’Angelo, C. A. (2022). Unveiling the
distinctive traits of a nation’s research
actuación: The case of Italy and
Norway. Estudios de ciencias cuantitativas,
3(3), 732–754. https://doi.org/10.1162
/qss_a_00198
DOI:
https://doi.org/10.1162/qss_a_00198
Revisión por pares:
https://publons.com/publon/10.1162
/qss_a_00198
Recibió: 24 Marzo 2022
Aceptado: 9 Puede 2022
Autor correspondiente:
Giovanni Abramo
giovanni.abramo@uniroma2.it
Editor de manejo:
Juego Waltman
Derechos de autor: © 2022 Giovanni Abramo,
Dag W. Aksnes, and Ciriaco Andrea
D’Angelo. Published under a Creative
Commons Attribution 4.0 Internacional
(CC POR 4.0) licencia.
La prensa del MIT
Unveiling the distinctive traits of a nation’s research performance
channeled, can be instrumental in continuous improvement. At the large-scale level of
national scientific systems, governments and policymakers might also benefit from such
information to complement that of their comparative world performance rank. It can inform
decisions on the dimensions along which to focus and orient policy measures.
This study deals with research efficiency. The quintessential indicator of efficiency in any
production systems is productivity (es decir., the rate of output per unit of input). En otras palabras, él
measures how efficiently production inputs are being used. Most bibliometricians define pro-
ductivity as the number of publications in the period of observation. Because publications
have different values (impacto), and the resources employed for research are not homogeneous
across individuals and organizations, we adopt the definition of productivity extracted from
the economic theory of production: the value of output per euro spent in research (es decir., costos
of labor and all other production factors, referred to as capital in the following).
Following on from our previous studies on the comparison of research productivity of
Italian and Norwegian academics (Abramo, Aksnes, & D’Angelo, 2020, 2021), in this work,
we present a methodology to measure the single components of research productivity, y
apply it to further scan and contrast the two countries’ academic systems.
In terms of their research production systems, Italy and Norway are rather different. Primero,
the total scientific output of Italy measured by number of journal articles is approximately five
times as large as Norway’s. En 2018 (latest available data at time of first of our previous
estudios), Italy ranked as the eighth largest science producing country in the world and Norway
was in 29th position (Norges forskningsråd, 2019). Sin embargo, compared with the population
tamaño, per capita production is 150% higher for Norway than for Italy. Además, the overall
citation impact of research output per capita is also higher for Norway. De este modo, Norway may be
considered a more research-intensive country than Italy. The underlying reasons for these dif-
ferences are interesting to analyze further, and we aim to contribute to additional knowledge
on the issue.
Abramo et al. (2020) for the first time ever applied the Fractional Scientific Strength, or FSS,
indicator of research performance of professors and universities to a country other than Italy
(es decir., Norway). FSS (Abramo & D’Angelo, 2014) differs from the most popular indicators, semejante
as the h-index (Hirsch, 2005), the MNCS (waltman, van Eck et al., 2011), and their variants
(alonso, Cabrerizo et al., 2009; waltman, 2016; Wildgaard, Schneider, & Larsen, 2014),
essentially because it accounts for the resources used to produce research output. Más-
más, the FSS adopts the publications’ fractional counting method, and values them through
citation indicators.
We refer the reader to Abramo et al. (2020) for details about the academic systems in the
two countries and the difficulties of achieving comparable measurements of performance, y
the ways to overcome them. The reader will also find the procedure for operationalizing the
measurements and all the limits and assumptions involved. Where appropriate, we extrapolate
from that study (and report in this) the performance scores and ranks in the two countries,
across disciplines.
En resumen, the previous study showed the following:
(cid:129) hardly any differences in the average research productivity of Italian and Norwegian
professors;
(cid:129) a higher concentration of Norwegian professors in the top and the bottom tails of the
productivity distribution; y
Estudios de ciencias cuantitativas
733
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
3
3
7
3
2
2
0
5
7
8
1
3
q
s
s
_
a
_
0
0
1
9
8
pag
d
/
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Unveiling the distinctive traits of a nation’s research performance
(cid:129) higher productivity of Norway in Mathematics and Earth and space sciences, and of Italy
in Biomedical research and Engineering.
en este estudio, we complement the above results with more in-depth analysis of performance
along the single dimensions of overall research productivity of Italian and Norwegian profes-
sors. En particular, we contrast the academics of the two countries in terms of yearly output,
fractional output, average citations per paper, and average impact factor (SI) por papel.
The elaborations are aimed at answering the following research questions:
Al hacerlo, do they engage in more collaborative work?
1. Do the academics of one country tend to publish more than those of the other?
2.
3. Do they pursue higher quality research products in terms of citation impact?
4. Do they publish their results in more prestigious journals?
Such questions are important in the context of evaluative use of bibliometrics. Por ejemplo,
in an upcoming evaluation of Norwegian STEM research1, the panels are asked to assess the
production of scholarly publications, the research quality, and national and international
cooperation—in addition to several other dimensions, such as research career and mobility,
leadership, and research infrastructure. The Italian national research assessment exercise
series, VQR2, of which the results of the latest edition have just been released, emphasizes
the importance of the quality of scientific production over volume, international research col-
laboration over national, etc.. De este modo, in-depth analyses of bibliometric patterns may be useful to
assess the aspects related to research output, where a unit of analysis may perform differently
across the dimensions. Por ejemplo, high publication volume does not necessarily imply high
citation impact and vice versa. Además, different indicators reflect different dimensions of
actuación, which are required to provide reliable assessments, a point which has long been
emphasized by bibliometric professionals (van Raan, 1993). Por lo tanto, we wish to underline
that none of the indicators that we present and measure in this paper can be considered alone
as performance indicators. Each one represents more or less a dimension of performance, pero
not the overall performance. They convey complementary information, useful to orient scien-
tists in focusing their efforts for continuous improvement, and managers and for policymakers
in formulating their interventions to the same aim.
Generally, a large number of factors may explain patterns identified through bibliometric
analiza. Some can be investigated bibliometrically, but often other approaches are needed.
Por ejemplo, some might be related to different policy and behavioral mechanisms. Semejante
explanatory attempts are not straightforward, as the systems are influenced by a large number
of factors, and at best indirect evidence can be provided. Por ejemplo, higher publication vol-
ume or quality may be due to more financial resources, more time for research, better man-
agement, changes in the staff composition, recruitment of talented individuals, responses to
funding incentives, Etcétera.
en este documento, we therefore mainly focus on the empirical aspects, and second-order inter-
pretations are addressed to a limited extent. In presenting our study, we devote special atten-
tion to the definition of each single indicator and the description of the operationalization of
1 Ver https://www.forskningsradet.no/en/statistics-evaluations/natural-sciences-2022-2023/ (accessed on May
9, 2022).
2 https://www.anvur.it/wp-content/uploads/2020/09/ Bando-VQR-2015-19_25-settembre_2020_signed.pdf
(accessed May 9, 2022).
Estudios de ciencias cuantitativas
734
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
3
3
7
3
2
2
0
5
7
8
1
3
q
s
s
_
a
_
0
0
1
9
8
pag
d
/
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Unveiling the distinctive traits of a nation’s research performance
their measurement to make hopeful future replications of the exercise in other countries more
straightforward.
The rest of the paper is structured as follows. In the next section we present the construction
of the data set. En la sección 3, we will recall the FSS definition and present those of the comple-
mentary indicators we are going to measure. En la sección 4, we will present the results of the
evaluación. Sección 5 will conclude the work with our considerations.
2. DATA
To follow on from the abovementioned previous studies of ours and their findings (Abramo
et al. 2020, 2021), we observe the research activity of Italian and Norwegian professors in
the same period, 2011–2015.
Data on Italian professors have been retrieved from the database on university personnel,
which is operated by the Italian Ministry of Universities and Research (MIUR). This database
contains information on the name of each individual, their gender, affiliation, field classifica-
ción, and academic rank at the end of each year3.
Data on the Norwegian professors have been retrieved from a similar database, the Norwe-
gian Research Personnel Register, operated by Nordic Institute for Studies in Innovation,
Research and Education (NIFU) (the database underlying the official Norwegian R&D
Estadísticas).
For reasons of significance, the analysis is limited to those professors who held formal fac-
ulty positions for at least three years over the 2011–2015 period. Además, the data set is
limited to individuals with at least one publication during the time period (nonpublishing per-
sonnel are not registered in Norwegian databases).
The data on the publication output of the Italian professors are retrieved from the Italian
Observatory of Public Research (ORP), a bibliometric database where data are available at
the level of authors. The database is developed and maintained by Abramo and D’Angelo,
and derived under license from the Clarivate Analytics Web of Science ( WoS) Core Collection.
For the Norwegian professors, WoS publication data have been retrieved from a biblio-
graphic database called Cristin (Current Research Information System in Norway). This is a
national information infrastructure, providing data for all institutions in the higher education
sector, research institutes and hospitals.
To perform a comparative assessment without distortions (see below), each individual is
classified in one and only one WoS subject category (CAROLINA DEL SUR)4.
The analysis covers professors in all fields, with the exception of arts and humanities and a
few SCs in the social sciences, where WoS has largest limitations in terms of coverage (Aksnes
& Sivertsen, 2019; Hicks, 1999; Larivière, Archambault et al., 2006). There are large variations
in the size of the different SCs. In some cases there are few observations, which might cause
random fluctuations in performance. Por lo tanto, we have further removed the SCs that did not
meet a minimum threshold of at least 10 professors in total, of both nationalities.
3 https://cercauniversita.cineca.it/php5/docenti/cerca.php (accessed May 9, 2022).
4 We assigned to each publication the SC or SCs of the hosting journal. We then classified each professor in
the most recurrent SC in their publication portfolio. We refer the reader to Abramo et al. (2020) for more
details of the procedure followed to classify professors.
Estudios de ciencias cuantitativas
735
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
3
3
7
3
2
2
0
5
7
8
1
3
q
s
s
_
a
_
0
0
1
9
8
pag
d
.
/
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Unveiling the distinctive traits of a nation’s research performance
Mesa 1. Data set of analysis
Italia
Norway
Discipline
Matemáticas
Physics
Chemistry
Earth and space
sciences
Biología
Biomedical
investigación
Clinical medicine
Psicología
Ingeniería
Political and social
sciences
Ciencias económicas
Total
No. de
SC
6
Total
professors
2,122
Assistant
(%)
22.8
Associate
(%)
40.4
16
7
11
28
14
36
6
34
11
2,918
1,896
1,873
5,635
3,707
7,571
475
5,522
442
8
177
1,848
34,009
23.8
28.1
29.6
34.9
37.8
35.8
31.2
26.5
20.6
19.0
30.6
43.3
43.8
41.6
37.6
36.3
36.0
39.8
40.3
41.6
40.4
39.0
Lleno
(%)
36.8
32.9
28.2
28.8
27.5
25.9
28.2
29.1
33.2
37.8
40.6
30.4
Total
professors
183
Assistant
(%)
2.2
Associate
(%)
29.0
256
122
413
736
245
958
148
426
438
402
4327
9.8
14.8
14.8
20.7
19.2
10.5
2.7
5.2
6.2
3.0
10.9
21.5
29.5
29.1
28.0
30.2
31.3
42.6
27.2
33.1
33.1
30.1
Lleno
(%)
68.9
68.8
55.7
56.2
51.4
50.6
58.1
54.7
67.6
60.7
63.9
59.0
The final data set consists of 34,009 Italian and 4,327 Norwegian professors, falling in 177
SC. Their distribution per academic rank and discipline5 is shown in Table 1.
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
3
3
7
3
2
2
0
5
7
8
1
3
q
s
s
_
a
_
0
0
1
9
8
pag
d
.
/
3. THE FSS INDICATOR AND ITS COMPONENTS
As already said, in the economic theory of production, productivity is defined as the rate of
output per unit of input. Because publications (producción) have different values (impacto), y el
resources employed for research are not homogeneous across individuals and organizations,
in research systems, a more appropriate definition of productivity is the value of output per
euro spent in research. The FSS indicator is a proxy measure of research productivity. A
thorough description of the FSS indicator, and the theory underlying it, can be found in
Abramo and D’Angelo (2014).
To measure the yearly average research productivity of Italian and Norwegian academics,
Abramo et al. (2020) used the following formula6:
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
FSS ¼
(cid:1)
1
þ k
(cid:3) ∙
1
t
XN
i¼1
ci
(cid:1)C
fi
wr
2
(1)
5 The SCs are classified and grouped into disciplines according to a system previously published on the web-
page of the ISI Journal Citation Reports. This page is no longer available at the current Clarivate site. debería
be noted that all SCs are assigned to only one discipline.
6 The underlying assumption is that labor and capital contribute equally to production.
Estudios de ciencias cuantitativas
736
Unveiling the distinctive traits of a nation’s research performance
dónde
wr = average yearly salary of the professor7
k = average yearly capital available for research to the professor8
t = number of years of work by the professor in period under observation
N = number of publications by the professor in period under observation
ci = citations received by publication i, until 31 Octubre 2018
(cid:1)c = average of distribution of citations received for all WoS-cited publications in same year
and SC of publication i 9
fi = fractional contribution of professor to publication i.
In the fields where the common practice is to place authors in simple alphabetical order,
the fractional contribution equals the inverse number of authors, while in other cases different
weights are applied (see Waltman, 2012). More specifically, for Biology, Biomedical research,
and Clinical medicine, the usual practice is to indicate the contributions of each author by
their position in the bylines. De este modo, for these disciplines, we have applied a system where dif-
ferent weights are credited to each coauthor depending on their position in the list of authors
and the type of coauthorship (intramural or extramural)10.
Looking at the formula, it can be seen that productivity is a function of output, or more
precisely of individual contribution to output, its quality (impacto), and the resources used
for production.
In this work we want to assess how Italian professors compare to Norwegian professors
along each dimension of productivity, a saber, in terms of output, fractional output, promedio
citations per paper, y, although not a dimension of productivity, average IF per paper. El
average IF per paper is informative about the degree of prestige of the journals chosen by
researchers to have their manuscripts published. The breakdown of the productivity indicator
along the above dimensions allows us to unveil the possible different traits of professors in
conducting research activities.
The relevant indicators are the following:
(cid:129) Output (oh), average yearly publications authored by the professor per euro spent in
investigación:
O ¼
(cid:1)
1
þ k
(cid:3) norte
t
wr
2
(2)
7 We halved labor costs, because we assumed that 50% of professors’ time is allocated to activities other than
investigación. En la sección 4.3, we will propose a specific analysis to test how sensitive the results are to such an
assumption.
8 Sources of input data and assumptions adopted in the measurement can be found in Abramo et al. (2020).
9 Abramo, Cicero, and D’Angelo (2012) demonstrated that the average of the distribution of citations received
for all cited publications of the same year and subject category is the most effective scaling factor.
10 This means that if the publication is the result of a collaboration that is exclusively intramural (a single affiliation
in the address list), 40% is credited to both the first and last authors, mientras que el resto 20% is shared among all
other authors. One the other hand, if a publication involves extramural collaboration, 30% is credited to both
the first and last authors; 15% to both the second and last but one authors; and the remaining 10% is divided
among all other authors. This weighting system has been designed according to advice from senior Italian pro-
fessors in the life sciences, and widely accepted by the customers of our research assessment services. El
crediting values of the system could be changed according to different practices in other national contexts.
Estudios de ciencias cuantitativas
737
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
3
3
7
3
2
2
0
5
7
8
1
3
q
s
s
_
a
_
0
0
1
9
8
pag
d
/
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Unveiling the distinctive traits of a nation’s research performance
dónde
w = average yearly salary of the professor
k = average yearly capital available for research to the professor
N = number of publications by the professor in period under observation
t = number of years on staff of the professor during the period under observation
(cid:129) Fractional Output (FO), average yearly total contribution to publications authored by the
professor per euro spent in research:
FO ¼
(cid:1)
1
þ k
(cid:3)
1
t
XN
i¼1
fi
wr
2
dónde
fi = fractional contribution of professor to publication i.
(cid:129) Average Citation (AC ), average standardized citations per publication:
dónde
AC ¼
1
norte
XN
i¼1
ci
(cid:1)C
ci = citations received by publication i
(cid:1)c = average of distribution of citations received for all WoS-cited publications in same
year and SC of publication i.
(cid:129) Average IF (AIF ), average standardized IF per publication:
AIF ¼
1
norte
XN
i¼1
IFi
SI(cid:1)
(5)
dónde
IF = IF of journal hosting publication i
SI(cid:1)= average of distribution of IFs of journals hosting all WoS-cited publications in same
year and SC of publication i.
The performance scores of professors belonging to different SCs cannot be compared directly.
De hecho, scientists’ intensity of publication remarkably varies across fields, en general (Lillquist &
Verde, 2010; Sandström & Sandström, 2009; Sorzano, Vargas et al., 2014), and in both countries
En particular (D’Angelo & Abramo, 2015; Piro, Aksnes & Rørstad, 2013); citation behavior varies
across fields (Stringer, Sales-Pardo, & Amaral, 2010; Vieira & Gomes, 2010); and the intensity of
colaboración (es decir., the average number of coauthors per publication) also varies across fields
(Abramo, D’Angelo, & Murgia, 2013; Glanzel & Schubert, 2004; Yoshikane & Kageura, 2004).
To avoid distortions, entonces, the performance rankings of professors are constructed at the SC
nivel.
For comparisons at higher levels of aggregation (es decir., discipline and overall), we normalize per-
formance scores to the average score of all professors of the same SC. To exemplify, an FSS score of
1.10 means that the professor’s performance is 10% above average, in his or her own SC11.
11 Note that the scaling is not referred to world distributions. As we are comparing Italy vs. Norway, the “aver-
age” used to rescale original distributions is calculated by collapsing Italian and Norwegian performance
distributions only.
Estudios de ciencias cuantitativas
738
(3)
(4)
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
3
3
7
3
2
2
0
5
7
8
1
3
q
s
s
_
a
_
0
0
1
9
8
pag
d
.
/
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Unveiling the distinctive traits of a nation’s research performance
Cifra 1. 2011–2015 average normalized research output and fractional output per euro spent
(oh, FO), impacto (AC, AIF ), and productivity (FSS ), at overall country level.
In the following tables, figures and text, all performance scores are normalized, but we keep
the same denomination for the relevant indicators.
4. RESULTADOS
En el siguiente, we present the comparison of Italian and Norwegian professors along each of
the above indicators, per discipline and overall.
Cifra 1 shows the average normalized scores of each indicator, for the overall 34,009 Ital-
ian and 4,327 Norwegian professors. While the two populations show practically the same
productivity, FSS12, noticeable differences occur for the other indicators. De término medio, italiano
professors publish more (1.4% above average) than Norwegian professors (11.1% below aver-
edad). Accounting for the real contribution to each publication, the gap decreases, but it is still
allá: Italy’s FO equals 1.00, while Norway’s is 0.97.
Because the FSS is the same, it follows that Norway’s AC needs to be higher (1.03) than
Italy’s (1.00). Además, Norwegian academics on average publish in more prestigious jour-
nal (AIF = 1.07) than Italians (AIF = 0.99).
En breve, while the two countries present the same research productivity, Italians publish
más, with larger research teams, while Norwegians publish higher quality products in more
prestigious journals.
We wondered whether the same traits could be found among top-performing scientists as
Bueno. Limiting the analysis to top 10% professors by FSS at SC level, we observe similar differ-
ences for all considered indicators (Cifra 2). Norwegian top professors (FSS = 4.15) son
slightly more productive than Italians (FSS = 4.10). The first build up their supremacy by pro-
ducing higher impact publications. De hecho, the average O by Italian top professors is 6.8%
higher than Norwegian (2.57 vs. 2.39), and although it diminishes by FO to 5.2% (2.77 vs.
12 Note that for FSS, both countries show average figures below one, because in rescaling the FSS of individ-
uals, we exclude nil values. This implies that the average of the overall distribution is below one, en general
and for both countries. As for impact indicators (AC, AIF ), the effect of removing nil values does not produce
the same “visual” effect, while for output indicators (oh, FO), we consider only professors with at least one
publication authored in the period under observation; consequently there are no nil values.
Estudios de ciencias cuantitativas
739
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
3
3
7
3
2
2
0
5
7
8
1
3
q
s
s
_
a
_
0
0
1
9
8
pag
d
/
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Unveiling the distinctive traits of a nation’s research performance
Cifra 2. 2011–2015 average normalized research output and fractional output per euro spent
(oh, FO), impacto (AC, AIF ), and productivity (FSS ), at overall country level for top 10% productive
professors.
2.62), the roles swap in terms of impact, whereby Norwegian top professors register an AC
(AJI ) 11.4% (12.4%) higher than Italians.
En el siguiente, we present the results at field level (discipline and SC), to assess possible
differences across fields between the two populations. We will start with output indicators;
impact will follow.
4.1. Output and Fractional Output per Euro Spent, at Field Level
Italian professors’ O is higher than that of Norwegians in all 11 disciplines except Mathemat-
circuitos integrados, as shown in Table 2. The biggest gap occurs in Biomedical research, donde el 3,707
Italian professors’ O is 1.02, mientras que la 245 Norwegian professors’ is 0.77. Noticeable differ-
ences occur also in Psychology (1.05 vs. 0.84), Biología (1.02 vs. 0.82), Ingeniería (1.01 vs.
0.82), and Clinical medicine (1.02 vs. 0.85).
When accounting for the real contribution to each paper, Italian professors’ contribution
(FO) is greater in only five disciplines, especially in Biomedical research (1.01 vs. 0.88) y
in Biology (1.01 vs. 0.90). Norwegians noticeably prevail in Mathematics (1.12 vs. 0.99),
Chemistry (1.13 vs. 0.99), and Earth and space sciences (1.08 vs. 0.98).
Próximo, we analyze the research output at the level of SCs. As noted in Section 2, SCs differ
considerably in size, and some of them include a rather small number of individuals, partícipe-
ularly for Norway. Sin embargo, an analysis at this level may reveal interesting differences
across the nations.
Mesa 3 shows how the differences by O and FO observed at discipline level vary across the
SCs of each single discipline. The FSS scores are reported as a reference. We note that Nor-
wegian professors outperform Italians’ FSS in 40% of SCs, en 24% by O, and in 38% by FO. En
particular, by O Norwegians never outperform Italians in Psychology, and they do so in only
two of the 28 SCs of Biology and in two of the 14 SCs of Biomedical research. En el otro
mano, in Mathematics, Italians outperform Norwegians in two out of 6 SC. In terms of FO,
Norway recovers and, differently from other disciplines, in Chemistry it outperforms Italy in
five out of seven SCs.
Estudios de ciencias cuantitativas
740
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
3
3
7
3
2
2
0
5
7
8
1
3
q
s
s
_
a
_
0
0
1
9
8
pag
d
.
/
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Unveiling the distinctive traits of a nation’s research performance
Mesa 2.
nivel.
2011–2015 normalized average research output and fractional output per euro spent (oh, FO), and productivity (FSS ) at discipline
Italia
Norway
Discipline
Matemáticas
Physics
Chemistry
Earth and space sciences
Biología
Biomedical research
Clinical medicine
Psicología
Ingeniería
Political and social sciences
Ciencias económicas
En general
No. de
professors
2,122
2,918
1,896
1,873
5,635
3,707
7,571
475
5,522
442
1,848
34,009
oh
0.99
1.01
1.01
1.00
1.02
1.02
1.02
1.05
1.01
1.03
1.01
1.01
FO
0.99
1.00
0.99
0.98
1.01
1.01
1.01
1.02
1.01
0.99
1.00
1.00
FSS
0.94
1.00
1.00
0.94
1.01
1.01
0.99
0.98
1.00
0.91
0.98
0.99
No. de
professors
183
256
122
413
736
245
958
148
426
438
402
4327
oh
1.10
0.94
0.91
0.98
0.82
0.77
0.85
0.84
0.82
0.97
0.95
0.89
FO
1.12
1.06
1.13
1.08
0.90
0.88
0.93
0.94
0.94
1.01
1.00
0.97
FSS
1.25
0.97
0.98
1.27
0.90
0.89
1.03
1.03
0.88
0.89
0.96
0.99
Mesa 3. Number and proportion of fields (SC) per discipline where 2011–2015 normalized
average research output and fractional output per euro spent (oh, FO) and productivity (FSS ) de
Norway is higher than Italy.
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
3
3
7
3
2
2
0
5
7
8
1
3
q
s
s
_
a
_
0
0
1
9
8
pag
d
.
/
Discipline
Matemáticas
Physics
Chemistry
Earth and space sciences
Biología
Biomedical research
Clinical medicine
Psicología
Ingeniería
Political and social sciences
Ciencias económicas
En general
No. de
SC
6
16
7
11
28
14
36
6
34
11
8
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
No. of SCs where Norway outperforms Italy
FO
4 (67%)
oh
4 (67%)
FSS
3 (50%)
7 (44%)
2 (29%)
4 (36%)
2 (7%)
2 (14%)
7 (19%)
0 (0%)
9 (26%)
4 (36%)
2 (25%)
10 (63%)
5 (71%)
4 (36%)
7 (25%)
4 (29%)
8 (50%)
3 (43%)
8 (73%)
7 (25%)
5 (36%)
11 (31%)
17 (47%)
1 (17%)
2 (33%)
13 (38%)
11 (32%)
4 (36%)
4 (50%)
5 (45%)
2 (25%)
177
43 (24%)
67 (38%)
71 (40%)
Estudios de ciencias cuantitativas
741
Unveiling the distinctive traits of a nation’s research performance
Mesa 4 shows the 10 SCs where the gap by O between Norway and Italy is highest, y
viceversa. At the top of the list, in line with the above findings, we find an SC of Mathematics
(Matemáticas, interdisciplinary applications), where the six Norwegian professors show an O
(3.77) about six times as high as that of their 46 Italian counterparts (0.64). The other top SCs
fall in four disciplines, a saber, Ingeniería (Remote sensing; Construction & building technol-
ogia), Clinical medicine (Anestesiología; Substance abuse), Physics (Acoustics; Thermody-
namics; Physics, particles & campos), and Political and social sciences (Area studies; Político
ciencia). On the other side, el 10 SCs with the highest gap in favor of Italy fall in Engineering
(Siete), Clinical medicine (two), and Biomedical research (uno).
Mesa 5 presents the same analysis by FO. The results are similar to those by O. Six of the
arriba 10 SCs where Norway outperforms Italy remain the same, while only one changes among
those where Italy outperforms Norway.
Mesa 4.
Top 10 campos (SC) by gap of output (oh) in favor of Norway, and the same for Italy.
CAROLINA DEL SUR
Matemáticas, interdisciplinary applications
Italia
No. of professors
46
Remote sensing
Acoustics
Anestesiología
Construction & building technology
Substance abuse
Thermodynamics
Physics, particles & campos
Area studies
Political science
…
Ingeniería, multidisciplinary
Materials science, composites
Computer science, hardware & architecture
Computer science, interdisciplinary applications
Medicamento, investigación & experimental
Computer science, cybernetics
Andrology
Rehabilitation
Medical informatics
Information science & library science
Estudios de ciencias cuantitativas
102
25
57
67
9
97
556
4
90
11
59
10
44
66
6
9
47
6
13
oh
0.64
0.92
0.78
0.96
0.91
0.69
0.98
0.97
0.51
0.85
1.13
1.03
1.08
1.17
1.03
1.39
1.10
1.47
1.72
2.04
Norway
No. of professors
6
8
9
3
11
10
4
30
73
43
2
2
1
10
2
4
1
38
6
24
oh
3.77
2.04
1.61
1.79
1.52
1.28
1.52
1.50
1.03
1.32
0.27
0.16
0.19
0.27
0.12
0.42
0.13
0.42
0.28
0.44
Δ
−3.13
−1.12
−0,83
−0,83
−0.61
−0.59
−0.54
−0.52
−0.52
−0.48
0.87
0.87
0.89
0.89
0.91
0.97
0.97
1.05
1.43
1.61
742
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
3
3
7
3
2
2
0
5
7
8
1
3
q
s
s
_
a
_
0
0
1
9
8
pag
d
.
/
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Unveiling the distinctive traits of a nation’s research performance
Mesa 5.
Top 10 campos (SC) by gap of fractional output (FO) in favor of Norway, and the same for Italy.
CAROLINA DEL SUR
Matemáticas, interdisciplinary applications
Italia
No. of professors
46
Hematology
Remote sensing
Chemistry, inorganic & nuclear
Thermodynamics
Anestesiología
Substance abuse
Instrumentos & instrumentation
Reproductive biology
Acoustics
…
Medical informatics
Computer science, interdisciplinary applications
Computer science, hardware & architecture
Medicamento, investigación & experimental
Ingeniería, aerospace
Materials science, composites
Computer science, cybernetics
Ingeniería, multidisciplinary
Andrology
Information science & library science
340
102
242
97
57
9
184
66
25
6
44
10
66
75
59
6
11
9
13
FO
0.69
0.99
0.92
0.98
0.96
0.96
0.60
0.97
0.98
0.85
1.38
1.14
1.07
1.02
1.01
1.03
1.37
1.15
1.11
1.87
Norway
No. of professors
6
4
8
4
4
3
10
7
2
9
6
10
1
2
1
2
4
2
1
24
FO
3.41
2.19
2.02
1.92
1.86
1.79
1.36
1.71
1.58
1.42
0.62
0.37
0.27
0.20
0.17
0.15
0.45
0.16
0.05
0.53
Δ
−2.72
−1.21
−1.09
−0.93
−0.89
−0,83
−0.75
−0.74
−0.60
−0.58
0.76
0.77
0.81
0.82
0.84
0.88
0.92
0.99
1.06
1.35
4.2.
Impact Analysis at Field Level
We now turn to the average impact analysis per discipline. Cifra 3 presents the normalized
scores of AC. Norwegian academics outperform Italians in six disciplines, most noticeably in
Earth and space sciences (0.96 for Italy vs. 1.16 for Norway), Matemáticas (0.98 vs. 1.15), y
Clinical medicine (0.99 vs. 1.09). Italians outperform Norwegians in five, most noticeably in
Ingeniería (1.01 vs. 0.90), Psicología (1.01 vs. 0.95), and Physics (1.00 vs. 0.95).
Norwegian professors publish on average in more prestigious journals than Italians in eight
disciplines, the only exceptions being Physics, Chemistry, and Engineering (Cifra 4).
As shown in Figure 1, Norway produces higher impact publications on average, but from
Mesa 6 we see that this occurs in fewer than 50% of the SCs under observations (86 out of
177): En particular, en 10 del 11 SCs of Earth and space sciences, in four of the six SCs of
Matemáticas, and in seven of the 11 SCs of Political and social sciences. En cambio, Italians
Estudios de ciencias cuantitativas
743
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
3
3
7
3
2
2
0
5
7
8
1
3
q
s
s
_
a
_
0
0
1
9
8
pag
d
/
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Unveiling the distinctive traits of a nation’s research performance
Cifra 3. 2011–2015 normalized average impact of publications (AC ) per discipline. The dashed
line represents the expected value.
outperform Norwegians in six of the eight SCs of Economics, five of the seven SCs of Chem-
istry, y 10 del 14 SCs of Biomedical research.
Norwegian research products are published in higher IF journals in 112 SC (63%).
Mesa 7 reports the 10 SCs with the highest difference of AC for each country. In five SCs the
average AC of Norwegian professors is above 2, while for Italians it is below 1. The highest gap
is registered in History & philosophy of science, with an average AC for 62 Italian professors of
0.74, against 2.59 para 10 Norwegians. Of the top 10 SCs where Italy outperforms Norway, five
SCs fall in Engineering (Robótica; Ingeniería, aerospace; Transportation science & technol-
ogia; Materials science, composites; Computer science, cybernetics), and three in
Economics/Political and social sciences (Public administration; Urban studies; Area studies).
Mesa 8 proposes the same view by the average IF of hosting journals per paper. Along this
indicator, five of the top 10 SCs where Norway outperforms Italy fall in Clinical medicine
(Andrology; Urology & nephrology; Medicamento, general & internal; Anestesiología; Medicamento,
legal). Similarmente, five of the top 10 SCs for Italy fall in Engineering (Robótica; Medical
Cifra 4. 2011–2015 normalized average impact factor of journals (AIF ) per discipline. El
dashed line represents the expected value.
Estudios de ciencias cuantitativas
744
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
3
3
7
3
2
2
0
5
7
8
1
3
q
s
s
_
a
_
0
0
1
9
8
pag
d
.
/
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Unveiling the distinctive traits of a nation’s research performance
Mesa 6. Number of fields (SC) and proportion per discipline where 2011–2015 average citations
per publication (AC ), average IF per publication (AIF ), and productivity (FSS ) of Norway is higher
than Italy.
Discipline
Matemáticas
Physics
Chemistry
Earth and space sciences
Biología
Biomedical research
Clinical medicine
Psicología
Ingeniería
Political and social sciences
Ciencias económicas
En general
No. of SCs
6
16
7
11
28
14
36
6
34
11
8
No. of SCs where Norway outperforms Italy
AIF
5 (83%)
FSS
3 (50%)
AC
4 (67%)
6 (38%)
2 (28%)
6 (38%)
3 (43%)
10 (91%)
10 (91%)
16 (57%)
21 (75%)
4 (29%)
11 (79%)
8 (50%)
3 (43%)
8 (73%)
7 (25%)
5 (36%)
20 (56%)
28 (78%)
17 (47%)
3 (50%)
4 (67%)
2 (33%)
12 (35%)
15 (44%)
11 (32%)
7 (64%)
2 (25%)
5 (45%)
4 (50%)
5 (45%)
2 (25%)
177
86 (49%)
112 (63%)
71 (40%)
informatics; Computer science, interdisciplinary applications; Nanoscience & nanotechnol-
ogia; Transportation science & tecnología).
4.3. Sensitivity Analysis
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
3
3
7
3
2
2
0
5
7
8
1
3
q
s
s
_
a
_
0
0
1
9
8
pag
d
/
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
The results of any evaluation exercise depend on the adopted methodology and, En particular,
on the assumptions underlying the measurement indicators used. As reported in Section 3, en
this study we used both output to input indicators (oh, FO, and FSS ) and average impact indi-
cators (AC and AIF ) that are independent of input.
The typical limits embedded in bibliometric analyses apply to our study too. En particular,
the new knowledge produced is not only that encoded in publications. Además, biblio-
graphic repertories (such as WoS, used here) do not register all publications. The measurement
of the impact of publications, before the end of their life cycle, using citation-based indicators
is a prediction of their overall impact, and therefore not definitive. Citations can also be neg-
ative or inappropriate, and in any case they certify only scholarly impact, forgoing other types
of impact. Finalmente, the results could be sensitive to the field classification schemes used for
publications and, especially in the proposed work, for professors.
To these limits, for O, FO, and FSS we must add those associated to the assumptions on
input data. When assessing the performance of individual scientists, if there are differences
in the production factors available to each, then one should normalize for them. Unfortu-
nately, relevant data are not available at the individual level, either in Italy or in Norway.
So, we are forced to assume that the same resources are available to all professors within
the same field. The second assumption is that the hours devoted to research are the same
for all professors.
Estudios de ciencias cuantitativas
745
Unveiling the distinctive traits of a nation’s research performance
Mesa 7.
Top 10 campos (SC) by gap of average citation per publication (AC ) in favor of Norway, and the same for Italy.
CAROLINA DEL SUR
Historia & philosophy of science
Ingeniería, multidisciplinary
Urology & nephrology
Medicamento, legal
Anestesiología
Estadísticas & probabilidad
Remote sensing
Matemáticas, interdisciplinary applications
Medicamento, general & internal
Physics, atómico, molecular & chemical
…
Robótica
Ingeniería, aerospace
Public administration
Urban studies
Transportation science & tecnología
Physics, multidisciplinary
Materials science, composites
Computer science, cybernetics
Developmental biology
Area studies
Italia
No. of professors
62
11
234
92
57
390
102
46
18
98
56
75
21
26
41
157
59
6
22
4
AC
0.74
0.79
0.98
0.94
0.94
0.94
0.95
0.92
0.97
0.94
1.01
1.01
1.26
1.14
1.06
1.05
1.03
1.35
1.04
1.98
Norway
No. of professors
10
2
3
5
3
29
8
6
1
12
1
1
12
6
3
11
2
4
1
73
AC
2.59
2.13
2.21
2.10
2.06
1.83
1.66
1.62
1.62
1.52
0.43
0.39
0.55
0.37
0.24
0.23
0.18
0.48
0.13
0.95
Δ
−1.84
−1.33
−1.23
−1.16
−1.12
−0.89
−0.71
−0.70
−0.65
−0.58
0.58
0.61
0.71
0.77
0.81
0.82
0.85
0.86
0.91
1.04
Given the characteristics of the Italian and Norwegian academic systems, this assumption
appears to be acceptable. De hecho, in keeping with the Humboldtian model, there are no
“teaching-only” universities included in the analysis, as all professors are required to carry
out both research and teaching, with a breakdown of the overall workload that, on average,
seems to justify the assumption that time available for research is 50% of the total work time.
Sin embargo, lacking a normative reference or a qualified on-field survey attesting the validity of
this assumption, we here propose a specific analysis to test how sensitive evaluation outcomes
are to such an assumption. En particular, for O, FO and FSS we repeat the analysis at an overall
level in four distinct scenarios:
(cid:129) Scenario 0: the one based on the assumption adopted in this study, and whose results are
como se muestra en la figura 1;
(cid:129) Scenario 1: where the time available for research is set at 40% of the total work hours for
all professors in the data set;
Estudios de ciencias cuantitativas
746
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
3
3
7
3
2
2
0
5
7
8
1
3
q
s
s
_
a
_
0
0
1
9
8
pag
d
.
/
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Unveiling the distinctive traits of a nation’s research performance
Mesa 8.
Top 10 campos (SC) by gap of average journal impact (AIF ) in favor of Norway, and the same for Italy.
CAROLINA DEL SUR
Historia & philosophy of science
Andrology
Urology & nephrology
Agriculture, multidisciplinary
Negocio, finance
Medicamento, general & internal
Anestesiología
Metallurgy & metallurgical engineering
Medicamento, legal
Horticulture
…
Robótica
Medical informatics
Comunicación
Biophysics
Computer science, interdisciplinary applications
Nanoscience & nanotechnology
Medicamento, investigación & experimental
Transportation science & tecnología
Physics, multidisciplinary
Developmental biology
Italia
No. of professors
62
9
234
67
101
18
57
47
92
162
56
6
25
17
44
32
66
41
157
22
AIF
0.82
0.92
0.99
0.96
0.90
0.97
0.97
0.92
0.97
0.99
1.01
1.15
1.18
1.02
1.06
1.01
1.01
1.03
1.03
1.02
Norway
No. of professors
10
1
3
4
20
1
3
9
5
2
1
6
34
1
10
1
2
3
11
1
AIF
2.14
1.73
1.72
1.61
1.53
1.54
1.54
1.44
1.49
1.50
0.72
0.85
0.87
0.69
0.72
0.65
0.62
0.60
0.53
0.46
Δ
−1.32
−0.82
−0.73
−0.65
−0.63
−0.58
−0.57
−0.52
−0.52
−0.50
0.29
0.29
0.32
0.33
0.34
0.36
0.40
0.43
0.50
0.56
(cid:129) Scenario 2: where the time available for research is set to 60% of the total work hours for
all professors in the data set;
(cid:129) Scenario 3: where the time available for research is set to 40% of the total work hours for
full professors, 50% for associate and 60% for assistant (assuming that the responsibili-
ties for management, coordination and institutional representation activities (y ahí-
fore the time dedicated to them) grows with academic rank.
It should be noted that differentiation by country would have the sole effect of increasing
the productivity of professors in one country with respect to the other.
En figura 5 we show the results of the sensitivity analysis. The variability of the average
value of the indicators among the four scenarios is very limited for Italy, while it is more evi-
dent for Norway. This result certainly depends on the different size of the two countries: Ni-
wegian professors represent 11% of the total data set, Italians 89%. En particular, for Norway
Estudios de ciencias cuantitativas
747
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
3
3
7
3
2
2
0
5
7
8
1
3
q
s
s
_
a
_
0
0
1
9
8
pag
d
.
/
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Unveiling the distinctive traits of a nation’s research performance
Cifra 5. 2011–2015 average normalized research output, fractional output, and productivity per
euro spent (oh, FO, FSS ) at overall country level, varying the share of time devoted to research.
Scenario 0 is certainly the most favorable. With respect to the single indicators, for O and FO
the differences between the two countries are always quite noticeable. A diferencia de, for FSS,
Scenario 0 shows practically no difference in average performance between the two countries,
while in the other three scenarios the difference is more noticeable.
We repeat the same analysis at the discipline level. For each country and discipline, nosotros
provide the average value and error bars of normalized research output (Cifra 6), fractional
producción (Cifra 7), and productivity (Cifra 8) for the four scenarios. The graphs confirm that the
variability among the four scenarios is practically undetectable for Italy, while more evident for
Norway. Sin embargo, in no case does a ranking inversion between the two countries emerge. Nosotros
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
3
3
7
3
2
2
0
5
7
8
1
3
q
s
s
_
a
_
0
0
1
9
8
pag
d
.
/
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Cifra 6. 2011–2015 average normalized research output per euro spent (oh) by UDA, with error
bars obtained by varying the share of time devoted to research*. *Scenario 0 (50% for all profes-
sors); Scenario 1 (40% for all professors); Scenario 2 (60% for all professors); Scenario 3 (40% para
full, 50% for associate, y 60% for assistant professors).
Estudios de ciencias cuantitativas
748
Unveiling the distinctive traits of a nation’s research performance
Cifra 7. 2011–2015 average normalized research fractional output per euro spent (FO) by UDA,
with error bars obtained by varying the share of time devoted to research*. *Scenario 0 (50% for all
professors); Scenario 1 (40% for all professors); Scenario 2 (60% for all professors); Scenario 3 (40%
for full, 50% for associate, y 60% for assistant professors).
can conclude that at the discipline level, the sensitivity of results of an evaluation exercise to
assumptions about the time available for research is very limited.
En cambio, at the overall level, the analysis reveals a higher sensitivity to input data and sug-
gests attention when setting assumptions and caution in the interpretation of results.
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
3
3
7
3
2
2
0
5
7
8
1
3
q
s
s
_
a
_
0
0
1
9
8
pag
d
.
/
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Cifra 8. 2011–2015 average normalized research productivity (FSS ), by UDA, with error bars
obtained varying the share of time devoted to research*. *Scenario 0 (50% for all professors); Sce-
nario 1 (40% for all professors); Scenario 2 (60% for all professors); Scenario 3 (40% for full, 50% para
associate, y 60% for assistant professors).
Estudios de ciencias cuantitativas
749
Unveiling the distinctive traits of a nation’s research performance
5. DISCUSSION AND CONCLUSIONS
Cross-national comparison of research performance is one of the main areas of application of
bibliometría. Por ejemplo, such analyses have a central position in science and technology
indicator reports for monitoring scientific development and performance (ver, Por ejemplo,
Junta Nacional de Ciencias, 2020; Norges forskningsråd, 2019; OECD, 2019). Through the great
attention that these reports receive, the bibliometric indicators play an important role in the
public perception of the scientific performance of countries. Our study provides new insight
into more advanced methods for comparing national research performance than is provided
through such reports.
Despite Italy and Norway differing considerably along such dimensions as research size
and profile, and the organization of their research systems, there are hardly any differences
at all in the average research productivity (FSS ) of Italian and Norwegian professors (Abramo
et al., 2020).
En este trabajo, we delve into the various dimensions of research performance. En general, el
study shows that an average Italian professor publishes more papers than a Norwegian. Este
difference is quite large when measuring publications on a whole count basis, but is smaller
using fractionalized measures (in which credit for a publication is allocated according to the
contribution of the participating coauthors). This means that Italian publications must have
more coauthors, which suggests that Italians tend to work in larger teams. This might be related
to the fact that the Italian research system is much larger than Norway’s. Possibly, italiano
researchers may benefit from larger universities, which can favor larger institutional collabo-
rations. The different collaboration behavior of Italian and Norwegian professors might reveal
an interesting topic for future investigation. Further future research may repeat this study,
restricting the observation to top performers only, to assess whether their distinctive traits along
each productivity component differ from the average population.
The discussion of the appropriateness of whole versus fractionalized measures of publica-
tion output has a long history in bibliometrics, dating back to the dispute on the decline of
British science during the 1980s (Braun, Glänzel, & Schubert, 1989; Irvine, Martin et al.,
1985). Moed (2005) argues that the two measurement principles should be seen as comple-
mentary, where whole counts measure participation and fractionalized counts measure the
number of creditable papers. Our analysis shows that the different measurement principles
yield quite different results, which has also been documented in previous studies (Aksnes,
Sivertsen et al., 2016; Gauffriau, Olesen Larsen et al., 2008). Actualmente, fractionalized mea-
surement seems to have gained increased popularity (waltman & van Eck, 2015), y esto
is also the principle underlying the FSS indicator.
While the number of publications produced per individual is higher on average for Italian
than for Norwegian professors, the pattern is reversed when looking at the citation impact and
journal profile. We observe national patterns where Italian academics produce more papers
but with lower impact and in less prestigious journals. De este modo, there is a quantity versus quality
tension in the national publication patterns, where the first dimension apparently is more
emphasized by Italians and the latter by Norwegians. However in the combined FSS indicators
these differences tend to level out.
We are not able to provide any final answer on the possible reasons underlying these dif-
ferences. The quality of the research is one obvious dimension, but there are also mechanisms
in the research systems that might favor certain publication behaviors more than others. Nosotros
will discuss this further below.
Estudios de ciencias cuantitativas
750
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
3
3
7
3
2
2
0
5
7
8
1
3
q
s
s
_
a
_
0
0
1
9
8
pag
d
/
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Unveiling the distinctive traits of a nation’s research performance
In Norway, there is a performance-based funding system where a bibliometric indicator is
applied for the allocation of funding across institutions (Sivertsen, 2018). Although this model
is designed to work at an overall national level, previous research has shown that it is some-
times applied at lower levels and may have an incentive effect at the level of individual
investigadores (Aagaard, 2015). Two important elements of the model should be emphasized.
Primero, extra points are given for publications in the most prestigious publication channels;
y segundo, all journal and book publications in accredited publication channels are
incluido, not only publications indexed in WoS. This means that there is an incentive to pub-
lish in prestigious publication outlets and no incentive to publish in WoS journals specifically.
Además, citations are not included as indicators in the Norwegian model.
In Italy, a proportion of public funds started being allocated to universities on the basis of
the outcome of the second national research assessment exercise, VQR 2006–201013. El
evaluation was based on a restricted number of research products submitted by universities,
and the bibliometric indicator applied to assess their relative quality was a combination of
citations and IF. The underlying incentive seems to have professors focus on the production
of a few very high-quality outputs, to be published in prestigious journals. In the same period,
the “national scientific habilitation” (ASN) for university appointments was introduced in
Italy14. The ASN required passing the threshold values in all or part of (depending on the field)
three bibliometric indicators, namely number of articles, number of citations, and contempo-
rary h-index15.
Based on these national differences, where citations are used as a component indicator in
Italy only, one might have expected Italian professors to be more highly cited. Sin embargo, nuestro
overall results show the contrary. The empirical paradox may suggest that the incentive effect
at the level of individual researchers may be limited, or that it takes a longer time for incentive
systems to show effects. Still, it should be emphasized that the Norwegian division of publi-
cation channels by quality levels means that the Norwegian model is not a pure productivity
measure. Además, the funding models are only a part of the picture, and citations indicators
are sometimes used in other contexts as well, such as in evaluations of research and in the
assessment of research proposals and candidates (Wilsdon, Allen et al., 2015). The national
differences in the number of publications are also relevant to discuss in the light of the two
funding models. While the Italian model is limited to WoS-indexed publications, the Norwe-
gian system covers all publications in channels accredited as scientific or scholarly. A previous
analysis showed that the WoS Core Collection covered 69% of the total Norwegian publica-
tion output, with large variations across fields (Aksnes & Sivertsen, 2019). This means that a
relatively large number of the publications of the professors analyzed would not be included in
the present analysis. This presumably holds for both Italy and Norway. Sin embargo, as there is no
specific incentive to publish in WoS journals in Norway, this issue might affect Norway more
than Italy.
Además, previous research has shown that there is a language bias in the coverage of
WoS, which affects various countries differently (van Leeuwen, Moed et al., 2001). A pesar de
journals in national languages as a general rule are not covered by WoS, at least not in the core
collection, there are many exceptions. While coverage of national journals will increase the
publication numbers of a country, it will have an opposite effect on the citation impact, como
13 https://www.anvur.it/en/homepage/ (accessed May 9, 2022).
14 https://abilitazione.miur.it/public/index.php?lang=eng (accessed May 9, 2022).
15 The contemporary index was introduced by Sidiropoulos, Katsaros, and Manolopoulos (2007).
Estudios de ciencias cuantitativas
751
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
3
3
7
3
2
2
0
5
7
8
1
3
q
s
s
_
a
_
0
0
1
9
8
pag
d
.
/
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Unveiling the distinctive traits of a nation’s research performance
articles in these journals generally tend to be little cited. There are hardly any national Nor-
wegian journals indexed in the core collection of WoS, while for Italy there are several, y
becoming more and more numerous, especially after the introduction of the VQR. Possibly,
this factor too might explain some of the observed country differences across the indicators
analyzed.
In interpreting the results of the performance analysis, all the usual limits, caveats, assump-
ciones, and qualifications of evaluative scientometrics apply. In addition to the bibliographic
data source and relative country coverage, these include general bibliometric issues, semejante
as that publications are not representative of all knowledge produced, and that citations have
limitations as performance measures. Además, the results are sensitive to the classification
schemes applied for the publication output and professors. Finalmente, due to limitations in the
availability of comparable input data, some adaptations have been introduced. The sensitivity
of results to varying input data suggests care when setting assumptions and caution in inter-
preting findings. We have shown in fact that varying the assumed 50% of professors’ time
devoted to research to 40% o 60% would tilt research productivity in the two countries from
being equal to favoring Italy. This is particularly true at overall level, while at discipline level
the sensitivity of results is very limited and the relative performance of the two countries seems
very robust.
From this, there arises a call for those governments and research institutions that intend to
benefit from ever more precise and reliable performance evaluations, useful for their decisions
and policy-making: They should be prepared to give scientometricians the underlying input
data necessary for the job. Subject to the availability of input data, the present study can be
replicated to include other countries.
Studies of the output dimension in other countries would be of particular interest. For a long
tiempo, various normalization procedures have been applied for calculating citation indicators
(Moed, 2005). Sin embargo, there are no cross-national reference standards for comparing pub-
lication output, although previous studies have shown the field dimension to be of particular
importance. Por ejemplo, a study of publication patterns in social sciences and humanities in
eight European countries showed large differences across fields but also across nations
(Kulczycki, Engels et al., 2018). Además, publication output has been shown to be influ-
enced by individual variables such as gender (Abramo et al., 2021; Sugimoto, Larivière et al.,
2013; Elsevier, 2020), edad (Abramo, D’Angelo, & Murgia, 2016; Gingras, Larivière et al., 2008;
Kyvik, 1990; Levin & Esteban, 1989), and academic rank (Abramo, D’Angelo, & Di Costa,
2011; Blackburn, Behymer, & Sala, 1978; ventura & Mombrù, 2006). De este modo, further research
would be required to provide better fundamentals for cross-national analyses along this dimen-
sion of performance.
Analyses based on comparable research assessment data may give stakeholders, como
gobiernos, Responsables políticos, universidades, business enterprises, and prospective students, valu-
able information on how various higher education institutions perform. En particular, in-depth
investigations along single dimensions of research performance can reveal those with higher
potential efficiency gains, and therefore orient the incentive systems for continuous
mejora.
CONTRIBUCIONES DE AUTOR
Giovanni Abramo: Conceptualización, Investigación, Metodología, Supervisión, Validación,
Escritura: borrador original, Escritura: revisión & edición. Dag W. Aksnes: Curación de datos, Fondos
adquisición, Investigación, Metodología, Supervisión, Validación, Visualización,
Estudios de ciencias cuantitativas
752
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
3
3
7
3
2
2
0
5
7
8
1
3
q
s
s
_
a
_
0
0
1
9
8
pag
d
/
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Unveiling the distinctive traits of a nation’s research performance
Escritura: revisión & edición. Ciriaco Andrea D’Angelo: Curación de datos, Análisis formal, Investi-
gation, Metodología, Validación, Escritura: borrador original.
CONFLICTO DE INTERESES
Los autores no tienen intereses en competencia.
INFORMACIÓN DE FINANCIACIÓN
The research project received funding by the Nordic Institute for Studies Innovation, Investigación
and Education (NIFU), in Oslo, Norway for open access publishing.
DISPONIBILIDAD DE DATOS
Being subject to Clarivate-WoS license restrictions, the raw data cannot be made publicly
disponible, but are available from the authors upon request by the reader for personal interest
solo.
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
REFERENCIAS
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
3
3
7
3
2
2
0
5
7
8
1
3
q
s
s
_
a
_
0
0
1
9
8
pag
d
.
/
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Aagaard, k. (2015). How incentives trickle down: Local use of a
national bibliometric indicator system. Science and Public
Política, 42(5), 725–737. https://doi.org/10.1093/scipol/scu087
Abramo, GRAMO., Aksnes, D. w., & D’Angelo, C. A. (2020). Comparison
of research productivity of Italian and Norwegian professors and
universidades. Journal of Informetrics, 14(2), 101023. https://doi.org
/10.1016/j.joi.2020.101023
Abramo, GRAMO., Aksnes, D. w., & D’Angelo, C. A. (2021). Gender dif-
ferences in research performance within and between countries:
Italy vs Norway. Journal of Informetrics, 15(2), 101144. https://
doi.org/10.1016/j.joi.2021.101144
Abramo, GRAMO., Cicero, T., & D’Angelo, C. A. (2012). Revisiting the
scaling of citations for research assessment. Journal of Infor-
métrica, 6(4), 470–479. https://doi.org/10.1016/j.joi.2012.03.005
Abramo, GRAMO., & D’Angelo, C. A. (2014). How do you define
and measure research productivity? cienciometria, 101(2),
1129–1144. https://doi.org/10.1007/s11192-014-1269-8
Abramo, GRAMO., D’Angelo, C. A., & Di Costa, F. (2011). Research pro-
ductivity: Are higher academic ranks more productive than lower
unos? cienciometria, 88(3), 915–928. https://doi.org/10.1007
/s11192-011-0426-6
Abramo, GRAMO., D’Angelo, C. A., & Murgia, GRAMO. (2013). The collabora-
tion behaviors of scientists in Italy: A field level analysis. Diario
of Informetrics, 7(2), 442–454. https://doi.org/10.1016/j.joi.2013
.01.009
Abramo, GRAMO., D’Angelo, C. A., & Murgia, GRAMO. (2016). The combined
effects of age and seniority on research performance of full pro-
fessors. Science and Public Policy, 43(3), 301–319. https://doi.org
/10.1093/scipol/scv037
Aksnes, D. w., & Sivertsen, GRAMO. (2019). A criteria-based assessment
of the coverage of Scopus and Web of Science. Journal of Data
and Information Science, 4(1), 1–21. https://doi.org/10.2478/jdis
-2019-0001
Aksnes, D. w., Sivertsen, GRAMO., van Leeuwen, t. norte. & Wendt, k. k.
(2016). Measuring the productivity of national R&D systems:
Challenges in cross-national comparisons of R&D input and
publication output indicators. Science and Public Policy, 44(2),
246–258. https://doi.org/10.1093/scipol/scw058
alonso, S., Cabrerizo, F. J., Herrera-Viedma, MI., & Herrera, F.
(2009). h-Index: A review focused in its variants, computation
and standardization for different scientific fields. Journal of Infor-
métrica, 3(4), 273–289. https://doi.org/10.1016/j.joi.2009.04.001
Blackburn, R. T., Behymer, C. MI., & Sala, D. mi. (1978). Investigación
notas: Correlates of faculty publication. Sociology of Education,
51(2), 132–141. https://doi.org/10.2307/2112245
Braun, T., Glänzel, w., & Schubert, A. (1989). Assessing assess-
ments of British science—Some facts and figures to accept or
decline. cienciometria, 15(3–4), 165–170. https://doi.org/10
.1007/BF02017195
D’Angelo, C. A., & Abramo, GRAMO. (2015). Publication rates in 192
research fields. In A. Salah, Y. Tonta, A. A. A. Salah, & C.
Sugimoto (Editores.), Proceedings of the 15th International Society
of Scientometrics and Informetrics Conference (páginas. 909–919).
Istanbul: Bogazici University Printhouse.
Elsevier. (2020). Informe: The researcher journey through a gender lens.
https://www.elsevier.com/research-intelligence/resource-library
/gender-report-2020?dgcid=_EC_Connect (accessed May 9, 2022).
Glanzel, w., Schubert, A. (2004). Analyzing scientific networks
through co-authorship. En H. Moed, W.. Glänzel, & Ud.. Schmoch
(Editores.), Handbook of bibliometric indicators for science and tech-
nología. Dordrecht: Kluwer Academic Publishers.
Gauffriau, METRO., Olesen Larsen, PAG., Maye, I., Roulin-Perriard, A., &
von Ins, METRO. (2008) Comparisons of results of publication count-
ing using different methods. cienciometria, 77(1), 147–176.
https://doi.org/10.1007/s11192-007-1934-2
Gingras, y., Larivière, v., Macaluso, B., & Robitaille, j. PAG. (2008).
The effects of aging on researchers’ publication and citation pat-
charranes. MÁS UNO, 3(12), 1–8. https://doi.org/10.1371/journal
.pone.0004048, PubMed: 19112502
Hicks, D. (1999). The difficulty of achieving full coverage of inter-
national social science literature and the bibliometric conse-
quences. cienciometria, 44(2), 193–215. https://doi.org/10
.1007/BF02457380
Hirsch, j. (2005). An index to quantify an individual’s scientific
research output. Proceedings of the National Academy of Sci-
ences, 102(46), 16569–16572. https://doi.org/10.1073/pnas
.0507655102, PubMed: 16275915
Irvine, J., Martín, B., Peacock, T., & Tornero, R. (1985). Charting the
decline in British science. Naturaleza, 316(6029), 587–590. https://
doi.org/10.1038/316587a0
Estudios de ciencias cuantitativas
753
Unveiling the distinctive traits of a nation’s research performance
Kyvik, S. (1990). Age and scientific productivity. Diferencias
between fields of learning. Higher Education, 19(1), 37–55.
https://doi.org/10.1007/BF00142022
Kulczycki, MI., Engels, t. C. MI., Polonen, J., Bruun, K., Duskova, METRO.,
… Zuccala, A. (2018) Publication patterns in the social sciences
and humanities: Evidence from eight European countries. ciencia-
tometrics, 116(1), 463–486. https://doi.org/10.1007/s11192-018
-2711-0
Larivière, v., Archambault, É., Gingras, y., & Vignola-Gagné, É.
(2006). The place of serials in referencing practices: Comparing
natural sciences and engineering with social sciences and human-
ities. Journal of the American Society for Information Science and
Tecnología, 57(8), 997–1004. https://doi.org/10.1002/asi.20349
Levin, S. GRAMO., & Esteban, PAG. mi. (1989). Age and research productivity
of academic scientists. Research in Higher Education, 30(5),
531–549. https://doi.org/10.1007/BF00992202
Lillquist, MI., & Verde, S. (2010). The discipline dependence of cita-
tion statistics. cienciometria, 84(3), 749–762. https://doi.org/10
.1007/s11192-010-0162-3
Moed, h. F. (2005). Citation analysis in research evaluation. Dor-
drecht: Saltador.
Junta Nacional de Ciencias. (2020). Science and Engineering Indicators
2020. NSB-2020-1. Alejandría, Virginia: National Science Founda-
ción. Available at https://ncses.nsf.gov/pubs/nsb20201 (accedido
Puede 9, 2022).
Norges forskningsråd. (2019). Det norske forsknings- og innovas-
jonssystemet – statistikk og indikatorer 2019. [Ciencia & Technol-
ogy Indicators for Norway 2019]. Oslo. Available at https://www
.forskningsradet.no/indikatorrapporten/ (accessed May 9, 2022).
OECD. (2019). Main science and technology indicators. Disponible
at https://www.oecd-ilibrary.org/science-and-technology/main
-science-and-technology-indicators_2304277x (accessed May
9, 2022).
Piro, F. NORTE., Aksnes, D. w., & Rørstad, k. (2013). A macro analysis of
productivity differences across fields: Challenges in the measure-
ment of scientific publishing. Journal of the American Society for
Information Science and Technology, 64(2), 307–320. https://doi
.org/10.1002/asi.22746
Sandström, Ud., & Sandström, mi. (2009). Meeting the micro-level
challenges: Bibliometrics at the individual level. 12th Interna-
tional Conference on Scientometrics and Informetrics, Rio de
Janeiro, Brasil.
Sidiropoulos, A., Katsaros, D., Manolopoulos, Y. (2007). General-
ized Hirsch h-index for disclosing latent facts in citation net-
obras. cienciometria, 72(2), 253–280. https://doi.org/10.1007
/s11192-007-1722-z
Sivertsen, GRAMO. (2018). The Norwegian model in Norway. Diario de
Data and Information Science, 3(4), 3–19. https://doi.org/10
.2478/jdis-2018-0017
Sorzano, C. oh. S., Vargas, J., Caffarena-Fernández, GRAMO., & Iriarte, A.
(2014). Comparing scientific performance among equals. ciencia-
tometrics, 101(3), 1731–1745. https://doi.org/10.1007/s11192
-014-1368-6
Stringer, METRO. J., Sales-Pardo, METRO., & Amaral, l. A. norte. (2010). Statistical
validation of a global model for the distribution of the ultimate
number of citations accrued by papers published in a scientific
journal. Journal of the American Society for Information Science,
61(7), 1377–1385. https://doi.org/10.1002/asi.21335, PubMed:
21858251
Sugimoto, C. r., Larivière, v., En, C., Gingras, y., & Cronin, B. (2013).
Global gender disparities in science. Naturaleza, 504(7479), 211–213.
https://doi.org/10.1038/504211a, PubMed: 24350369
van Raan, A. F. j. (1993). Advanced bibliometric methods to assess
research performance and scientific development: Basic princi-
ples and recent practical applications. Research Evaluation,
3(3), 151–166. https://doi.org/10.1093/rev/3.3.151
van Leeuwen, t. NORTE., Moed, h. F., Tijssen, R. J., Visser, METRO. S., &
Raan, A. F. (2001) Language biases in the coverage of the science
Citation Index and its consequences for international compari-
sons of national research performance. cienciometria, 51(1),
335–346. https://doi.org/10.1023/A:1010549719484
ventura, oh. NORTE., & Mombrù, A. W.. (2006). Use of bibliometric infor-
mation to assist research policy making. A comparison of publi-
cation and citation profiles of full and associate professors at a
school of chemistry in Uruguay. cienciometria, 69(2), 287–313.
https://doi.org/10.1007/s11192-006-0154-5
Vieira, mi. S., & Gomes, j. A. norte. F. (2010). Citations to scientific arti-
cles: Its distribution and dependence on the article features. Jour-
nal of Informetrics, 4(1), 1–13. https://doi.org/10.1016/j.joi.2009
.06.002
waltman, l. (2012). An empirical analysis of the use of alphabetical
authorship in scientific publishing. Journal of Informetrics, 6(4),
700–711. https://doi.org/10.1016/j.joi.2012.07.008
waltman, l. (2016). A review of the literature on citation impact
indicators. Journal of Informetrics, 10(2), 365–391. https://doi
.org/10.1016/j.joi.2016.02.007
waltman, l., & van Eck, norte. j. (2015). Field-normalized citation
impact indicators and the choice of an appropriate counting
método. Journal of Informetrics, 9(4), 872–894. https://doi.org
/10.1016/j.joi.2015.08.001
waltman, l., van Eck, norte. J., van Leeuwen, t. NORTE., Visser, METRO. S., &
van Raan, A. F. j. (2011). Towards a new crown indicator: Alguno
theoretical considerations. Journal of Informetrics, 5(1), 37–47.
https://doi.org/10.1016/j.joi.2010.08.001
Wildgaard, l., Schneider, j. w., & Larsen, B. (2014). A review of the
characteristics of 108 author-level bibliometric indicators. ciencia-
tometrics, 101(1), 125–158. https://doi.org/10.1007/s11192-014
-1423-3
Wilsdon, J., allen, l., Belfiore, MI., Campbell, PAG., Curry, S., …
Johnson, B. (2015). The metric tide: Report of the independent
review of the role of metrics in research assessment and manage-
mento. HEFCE. https://doi.org/10.4135/9781473978782
Yoshikane, F., & Kageura, k. (2004). Comparative analysis of coau-
thorship networks of different domains: The growth and change
of networks. cienciometria, 60(3), 433–444. https://doi.org/10
.1023/B:SCIE.0000034385.05897.46
Estudios de ciencias cuantitativas
754
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
3
3
7
3
2
2
0
5
7
8
1
3
q
s
s
_
a
_
0
0
1
9
8
pag
d
/
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
