ARTÍCULO DE INVESTIGACIÓN

Might Europe one day again be a global scientific
powerhouse? Analysis of ERC publications
suggests it will not be possible without changes in
research policy

un acceso abierto

diario

Alonso Rodríguez-Navarro1,2

and Ricardo Brito1

1Departamento de Estructura de la Materia, Física Térmica y Electrónica y GISC, Universidad Complutense de Madrid, Plaza de
las Ciencias 3, 28040, Madrid, España
2Departamento de Biotecnología-Biología Vegetal, Universidad Politécnica de Madrid, Avenida Puerta de Hierro 2, 28040,
Madrid, España

Palabras clave: ep index, European research, research assessment, research policy

ABSTRACTO

Numerous EU documents praise the excellence of EU research without empirical evidence
and in contradiction of academic studies. We investigated research performance in two fields
of high socioeconomic importance, advanced technology and basic medical research, in two
sets of European countries, Alemania, Francia, Italia, and Spain (GFIS), and the UK, El
Países Bajos, and Switzerland (UKNCH). Despite their historical and geographical proximity,
research performance in GFIS is much lower than in UKNCH, and well below the world
promedio. Funding from the European Research Council (ERC) greatly improves performance in
both GFIS and UKNCH, but ERC-GFIS publications are less cited than ERC-UKNCH
publicaciones. We conclude that research performance in GFIS and in other EU countries is
intrinsically low, even in highly selected and generously funded projects. The technological
and economic future of the EU depends on improving research, which requires structural
changes in research policy within the EU, and in most EU countries.

1.

INTRODUCCIÓN

Actualmente, research is a key activity in all nations. This is widely recognized and highlighted by
leading research journals, corporations, and government institutions. Por ejemplo: “Scientific
descubrimientos, new technologies, and the aggressive application of cutting edge knowledge are
essential for success in a competitive global economy. Tal como, the strength of a country’s
overall R&D enterprise—including both the public and private realms of this system—is an
important marker of current and future national economic advantage.” (National Science
Board, 2018, pag. 4–7).

The EU is one of the strongest economies in the world, together with those of China and the
EE.UU. Considering the role that knowledge currently plays in an economy, to maintain this
strong position, the EU must also hold a leading position in research. Sin embargo, whether this
occurs is a controversial matter, mainly due to the difficulties of assessing the real performance
of a research system given the intangible nature of much of the output of research (Martín &
Irvine, 1983).

Citación: Rodríguez-Navarro, A. &
Brito, R. (2020). Might Europe one day
again be a global scientific
powerhouse? Analysis of ERC
publications suggests it will not be
possible without changes in research
política. Estudios de ciencias cuantitativas,
1(2), 872–893. https://doi.org/10.1162/
qss_a_00039

DOI:
https://doi.org/10.1162/qss_a_00039

Recibió: 23 Octubre 2019
Aceptado: 20 Enero 2020

Autor correspondiente:
Alonso Rodríguez-Navarro
alonso.rodriguez@upm.es

Editor de manejo:
Juego Waltman

Derechos de autor: © 2020 Alonso Rodríguez-
Navarro and Ricardo Brito. Publicado
bajo una atribución Creative Commons
4.0 Internacional (CC POR 4.0) licencia.

La prensa del MIT

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

1
2
8
7
2
1
8
8
5
8
8
5
q
s
s
_
a
_
0
0
0
3
9
pag
d

.

/

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Might Europe one day again be a global scientific powerhouse?

In view of these considerations, this study aims to investigate EU research performance,
especially considering the role of landmark publications, which comprise a small proportion
of the total number of publications (Bornmann, S.M, & S.M, 2018). Además, it focuses on
two research fields of high socioeconomic importance: advanced technology and basic med-
ical research (Brito & Rodríguez-Navarro, 2018b; Rodríguez-Navarro & Brito, 2018b), y
compares the performance of a small number of countries within the European Research
Area (ERA) with excellent research indicators with that of others of lower performance. Como
a second tool, this study focuses on the research funded by the European Research Council
(ERC), as a means to single out elite and generously funded research in high and low-
performance countries.

1.1. Current Controversy About EU Research Performance

Desde 2006, the research level in the EU has been controversial. While academic studies have
demonstrated the weakness of European research (Albarrán, Crespo, et al., 2010; Bauwens,
Mion, & Thisse, 2011; Bonaccorsi, 2007; Bonaccorsi, Cicero, et al., 2017; Dosi, Llerena, &
Labini, 2006; Herranz & Ruiz-Castillo, 2013; Rodríguez-Navarro & Brito, 2018b; Rodríguez-
Navarro & Narin, 2018), especially in fields that are at the forefront of technological knowl-
borde (Bonaccorsi, 2007; Rodríguez-Navarro & Narin, 2018; Sachwald, 2015), the European
Commission (EC) has developed research policies that assume that European research is ex-
cellent. For many years, EU research policy has been based on the “European paradox,” which
proclaims strong research and weak innovation in the EU. Recientemente, references to this paradox
have disappeared from EC documents, but the notion is as present as it was when it was de-
fined (European Commission, 1995). Por ejemplo, a report from an independent High Level
Group appointed by the EC (European Commission, 2016) begins by describing EU research
as in the European paradox: “When looking ahead to the future of Europe in a globalising
world, the contrast is striking between Europe’s comparative advantage in producing knowl-
edge and its comparative disadvantage in turning that knowledge into innovation and growth.”
A subsequent report contains a motto: “Europe is a global scientific powerhouse” (European
Commission, 2017b, pag. 7).

Lamentablemente, the report of the High Level Group was the foundation for the preparation of
the Horizon Europe research program for 2021–2027. Por lo tanto, if the policy is not changed
the new research program will be based on the notion that “Europe is a world leader in sci-
ence” (European Commission, 2018a, pag. 11), and that “In a swiftly changing world, Europe’s
success increasingly depends on its ability to transform excellent scientific results into inno-
vation that have a real beneficial impact” (European Commission, 2018d, pag. 1). Press releases
also suggest that the new research program will be based on a positive view of EU research, para
ejemplo: “A new programme – Horizon Europe – will build on the achievements and success
of the previous research and innovation programme (Horizon 2020) and keep the EU at the
forefront of global research and innovation” (http://europa.eu/rapid/press-release_IP-18-
4041_en.htm, accessed on March 7, 2019).

In contrast with this positive view of the EC, if the aforementioned academic studies are
correcto, EU research might remain below the level that the EU’s economy requires for many
años. Considering the importance of research, this controversy about its level needs to be
resolved as soon as possible. Sin embargo, because the origin of the controversy lies in the lack
of agreement on how to assess research performance, all efforts must be addressed to applying
methods of research assessment that are convincing for all actors. This purpose, sin embargo, es
more complex than it would be in a developing country, where simple indicators can be used.

Estudios de ciencias cuantitativas

873

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

1
2
8
7
2
1
8
8
5
8
8
5
q
s
s
_
a
_
0
0
0
3
9
pag
d

.

/

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Might Europe one day again be a global scientific powerhouse?

The EU is a powerful economy, it was a world scientific leader not so long ago, it still produces
an enormous amount of scientific research, and it has competitive industries that are able to
maintain a high level of incremental innovation. EU research cannot, por lo tanto, be correctly
described using simple indicators that are unable to detect complex problems. Por ejemplo,
the statements: “The EU is a global research powerhouse responsible for one-fifth of all R&D
investments worldwide” and “In terms of overall scientific production, Europe is in the lead,
ahead of the United States and China” (European Commission, 2018mi, páginas. 78 y 154) son
true, but these truths are not relevant. Investment and overall scientific production say nothing
about discoveries and knowledge advancement, which are the driving force of technological
advances and the roots of radical innovations.

1.2. EU Research Is Not Homogeneous Across Countries

Academic research assessments of individual countries have shown that the low performance
reported for the EU as a whole does not occur in all EU countries; En realidad, the UK and the
Países Bajos (in the EU) and Switzerland (in the ERA) maintain highly competitive research
(Brito & Rodríguez-Navarro, 2018b; Rodríguez-Navarro & Brito, 2018b).

These large differences across countries can lead to an incomplete perception of the situ-
ation of EU research if they are not taken into consideration; sin embargo, most of the studies that
have so far examined EU research have considered the EU as a whole (p.ej., Albarrán et al.,
2010; Dosi et al., 2006; Herranz & Ruiz-Castillo, 2013; Rodríguez-Navarro & Narin, 2018),
and this does not help to describe a problem that needs to be addressed at the country level.
According to population, tamaño, and degree of socioeconomic development, the set of countries
that would have the greatest responsibility for weak research performance in the EU is
Alemania, Francia, Italia, and Spain (GFIS; ver tabla 2 in Bauwens et al., 2011; Tables 1 y
3 in Brito & Rodríguez-Navarro, 2018b; and Table 5 in Rodríguez-Navarro & Brito, 2018b).
These countries represent approximately 50% of the EU population and 55% of its GNP,
y, por lo tanto, the study of these four countries, rather than the study of the whole EU,
should give a clearer diagnosis of the difficulties of EU research in competing at the forefront
of knowledge.

1.3. Research Assessment Methods

The controversial views regarding the level of research in the EU lie in the absence of agree-
ment about how to measure success in research. Bibliometric indicators based on counting
publications and citations have been used for a long time (Godin, 2003, 2006). Sin embargo, si
the purpose is to estimate the contribution to the advancement of knowledge then most of
these indicators give misleading information. This occurs because only a low proportion of
the total number of publications report important scientific breakthroughs, which is an exten-
sion of the differences that exist between “normal” and “revolutionary” science (Kuhn, 1970).
This idea can also be deduced by taking into consideration that so far “the benefits of scientific
discoveries have been heavy-tailed” (Prensa, 2013, pag. 822). This implies that the evaluation of
breakthroughs and publications in this heavy tail are of crucial importance in research
evaluación.

Counting the number of Nobel prizes has occasionally been used as a reliable indicator of
research success (p.ej., Braun, Szabadi-Peresztegi, & Kovács-Németh, 2003; Charlton, 2007a,
2007b; Heinze, Jappe, & Pithan, 2019; Schlagberger, Bornmann, & Bauer, 2016), but this method
has serious limitations because Nobel prizes are awarded to extremely infrequent scientific
achievements. It can therefore be applied to big science producers (p.ej., the USA, the EU,

Estudios de ciencias cuantitativas

874

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

1
2
8
7
2
1
8
8
5
8
8
5
q
s
s
_
a
_
0
0
0
3
9
pag
d

.

/

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Might Europe one day again be a global scientific powerhouse?

Instituto de Tecnología de Massachusetts [CON], Harvard University, University of Cambridge) pero
in most countries and institutions it is a useless measure, because there are no Nobel prizes to be
counted. Although Nobel prizes cannot be used as indicators of research performance, sin embargo,
they can be used to validate other indicators, which demonstrates that many bibliometric
indicators do not correlate with the number of Nobel prizes (Rodríguez-Navarro, 2011).

Accepting that what really counts in the progress of science are the infrequent scientific
breakthroughs that cannot be counted in most countries and institutions, the conclusion is that
research indicators cannot be formulated by simply counting something. Some dichotomous
indicators count the number of papers in a certain fraction of the upper tail of citation distri-
butions, assuming that these are the most influential papers (Albarrán, Herrero, et al., 2017).
Sin embargo, these indicators count a fraction of the upper tail that is too large and do not provide
information about the number of less frequent papers that are really influential. Por ejemplo,
the number of papers in the top 1% ranked by citations does not provide information about the
number of highly influential publications that are in the more exclusive top 0.02% (Bornmann
et al., 2018).

The alternative to this type of indicator is to characterize the complete distribution of citations.
Once the distribution is known, the probability of publishing highly cited papers at any level can
be calculated (Rodríguez-Navarro & Brito, 2019). En la sección 2.2 we describe the methods.

2. MÉTODOS

2.1. Rationale and Design of This Study

As presented in Section 1.2, previous studies suggest that responsibility for the weak research
performance in the EU lies mainly with GFIS; a study of these four countries as a single set has
advantages in comparison to the study of independent countries, not only because the con-
clusions are more representative—GFIS represent around 50% of the EU’s population—but,
more importantly, because the study of a set of countries allows for a larger base of research
publicaciones, making it statistically more robust. According to previous studies in technologi-
cal and biotechnological areas, and basic medical research, research performance in
Germany is better than in France, Italia, and Spain; sin embargo, the differences are small when
taking a competitive country, such as Switzerland, as a reference (Brito & Rodríguez-Navarro,
2018b; Rodríguez-Navarro & Brito, 2018b). GFIS are thus more similar in their weakness, como
compared to Switzerland, than different in their degrees of weakness.

The study of a single set of countries with weak research performances has, nonetheless, el
inconvenience of lacking a reference of similar countries—in Europe—with competitive
research performance. We therefore selected a reference set comprising the UK, el
Países Bajos, and Switzerland (henceforth UKNCH) for comparison in the analyses. El COM-
parative research performances of UKNCH in fast-evolving technologies and basic medical
research vary notably (Brito & Rodríguez-Navarro, 2018b; Rodríguez-Navarro & Brito,
2018b) because university specialization in Europe (Bonaccorsi, Haddawy, et al., 2017) tiene
a strong effect on small countries. Por lo tanto, although UKNCH research performance is taken
as a reference in this study, our results are not indicative of the research performances of the
individual countries.

We studied domestic papers at the ERA level—that is, papers authored by at least one GFIS
or UKNH researcher plus others from ERA countries external to GFIS and UKNCH, but none
from nonERA countries. Por lo tanto, collaborations of ERA countries external to GFIS and
UKNCH could exist in any of the two sets of independent papers.

Estudios de ciencias cuantitativas

875

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

1
2
8
7
2
1
8
8
5
8
8
5
q
s
s
_
a
_
0
0
0
3
9
pag
d

.

/

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Might Europe one day again be a global scientific powerhouse?

As already advanced in Section 1, another basic design of our study involves taking advan-
tage of the EU funding programs, especially the ERC program, to select for different levels of
excellence. The rationale is simple: The scientific success of a certain country or association of
countries depends on a combination of the ability of its researchers, their funding resources,
and the research environment—we use this term in a broad sense, including everything from
stocks of knowledge to national evaluation methods (Sandström & van den Besselaar, 2018).
By selecting a specific type of funding some of the country differences are eliminated and fur-
ther insights are possible.

Statistically, citations follow a lognormal distribution (Rodríguez-Navarro & Brito, 2018a;
Viiu, 2018; and references therein) where the μ and σ parameters of the lognormal function
increase in parallel with research success. If a subpopulation in a given country is made up of
the most capable researchers—for example, in an elite research university—it is certain that,
ceteris paribus, the lognormal citation distribution of their publications will have higher μ and
σ parameters than those of the total population. Because countries have complex research
systems made up of multiple subpopulations of researchers, a low average capacity can be
explained by (a) the low performance of all researchers due to a generally poor research en-
vironment and insufficient funding, y (b) the existence of many low research performance
institutions that conceal the high performance of a few elite institutions.

These two possibilities can be distinguished by studying the ERC-funded GFIS and UKNCH
publicaciones. The ERC was created to promote excellence in science (Celis & Gago, 2014;
Luukkonen, 2014), and it is based on generous funding and the selection of the most excellent
research projects. Generous funding could correct insufficient funding in a country or set of
countries, but not a poor research environment. It is therefore likely that studying the ERC-funded
subpopulations will provide information about the research environment in GFIS and UKNCH.

ERC-funded GFIS research was also compared with the research of an elite institution. El
ERC funding of research projects represents an ex ante selection that is significantly different
from the selection of researchers in elite research institutions, whose members are mainly selected
via an ex post research assessment improved by multiple considerations of future perspectives.
The selected researchers then obtain generous funding. We studied the effect of these two models
for selecting excellence by comparing MIT- and ERC-funded GFIS publications.

2.2. Research Assessment Approaches

The basis of this study was to calculate the probability that a random publication from a country
or institution reaches a certain level of citations. For this purpose, most results were obtained
through the use of the ep index, which is based on analysis of the distribution of local publications
among global publications (Rodríguez-Navarro & Brito, 2019). Sin embargo, citation distributions
are lognormal (Rodríguez-Navarro & Brito, 2018a; Viiu, 2018; and references therein) and local
comparisons can also be performed by omitting the comparison with global publications through
the more direct but less accurate method of comparing lognormal distributions of citations.

The first approach is based on the fact that considering their number of citations, the rank of
local papers expressed as a function of their global rank follows a power law (Rodríguez-
Navarro & Brito, 2018a). As a mathematical consequence, the distribution of local papers in
global percentiles attending to their number of citations also follows a power law (Brito &
Rodríguez-Navarro, 2018a).

The ep index is a derivative of the exponent of the power law that percentile frequencies
obey; eso es, a mathematical parameter that characterizes the distribution of local papers

Estudios de ciencias cuantitativas

876

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

1
2
8
7
2
1
8
8
5
8
8
5
q
s
s
_
a
_
0
0
0
3
9
pag
d

/

.

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Might Europe one day again be a global scientific powerhouse?

among the global papers. It reveals the breakthrough potential (Rodríguez-Navarro & Brito,
2018b), namely the efficiency of the system in scaling up from less cited papers to highly cited
documentos: Por ejemplo, the decrease of the number of papers in the top 1% of most globally cited
papers with reference to the number of papers in the top 10% of most globally cited papers. Un
ep index of 0.1 indicates that as the percentile decreases, the number of papers in a country or
institutions decreases at the same rate as in global publications. Por lo tanto, if the ep index is
lower than 0.1 in a country or institution, the research performance of that country or institu-
tion is worse than the global average. Como consecuencia, in countries that are international leaders
in research, the ep index must be significantly higher than 0.1—excellent research systems
have ep index values of around 0.2.

The probability that a random paper from a given country or institution reaches a top per-
centile x is calculated by simply raising ep to a power, as shown by the formula (Rodríguez-
Navarro & Brito, 2019):

P xð Þ ¼ ep

Þ
2 – lgx
d

(1)

As a first approximation, it can be assumed that important breakthrough papers are linked
to radical innovations and future technologies, while incremental innovations in present tech-
nologies are linked to less cited papers (Dewar & Dutton, 1986); por lo tanto, a meaningful re-
search assessment should also provide the probability and expected frequencies of these
documentos. The ep index fulfills this condition because probabilities at all citation levels are pow-
ers of the ep index; probabilities at high citation levels— infrequent achievements/high powers
of the ep index—correlate with the number of Nobel prizes (Brito & Rodríguez-Navarro,
2018a) and probabilities at low citation levels—more frequent achievements/low powers of
the ep index—correlate with peer reviews (Rodríguez-Navarro & Brito, 2020; Traag &
waltman, 2019).

In addition to this method, cross-country comparisons can also be performed by omitting
the comparison with global publications through the more direct but less accurate method of
comparing lognormal distributions of citations. In this method, once the parameters of the log-
normal distributions of a country, institución, or subpopulation of papers have been calculated,
the cumulative probability at any citation level can be calculated. In absolute terms, local
probabilities do not have a clear meaning, but this meaning can be obtained through compar-
ison with a reference or gold standard that publishes a similar number of papers. As already
mentioned, MIT publications were used as a reference.

The formulas of the lognormal distribution and upper cumulative distribution for a paper to

receive more than Ca citations are the following (Aitchison & Marrón, 1963):

»

#

d
p C; μ; pag

Þ ¼

pag

1ffiffiffiffiffi
2Pi

Cσ

Þ2
d
exp − lnC − μ
2p2

p Cað

Þ ¼

Z ∞

Ca

d
p C; μ; pag

Þ dC

(2)

(3)

2.3. Technologies Selected in This Study

The EC reasonably places emphasis on the importance of research for the economic future of
the EU. Por ejemplo, one of the main features of the Horizon Europe research program is to
“foster the EU’s industrial competitiveness and its innovation performance, notably supporting

Estudios de ciencias cuantitativas

877

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

1
2
8
7
2
1
8
8
5
8
8
5
q
s
s
_
a
_
0
0
0
3
9
pag
d

.

/

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Might Europe one day again be a global scientific powerhouse?

market-creating innovation via the European Innovation Council and the European Institute of
Innovation and Technology” (European Commission, 2018b).

The research areas and topics to be investigated were selected in two independent fields:
physical and chemical technologies (henceforth TECH) and biological technologies and basic
investigación médica (henceforth BIO-MED), which are currently at the forefront of knowledge.
They were selected in two previous studies (Brito & Rodríguez-Navarro, 2018b; Rodríguez-
Navarro & Brito, 2018b), with a slight increase in the scope in the case of technology (ver
Sección 2.5). These research areas and topics will probably continue being important and sup-
porting market-creating innovations and healthcare advancements in the near and medium
future.

2.4. Calculations of the e

p Index

As described in Section 2.2, the ep index is a derivative of the exponent of the power law that
percentile frequencies obey. For its calculation, we counted the number of papers from the
two sets of countries and MIT that were in global top percentiles according to the number
de citas, as described previously (Brito & Rodríguez-Navarro, 2018a; Rodríguez-Navarro
& Brito, 2018b). In previous cases, we counted the number of papers in six top percentiles
de 7 a 35, but for ERC publications we added top percentiles 3 y 1, porque, in some
casos, the percentile distribution of ERC publications deviates slightly from the power law. En
these cases, the ep index obtained from fitting the power law to the higher percentiles was
higher than those obtained from fitting to the lower percentiles. When this occurs, to better
predict the probability of breakthrough papers in the top 0.01%, we normally fitted the power
law to the lower percentiles instead of fitting the power law to all data points. In the cases
presented in this study the fits to the power law showed R2 and p values calculated by using
the Χ2 statistics (Prensa, Flannery, et al., 1989) that were higher than 0.99.

2.5. Bibliometric Searches

Bibliometric searches were performed in the Science Citation Index Expanded of the Web of
Science Core Collection ( WoS), using the “Advanced Search” feature. For TECH searches we
usado (TS=(ionic liquid* OR liquid electrolyte* OR liquid salt* OR energy transfer OR fuel cell*
OR quantum dot* OR composite material* OR transistor* OR semiconductor OR supercon-
ductor OR graphene OR batter* OR solar cell* OR electronic OR metal organic framework*
OR nano*) OR SU=Telecommunications)). For BIO-MED searches we used (SU=((biochemis-
intentar & molecular biology OR biotechnology & applied biotechnology OR cell biology OR mi-
crobiology) NOT (computer science OR mathematical & computational biology)) OR TS=
((cancer OR crispr* OR microbiota OR stem cell* OR immunity OR inflamma*) NOT (Estadísticas
OR trial OR survey))). For ERC-funded publications we used FT=(ERC OR (European Research
Council)); for EU-funded research excluding ERC- and Marie Curie (MC)- funded publications,
we used FT=(((COST OR FEDER OR FP7 OR FP6 OR (European Social Fund) O (European
Regional Development Fund) O (European Commission)) NOT (ERC OR (European Research
Council) OR Marie Curie))). Although MC funding was excluded in these searches, MC pub-
lications were not treated as ERC publications because they have a lower level of excellence
(results not shown).

We retrieved only “articles,” which excludes review papers, because review papers often
receive more citations than the original articles on which they are based. Searches were per-
formed between February 23 and March 5, 2019. Some countries or sets of countries were
analyzed on different days, but each analysis on a different day was complete, including world

Estudios de ciencias cuantitativas

878

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

1
2
8
7
2
1
8
8
5
8
8
5
q
s
s
_
a
_
0
0
0
3
9
pag
d

/

.

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Might Europe one day again be a global scientific powerhouse?

and country citation distributions. Because the ERC program started in 2007, our citation anal-
yses were performed for the years 2011–2014.

To calculate the μ and σ parameters of lognormal distributions of citations of MIT and ERC-
GFIS publications, we retrieved the number of citations of all publications and used the max-
imum likelihood method to fit the empirical data to a lognormal function.

3. RESULTADOS

3.1. General Appraisal of EU Research Performance

Before going into more detail, we obtained a general appraisal of the research performance in
the EU from Science & Engineering Indicators, published by the National Science Board of the
Fundación Nacional de Ciencia (Junta Nacional de Ciencias, 2016, 2018). Among other indicators
they report the proportions of papers within five citation-based global percentiles in 13 re-
search areas for the years 2002, 2004, 2012, y 2014, which allow the calculation of the
ep index values. Although the maximum differences between USA and EU research occur
in topics that are at the forefront of technological knowledge (Bonaccorsi, 2007; Rodríguez-
Navarro & Narin, 2018; Sachwald, 2015), the National Science Board data enables a general
appraisal of research performances in the EU and the USA. Mesa 1 records the ep index values

Mesa 1. Values of the ep index calculated from the data reported in Science and Engineering Indicators. Data for the years 2002 y 2012 son
en el 2016 informe, and data for the years 2004 y 2014 are in the 2018 informe.

EE.UU

EU

Año

Research field
Ingeniería

Astronomy

Chemistry

Physics

Geosciences

Matemáticas

Computer sciences

Agricultural sciences

Biological sciences

Medical sciences

Other life sciences

Psicología

Social sciences

2002
0.126

0.127

0.134

0.128

0.118

0.121

0.137

0.129

0.119

0.127

0.103

0.115

0.120

2004
0.138

0.119

0.131

0.134

0.119

0.126

0.139

0.118

0.119

0.126

0.102

0.111

0.119

2012
0.133

0.125

0.127

0.145

0.131

0.109

0.142

0.127

0.129

0.132

0.108

0.121

0.118

2014
0.134

0.132

0.114

0.143

0.130

0.112

0.138

0.128

0.132

0.135

0.102

0.110

0.119

2002
0.096

0.102

0.093

0.102

0.095

0.091

0.082

0.100

0.095

0.096

0.104

0.084

0.083

2004
0.105

0.109

0.093

0.099

0.103

0.096

0.082

0.106

0.100

0.100

0.104

0.102

0.094

2012
0.109

0.109

0.093

0.113

0.109

0.095

0.100

0.118

0.113

0.117

0.112

0.096

0.100

2014
0.102

0.107

0.091

0.116

0.111

0.106

0.090

0.112

0.111

0.118

0.111

0.105

0.107

Significar

0.123

0.123

0.127

0.125

0.094

0.100

0.106

0.107

Science and Engineering Indicators 2016 y 2018, Junta Nacional de Ciencias, Fundación Nacional de Ciencia, Arlington, Virginia. Appendix Tables 5-59 y 5-48,
respectivamente. In the ep index, the higher the better.

Estudios de ciencias cuantitativas

879

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

1
2
8
7
2
1
8
8
5
8
8
5
q
s
s
_
a
_
0
0
0
3
9
pag
d

.

/

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Might Europe one day again be a global scientific powerhouse?

in all these 52 cases—4 years and 13 research areas—which clearly indicate that while the
≈
USA is above the world average (ep
0.10). Only in the case of “Other Life Sciences” was the EU slightly ahead of, or on par with,
the USA.

≈ 0.12) the EU is almost exactly at the world average (ep

In “Engineering,” the means of the ep index values of the four evaluations are 0.13 para el
USA and 0.10 for the EU. The top 0.01% of most cited papers reasonably represents the per-
centile where most breakthrough and landmark publications concentrate (Bornmann et al.,
2018; Brito & Rodríguez-Navarro, 2018a), and the probability that a random paper is located
in this percentile—ep index raised to the fourth power, ecuación. (1)—reveals the efficiency of re-
search systems in making discoveries. The probability that a random paper reaches the 0.01
−4 in the EU, 2.9
percentile in the field of “Engineering” is 2.9 × 10
times higher in the USA than in the EU. This is a significant difference that implies that the EU
should publish 2.9 times more papers to obtain the same number of achievements.

−4 in the USA and 1.0 × 10

The Science & Engineering Indicators reports have no data that allow a comparison of GFIS
and UKNCH research. Por lo tanto, we obtained a general overview of their differences by
counting the number of universities among the top universities in the Leiden Ranking
(Ranking, 2019). For this purpose, we counted the number of GFIS and UKNCH universities
among the top 25, 50, y 100 universities ordered by the Ptop 1% in the field of “Physical
sciences and engineering,” in two periods, 2006–2009 and 2014–2017 (Mesa 2). The differ-
ences between UKNCH and GFIS were striking in the top 25, because in the two periods
UKNCH had four and five universities while GFIS had none. In the top 50, UKNCH has eight
and seven, and GFIS has three and zero universities in the two periods, respectivamente. In the top
100, again UKNCH performed better than GFIS. In the first period GFIS has 10 universities that
decreased to only four in the second period. The differences between the two periods were
due to the emergence of China as a global scientific powerhouse, which impaired the rank of
GFIS universities the most, and had a lesser effect on UKNCH than on the US universities.

3.2. EU-Funded Publications in TECH and BIO-MED Research

This study focuses on TECH and BIO-MED; the number of papers published by GFIS and
UKNCH in BIO-MED is rather high. In BIO-MED, it represents about 17–18% of the total num-
ber of publications; in TECH the proportion decreases to approximately 13% y 10% in GFIS
and UKNCH, respectivamente (Mesa 3). These figures show that in addition to its economic im-
portance, we are dealing with a paramount research activity in the investigated countries. El

Mesa 2. Number of universities among the top 25, 50, y 100 in the CWTS Leiden Ranking in
periods 2006–2009 and 2014–2017 in the research field of “Physical sciences and engineering,"
Ptop 1% indicator

Country/Set
EE.UU

Top 25
17

2006–2009
Top 50
28

Top 100
45

Top 25
11

2014–2017
Top 50
20

Top 100
31

UKNCH

GFIS

Porcelana

4

0

1

8

3

3

17

10

7

5

0

7

7

0

17

11

4

35

UKNCH: the UK, Países Bajos, and Switzerland; GFIS: Alemania, Francia, Italia, and Spain. Fuente: https://www.
leidenranking.com/ranking/2019/list, accessed May 22, 2019.

Estudios de ciencias cuantitativas

880

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

1
2
8
7
2
1
8
8
5
8
8
5
q
s
s
_
a
_
0
0
0
3
9
pag
d

.

/

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Might Europe one day again be a global scientific powerhouse?

Mesa 3. Total number of publications, and number of publications in chemical and physical
tecnologías (TECH), and biotechnology and basic medical research (BIO-MED) from Germany,
Francia, Italia, and Spain (GFIS), and from UK, Países Bajos, and Switzerland (UKNCH)

Año
2011

2014

2018

Total
143,017

GFIS
TECH
17,801

149,224

20,080

144,404

19,806

BIO-MED
24,994

26,828

24,453

Total
62,048

63,066

61,590

UKNCH
TECH
5,606

6,250

6,533

BIO-MED
11,171

10,884

10,030

difference in the total number of papers between UKNCH and GFIS roughly reflects their pop-
ulation difference.

The data in Table 3 show a situation of concern for both UKNCH and GFIS, because the
number of publications did not increase in the period from 2011 a 2018, when it is well
known that several Asian countries were contributing to a notable increase in the number
of publications and research advances in these technological areas (Brito & Rodríguez-
Navarro, 2018b; Rodríguez-Navarro & Brito, 2018b; Rodríguez-Navarro & Narin, 2018).
The consequence is that the share of global publications in TECH and BIO-MED has been
continually decreasing from 2011 a 2018 in both GFIS and UKNCH. This loss of share is
particularly important in TECH, but it also occurs for the USA.

Although this loss of share seems to be a general trend for the EU and the USA, it is espe-
cially worrying for GFIS because the ep index values for TECH and BIO-MED publications
(0.06–0.07) were significantly lower than the world average (Mesa 4). Taking these two facts
together, the effect is that research by GFIS in technological areas and basic medicine could
eventually become of low global relevance in terms of scientific advancement. Our next step
was thus to study the reasons for this low research performance by analyzing the subpopula-
tions of elite papers selected according to their funding sources.

The subpopulation of papers selected by their EU funding in programs other than ERC and
MC was higher in GFIS than in UKNCH in the two technologies (Mesa 4; 14% versus 12% en

Mesa 4. Publications in TECH and BIO-MED from Germany, Francia, Italia, and Spain (GFIS) y
from the UK, Los países bajos, and Switzerland (UKNCH): number and proportion of publications,
and ep index by type of funding (año 2014)

Country set/funding
GFIS

TECH publications

BIO-MED publications

Número

%

ep

Número

%

ep

All papers published

20,080

100

0.062

26,828

100

EU funded not (ERC or MC)

2,813

14.0

0.066

2,305

ERC funded

UKNS

726

3.6

0.143

510

8.6

1.9

All papers published

6,250

100

0.104

10,884

100

EU funded not (ERC or MC)

ERC funded

751

534

12.0

0.145

8.5

0.191

592

414

5.4

3.8

0.066

0.075

0.166

0.103

0.144

0.271

881

Estudios de ciencias cuantitativas

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

1
2
8
7
2
1
8
8
5
8
8
5
q
s
s
_
a
_
0
0
0
3
9
pag
d

.

/

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Might Europe one day again be a global scientific powerhouse?

TECH, y 8.6% versus 5.4% in BIO-MED, for GFIS and UKNCH respectively). In GFIS, se-
lection by this EU funding involved a weak increase in the ep index, apenas 10%, with ref-
erence to that of the total number of papers. This indicates a practically nonexistent selection
of above-average research in these programs. A diferencia de, this funding clearly increased the ep
index in UKNCH, de 0.10 a 0.14, which indicates a significant selection of above-average
investigación.

The study of ERC-funded papers is more interesting, because ERC funding is based on the
selection of the highest research excellence through high-quality peer review (Celis & Gago,
2014; Luukkonen, 2014). The proportion of this elite subpopulation (Mesa 4) was consider-
ably lower in GFIS than in UKNCH—3.6% versus 8.5% y 1.9% versus 3.8% in TECH and
BIO-MED, respectively—which once more denotes the lower research competitiveness of
GFIS. Curiosamente, despite the apparently more stringent selection by ERC funding in GFIS
than in UKNCH, the ep index had lower values for ERC-GFIS than for ERC-UKNCH papers:
0.14 y 0.17 versus 0.19 y 0.27 in TECH and BIO-MED, and GFIS and UKNCH,
respectivamente.

This lower success of GFIS versus UKNCH in ERC-funded publications in TECH and BIO-
MED is a general trend also seen in the number of ERC grants in all fields over the years
(Cifra 1). This number increased continually from 2007 a 2012, when it reached a plateau.
Similarmente, the number of ERC publications in TECH and BIO-MED increased from 2012 a
2016 and has remained almost constant since then (Cifra 2). Interpretation of these data
needs to consider that the size of the GFIS research system is more than twice that of the
UKNCH system (Mesa 3).

3.3. Evolution of the ERC-GFIS, ERC-UKNCH, and MIT Publications

As already argued (Sección 2.1), low ep index values in a given country might arise in two dif-
ferent research scenarios: Publications are produced by either a homogeneous population of
researchers of low competitiveness—most probably because of a poor research environment—
or by a heterogeneous population of researchers, where some were highly competitive but
others were less competitive. These two cases may be distinguished through the study of

Cifra 1. Number of grants from the European Research Council (ERC) awarded to Germany,
Francia, Italia, and Spain (GFIS) and to the UK, Los países bajos, and Switzerland (UKNCH) de
2007 a 2017. Fuente: https://erc.europa.eu/, accedido 9 Abril 2019; Proof of Concept grants were
not included.

Estudios de ciencias cuantitativas

882

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

1
2
8
7
2
1
8
8
5
8
8
5
q
s
s
_
a
_
0
0
0
3
9
pag
d

/

.

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Might Europe one day again be a global scientific powerhouse?

Cifra 2. Number of ERC publications in rapid evolving physical and chemical rapidly evolving
tecnologías (TECH) and in biotechnological and basic medical research (BIO-MED) de
Alemania, Francia, Italia, and Spain (GFIS), and from the UK, Los países bajos, and Switzerland
(UKNCH) in years 2011−2018.

ERC-funded publications because ERC funding implies a selection for excellence. For better
representativeness, the study must include several years, because the ep index shows annual
variations in the hot topics of this study.

Cifra 3 shows the ep index values for 2011–2014 for ERC-GFIS, ERC-UKNCH, and MIT
publications in TECH and BIO-MED. In TECH there was no difference between ERC-UNKCH
and MIT publications, while ERC-GFIS publications showed lower ep index values. The ep
index in ERC-UKNCH and MIT publications varies around 0.2, while in ERC-GFIS publica-
tions it was around 0.15; the ep index values for the three actors can be considered constant
de 2011 a 2014. In BIO-MED, the ep index values in ERC-UKNCH publications varies from
0.15 a 0.28, always below those of MIT, which varies from 0.23 a 0.33. In the case of GFIS
the ep index values varies around 0.15. In both ERC-UKNCH and MIT publications, había
a clear increase in the ep index from 2011 a 2014, while it remained constant in the case of
ERC-GFIS publications. Although irrelevant regarding the ep index, it is worth noting that the

Cifra 3. Values of the e(pag) index in years 2011−2014 of the ERC-funded publications from
Alemania, Francia, Italia, and Spain (GFIS) and the UK, Los países bajos, and Switzerland
(UKNCH); and publications from the Massachusetts Institute of Technology (CON). Physical and
chemical technologies (TECH; left panel A) and biotechnology and basic medical research (BIO-
MED; right panel B). Lines are drawn to guide the eye.

Estudios de ciencias cuantitativas

883

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

1
2
8
7
2
1
8
8
5
8
8
5
q
s
s
_
a
_
0
0
0
3
9
pag
d

/

.

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Might Europe one day again be a global scientific powerhouse?

Mesa 5. Probability that a random ERC-GFIS or MIT paper is highly cited, calculated from the lognormal distribution of citations and from the
ep index; and MIT/ERC-GFIS probability ratio. The top 0.01 percentile was fixed for the ep index method; for lognormal calculations, el
number of citations was arbitrarily selected as described in the text

Lognormal
ERC-GFIS
pag
μ

Lognormal
CON

μ

pag

Lognormal
probabilidad
ERC-GFIS

CON

ep probability
arriba 0.01 percentile
CON

ERC-GFIS

MIT/ERC-GFIS
problema. ratio

ep

Lognor.

Citations

Año
TECH

2011

3.458

1.196

3.420

1.339

1000

0.00195

0.00461

0.00051

0.00163

2012

3.240

1.118

3.412

1.191

2013

3.138

1.129

3.250

1.203

2014

2.922

1.062

3.028

1.196

850

700

500

BIO-MED

0.00086

0.00257

0.00031

0.00084

0.00126

0.00304

0.00043

0.00184

0.00097

0.00385

0.00017

0.00187

10.7

3.2

2.7

4.3

2011

3.755

1.030

3.786

1.248

1000

0.00110

0.00617

0.00107

0.00300

2.8

2012

3.398

0.934

3.592

1.212

2013

3.325

1.068

3.566

1.290

2014

3.081

0.954

3.491

1.262

850

700

500

0.00017

0.00463

0.00034

0.00555

16.2

0.00126

0.01036

0.00078

0.00516

6.6

0.00051

0.01543

0.00071

0.01321

18.7

2.4

3.0

2.4

4.0

5.6

27.1

8.3

30.1

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

1
2
8
7
2
1
8
8
5
8
8
5
q
s
s
_
a
_
0
0
0
3
9
pag
d

.

/

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

number of ERC-funded publications was continually increasing from 2011 a 2014 (Cifra 2)
while the MIT papers remained constant at around 550 (results not shown).

3.4. Calculations Based on the Lognormal Distributions: ERC-GFIS Versus MIT Publications

One way of comparing the research performances of two countries or institutions is to com-
pare the probabilities of publishing highly cited papers. These probabilities can be
calculated from the ep index values Eq. (1) and from the their lognormal citation distributions
ecuación. (3). These two methods can be made equivalent, without any reference to the global
documentos, by selecting a number of citations for Eq. (3) that reasonably represent the highly cited
tail that corresponds to the percentile used in Eq. (1). We used 1,000 citations for papers pub-
lished in 2011 and reduced the numbers in proportion to the lower citation window for papers
published in 2012–2014.

Mesa 5 summarizes the MIT and ERC-GFIS probabilities and their ratios calculated follow-
ing both approaches—Eq. (1) and Eq. (3). The results could not be equal because there was not
an exact equivalence between the 0.01 percentile applying the ep index method and the num-
ber of citations reasonably but arbitrarily selected for the lognormal calculation. Sin embargo,
the results are totally consistent and exhibit similar annual variations; they unequivocally show
that ERC-GFIS research does not compete well with MIT research in TECH and BIO-MED.

4. DISCUSIÓN

4.1. Wrong Diagnoses and Misguided Policies

This was the title of the final section and conclusion of a paper about European research that
was written 13 years ago (Dosi et al., 2006). There have been some changes since then,
especially regarding basic research and the creation of the ERC, but essentially the situation

Estudios de ciencias cuantitativas

884

Might Europe one day again be a global scientific powerhouse?

has not altered. The EC brags about excellent research in contradiction to academic studies
(Sección 1); the question is whether this excellence is a misjudgment based on incorrect
evaluaciones.

As a first approach to evaluate European research, we took advantage of the large amount
of data provided by two successive reports of the Science & Engineering Indicators (National
Science Board, 2016, 2018). The advantage of using these data is that they have been gener-
ated by the USA’s National Science Board and not by EU academics. The data recorded by the
Ciencia & Engineering Indicators include the distribution of papers in 13 research areas and
five percentiles for four years across a span of 13 years from 2002 a 2014.

As discussed in Section 3.1, according to the ep index values (Mesa 1), the advantage of the
USA over the EU is high. Only in the case of “Other Life Sciences” was the EU slightly ahead
de, or on par with, the USA. The top 0.01% of most cited papers reasonably represents the
percentile where most breakthrough and landmark publications concentrate (Bornmann
et al., 2018; Brito & Rodríguez-Navarro, 2018a). In “Engineering,” the probability that a ran-
dom paper locates in this percentile is 2.9 times higher for a US paper than for an EU paper.

It is worth highlighting that the engineering research field mixes different technologies and
that the 2.9 times difference is just the tip of the iceberg. In rapidly evolving technology topics,
in the EU excluding the UK, the difference can be up to eight times (Rodríguez-Navarro &
Brito, 2018b). The question raised by these results is whether the EU, excluding the UK, es
prepared to invest eight times more in research than the USA to obtain a similar number of
scientific advances.

En resumen, academic publications and simple analyses (Mesa 1) show that the EU is not a
global scientific powerhouse. If current research policy continues to be based on the same
wrong diagnoses of scientific excellence that have oriented EU research policy for more than
20 años, then EU research will not correct its weak position. Even worse, the EU had only two
relevant competitors 20 years ago (the USA and Japan), but China (together with other Asian
countries) is currently in the competition and becoming even stronger than the USA and
Japón.

The EU’s sustained praise of its research excellence based on wrong diagnoses gives rise to
misguided research policies and failures in reaching predicted targets. Por ejemplo, the objec-
tive of the Lisbon Strategy, launched in 2000, that the EU should become “the most compet-
itive knowledge based economy in the world by 2010” (European Commission, 2010, pag. 2),
was far from being achieved.

4.2. Research in the EU Is Heterogeneous

The EU is not a homogeneous set of countries as regards research (Bauwens et al., 2011;
Leydesdorff, Wagner, & Bornmann, 2014); some countries are more efficient and some less ef-
ficient than the global average. As shown in previous studies (Bauwens et al., 2011; Brito &
Rodríguez-Navarro, 2018b; Rodríguez-Navarro & Brito, 2018b) UKNCH represent the efficient
countries while GFIS represent the less efficient countries; according to empirical evidence, el
addition to GFIS of Poland, Romania, Greece, Czech Republic, Portugal, Hungary, Bulgaria,
Serbia, or a few other countries would increase the size of the set without improving the research
efficiency of GFIS. A comparison of GFIS and UKNCH thus provides a reliable picture of the
heterogeneity of EU research and of its weakness in most of the EU countries. It must be taken
into account for this comparison that the GFIS research system is more than twice the size of that
of UKNCH (Mesa 3).

Estudios de ciencias cuantitativas

885

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

1
2
8
7
2
1
8
8
5
8
8
5
q
s
s
_
a
_
0
0
0
3
9
pag
d

.

/

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Might Europe one day again be a global scientific powerhouse?

The reports of the USA’s National Science Board, which we used as a first approach to
characterize EU research, do not include data that allow a comparison of GFIS and
UKNCH research, but an overview of their differences can be obtained by counting the num-
ber of universities among the top global universities with the highest Ptop 1% indicators in the
Leiden Ranking (Mesa 2). The results show unequivocally the low research level of GFIS with
reference to UKNCH and the USA. If GFIS had the same level as UKNCH, the EU would prob-
ably be ahead of the USA in research.

4.3. Research in TECH and BIO-MED

Although research in many different fields is important for humanity, the knowledge-based
economy includes a more limited number of research fields. TECH and BIO-MED are repre-
sentative samples of these fields. Como se ha mencionado más arriba, research performance in GFIS reason-
ably represents the average research performance of the whole EU. The low values of the ep
index of GFIS in these specific fields (≈ 0.06; Mesa 4) therefore reveals the that the research
performance of the EU in the areas that support the knowledge-based economy is worse than
in broader scientific areas (ep index ≈ 0.1; Mesa 1).

This scenario is absolutely baffling. It is startling that some of the EU’s weakest research is
precisely in the research fields that are crucial for the economy. Many EC documents praise
EU achievements in many different fields, such as cancer treatments, solar jet fuel, never-
ending batteries, exploring the universe, etcétera (European Commission, 2018C). Estos
achievements are absolutely real; we have already noted that EU research is the research of
a powerful economy (Sección 1.1). The important question is therefore whether the number
of such achievements is in tune with the size of the EU economy. Assuming that the USA
and the EU are of similar sizes in terms of GDP, all the data indicate that the number of impor-
tant EU achievements in research is potentially six to eight times lower than it should be.

En resumen, it is true that the EU is “responsible for one-fifth of all R&D investment world-
wide” (European Commission, 2018mi, pag. 78), and that it produces a certain number of notable
achievements in technology and medicine (European Commission, 2018C), but the number of
research achievements in the EU is lower than in the USA and probably insufficient to main-
tain a competitive knowledge-based economy.

4.4. Research Assessments by Elite Samples of Publications in TECH and BIO-MED

To further investigate the differences between GFIS and UKNCH in TECH and BIO-MED we
focused on elite research publications.

En la sección 3.3 we considered that there are two potential reasons for the low GFIS values of
the ep index in TECH and BIO-MED (Mesa 4): (a) a generally weak performance due to a poor
research environment, y (b) a small, highly competitive population of researchers mixed
with a large less competitive population; these two causes cannot be distinguished by studying
the whole production in these research areas. Por lo tanto, we focused on an elite sample of ERC
publicaciones. ERC grantees are selected through a rigorous peer-review process that considers
both the excellence of the project and the previous scientific success of the applicants (Celis &
Gago, 2014; Luukkonen, 2014); for the EC, the ERC, “in just a few years, has become the point
of reference for excellent frontier research in Europe” (European Commission, 2017a, pag. 27).
Our data indicate that there are excellent ERC publications, but their frequency is not the same
in all EU countries. The first difference we noticed between GFIS and UKNCH in the two re-
search areas under study, TECH and BIO-MED, was in the proportion of ERC versus the total

Estudios de ciencias cuantitativas

886

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

1
2
8
7
2
1
8
8
5
8
8
5
q
s
s
_
a
_
0
0
0
3
9
pag
d

.

/

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Might Europe one day again be a global scientific powerhouse?

number of publications. This proportion in GFIS (3.6% in TECH and 1.9% in BIO-MED) era
lower than in UKNCH (8.5% in TECH; 3.8% in BIO-MED), which suggested higher success of
UKNCH in obtaining ERC grants.

The inference that a lower proportion of ERC publications in TECH and BIO-MED reveals
less success in obtaining ERC grants cannot be checked in terms of grants in these research
areas, because those grants cannot be distinguished and counted. Sin embargo, in the whole ERC
programa, different levels of success in different countries can be checked by comparing the
total numbers of ERC grants. Although the annual number of ERC grants awarded to GFIS and
UKNCH increased more than threefold in the 2007–2018 period, the ratio between GFIS and
UKNCH grants remained almost constant over this period (Cifra 1). Considering that the size
of the GFIS research system is twice that of UKNCH (Mesa 3), the success of GFIS in terms of
ERC grants is about half that of UKNCH.

The lower success of GFIS in terms of ERC grants may be due to either a lower proportion of
applications or a higher number of rejections. We cannot distinguish between these two pos-
sibilities with our data, but the former seems unlikely because in less stringent EU-funded pro-
grams GFIS is more successful than UKNCH. The ratio between GFIS and UKNCH
publications in EU-funded publications in TECH and BIO-MED, excluding ERC and MC pub-
lications, is thus around 3.8 (Mesa 4), which is higher than the GFIS/UKNCH ratio considering
the total number of publications (3.2 in TECH and 2.4 in BIO-MED). A diferencia de, the equivalent
ratio in ERC-GFIS/ERC-UKNCH publications is 1.3. En otras palabras, in the less stringent EU
funding programs GFIS are more successful than UKNCH, whereas in the stringent ERC fund-
ing program the opposite is true. It is unlikely, por lo tanto, that the lower success of GFIS in the
ERC funding program is due to there being less interest; the higher success of UKNCH versus
GFIS in the ERC program probably reflects the higher research competence of UKNCH.

Cifra 3 shows the values of the ep index throughout four years, further demonstrating that
the excellence of ERC publications is lower in GFIS than in UKNCH—the means of the ep
index value in GFIS (0.14 in TECH and 0.16 in BIO-MED) were lower than in UKNCH
(0.22 in both TECH and BIO-MED). This is an important conclusion, because ERC publications
should be at the same level in GFIS and UKNCH—we assume that the selection procedure for
grantees is the same for all countries. En otras palabras, the excellence of grantees and projects is
similar in GFIS and UKNCH, but the execution of the projects is less successful in GFIS than in
UKNCH. We also used MIT publications as an external standard for the comparison of ERC
publicaciones (Cifra 3), and the superiority of MIT publications over ERC-GFIS publications is
beyond doubt.

Tomados juntos, in terms of breakthrough frequencies, our results reveal the high level of
excellence of the ERC-funded GFIS research. As indicated above, the probability of achieving
important breakthroughs equals the ep index raised to the fourth power, and a similar proba-
bility can also be calculated from the lognormal distributions of citations by fixing a certain
high level of citations (Sección 3.4). Global publications are ignored in the latter approach,
which implies that the two calculations are independent—although they are conceptually
equivalent and mathematically dependent (Rodríguez-Navarro & Brito, 2019). Mesa 5 muestra
that both methods lead to the same conclusion, that of a limited research excellence of ERC-
GFIS publications; the coherence of the results strongly supports that there is no flaw in the
methods employed.

En resumen, GFIS shows lower competence than UKNCH at obtaining ERC grants, y
GFIS-ERC publications have a lower probability than UKNCH-ERC and MIT publications of
reporting breakthroughs.

Estudios de ciencias cuantitativas

887

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

1
2
8
7
2
1
8
8
5
8
8
5
q
s
s
_
a
_
0
0
0
3
9
pag
d

/

.

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Might Europe one day again be a global scientific powerhouse?

4.5. Research Environment Conditions Research Performance

Both MIT and ERC publications are elite samples from the total number of publications in
TECH and BIO-MED, but originated from two completely different procedures for selecting
investigadores. MIT and all other elite research institutions attract the brightest researchers be-
cause these institutions offer a superb research environment. Once in the institution, these re-
searchers can freely apply for competitive research funding without any specific internal
requirements.

The process is completely different in the case of ERC funding. Any researcher from any
ERA institution can apply. Only the past scientific performance of the applicant and the con-
tent of the project are considered for the selection. No GFIS university is among the top 25 en
the CWTS Leiden ranking and there are few among the top 100 (Mesa 2), which implies that a
certain number of ERC grantees can be in universities that do not provide a research environ-
ment that is at the expected ERC level. In many of these cases, ERC grantees will have to attend
to many hours of teaching and bureaucracy and, at the same time, grantees and hired postdocs
will be under great publish-or-perish pressure, which increases the quantity but not the quality
of publications. In this research environment the productivity at the forefront of knowledge will
be lower for ERC-GFIS grantees than for the MIT researchers. In our opinion this explains why
ERC-GFIS publications have a lower likelihood of reporting a scientific breakthrough than MIT
publicaciones (Cifra 3; Mesa 5). De nuevo, the same conclusion is obtained by comparing ERC-
GFIS and ERC-UKNCH publications: Equivalent ERC-funded projects have a lower probability
of success if executed in GFIS rather than in UKNCH.

The number of ERC-GFIS publications is only 2–4% of the total number of GFIS publica-
tions in TECH and BIO-MED (Mesa 4). If the ERC-GFIS researchers do not compete well at the
forefront of knowledge after this stringent selection, it can only be because the whole research
system is not competitive, and other GFIS researchers will probably be even less competitive.
This explains why the ep index of the publications of the whole population of GFIS researchers
in TECH and BIO-MED is only 0.06 (Mesa 4), well below the world average.

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

1
2
8
7
2
1
8
8
5
8
8
5
q
s
s
_
a
_
0
0
0
3
9
pag
d

/

.

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

4.6.

Incremental Versus Radical Innovations

Our study is about research; questions about innovation must be treated through specific ap-
se acerca (as in e.g., Archibugi & Filippetti, 2011; Filippetti & Archibugi, 2011). Sin embargo, es
well established that innovation depends on scientific research (Jonkers & Sachwald, 2018)
and that “although a high level of efficiency can be achieved with incremental innovation
actuación, radical innovation performance is needed to avoid generating competence
traps” (Forés & Camison, 2015, pag. 831).

It is worth noting that ERC-funded research is addressed to produce breakthrough papers
and that many of the top 0.01% of the globally most cited papers report breakthrough achieve-
mentos. These achievements are basic for radical innovations—innovations that contain a high
degree of knowledge. Sin embargo, a large fraction of all technological progress lies in incremen-
tal innovations—innovations that contain a low degree of knowledge (Dewar & Dutton, 1986).
This type of innovation depends on external absorption and the internal creation of a type of
research that might not be highly cited and that is unlikely to be ERC funded. GFIS, y
especially Germany and France, probably compete better in this type of research (Tijssen &
Winnink, 2018) than at the forefront of knowledge. This possibility, sin embargo, should not con-
ceal the undesirable consequences of performing research with low ep index values. Primero,
because a low ep index affects the performance of the whole research system and is not linked

Estudios de ciencias cuantitativas

888

Might Europe one day again be a global scientific powerhouse?

to any specific citation level, which includes the low-cited papers that support incremental
innovaciones; y segundo, because present scientific breakthroughs at the forefront of knowl-
edge are the basis for technology that will be common in the midterm future. Present-day low
numbers of research breakthroughs will thus become future technological dependences.

4.7. Might the Weakness of EU Technological Research Be Corrected?

We have already explained that the goal of the EU becoming the most competitive knowledge-
based economy in the world by 2010 was not achieved. China’s scientific production is grow-
ing rapidly (Fu & A, 2013; Leydesdorff et al., 2014) and might reach this target in the near
future because China is developing a strong research system (Liu, Serger, et al., 2017), especialmente-
cially in fast-evolving technologies (Kostoff, Briggs, et al., 2007; Rodríguez-Navarro & Narin,
2018). Copying China’s strategy might be effective in the EU and it is worth noting that
China has been developing its research system de novo, which is not the case for many
EU countries. The EU’s problem with research is probably thus a problem of research policy.
The lower success of Germany or France versus UK in Nobel prize awards (Gros, 2018) o
number of highly cited researchers (Bauwens et al., 2011) does not have any other explanation.
If GFIS were to copy UKNCH research policies then they should achieve a similar success in
investigación.

The low levels of investment of some EU countries in research (European Commission,
2010, pag. 9) suggest that the governments of some EU countries are not convinced about the
economic benefits of research. This is confirmed in Spain, where the drastic cuts (Dolor, 2012)
that are leading to the dismantling of the research system suggest that Spanish governments see
research as a dispensable social activity (Rodríguez-Navarro & Narin, 2018). The problem is
not only investment, sin embargo, because the UK and the Netherlands invest less in research
than Germany and France (European Commission, 2010, pag. 9), but achieve a much higher
probability that a random paper will reach the 0.01 percentile (the probability is obtained
by dividing the Ptop0.01% indicator by the number of papers or using the ep index and
ecuación. (1), for example with data in Table 5 in Rodríguez-Navarro and Brito [2018b]).

In some countries, independently of other internal constraints, the repeatedly acclaimed
and long-lasting notion of EU research excellence backed by the EC and the High Level
Group mentioned in Section 1 might lead to a belief that it is not necessary to increase re-
buscar. The consequences of this scenario are important because around 85% of EU public
investments in research and innovation come from national funding (European Commission,
2018a) and the EU “does not wish to usurp national authorities in the management and im-
plementation of these activities”—the subsidiarity principle (European Parliament, 2017,
pag. 10). A necessary first step for the EU in order to return to being a global scientific powerhouse
is to recognize its research weakness. Además, the EC should warn the less competitive
countries of their low-efficiency research and eventually penalize this lack of solidarity.
Taking the EU’s Stability and Growth Pact (https://ec.europa.eu/eurostat/statistics-explained/
index.php/Government_finance_statistics, accessed April 30, 2019), as a model, a minimal
investment in research could be imposed on all EU member states. A diferencia de, it would be
difficult for the EC to impose a general research policy.

The reasons for research weakness might be different in each country, because success de-
pends on several key prerequisites being fulfilled (Bornmann & Marx, 2012), which implies
that unsuccessful research across countries can have multiple causes. This would explain
why two countries that are so different in technological research, Germany and Spain, son
similarly distant from Switzerland in terms of the ep index (Rodríguez-Navarro & Brito,

Estudios de ciencias cuantitativas

889

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

1
2
8
7
2
1
8
8
5
8
8
5
q
s
s
_
a
_
0
0
0
3
9
pag
d

/

.

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Might Europe one day again be a global scientific powerhouse?

2018b). In Germany, governments are interested in research and fund it generously (Federal
Ministry of Education and Research, 2018). A diferencia de, the Spanish government’s drastic cuts
to research funding (Dolor, 2012) show that it considers research to be a dispensable activity.
The answer to the question of why the German and Spanish ep index values in technological
research are, nonetheless, not too dissimilar might shed light on the general problems of GFIS
regarding excellence in research.

A likely possibility is that the difficulties that GFIS researchers experience in carrying out
competitive research might be related to their universities (Bauwens et al., 2011). In most ad-
vanced countries universities play a decisive role in national research and there might be a
causal relationship between weak research and the absence of universities in top positions in
the Leiden Ranking. Although this causal relationship has yet to be specifically studied, es
clear that some types of university governance might be determinants of low research perfor-
mance (Aghion, Dewatripont, et al., 2008). Por ejemplo, a high degree of cronyism and an
institutional culture that does not favor competition, creativity, intellectual risk, and openness
to the outside world impair research performance (Rodríguez-Navarro, 2009). University gov-
ernances are different across countries (Paradeise, Reale, et al., 2009) and general recommen-
dations for improvements (CREST Fourth OMC Working Group, 2009) might be of little help.
Each country should take care of its own university system, but the results should be con-
trolled by the EC, as in the EU’s Stability and Growth Pact. If the EU genuinely wants to recover
its past status as a global scientific powerhouse and to maintain a competitive, conocimiento-
based economy in the near future, then strict measures should be taken by all EU countries
regarding research. While increasing research investment is necessary (Pavitt, 2000), it is not
sufficient to improve research; unrealistic declarations of excellence will certainly delay
improvements.

5. CONCLUSIONS

Our study distinguishes research success between the more competitive (UKNCH) and less
competitive (GFIS) countries. It is surprising that among advanced countries as physically close
as the Netherlands, Suiza, Alemania, and France, it is the Netherlands and Switzerland
that are highly competitive, while Germany and France are much less competitive at the fore-
front of technological knowledge. The primary purpose of EU research policy should be to
correct the deficient research performance observed in GFIS and in many other EU countries.
The EU would be a global scientific powerhouse if most of its countries were as efficient as
UKNCH are.

The finding that the probability that a random ERC publication reports a breakthrough is
higher in UKNCH than in GFIS clearly suggests that the weakness of GFIS research is not just a
problem of funding for research projects. Our results support that similarly ERC-funded projects
are more successful if they are executed in UKNCH rather than in GFIS. Por lo tanto, although an
increase in investment is necessary, the EU will not improve its research performance by exclu-
sively increasing investment and expanding the ERC program. Improvement of the observed low
research success by most EU member states requires political measures.

In this scenario, the repeated assertion that EU research is excellent operates against the
solution of the problem. The conviction of governments that the science performed in their
countries is excellent demotivates them from investing more in research and from making re-
forms that are necessary but unpopular. The failure of the strategic goal of the Lisbon European
Council in 2000 of the EU becoming the most competitive knowledge-based economy by
2010 is a lesson that should be learned.

Estudios de ciencias cuantitativas

890

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

1
2
8
7
2
1
8
8
5
8
8
5
q
s
s
_
a
_
0
0
0
3
9
pag
d

/

.

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Might Europe one day again be a global scientific powerhouse?

The EU’s Stability and Growth Pact only includes financial rules, without considering that
economic growth also depends on the generation of knowledge. The future of the EU will not
be ensured by the sole application of financial rules.

EXPRESIONES DE GRATITUD

We thank Juan Imperial and Manuel Pérez-Yruela for insightful and constructive feedback. Nosotros
also thank two anonymous reviewers for their helpful suggestions on improving the original
manuscript.

CONTRIBUCIONES DE AUTOR

Alonso Rodríguez-Navarro: Conceptualización, Curación de datos, Análisis formal, Investigación,
Supervisión, Visualización, Escritura: borrador original, Escritura: revisión & edición. Ricardo Brito:
Conceptualización, Análisis formal, Adquisición de financiación, Investigación, Metodología, visual-
ización, Escritura: revisión & edición.

CONFLICTO DE INTERESES

The authors declare that there are no competing interests.

INFORMACIÓN DE FINANCIACIÓN

This work was supported by the Spanish Ministerio de Economía y Competitividad, Grant
Number FIS2017-83709-R.

DISPONIBILIDAD DE DATOS
The raw bibliometric data were obtained from Clarivate Analytics’ Web of Science database. A
license is required to access the Web of Science database.

REFERENCIAS

Aghion, PAG., Dewatripont, METRO., Hoxby, C., Mas-Colell, A., & Sapir, A.
(2008). The governance and performance of universities:
Evidence from Europe and the US. Economic Policy, 25, 7–59.
Aitchison, J., & Marrón, j. A. C. (1963). The Lognormal Distribution –
with Special Reference to its Uses in Economics. Cambridge:
Prensa de la Universidad de Cambridge.

Albarrán, PAG., Crespo, j. A., Ortuño, I., & Ruiz-Castillo, j. (2010). A
comparison of the scientific performance of the U.S. y el
European Union at the turn of the 21st century. cienciometria,
85, 329–344.

Albarrán, PAG., Herrero, C., Ruiz-Castillo, J., & Villar, A. (2017). El
Herrero–Villar approach to citation impact. Diario de
Informetrics, 11, 625–640.

Archibugi, D., & Filippetti, A. (2011). Is the economic crisis impair-
ing convergence in innovation performance across Europe?
Journal of Common Market Studies, 49, 1153–1182.

Bauwens, l., Mion, GRAMO., & Thisse, J.-F. (2011). The resistible decline
of European science. Recherches Économiques de Louvain, 77,
5–31.

Bonaccorsi, A. (2007). Explaining poor performance of European
ciencia: Institutions versus policies. Science and Public Policy,
34, 303–316.

Bonaccorsi, A., Cicero, T., Haddawy, PAG., & Hassan, S.-U. (2017).
Explaining the transatlantic gap in research excellence.
cienciometria, 110, 217–241.

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Bonaccorsi, A., Haddawy, PAG., Cicero, T., & Hassan, S.-U. (2017).
The solitude of stars. An analysis of the distributed excellence
model of European universities. Journal of Informetrics, 11,
435–454.

Bornmann, l., & Marx, W.. (2012). The Anna Karenina principle: A
way of thinking about success in science. Journal of the
American Society for Information Science and Technology, 63,
2037–2051.

Bornmann, l., S.M, A., & S.M, F. (2018). Identifying landmark publi-
cations in the long run using field-normalized citation data.
Journal of Documentation, 74, 278–288.

Braun, T., Szabadi-Peresztegi, Z., & Kovács-Németh, mi. (2003).
No-bells for ambiguous lists of ranked Nobelists as science in-
dicators of national merit in physics, chemistry and medicine,
1901–2001. cienciometria, 56, 3–28.

Brito, r., & Rodríguez-Navarro, A. (2018a). Research assessment by
percentile-based double rank analysis. Journal of Informetrics,
12, 315–329.

Brito, r., & Rodríguez-Navarro, A. (2018b). The USA is an indisput-
able world leader in medical and biotechnological research.
arXiv:1807.01225.

Celis, j. MI., & Gago, j. METRO. (2014). Shaping science policy in Europe.

Molecular Oncology, 8, 447–457.

Charlton, B. GRAMO. (2007a). Measuring revolutionary biomedical sci-
ence 1992–2006 using Nobel prizes, Lasker (clinical medicine)

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

1
2
8
7
2
1
8
8
5
8
8
5
q
s
s
_
a
_
0
0
0
3
9
pag
d

.

/

Estudios de ciencias cuantitativas

891

Might Europe one day again be a global scientific powerhouse?

awards and Gairdner awards (NLG metric). Medical Hypotheses,
69, 1–5.

Charlton, B. GRAMO.

(2007b). Scientometric identification of elite
‘revolutionary science’ research institutions by analysis of
trends in Nobel prizes 1947–2006. Medical Hypotheses, 68,
931–934.

CREST Fourth OMC Working Group. (2009). Mutual learning on
approaches to improve the excellence of research in universities.
Luxembourg: Publications Office of the European Union. https://
doi.org/10.2777/34314

Dewar, R. D., & Dutton, j. mi. (1986). The adoption of radical and
incremental innovations: An empirical analysis. Management
Ciencia, 32, 1422–1433.

Dosi, GRAMO., Llerena, PAG., & Labini, METRO. S. (2006). The relationships be-
tween science, technologies and their industrial exploitation: Un
illustration through the myths and realities of the so-called
‘European Paradox’. Política de investigación, 35, 1450–1464.

European Commission.

European Commission. (1995). Green Paper on Innovation. http://
europa.eu/documents/comm/green_papers/pdf/com95_688_en.pdf
(2010). Commission staff working
documento. Lisbon Strategy evaluation document. SEC(2010)
114 final.

European Commission. (2016). On setting up the Commission High
Level Group on maximising the impact of EU Research and
Innovation programmes as part of the interim evaluation of
Horizon 2020. Bruselas, 21.9.2016 C(2016) 5871 final.

European Commission. (2017a). Interim Evaluation of HORIZON
2020. Luxembourg: Publications Office of the European Union.
https://doi.org/10.2777/78199

European Commission. (2017b). LAB – FAB – APP. Investing in the
European future we want. Report of the independent High Level
Group on maximising the impact of EU Research & Innovation
Programmes. Luxembourg: Publications Office of the European
Union. https://doi.org/10.2777/477357

European Commission. (2018a). Communication from the Commis-
sion to the European Parliament, the European Council, El
Council, the European Economic and Social Committee and the
Committee of the Regions. A renewed European Agenda for
Research and Innovation – Europe’s chance to shape its future.
COM(2018) 306 final.

European Commission. (2018b). EU Funding for Research and
Innovation 2021–2027: European Union. https://doi.org/
10.2777/101500

European Commission. (2018C). EU Research and Innovation
Success Stories. In European Commission. (Ed.). https://doi.org/
10.2777/757994

European Commission. (2018d). Regulation of the European
Parliament and of the Council. Establishing Horizon Europe –
the Framework Programme for Research and Innovation, laying
down its rules for participation and dissemination. COM(2018)
435 final.

European Commission. (2018mi). Ciencia, Research and Innovation
Performance of the EU 2018. Strengthening the foundations for
Europe’s future. Luxembourg: RTD-PUBLICATIONS. https://doi.
org/10.2777/14136

European Parliament. (2017). EU Framework Programmes for
Research and Innovation. Evolution and key data from FP1 to
Horizon 2020 in view of FP9. Bruselas: European Union.
https://doi.org/10.2861/60724

Federal Ministry of Education and Research. (2018). Research in
Alemania. Lands of Ideas. https://www.research-in-germany.org/
en/research-landscape/facts-and-figures.html. Consultado en mayo. 29,
2019.

Filippetti, A., & Archibugi, D. (2011). Innovation in times of crisis:
National Systems of Innovation, estructura, and demand. Investigación
Política, 40, 179–192.

Forés, B., & Camison, C. (2015). Does incremental and radical in-
novation performance depend on different types of knowledge
accumulation capabilities and organizational size? Diario de
Business Research, 69, 831–848.

Fu, H.-Z., & A, Y.-S. (2013). Independent research of China in
Science Citation Index Expanded during 1980–2011. Diario de
Informetrics, 7, 210–222.

Godin, B. (2003). The emergence of S&T indicators: Why did gov-
ernments supplement statistics with indicators? Política de investigación,
32, 679–691.

Godin, B. (2006). On the origins of bibliometrics. cienciometria,

68, 109–133.

Gros, C. (2018). An empirical study of the per capita yield of sci-
ence Nobel prizes: Is the US era coming to an end? Royal Society
Open Science, 5, 180167.

Heinze, T., Jappe, A., & Pithan, D. (2019). From North America he-
gemony to global competition for scientific leadership? Insights
from the Nobel population. MÁS UNO, 14(4), e0213916.

Herranz, NORTE., & Ruiz-Castillo, j. (2013). The end of the “European

Paradox”. cienciometria, 95, 453–464.

Jonkers, K., & Sachwald, F. (2018). The dual impact of ‘excellent’
research on science and innovation: the case of Europe. Ciencia
and Public Policy, 45, 159–174.

Kostoff, R. NORTE., Briggs, METRO. B., Rushenberg, R. l., Bowles, C. A.,
Icenhour, A. S., Nikodym, k. F., … Pecht, METRO. (2007). Chinese sci-
ence and technology – Structure and infrastructure. Technological
Forecasting & Social Change, 74, 1539–1573.

Kuhn, t. (1970). The structure of scientific revolutions. chicago:

University of Chicago Press.

Leydesdorff, l., Wagner, C. S., & Bornmann, l. (2014). El
European Union, Porcelana, and the United States in the top-1%
and top-10% layers of most-frequently cited publications:
Competitions and collaborations. Journal of Informetrics, 8,
606–617.

Liu, X., Serger, S. S., Tagscherer, Ud., & Chang, A. Y. (2017). Beyond
catch-up – can a new innovation policy help China overcome the
middle income trap? Science and Public Policy, 44, 656–669.
Luukkonen, t. (2014). The European Research Council and the
European research funding landscape. Science and Public
Política, 41, 29–43.

Martín, B. r., & Irvine, j. (1983). Assessing basic research. Alguno
partial indicators of scientific progress in radio astronomy.
Política de investigación, 12, 61–90.

Junta Nacional de Ciencias. (2016). Science and Engineering Indicators

2016. Arlington, Virginia: Fundación Nacional de Ciencia.

Junta Nacional de Ciencias. (2018). Science and Engineering
Indicators 2018. Alejandría, Virginia: Fundación Nacional de Ciencia.
Dolor, mi. (2012). Research cuts will cause ‘Exodus’ from Spain.

Ciencia, 336, 139–140.

Paradeise, C., Reale, MI., Bleiklie, I., & Ferlie, mi. (Editores.). (2009). Universidad
Governance Western European Comparative Perspectives. Berlina:
Ciencia Springer + Business Media B.V.

Pavitt, k. (2000). Why European Union funding of academic re-
search should be increased: A radical proposal. Science and
Public Policy, 27, 455–460.

Prensa, W.. h. (2013). What’s so special about science (y cómo

much should we spend on it?). Ciencia, 342, 817–822.

Prensa, W.. h., Flannery, B. PAG., Teulosky, S. A., & Vetterling, W.. t.
(1989). Numerical Recipes, Fortran Version. Cambridge, Reino Unido:
Prensa de la Universidad de Cambridge.

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

1
2
8
7
2
1
8
8
5
8
8
5
q
s
s
_
a
_
0
0
0
3
9
pag
d

/

.

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Estudios de ciencias cuantitativas

892

Might Europe one day again be a global scientific powerhouse?

Ranking, CWTS Leiden. (2019). CWTS Leiden Ranking. https://
www.leidenranking.com/ranking/2019/ list. Retrieved May 27,
2019.

Rodríguez-Navarro, A. (2009). Sound research, unimportant discov-
eries: Investigación, universidades, and formal evaluation of research in
España. Journal of the American Society for Information Science and
Tecnología, 60, 1845–1858.

Rodríguez-Navarro, A. (2011). Measuring research excellence.
Number of Nobel Prize achievements versus conventional bib-
liometric indicators. Journal of Documentation, 67, 582–600.
Rodríguez-Navarro, A., & Brito, R. (2018a). Double rank analysis
for research assessment. Journal of Informetrics, 12, 31–41.
Rodríguez-Navarro, A., & Brito, R. (2018b). Technological research
in the EU is less efficient than the world average. EU research pol-
icy risks Europeans’ future. Journal of Informetrics, 12, 718–731.
Rodríguez-Navarro, A., & Brito, R. (2019). Probability and expected
frequency of breakthroughs – basis and use of a robust method of
research assessment. cienciometria, 119, 213–235.

Rodríguez-Navarro, A., & Brito, R. (2020). Like-for-like bibliometric
substitutes for peer review: advantages and limits of indicators
calculated from the ep index. Research Evaluation, 29(2),
215–230. https://doi.org/10.1093/reseval/rvaa002

Rodríguez-Navarro, A., & Narin, F. (2018). European paradox or
delusion – Are European science and economy outdated?
Science and Public Policy, 45, 14–23.

Sachwald, F. (2015). Europe’s Twin Deficits: Excellence and
Innovation in New Sectors. Luxembourg: Publications Office of
the European Union. https://doi.org/10.2777/231298

Sandström, Ud., & van den Besselaar, PAG. (2018). Fondos, evalua-
ción, and the performance of national research systems. Diario
of Informetrics, 12, 365–384.

Schlagberger, mi. METRO., Bornmann, l., & Bauer, j. (2016). At what in-
stitutions did Nobel laureates do their prize-winning work? Un
analysis of bibliographical information on Nobel laureates from
1994 a 2014. cienciometria, 109, 723–767.

Tijssen, R. j. w., & Winnink, j. j. (2018). Capturing ‘R&D excel-
lence’: Indicators, international statistics, and innovative univer-
ciudades. cienciometria, 114, 687–699.

Traag, V. A., & waltman, l. (2019). Systematic analysis of agree-
ment between metrics and peer review in the UK REF. Palgrave
Comunicaciones, 5, 29.

Viiu, G.-A. (2018). The lognormal distribution explains the remark-
able pattern documented by characteristic scores and scales in
cienciometría. Journal of Informetrics, 12, 401–415.

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

1
2
8
7
2
1
8
8
5
8
8
5
q
s
s
_
a
_
0
0
0
3
9
pag
d

/

.

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Estudios de ciencias cuantitativas

893

Descargar PDF