RESEARCH ARTICLE
Noncumulative measures of
researcher citation impact
Mark C. Wilson
and Zhou Tang
Department of Mathematics and Statistics, University of Massachusetts Amherst
a n o p e n a c c e s s
j o u r n a l
Keywords: bibliometrics, citation indicator, scientometrics
Citation: Wilson, M. C., & Tang, Z.
(2020). Noncumulative measures of
researcher citation impact. Quantitative
Science Studies, 1(3), 1309–1320.
https://doi.org/10.1162/qss_a_00074
DOI:
https://doi.org/10.1162/qss_a_00074
Supporting Information:
https://doi.org/10.1162/qss_a_00074
Received: 22 Marzo 2020
Accepted: 23 May 2020
Corresponding Author:
Mark C. Wilson
markwilson@umass.edu
Handling Editor:
Ludo Waltman
Copyright: © 2020 Mark C. Wilson and
Zhou Tang. Published under a Creative
Commons Attribution 4.0 Internazionale
(CC BY 4.0) licenza.
The MIT Press
ABSTRACT
The most commonly used publication metrics for individual researchers are the total number of
publications, the total number of citations, and Hirsch’s h-index. Each of these is cumulative, E
hence increases throughout a researcher’s career, making it less suitable for evaluation of junior
researchers or assessing recent impact. Most other author-level measures in the literature share
this cumulative property. By contrast, we aim to study noncumulative measures that answer the
question “In terms of citation impact, what have you done lately?” We single out six measures
from the rather sparse literature, including Hirsch’s m-index, a time-scaled version of the h-index.
We introduce new measures based on the idea of “citation acceleration.” After presenting several
axioms for noncumulative measures, we conclude that one of our new measures has much better
theoretical justification. We present a small-scale study of its performance on real data and
conclude that it shows substantial promise for future use.
1.
INTRODUCTION
Despite strong opinion to the contrary among researchers, it is deemed necessary by bureaucrats
worldwide to use simple measures of researcher impact. Measures based on research publica-
zioni (mostly research monographs and peer-reviewed articles) are heavily used, the most com-
mon being the cumulative number of citations N(T), cumulative number of papers P(T), and the
h-index h(T) (Hirsch, 2005) (defined as the greatest integer h such that the author has at least
h papers each of which has at least h citations). All three quantities above are biased toward senior
scholars, being cumulative and therefore automatically increasing over time, even after the end of
the researcher’s career. Also, they provide information on overall career citation impact, but no
answer to “What have you done lately?” For many purposes it is not particularly useful to know
the h-index of Isaac Newton or total number of citations of Albert Einstein. Comparing researchers
near the start of their careers, comparing them with more senior researchers, or trying to predict
the future productivity and impact of a researcher clearly require different metrics.
Citation metrics that are not automatically increasing have received much less discussion in
the literature. Per esempio, the survey (Wildgaard, Schneider, & Larsen, 2014) Di 108 author-level
metrics contains at most 15 that are not automatically increasing and which attempt to measure
time-varying performance. The earlier survey (Bornmann, Mutz, et al., 2011) of variants of the
h-index included only six variants that attempted to adjust for career age, out of 37 indicators.
Ovviamente, nonincreasing measures intended to account for career age have been covered by
some researchers. Infatti, Hirsch in his original paper (Hirsch, 2005) devoted substantial analysis
to the rate of growth of h with the number of years t since the author’s first publication, and defined
the m-index by m(T) = h(T)/T. He calculated m for a selection of physicists (using a single fixed year,
l
D
o
w
N
o
UN
D
e
D
F
R
o
M
H
T
T
P
:
/
/
D
io
R
e
C
T
.
M
io
T
.
/
e
D
tu
q
S
S
/
UN
R
T
io
C
e
–
P
D
l
F
/
/
/
/
1
3
1
3
0
9
1
8
6
9
8
8
1
q
S
S
_
UN
_
0
0
0
7
4
P
D
/
.
F
B
sì
G
tu
e
S
T
T
o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
Noncumulative measures of researcher citation impact
presumably 2005) and concluded that a value of m around 1, 2, 3 correlated with his judgment of
distinguishing between “successful scientist,” “outstanding scientist,” and “truly unique” individ-
ual respectively. The m-index was immediately studied by others (Burrell, 2007; Liang, 2006)
but has been relatively little explored since. Another measure based on the h-index and attempt-
ing to measure recent performance is the contemporary h-index (Sidiropoulos, Katsaros, &
Manolopoulos, 2007). Other measures based on the h-index and attempting to adjust for career
age include the AR-index (Jin, Liang, et al., 2007), the square root of the sum over papers of
citations per year, restricted to publications in the h-core (the minimal set needed for computa-
tion of the h-index). The literature contains fewer nonincreasing measures not involving the
h-index. One could of course also look at the analog of the AR-index where all publications
are considered. We stop here, conscious that we must draw a line somewhere—given the axiom-
atic approach to be taken below, it already seems clear that most of the above measures will fail
many of the axioms.
1.1. Our Contribution
We argue that the “instantaneous rate of accumulation of citations owing to recent work” is the
relevant measure of recent citation productivity. We claim that this is precisely the “citation
acceleration,” the second time-derivative of the number of citations accumulated by an author.
As this quantity is not directly observable, we explore measures aimed at approximating it.
These measures are
w tð Þ : ¼
2N tð Þ
t2
W tð Þ : ¼ N tð Þ − 2N t − 1
ð
Þ
Þ þ N t − 2
ð
W5 tð Þ : ¼
1
7
ð
2N tð Þ − N t − 1
ð
ð
Þ − 2N t − 2
ð
Þ−N t − 3
Þ
Þ þ 2N t−4ð
Þ:
l
D
o
w
N
o
UN
D
e
D
F
R
o
M
H
T
T
P
:
/
/
D
io
R
e
C
T
.
M
io
T
.
/
e
D
tu
q
S
S
/
UN
R
T
io
C
e
–
P
D
l
F
/
/
/
/
1
3
1
3
0
9
1
8
6
9
8
8
1
q
S
S
_
UN
_
0
0
0
7
4
P
D
.
/
We single out six existing measures from the literature that are noncumulative and intended
to adjust for career age. These are
(cid:129) Hirsch’s (Hirsch, 2005) m-index defined by
m tð Þ : ¼ h tð Þ=t
(cid:129) Mannella and Rossi’s (2013) measure
α1 tð Þ : ¼
h tð Þ
P
ffiffiffiffiffiffiffiffiffi
N tð Þ
F
B
sì
G
tu
e
S
T
T
o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
(cid:129) The contemporary h-index hc(T) (Sidiropoulos et al., 2007), defined below.
(cid:129) The trend h-index ht(T) (Sidiropoulos et al., 2007), defined below.
(cid:129) The age-weighted citation rate A(T), defined below.
(cid:129) The average number of citations per year μ(T), defined below.
In Section 3 we evaluate all nine citation measures against axiomatic criteria. The difference
between “theoretical” and “empirical” work in bibliometrics has been well described by
Quantitative Science Studies
1310
Noncumulative measures of researcher citation impact
Waltman and Van Eck (2012). Our approach here is grounded in a theoretical analysis. IL
axiomatic approach to bibliometric indicators has been applied to the h-index by several authors
following the initial paper (Woeginger, 2008)—we single out Quesada (2011) and Bouyssou and
Marchant (2013). To our knowledge, axiomatics have been applied to few other indicators. Noi
single out Waltman and van Eck (2009) and Bouyssou and Marchant (2014), which gives axiom-
atic characterizations of several well-known indicators such as N and P.
To check that we have not strayed too far from reality, we evaluate the performance of the
theoretically best measures W and W5 on several small data sets of mathematical researchers.
The results, as shown in Section 4, are promising with respect to predictive value and correlation
with expert judgment.
1.2. Definition of Remaining Measures
The age-weighted citation rate A(T) is obtained by summing over all publications the average
number of citations per year, evaluated at time t. The measure μ(T) is simply the average number
of citations per year, evaluated at time t.
The contemporary h-index is defined analogously to the h-index, where citations gradually
lose value. The contemporary h-index uses an adjusted weight of citation, and requires the
researcher to have published at least h papers, each with adjusted citation weight to be at least
H. The adjusted citation weight of a paper published in year t0 and having C citations up to year t is
given by
γ t− t0 þ 1
ð
Þ−δ
C;
Dove (cid:2) E (cid:3) are fixed positive constants. Per esempio, if (cid:3) = (cid:2) = 1, the adjusted citation weight of
a paper is the number of citations per year since the year before its publication.
The trend h-index is defined using similar reasoning to the contemporary h-index, except
that the weight depends on the year of citation, not of publication. Each citing paper in year t
contributes a weight of
γ t − t0 þ 1
ð
Þ−δ:
2. BACKGROUND AND DEFINITIONS
Consider a researcher emitting research publications starting from time t = 0. These publica-
tions accumulate citations at a certain rate, dependent both on such factors as the number of
publications, the size of the research field, the citation practices of the field, and the attrac-
tiveness of the papers to other researchers. As mentioned above, the most commonly used
metrics are
(cid:129) P(T), the number of publications up to time t;
(cid:129) N(T), the number of citations up to time t; E
(cid:129) H(T), IL (Hirsch) h-index at time t.
Note that these each increase in t, even after the end of the researcher’s career.
2.1. The Simple Citation Model
For definiteness we measure time in years and t = 0 corresponds to the date of the first publication.
We first consider what we call the “simple model” in continuous time. In questo caso, a researcher
Quantitative Science Studies
1311
l
D
o
w
N
o
UN
D
e
D
F
R
o
M
H
T
T
P
:
/
/
D
io
R
e
C
T
.
M
io
T
.
/
e
D
tu
q
S
S
/
UN
R
T
io
C
e
–
P
D
l
F
/
/
/
/
1
3
1
3
0
9
1
8
6
9
8
8
1
q
S
S
_
UN
_
0
0
0
7
4
P
D
.
/
F
B
sì
G
tu
e
S
T
T
o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
Noncumulative measures of researcher citation impact
publishes at a constant rate of p papers per unit time and each paper attracts citations at a rate of
c per paper per unit time, forever. A discrete version of this model was used by Hirsch (2005).
Then the number of publications by time t is P(T) = pt, and the total number of citations is
N tð Þ ¼
Z
T
0
pcs ds ¼ pct
2=2:
The “acceleration” in citation accumulation is therefore
00
N
tð Þ ¼ pc:
(1)
(2)
Our main idea is that the quantity pc measures the instantaneous accumulation of citations
from new work. This “acceleration” is a key measure of recent productivity and citation impact.
For a given research field, larger values indicate researchers with greater recent impact.
The instantaneous citation acceleration cannot be measured directly owing to the discreteness
of available publication data, so we introduce the quantity
δ
W
tð Þ ¼ N tð Þ − 2N t − δ
δ2
ð
Þ
Þ þ N t − 2δ
ð
;
which is the usual backward difference approximation to the second derivative. When (cid:2) = 1 year,
this reduces to the measure W defined above.
We also consider the measure W5, obtained by fitting quadratics to N through successive win-
dows of five data points (separated by 1-year intervals) and approximating the second derivative
on each. This is an example of a Savitzky-Golay filter, widely used to smooth discrete data of this
type. In fact W5 is the simplest such filter, and we could define W2k+1 analogously for k > 2 by
using larger numbers of points.
The measure w is clearly constant in the simple model, with value pc. So is W
(cid:2)
, as shown by
δ
W
tð Þ ¼ pc
(cid:3)
2 T
2 −2 t − δ
ð
Þ2 þ t − 2δ
ð
Þ2
(cid:4)
¼ pc:
So is W5, because the filter used is exact for quadratics and hence reproduces the (constant)
second derivative.
As explained by Hirsch (2005), in this model we may assume that the papers in the h-core
at time t have been published up to time s ≤ t, so that ps = h(T), while the number of citations of
the least cited element of the h-core is c(t − s) = h(T). This leads immediately to
so that
h tð Þ ¼
pc
p þ c
T;
m tð Þ ¼
pc
p þ c
:
As an aside, note that this equals the harmonic mean of p and c.
From above it follows that in the simple model
so that
ð
N tð Þ ¼ p þ c
2pc
Þ2
h tð Þ2;
h tð Þ ¼ β
P
ffiffiffiffiffiffiffiffiffi
;
N tð Þ
(3)
(4)
1312
Quantitative Science Studies
l
D
o
w
N
o
UN
D
e
D
F
R
o
M
H
T
T
P
:
/
/
D
io
R
e
C
T
.
M
io
T
.
/
e
D
tu
q
S
S
/
UN
R
T
io
C
e
–
P
D
l
F
/
/
/
/
1
3
1
3
0
9
1
8
6
9
8
8
1
q
S
S
_
UN
_
0
0
0
7
4
P
D
.
/
F
B
sì
G
tu
e
S
T
T
o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
Noncumulative measures of researcher citation impact
P
ffiffiffiffiffiffi
2pc
p þ c
Dove (cid:4) =
we have
:This is the reasoning behind the definition of (cid:5)
1, and clearly in the simple model
α1 tð Þ ¼
P
ffiffiffiffiffiffiffiffi
2pc
p þ c
:
Thus in the simple model several measures are constant. Tuttavia, not all constants are equally
valid. The units of acceleration should be citation/(year)2, and indeed w, W, W5 have these units.
However the units of m and (cid:5)
1 are inconsistent with this requirement.
The value of μ(T) is clearly pct/2. The age-weighted citation measure A(T) equals pct in this model,
because in Eq. 1 we divide the integrand by s. The contemporary h-index and trend h-index are
harder to compute, but are also nonconstant unless (cid:2) = 1 (we use the method shown above for the
h-index, and omit the details here).
2.2. A Model Incorporating Retirement
We introduce another simple model in order to analyze the prediction value and behavior of
measures when a researcher stops publishing papers due to any reason. In the second model
we suppose that publications in the simple model stop after time T. In questo caso, the number
of citations is N(T ) = pcT2
2 , while for t > T we have an additional pcT(t − T ) citations. Thus if
t ≥ T,
N tð Þ ¼ pcT t−T=2
ð
Þ:
(5)
Direct computation shows that the measure W takes the value zero provided T ≥ t + 2, so that
sufficient time has elapsed for measurements to be taken. Tuttavia, the other measures above
take on nonzero values when t > T. We see that w has the value pcT(2t − T )/t2. For example if
t = 2T, so the total time elapsed after retirement is as long as the researcher’s entire career, w has
reduced by only 25%, to 3pc/4, from its previous constant value pc during the career.
Hirsch’s original argument shows that the h-index has the same value pct/( P + C) in the second
model until time t = T(1 + p/c) as it had in the simple model, but that after this time it has value pT/t.
È interessante notare, this is independent of c. The measure (cid:5)
when t = 2T. We do
1 takes the value
q
ffiffiffiffiffiffi
P
3cT
not compute the details of the contemporary and trend h-index here, because we can see enough
to rule them out axiomatically below.
It is of course possible to explore more complicated models, but the focus of this article is
the introduction of a new measure with axiomatic justification, so we now proceed to that.
3. AXIOMS
As mentioned above, the number of possible measures is enormous, but without axiomatic foun-
dations, it seems pointless to study them in detail. We now present several axioms for measures
intended to describe noncumulative citation impact. All except the last seem to us to be
uncontroversial.
(cid:129) Computability: The measure should be easily computable from citation counts, paper
conta, and academic age.
(cid:129) Units: The units of the measure should be (citation)/(time)2.
(cid:129) Locality: If no citations are gained during a time interval, the measure is zero during that
interval.
Quantitative Science Studies
1313
l
D
o
w
N
o
UN
D
e
D
F
R
o
M
H
T
T
P
:
/
/
D
io
R
e
C
T
.
M
io
T
.
/
e
D
tu
q
S
S
/
UN
R
T
io
C
e
–
P
D
l
F
/
/
/
/
1
3
1
3
0
9
1
8
6
9
8
8
1
q
S
S
_
UN
_
0
0
0
7
4
P
D
.
/
F
B
sì
G
tu
e
S
T
T
o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
Noncumulative measures of researcher citation impact
(cid:129) Constancy: In the first model, the measure is constant.
(cid:129) End of career: In the second model, the measure is zero for t > T.
(cid:129) Packaging-independence: The measure should not depend on P: It is computable only
from citation counts and academic age.
The Packaging-independence axiom requires more explanation. We argue that the impact
of a researcher with 10 papers each attracting 100 citations is the same as if all 10 papers had
been combined into a book that receives 1,000 citations. The packaging into publications may
in practice affect the number of citations in a more complicated way, but if publications have
no overlap and citations reflect intellectual influence only, this should not occur. Ovviamente,
this is also a strong argument against using the h-index.
The Locality axiom may need to be interpreted slightly differently when time is discrete. For
esempio, the measure W satisfies this axiom provided the point chosen is 2 years past the left
endpoint of the interval in question.
Tavolo 1 shows the performance of the abovementioned measures against these axiomatic criteria.
Clearly, W and W5 perform much better than the others, and we consider only these measures in
the next section. Note that hc requires specification of two free parameters before we can even
compute it, and we know of no principled way to do that, hence the failure of the Computability
axiom. The other axioms were evaluated for arbitrary (cid:3) E (cid:2), and give the same answer for all
choices of these constants (with an exception for the constancy axiom, as noted above).
Based on the discussion above, we move on to consider only W(T) and W5(T), which seem
the most promising measures.
4. CASE STUDY OF MATHEMATICAL RESEARCHERS
We now discuss the performance of the measures on real data.
4.1. Methods
The experiments described below were carried out in February 2020. All raw data and com-
puted data are publicly available (Wilson, 2020).
We concentrated for this article on the research area most familiar to us, namely mathematics.
We compiled several data sets based on specific sets of researchers. The lack of availability of this
data in an open format, or even in a proprietary one that allows for comprehensive analysis, is a
major factor in hampering studies of this type. We resorted to making many time-intensive manual
Tavolo 1.
Performance of measures with respect to axioms
w
W
W5
M
Measure
(cid:5)
1
hc
ht
UN
μ
Axiom
Computability
Units
Locality
Constancy
End of career
Packaging-independence
Quantitative Science Studies
1314
l
D
o
w
N
o
UN
D
e
D
F
R
o
M
H
T
T
P
:
/
/
D
io
R
e
C
T
.
M
io
T
.
/
e
D
tu
q
S
S
/
UN
R
T
io
C
e
–
P
D
l
F
/
/
/
/
1
3
1
3
0
9
1
8
6
9
8
8
1
q
S
S
_
UN
_
0
0
0
7
4
P
D
.
/
F
B
sì
G
tu
e
S
T
T
o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
Noncumulative measures of researcher citation impact
web-based queries. To test our hypotheses, for each researcher we require the total number of
citations to his or her works in each calendar year. We used Web of Science Core Collection,
which we chose because of its availability and reasonably wide coverage.
We extracted four sets of mathematicians and categorized them. The first set used consisted of all
Abel Prize winners still living at May 1, 2018 (with the exception of two for whom author name
disambiguation was too difficult) and included 17 authors. The second consisted of 10 mathemati-
cians from a single department (University of Massachusetts Amherst, Mathematics & Statistics;
UMass) with interests in a common subfield (algebraic geometry). The third data set consisted of
10 authors generated “randomly” from MathSciNet (we chose authors of the most recent papers
in algebraic geometry according to MathSciNet). The fourth consisted of all living winners of the
Fields Medal from 2006 A 2018 inclusive, and consisted of 13 authors. This gave 50 mathematical
researchers. For each researcher we took year 1 to be the first year t for which N(T) ≥ 10.
Larger data sets would give more confidence in the results below, but they are clear enough to
show that the axiomatically well-founded measures W and W5 measure something about a
researcher that enables us to distinguish between randomly chosen, successful and outstanding
researchers.
4.2. Results
4.2.1. Variation of Measures
In Figure 1 we graph, for three Abel and three Fields prizewinners, W(T) and W5(T) for t from
5 years after career start (defined as the first year for which N exceeds 10). As can be seen,
there is substantial variation over time for each author. In Table 2 we give the mean and stan-
dard variation of the values of the measures over the same time period.
l
D
o
w
N
o
UN
D
e
D
F
R
o
M
H
T
T
P
:
/
/
D
io
R
e
C
T
.
M
io
T
.
/
e
D
tu
q
S
S
/
UN
R
T
io
C
e
–
P
D
l
F
/
/
/
/
1
3
1
3
0
9
1
8
6
9
8
8
1
q
S
S
_
UN
_
0
0
0
7
4
P
D
.
/
F
B
sì
G
tu
e
S
T
T
o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
Figura 1. Variation over time of values of W and W5 for three selected Abel and Fields prizewinners.
Quantitative Science Studies
1315
Noncumulative measures of researcher citation impact
Tavolo 2. Mean and standard deviation for W and W5 for Abel prizewinners
Serre, JP
Atiyah, MF
Singer, IM
Lax, PD
Carleson, l.
Varadhan, SRS
Tits, J
Tate, JT
Milnor, J
Szemeredi, E
Deligne, P
Sinai, YG
Nirenberg, l
Wiles, UN.
Meyer, Y
Langlands, RP
Uhlenbeck, K
mean(W )
1.00
9.27
1.91
5.91
1.45
2.45
1.09
1.45
3.91
2.45
0.09
0.55
3.36
2.73
9.64
0.82
2.82
sd(W )
4.05
34.88
4.21
19.11
8.49
27.89
7.81
6.36
11.08
7.94
2.39
2.46
8.71
8.43
12.17
5.42
18.70
mean(W5)
−0.36
10.73
sd(W5)
2.04
14.95
0.56
5.14
1.14
6.10
0.90
−0.43
3.64
2.19
−0.08
0.17
2.95
1.09
5.94
0.99
5.78
1.20
2.94
2.67
5.85
2.01
2.27
5.99
1.24
0.87
0.60
1.86
2.23
5.63
1.21
5.45
l
D
o
w
N
o
UN
D
e
D
F
R
o
M
H
T
T
P
:
/
/
D
io
R
e
C
T
.
M
io
T
.
/
e
D
tu
q
S
S
/
UN
R
T
io
C
e
–
P
D
l
F
/
/
/
/
1
3
1
3
0
9
1
8
6
9
8
8
1
q
S
S
_
UN
_
0
0
0
7
4
P
D
/
.
F
B
sì
G
tu
e
S
T
T
o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
4.2.2. Predictive Value of Measures
As explained by Penner, Pan, et al. (2013), cumulative increasing measures such as the h-index
contain intrinsic autocorrelation, which vastly overstates their predictive power. They find that
the actual ability of the h-index to predict future citations from future publications is rather low.
In our case, we are dealing with noncumulative measures, whose predictive power is not so clear.
The results of Section 4.2.1 show considerable variation in the year-to-year values of W(T) and W5(T),
so the idea of the simple model, that these measures are constant and hence precisely determine
something intrinsic to the researcher, is not plausible. We do not expect to be able to predict the
value of W(T + 5), Per esempio, from W(T) only. Tuttavia, the results in Section 4.2.3 show that gross
distinctions between researchers at different levels of impact can be made (when dealing with
researchers in the same fairly narrowly defined field), and these seem to mean something.
To obtain a better idea of predictive power, for the union of our data sets we computed the mean
in years 3–5 of career of W, and used this to attempt to predict the mean in years 6–8 of the same
measure. The ordinary least squares linear regression results are displayed in Figure 2 (note that the
extreme outlier Terence Tao was removed from the data set, as was the very young Peter Scholze,
in partenza 48 researchers). The value R2 = 0.748 shows a high level of predictive power. Note that the
definition of W means that we are trying to predict a linear combination of N(8), N(7), N(5), and N(4)
from a linear combination (with the same coefficients) of N(5), N(4), N(2), and N(1), and there is no
reason a priori to expect this to have such a high coefficient of determination.
Quantitative Science Studies
1316
Noncumulative measures of researcher citation impact
l
D
o
w
N
o
UN
D
e
D
F
R
o
M
H
T
T
P
:
/
/
D
io
R
e
C
T
.
M
io
T
.
Figura 2. Regression of mean of W in years 4–6 against mean in years 1–3.
4.2.3. Relation to Expert Judgment
We expect that the citation acceleration for randomly chosen authors should be lower overall than
that of the UMass researchers. Also, we expect the Fields data set and Abel data set to have higher
values of the measures than the UMass data set. Given the age of the members of the Abel data set,
we expect a small advantage to the Fields data set. All our expectations are borne out by the results
shown in Figure 3.
/
e
D
tu
q
S
S
/
UN
R
T
io
C
e
–
P
D
l
F
/
/
/
/
1
3
1
3
0
9
1
8
6
9
8
8
1
q
S
S
_
UN
_
0
0
0
7
4
P
D
/
.
F
B
sì
G
tu
e
S
T
T
o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
Figura 3. Scatterplot of values of W for four different data sets.
Quantitative Science Studies
1317
Noncumulative measures of researcher citation impact
5. DISCUSSION
5.1. Limitations
Before summarizing our main points, we acknowledge that the entire subject of bibliometrics is
suspect in the eyes of many researchers, because of its perceived misapplication by unimagina-
tive bureaucrats having influence over researcher careers. We share these concerns, and the
present article was motivated by the idea that if we researchers are forced to be evaluated by
simple metrics, we can at least have some agency in their design. The popularity of the h-index,
Per esempio, is very mysterious to us, given its weak theoretical foundations (Waltman & Van Eck,
2012). Any one measure can be strategically manipulated, but we hope that by use of sufficiently
many metrics with low theoretical correlation, incentives for researchers to act in ways that are
not helpful to science overall will be reduced.
All of the citation measures in the literature are susceptible to many problems (including miss-
ing data, author name disambiguation, negative citations, contributions of multiple authors,
citation inflation owing to growth in number of researchers). Also, there is the problem of normal-
ization across different fields (Per esempio, one citation in mathematics corresponds roughly to 19
in physics and 78 in biomedicine [Podlubny, 2005]). This question of “field” is a difficult one—it
is obvious that certain areas of mathematics have communities of different sizes, leading to
substantial variation in the number of citations across areas. Così, as usual, there is no completely
automated substitute for human judgment.
5.2. Positive Outcomes
The index W(T) introduced in this article seems to measure something specific to a researcher that is
related to his or her recent productivity and impact, and seems promising as a way to make coarse
distinctions between researchers in the same field who may be at different career stages. It behaves
well with respect to natural axioms. It seems fairly well correlated with subjective measures of
research impact or quality. It is less sensitive to the way in which ideas are packaged into individ-
ual publications, and considerably easier to compute, than the m-index (under the assumption that
splitting a paper splits the citations in the obvious way, the m index discourages extreme “salami-
slicing,” whereas W is indifferent to it and P encourages it).
Insofar as citation metrics are to be increasingly used for evaluation of researchers and espe-
cially for allocation of resources to them, the W-index provides another useful (perhaps the single
most useful found so far) measure of recent publication activity leading to citation impact, and one
that has decent predictive value.
5.3. Future Work
We are grateful to an anonymous referee for informing us about two papers that deal with the issue
of discrete data in bibliometrics (Liu & Rousseau, 2012, 2014). The idea used in those papers, A
approximate (in their case the citations to a single research article) by a continuous cubic spline,
could also be used in the situation of the present paper. Per esempio, we could approximate N(T)
by a cubic spline and then estimate the second derivative by taking the second derivative of the
spline. The measure so derived would satisfy all our axioms. We do not expect substantially dif-
ferent results from using interpolation rather than filtering as we have done, but it may be worth
further investigation.
Bouyssou and Marchant (2014) state that their paper explicitly does not deal with any
indicators intended to adjust for career age, and the last part of the paper suggests further work
in such a direction. We offer the present work as an initial contribution, and intend to follow
Quantitative Science Studies
1318
l
D
o
w
N
o
UN
D
e
D
F
R
o
M
H
T
T
P
:
/
/
D
io
R
e
C
T
.
M
io
T
.
/
e
D
tu
q
S
S
/
UN
R
T
io
C
e
–
P
D
l
F
/
/
/
/
1
3
1
3
0
9
1
8
6
9
8
8
1
q
S
S
_
UN
_
0
0
0
7
4
P
D
/
.
F
B
sì
G
tu
e
S
T
T
o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
Noncumulative measures of researcher citation impact
su. A stream of research initiated by Woeginger (2008) deals with axiomatic characterizations
of the h-index—that is, a set of axioms that, taken together, uniquely determine the h-index.
Our experience with characterization theorems (and also impossibility theorems where “too
many” axioms are chosen, a prominent approach in social choice theory, Per esempio) is that
very often the axiom systems consist of a few innocuous assumptions and one that is much less
intuitive and essentially encodes the desired result. Nevertheless, it would be interesting to
obtain an axiomatic characterization of our measure W, Per esempio.
The relation of Eq. 4 derived from the simple model has wider validity than might be expected
at first sight. Mannella and Rossi (2013) find via a study of 1,400 Italian physicists that this quad-
ratic relationship holds well on real data, and empirically find the best fit value of (cid:4) = 0.53 In
Eq. 4, agreeing with the rough calculations of Hirsch based on a smaller data set of physicists.
Yong (2014) showed analytically, based on theory of random partitions, that a very good estimate
should be (cid:4) =
ln 2/π ≈ 0.54. He also demonstrated the accuracy of this approximation on a
small data set of prominent mathematicians.
ffiffiffi
6
P
Tuttavia, Mannella and Rossi also showed that the time-scaled index
α2 tð Þ ¼ h tð Þ
P
ffiffi
T
(6)
is approximately time independent on their data set. This implies that the number of citations is
better described as linear rather than quadratic, which is clearly inconsistent with the simple
modello. Attempts to estimate citation acceleration depend on the quadratic growth of citations
with time, so a possible linear relationship will have an effect on the measures used in this paper:
In that case, the acceleration would be identically zero. We feel that the growth of citations by an
author with time should be studied seriously on much larger data sets than we have treated here.
To concentrate on the main concept of citation acceleration, we have omitted more subtle
issues, such as weights for coauthored papers and normalization by the size of the research field.
These of course could be explored.
l
D
o
w
N
o
UN
D
e
D
F
R
o
M
H
T
T
P
:
/
/
D
io
R
e
C
T
.
M
io
T
.
/
e
D
tu
q
S
S
/
UN
R
T
io
C
e
–
P
D
l
F
/
/
/
/
1
3
1
3
0
9
1
8
6
9
8
8
1
q
S
S
_
UN
_
0
0
0
7
4
P
D
.
/
ACKNOWLEDGMENTS
We thank Hooman Alavizadeh for preliminary discussions and Thierry Marchant for feedback
on a draft of this article. We thank the editor and referees of this journal for their constructive
comments on the initial submission.
AUTHOR CONTRIBUTIONS
Mark Wilson: Conceptualization, Formal analysis, Investigation, Methodology, Project adminis-
tration, Supervision, Writing—original draft, Writing—review & editing. Zhou Tang: Formal
analysis, Software, Visualization.
F
B
sì
G
tu
e
S
T
T
o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
COMPETING INTERESTS
The authors have no competing interests.
FUNDING INFORMATION
This project received no funding.
Quantitative Science Studies
1319
Noncumulative measures of researcher citation impact
DATA AVAILABILITY
Data are available from Harvard Dataverse.
REFERENCES
Bornmann, L., Mutz, R., Hug, S. E., & Daniel, H.-D. (2011). A multi-
level meta-analysis of studies reporting correlations between the h
index and 37 different h index variants. Journal of Informetrics, 5(3),
346–359.
Bouyssou, D., & Marchant, T. (2013). An interpretable axiomatiza-
tion of the Hirsch-index. In 14th ISSI Conference (pag. 2024–2026).
Leuven: International Society for Scientometrics and Informetrics.
Bouyssou, D., & Marchant, T. (2014). An axiomatic approach to biblio-
metric rankings and indices. Journal of Informetrics, 8(3), 449–477.
Burrell, Q. (2007). Hirsch index or Hirsch rate? Some thoughts
arising from Liang’s data. Scientometrics, 73(1), 19–28.
Hirsch, J. E. (2005). An index to quantify an individual’s scientific
research output. Proceedings of the National Academy of Sciences,
102(46), 16569–16572.
Jin, B., Liang, L., Rousseau, R., & Egghe, l. (2007). The R-and
AR-indices: Complementing the h-index. Chinese Science Bulletin,
52(6), 855–863.
Liang, l. (2006). h-index sequence and h-index matrix: Constructions
e applicazioni. Scientometrics, 69(1), 153–159.
Liu, Y., & Rousseau, R. (2012). Towards a representation of diffusion
and interaction of scientific ideas: The case of fiber optics commu-
nication. Information Processing & Management, 48(4), 791–801.
Liu, Y., & Rousseau, R. (2014). Citation analysis and the develop-
ment of science: A case study using articles by some Nobel prize
winners. Journal of the Association for Information Science and
Tecnologia, 65(2), 281–289.
Mannella, R., & Rossi, P. (2013). On the time dependence of the
h-index. Journal of Informetrics, 7(1), 176–182.
Penner, O., Pan, R. K., Petersen, UN. M., Kaski, K., & Fortunato, S.
(2013). On the predictability of future impact in science. Scientific
Reports, 3(1), 1–8.
Podlubny, IO. (2005). Comparison of scientific impact expressed by the
number of citations in different fields of science. Scientometrics,
64(1), 95–99.
Quesada, UN. (2011). Further characterizations of the Hirsch index.
Scientometrics, 87(1), 107–114.
Sidiropoulos, A., Katsaros, D., & Manolopoulos, Y. (2007).
Generalized Hirsch h-index for disclosing latent facts in citation
networks. Scientometrics, 72(2), 253–280.
Waltman, L., & van Eck, N. J. (2009). A taxonomy of bibliometric
performance indicators based on the property of consistency
(Tech. Rep. No. ERS-2009-014-LIS). Rotterdam: Erasmus
Università.
Waltman, L., & Van Eck, N. J. (2012). The inconsistency of the h-index.
Journal of the American Society for Information Science and
Tecnologia, 63(2), 406–415.
Wildgaard, L., Schneider, J. W., & Larsen, B. (2014). A review of
the characteristics of 108 author-level bibliometric indicators.
Scientometrics, 101(1), 125–158.
Wilson, M. C. (2020). Replication Data for: Non-cumulative measures
of researcher citation impact. Harvard Dataverse. https://doi.org/
10.7910/DVN/SQE29Z
Woeginger, G. J. (2008). An axiomatic characterization of the
Hirsch-index. Mathematical Social Sciences, 56(2), 224–232.
Yong, UN. (2014). Critique of Hirsch’s citation index: A combinatorial
Fermi problem. Notices of the AMS, 61(9), 1040–1050.
l
D
o
w
N
o
UN
D
e
D
F
R
o
M
H
T
T
P
:
/
/
D
io
R
e
C
T
.
M
io
T
.
/
e
D
tu
q
S
S
/
UN
R
T
io
C
e
–
P
D
l
F
/
/
/
/
1
3
1
3
0
9
1
8
6
9
8
8
1
q
S
S
_
UN
_
0
0
0
7
4
P
D
.
/
F
B
sì
G
tu
e
S
T
T
o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
Quantitative Science Studies
1320
