RESEARCH ARTICLE

Inferring the causal effect of journals on citations

V. A. Traag

Centre for Science and Technology Studies (CWTS), 莱顿大学, 荷兰人

关键词: Bayesian model, causal inference, citations, journal effects, science of science

开放访问

杂志

抽象的

引文: Traag, V. A. (2021). Inferring
the causal effect of journals on
citations. Quantitative Science Studies,
2(2), 496–504. https://doi.org/10.1162
/qss_a_00128

DOI:
https://doi.org/10.1162/qss_a_00128

Peer Review:
https://publons.com/publon/10.1162
/qss_a_00128

支持信息:
https://doi.org/10.1162/qss_a_00128

已收到: 6 十一月 2020
公认: 10 一月 2021

通讯作者:
V. A. Traag
v.a.traag@cwts.leidenuniv.nl

处理编辑器:
Staša Milojević

版权: © 2021 V. A. Traag.
在知识共享下发布
归因 4.0 国际的
(抄送 4.0) 执照.

Articles in high-impact journals are, 一般, more frequently cited. But are they cited more
often because those articles are somehow more “citable”? Or are they cited more often simply
because they are published in a high-impact journal? Although some evidence suggests the
后者, the causal relationship is not clear. We here compare citations of preprints to citations of
the published version to uncover the causal mechanism. We build on an earlier model of citation
dynamics to infer the causal effect of journals on citations. We find that high-impact journals
select articles that tend to attract more citations. 同时, we find that high-impact
journals augment the citation rate of published articles. Our results yield a deeper understanding
of the role of journals in the research system. The use of journal metrics in research evaluation
has been increasingly criticized in recent years and article-level citations are sometimes
suggested as an alternative. Our results show that removing impact factors from evaluation does
not negate the influence of journals. This insight has important implications for changing
practices of research evaluation.

介绍

Journals play a central role in scholarly communication, yet their role is also contested. The jour-
nal impact factor in particular has been criticized on several accounts (Larivière & Sugimoto,
2019). The main critique is its pervasive use in the context of research evaluation, 例如
in tenure decisions (McKiernan, Schimanski et al., 2019). Scientists shape their research with
impact factors in mind (穆勒 & de Rijcke, 2017; Rushforth & de Rijcke, 2015). In a meeting
in San Francisco in 2012, cell biologists called for a ban on the impact factor from research eval-
uation, and conjoined the “San Francisco Declaration on Research Assessment”1 (DORA). A
group of researchers and editors called for publishing entire citation distributions instead of
impact factors, to counter inappropriate use (Larivière, Kiermer et al., 2016). 最近, A
group of editors and researchers came together and called for “rethinking impact factors”
(Wouters, Sugimoto et al., 2019).

同时, journal impact is one of the most clear predictors of future citations
(Abramo, D’Angelo, & Felici, 2019; Callaham, 2002; 莱维特 & Thelwall, 2011; Stegehuis,
Litvak, & Waltman, 2015). The question is why. Possibly, high-impact journals select articles
that somehow tend to be cited frequently. Another possibility is that articles are cited more fre-
quently because they are published in a high-impact journal, not because they tend to be cited
frequently per se. Neither citations of an article nor the journal in which it is published needs to

麻省理工学院出版社

1 https://sfdora.org

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你
q
s
s
/
A
r
t
我
C
e
–
p
d

我

F
/

2
2
4
9
6
1
9
3
0
6
8
5
q
s
s
_
A
_
0
0
1
2
8
p
d

乙
y
G
你
e
s
t

哦
n
0
7
S
e
p
e
米
乙
e
r
2
0
2
3

Inferring the causal effect of journals on citations

be representative of “quality.” Here, we simply study whether citations of an article are influ-
enced by the journal in which it is published, not their relationship to “quality.”

Answering this question is not straightforward. In rare cases, publications appear in multiple
journals, and researchers found that the version in a higher impact journal was more frequently
cited than its twin in a lower impact journal (Cantrill, 2016; Larivière & Gingras, 2010; Perneger,
2010). 然而, duplicate publications are quite special, limiting the generalizability of this
观察. Some other earlier work claimed that citations were not affected by the journal
(Seglen, 1994).

We answer this question by comparing citations of preprints with citations of the published
version. The number of citations C may be influenced by both the latent citation rate (西德:1) 和
journal J in which the article is published (数字 1). Possibly, high-impact journals perform a
stringent peer review of articles, selecting only articles with a high latent citation rate, so that
(西德:1) influences the journal J. The latent citation rate itself may be influenced by many factors
and characteristics (Onodera & Yoshikane, 2015) and motivations for citing the paper
(Bornmann & Daniel, 2008). These factors are not limited to the characteristics of the paper itself,
but may also include author reputation (彼得森, Fortunato et al., 2014) or institutional reputa-
的 (Medoff, 2006). Regardless of which factors influence the latent citation rate, 的数量
0
citations of the preprint before it is published in a journal C
is unaffected by where it will be
published and is affected only by the latent citation rate (西德:1). We rely on this insight to estimate
the causal effect of the journal on citations Pr(C | 做( J )). The identification of the causal effect is
possible because of the so-called “effect restoration” (Kuroki & Pearl, 2014), provided we can
0
estimate Pr(C

| (西德:1)). We construct a parametric model that provides exactly such an estimate.

2. METHODOLOGY

We gathered information about 1,341,016 preprints from arXiv, and identified the published
version for 727,186 preprints (54%; see Supplementary Material for more details). We extracted
citations of both the preprint version and the published version from references in Scopus.
Preprint dates, publication dates, and citation dates are all extracted from Crossref, using a daily
granularity. We used the major subject headings of arXiv as field definitions. The impact of
journals is calculated as the average number of citations received in the first 5 years after
publication for all research articles and reviews in Scopus. We perform our analysis per year
(2000–2016) and field, as the journal effect may vary per year and field. 而且, we restrict

数字 1. Simple causal model of the confounding effect of the latent citation rate (西德:1) of an article
0
being published in a journal J and the citations it accrues C. 相比之下, citations of preprints C
是
affected by the latent citation rate (西德:1) 仅有的. The selection bias on arXiv preprints A does not bias the
causal effect of J on C once (西德:1) is controlled for. The time before publication T
affects preprint cita-
0
tions C

and complicates the analysis.

Quantitative Science Studies

497

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你
q
s
s
/
A
r
t
我
C
e
–
p
d

我

F
/

2
2
4
9
6
1
9
3
0
6
8
5
q
s
s
_
A
_
0
0
1
2
8
p
d

乙
y
G
你
e
s
t

哦
n
0
7
S
e
p
e
米
乙
e
r
2
0
2
3

Inferring the causal effect of journals on citations

our analysis to journals that have at least 20 articles that were published at least 30 days after
appearing as a preprint on arXiv (Figure S1). 清楚地, our data has a selection bias (Bareinboim &
Pearl, 2012) on papers being submitted to arXiv or not (A). 然而, we can show that this does
not affect our estimate of the causal effect Pr(C | 做( J )) (see Supplementary Material).

Time complicates our analysis. The time T

before a preprint was published, the preprint
0
duration, will clearly affect the number of prepublication citations C
, while the total time since
publication T will affect the postpublication citations C. Preprints with a higher latent citation
rate may perhaps be more quickly published, thus affecting T
. To tackle this problem, we model
the full temporal dynamics of both pre- and postpublication citations.

Citation dynamics are influenced by a wide range of factors, such as a rich-get-richer effect
and a clear temporal decay (Fortunato, Bergstrom et al., 2018), but was captured reasonably well
by a recent model by Wang, 歌曲, and Barabási (2013). We build on that model and include a
parameter that modulates the citation rate based on where the article is published. We assume
that the number of citations ci (t) that article i receives at time t is distributed as

ci tð Þ (西德:2) Poisson λi tð Þfi tð Þ m þ Ci t − 1
磷

Þ
Þ

(西德:3);

(1)

τ¼0 ci ((西德:3)) the cumulative number of citations, 和
with effective citation rate λi(t) and Ci (t) =
t
m a parameter affecting the initial citation accumulation. The temporal decay of the accumu-
lation of citations is captured by fi (t), which is modeled by an exponential distribution, 和
inverse rate (西德:4)
我. We assume that preprint i attracts citations at an effective rate of (西德:1)
i is
the latent citation rate of article i. The published version attracts citations at an effective rate
的 (西德:1)
我
Ji is the journal citation multiplier for journal Ji in which article i is published.
我
We equate θ
j with the causal effect on citations of publishing in journal j, which is identical
for all articles published in journal j, regardless of the characteristics of those papers. We call
Ci
) the postpublication citations.
The expected number of long-term citations is about

) the prepublication citations and Ci = Ci (Ti) − Ci (Ti

吉, where θ

我, 在哪里 (西德:1)

= Ci (Ti

(西德:2)
m e

(西德:1)
我

θJi − 1

(西德:3)
;

(2)

assuming prepublication citations are negligible (see Supplementary Material).

The selection of articles by peer review is assumed to lead to a distribution of latent citation

rates for journal j,

(西德:2) LogNormal Φj; (西德:5)j

(西德:2)

(西德:3)

(西德:1)
我

(3)

如果 (西德:1)j is high, journal j will tend to publish articles of higher latent citation rates (西德:1)我. The median
(西德:1)j. 有效地, this is a Bayesian hierarchical model, 和我们
latent citation rate of journal j is e
specify informed prior distributions based on earlier results (王等人。, 2013) (see Supplementary
Material for full details and analysis of the model). We illustrate the model in Figure 2.

3. 结果

The numbers of pre- and postpublication citations are not clearly related (数字 3, panel A).
The numbers of prepublication citations also do not clearly relate to journal impact (数字 3,
panel B). The relation between preprint duration and the number of prepublication citations is
also not clear (数字 3, panel C). The ratio of postpublication citations and prepublication
citations is higher for high-impact journals (数字 3, panel D). Articles in high-impact journals
accumulate more postpublication citations relative to prepublication citations compared to
articles that have appeared in lower impact journals. These results are possibly obfuscated by
two counteracting effects: Higher latent citation rates lead to higher prepublication citations,

Quantitative Science Studies

498

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你
q
s
s
/
A
r
t
我
C
e
–
p
d

我

F
/

2
2
4
9
6
1
9
3
0
6
8
5
q
s
s
_
A
_
0
0
1
2
8
p
d

乙
y
G
你
e
s
t

哦
n
0
7
S
e
p
e
米
乙
e
r
2
0
2
3

Inferring the causal effect of journals on citations

Illustration of citation dynamics. This example, astro-ph/0405353, was first submit-
数字 2.
ted to arXiv in 2004 and was published in Journal of Cosmology and Astroparticle Physics almost
4 多年后 (Ti
= 1,385). It was cited 33 times before it was published (Ci
= 33), 和 29 times after
it was published (Ci = 29). We assume citations are attracted at a rate of (西德:1)
i before it was published
and at a rate of (西德:1)
我
Ji after it was published. The thick solid line represents the empirically observed
number of citations. The thin lines in the background represent samples from the posterior predictive
distribution of our model.

我

but perhaps also to shorter preprint durations, reducing the time to attract prepublication
citations. The model that we constructed is intended to address this issue.

We here report results from our model for the five largest fields and the publication year 2016.
Other fields and years show qualitatively similar results (see Figures S2 and S4). Our model pre-
sents a good fit of both pre- and postpublication citations (Figure S5).

The journal citation multiplier is consistently higher than 1 (数字 4, panel A). Publishing in
journals, compared to being available on arXiv only, multiplies the citation rate substantially, 作为
预期的. 例如, Nature shows a multiplier of 6.0–9.9 (95% CI) for papers published in
2016 in the subject of Condensed Matter and Science shows a multiplier of 7.5–12.0 (95% CI)
for such papers. Using the median estimates and the approximation in Eq. 2, this implies that a

数字 3.

Impact versus pre- and postpublication citations.

Quantitative Science Studies

499

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你
q
s
s
/
A
r
t
我
C
e
–
p
d

我

F
/

2
2
4
9
6
1
9
3
0
6
8
5
q
s
s
_
A
_
0
0
1
2
8
p
d

乙
y
G
你
e
s
t

哦
n
0
7
S
e
p
e
米
乙
e
r
2
0
2
3

Inferring the causal effect of journals on citations

Condensed Matter article published in Nature in 2016 that obtained about 200 citations would
not have obtained even 10 citations had it been available on arXiv only. Had it been published in
Science instead, it would have obtained almost 350 citations. This is only an illustration: 两个都
parameter estimates and the citation dynamics themselves exhibit considerable uncertainty
(see Supplementary Material).

Most relevant to our question, higher impact journals tend to show higher citation multipliers.
The correlation between the (logarithm of ) the journal impact and the (logarithm of ) the median
journal citation multiplier θ
j is on average 0.45 for each combination of field and year. It ranges
从 0.063 for High Energy Physics in 2002 到 0.79 for Astrophysics in 2012. 有趣的是, 这
correlation grows stronger for High Energy Physics and Astrophysics over time, hovering around
0.6–0.7 for recent years (Figure S3).

(西德:1)

同时, the median latent citation rate e

j is also clearly increasing with journal
impact (数字 4, panel B). 例如, the U.S.-based Physical Review Letters has a relatively
high journal impact and shows a latent citation rate of 0.15–0.17 (95% CI) for Condensed Matter
在 2016. Its lower impact European counterpart Europhysics Letters shows a latent citation rate of
0.013–0.027 (95% CI) in the same field and year. 全面的, the correlation between the (loga-
rithm of ) the journal impact and (西德:1)
j is on average 0.54 for each combination of field and year.
For High Energy Physics in 2002 the correlation is 0.72, while for Astrophysics in 2012 这
correlation is 0.050. The highest correlation of 0.85 is observed for Astrophysics in 2006. 这
correlation grows weaker for High Energy Physics and Astrophysics over time (Figure S3). 这
median effective citation rate of a journal is e
j, which aligns closely with the observed journal
impact (Figure S6).

(西德:1)jθ

The latent citation rates also vary within journals, and are controlled by (西德:5)

j. Journals with a
更高 (西德:5)
j tend to publish articles with a larger variety of latent citation rates. 例如,
Europhysics Letters shows an (西德:5)j of 0.7–1.1 (95% CI), while Science shows an (西德:5)j of 0.2–0.3
(95% CI), resulting in a broader distribution of (西德:1)
i for Europhysics Letters than Science. 一般来说,
high-impact journals show more narrow distributions of latent citation rates than lower impact
journals (数字 4, panel C).

数字 4. Posterior results for model of citation dynamics for five largest fields and publication year 2016. Error bars represent the average
95% 可信区间. Highlighted journals indicate results in the field of Condensed Matter.

Quantitative Science Studies

500

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你
q
s
s
/
A
r
t
我
C
e
–
p
d

我

F
/

2
2
4
9
6
1
9
3
0
6
8
5
q
s
s
_
A
_
0
0
1
2
8
p
d

乙
y
G
你
e
s
t

哦
n
0
7
S
e
p
e
米
乙
e
r
2
0
2
3

Inferring the causal effect of journals on citations

4. 讨论

Why articles in high-impact journals attract more citations is a fundamental question. 我们有
provided clear evidence that articles in high-impact journals are highly cited because of two
effects. 一方面, articles that attract more citations are more likely to be published in
high-impact journals. 另一方面, articles in high-impact journals will be cited even more
frequently because of the publication venue. This amplifies the cumulative advantage effect for
citations (Price, 1976).

A recent publication (Kim, Portenoy et al., 2020) took a similar approach and compared
citations of preprints with citations of the published version. Using a more rudimentary model
they obtained similar results and also find an influence of the journal on citations, 虽然他们
do not address the causal mechanism. They also find that preprints with more citations are more
likely to be published, but do not analyze in what journals they are published.

Several mechanisms may play a role in the causal effect of journals on citations. High-impact
journals tend to have a higher circulation (Peritz, 1995), and reach a wider audience. 此外,
researchers may prefer to cite an article from a high-impact journal over an article from a low-
impact journal, even if both articles would be equally fitting. Both mechanisms are consistent
with our results and earlier results (Cantrill, 2016; Kim et al., 2020; Larivière & Gingras, 2010;
Perneger, 2010). Distinguishing between these two causal mechanisms is difficult (戴维斯, 2010)
and should be investigated further.

An alternative explanation may be that published preprints are more highly cited because the
preprints were improved by high-quality peer review in high-impact journals. We deem this an
unlikely scenario. Differences between the preprint and the published version are textually
minor (克莱因, Broadwell et al., 2016). Those modifications can of course be substantively impor-
坦特. Peer review may substantially improve and strengthen a manuscript. 尽管如此, we think
it is unlikely to alter a paper’s core contribution so as to affect its citation rate considerably.

Our analysis is limited to mostly physics and mathematics because of our reliance on arXiv.
We expect to see similar effects in the medical sciences and the social sciences, in line with
earlier results (Cantrill, 2016; Larivière & Gingras, 2010; Perneger, 2010). It would be interesting
to replicate our analysis on younger preprint repositories, such as bioRxiv or SocArxiv, once they
have had more time to accumulate citations. Another limitation is that we considered references
from published articles only. It would be interesting to include also the references of preprints.
This presumably increases the number of prepublication citations (Larivière, Sugimoto et al.,
2014), which may decrease the overall inferred journal causal effect.

In our model we assumed that the effect of publishing in a journal is identical for all articles
published in that journal. 然而, the effect of publishing in a journal may possibly vary for
different articles. 例如, articles from well-known authors may be cited frequently regard-
less of the exact journal in which they are published, while articles from more junior authors may
benefit more from publishing in high-impact journals. Teasing out these different effects is not
直截了当, but presents an interesting avenue for future research.

The latent citation rate itself may be influenced by many factors and characteristics of the
纸 (Onodera & Yoshikane, 2015) and motivations for citing the paper (Bornmann &
Daniel, 2008). 全面的, our results suggest that the characteristics (X1, X2, …… ) that drive citations
(C ) overlap or correlate with factors that drive journal ( J ) peer review (数字 5). 例如,
新奇, 关联, and scientific breadth (X2 to X4) may affect both journal evaluation and
citations directly, while methodological aspects affect journal evaluation (X1) and authors’
reputation (X5) only affects citations. Because the journal also affects citations, methodological

Quantitative Science Studies

501

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你
q
s
s
/
A
r
t
我
C
e
–
p
d

我

F
/

2
2
4
9
6
1
9
3
0
6
8
5
q
s
s
_
A
_
0
0
1
2
8
p
d

乙
y
G
你
e
s
t

哦
n
0
7
S
e
p
e
米
乙
e
r
2
0
2
3

Inferring the causal effect of journals on citations

数字 5. Causal model of factors and characteristics X1, X2, ……, journals J, citations C, and eval-
uation E.

aspects would have an indirect effect on citations in this example. What factors drive journal
evaluation and what factors drive citations is not clear and should be further investigated.

We hypothesize that a subset of factors that are used in journal evaluation are also used in
postpublication research evaluation, such as the UK REF (Traag & Waltman, 2019). This means
that research evaluation (乙 ) tends to correlate with journals ( J ) because of underlying common
因素 (数字 5). Even if factors that influence research evaluation do not influence citations
直接地, they will still correlate because of the mediating effect of the journal. 例如, 如果
methodological aspects (X1) affect research evaluation (乙 ), it would correlate with citations (C )
only because methodological aspects affect the journal ( J ). If our hypothesis holds true, citations
would be indicative of the evaluation of articles only because they were published in a particular
journal. In that case, citations should not be normalized based on the journal in which they are
published, as was attempted by Zitt, Ramanana-Rahary, and Bassecoulard (2005). Doing so
would effectively control for the journal, thereby blocking these causal pathways. 的确,
Adams, Gurney, and Jackson (2008) find that journal-normalized citations do not correlate with
评估. 相似地, Eyre-Walker and Stoletzki (2013) report an absence of various correlations
with evaluations when controlling for the journal. These results provide some evidence for
我们的假设. Journal metrics might even be a more appropriate indicator than citations to in-
dividual articles, as was suggested by Waltman and Traag (2020), although our results neither
affirm nor refute this possibility.

Possibly, evaluation itself is also affected directly by the journal in which an article is pub-
列出, and depending on the context, perhaps also by its citations. 的确, the proposed causal
diagram only captures part of a larger web of entanglement.

The use of citations and journals in research evaluation is often debated. Removing the use of
journal metrics from research evaluation, as for example advocated by DORA, may decrease the
pressure on authors to publish in high-impact journals. The use of article-level citations for evalu-
ation could be condoned by DORA, but the use of journal metrics could not. Even if journal metrics
were to be removed from research evaluation, journals would continue to play a role in research
评估, 尽管是间接的. Evaluating researchers based on citations then may still reward
authors who publish in high-impact journals. This may effectively exert selective pressures that
drive the evolution of the research system (Smaldino & McElreath, 2016). Simply removing
impact factors from research evaluation therefore does not negate the influence of journals.

致谢

I thank Rodrigo Costas, Ludo Waltman, Jesper Schneider, and other colleagues from CWTS.
I gratefully acknowledge use of the Shark cluster of the LUMC for computation time.

Quantitative Science Studies

502

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你
q
s
s
/
A
r
t
我
C
e
–
p
d

我

F
/

2
2
4
9
6
1
9
3
0
6
8
5
q
s
s
_
A
_
0
0
1
2
8
p
d

乙
y
G
你
e
s
t

哦
n
0
7
S
e
p
e
米
乙
e
r
2
0
2
3

Inferring the causal effect of journals on citations

COMPETING INTERESTS

The author has no competing interests.

DATA AVAILABILITY

All data necessary to reproduce the results in this analysis is available from Traag (2020A) 和
all source code is available from Traag (2020乙).

参考

Abramo, G。, D’Angelo, C. A。, & Felici, G. (2019). Predicting publi-
cation long-term impact through a combination of early citations
and journal impact factor. Journal of Informetrics, 13, 32–49. DOI:
https://doi.org/10.1016/j.joi.2018.11.003

Adams, J。, Gurney, K., & Jackson, L. (2008). Calibrating the zoom—
A test of Zitt’s hypothesis. Scientometrics, 75, 81–95. DOI: https://
doi.org/10.1007/s11192-007-1832-7

Bareinboim, E., & Pearl, J. (2012). Controlling selection bias in
causal inference. In International Conference on Artificial
Intelligence and Statistics, 卷. 22, (PP. 100–108).

Bornmann, L。, & Daniel, H. (2008). What do citation counts
措施? A review of studies on citing behavior. 杂志
Documentation, 64, 45–80. DOI: https://doi.org/10.1108
/00220410810844150

Callaham, 中号. (2002). Journal prestige, publication bias, 和别的
characteristics associated with citation of published studies in
peer-reviewed journals. JAMA, 287, 2847. DOI: https://doi.org
/10.1001/jama.287.21.2847, PMID: 12038930

Cantrill, S. (2016). Imperfect impact. Chemical connections blog.
一月 23. https://stuartcantrill.com/2016/01/23/imperfect
-impact/

戴维斯, 磷. (2010). Impact factors – A self-fulfilling prophecy? 这
Scholarly Kitchen blog, 六月 9. https://scholarlykitchen.sspnet
.org/2010/06/09/impact-factors-a-self-fulfilling-prophecy/.

Eyre-Walker, A。, & Stoletzki, 氮. (2013). The assessment of science:
The relative merits of post-publication review, the impact factor,
and the number of citations. PLOS Biology, 11, e1001675. DOI:
https://doi.org/10.1371/journal.pbio.1001675, PMID:
24115908, PMCID: PMC3792863

Fortunato, S。, 伯格斯特罗姆, C. T。, Börner, K., 埃文斯, J. A。, Helbing, D .,
… Barabási, A.-L. (2018). Science of science. 科学, 359. DOI:
https://doi.org/10.1126/science.aao0185, PMID: 29496846,
PMCID: PMC5949209

Kim, L。, Portenoy, J. H。, 西方, J. D ., & Stovel, K. 瓦. (2020).
Scientific journals still matter in the era of academic search
engines and preprint archives. Journal of the Association for
Information Science and Technology, 71, 1218–1226. DOI:
https://doi.org/10.1002/asi.24326

克莱因, M。, Broadwell, P。, Farb, S. E., & Grappone, 时间. (2016).
Comparing published scientific journal articles to their pre-print
versions. In Proceedings of the 16th ACM/ IEEE-CS on Joint
Conference on Digital Libraries – JCDL ’16 (PP. 153–162).
纽约: ACM Press. DOI: https://doi.org/10.1145/2910896
.2910909

Kuroki, M。, & Pearl, J. (2014). Measurement bias and effect restora-
tion in causal inference. Biometrika, 101, 423–437. DOI: https://
doi.org/10.1093/biomet/ast066

Larivière, 五、, & Gingras, 是. (2010). The impact factor’s Matthew
Effect: A natural experiment in bibliometrics. Journal of the
American Society for Information Science and Technology, 61,
424–427. DOI: https://doi.org/10.1002/asi.21232

Larivière, 五、, Kiermer, 五、, MacCallum, C. J。, McNutt, M。, 帕特森,
M。, … Curry, S. (2016). A simple proposal for the publication of
journal citation distributions. bioRxiv, 062109. DOI: https://土井
.org/10.1101/062109

Larivière, 五、, & Sugimoto, C. 右. (2019). The journal impact factor:
A brief history, critique, and discussion of adverse effects. 在
瓦. Glänzel, H. F. Moed, U. Schmoch, & 中号. Thelwall (编辑。),
Springer handbook of science and technology indicators
(PP. 3–24). 占婆: 施普林格. DOI: https://doi.org/10.1007/978-3
-030-02511-3_1

Larivière, 五、, Sugimoto, C. R。, Macaluso, B., Milojevic(西德:3), S。, 克朗的,
B., & Thelwall, 中号. (2014). arXiv E-prints and the journal of
record: An analysis of roles and relationships. Journal of the
Association for Information Science and Technology, 65,
1157–1169. DOI: https://doi.org/10.1002/asi.23044

莱维特, J. M。, & Thelwall, 中号. (2011). A combined bibliometric indi-
cator to predict article impact. Information Processing and
管理, 47, 300–308. DOI: https://doi.org/10.1016/j.ipm
.2010.09.005

McKiernan, 乙. C。, Schimanski, L. A。, Muñoz Nieves, C。, 马蒂亚斯,
L。, Niles, 中号. T。, & Alperin, J. 磷. (2019). Use of the Journal Impact
Factor in academic review, promotion, and tenure evaluations.
电子生活, 8, e47388. DOI: https://doi.org/10.7554/eLife.47338,
PMID: 31364991, PMCID: PMC6668985

Medoff, 中号. H. (2006). Evidence of a Harvard and Chicago Matthew
Effect. Journal of Economic Methodology, 13, 485–506. DOI:
https://doi.org/10.1080/13501780601049079

穆勒, R。, & de Rijcke, S. (2017). Thinking with indicators.
Exploring the epistemic impacts of academic performance indi-
cators in the life sciences. Research Evaluation, 26, 157–168.
DOI: https://doi.org/10.1093/reseval/rvx023

Onodera, N。, & Yoshikane, F. (2015). Factors affecting citation rates
of research articles. Journal of the Association for Information
Science and Technology, 66, 739–764. DOI: https://doi.org/10
.1002/asi.23209

Peritz, 乙. C. (1995). On the association between journal circulation
and impact factor. Journal of Information Science, 21, 63–67.
DOI: https://doi.org/10.1177/016555159502100108

Perneger, 时间. V. (2010). Citation analysis of identical consensus
statements revealed journal-related bias. Journal of Clinical
流行病学, 63, 660–664. DOI: https://doi.org/10.1016/j
.jclinepi.2009.09.012, PMID: 20097531

彼得森, A. M。, Fortunato, S。, Pan, 右. K., Kaski, K., Penner, 奥。, ……
Pammolli, F.
in academic
(2014). Reputation and impact
careers. 美国国家科学院院刊, 111,
15316–15321. DOI: https://doi.org/10.1073/pnas.1323111111,
PMID: 25288774, PMCID: PMC4217436

Price, D. D. S. (1976). A general theory of bibliometric and other
cumulative advantage processes. Journal of the American Society
for Information Science, 27, 292–306. DOI: https://doi.org/10
.1002/asi.4630270505

Quantitative Science Studies

503

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你
q
s
s
/
A
r
t
我
C
e
–
p
d

我

F
/

2
2
4
9
6
1
9
3
0
6
8
5
q
s
s
_
A
_
0
0
1
2
8
p
d

乙
y
G
你
e
s
t

哦
n
0
7
S
e
p
e
米
乙
e
r
2
0
2
3

Inferring the causal effect of journals on citations

Rushforth, A。, & de Rijcke, S. (2015). Accounting for impact? 这
Journal Impact Factor and the making of biomedical research in the
荷兰. Minerva, 53, 117–139. DOI: https://doi.org/10.1007
/s11024-015-9274-5, PMID: 26097258, PMCID: PMC4469321
Seglen, 磷. 氧. (1994). Causal relationship between article citedness
and journal impact. Journal of the American Society for Information
科学, 45, 1–11. DOI: https://doi.org/10.1002/(SICI)1097-4571
(199401)45:1<1::AID-ASI1>3.0.一氧化碳;2-是

Smaldino, 磷. E., & McElreath, 右. (2016). The natural selection of bad
科学. Royal Society Open Science, 3, 160384. DOI: https://土井
.org/10.1098/rsos.160384, PMID: 27703703, PMCID:
PMC5043322

Stegehuis, C。, Litvak, N。, & Waltman, L. (2015). Predicting the long-
term citation impact of recent publications. Journal of Informetrics,
9, 642–657. DOI: https://doi.org/10.1016/j.joi.2015.06.005
Traag, V. A. (2020A). Replication data: Inferring the causal effect of
journals on citations. Zenodo. https://zenodo.org/record/3582974
Traag, V. A. (2020乙), Replication source code: Inferring the causal
effect of journals on citations. Zenodo. https://zenodo.org/record
/3583012

Traag, V. A。, & Waltman, L. (2019). Systematic analysis of agreement
between metrics and peer review in the UK REF. Palgrave
通讯, 5, 29. DOI: https://doi.org/10.1057/s41599-019
-0233-X

Waltman, L。, & Traag, V. A. (2020). Use of the journal impact factor for
assessing individual articles need not be wrong. F1000Research, 9.
DOI: https://doi.org/10.12688/f1000research.23418.1

王, D ., 歌曲, C。, & 巴拉巴斯, A.-L. (2013). Quantifying long-
term scientific impact. 科学, 342, 127–132. DOI: https://土井
.org/10.1126/science.1237825, PMID: 24092745

Wouters, P。, Sugimoto, C. R。, Larivière, 五、, McVeigh, 中号. E.,
Pulverer, B., … Waltman, L.
fac-
托尔斯: Better ways to judge a journal. 自然, 569, 621–623.
DOI: https://doi.org/10.1038/d41586-019-01643-3, PMID:
31138924

(2019). Rethinking impact

Zitt, M。, Ramanana-Rahary, S。, & Bassecoulard, 乙. (2005).
Relativity of citation performance and excellence measures:
From cross-field to cross-scale effects of field-normalisation.
Scientometrics, 63, 373–401. DOI: https://doi.org/10.1007
/s11192-005-0218-y

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你
q
s
s
/
A
r
t
我
C
e
–
p
d

我

F
/

2
2
4
9
6
1
9
3
0
6
8
5
q
s
s
_
A
_
0
0
1
2
8
p
d