LETTER
Towards a moralization of bibliometrics?
A response to Kyle Siler
Département d’histoire, Université du Québec à Montréal, Montréal, Quebec, Canada
Yves Gingras
a n o p e n a c c e s s
j o u r n a l
Keywords: argumentation, ethics, bibliometrics, methodology, moralization
Citation: Gingras, Y. (2022). Towards a
moralization of bibliometrics? UN
response to Kyle Siler. Quantitative
Science Studies, 3(1), 315–318. https://
doi.org/10.1162/qss_c_00178
DOI:
https://doi.org/10.1162/qss_c_00178
Received: 3 Gennaio 2022
Accepted: 3 Gennaio 2022
Corresponding Author:
Yves Gingras
gingras.yves@uqam.ca
Handling Editor:
Staša Milojević
Copyright: © 2022 Yves Gingras.
Pubblicato sotto Creative Commons
Attribuzione 4.0 Internazionale (CC BY 4.0)
licenza.
The MIT Press
ABSTRACT
In a recent letter to QSS, Kyle Siler (2021), made harsh comments against the decision of the editors
to publish a controversial paper signed by Alessandro Strumia (2021) about gender differences
in high-energy physics. My aim here is to point to the elements in Siler’s letter that are typical of
a new tendency to replace rational and technical arguments with a series of moral statements
and ex cathedra affirmations that are not supported by cogent arguments. Such an approach
can only be detrimental to rational debates within the bibliometric research community.
In a recent letter to QSS, Kyle Siler (2021), made harsh comments against the decision of the
editors to publish a controversial paper signed by Alessandro Strumia (2021) about gender
differences in high-energy physics. Though, like him, “I would rather be doing other things
than writing this letter,” I think it is important not to pass over this letter in silence as it raises
important questions about what I see as a recent and very problematic trend in scholarship: IL
tendency to replace rational and technical arguments with a series of moral statements and
ex cathedra affirmations that are not supported by cogent arguments.
My aim here is not to add another layer of technical comments on Strumia’s paper but to
point to the elements in Siler’s letter that are typical of this new tendency that now also begins
to affect debates within the bibliometric research community.
The most striking feature of Siler’s letter is the moral high horse from which he formulates
his many questionable statements, which are often based on insinuations about supposed
“questionable values” or his being “disappointed” by decisions he disapproves. According
to him, Waltman allegedly offers “vacillating”’ justifications for publishing a “bad” paper,
and even shows “stubbornness” and “arrogance.”
Though it is reassuring that Siler adds that “there are many good people affiliated with
QSS,” we are not told what the basis is for his strong convictions other than that he just
happens to dislike the paper. But anyone with a modicum of understanding of the dynamics
of scientific research would not be surprised that papers are often “widely criticized.” And
invoking the Sokal Affair shows a surprising misunderstanding on his part of the nature of
the debate surrounding Strumia’s paper, which is not a hoax! The nearly hysterical tone of
the letter is also apparent in his comparison of Strumia’s paper with “Flat Earthers” and his
bizarre references to African Colonialism, eugenics, and LGBT parents! Another example of
the ignorance of the nature of publishing is Siler’s character assassination and talk of
“cowardice” for not publishing the review reports. It should be obvious that editors cannot
do that without the permission of the authors. And why should experts in bibliometrics be
l
D
o
w
N
o
UN
D
e
D
F
R
o
M
H
T
T
P
:
/
/
D
io
R
e
C
T
.
M
io
T
.
/
e
D
tu
q
S
S
/
UN
R
T
io
C
e
–
P
D
l
F
/
/
/
/
3
1
3
1
5
2
0
0
8
3
5
8
q
S
S
_
C
_
0
0
1
7
8
P
D
.
/
F
B
sì
G
tu
e
S
T
T
o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
Towards a moralization of bibliometrics?
impressed by the fact that “thousands of physicists signed a petition condemning Strumia’s
work and ideas”? Since when is a petition an argument? I will not comment on the fact that
the very speed at which the petition was signed probably means that most of the signatories
never analyzed the data and were simply reacting emotionally.
The moral tone of Siler’s indictments is also evident in the fact that he seems to consider
that “the implications of the article” which could be “damaging” (for whom?) are more impor-
tant than its empirical validity. After all, suppose some bibliometric results published by two
female researchers using the usual valid methods would somehow contradict the personal
moral convictions of some activist scholars about, Dire, “equity”? Should the results be burned?
Retracted if published? Seemingly having the capacity of a medium, Siler even affirms that
having published Strumia’s paper is “a bad omen for QSS on gender and science issues”!
And as Siler mentions the case of the retracted paper by Nature Communications in November
2020, let us look into that case more carefully, as it is very instructive. That controversy also
reflects a moralizing trend that is detrimental to rational discussions of empirical results.
In a paper titled “The association between early career informal mentorship in academic
collaborations and junior author performance,” the authors (two women and one man), found that
“mentorship quality predicts the scientific impact of the papers written by protégés post mentorship
without their mentors” (AlShebli, Makovi, & Talal Rahwan, 2020UN). They also found that “increas-
ing the proportion of female mentors is associated not only with a reduction in post-mentorship
impact of female protégés, but also a reduction in the gain of female mentors.” In the weeks follow-
ing the publication, and without having the possibility to closely analyze the complex method used
by the authors, these conclusions have been read by many as a “critique” of female mentorship, E
social media hysteria led to the retractation of the paper. Now, let us consider the reasons
invoked by the authors to justify retracting their paper. While reading them, keep in mind that
the reasons to retract a paper are usually related to the fact that the results are somehow
flawed (fake or manipulated data, lack of reproducibility, plagiarism, eccetera. [Shi, Wang et al., 2020]).
So, here is the explanation provided by the authors (AlShebli, Makovi, & Talal Rahwan, 2020B):
“The Authors are retracting this Article in response to criticisms about the assumptions
underpinning the Article in terms of the identification of mentorship relationships and
the measure of mentorship quality, challenging the interpretation of the conclusions.”
What is striking here is that they talk about criticism of their assumptions and interpreta-
zioni. But criticism is usually the ground for discussion, not retraction, right? They then add
Although we believe that all the key findings of the paper with regards to co-authorship
between junior and senior researchers are still valid, given the issues identified by
reviewers about the validation of key measures, we have concluded that the most appro-
priate course of action is to retract the Article.
So, “although” they consider “all the key findings” valid, they still conclude that the “most
appropriate” action is to retract the paper! Here again, that kind of debate is usually found in
letters and rejoinders criticizing the interpretation or method, and do not lead to retraction. Ma
I think we better understand what is really going on here when we read the last paragraph of
their explanation:
We are an interdisciplinary team of scientists with an unwavering commitment to gender
equity, […] We feel deep regret that the publication of our research has both caused pain
Quantitative Science Studies
316
l
D
o
w
N
o
UN
D
e
D
F
R
o
M
H
T
T
P
:
/
/
D
io
R
e
C
T
.
M
io
T
.
/
e
D
tu
q
S
S
/
UN
R
T
io
C
e
–
P
D
l
F
/
/
/
/
3
1
3
1
5
2
0
0
8
3
5
8
q
S
S
_
C
_
0
0
1
7
8
P
D
.
/
F
B
sì
G
tu
e
S
T
T
o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
Towards a moralization of bibliometrics?
on an individual level and triggered such a profound response among many in the sci-
entific community. Many women have personally been extremely influential in our own
careers, and we express our steadfast solidarity with and support of the countless women
who have been a driving force in scientific advancement. We hope the academic debate
continues on how to achieve true equity in science–a debate that thrives on robust and
vivid scientific exchange.
One can of course appreciate that the authors express their “steadfast solidarity” with
women and that they have strong feelings about “equity,” but who could seriously be against
equity? These are certainly laudable moral statements expressing publicly their personal con-
victions, but how are they related to the method and technical content of the paper? And how
do these moral convictions justify retracting a paper when the authors clearly believe that their
findings are valid? And how is that compatible with their hope that academic debate continues
and should “thrive on robust and vivid scientific exchange,” when their action is the very
negation of such debates?
It is obvious to me that the authors have in fact caved in to moral pressures instead of better
explaining that their assumptions and results were warranted after all. For just suppose, for the
sake of the argument, that the scientific field is male dominated and that female scientists are
indeed kept in subjugated positions. In such a world, it cannot be a surprise to find that a
woman mentored by a man will have a greater chance of being integrated in a central network
than a woman mentored by a woman who is part of the subjugated group and thus has a less
powerful network. Finding the contrary would in fact be surprising. Ovviamente, the details of
their methodology are complex and can always be discussed, but their result does not come as
a surprise. In light of this brief analysis, here is my explanation of that controversy: Those who,
for moral and ideological reasons, do not like to read such a conclusion, will try to demolish
the methods. Now, suppose that the same method had found the opposite result: Being men-
tored by women gives rise to a larger “scientific impact” than being mentored by men. Would
that have led to a scandal and social media hysteria? I bet not. And here is an empirical argu-
ment supporting my prediction: The very same authors, using essentially the same complex
method, published a paper in the same journal in 2018 about “The preeminence of ethnic
diversity in scientific collaboration” and concluded that “ethnic diversity had the strongest
correlation with scientific impact” (AlShebli, Rahwan, & Woon, 2018). This is a nice conclu-
sion as we are certainly happy to learn that “ethnic diversity” is not only a good thing in itself
but that it also leads to better science! So, nobody ever complained about that paper despite its
complex method of identifying “ethnicity, discipline, genere, affiliation, and academic age.”
What all this calls to mind is the classic “confirmation bias,” which implies that we are
much less critical of results we like than of results we don’t like. The difference now is that
bias is expressed through moral injunctions and bullying by self-appointed moral entrepre-
neurs instead of arguments.
The content and tone of Siler’s letter is part and parcel of this new trend and one can seri-
ously doubt that it can lead to producing better and more valid results that really shed light on
existing inequalities.
So, far from criticizing the editors for having created a space for rationally debating Strumia’s
paper, we should congratulate them for having done so. QSS should thus stand for robust aca-
demic criteria and reject moralizing comments containing personal attacks or insinuations about
the supposed hidden motives of the paper criticized. Editors should simply ask the critics to stick
to arguments directly related to the content of papers discussed. This is the price to pay to
Quantitative Science Studies
317
l
D
o
w
N
o
UN
D
e
D
F
R
o
M
H
T
T
P
:
/
/
D
io
R
e
C
T
.
M
io
T
.
/
e
D
tu
q
S
S
/
UN
R
T
io
C
e
–
P
D
l
F
/
/
/
/
3
1
3
1
5
2
0
0
8
3
5
8
q
S
S
_
C
_
0
0
1
7
8
P
D
/
.
F
B
sì
G
tu
e
S
T
T
o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
Towards a moralization of bibliometrics?
preserve the autonomy of scientific research, keeping it free from any extrinsic endeavors to
moralize, be it from churches, states or pressure groups convinced that they embody the only
valid version of “social justice.”
For, as the old proverb says: “The road to hell is paved with good intentions.”
REFERENCES
AlShebli, B. K., Rahwan, T., & Woon, l. W. (2018). The preemi-
nence of ethnic diversity in scientific collaboration. Nature
Communications, 9, 5163. https://doi.org/10.1038/s41467-018
-07634-8, PubMed: 30514841
AlShebli, B., Makovi, K., & Talal Rahwan, T. (2020UN). The asso-
ciation between early career informal mentorship in academic
collaborations and junior author performance. Nature Communi-
cations, 11, 5855. https://doi.org/10.1038/s41467-020-19723-8,
PubMed: 33203848
AlShebli, B., Makovi, K., & Talal Rahwan, T. (2020B). Retraction
Note: The association between early career informal mentorship
in academic collaborations and junior author performance.
Nature Communications, 11, 6446. https://doi.org/10.1038
/s41467-020-20617-y, PubMed: 33349637
Shi, Q., Wang, Z., Zhou, Q., Hou, R., Gao, X., … Chen, Y. (2020).
An overview of retraction status and reasons of non-Cochrane
systematic reviews in medicine. MedRXiv. https://doi.org/10
.1101/2020.10.10.20210666
Siler, K. (2021). L’Affaire Strumia reveals troubling gatekeeping values
and outcomes at Quantitative Science Studies. Quantitative Science
Studi, 2(3), 1119–1122. https://doi.org/10.1162/qss_c_00153
Strumia, UN. (2021). Gender issues in fundamental physics: A biblio-
metric analysis. Quantitative Science Studies, 2(1), 225–253.
https://doi.org/10.1162/qss_a_00114
l
D
o
w
N
o
UN
D
e
D
F
R
o
M
H
T
T
P
:
/
/
D
io
R
e
C
T
.
M
io
T
.
/
e
D
tu
q
S
S
/
UN
R
T
io
C
e
–
P
D
l
F
/
/
/
/
3
1
3
1
5
2
0
0
8
3
5
8
q
S
S
_
C
_
0
0
1
7
8
P
D
.
/
F
B
sì
G
tu
e
S
T
T
o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3
Quantitative Science Studies
318
