RESEARCH ARTICLE - IA de Investigación especializada en el MIT

ARTÍCULO DE INVESTIGACIÓN

The reinstrumentalization of the Diagnostic
and Statistical Manual of Mental Disorders
(DSM) in psychological publications:
A citation context analysis

un acceso abierto

diario

Kai Li

School of Information Resource Management, Renmin University of China

Palabras clave: citation context analysis, DSM, instrumentalization, psicología

Citación: li, k. (2021). El
reinstrumentalization of the Diagnostic
and Statistical Manual of Mental
Disorders (DSM ) in psychological
publicaciones: A citation context
análisis. Estudios de ciencias cuantitativas,
2(2), 678–697. https://doi.org
/10.1162/qss_a_00124

DOI:
https://doi.org/10.1162/qss_a_00124

Revisión por pares:
https://publons.com/publon/10.1162
/qss_a_00124

Recibió: 21 Octubre 2020
Aceptado: 3 Febrero 2021

Autor correspondiente:
Kai Li
kai.li@ruc.edu.cn

Editor de manejo:
Juego Waltman

ABSTRACTO

Research instruments play significant roles in the construction of scientific knowledge, incluso
though we have only acquired very limited knowledge about their life cycles from quantitative
estudios. This paper aims to address this gap by quantitatively examining the citation contexts of
an exemplary research instrument, the Diagnostic and Statistical Manual of Mental Disorders
(DSM ), in full-text psychological publications. We investigated the relationship between the
citation contexts of the DSM and its status as a valid instrument being used and described by
psychological researchers. We specifically focused on how this relationship has changed over
the DSM’s citation histories, especially through the temporal framework of its versions. Nosotros
found that a new version of the DSM is increasingly regarded as a valid instrument after its
publicación; this is reflected in various key citation contexts, such as the use of hedges, atención
markers, and the verb profile in sentences where the DSM is cited. We call this process the
reinstrumentalization of the DSM in the space of scientific publications. Our findings bridge an
important gap between quantitative and qualitative science studies and shed light on an aspect
of the social process of scientific instrument development that is not addressed by the current
qualitative literature.

INTRODUCCIÓN

Research instruments (es decir., objects that are instrumental to scientific works), are an important class
of material objects involved in scientific research. Studies on scientific practices have found that
instruments serve as a fundamental device through which researchers can gain access to “nature”
that is otherwise invisible to human beings (Fraassen, 2008; Rheinberger, 1997) and as an impor-
tant epistemological foundation of scientific objectivity (Daston & Galison, 2010). As quantitative
researchers strive to understand the scientific system from broader perspectives (Leydesdorff,
Ràfols, & Milojevic, 2020), the representation of scientific material objects, especially instru-
mentos, in scientific citations and texts has become an important topic yet to be fully investigated.
Específicamente, with an increasing number of publications addressing the scientific impact of
research data sets and software entities (Howison & Bullard, 2015; li, yan, & feng, 2017;
zhao, yan, & li, 2018), two key examples of scientific instruments, we need deeper knowledge
about how the use and development of these research instruments are situated in epistemic
cultures of science (Knorr-Cetina, 1999).

Derechos de autor: © 2021 Kai Li. Publicado
bajo una atribución Creative Commons
4.0 Internacional (CC POR 4.0) licencia.

La prensa del MIT

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
6
7
8
1
9
3
0
7
0
8
q
s
s
_
a
_
0
0
1
2
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

The reinstrumentalization of the DSM in psychological publications

To address this gap, this paper presents an analysis of the citation contexts of a classic re-
search instrument in the scholarship of mental disorder, the Diagnostic and Statistical Manual
of Mental Disorders (DSM ), as a response to the concept of instrumentalization in the field of
science and technology studies (STS). Works by Bruno Latour and his colleagues (Latour, 1987;
Latour & Woolgar, 1979) successfully established a materialist and practice-oriented tradition in
science studies. In this line of research, a frequently recurring theme is that the status of research
instruments is contingent and dependent on specific research contexts in scientific laboratories
that are both temporal and local. Two major arguments have been proposed: that there are
highly blurry and fluid boundaries between research instruments and other types of research
objects (Engeström, 1990; Rheinberger, 1997) and that scientific knowledge and instruments
are always coproduced (Jasanoff, 2004). These ideas are well summarized by Clarke and
Fujimura (1992), who famously stated that scientific instruments are constructed through stabi-
lization of scientific knowledge: A knowledge object becomes a tool when it is “no longer ques-
cionado, examined, or viewed as problematic, but is taken for granted” (páginas. 10–11).

Given this conceptualization of instrumentalization, this study aims to examine how the DSM
gained the status of a well-accepted research instrument in the research domain of psychology.
Developed by the American Psychiatric Association (QUÉ), the DSM is one of the most widely
used classification systems for mental disorders. Originally designed as a tool for interhospital
communication in the 1950s, the DSM was gradually developed into a diagnostic scheme used
by “psychiatrists, other physicians, and other mental health professionals that described the
essential features of the full range of mental disorders” (American Psychiatric Association, 2013).
Bowker and Star (2000) offered the observation that the DSM serves as the lingua franca for medical
insurance companies, because of a lack of competing standards in the market. Además, the DSM
is publicly regarded as an authoritative resource for both research and teaching in such fields as
psychiatry and psychology, especially in North America (Millon & Klerman, 1986; Joven, 1997).

en este estudio, we aim to examine the instrumentality of DSM in a different space other than
scientific laboratories: publicaciones cientificas. We assume that instrumentalization of DSM also
happened in scientific publications after this instrument was developed, even though this process
is rather different from how DSM was physically developed through scientific activities. Inspired
by Small’s recent work (Pequeño, 2018), we assume that the references representing research instru-
ments are cited in different citation contexts over their life cycles after being published. As this
process happens after the development of the instrument, we call it “reinstrumentalization.”
Comparing with the original concept of instrumentalization, reinstrumentalization is also a tem-
poral process during which the cited or mentioned instrument is differently regarded or adopted as
a valid tool: The more instrumentalized a tool’s representation is, the more it is used as a tool,
instead of a conceptual entity, in publications. Sin embargo, due to the fact that these differences
are inevitably represented as textual features in publications and the diversity of language use
in research, reinstrumentalization is expressed as a continuous scale. Es decir, using all
publicaciones cientificas, we will be able to understand the level of reinstrumentalization of a tool
from the perspective of the extent to which it is mentioned within certain linguistic contexts.

According to Clarke and Fujimura (1992), temporality is an important scale along which
research objects show instrumentality. This is also the focus of the present study: We aim to
understand how the level of re-instrumentality of the DSM has changed over its citation histories.
The plural form of history is used here because there are multiple citation histories of the DSM:
while we can take the DSM as a single object being cited in the scientific literature, each of its
different versions (as shown in Table 1) may be deemed to have its own citation history.

Based on the discussions above, this work aims to establish the temporal connections between
the level of re-instrumentality and various linguistic features along with citations to DSM versions.

Estudios de ciencias cuantitativas

679

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
6
7
8
1
9
3
0
7
0
8
q
s
s
_
a
_
0
0
1
2
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

The reinstrumentalization of the DSM in psychological publications

Versión
1st Edition

2nd Edition

3rd Edition

3rd Edition (Text Revision)

4th Edition

4th Edition (Text Revision)

5th Edition

Mesa 1.

Versions of the DSM

Abbreviation

3-TR

4-TR

Publication year
1952

1968

1980

1987

1994

2000

2013

This research is deeply rooted in the tradition of citation context analysis. First proposed in the
early 1980s (Pequeño, 1982), citation context analysis deals with the “particular message or statement
within the citing document containing the reference” (pag. 288), so that deeper meanings of citations
can be extracted from publications. This method has frequently been adopted by researchers in
quantitative science studies dating back to the 1980s, from Garfield’s analysis of how Robert
Merton’s works are cited in different knowledge domains (garfield, 1980) to McCain’s Mean
Utility Index, which considered both the location and context of citations (McCain & Tornero,
1989), and her study of the longitudinal citation contexts of Frederick Brooks’ book The
Mythical Man-Month (McCain & Salvucci, 2006).

Además, following Small’s work (Pequeño, 2018), we likewise assume that more frequent use
of the DSM (or a specific DSM version) in the Method section of research articles demonstrates a
concomitantly higher level of re-instrumentality. Based on this assumption, we have extended
the research framework adopted by Small (2018). We specifically examined the five metadis-
coursal resources proposed by Hyland (2005a) as well as verbs used in citation sentences (más
details of these measurements are discussed in Section 2.2) from more than 100,000 full-text
psychology research articles included in the Elsevier Text and Data Mining service. We hope
our study is a first step towards a more profound appreciation of our scholarly communication
system from a material-oriented perspective, which will help us shift away from document-
centric bias and construct a fairer reward system for all kinds of scholarly outputs.

2. METHOD

2.1. Sample

To conduct this analysis, we acquired all English research articles in psychological journals in-
cluded in the Elsevier Text and Data Mining (TDM) service1, Elsevier’s official API service offer-
ing access to all content in the Elsevier ScienceDirect full-text database. This collection includes
both open-access and institution-subscribed articles. Up to 2018, the ScienceDirect platform
has the full text of over 15 million publications from more than 20,000 journals published as
early as 18232. This research is conducted on the full Elsevier TDM data set as collected by
the Centre for Science and Technology Studies (CWTS) at Leiden University. Using this data-
base, the following data collection steps were undertaken.

1 https://www.elsevier.com/about/policies/text-and-data-mining.
2 https://www.elsevier.com/__data/assets/pdf_file/0005/53528/0597-ScienceDirect-Factsheet-v4-HI-no-ticks.pdf.

Estudios de ciencias cuantitativas

680

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
6
7
8
1
9
3
0
7
0
8
q
s
s
_
a
_
0
0
1
2
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

The reinstrumentalization of the DSM in psychological publications

Primero, todo 203 journals classified under psychology were acquired on April 15, 2019. Todo
English research articles published in these journals, as collected by CWTS, were included in
our sample. We retrieved 203,803 unique DOIs from this step.

Segundo, from the above collection, we selected only the 145,892 papers with structured

full-text data for inclusion in the final sample.

Tercero, we included only papers with at least one Method section. The following four criteria

were used to classify paper sections as either Method or non-Method:

(cid:129) Criterion #1: A Method section title should contain the following terms: “experimental,"

“methodology,” “methodologies,” “method,” or “methods”;

(cid:129) Criterion #2 A Method section title must not contain the terms “experiment” or

“experiments”;

(cid:129) Criterion #3: Any subsection under a Method section is a Method section; y
(cid:129) Criterion #4: A Method section can be a subsection of a non-Method section.

This list of criteria considers the fact that psychological papers sometimes contain parallel
sections with titles such as “Experiment 1” and “Experiment 2.” These sections serve as the
container for a separate set of Method, Resultados, and sometimes Discussion sections. probamos
some additional keywords in our query, especially “data,” but given that these failed to
produce any significant increase in the number of retrieved paper sections, we maintained
our original query.

This approach to classifying paper sections was tested on the full paper sample and was eval-
uated manually. After it was applied on our paper sample, we used all 22 Method-section titles that
were manually validated and appeared at least 100 times in the sample. All the selected Method-
section titles are listed in Appendix A. These titles have 105,220 instancias (en 104,094 unique
documentos); both figures are comparable to the 119,166 instances of the Introduction section found
in our sample. Todo 104,094 articles were thus included in our final sample for this analysis. Estos
papers were published between 1997 y 2016. A more detailed description of the paper sample
is offered in Section 2.4. Based on this sample, we acquired all citances parsed by CWTS from
their database.

Cuatro, based on our final sample, we identified all references related to the DSM. En el
CWTS database, a reference key is given to every unique indexed reference. This key is com-
posed of the name of the leading contributor, the publication year, and some other identifying
información (such as the first few characters of the object title). Por ejemplo, one key for the
fifth edition of the DSM is “americanpsychiatricassoci_2013_the.” However, nonpublication
objects are frequently cited with variant titles and other metadata elements (li, Chen, & yan,
2019), which explains why multiple keys (p.ej. “americanpsychiatricassoci_2013_ame”) eran
found for most DSM versions. To address this issue, we used the following criteria to integrate
keys representing the same DSM version.

#1: The author of the reference is the American Psychiatric Association (“americanpsychiatricassoci”);

#2: The object must be published in a year in which a major DSM version was published (as in

Mesa 1);

#3: The initial letters of the title must be either “the,” “dia,” “dsm,” or “ame,” so that it is pos-
sible to cover different name forms of the DSM without including other resources published by
APA in the same years. The selected letters cover such titles as “The Diagnostic and Statistical
Manual of Mental Disorder” and “Diagnostic and Statistical Manual of Mental Disorder.”

Estudios de ciencias cuantitativas

681

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
6
7
8
1
9
3
0
7
0
8
q
s
s
_
a
_
0
0
1
2
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

The reinstrumentalization of the DSM in psychological publications

In total, 87 unique keys were identified for the seven major versions of the DSM. It should be
noted that there are a few other keys that meet criteria #1 y #3, pero no #2 (es decir., keys with a
different year from those in Table 1). These may be correct references that were given a wrong
publication year, mistakenly or intendedly. Sin embargo, we decided not to use them because of
the difficulty of accurately classifying them into the seven versions.

2.2. Measurements

Based on the sample discussed above, the following measurements were examined in this study.

2.2.1. Ratio of citations in the Method section

It has been commonly accepted that different sections within a paper have distinct narrative
functions and significance (Swales, 1990). An assumption made in this study is that citations
in the Method section are more strongly connected to scientific instrumentality; this assumption
is supported by various prior works in quantitative science studies (Bertin, Atanassova et al.,
2016; Thelwall, 2019) and applied linguistics (Huang, 2014; Kanoksilapatham, 2012). A
calculate this parameter, we measured the ratio of DSM citations used in the Method section
to all DSM citations in a paper. By tracing this ratio and especially how it changes over time,
we strive to illustrate the extent to which a specific version of the DSM is regarded as an estab-
lished research instrument, how this pattern changes over time, and how well this temporal trend
is correlated with other linguistic attributes of DSM citances.

2.2.2. Number of times a reference is cited in a paper

A large number of studies focus on the meanings of a reference when it is cited multiple times in
another article. A number of earlier studies reported evidence that the more times a reference is
cited in a publication, there is a stronger and more meaningful relationship between the citing and
cited documents (Chubin & Moitra, 1975; Hooten, 1991; Espiga & Safer, 2008; Voos & Dagaev,
1976). Específicamente, this parameter can be an even more effective predictor of citation functions
when it is combined with the location of citation, as indicated by multiple studies (McCain &
Tornero, 1989; zhao, Cappello, & Johnston, 2017; zhao & Strotmann, 2020). Sin embargo, most of
these studies were conducted on all research articles and their findings have not been examined
in the context of cited research instruments.

2.2.3.

Linguistic attributes of citances

Linguistic attributes have been increasingly studied as a type of citation context (Ding, Liu et al.,
2013; Jha, Jbara et al., 2017; Pequeño, 2018). en este estudio, we focused on five classes of interactional
markers identified by Hyland (1999): hedges, boosters, attitude markers, self-mentions, and engage-
ment markers. Ellos son, according to Hyland, resources available to writers to interact with their
readers through their writings—for instance, in expressing their views or acknowledging uncer-
tainties. Hedges are the only category that has been deliberately examined in quantitative studies.
Como resultado, the present study aims to offer a more comprehensive analysis of how these resources
are used in the textual description of scientific instruments. The five resources are discussed below
and examples of how these resources are used in our sample are given in Appendix B.

(cid:129) Hedges: Hedges, such as possible and perhaps, are devices indicating uncertainties in
writing (p.ej., in acknowledging an alternative interpretation). Earlier quantitative studies
have proven that hedges are inversely related to the Method section and method-oriented
references, because of the certain tone that is supposed to be expressed in this section
(Chen, Song, & Heo, 2018; Pequeño, 2018; Pequeño, Boyack, & Klavans, 2019).

Estudios de ciencias cuantitativas

682

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
6
7
8
1
9
3
0
7
0
8
q
s
s
_
a
_
0
0
1
2
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

The reinstrumentalization of the DSM in psychological publications

(cid:129) Boosters: Contrary to hedges, boosters, such as obviously and demonstrate, are used to
express certainties. Despite this opposition of function, Hyland (2005b) commented that
the excessive use of either type of marker is discouraged by the research community.
(cid:129) Attitude markers: Attitude markers are those words or phrases that express the writer’s
subjective attitudes towards the topics. Examples include attitude verbs (agree), oración
adverbs (unfortunately), and adjectives (remarkable). While no research has focused on
their usage in scientific writings, we can assume that attitudes, like uncertainties, are less
likely to be expressed in the Method section than in other paper sections.

(cid:129) Self-mention: Self-mention includes first-person pronouns (nosotros) and possessive adjectives
(nuestro). In an earlier study, Hyland (2003) interpreted the ratios of self-mention in scientific
publications across multiple research fields using the different research and persuasion
strategies employed in these communities. Por ejemplo, in the hard sciences, uniformities
in the procedures and results are more important in convincing readers than is personal
authority; de este modo, the texts in these fields are less personal in style than in the soft sciences.
Despite such disciplinary differences, there is a lack of discussion as to how self-mention
phrases are used differently between paper sections.

(cid:129) Engagement markers: Engagement markers are devices to guide readers’ attention or
include them in the discourse. Such expressions include by the way, you may notice,
and note.

2.2.4. Readability of citances

Readability is a frequently used linguistic attribute in evaluating the writing styles of scientific
publicaciones (Hartley, Pennebaker, & Fox, 2003; Hayden, 2008). Despite recent criticism
(Hartley, 2016), we used the Flesch Scale in this study, as it is still widely used in quantitative
studies of texts and especially scientific texts, as a marker for the differences between individual
textos (Didegah, Bowman, & Holmberg, 2018; Oleinik, Kirdina-Chandler et al., 2017; Van
Wesel, Wyatt, & ten Haaf, 2014).

2.2.5. Verbs used in citances

Verbs bear strong rhetorical functions in the construction of scientific texts and are especially
useful for understanding the relationship between citing and cited documents (Bertin et al.,
2016; Bloch, 2010). Several studies have evaluated which verbs are the most frequently used
in the Method section (Bertin & Atanassova, 2014; Lamers, van Eck et al., 2018; Pequeño, 2018).

2.3. Analysis Method

Using the 87 keys mentioned above, we acquired all citances of the DSM and conducted the
following analyses.

For the linguistic analysis, we used the Spacy library in Python (Honnibal & Montani, 2017) a
parse the sentences and extract verbs from them. Based on the statistics reported on its website,
Spacy’s parser reaches an accuracy of 94.48% as tested on a Wall Street Journal data set3.
Además, this library has been increasingly used to examine scientific corpora in recent years
(Lamurias & Couto, 2019; Roth & Basov, 2020). Verbs were extracted based on the part-of-
speech tagger implemented in the library. en este estudio, we extracted and analyzed only the
primary verbs in citances, or the verbs in the main clause. If there were multiple main verbs
en la misma frase, we included all of them.

3 https://spacy.io/usage/facts-figures.

Estudios de ciencias cuantitativas

683

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
6
7
8
1
9
3
0
7
0
8
q
s
s
_
a
_
0
0
1
2
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

The reinstrumentalization of the DSM in psychological publications

Cifra 1. Ratio of articles citing the DSM over time.

To determine whether a sentence used any of the five interactional resources identified by
Hyland, we used the list of phrases offered in his book (Hyland, 2005a). All the included phrases
were matched with the citances described above.

We used the Flesch Reading Ease Scale to calculate the readability of each citance. Este
domain-independent readability scale was developed by Rudolf Franz Flesch in 1943 (Flesch,
1948) and considers sentence length and the number of syllables to calculate the ease of reading
for a corpus. It assigns a score from 0 a 100 to a corpus, con 100 representing the easiest and 0
the most difficult. The textstat Python package4 was used to calculate this parameter.

2.4. Description of the Sample

In aggregate, the identified keys representing the DSM are cited in 17,695 citances belonging
a 12,435 documentos. Cifra 1 summarizes the ratio of citing articles among all sampled articles
(norte = 145,892) in a given year, and thus the relative importance of the DSM in our general
paper sample. Despite the growing numbers of citations over time, the results show that the
DSM has been cited in a relatively stable proportion of papers over the citation window.

We also broke up the total number of citing publications by DSM version. The result is shown
En figura 2, with the y-axis representing the ratio of papers citing each version among the total
number of papers in our overall sample. We used the object history (the year difference between
a version’s publication date and the citation date) as the x-axis, so that the patterns for all DSM
versions can be standardized against their different relationships with the citation window. En
this graph, the four most recent DSM versions are highlighted. The three other versions
( Versions 1–3) are also shown in the graph, but their names are not printed out.

In our research design, only two DSM versions are fully covered by the citation window (de
1997 a 2016): Version 4-TR (2000) and Version 5 (2013). Además, Versión 4 was published
just ahead of the citation window (1994). We can see that each of these three versions is increas-
ingly cited in its early years, despite the very different slopes. After a new version is published,

4 https://github.com/shivam5992/textstat.

Estudios de ciencias cuantitativas

684

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
6
7
8
1
9
3
0
7
0
8
q
s
s
_
a
_
0
0
1
2
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

The reinstrumentalization of the DSM in psychological publications

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

Cifra 2. Ratios of citing articles citing specific versions of the DSM.

fewer citations are given to older versions. This can be explained by the fact that up-to-dateness
is an important factor for researchers in selecting an instrument such as the DSM.

A major focus of this study is to understand how linguistic attributes are used differently in
citances over the histories of the DSM. Mesa 2 summarizes the measurements discussed
arriba. In this table, as well as the rest of the study, we included only versions from V3-TR
to V5 because of the small numbers of citations the first three DSM versions received (17, 29,
y 399, respectivamente). The table shows that some measurements have strong variances among
these four versions, which is the starting point of this study.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
6
7
8
1
9
3
0
7
0
8
q
s
s
_
a
_
0
0
1
2
4
pag
d

Mesa 2.

Summary of key attributes of DSM versions

No. of citing papers

Ratio of citing papers with Method-section citance

V3-TR
980

64.8%

V4
5419

V4-TR

3890

V5
1709

63.7%

53.2%

16.8%

No. of citances

1143

7630

5725

2702

No. of citances in Method section

59.8%

52.1%

42.1%

11.8%

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Citances per paper

Ratio of citances with attention markers

Ratio of citances with boosters

Ratio of citances with self-mentions

Ratio of citances with engagement markers

Ratio of citances with hedging

Mean readability score

Estudios de ciencias cuantitativas

1.17

2.3%

9.4%

3.4%

21.7%

10.7%

22.62

1.4

4.4%

8.7%

4.6%

20.6%

16.2%

18.96

1.47

5.6%

9.2%

3.6%

19.9%

19.6%

15.56

1.58

7.5%

8.9%

5.2%

20.3%

26.6%

13.86

685

The reinstrumentalization of the DSM in psychological publications

Cifra 3. Ratio of Method section citances by DSM version.

3. RESULTADOS

3.1. How Often Is the DSM Cited in the Method Section?

One of the most notable differences among the DSM versions in Table 2 is the ratio of citances
used in the Method section. The likelihood for a citation to be given in this section decreased
sharply from Version 3-TR to Version 5, as shown in Figure 3, where each data point is an
aggregated ratio for a specific year. (This figure omits data points where a version has fewer than
10 citances in a year, to reduce radical outliers.)

Given the fact that these four versions have distinct relationships with the citation window, nosotros
further evaluated whether the differences were caused by differences in the portion of the life cycle
covered by the citation window. En figura 4, we plotted how the Method-section citance ratio
changed over each version’s citation history. Despite the fact that our citation window does not
fully cover most of the DSM versions, there is a similar upward trend in V4 and V4-TR during their
fifth to 15th years after publication. This may suggest a similar trend to be observed in V5 during the

Cifra 4. Ratio of Method section citances over years by DSM version.

Estudios de ciencias cuantitativas

686

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
6
7
8
1
9
3
0
7
0
8
q
s
s
_
a
_
0
0
1
2
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

The reinstrumentalization of the DSM in psychological publications

Cifra 5. Mean number of citances per paper by DSM version.

next few years, given its similar pattern with V4-TR in its earliest citation history. To a lesser extent,
this may also indicate how V3-TR was cited before the 1980s, assuming the general citation
pattern of DSM has remained the same since the 1970s. En general, this result is an indication that
the DSM, as a well-developed scientific instrument, still takes time to be regarded as an instrument
that can be reliably used in scientific research in the space of scientific publications.

3.2. Density of Citance in Papers

Another pattern with a strong linear trend among various versions is the number of citances per
paper. As shown in Figure 5, from versions 3-TR to 5, the mean number of citances per paper
keeps increasing, while that in the Method section decreases significantly.

Cifra 6 shows how both trends change over the citation histories of all DSM versions. Ellos
are once again associated with the different life cycles of these versions. In its beginning years, a

Cifra 6. Mean number of citances per paper by DSM history.

Estudios de ciencias cuantitativas

687

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
6
7
8
1
9
3
0
7
0
8
q
s
s
_
a
_
0
0
1
2
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

The reinstrumentalization of the DSM in psychological publications

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
6
7
8
1
9
3
0
7
0
8
q
s
s
_
a
_
0
0
1
2
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Cifra 7. Mean ratio of citances with Hyland’s interactional markers.

new version is mentioned in non-Method sections more frequently but barely mentioned in the
Method section. In light of the results from the previous section, the decreasing number of
citances in non-Method sections in the earlier years may be explained by the need for researchers
to introduce or justify the DSM as a research instrument after it is published. This need, por supuesto,
gradually diminishes as the DSM is increasingly accepted as an established instrument.

3.3. Linguistic Analysis

We further examined the linguistic attributes related to DSM citances. Cifra 7 illustrates how
the five interactional markers are used differently over the citation histories of DSM versions.

As shown in Figure 7, hedges and attention markers are the only two categories with linear
changes over the citation history. The decreasing trend in both variables is consistent with our
overall assumption that, as the DSM is more established, it is increasingly used as an instrument
and thus is described with a lower level of uncertainty (decreasing use of hedges) and in more
factual tones (decreasing use of attention markers). The trends for self-mention markers and
boosters are essentially flat, and that for engagement markers appears to be subject to radical
fluctuation, even though we removed all data points with fewer than 10 citances.

On the other side of the story, some variables examined above display quite different usage
patterns in Method and non-Method sections, respectivamente, even though they may or may not
show any temporal trends. The results are summarized in Table 3, where the “All versions”
category contains only versions from V3-TR onward, instead of all seven versions of DSM.

A few observations can be drawn from this table. Primero, both hedges and attention markers
are used differently across paper sections and over time. Given the fact that there is a higher

Estudios de ciencias cuantitativas

688

The reinstrumentalization of the DSM in psychological publications

Versión
V3-TR

V3-TR

V4-TR

Sección

Método

Non-Method

Método

Non-Method

Método

Non-Method

Método

Non-Method

All versions

Método

All versions

Non-Method

Mesa 3.

Summary of linguistic attributes by DSM version

Atención
1.3%

Boosters
7.4%

Self-mention
3.6%

Compromiso
21.4%

Hedging
7%

Readability
22.77

4.3%

1.4%

8.6%

8.8%

1.3%

8.5%

1.5%

8.4%

13.4%

12.5%

6.6%

11.5%

7.5%

9.2%

6.4%

11.3%

4.3%

4.9%

3.2%

9.7%

4.5%

22.2%

20%

21.6%

20.8%

19.1%

27.3%

19.2%

20.8%

20.1%

18.3%

6.9%

29.4%

7.6%

30.2%

15.1%

28.4%

7.5%

28.8%

22.3

22.83

13.39

22.11

9.78

28.45

11.65

22.84

12.25

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
6
7
8
1
9
3
0
7
0
8
q
s
s
_
a
_
0
0
1
2
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

ratio of citations being given in the Method section as a reference grows older, this result is
consistent with our findings in Figure 7. De término medio, a citance in a non-Method section is
almost three times more likely to have a hedging phrase and four times more likely to have
an attention marker than one in the Method section, which makes these features strong pre-
dictors of the citation function of a reference. Además, there is an especially high use of
hedges in citances of the DSM-V5 in the Method section (15.1%), showing that even though
it is in the Method section, there is still a higher level of uncertainty being expressed in the
very early years of this version.

Segundo, like hedging and attention markers, boosters, and the readability score exhibit
different patterns in the Method section than in other paper sections. Específicamente, boosters are
also used more frequently in non-Method sections; this is consistent with Hyland’s comment

Mesa 4.

Top 10 verbs from our sample

Rango
1

Todo
include

meet

characterize

diagnose

usar

assess

define

make

base

recruit

Método
include

meet

diagnose

assess

usar

make

base

recruit

consist

establish

Non-Method
characterize

include

define

classify

consider

informe

sugerir

usar

diagnose

find

689

Estudios de ciencias cuantitativas

The reinstrumentalization of the DSM in psychological publications

Mesa 5.

Spearman rank correlation among verb lists from Method and non-Method citances

Category
All citances—Method citances

All citances—Non-Method citances

Method citances—Non-Method citances

Rho value
0.708

0.509

−0.547

that even in the Method section, the expression of certainty is discouraged, even though it is not
consistent with our findings regarding the temporal trend.

3.4. Verbs Used in Citances

El 10 most frequently used verbs extracted from all citances, Method section citances, y
non-Method section citances are summarized in Table 4.

It can be observed that our lists of verbs are very different from the top verbs reported in other
similar works (Bertin & Atanassova, 2014; Pequeño, Tseng, & Patek, 2017), where verbs that are
most frequently used in the Method section normally include use, perform, seguir, etc.. Most of
these verbs rank very low in our results. Además, many verbs in our lists do not fall into the
category of research verbs (es decir., verbs aiming to describe the procedure or acts of research).
Sin embargo, the top verbs identified from this study are consistent with the distinct ways in which
the DSM is involved in research: It is a standard used to diagnose a mental disorder and to
support decisions about recruiting participants.

Notwithstanding the differences between our verb lists and those of previous works, we found
that verbs could function as a valid means of measuring the text similarity of citances between
the Method and non-Method sections. We calculated the frequencies of all 153 verbs that
appeared in our sample at least 10 veces (with “be” and “have” removed because they are used
too broadly). To measure the similarity between these two sections, we applied Spearman rank
correlation to the ranks of verbs from the two corpora. Mesa 5 summarizes the relationships
between the three broad citance groups: those in both the Method and non-Method sections,
those in the Method section only, and those in the non-Method sections only. All values in the
table are significant at the p < 0.05 level. The result shows that the rankings of verbs in the Method and non-Method sections are strongly negatively correlated with each other. Moreover, we also compared how verbs from each version of the DSM are distributed relative to the abovementioned verb lists. The results, summarized in Table 6, show that verb distribu- tions in newer DSM versions are more similar to those in non-Method sections and vice versa, with all values significant at the p < 0.05 level. This strongly supports our earlier observation that Category V3-TR V4 V4-TR V5 Table 6. Spearman rank correlation between verbs in specific versions and general lists All to Method 0.707 All to non-Method 0.175 Method to Method 0.674 Non-Method to non-Method 0.502 0.799 0.602 0.030 0.298 0.507 0.801 0.865 0.774 0.546 0.91 0.892 0.877 Quantitative Science Studies 690 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / e d u q s s / a r t i c e - p d l f / / / / 2 2 6 7 8 1 9 3 0 7 0 8 q s s _ a _ 0 0 1 2 4 p d / . f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 The reinstrumentalization of the DSM in psychological publications DSM versions are increasingly used in the Method section by showing that the use of verbs can be reliably used to evaluate the instrumentality of references, even though the use of individual verbs may not work very well because of the low density of these verbs in citances. 4. DISCUSSION 4.1. The Reinstrumentalization of the DSM in Scientific Texts The present research offers a case study of how citation contexts shift over the citation histories of the DSM in the field of psychology. By conducting a citation context analysis, we examined the relationship between the instrumentality of the DSM and various linguistic attributes, such as the use of interactional markers and verbs in the citation sentences. The most interesting finding from this research is the fact that, even though every version of the DSM was meticulously developed into a blackboxed instrument (Regier, Narrow et al., 2009), it does not automatically become one after being published, at least in the space of sci- entific publications. Instead, it still takes time for the DSM to be accepted as a valid instrument by researchers. This is primarily shown by the fact that as a new DSM version matures, it is increas- ingly used in the Method section compared to its earlier years. This general conclusion is sup- ported by existing works on the rhetorical functions of paper sections (Bertin et al., 2016; Huang, 2014; Kanoksilapatham, 2012) and is also consistent with the citation contexts of sentences in different paper sections, such as the increasing use of DSM citations in the Method section and the opposite trend for the use of hedges and attention markers in DSM citances. The fact that these patterns are shown for all major DSM versions adds further validity to our conclusion. This conclusion sheds light on the processes of the construction of scientific instruments in two significant ways. First, it aims to bridge the gap between quantitative and qualitative science studies on the topic of scientific instruments. This gap was largely created by various barriers to proper and sufficient representations of research instruments in citation data (Li et al., 2017, 2019). As a result, quantitative researchers have only gathered very limited evidence about the performance of these material objects in the scholarly communication system, even less so from the perspective of their life cycles. This reality makes it very difficult for a material- oriented perspective to be established in quantitative science studies and thus for more conver- sations to transpire between these two research communities. Second, our evidence shows that, after its physical development and publication, the DSM does not automatically become a research instrument. This process of reinstrumentalization is not covered by science and tech- nology studies literature concerning how research instruments are produced, which shows how quantitative evidence can help to expand theories developed in qualitative communities. Moreover, in our discussion of the process of reinstrumentalization, we have also shown the significant roles played by the temporal framework in the appreciation of the citation context— in particular, how multiple time frames may exist in an object’s citation history. The concept of version is a vital yet highly underexamined topic in quantitative science studies; versions are critical to the identification of many nonpublication objects, such as research data sets (Pröll & Rauber, 2013) and software objects (Smith, Katz, & Niemeyer, 2016). Moreover, versions are gaining in relevance as prepublication paper repositories are increasingly used by researchers from nearly every field, which creates multiple versions of research articles (Larivière, Sugimoto et al., 2014). This study offers some preliminary evidence about the roles played by versioning in the citation history of an object. Specifically, we have shown that each version of the DSM can be treated as a unique epistemic object, with its own citation history and similar patterns of citation contexts. However, to understand the concept of version in scholarly communication Quantitative Science Studies 691 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / e d u q s s / a r t i c e - p d l f / / / / 2 2 6 7 8 1 9 3 0 7 0 8 q s s _ a _ 0 0 1 2 4 p d . / f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 The reinstrumentalization of the DSM in psychological publications in more depth, we plan to conduct more studies in the future to elucidate how it may work differently for other types of research object and in different knowledge domains. 4.2. Citation Contexts of Research Instruments Another major contribution of the present work is to offer a more comprehensive examination of the relationships between citation contexts and method-related rhetorical functions, inspired by recent work concerning citation contexts of method papers5 (Small, 2018; Small et al., 2019). Our results supported Small’s key finding that hedges are a central predictor for method-related citation context, especially the distinctively low level of uncertainties expressed in the Method section. However, apart from hedges, we also evaluated how other interactional phrases are used along the life cycle of the DSM. We found that attention markers are another strong indi- cator for how the DSM is cited. Like hedges, these show strong variances not only between the Method and non-Method sections, but also between different stages of the DSM’s citation his- tories. In addition to hedges and attention markers, boosters are shown to be used differently between the Method section and other paper sections. However, this difference does not trans- late into temporal patterns. Instead, the use of boosters is relatively stable over the citation his- tories of the DSM. The different patterns for these interactional markers point to the complexity of human language in scientific writings. One example of such complexity is that citations to the DSM are not consistently given in an instrumental context. This may be applicable to other research instruments, given the diversity of citation practices for material objects (Li et al., 2019). To address this complexity, these linguistic attributes can be employed as useful instruments for future works to automatically identify research instruments from scientific citations and texts, which will further help to construct a material-oriented history of science. This study also evaluated the distribution of main verbs in all citances of the DSM. Two conclusions are drawn from the analysis. First, the verb profile of the DSM (even when based on all Method section citances) is very different from earlier findings based on all citations in the Method section (Bertin et al., 2016). This finding indicates that even under the citation contexts of the Method section, there are different and highly individual connections between individual citations (or instruments) and verbs (or actions taken in the research). This is consistent with how the Method section is conceptualized in the Create a Research Space (CARS) model, where the section serves to describe a localized research setting (Swales, 1990). This individualized connection between verbs and citations raises questions about the extent to which we can use a general verb ranking or one scheme of citation context to analyze the Method section, given the vast diversity of research actions taken by researchers. Before fully transforming verbs into a valid research instrument, we need to build better knowledge about the categorization of research actions expressed by action verbs, with a comprehensive con- sideration of local factors, such as disciplinarity and the life cycle of the research objects rep- resented by the citation. This research direction will be an important complement to existing works on full-text scientific publications. Despite the individual nature of the connections between verbs and citations, verbs have proven to be a solid instrument to evaluate citation contexts at a somewhat aggregated level. 5 Many, if not most, of these method papers are representations of research instruments, especially when they are cited in the Method section, even though “research instrument” is not the framework adopted by Dr. Henry Small. Quantitative Science Studies 692 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / e d u q s s / a r t i c e - p d l f / / / / 2 2 6 7 8 1 9 3 0 7 0 8 q s s _ a _ 0 0 1 2 4 p d . / f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 The reinstrumentalization of the DSM in psychological publications By analyzing the similarity of verb profiles, we draw the same conclusion that older DSM versions are more strongly connected to the Method section than are newer ones. 5. CONCLUSION The research reported in this paper offers a citation context analysis of how the DSM is cited in full-text psychological publications. Our results show that the re-instrumentality, or the extent to which the DSM is used in the Method section, varies as specific versions of the DSM go through their respective life stages. Over the first few years after its publication, a DSM version is increas- ingly cited in the Method section, which indicates that it is increasingly cited as a research in- strument over time. This ratio does not seem to be influenced by the publication of the next version and only reaches its peak after around 10–15 years. Moreover, this changing level of instrumentality is accompanied by shifts in the use of some other important citation contexts within citation sentences. We found significantly different uses of hedges, attention markers, and some verbs as the DSM becomes more mature. Our study offers an important quantitative examination of how citations serve as research instruments, one citation function that has not been extensively studied in quantitative science studies. Specifically, this research builds connections between quantitative evidence and theories of instrumentalization developed in qualitative studies in the field of science and tech- nology studies. We found another aspect of how research instruments are constructed that is not fully addressed in the qualitative literature: After “physical” packaging, research instruments need to be re-instrumentalized in the space of scientific texts, as reflected in those texts and their citations. This paper, despite the significance of its findings, represents only the first step towards a deeper understanding of how research instruments are represented in scientific texts. After all, it is only a case study of a unique research instrument on many different levels. To address this limitation, as the next step of this research project, we plan to conduct larger-scale quantitative analyses to examine the findings from the present study more comprehensively, especially the manner in which these research instruments are used differently across knowledge domains. Moreover, it is also possible that DSM is mentioned, instead of cited, in these publications, as DSM versions are “obliterated” into mere mentions over their citation histories (McCain, 2014). While this question cannot be investigated by this current study, it is an important question to be pursued in future works. ACKNOWLEDGMENTS Part of this work was done while the author was a visiting researcher at the Centre for Science and Technology Studies (CWTS) at Leiden University. The author thanks CWTS for providing access to its data infrastructure. The author would also like to thank Drs. Erjia Yan, Cassidy Sugimoto, Chaomei Chen, Jake Williams, Jane Greenberg, and Ali Kenner for their helps in the composition of this article. My thanks also go to Dr. Ludo Waltman and peer reviewers for their comments on the manuscript. FUNDING INFORMATION No funding was received for this research. COMPETING INTERESTS The author has no competing interests. Quantitative Science Studies 693 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / e d u q s s / a r t i c e - p d l f / / / / 2 2 6 7 8 1 9 3 0 7 0 8 q s s _ a _ 0 0 1 2 4 p d / . f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 The reinstrumentalization of the DSM in psychological publications DATA AVAILABILITY The data cannot be made available in a data repository because the provider of the data (Elsevier) does not allow this. REFERENCES American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Washington, DC: American Psychiatric Publishing. DOI: https://doi.org/10.1176/appi.books .9780890425596 Bertin, M., & Atanassova, I. (2014). A study of lexical distribution in citation contexts through the IMRaD standard. PLOS Neglected Tropical Diseases, 1(200,920), 83–402. Bertin, M., Atanassova, I., Sugimoto, C. R., & Lariviere, V. (2016). The linguistic patterns and rhetorical structure of citation context: an approach using n-grams. Scientometrics, 109(3), 1417–1434. DOI: https://doi.org/10.1007/s11192-016-2134-8 Bloch, J. (2010). A concordance-based study of the use of reporting verbs as rhetorical devices in academic papers. Journal of Writing Research, 2(2), 219–244. DOI: https://doi.org/10.17239/jowr -2010.02.02.7 Bowker, G. C., & Star, S. L. (2000). Sorting things out: Classification and its consequences. Cambridge, MA: MIT Press. DOI: https:// doi.org/10.7551/mitpress/6352.001.0001 Chen, C., Song, M., & Heo, G. E. (2018). A scalable and adaptive method for finding semantically equivalent cue words of uncer- tainty. Journal of Informetrics, 12(1), 158–180. DOI: https://doi .org/10.1016/j.joi.2017.12.004 Chubin, D. E., & Moitra, S. D. (1975). Content analysis of refer- ences: Adjunct or alternative to citation counting? Social Studies of Science, 5(4), 423–441. DOI: https://doi.org/10 .1177/030631277500500403 Clarke, A. E., & Fujimura, J. H. (1992). What tools? Which jobs? Why right? In A. E. Clarke & J. H. Fujimura (Eds.) The right tools for the job: At work in twentieth-century life sciences (pp. 3–44). Princeton, NJ: Princeton University Press. DOI: https://doi.org/10 .1515/9781400863136.3 Daston, L., & Galison, P. (2010). Objectivity. New York, NY: Zone Books. Didegah, F., Bowman, T. D., & Holmberg, K. (2018). On the differ- ences between citations and altmetrics: An investigation of fac- tors driving altmetrics versus citations for Finnish articles. Journal of the Association for Information Science and Technology, 69(6), 832–843. DOI: https://doi.org/10.1002/asi.23934 Ding, Y., Liu, X., Guo, C., & Cronin, B. (2013). The distribution of references across texts: Some implications for citation analysis. Journal of Informetrics, 7(3), 583–592. DOI: https://doi.org/10 .1016/j.joi.2013.03.003 Engeström, Y. (1990). When is a tool? Multiple meanings of arti- facts in human activity. In Y. Engeström (Ed.), Learning, Working and Imagining: Twelve Studies in Activity Theory (pp. 171–195). Helsinki: Orienta-Konsultit Oy. Flesch, R. (1948). A new readability yardstick. Journal of Applied Psychology, 32(3), 221. DOI: https://doi.org/10.1037/h0057532, PMID: 18867058 Fraassen, B. C. van. (2008). Scientific Representation: Paradoxes of Perspective. Oxford: Oxford University Press. DOI: https://doi.org /10.1093/acprof:oso/9780199278220.001.0001 Garfield, E. (1980). Citation measures of the influence of Robert K. Merton. Transactions of the New York Academy of Sciences, 39 (1 Series II), 61–74. DOI: https://doi.org/10.1111/j.2164-0947 .1980.tb02769.x Hartley, J. (2016). Is time up for the Flesch measure of reading ease? Scientometrics, 107(3), 1523–1526. DOI: https://doi.org/10.1007 /s11192-016-1920-7 Hartley, J., Pennebaker, J., & Fox, C. (2003). Abstracts, introductions and discussions: How far do they differ in style? Scientometrics, 57(3), 389–398. DOI: https://doi.org/10.1023/A:1025008802657 Hayden, J. D. (2008). Readability in the British Journal of Surgery. British Journal of Surgery, 95(1), 119–124. DOI: https://doi.org /10.1002/bjs.5994, PMID: 18076017 Honnibal, M., & Montani, I. (2017). spacy 2: Natural language un- derstanding with bloom embeddings, convolutional neural net- works and incremental parsing. To Appear. Hooten, P. A. (1991). Frequency and functional use of cited documents in information science. Journal of the American Society for Information Science, 42(6), 397–404. DOI: https://doi.org/10.1002/(SICI)1097 -4571(199107)42:6<397::AID-ASI2>3.0.CO;2-norte

Howison, J., & Bullard, j. (2015). Software in the scientific literature:
Problems with seeing, finding, and using software mentioned in
the biology literature. Journal of the Association for Information
Science and Technology, 67(9), 2137–2155. DOI: https://doi.org
/10.1002/asi.23538

Huang, D. (2014). Genre analysis of moves in medical research

artículos. Stylus, 5(1), 7–17.

Hyland, k. (1999). Academic attribution: Citation and the con-
struction of disciplinary knowledge. Applied Linguistics, 20(3),
341–367. DOI: https://doi.org/10.1093/applin/20.3.341

Hyland, k. (2003). Self-citation and self-reference: Credibility and
promotion in academic publication. Journal of the American
Society for Information Science and Technology, 54(3), 251–259.
DOI: https://doi.org/10.1002/asi.10204

Hyland, k. (2005a). Metadiscourse: Exploring interaction in writing.

Londres: A&C Black.

Hyland, k. (2005b). Stance and engagement: a model of interaction
in academic discourse. Discourse Studies, 7(2), 173–192. DOI:
https://doi.org/10.1177/1461445605050365

Jasanoff, S. (2004). States of knowledge: the co-production of sci-
ence and the social order. Londres: Routledge. DOI: https://doi
.org/10.4324/9780203413845

Jha, r., Jbara, A.-A., Qazvinian, v., & Radev, D. R. (2017). NLP-driven
citation analysis for scientometrics. Natural Language Engineering,
23(1), 93–130. DOI: https://doi.org/10.1017/S1351324915000443
Kanoksilapatham, B. (2012). Facilitating scholarly publication: Genre
characteristics of English research article Introductions and
Métodos. 3l: Idioma, Lingüística, Literature, 18(4).

Knorr-Cetina, k. (1999). Epistemic cultures: How the sciences make

conocimiento. Cambridge, MAMÁ: Prensa de la Universidad de Harvard.

Lamers, w., van Eck, norte. J., waltman, l., & Hoos, h. (2018). Patrones
in citation context: The case of the field of scientometrics. 23rd
International Conference on Science and Technology Indicators
(STI 2018), September 12–14, Leiden, Los países bajos.

Lamurias, A., & Couto, F. (2019). Text mining for bioinformatics
using biomedical literature. Encyclopedia of Bioinformatics and
Biología Computacional, 1, 602–611. DOI: https://doi.org/10
.1016/B978-0-12-809633-8.20409-3

Larivière, v., Sugimoto, C. r., Macaluso, B., Milojevic(cid:1), S., Cronin, B.,
& Thelwall, METRO. (2014). arXiv E-prints and the journal of record: Un

Estudios de ciencias cuantitativas

694

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
6
7
8
1
9
3
0
7
0
8
q
s
s
_
a
_
0
0
1
2
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

The reinstrumentalization of the DSM in psychological publications

analysis of roles and relationships. Journal of the Association for
Information Science and Technology, 65(6), 1157–1169. DOI:
https://doi.org/10.1002/asi.23044

Latour, B. (1987). Science in action: How to follow scientists and
engineers through society. Cambridge, MAMÁ: Harvard University
Prensa.

Latour, B., & Woolgar, S. (1979). Laboratory life: The construction
of scientific facts. Princeton, Nueva Jersey: Prensa de la Universidad de Princeton.
Leydesdorff, l., Ràfols, I., & Milojevic, S. (2020). Bridging the divide
between qualitative and quantitative science studies. Cambridge,
MAMÁ: CON prensa. DOI: https://doi.org/10.1162/qss_e_00061

li, K., Chen, P.-Y., & yan, mi. (2019). Challenges of measuring soft-
ware impact through citations: An examination of the lme4 R
package. Journal of Informetrics, 13(1), 449–461. DOI: https://
doi.org/10.1016/j.joi.2019.02.007

li, K., yan, MI., & feng, Y. (2017). How is R cited in research outputs?
Estructura, impacts, and citation standard. Journal of Informetrics,
11(4), 989–1002. DOI: https://doi.org/10.1016/j.joi.2017.08.003
McCain, k. W.. (2014). Obliteration by incorporation. In B. Cronin
& C. R. Sugimoto (Editores.), Beyond bibliometrics: Harnessing
multidimensional indicators of scholarly impact (páginas. 129–149).
Cambridge, MAMÁ: CON prensa.

McCain, k. w., & Salvucci, l. j. (2006). How influential is Brooks’
law? A longitudinal citation context analysis of Frederick Brooks’
The Mythical Man-Month. Journal of Information Science, 32(3),
277–295. DOI: https://doi.org/10.1177/0165551506064397
McCain, k. w., & Tornero, k. (1989). Citation context analysis and aging
patterns of journal articles in molecular genetics. cienciometria,
17(1), 127–163. DOI: https://doi.org/10.1007/BF02017729
Millon, t. MI., & Klerman, GRAMO. l. (1986). Contemporary directions in
psychopathology: Toward the DSM-IV. Nueva York, Nueva York: Guilford
Prensa.

Nakhli, J., El Kissi, y., Bouhlel, S., Amamou, B., Nabli, t. A., Nasr,
S. Ben, & Alí, B. B. h. (2014). Reliability and validity of the
Arizona Sexual Experiences Scale-Arabic version in Tunisian
patients with schizophrenia. Comprehensive Psychiatry, 55(6),
1473–1477. DOI: https://doi.org/10.1016/j.comppsych.2014
.04.006, PMID: 24850067

Oleinik, A., Kirdina-Chandler, S., Popova, I., & Shatalova, t.
(2017). On academic reading: citation patterns and beyond.
cienciometria, 113(1), 417–435. DOI: https://doi.org/10.1007
/s11192-017-2466-z

Pröll, S., & Rauber, A. (2013). Scalable data citation in dynamic,
large databases: Model and reference implementation.
Actas de la 2013 IEEE International Conference on Big
Datos (Big Data 2013) (páginas. 307–312). DOI: https://doi.org
/10.1109/BigData.2013.6691588

Regier, D. A., Narrow, W.. MI., kühl, mi. A., & Kupfer, D. j. (2009).
The conceptual development of DSM-V. American Journal of
Psiquiatría, 166(6), 645–650. DOI: https://doi.org/10.1176/appi
.ajp.2009.09020279, PMID: 19487400

Rheinberger, H.-J. (1997). Toward a history of epistemic things:
Synthesizing proteins in the test tube. Redwood City, California: stanford
Prensa universitaria.

Roth, C., & Basov, norte. (2020). The socio-semantic space of John
Mohr. Poetics, 101437. DOI: https://doi.org/10.1016/j.poetic
.2020.101437.

Pequeño, h. (1982). Citation context analysis. Progress in Communication

Ciencias, 3, 287–310.

Pequeño, h. (2018). Characterizing highly cited method and non-
method papers using citation contexts: The role of uncertainty.
Journal of Informetrics, 12(2), 461–480. DOI: https://doi.org/10
.1016/j.joi.2018.03.007

Pequeño, h., Boyack, k. w., & Klavans, R. (2019). Citations and cer-
tainty: A new interpretation of citation counts. cienciometria,
118(3), 1079–1092. DOI: https://doi.org/10.1007/s11192-019
-03016-z

Pequeño, h., Tseng, h., & Patek, METRO. (2017). Discovering discoveries:
Identifying biomedical discoveries using citation contexts. Diario
of Informetrics, 11(1), 46–62. DOI: https://doi.org/10.1016/j.joi
.2016.11.001

Herrero, A. METRO., katz, D. S., & Niemeyer, k. mi. (2016). Software cita-
tion principles. PeerJ Computer Science, 2, e86. DOI: https://doi
.org/10.7717/peerj-cs.86

Swales, j. (1990). Genre analysis: English in academic and research

settings. Cambridge: Prensa de la Universidad de Cambridge.

Tagay, S., Herpertz, S., Langkafel, METRO., & Senf, W.. (2005).
Posttraumatic stress disorder in a psychosomatic outpatient clinic:
Gender effects, psychosocial functioning, sense of coherence, y
service utilization. Journal of Psychosomatic Research, 58(5),
439–446. DOI: https://doi.org/10.1016/j.jpsychores.2004
.09.007, PMID: 16026660

Espiga, r., & Safer, METRO. A. (2008). Author-rated importance of cited
references in biology and psychology publications. Diario de
Documentation, 64(2), 246–272. DOI: https://doi.org/10.1108
/00220410810858047

Thelwall, METRO. (2019). The rhetorical structure of science? A multidisci-
plinary analysis of article headings. Journal of Informetrics, 13(2),
555–563. DOI: https://doi.org/10.1016/j.joi.2019.03.002

Torres, A. r., Ferrão, Y. A., Shavitt, R. GRAMO., Diniz, j. B., Costa, D. l. C.,
Do Rosário, METRO. C., Miguel, mi. C., & Fontenelle, l. F. (2014).
Panic disorder and agoraphobia in OCD patients: Clinical profile
and possible treatment implications. Comprehensive Psychiatry,
55(3), 588–597. DOI: https://doi.org/10.1016/j.comppsych
.2013.11.017, PMID: 24374170

van Strien, T., van der Zwaluw, C. S., & Engels, R. C. METRO. mi. (2010).
Emotional eating in adolescents: A gene (SLC6A4/5-HTT)–
Depressive feelings interaction analysis. Journal of Psychiatric
Investigación, 44(15), 1035–1042. DOI: https://doi.org/10.1016
/j.jpsychires.2010.03.012, PMID: 20416884

Van Wesel, METRO., Wyatt, S., & ten Haaf, j. (2014). What a difference a
colon makes: How superficial factors influence subsequent cita-
ción. cienciometria, 98(3), 1601–1615. DOI: https://doi.org/10
.1007/s11192-013-1154-x

Voos, h., & Dagaev, k. S. (1976). Are all citations equal? O, did
we op. cit. your idem? Journal of Academic Librarianship, 1(6),
19–21.

watson, PAG. J., ross, D. F., & morris, R. j. (2003). Borderline person-
ality traits correlate with death penalty decisions. Personality and
Individual Differences, 35(2), 421–429. DOI: https://doi.org/10
.1016/S0191-8869(02)00204-0

Joven, A. (1997). The harmony of illusions: Inventing post-traumatic
stress disorder. Princeton, Nueva Jersey: Prensa de la Universidad de Princeton. DOI:
https://doi.org/10.1515/9781400821938

zhao, D., Cappello, A., & Johnston, l. (2017). Functions of uni-and
multi-citations: Implications for weighted citation analysis.
Journal of Data and Information Science, 2(1), 51–69. DOI:
https://doi.org/10.1515/jdis-2017-0003

zhao, D., & Strotmann, A. (2020). Deep and narrow impact:
Introducing location filtered citation counting. cienciometria,
122(1), 503–517. DOI: https://doi.org/10.1007/s11192-019
-03280-z

zhao, METRO., yan, MI., & li, k. (2018). Data set mentions and citations:
A content analysis of full-text publications. Journal of the
Association for Information Science and Technology, 69(1),
32–46. DOI: https://doi.org/10.1002/asi.23919

Estudios de ciencias cuantitativas

695

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
6
7
8
1
9
3
0
7
0
8
q
s
s
_
a
_
0
0
1
2
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

The reinstrumentalization of the DSM in psychological publications

APPENDIX A: SELECTED METHOD-SECTION TITLES

Título
methods

método

materials and methods

methodology

material and methods

experimental procedures

data and methods

methods and materials

subjects and methods

experimental design

experimental results

patients and methods

research methodology

research method

materials and method

research methods

data and method

data and methodology

general method

general methods

participants and methods

material and method

Count in paper sample
40,735

39,616

11,901

2,981

2,699

1,380

1,237

953

569

433

387

375

312

286

258

230

214

173

134

123

112

Estudios de ciencias cuantitativas

696

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
6
7
8
1
9
3
0
7
0
8
q
s
s
_
a
_
0
0
1
2
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

The reinstrumentalization of the DSM in psychological publications

APPENDIX B: EXAMPLES OF HOW HYLAND’S METADISCOURSE EXPRESSIONS ARE USED IN
OUR PAPER SAMPLE

Category
Hedges

Boosters

DSM-citation sentence with the target phrase highlighted
“Perhaps with these cautionary notes in mind, the American Psychiatric
Association has thus far taken a conservative approach to proposing
specific new diagnoses for electronic media in the forthcoming DSM-V
(American Psychiatric Association, 2010)."

“Our differentiated evaluation of the prevalence of trauma or PTSD
demonstrates that not every trauma on the PDS trauma checklist
equates to a psychic trauma according to DSM-IV [1]."

Citación

(van Strien, van der Zwaluw

et al., 2010)

(Tagay, Herpertz et al., 2005)

Attitude markers

“Unfortunately, sin embargo, we were unable to investigate how the

(Torres, Ferrão et al., 2014)

recently proposed DSM-5 subtypes of hypochondria (es decir. care seeking
vs. care avoidant types) [63] relate to OCD with PD/AG vs. OCD
without PD/AG.”

Self-mention

“We have performed this study with one hundred first outpatients with a

(Nakhli, El Kissi et al., 2014)

Compromiso
markers

diagnosis of schizophrenia (DSM-IV TR criteria) [12] from the outpatient
psychiatric service of Farhat Hached Hospital in Sousse.”

“Such an analysis obviously receives some confirmation in observations
that borderline personalities display an unstable identity (Americano
Psychiatric Association, 1994) and that borderline traits correlated
inversely with death penalty support in the present project.”

(watson, ross, & morris, 2003)

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d