ARTÍCULO DE INVESTIGACIÓN

Investigating dissemination of scientific
information on Twitter: A study of topic
networks in opioid publications

Robin Haunschild1

, Lutz Bornmann2

, Devendra Potnis3

, and Iman Tahamtan4

1Max Planck Institute for Solid State Research, Stuttgart, Alemania
2Division for Science and Innovation Studies, Administrative Headquarters of the Max Planck Society, Munich, Alemania
3School of Information Sciences, College of Communication and Information, University of Tennessee, Knoxville, TN, EE.UU
4School of Information Sciences, Comunicación, and Information, University of Tennessee, Knoxville, TN, EE.UU

Palabras clave: altmetrics, bots, hashtags, network analysis, opioid publications, Twitter

ABSTRACTO
While previous research has mostly focused on the “number of mentions” of scientific
research on social media, the current study applies “topic networks” to measure public
attention to scientific research on Twitter. Topic networks are the networks of co-occurring
author keywords in scholarly publications and networks of co-occurring hashtags in the tweets
mentioning those publications. We investigate which topics in opioid scholarly publications
have received public attention on Twitter. Además, we investigate whether the topic
networks generated from the publications tweeted by all accounts (bot and nonbot accounts)
differ from those generated by nonbot accounts. Our analysis is based on a set of opioid
publications from 2011 a 2019 and the tweets associated with them. Results indicated that
Twitter users have mostly used generic terms to discuss opioid publications, such as “Pain,"
“Addiction,” “Analgesics,” “Abuse,” “Overdose,” and “Disorders.” A considerable amount of
tweets is produced by accounts that were identified as automated social media accounts,
known as bots. There was a substantial overlap between the topic networks based on the
tweets by all accounts (bot and nonbot accounts). This result indicates that it might not be
necessary to exclude bot accounts for generating topic networks as they have a negligible
impact on the results. This study provided some preliminary evidence that scholarly
publications have a network agenda-setting effect on Twitter.

1.

INTRODUCCIÓN

1.1. Social Media Mentions in Research Evaluation

Social media increasingly play an important role in the dissemination of scientific information
to the public. The public can then engage in discussions around scientific topics shared on
redes sociales (chan, Nickson et al., 2020; Murphy & Salomone, 2013; patel, Haunschild
et al., 2020). The transfer of scientific information to the public is an ongoing activity in which
knowledge is obtained from those who own it (p.ej., the authors and journals), is learned by
those who do not have it (p.ej., social media users), and is being passed to other people through
social networks (Havakhor, Soror, & Sabherwal, 2018). People use social media to share and
discuss complex scientific information (Murphy & Salomone, 2013), motivating public conver-
sations around various issues such as climate change. The diffusion of scholarly publications

un acceso abierto

diario

Citación: Haunschild, r., Bornmann, l.,
Potnis, D., & Tahamtan, I. (2021).
Investigating dissemination of
scientific information on Twitter: A
study of topic networks in opioid
publicaciones. Quantitative Science
Estudios, 2(4), 1486–1510. https://doi.org
/10.1162/qss_a_00168

DOI:
https://doi.org/10.1162/qss_a_00168

Autor correspondiente:
Iman Tahamtan
tahamtan@vols.utk.edu

Derechos de autor: © 2021 Robin Haunschild,
Lutz Bornmann, Devendra Potnis, y
Iman Tahamtan. Published under a
Creative Commons Attribution 4.0
Internacional (CC POR 4.0) licencia.

La prensa del MIT

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

2
4
1
4
8
6
2
0
0
7
8
8
7
q
s
s
_
a
_
0
0
1
6
8
pag
d

/

.

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Investigating dissemination of scientific information on Twitter

and scientific information on social media can have a positive societal impact, por ejemplo, por
educating and promoting health literacy and changing public behavior (Korda & Itani, 2013).

To measure the impact of scientific information diffused on social media, most past studies
have focused on the number of times scholarly publications have been mentioned on social
media. The “number of mentions” of scholarly publications on social media has been used as
a measure to evaluate the impact and value of scientific research (for a review see Bornmann,
2014; Sugimoto, Work et al., 2017; Tahamtan & Bornmann, 2020). Sin embargo, the number of
mentions of scholarly publications on social media may only reflect public interest and atten-
tion to scientific research rather than their impact (Tahamtan & Bornmann, 2020).

1.2. Network of Topics in Scholarly Publications in Research Evaluation

Haunschild, Leydesdorff, and Bornmann (2020a) and Haunschild, Leydesdorff, Bornmann
et al. (2019b) suggested that besides “number of mentions,” the impact of scientific informa-
tion can also be evaluated by determining which topics in scholarly publications are more
frequently discussed on social media. They noted that determining the topics in scholarly pub-
lications that have received public attention on social media provides a reasonable way to
evaluate their impact beyond their academic impact that is often measured by citation counts
(Haunschild et al., 2020a; Haunschild et al., 2019b).

Haunschild et al. (2019b) proposed a new method that not only measures public attention
to scholarly publications on social media but also demonstrates how Twitter users (representar-
ing the public) discuss scientific research. They used a network approach in which a topic
network (co-occurrence network of author keywords) in scholarly publications would be com-
pared with a topic network of author keywords in scholarly publications that are mentioned on
Twitter (or any other social media platform). This approach assumes that the topics in scholarly
publications with broader societal impact would receive greater public attention on Twitter.
The co-occurrence network approach focuses on the topics in a scholarly publication shared
on Twitter rather than “number of mentions” or how many times the publication is mentioned
on social media. An advantage of the co-occurrence network-based approach over previous
approaches (measuring number of mentions) is that it can be used to analyze, map, and com-
pare scientific discussions around a given topic (represented in the author keywords in those
publicaciones) with public discussions around that topic (assessed by the author keywords in the
publications mentioned on Twitter).

1.3. Opioid Scholarly Publications on Social Media

The public has barely any access to scholarly publications about opioids (or any other topic)
unless they are shared on social media or other platforms. Scholarly publications and the
knowledge associated with them once shared on social media are opened to the public
and create value for them.

The influence of opioid scholarly publications on public attention on social media and their
role in creating public awareness is an understudied topic. De este modo, it is important to study how
scientific knowledge shared on social media in the case of opioid scholarly publications cre-
ates value for users. en este estudio, we assess the topics in opioid scholarly publications that
have received public attention on social media. The reason for studying opioids is declaring
it as a public health emergency in the United States in 2017 (Casa Blanca, 2017). Opioids
cause the death of thousands of people worldwide every year (Rudd, 2016; Rudd, Aleshire
et al., 2016). It is an important issue for the public which has many implications for areas such

Estudios de ciencias cuantitativas

1487

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

2
4
1
4
8
6
2
0
0
7
8
8
7
q
s
s
_
a
_
0
0
1
6
8
pag
d

.

/

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Investigating dissemination of scientific information on Twitter

as public health, mental health, and economics. The results can be used to learn which topics
in opioid scholarly publications have more impact or are more popular on social media.
Por lo tanto, it is of merit to study how opioid scholarly publications are discussed on Twitter.

1.4. The Knowledge Gap and Study Objectives

There has been considerable research on using numbers of mentions of scholarly publications
on social media to evaluate the impact of research. Sin embargo, little is known about using topic
networks in research impact and assessing public attention to scholarly publications. As one
way to evaluate the value of scientific research is to measure the public attention it receives
on social media, and given the advantages of topic networks for assessing public attention to
scientific research (Haunschild et al., 2020b; Haunschild et al., 2019b), this study uses the
co-occurrence network analysis approach to assess which topics in “opioid” scholarly pub-
lications have received public attention on Twitter.

We also study the influence of bots in measuring public attention to scholarly publications
on social media. A considerable amount of tweets are produced by automated social media
accounts, known as bots (Ferrara, 2020b; Hegelich & Janetzko, 2016). Bots can impact public
opinion and social media discussions by presenting a distorted reality or artificially and force-
fully changing or influencing the public discourse. Bots can manipulate public attention to and
discussions on critical public issues such as COVID-19 (Ferrara, 2020a). Por lo tanto, it is impor-
tant to know if bots influence public attention to scholarly publications on social media. El
influence of bots has not been investigated in previous Twitter network studies (Haunschild
et al., 2020a; Haunschild et al., 2019b).

To address our research questions, we created co-occurrence networks of the author
keywords in opioid scholarly publications from 2011–2019 in the Web of Science ( WoS,
Clarivate Analytics, Filadelfia, Pensilvania, United States). We analyzed the topics asso-
ciated with opioid scholarly publications shared on Twitter in comparison to the topics of
opioid scholarly publications. The topic networks are shown in two different versions: net-
works created by including the publications shared by all Twitter accounts (bot accounts and
nonbot accounts) and networks created by including the publications shared by only nonbot
Twitter accounts.

2. BACKGROUND

2.1. Application of Topic Networks of Scholarly Publications Shared on Social Media in

Research Evaluation

Network analysis approaches can be used to analyze the diffusion of scholarly publications on
redes sociales. Alperin, Gómez, and Haustein (2019) studied the diffusion patterns of peer-
reviewed scholarly publications on Twitter. They analyzed 1,590 tweets mentioning 11 artículos
in biology and found that the users were connected through common followers that mostly
shared the same article: Most scholarly publications are spread on Twitter within a tightly
connected single community. Además, almost half of the tweeted publications were dis-
seminated on Twitter through retweets. Hellsten and Leydesdorff (2020) used a new network
analysis approach to analyze the discussions around scholarly publications on Twitter for
understanding and visualizing online science-related communications. Their approach indi-
cated which Twitter users were connected with which hashtags.

Scholars such as Haunschild et al. (2019b) and Haunschild, Leydesdorff et al. (2020b) pro-
posed a network-based approach to measure public attention to scientific research on Twitter.

Estudios de ciencias cuantitativas

1488

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

2
4
1
4
8
6
2
0
0
7
8
8
7
q
s
s
_
a
_
0
0
1
6
8
pag
d

.

/

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Investigating dissemination of scientific information on Twitter

Their network analysis approach can be used to identify which scholarly publications have
entered the public discussion on social media; which topics in scholarly publications have
received greater public attention on social media; and how the public perceives and discusses
scholarly publications differently from the scientific community (Haunschild et al., 2020b;
Haunschild et al., 2019b).

Haunschild et al. (2020b) compared the topic network (network of author keywords) generación-
erated from approximately 46,000 climate change scholarly publications between 2011 y
2017 with the topic network in 775,499 tweets containing a link to those publications. Ellos
found that the climate change research topics that had achieved public attention on Twitter
were generally related to the consequences of climate change for humans. They reported that
publications with more general keywords were more likely to be tweeted than those with
scientific jargon. In a similar study, using another topic, Haunschild et al. (2020a) examined
how Library and Information Science (LIS) publications were discussed on Twitter. Su
results demonstrated that only certain topics in LIS publications received public attention
on Twitter, such as librarians, libraries, investigación, and social media. Haunschild et al.
(2020a) also indicated that while all LIS publications were generally more focused on
theoretical applications and methodologies, the topics in the tweeted LIS publications (y
publications mentioned in the news) were related to health applications, redes sociales, pri-
vacy issues, and sociological studies.

The studies that have employed network analysis approaches (p.ej., Haunschild et al.,
2020a; Haunschild et al., 2019b) indicate that the dissemination of scholarly publications
on social media brings public attention to some topics of scholarly outputs more than others.
These studies also indicate that topics of scholarly publications are transferred to the public
discussions as a bundle of networked topics, which is discussed in the following section.

2.2. Network-Agenda-Setting Effect of Scholarly Publications Shared on Social Media

Scholarly publications shared on social media can set an agenda for social media users, estafa-
sequently impacting and shaping their opinion. We adopt this idea from the network agenda-
setting model (guo & McCombs, 2011) which states the repetition of an issue (p.ej., opioid)
and topics related to that issue (p.ej., addiction, abuso, pain) will be transferred to and impact
public opinion as a bundle of networked topics (guo & McCombs, 2011).

According to the network agenda-setting model, it can be argued that the scholarly pub-
lications around any topic such as opioids shared on social media, specifically by influential
authors and high-impact journals, can set a public agenda, consequently attracting public
attention and influencing the public opinion. Por ejemplo, Haunschild et al. (2019b) dem-
onstrated that the terms related to climate policy such as “food security,” “governance,"
“renewable energy,” and “redd” (reducing emissions from deforestation and forest degrada-
ción) were linked to each other in a cluster of networked topics on Twitter. The topics adja-
cent to each other demonstrate that people would link and make connections between those
topics in their minds, consequently impacting their opinion (guo, 2015; guo & McCombs,
2011).

To assess the network agenda-setting effect of scholarly publications on social media, el
superposición (and/or correlation) between two topic networks can be evaluated: a topic network of
author keywords in scholarly publications and a topic network of scholarly publications
shared on social media and/or mentioned in the news. When there is a high overlap or cor-
relation between the two networks, it can be said that the agenda set by those scholarly pub-
lications has impacted public attention (guo, 2015; guo & McCombs, 2011).

Estudios de ciencias cuantitativas

1489

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

2
4
1
4
8
6
2
0
0
7
8
8
7
q
s
s
_
a
_
0
0
1
6
8
pag
d

/

.

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Investigating dissemination of scientific information on Twitter

2.3. Bot Accounts Activity on Social Media

Bots can perform many actions, such as information collection and distribution, generating
clicks and content (p.ej., comments), and editing articles on Wikipedia (Leidl, 2019). Bots
may intervene in online discussions on critical public issues to manipulate public opinion
(Ferrara, 2020a). Some bots behave like humans (social bots) and are hard to detect (Hegelich
& Janetzko, 2016). Previous studies have indicated that bots can impact public discussion on
redes sociales. Por ejemplo, Hegelich and Janetzko (2016) showed that bots shape the public
agenda on political issues. They noted that bots conceal their bot identity, prompt topics and
hashtags to appeal to the public, and retweet selected tweets. Por ejemplo, bots take a popular
tweet and retweet it by adding specific hashtags. Bots can prompt (political) topics and hash-
tags that may sound interesting to the public (Hegelich & Janetzko, 2016).

Only a few studies have investigated how bots impact scientific information diffusion on
redes sociales. Haustein et al. (2016) studied bot accounts that tweeted scholarly publications
deposited on the preprint repository arXiv in 2012. They showed that bots created 9% de
tweets to scholarly publications submitted to arXiv and were subsequently published in jour-
nals indexed in WoS. Bots distributing scholarly publications undermine the usefulness of
tweets-based metrics for research impact (Haustein, Bowman et al., 2016). Didegah,
Mejlgaard, and Sørensen (2018) studied the impact of bots in distributing scholarly publica-
tions in five different fields. They found that 65% of Twitter accounts were bots that contrib-
uted to disseminating scientific information, particularly in life and earth science.

3. DATA AND METHODS

3.1.

Inclusion Criteria for Studies

In the first step, we collected all the scholarly publications indexed in the WoS, published
entre 2011 y 2019, and contained opioid-related terms in their titles (ver tabla 1).
The period of 2011 a 2019 was chosen because Altmetric.com started covering Twitter data
en 2011 (Haunschild et al., 2019b). To retrieve publications from WoS, we needed a list of

Mesa 1.

List of sources that were used to determine the keywords used to perform the search in WoS

Fuente
Cochrane review: Candy et al. (2018),

Doleman et al. (2018), Moe-Byrne et al.
(2018), and Smith et al. (2018)

Centers for Disease Control and Prevention
(2018b), and Centers for Disease Control
and Prevention (2019b)

Palabras clave
Narcotic, Opiate, Morphine, Diamorphine, Fentanyl, Remifentanil, Alfentanil,

Meperidine, Pethidine, Tramadol, and Ketobemidone

Prescription pain relievers included opioids and covered the following drug
subcategories: Hydrocodone, Oxycodone, Tramadol, Codeine, Morphine,
Fentanyl, Buprenorphine, Butrans, Belbuca, Oxymorphone, Hydromorphone,
Methadone, Tapentadol, and Propoxyphene

National Institute on Drug Abuse (2019)

Common prescription opioids: Hydrocodone ( Vicodin), Oxycodone

(OxyContin, Percocet), Oxymorphone (Opana), Morphine (Kadian, Avinza),
Codeine, and Fentanyl; Slang terms used for opioid: Oxy, Percs, and Vikes

Rubinstein and Carpenter (2017)

Commonly prescribed opioids: Codeine, Fentanyl, Hydrocodone,

Hydromorphone, Methadone, Morphine, and Oxycodone

Public Health Experts

Suboxone, Subutex, and Heroin

Estudios de ciencias cuantitativas

1490

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

2
4
1
4
8
6
2
0
0
7
8
8
7
q
s
s
_
a
_
0
0
1
6
8
pag
d

/

.

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Investigating dissemination of scientific information on Twitter

keywords relevant to opioids to be searched in WoS. The following section explains how the
list of keywords was determined.

3.2. Methods to Identify Search Terms

To find a list of keywords synonymous with “opioid,” we performed the following steps. El
list of keywords is presented in Table 1.

Primero, we searched the Cochrane Database of Systematic Reviews (in the Cochrane Library,
https://www.cochranelibrary.com/) in February 2020 to find the reviews that contained opioids
or opioids in their titles. Fifty-seven reviews were retrieved. Cochrane reviews meticulously list
the keywords used in their search strategy. We collected a list of opioid-related terms and syn-
onyms from the “Search Strategy” section of Cochrane reviews. Among the 57 retrieved
reviews, we extracted the synonyms from four reviews published in the last 2 años. The most
recent reviews dated back to 2019 (these studies include Candy, Jones et al., 2018; Doleman,
Leonardi-Bee et al., 2018; Moe-Byrne, Marrón, & McGuire, 2018; Herrero, Burns, & Cuthbert,
2018). This approach provided us with a rich list of opioid synonyms, such as Narcotics,
Opiate, Morphine, and Diamorphine (ver tabla 1).

Segundo, we found other relevant keywords from the following resources: the Centers for
Disease Control and Prevention (CDC) annual surveillance report of drug-related risks and out-
comes (Centers for Disease Control and Prevention, 2018a, 2019a), CDC guideline for pre-
scribing opioids for chronic pain (Centers for Disease Control and Prevention, 2019C), el
National Institute on Drug Abuse (2019), and Rubinstein and Carpenter (2017).

Además, two public health experts (with research background on opioids) suggested

“Suboxone,” “Subutex,” and “Heroin” to be added to the list.

3.3. Bibliometric Data Sources

The bibliometric data was collected from an in-house database developed and maintained by
the Competence Centre for Bibliometrics (CCB, https://www.bibliometrie.info/) and retrieved
from the Science Citation Index Expanded (SCI-EXPANDED), Social Sciences Citation Index
(SSCI), and Arts and Humanities Citation Index (A&HCI), produced by Clarivate Analytics.
The in-house database was last updated in April 2019. The search was performed in the
WoS online interface. The export of the results was done using the “Fast 5K” mode. Only a
few metadata can be retrieved this way, (p.ej., author keywords are excluded). De este modo, nosotros
extracted the WoS UTs (a unique accession number of a record in the WoS) and DOIs and
appended the author keywords from the in-house database.

In the first step, we searched a combination of search terms mentioned in Table 1 in the title
(TI) field of documents. Our initial search indicated that some of the retrieved documents
were not related to opioids. Por ejemplo, a document that included “Oxy-fuel” in its title
was assessed as irrelevant. Besides, some search terms such as “Percs” only retrieved two
documentos, and “Vikes” did not retrieve any documents. These search terms, which are the
slang used for opioids (National Institute on Drug Abuse, 2019) were excluded from our
buscar. We performed our search, including the final list of search terms in WoS, in three steps
(see Appendix A).

The publication year 2019 was incomplete at the time of data retrieval (Febrero 2020).
Sin embargo, this is irrelevant, as our in-house database was last updated at the end of April
2019. We were able to match 14,381 UTs in the in-house database. For all practical purposes,
we expect all opioid publications between 2011 y 2018, with a few early indexed opioid

Estudios de ciencias cuantitativas

1491

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

2
4
1
4
8
6
2
0
0
7
8
8
7
q
s
s
_
a
_
0
0
1
6
8
pag
d

/

.

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Investigating dissemination of scientific information on Twitter

publications from 2019, to be in our data set. Of those 14,381 publicaciones, 10,855 contained
author keywords.

3.4. Data Extraction From Twitter and News Outlets

Altmetric.com (https://www.altmetric.com) is a company that tracks mentions of scholarly
publications in various sources such as Twitter, Facebook, news outlets, and Wikipedia. Schol-
arly publications’ mentions can be accessed at no cost for research purposes via the Altmetric
.com API or snapshots. Altmetric.com provides access to the IDs of tweets (a unique identifi-
cation number assigned to each tweet by Twitter). These IDs were used to download 173,187
tweets (including retweets) associated with 6,433 opioid publications tweeted by at least two
different accounts via the Twitter API. We did not include publications tweeted only once to
reduce noise in the data because we assume they may have been tweeted by the publisher or
the authors for self-promotion purposes. We also downloaded other available information
besides the tweet texts from the Twitter API using the R software (R Core Team, 2019), semejante
as Twitter user names (see Appendix B). The number of times a paper was mentioned in tweets
or news outlets was taken from the Altmetric.com database snapshot.

Some tweets were not available and accessible, such as “private tweets,” “deleted tweets,"
and “suspended accounts,” and therefore were not included in our analysis (see Appendix C).

The analysis units in this study were author keywords in opioid publications and hashtags
associated with the tweets mentioning opioid publications (as the representation of topics).
Three sets of author keywords were extracted: author keywords in all opioid publications;
author keywords in publications tweeted at least twice; and author keywords of publications

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

2
4
1
4
8
6
2
0
0
7
8
8
7
q
s
s
_
a
_
0
0
1
6
8
pag
d

/

.

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Cifra 1. The plot of bot probability scores for Twitter accounts discussing the opioid scholarly
publications published between 2011 y 2019 and indexed in WoS. Accounts with a probability
score up to 0.5 are considered as nonbots (norte = 28,985), and accounts with a score above 0.5 son
considered bots (norte = 26,489).

Estudios de ciencias cuantitativas

1492

Investigating dissemination of scientific information on Twitter

tweeted at least twice and mentioned in the news outlets. As mentioned, publications tweeted
only once were excluded because they may have been tweeted by the publisher or the authors
for self-promotion purposes.

3.5. Detection of Bots

Detecting bots is very hard due to the evolving capabilities of bots. Sin embargo, many attempts
have been made to detect bots in recent years (Ferrara, 2020b). This study used the default
model of the R package “tweetbotornot” designed by Kearney (2019) to detect bots. El
default model of “tweetbotornot” is 93.53% accurate in classifying bots and 95.32% accurate
in classifying nonbots (Kearney, 2019). It uses two sets of data to classify accounts to bots and
nonbots: user-level data, such as biography, ubicación, number of followers and friends; y
tweet-level data, such as the number of hashtags, menciona, and capital letters in the 100 mayoría
recent tweets of a Twitter user (Kearney, 2019). This model can only classify 180 users as bots
or nonbots every 15 minutos. Accounts that receive a score of at least 0.5 (a probability of
50%) are considered bots (davis, Varol et al., 2016).

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

2
4
1
4
8
6
2
0
0
7
8
8
7
q
s
s
_
a
_
0
0
1
6
8
pag
d

.

/

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Cifra 2. Network of top 70 author keywords in all opioid publications published between 2011 y 2019. An interactive version of this
network can be viewed at https://s.gwdg.de/gWGMIa.

Estudios de ciencias cuantitativas

1493

Investigating dissemination of scientific information on Twitter

We ran the “tweetbotornot” package on the Twitter accounts (usernames) in our data set
(56,266 distinct users), que se llevó 3.256 días (180 users per 15 minutos) to classify Twitter
accounts to bots and nonbots. Running the package on our data set, we obtained two warning
messages for some accounts: “sorry, that page does not exist” and “not authorized.” We used a
self-consistent methodology to find the bot probability of the profiles that returned a “not
authorized” error (norte = 22,396). We reran the package on these 22,396 profiles. This resulted
in identifying a bot probability for 21,416 of the accounts and an error for 980. We reran the
package on the 980 accounts and received a valid bot estimate for 188 accounts and an error
para 792. Rerunning the package on these 792 accounts resulted in errors for all. De este modo, nosotros
stopped rerunning the package on these accounts. En general, we found the bot probability of
hasta 50% para 28,985 accounts (nonbots) and above 50% para 26,489 accounts (bots). Cifra 1
shows the plot of the bot probability scores, with a red line at 0.5.

3.6. Visualization of Networks

To visualize the data, we used VOSviewer v.1.6.12 (https://www.vosviewer.com). Following
Haunschild, Leydesdorff and Bornmann (2019a, 2020a), the algorithm designed by Kamada

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

2
4
1
4
8
6
2
0
0
7
8
8
7
q
s
s
_
a
_
0
0
1
6
8
pag
d

/

.

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Cifra 3. Network of top 69 author keywords in opioid publications published between 2011 y 2019, tweeted by at least two accounts
(considering all accounts). The interactive network can be found at https://s.gwdg.de/EhmvtM.

Estudios de ciencias cuantitativas

1494

Investigating dissemination of scientific information on Twitter

and Kawai (1989) for drawing undirected graphs and weighted graphs was used to lay out the
resulting files (containing cosine-normalized distributions of terms in the Pajek format; ver
https://mrvar.fdv.uni-lj.si/pajek). The community-searching algorithm in VOSviewer was
employed with a resolution parameter of 1.0, the minimum cluster size of 1, ten random starts,
ten iterations, a random seed of 0, and the option “merge small clusters” enabled. The node’s
size indicates the frequency of co-occurrence of a specific term with all other terms on the
network. Lines between two nodes and their thickness indicate the co-occurrence frequency
of these particular terms.

Before creating the networks, we unified some synonyms in our data set using Excel. Para examen-
por ejemplo, “Drug_abuse,” “Substance_abuse,” and “Opioid_abuse” were merged into “Opioid_abuse.”
Sin embargo, we did not combine general terms (es decir., “Abuse,” “Treatment,” “Prescribing,” “Addic-
ción,” “Dependence,” “Overdose,” “Analgesic,” “Constipation,” and “Hyperalgesia”) a
the categories with more specific terms. Por ejemplo, we considered “Prescribing” as a general
term, but more specific terms such as “Opioid_prescribing,” “Prescription_drugs,” and
“Prescription_opioid” were merged into “Prescription_opioid.” In addition, “Opioids,” “Opiates,"
“Opiate,” and “Opioid” were all grouped as “Opioid.” We also did the same grouping for
other general terms such as “Analgesia” and “Analgesics,” grouped as “Analgesics,” or
“Narcotics” and “Narcotic,” grouped as “Narcotic.”

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

2
4
1
4
8
6
2
0
0
7
8
8
7
q
s
s
_
a
_
0
0
1
6
8
pag
d

/

.

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Cifra 4. Network of top 64 author keywords in opioid publications published between 2011 y 2019, tweeted by at least two accounts
(considering only nonbot accounts). The interactive network can be found at https://s.gwdg.de/cRGyNN.

Estudios de ciencias cuantitativas

1495

Investigating dissemination of scientific information on Twitter

We only included the most frequently occurring author keywords and hashtags in our anal-
ysis: We used the author keywords that appeared more than five times in opioid publications
and were tweeted by at least two accounts and were mentioned at least once in news outlets.
The resulting number of top author keywords was 70. The other sets of publications and their
author keywords were larger. To compare networks and data of similar sizes, we also tried to
use the top 70 author keywords for the other sets (all publications and tweeted publications).
Sin embargo, due to the tied author keywords, a slightly different number of most frequent author
keywords had to be used in some cases, p.ej., arriba 69 author keywords for the publications that
were tweeted by at least two accounts.

One network from the top 70 author keywords in all opioid publications was created
(Cifra 2). También, five networks from the tweets that were posted by all accounts (bot and non-
bot accounts) were created: one network with the top 69 author keywords of opioid publica-
tions that were tweeted by at least two accounts (Cifra 3), one network with the top 64 author
keywords of publications tweeted by at least two accounts and mentioned in the news
(Cifra 5), one network with the top 70 hashtags of tweets (Cifra 7), and one network of
the top 35 author keywords tweeted by at least two accounts and top 35 hashtags of tweets
(Cifra 9).

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

2
4
1
4
8
6
2
0
0
7
8
8
7
q
s
s
_
a
_
0
0
1
6
8
pag
d

.

/

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Cifra 5. Network of top 64 author keywords in publications published between 2011 y 2019, tweeted by at least two accounts (estafa-
sidering all accounts) and mentioned in the news. The interactive network can be found at https://s.gwdg.de/ YfAXwY.

Estudios de ciencias cuantitativas

1496

Investigating dissemination of scientific information on Twitter

We also created five networks from the tweets that were posted by only nonbot accounts:
one network with the top 64 author keywords of opioid publications that were tweeted by at
least two accounts (Cifra 4), one network with the top-64 author keywords of publications
tweeted by at least two accounts and mentioned in the news (Cifra 6), one network with the
arriba 64 hashtags of tweets (Cifra 8), and one network of the top 35 author keywords tweeted
by at least two accounts and top 35 hashtags of tweets (Cifra 10, a network of 70
hashtags/keywords).

4. RESULTADOS

The following section presents various networks used to explore the similarities and differ-
ences of scientific and public communications around opioids on Twitter.

4.1. Author Keywords

Cifra 2 shows a network of the top 70 author keywords in the opioid scholarly publications
published between 2011 y 2019. The six clusters of author keywords marked by respective
colors are visualized in the figure, which provides a general overview of the topics presented

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

2
4
1
4
8
6
2
0
0
7
8
8
7
q
s
s
_
a
_
0
0
1
6
8
pag
d

.

/

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Cifra 6. Network of top 64 author keywords in publications published between 2011 y 2019,
tweeted by at least two accounts (considering only nonbot accounts) and mentioned in the news.
The interactive network can be found at https://s.gwdg.de/LvH6dS.

Estudios de ciencias cuantitativas

1497

Investigating dissemination of scientific information on Twitter

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

2
4
1
4
8
6
2
0
0
7
8
8
7
q
s
s
_
a
_
0
0
1
6
8
pag
d

.

/

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Cifra 7. Network of top 70 hashtags in the tweets linked to opioid scholarly publications published between 2011 y 2019 (considering all
accounts). The interactive network can be found at https://s.gwdg.de/xumAgn.

in opioid scholarly publications from 2011 a 2019. Each cluster includes a set of related key-
words in each network.

The largest node in Figure 2 is the term “Opioid” in the center of the blue network, sur-
rounded by terms that have close ties with opioids, such as opioid disorders, opioid addiction,
opioid treatment, and opioid use. The green cluster contains pain-related terms, including pain
management, neuropathic pain, postoperative pain, acute pain, and analgesic. The red cluster
comprises terms related to the use of and dependence on substances such as morphine, dynor-
phin, dopamina, and cocaine, which are also used as pain relievers. This cluster also contains
mental disorders and issues related to the consumption of opioids, such as anxiety, depresión,
and stress. The light blue cluster deals with opioid prescriptions, usar, abuso, and overdose. El
light blue and purple clusters comprise author keywords related to the consequences of taking
opioids for a long time, mainly constipation and abnormally increased sensitivity to pain
(hyperalgesia).

The network of opioid publications tweeted by at least two accounts is illustrated in
Cifra 3 (bot and nonbot) y figura 4 (only nonbot accounts). There was a high overlap
(95.3%) between the keywords in these two networks, which might indicate that bots do

Estudios de ciencias cuantitativas

1498

Investigating dissemination of scientific information on Twitter

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

2
4
1
4
8
6
2
0
0
7
8
8
7
q
s
s
_
a
_
0
0
1
6
8
pag
d

.

/

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Cifra 8. Network of top 64 hashtags in the tweets linked to opioid scholarly publications published between 2011 y 2019 (considering
only nonbot accounts). The interactive network can be found at https://s.gwdg.de/jK4LHd.

not impact public attention to opioid scholarly publications on Twitter or that bot accounts
communicate scholarly publications in a similar way to humans.

The green cluster in Figure 3 consists of terms related to opioid misuse, overdose, analge-
sics, epidemic, prescription, and primary care. The red cluster includes author keywords that
deal with opioid use, addiction, disorders, and treatment. The blue cluster contains keywords
related to pain management and palliative care. The yellow cluster is pertinent to the use of
postoperative analgesic drugs. Most of the keywords in Figure 3 also appear in Figure 4, y
the two networks have 61 keywords in common (95.3%).

Cifra 5 shows the top 64 author keywords in publications tweeted by at least two accounts
(considering all accounts) and mentioned in the news. Cifra 6 shows the top 64 author key-
words in publications tweeted by at least two accounts (considering only nonbot accounts)
and mentioned in the news. We tried to focus more specifically on the public discourse by
including the news outlets because we expect the news editors to select topics that are most
certainly of public interest.

En figura 5, the red cluster is centered around pain management as well as primary and
palliative care. The green cluster consists of opioid misuse, ansiedad, and depression. The blue
color is focused on self-administration of and dependence on alcohol, cocaína, codeine,
tramadol, and heroin. The yellow cluster is comprised of opioid use, addiction, and treatment.
The purple cluster seems pertinent to making opioid-related policies, such as policies

Estudios de ciencias cuantitativas

1499

Investigating dissemination of scientific information on Twitter

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

Cifra 9. Network of top 35 author keywords in opioid scholarly publications published between 2011 y 2019, tweeted by at least two
accounts (considering all accounts) and the top 35 hashtags in the tweets. The interactive network can be found at https://s.gwdg.de/UPnfPm.

/

/

/

2
4
1
4
8
6
2
0
0
7
8
8
7
q
s
s
_
a
_
0
0
1
6
8
pag
d

/

.

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

regarding opioid prescription, and policies about the use of marijuana and cannabis. Uno
hundred per cent (norte = 64) of the author keywords in Figure 5 (considering all accounts) también
appear in Figure 6 (considering only nonbot accounts), yet they may appear in different
grupos (with different colors). This high overlap provides further evidence that bots may
not impact topic networks of opioid scholarly publications.

4.2. Hashtags

Hashtags are metadata that are often used strategically to label and describe social media
posts. We analyzed hashtags in the tweets linked to opioid scholarly publications to under-
stand how people describe those publications by using hashtags. Cifra 7 shows the network
of top 70 hashtags in the tweets posted by all accounts. The red color reflects the largest
grupo, con 27 hashtags that mostly represent palliative care and pain management. Este
cluster also includes research-related hashtags such as “#SCIENCE,” “#OPENACCESS,"
“#PAINJOURNAL,” and “#COCHRANE.” The blue cluster is related to the opioid crisis, epi-
demic, and addiction. The green cluster contains hashtags such as “#IDU,” “#PWID,” “#SUD,"
and “#OD,” which refer to the use and injection of drugs, substance use, and substance
overdose. The overlap between Figure 7 (all accounts) y figura 8 (only nonbot accounts)
es 81.3%.

Estudios de ciencias cuantitativas

1500

Investigating dissemination of scientific information on Twitter

4.3. Author Keywords and Hashtags

We also generated the co-occurrence network of the top 35 hashtags and top 35 author key-
palabras, including bot accounts (Cifra 9) and nonbot accounts (Cifra 10). We found a high
superposición (90.06%) between the hashtags and author keywords in the two networks. Cifra 9
indicates that some hashtags and author keywords in the red cluster were synonyms, como
“#PAIN” and “#PAINEVIDENCE,” which are associated with pain, pain management, pallia-
tive care, and cancer pain, or “#CANCER,” which is related to the author keywords cancer and
cancer pain. The blue cluster also consists of hashtags and keywords with similar concepts,
such as “#HEROIN” and “Heroin.” “#OVERDOSE” and “#OD” are associated with author
keywords such as overdose and opioid overdose. Other hashtags in this cluster also seem
to be related to opioid overdoses, such as “#PWID” and “#PWUD,” which refer to the people
who inject and use opioids. In the green cluster, opioid addiction, disorders, usar, and treat-
ment are far away and less connected to hashtags such as “#HCV,” “#HIV,” “#METHADONE,"
“#CANNABIS,” and “#BUPRENORPHINE.”

Mesa 2 shows the overlap of networks generated from the tweets posted by all accounts
and nonbot accounts. There is a 95.3% overlap between the top author keywords of opioid
scholarly publications tweeted by all accounts and the top author keywords of publications

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

2
4
1
4
8
6
2
0
0
7
8
8
7
q
s
s
_
a
_
0
0
1
6
8
pag
d

.

/

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Cifra 10. Network of top 35 author keywords in opioid scholarly publications published between 2011 y 2019, tweeted by at least two
accounts (considering only nonbot accounts) and the top 35 hashtags in the tweets. The interactive network can be found at https://s.gwdg.de
/IOw481.

Estudios de ciencias cuantitativas

1501

Investigating dissemination of scientific information on Twitter

Mesa 2. Overlap between networks of author keywords in opioid scholarly publications posted on Twitter by bot accounts and without
bot accounts

Networks
Overlap between Figures 3 y 4: Top author keywords of
publications that were tweeted by at least two accounts

Overlap between Figures 5 y 6: Top author keywords of
publications that were tweeted by at least two accounts
and mentioned in the news

Overlap between Figures 7 y 8: Top hashtags of tweets

Overlap between Figures 9 y 10: Author keywords of

publications that were tweeted by at least two accounts
with top hashtags of tweets

Frequency of terms that
appeared in both networks
61

Percentage of terms that
appeared in both networks
95.3

64

52

58

100.0

81.3

90.6

tweeted by only nonbot accounts. This high overlap shows that including bot accounts in the
final analysis did not impact the topic networks of opioid scholarly publications.

Además, there is a 100% overlap between top author keywords in publications tweeted
by all accounts and mentioned in the news and author keywords in publications tweeted by
only nonbot accounts and mentioned in the news. Mesa 2 indicates a high (81.3%) superposición
between the top hashtags of tweets by all accounts and the top hashtags of tweets by only
nonbot accounts. There is a 90.6% overlap between the network of top hashtags/author
keywords (considering all accounts) and the network of top hashtags/author keywords (estafa-
sidering only nonbot accounts).

As evident in Table 3, there was a high overlap (84.1%) between author keywords in all
opioid scholarly publications (Cifra 2) and author keywords in publications that were tweeted
by at least two accounts (Cifra 3). A much lower overlap was found between author key-
words in all publications and author keywords in publications that were tweeted by at least
two accounts and mentioned in the news (Cifra 5). Mesa 3 also shows that there was a 72.5%
overlap between terms in the publications that were tweeted by at least two accounts and
publications that were both tweeted by at least two accounts and mentioned in the news.

Mesa 4 shows a 78.1% overlap between the terms in all publications (Cifra 2) and pub-
lications that were tweeted by at least two accounts (Cifra 4). The overlap between the terms

Mesa 3. Overlap between networks of author keywords in all opioid scholarly publications and the publications tweeted that included
both bot and nonbot accounts and mentioned in the news

Network of all publications (Cifra 2)

Network of publications tweeted by at

least two accounts (Cifra 3)

Network of publications tweeted by at
least two accounts and mentioned
in the news (Cifra 5)

Network of all
publicaciones
(Cifra 2)
70

Network of publications
tweeted by at least two
accounts (Cifra 3)
84.1%

Network of publications tweeted
by at least two accounts and
mentioned in the news (Cifra 5)
64.3%

58

45

69

50

72.5%

70

Estudios de ciencias cuantitativas

1502

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

2
4
1
4
8
6
2
0
0
7
8
8
7
q
s
s
_
a
_
0
0
1
6
8
pag
d

.

/

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Investigating dissemination of scientific information on Twitter

Mesa 4. Overlap between networks of author keywords in all opioid scholarly publications and the publications tweeted that included
only nonbot accounts and mentioned in the news

Network of all publications (Cifra 2)

Network of publications tweeted by
at least two accounts (Cifra 4)

Network of publications tweeted by at
least two accounts and mentioned
in the news (Cifra 6)

Network of all
publicaciones
(Cifra 2)
64

Network of publications
tweeted by at least two
accounts (Cifra 4)
78.1%

Network of publications tweeted
by at least two accounts and
mentioned in the news (Cifra 6)
65.6%

50

42

64

51

79.7%

64

in all publications (Cifra 2) and publications that were tweeted by at least two accounts and
mentioned in the news (Cifra 6) era 65.6%. Además, Mesa 4 shows a 79.7% superposición
between terms in the publications tweeted by at least two accounts (Cifra 4), and the publi-
cations tweeted by at least two accounts and mentioned in the news (Cifra 6).

5. DISCUSIÓN

The amount of information embedded in a message (p.ej., a tweet) does not necessarily prompt
users to share the message on social media (Potnis et al., 2020). We also found that the
topics in opioid scholarly publications, which are possibly more popular, important, or appeal-
En g, have received more public attention on Twitter. Public attention to specific topics might
be a helpful metric to be used in certain aspects of research evaluation. Por ejemplo, fondos
agencies can evaluate research proposals based on whether the topics presented in the pro-
posals have received public attention on Twitter.

We found a high overlap between all networks that were presented in the results section.
An explanation for the large overlap between networks could be that publishers, autores, y
Twitter users largely use paper keywords to tweet about newly published papers. It seems the
most obvious thing to do, especially for publishers and Twitter users who do not understand
the content of papers and couldn’t come up with a better keyword list than the one given by
the authors in the paper.

This study has several theoretical contributions to the research evaluation literature, cual

are mentioned below.

5.1. Communication Channels for Scholarly Publications on Twitter

Our results indicated that there are two channels of communication in the diffusion of scien-
tific information on Twitter: networked topics and networked hashtags. This provides evidence
that people’s communications about scholarly publications on social media have patterns in
the form of networked topics. Patterns refer to the fact that people’s interactions are not ran-
dom but take place in patterned and systematic ways (Leeds-Hurwitz, 2009). These systematic
patterns could be seen in the networks and clusters within networks. Each network consists of
clusters of keywords that are related to each other by co-occurrence.

Another channel of communication was the network of topics and hashtags (Figures 9 y
10). The networks of topics and hashtags reflect the multimodal nature of communication
(Leeds-Hurwitz, 2009) on Twitter; eso es, both hashtags and author keywords are used by

Estudios de ciencias cuantitativas

1503

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

2
4
1
4
8
6
2
0
0
7
8
8
7
q
s
s
_
a
_
0
0
1
6
8
pag
d

.

/

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Investigating dissemination of scientific information on Twitter

Twitter users to make sense of scholarly publications, and both should be studied to under-
stand how people communicate about scholarly publications on Twitter.

This may provide evidence that people understand, conversar, and remember scientific infor-
mation as clusters of networked topics and hashtags. Each cluster within networks represents a
community of users with similar interests discussing topics of interest. Por ejemplo, in the red
cluster in Figure 9, “#PAIN” and “#PAINEVIDENCE” are associated with terms such as “Pain,"
“Pain management,” “Palliative care,” and “Cancer pain.”

Linking scientific information to hashtags on social media can also reflect the meta-
communication function (Leeds-Hurwitz, 2009) of hashtags, in that they are used to clarify
what the tweet is about. The advantage of using hashtags to transfer topics of scholarly
publications on Twitter is that it allows users with similar interests to interact with each other
without relying on gatekeepers such as influential authors. The hashtags and topics in networks
form communities and act as information organizers. Each cluster shows a subcommunity of
people with similar interests. People with similar interests come across each other and form
virtual communities without any boundaries. If someone is interested in scholarly publications
related to specific topics, they would likely meet people with similar interests. The networks
not only show connections, but also the possibility of people with similar interests coming
together.

5.2. Using Topic Networks to Identify Topics of Interest in the Public Discourse on Twitter

Topic networks can be used to determine topics of interest related to scholarly publications in
the public discourse on Twitter.

5.2.1. Publications with generic topics

Haunschild et al. (2019b) noted that climate change publications with more general keywords
were more likely to be tweeted than those with jargon. Our results also indicated that generic
topics were more noticeable in all topic networks. We found that the most tweeted topics were
general keywords such as “Opioid,” “Pain,” “Addiction,” “Abuse,” “Depression,” “Treatment,"
and “Analgesics.” The network of opioid publications tweeted by at least two accounts
(Cifra 3), was also focused on similar generic topics. It was also indicated that generic hash-
tags were often used in the tweets linked to opioid scholarly publications, such as “#PWUD”
(people who use drugs), “#PWID” (people who inject drugs), “#SUD” (substance use drugs),
“#ADDICTION,” “#OPIOID,” “#OPIOIDCRISIS,” “#OPIOIDEPIDEMIC,” and “#OVERDOSE.”

Besides publications with generic terms, some networks indicated public attention to

specific topics related to opioids, such as policy and scientific evidence about opioids.

5.2.2. Policy-related topics

Publications always make recommendations such as policy recommendations relevant to
agencias gubernamentales, el público, and different communities. The topic networks of author
keywords in publications tweeted by at least two accounts and mentioned in the news
(Figures 5 y 6) revealed policy-related topics. En figura 5, “policy” is in the center of the
purple cluster and linked to topics such as “Health policy,” “Marijuana,” “Medicaid,” “Opioid
prescription,” and “Overdose.” This result indicates the importance of policy-making
regarding opioid use and prescription and its impact on health. Haunschild et al. (2019b) también
found policy-related issues such as “Food_security,” “Governance,” and “Renewable_energy”
in the tweeted publications about climate change. Policy-related topics may be at the focal
point of public attention in other disciplines too.

Estudios de ciencias cuantitativas

1504

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

2
4
1
4
8
6
2
0
0
7
8
8
7
q
s
s
_
a
_
0
0
1
6
8
pag
d

.

/

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Investigating dissemination of scientific information on Twitter

5.2.3.

Scientific evidence about opioids

We found some science-related hashtags such as “#SCIENCE,” “#OPENACCESS,"
“#PAINJOURNAL,” “#COCHRANE,” “#RESEARCH,” and “#STUDY.” These hashtags may
reflect the fact that scientists are also active on Twitter posting scholarly publications or that
people attribute such hashtags to tweets to cite scientific evidence to verify the credibility of
their opioid-related posts.

5.3. The Network-Agenda-Setting Effect of Scholarly Publications on Public Attention

We examined which author keywords in opioid scholarly publications from 2011 a 2019
have received public attention on Twitter. We found a high overlap between the network of
author keywords in all publications (Cifra 2) and the networks of author keywords in the
publications shared on Twitter by either bot accounts or nonbot accounts. These results show
that the topics frequently discussed in scholarly publications also get much attention on
Twitter. The high overlap may also represent the presence of a network agenda-setting effect
of scholarly publications on public attention on Twitter (guo, 2015). In line with the network
agenda-setting model (guo & McCombs, 2011), it can be the case that topics of scholarly
publications are transferred to the public discussion on Twitter as clusters of networked topics
and impact public opinion. This assumption needs to be tested in future studies using statistical
tests such as Granger causality (Vargo & guo, 2017).

The use of hashtags in tweets can also bring users with similar interests together on social
media, forming communities (Potnis & Tahamtan, 2021), and consequently setting a public
agenda on social media (Hemsley, 2019; Potnis & Tahamtan, 2021). Sin embargo, this study pro-
vides some preliminary evidence on the presence of a network agenda-setting effect on social
media by the shared scholarly publications. The topics in scholarly publications that have the
potential to set an agenda on social media can be said to have an impact on social media users.

5.4. Effect of Bots on Public Attention to Scholarly Publications on Social Media

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

2
4
1
4
8
6
2
0
0
7
8
8
7
q
s
s
_
a
_
0
0
1
6
8
pag
d

.

/

We studied the impact of different actors (bot accounts versus nonbot actors) on public atten-
tion to scholarly publications. If bot accounts impact public attention to scholarly publications,
they should be removed in studies that use social media data.

We investigated whether the networks generated by all accounts (bot and nonbot accounts)
were different from those generated by nonbot accounts. In line with past studies (Didegah
et al., 2018; Haustein et al., 2016), the results in the current research demonstrated that bot
accounts were extensively involved in disseminating opioid scholarly publications on Twitter.
Sin embargo, because of the high overlap between the networks that included bot accounts and those
with nonbot accounts, it can be said that bots do not manipulate but possibly magnify public atten-
tion to scholarly publications on Twitter. Investigating this assumption requires further studies and
analiza, such as multiple regression models, including larger data sets and other disciplines. En
this regard, Lokot and Diakopoulos (2016) maintained that bots could be useful in automating the
spread of news by news agencies and citizen journalists. Sin embargo, their result on bots is in dis-
agreement with parts of the literature maintaining that bots manipulate and shape public opinion
and attention regarding ideological and political debates (Kollanyi, Howard, & Woolley, 2016).

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

5.5. Study Limitations and Future Research

We analyzed opioid scholarly publications published between 2011 y 2019 (and their
corresponding tweets) that have been mentioned by at least two accounts. We did not check

Estudios de ciencias cuantitativas

1505

Investigating dissemination of scientific information on Twitter

if the same person owned the two accounts. Sin embargo, checking such a possibility seems to be
impossible because some persons may not use their real names for their Twitter accounts. Es
indeed impossible, at least with current social media platforms, to reliably tell whether the
same person owns two social media accounts.

We only considered top (64 a 70) author keywords/hashtags in generating the networks and
assessing the overlap between networks. Including all author keywords/hashtags may result in
different overlap scores but may generate complex networks that are difficult to analyze.

The majority of the authors of scholarly publications are not on Twitter. This is a limitation
that is beyond our control. Sin embargo, the approach proposed in this study can still be
applied to different domains to evaluate the impact of scholarly publications on social media.

We noted that one way to assess public attention to scholarly publications is to analyze the
contents of scholarly publications shared on social media. This statement may not hold if the
people sharing content on social media are the authors of opioid research publications or
researchers from other fields. Por lo tanto, future studies should find ways to classify Twitter
accounts to nonacademic users (representing the public) and nonpublic (p.ej., healthcare pro-
fessionals, journalists, organizaciones). A este respecto, two recent studies by Mohammadi,
Barahmand, and Thelwall (2020) and Mohammadi, Thelwall et al. (2018) indicated that
almost half of scholarly publications in the study samples were discussed by nonacademic
users on Twitter and Facebook. Mohammadi et al. (2018) conducted a survey study on
1,912 Twitter users who had tweeted journal articles and indicated that half of them did not
work in academia. Mohammadi et al. (2020) manually classified users who had mentioned
500 journal articles on Facebook and indicated that 58% of users were nonacademics. Estos
two studies suggested that half of the discussions on scholarly publications on social media are
performed by the public. We suggest that future studies focus on methods and ways to analyze
and interpret results in terms of nonacademics and academics. En otras palabras, the focus
should be on topics of scholarly publications receiving more attention from nonacademics
compared to academics on social media.

The high overlap between networks with all accounts and networks with only nonbot
accounts raises several questions that can be addressed in future studies: Are bots simply rep-
licating and amplifying the message in tweets by humans? Does it mean that bot-generated
tweet topics are subsets of human-generated tweet topics or vice versa? Do bots tweet the
same tweets after human-generated tweets? What is the timeline correlation between human
and bot-generated tweets? Future studies should also identify how many people read the
tweets by bots and nonbot accounts? The number of people who read or engage with that
tweet might partially represent the magnitude of public attention.

Future research might investigate how the approach proposed in the current study works in
other domains. Further research is required to confirm our result that bot accounts can impact
or manipulate public attention to scholarly publications on Twitter.

6. CONCLUSIONS
Unlike most previous studies that have used “the number of mentions” of scholarly publica-
tions on social media to measure research impact, we used topic networks to measure
public attention to opioid publications. The results indicated that Twitter provides generic
information about scholarly publications in the form of clusters of networked topics and hash-
tags. Bots are greatly involved in the distribution of scholarly publications on Twitter. Sin embargo,
they have a negligible impact on the networks generated from author keywords in

Estudios de ciencias cuantitativas

1506

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

2
4
1
4
8
6
2
0
0
7
8
8
7
q
s
s
_
a
_
0
0
1
6
8
pag
d

/

.

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Investigating dissemination of scientific information on Twitter

publicaciones. This study provided some preliminary evidence that scholarly publications might
have a network agenda-setting effect on Twitter, in that the networks of topics in scholarly
publications can impact public attention on Twitter.

EXPRESIONES DE GRATITUD

We appreciate the editors and reviewers for their valuable feedback and comments which
improved our work.

CONTRIBUCIONES DE AUTOR

All authors have contributed to this manuscript equally.

CONFLICTO DE INTERESES

Los autores no tienen intereses en competencia.

DISPONIBILIDAD DE DATOS

The bibliometric data used in this study are from the bibliometric in-house database of the
Competence Centre for Bibliometrics (CCB; https://www.bibliometrie.info/). The CCB’s
database is developed and maintained by a cooperation of various German research orga-
nizations. The database is derived from the Science Citation Index Expanded (SCI-E), Social
Sciences Citation Index (SSCI), Arts and Humanities Citation Index (AHCI) prepared by
Clarivate Analytics (Filadelfia, Pensilvania, EE.UU). The Twitter data are retrieved from
our locally maintained database at the Max Planck Institute for Solid State Research (MPI-FKF,
Stuttgart) and derived from data shared with some of us (RH and LB) by the company Altmetric
.com on October 30, 2019. Tweets and other metadata associated with them were retrieved
from the Twitter API. Sharing the data is prohibited by licensing issues.

REFERENCIAS

Alperin, j. PAG., Gómez, C. J., & Haustein, S. (2019). Identifying diffu-
sion patterns of research articles on Twitter: Un estudio de caso de
online engagement with open access articles. Public Under-
standing of Science, 28(1), 2–18. https://doi.org/10.1177
/0963662518761733, PubMed: 29607775

Bornmann, l. (2014). Do altmetrics point to the broader impact of
investigación? An overview of benefits and disadvantages of
altmetrics. Journal of Informetrics, 8(4), 895–903. https://doi.org
/10.1016/j.joi.2014.09.005

Candy, B., jones, l., Vickerstaff, v., Larkin, PAG. J., & Piedra, PAG. (2018).
Mu-opioid antagonists for opioid-induced bowel dysfunction in
people with cancer and people receiving palliative care.
Cochrane Database of Systematic Reviews (6). https://doi.org/10
.1002/14651858.CD006332.pub3, PubMed: 29869799

Centers for Disease Control and Prevention. (2018a). 2018 Annual
Surveillance Report of Drug-Related Risks and Outcomes—
United States Surveillance Special Report. Centers for Disease
Control and Prevention, US Department of Health and Human
Services. https://www.cdc.gov/drugoverdose/pdf/pubs/2018-cdc
-drug-surveillance-report.pdf

Centers for Disease Control and Prevention. (2018b). 2018 Annual
Surveillance Report of Drug-Related Risks and Outcomes—
United States Surveillance Special Report. Centers for Disease
Control and Prevention, US Department of Health and Human
Services. https://www.cdc.gov/drugoverdose/pdf/pubs/2018-cdc
-drug-surveillance-report.pdf

Centers for Disease Control and Prevention. (2019a). 2019 Annual
Surveillance Report of Drug-Related Risks and Outcomes—
United States Surveillance Special Report. Centers for Disease
Control and Prevention, US Department of Health and Human
Services. https://www.cdc.gov/drugoverdose/pdf/pubs/2019-cdc
-drug-surveillance-report.pdf

Centers for Disease Control and Prevention. (2019b). 2019 Annual
Surveillance Report of Drug-Related Risks and Outcomes—
United States Surveillance Special Report. Centers for Disease
Control and Prevention, US Department of Health and Human
Services. https://www.cdc.gov/drugoverdose/pdf/pubs/2019-cdc
-drug-surveillance-report.pdf

Centers for Disease Control and Prevention. (2019C). CDC Guide-
line for Prescribing Opioids for Chronic Pain. Centers for Disease
Control and Prevention, US Department of Health and Human
Services. https://www.cdc.gov/drugoverdose/prescribing
/guideline.html

chan, A. k. METRO, Nickson, C. PAG., Rudolph,

j. w., Sotavento, A., &
Joynt, GRAMO. METRO.
(2020). Social media for rapid knowledge dis-
semination: Early experience from the COVID-19 pandemic.
Anaesthesia. https://doi.org/10.1111/anae.15057, PubMed:
32227594

Clarivate Analytics. (2018). KeyWords Plus generation, creation,
a n d c h a n g e s . h t t p s : / / s u p p o r t . c l a r i v a t e . c o m
/ScientificandAcademicResearch/s/article/ KeyWords-Plus
-generation-creation-and-changes?language=en_US

Estudios de ciencias cuantitativas

1507

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

2
4
1
4
8
6
2
0
0
7
8
8
7
q
s
s
_
a
_
0
0
1
6
8
pag
d

.

/

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Investigating dissemination of scientific information on Twitter

davis, C. A., Varol, o., Ferrara, MI., Llamas, A., & Menczer, F.
(2016). BotOrNot: A system to evaluate social bots. WWW '16
Companion, Montréal, Québec, Canada. https://doi.org/10
.1145/2872518.2889302

Didegah, F., Mejlgaard, NORTE., & Sørensen, METRO. PAG. (2018). Investigating
the quality of interactions and public engagement around scien-
tific papers on Twitter. Journal of Informetrics, 12(3), 960–971.
https://doi.org/10.1016/j.joi.2018.08.002

Doleman, B., Leonardi-Bee, J., Heinink, t. PAG., Bhattacharjee, D.,
Lund, j. NORTE., & williams, j. PAG. (2018). Pre-emptive and preventive
opioids for postoperative pain in adults undergoing all types of
surgery. Cochrane Database of Systematic Reviews (12). https://
doi.org/10.1002/14651858.CD012624.pub2, PubMed: 30521692
Ferrara, mi. (2020a). #covid-19 on Twitter: Bots, conspiracies, y

social media activism. arXiv preprint, arXiv:2004.09531.

Ferrara, mi. (2020b). What types of COVID-19 conspiracies are
populated by Twitter bots? First Monday, 25(6). https://doi.org
/10.5210/fm.v25i6.10633

guo, l. (2015). Semantic network analysis, mind mapping and visu-
alización: A methodological exploration of the network agenda
setting model. en l. guo & METRO. McCombs (Editores.), The power of
information networks (páginas. 37–51). Nueva York: Routledge.

guo, l., & McCombs, METRO. (2011). Network agenda setting: A third
level of media effects. Annual Conference of the International
Communication Association, Bostón, MAMÁ.

Haunschild, r., Leydesdorff, l., & Bornmann, l. (2019a). Library and
information science papers as topics on Twitter: A network
approach to measuring public attention. ISSI 2019 – 17th Interna-
tional Conference of the International Society for Scientometrics
and Informetrics, Roma, Italia. https://doi.org/10.2478/jdis-2020-0017
Haunschild, r., Leydesdorff, l., Bornmann, l., Hellsten, I., & Marx,
W.. (2019b). Does the public discuss other topics on climate
change than researchers? A comparison of explorative networks
based on author keywords and hashtags. Journal of Informetrics,
13(2), 695–707. https://doi.org/10.1016/j.joi.2019.03.008

Haunschild, r., Leydesdorff, l., & Bornmann, l. (2020a). Library
and information science papers discussed on Twitter: A new
network-based approach for measuring public attention. Diario
of Data and Information Science, 5(3), 5–17. https://doi.org/10
.2478/jdis-2020-0017

Haunschild, r., Leydesdorff, l., Bornmann, l., Hellsten, I., & Marx,
W.. (2020b). Corrigendum to “Does the public discuss other
topics on climate change than researchers? A comparison of
explorative networks based on author keywords and hashtags”
[j. Informetrics 13 (2019) 695–707]. Journal of Informetrics,
14(1), 101020. https://doi.org/10.1016/j.joi.2020.101020

Haustein, S., Bowman, t. D., Holmberg, K., Tsou, A., Sugimoto,
C. r., & Larivière, V. (2016). Tweets as impact indicators:
Examining the implications of automated “bot” accounts on
Twitter. Journal of the Association for Information Science and
Tecnología, 67(1), 232–238. https://doi.org/10.1002/asi.23456
Havakhor, T., Soror, A. A., & Sabherwal, R. (2018). Diffusion of
knowledge in social media networks: Effects of reputation mech-
anisms and distribution of knowledge roles. Information Systems
Diario, 28(1), 104–141. https://doi.org/10.1111/isj.12127

Hegelich, S., & Janetzko, D. (2016). Are social bots on Twitter polit-
ical actors? Empirical evidence from a Ukrainian social botnet.
Tenth International AAAI Conference on Web and Social Media.

Hellsten, I., & Leydesdorff, l. (2020). Automated analysis of

actor–topic networks on Twitter: New approaches to the analysis
of socio-semantic networks. Journal of the Association for Infor-
mation Science and Technology, 71(1), 3-15. https://doi.org/10
.1002/asi.24207

Hemsley, j. (2019). Followers retweet! The influence of middle-level
gatekeepers on the spread of political information on Twitter.
Política & Internet, 11(3), 280–304. https://doi.org/10.1002/poi3.202
Kamada, T., & Kawai, S. (1989). An algorithm for drawing general
undirected graphs. Information Processing Letters, 31(1), 7-15.
https://doi.org/10.1016/0020-0190(89)90102-6

Kearney, METRO. W.. (2019). tweetbotornot. https://github.com

/mkearney/tweetbotornot (accessed January 28, 2021).

Kollanyi, B., Howard, PAG. NORTE., & Woolley, S. C. (2016). Bots and
automation over Twitter during the first US presidential debate.
Comprop Data Memo, 1, 1–4.

Korda, h., & Itani, z. (2013). Harnessing social media for health
promotion and behavior change. Health Promotion Practice,
14(1), 15–23. https://doi.org/10.1177/1524839911405850,
PubMed: 21558472

Leeds-Hurwitz, W.. (2009). Encyclopedia of communication theory.
In S. W.. Littlejohn & k. A. Foss (Editores.), (volumen. 1, páginas. 899–905). Sage.
Leidl, R. (2019). Social media, bots and research performance.
European Journal of Public Health, 29(1), 1. https://doi.org/10
.1093/eurpub/cky123, PubMed: 30689925

Lokot, T., & Diakopoulos, norte. (2016). News bots: Automating news
and information dissemination on Twitter. Digital Journalism,
4(6), 682–699. https://doi.org/10.1080/21670811.2015.1081822
Moe-Byrne, T., Marrón, j. V. mi, & McGuire, W.. (2018). Naloxone for
opioid-exposed newborn infants. Cochrane Database of System-
atic Reviews (10). https://doi.org/10.1002/14651858.CD003483
.pub3, PubMed: 30311212

Mohammadi, MI., Barahmand, NORTE., & Thelwall, METRO. (2020). Who
shares health and medical scholarly articles on Facebook?
Learned Publishing, 33(2), 111–118. https://doi.org/10.1002
/leap.1271

Mohammadi, MI., Thelwall, METRO., Kwasny, METRO., & holmes, k. l. (2018).
Academic information on Twitter: A user survey. MÁS UNO,
13(5), e0197265. https://doi.org/10.1371/journal.pone
.0197265, PubMed: 29771947

Müller, K., Wickham, h., James, D. A., & Falcon, S. (2017).
RSQLite: SQLite Interface for R. R package version 1.1-2.
https://rsqlite.r-dbi.org/

Murphy, GRAMO., & Salomone, S. (2013). Using social media to facilitate
knowledge transfer in complex engineering environments: A
primer for educators. European Journal of Engineering Education,
38(1), 70–84. https://doi.org/10.1080/03043797.2012.742871
National Institute on Drug Abuse. (2019). What are prescription
opioids? https://www.drugabuse.gov/publications/drugfacts
/prescription-opioids

patel, v., Haunschild, r., Bornmann, l., & Garas, GRAMO. (2020). A call
for governments to pause Twitter censorship: A cross-sectional
s t u d y u s i n g Tw i t t e r d a t a a s s oc i a l – s p a t i a l s e n s o r s o f
COVID-19/SARS-CoV-2 research diffusion. medRxiv. https://doi
.org/10.1101/2020.05.27.20114983

Potnis, D. D., Gala, B., & Deosthali, k. (2020). Investigating “mes-
sage forwarding behavior” of mobile phone users: Exploring the
link between message content, user sentiment, and user intention
to forward messages on social media-based instant messaging
platforms. First Monday, 25(8). https://doi.org/10.5210/fm.v25i8
.10651

Potnis, D., & Tahamtan, I. (2021). Hashtags for gatekeeping of infor-
mation on social media. Revista de la Asociación de Informa-
ción Ciencia y Tecnología, 72, 1234–1246. https://doi.org/10
.1002/asi.24467

R Core Team. (2019). R: A language and environment for statistical
computing ( Versión 3.6.0). R Foundation for Statistical Com-
puting. https://www.r-project.org/

Estudios de ciencias cuantitativas

1508

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

2
4
1
4
8
6
2
0
0
7
8
8
7
q
s
s
_
a
_
0
0
1
6
8
pag
d

/

.

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Investigating dissemination of scientific information on Twitter

R Special Interest Group on Databases (R-SIG-DB), Wickham, h., &

Müller, k. (2018). https://dbi.r-dbi.org/

Rubinstein, A. l., & Carpintero, D. METRO. (2017). Association between
commonly prescribed opioids and androgen deficiency in men:
a retrospective cohort analysis. Pain Medicine, 18(4), 637–644.
https://doi.org/10.1093/pm/pnw182, PubMed: 27516365

Rudd, R. A. (2016). Increases in drug and opioid-involved overdose
deaths—United States, 2010–2015. Morbidity and Mortality
Weekly Report, 65(50–51), 1445–1452. https://doi.org/10
.15585/mmwr.mm655051e1, PubMed: 28033313

Rudd, R. A., Aleshire, NORTE., Zibbell, j. MI., & Matthew Gladden, R.
(2016). Increases in drug and opioid overdose deaths—United
Estados, 2000–2014. American Journal of Transplantation, 16(4),
1323–1327. https://doi.org/10.1111/ajt.13776

Herrero, l. A., Burns, MI., & Cuthbert, A. (2018). Parenteral opioids for
maternal pain management in labour. Cochrane Database of

Systematic Reviews (6). https://doi.org/10.1002/14651858
.CD007396.pub3, PubMed: 29870574

Sugimoto, C. r., Work, S., Larivière, v., & Haustein, S. (2017). Schol-
arly use of social media and altmetrics: A review of the literature.
Journal of the Association for Information Science and Technology,
68(9), 2037–2062. https://doi.org/10.1002/asi.23833

Tahamtan, I., & Bornmann, l. (2020). Altmetrics and societal
impact measurements: Match or mismatch? A literature review.
El profesional de la información (EPI), 29(1). https://doi.org/10
.3145/epi.2020.ene.02

Vargo, C. J., & guo, l. (2017). Networks, grandes datos, and intermedia
agenda setting: An analysis of traditional, partisan, and emerging
online US news. Journalism & Mass Communication Quarterly,
94(4), 1031–1055. https://doi.org/10.1177/1077699016679976
Casa Blanca, El. (2017). Ending America’s Opioid Crisis. https://

www.whitehouse.gov/opioids/

APPENDIX A

We performed our search including the final list of search terms in WoS in three steps (#1, #2,
y #3), como sigue:

#1: We assume that English publications between 2011 y 2019 that contain “Opiate* OR

Opioid*” in their titles are related to opioid research:

(cid:129) (TI=(Opiate* OR Opioid*)) AND LANGUAGE: (Inglés) AND DOCUMENT TYPES:
(Article OR Review) Indexes=SCI-EXPANDED, SSCI, A&HCI Timespan=2011-2019

#2: We then executed the following search in WoS to retrieve additional English publica-
tions that also appeared between 2011 a 2019. This search strategy retrieves schol-
arly publications that contain opioid-related terms in their titles (TI field) and also
contain “Opiate* OR Opioid*” in their title, abstract, or author keywords (TS1). El
asterisk sign (*) at the end of the search terms was used to find variations of the search
terms (p.ej., “Opioid*” finds both “Opioid” and “Opioids”):

(cid:129) (TI=(Narcotic* OR Morphine OR Heroin OR Suboxone OR Subutex OR Kadian OR
Avinza OR Diamorphine OR Fentanyl OR Remifentanil OR Alfentanil OR Meperi-
dine OR Pethidine OR Tramadol OR Ketobemidone OR Hydrocodone OR Vicodin
OR Hydromorphone OR Methadone OR Oxycodone OR OxyContin OR Percocet
OR Oxymorphone OR Opana OR Tapentadol OR Codeine OR Buprenorphine OR
Butrans OR Belbuca OR Propoxyphene) AND TS=(Opiate* OR Opioid*)) Y
LANGUAGE: (Inglés) AND DOCUMENT TYPES: (Article OR Review) Indexes=
SCI-EXPANDED, SSCI, A&HCI Timespan=2011-2019

#3: The above two searches provided us with two sets of opioid scholarly publications.
Sin embargo, there is some overlap between the publications retrieved in steps #1 y
#2. Por lo tanto, we combined the two sets using the OR operator (#1 O #2). Este
approach removed duplicated publications and resulted in 16,889 publicaciones.

1 TS or the topic field searches in title, abstract, author keywords, and KeyWords Plus. KeyWords Plus are
generated by the database provider from the titles of cited documents. “The data in KeyWords Plus are words
or phrases that frequently appear in the titles of an article’s references but do not appear in the title of the
article itself” (Clarivate Analytics, 2018).

Estudios de ciencias cuantitativas

1509

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

2
4
1
4
8
6
2
0
0
7
8
8
7
q
s
s
_
a
_
0
0
1
6
8
pag
d

.

/

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Investigating dissemination of scientific information on Twitter

APPENDIX B

The downloaded data was stored in a local SQLite database file using the R package RSQLite
(Müller, Wickham et al., 2017). Functions from the R package DBI (R Special Interest Group on
Databases (R-SIG-DB), Wickham, & Müller, 2018) were used for sending database queries.
The R package “tweetbotornot” designed by Kearney (2019) was used to detect bots.

APPENDIX C
We used a threshold to exclude the tweets that were not available. Because the tweets’ texts
and meta information of available tweets are usually longer than 90 characters, we used this as
a threshold to filter for available or unavailable tweets. Encontramos 165,660 available tweets
y 7,527 unavailable tweets. También, some tweets were not downloaded because of Twitter’s
internal errors (internal and overcapacity errors). Sin embargo, this applied to only three tweets:
two with the “overcapacity” error and one with the “internal” error. Además, 1,516 tweets
from accounts suspended by Twitter were not included in our analysis.

yo

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

/

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

yo

F
/

/

/

/

2
4
1
4
8
6
2
0
0
7
8
8
7
q
s
s
_
a
_
0
0
1
6
8
pag
d

/

.

F

b
y
gramo
tu
mi
s
t

t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Estudios de ciencias cuantitativas

1510

Descargar PDF