RESEARCH ARTICLE - IA de Investigación especializada en el MIT

ARTÍCULO DE INVESTIGACIÓN

Performance-based publisher ratings and the
visibility/impact of books: Small fish in a
big pond, or big fish in a small pond?

Alesia A. Zuccala1

, Janne Pölönen2

, Raf Guns3

, Vidar Røeggen4

Emanuel Kulczycki5

, Kasper Bruun6

, and Eeva Savolainen7

1Department of Communication, Universidad de Copenhague, Karen Blixens Plads 8, Copenhague, Dinamarca
2Federation of Finnish Learned Societies, Helsinki, Finland
3Centre for R&D Monitoring (ECOOM), Faculty of Social Sciences, University of Antwerp,
Middelheimlaan 1, 2020 Antwerp, Bélgica
4Universities Norway, Stortorvet 2, 0155 Oslo, Norway
5Scholarly Communication Research Group, Adam Mickiewicz University in Poznan, Poznan, Poland
6Roskilde University Library, Roskilde University, Universitetsvej 1 DK-4000 Roskilde, Dinamarca
7Aalto University, Research Services, Otaniementie 9, 02150 Espoo, Finland

Palabras clave: book evaluations, book publishing, performance-based funding, basado en el rendimiento
research funding systems (PRFS)

un acceso abierto

diario

Citación: Zuccala, A. A., Pölönen, J.,
Guns, r., Røeggen, v., Kulczycki, MI.,
Bruun, K., & Savolainen, mi. (2021).
Performance-based publisher ratings
and the visibility/impact of books:
Small fish in a big pond, or big fish in
a small pond? Quantitative Science
Estudios, 2(2), 588–615. https://doi.org
/10.1162/qss_a_00134

DOI:
https://doi.org/10.1162/qss_a_00134

ABSTRACTO

Recibió: 28 Abril 2020
Aceptado: 24 Marzo 2021

Autor correspondiente:
Alesia A. Zuccala
a.zuccala@hum.ku.dk

Editor de manejo:
Juego Waltman

Derechos de autor: © 2021 Alesia A. Zuccala,
Janne Pölönen, Raf Guns, Vidar
Røeggen, Emanuel Kulczycki, Kasper
Bruun, and Eeva Savolainen.
Publicado bajo Creative Commons
Atribución 4.0 Internacional
(CC POR 4.0) licencia.

La prensa del MIT

This study compares publisher ratings to the visibility and impact of individual books, based
en un 2017 data set from three Nordic performance-based research funding systems (PRFS)
(Dinamarca, Norway, and Finland). Although there are Journal Impact Factors ( JIFs) for journals,
there is no similar indicator for book publishers. National publisher lists are used instead to
account for the general “quality” of books, leading to institutional rewards. Pero, just as the JIF is
not recommended as a proxy for the “citedness” of a paper, a publisher rating is also not
recommended as a proxy for the impact of an individual book. We introduce a small fish in a
big pond versus big fish in a small pond metaphor, where a “fish” is a book and “the pond”
represents its publishing house. We investigate how books fit on this metaphorical fish and
pond continuum, using WorldCat holdings (visibility) and Google Scholar citations (impacto),
and test other variables to determine their predictive value with respect to these two indicators.
Our statistics show that publisher levels do not have predictive value when other variables are
held constant. This has implications for PRFS and book evaluations in general, así como
ongoing developments related to a newly proposed international publisher registry.

INTRODUCCIÓN

The evaluation of scholarly books is a growing theme across Europe, mainly because of their
significance to SSH scholarly communities (Zuccala & Robinson-Garcia, 2019) but also due to
their registration in national databases (Giménez-Toledo, Mañana-Rodríguez et al., 2016). Este
has inspired a number of scholars to investigate book publishing more intensively, often in relation
to national performance-based research funding systems (PRFS) (Zuccala, Giménez-Toledo, &
Peruginelli, 2018). Some have investigated the prestige of publishers (Giménez-Toledo &
Román-Román, 2009) as well as their specialization (Mannana-Rodriguez & Giménez-Toledo,
2018), and others have explained how peer-review labels in books can be used as an aid to
evaluation procedures (Kulczycki, Rozkosz et al., 2019; Verleysen & Engels, 2014).

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
5
8
8
1
9
3
0
6
9
3
q
s
s
_
a
_
0
0
1
3
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Performance-based publisher lists

Peer review is as vital to book publishing as it is to the production of journal articles; hence,
the need to identify “quality” publishing outlets (es decir., those adhering to review standards) es
becoming a big incentive for establishing a European-wide publisher registry (Giménez-
Toledo, Sivertsen, & Mañana-Rodríguez, 2019). Before we introduce the purpose and potential
design of this proposed registry (en la sección 3), we begin with a brief explanation of what
we call a top-down (basado en el rendimiento) versus bottom-up (individual level) approach to the
evaluation of scholarly books. Following this, we introduce our study framework and research
hypotheses.

A top-down versus bottom-up evaluation can be explained on the basis of what is known
currently about evaluations in general, and about research articles. Por ejemplo, a bottom-up
approach to evaluation starts at a low level of aggregation, where the lowest unit is the research
paper, or the oeuvre of a particular researcher. En cambio, we can also take a top-down
acercarse, or higher level of aggregation, where the focus is on a department, university, or journal
(van Leeuwen, 2007). Different approaches lead to different insights into scholarly performance
and impact (p.ej., Zuccala, costas, & van Leeuwen, 2010).

At the aggregate level of a journal, the most commonly used proxy for “quality” is the journal
impact factor ( JIF). Sin embargo, it is well known that the JIF is a relatively poor predictor of the
citation impact for individual articles (Hegarty & Walton, 2012). According to Pudovkin
(2018) it is “a great mistake to consider the JIF value as a proxy for the citedness of individual
papers in a journal” or “as a proxy for the influence of an author” (pag. 1). Each research paper is
likely to achieve its own degree of impact regardless of the journal in which it was published,
with a “minority receiving a high number of citations, and a majority of documents receiving
the few remaining citations” (Sugimoto & Larivière, 2018, pag. 96). Whilst Waltman and Traag
(2021) statistically show that the impact factor can be an accurate indicator of the value of an
individual article (es decir., more than the number of citations it has received), emphasis is still
placed on this indicator’s sociotechnical implications, not the statistics.

In the PRFS of the Nordic countries, specifically Norway, Dinamarca, and Finland, atten-
tion has been given to the development of journal lists with level assignments as opposed to
JIFs or other journal-level indicators. Yet similar to the JIF, even if a correlation is found at
the macrolevel between journal level ratings and citation scores, there are still many highly
cited individual articles in lower rated journals, and articles without citations in higher level
journals (Aksnes, 2017; Auranen & Pölönen, 2012).

In the case of book publishers, there is no established “publisher impact factor” because
commercial citation databases (es decir., Web of Science/Scopus) do not accurately index pub-
lishers, international monographs, and book chapters. Journals are also far more homogeneous
as publication channels than publishers are for books. To account for book publications, muchos
European nations have, like journal ratings, developed publisher level ratings for their PRFSs
(Hicks, 2012). In Denmark, Norway, and Finland, book publishers are assigned to level
categories by experts in a given field (p.ej., niveles 0 a 3), and the levels determine the number
of performance points (p.ej., nivel 1 = 5 puntos) (Sivertsen, 2016). All points are allocated on an
institutional level, eso es, to a university, which in turn leads to performance-based funding
from the country’s government. As indicators, these ratings are complementary to journal
ratings, and also comparable to the JIF in the sense that they attempt to account for differ-
ences in the average “quality” of outputs at the level of the publication channel (see more in
Sección 2.2).

Little is known; sin embargo, about the extent to which publisher levels relate to the visibility or
impact of individual books. Respectivamente, we introduce a small fish in a big pond versus big fish

Estudios de ciencias cuantitativas

589

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
5
8
8
1
9
3
0
6
9
3
q
s
s
_
a
_
0
0
1
3
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Performance-based publisher lists

in a small pond metaphor, where a “fish” is a book and “the pond” represents its publishing
house (ver figura 1). Utilizing this metaphor, we investigate book titles:

(cid:129) that were published by a variety of presses and publishing houses; some from the “big
pond” of level 2 a nivel 3 publicación (presumably international), and others from the
“small pond” of level 0 a nivel 1 publicación (presumably national or regional), y
(cid:129) that can be evaluated according to different measures (indicators) of an individual book’s
overall visibility and impact as a “big fish” or “small fish.” The indicators that we have
retrieved include WorldCat holding and country distribution counts, Google Scholar
and Scopus citations, and PlumX usage, captures, menciona, and social media counts.

Our assumption, based on data from three Nordic PRFS databases (Dinamarca, Norway, y
Finland), is that there can be smaller fish books from the big pond of publishing as well as bigger
fish books in the small pond of publishing. Cifra 1 illustrates this on two axes, with high visibility
or impact versus lower visibility or impact along the vertical axis, and publisher levels from 0 a 2
(national/regional) versus levels 2 a 3 (international) on the horizontal axis. Tenga en cuenta que, dentro
these categories, books will, in reality, exist on a continuum. We also expect that the publisher
nivel, like the JIF, will be a limited predictor of an individual book’s impact and visibility.

This research has implications for the use of book publisher ratings in research evaluation,
where the book as scholarly output is valued according to its publisher. A book’s value, cómo-
alguna vez, can also be determined by an expert review of its content, by librarians who select them for
library collections, and through the application of alternative metrics. There are considerable
implications for researchers as well. Many are concerned with the overall visibility, perceived
scholarly/cultural relevance, and impact of their book titles. With this, there are three primary
aims to this study.

The first aim is to determine the degree to which a PRFS rating is related to an individual
book title’s visibility or impact. The second aim is to identify the characteristics of a publisher

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
5
8
8
1
9
3
0
6
9
3
q
s
s
_
a
_
0
0
1
3
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Cifra 1.

International publishing versus national publishing “ponds” and “smaller/bigger fish” books.

Estudios de ciencias cuantitativas

590

Performance-based publisher lists

or book, in addition to the PRFS rating, which may play a role in a book’s visibility or impact.
And finally, the third aim is to provide new insight into the worldwide library distribution
potential of publishers, as they fit into the overall schema of PRFSs. Some of what is clarified
about performance-based systems, via our individual level analyses, might help to identify the
essential criteria for a publisher’s inclusion in the registry. Currently established as well as future
PRFSs might also benefit from this registry’s development.

2. BACKGROUND LITERATURE

2.1. Publisher Prestige, Specialization, and Quality
For the scientist preparing a research article, emphasis is often placed on selecting a “high-impact”
journal. The choice of publisher is equally as important to the scholar who writes a book. Todavía, el
prestige, specialization, or quality of a scholarly book publisher is not easily recognized, and has
been less transparent to evaluation communities than the accounting and ranking systems currently
in place for journals (p.ej., the JIF, SNIP, SJR; Archambault & Larivière, 2009; garfield, 2006;
González-Pereira, Guerrero-Bote, & Moya-Anegón, 2010; Moed, 2011). Además, in some fields
a consensus regarding publisher prestige, specialization, and quality can be reached, while in
others not. As publishers’ names are not adequately standardized, categorized, or ranked in sources
such as the Book Citation Index, or in other commercial databases (p.ej., Scopus), it is therefore
useful to review how the book publishing industry works.

Thomson’s (2005) volume Books in the digital age provides a thorough overview, though it
focuses primarily on English-language publishing across Great Britain and the United States.
Little is known about scholarly book publishing from a broader international or multilanguage
perspectiva. What the volume does do, sin embargo, is convey how prestige, specialization, y
quality cannot be easily extricated from other factors, such as a publisher’s economic status, histor-
ical context, approach to commercialization or lack of commercialization, and symbolic capital.
There is and continues to be a difference between the directives of a university press versus a com-
mercial publisher, with the latter “subject to the commercial constraints of the organization and the
financial objectives set by its management” (Thomson, 2005, pag. 87). Details pertaining to these
different units could potentially be added to a European publisher registry, and this is discussed
in more detail in Section 3. Primero, we explore some of the background literature relevant to the
registry objective.

2.1.1. Prestige

Publisher prestige is often described as a feature that scholars intuitively recognize, if not think
acerca de, when preparing a new book. Studies designed to investigate this concept have either
utilized interviews or distributed surveys to academic communities concerning scholars’ rating
of specific publishers (Garand & Giles, 2011; Giménez-Toledo & Román-Román, 2009;
Goodson, Dillman, & Hira, 1999). Research by Goodson et al. (1999) found that interviewees
were candid about their perceptions, explaining that there can be “a decided ‘unwritten’ hierarchy
of publishers, not only in political science, en efecto, not only in academia” (pag. 257). With this notion
of prestige, there is often an implied ranking. A publishing house that is at the “top” of the hierarchy
is usually prestigious, but how a publisher is perceived in the minds of scholars might not neces-
sarily turn up in the exploration of hard data. Zuccala, Guns et al. (2015) have therefore studied this
by examining how citations could potentially be the focus of a publisher ranking exercise. Su
research has shown that citations do, hasta cierto punto, reflect prestige, though they can also indicate
the economic strength of the publishing house, based on revenues and international status (p.ej.,
a head office in one country, and subsidiaries in others).

Estudios de ciencias cuantitativas

591

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
5
8
8
1
9
3
0
6
9
3
q
s
s
_
a
_
0
0
1
3
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Performance-based publisher lists

The most lucrative, historically famous, and productive publishing houses, across many
scholarly subjects, produce books that are cited frequently, with the corollary that works pub-
lished by smaller publishers might almost never be cited at all (Torres-Salinas et al., 2014). Este
is because publishing houses in general tend to be concentrated. De hecho, the largest and more
prolific publishers of journals are now concentrated in an oligopoly (Larivière, Haustein, &
Mongeon, 2015). Research by Guns (2018) found a similar tendency with publishers of peer-
reviewed books registered in the Flemish VABB-SHW. Still, when assessing book publishers
who dominate market share—such as Cambridge University Press, Prensa de la Universidad de Harvard,
or Oxford University Press—rank changes have been observed (at least in the field of history),
with a shift from overall citation counts to average citation counts per book (Zuccala et al.,
2015). Certain books and published topics are thus likely to receive more attention than others,
even when produced by a top publisher.

2.1.2. Specialization

Specialization offers a different perspective on publishing and can occur in environments of
interorganizational rivalry. To avoid unnecessary rivalry, some publishers establish a market
niche by producing texts for specific academic fields and subjects or install multiple imprints.
Rodaballo, Por ejemplo, is a Leiden-based publishing company, known for producing social sciences
and humanities books (in history, philosophy, religious studies, theology and world Christianity,
literature and cultural studies, etc.).

A publisher may also become specialized or differentiated for other reasons. Thomson (2005)
describes how AC2, a medium-sized academic publisher from the United Kingdom, experimentado
a decline in scholarly monograph sales and thus focused on cutting costs by both simplifying and
standardizing production processes. A separate organizational unit was developed at this firm,
which was designed to focus “exclusively on publishing monographs”: a strategy known as
“sequestering the monograph” (Thomson, 2005, pag. 163).

For collection development librarians in particular, publisher specialization is said to be prob-
lematic (Kousha & Thelwall, 2015; Metz & Stemmer, 1996). According to Metz and Stemmer
“judgements are subjective and vary greatly from one library to another.” Moreover, “publishers
are so specialized that only those familiar with a field can evaluate their work” (pag. 246).

Hasta ahora, little research attention has been given to the link between publisher specialization
and research evaluation (p.ej., Giménez-Toledo, Mañana-Rodríguez, & Tejada-Artigas, 2015;
Mannana-Rodriguez & Giménez-Toledo, 2018), though scholars primarily understand that “few
presses can successfully publish in all fields” (Parsons, 1990, pag. 3). Sin embargo, when comparisons
are made between commercial publishers and university presses, it is the latter that are less likely to
become fully specialized. University presses in general cannot afford to be specialized when
required to serve the research areas or fields covered by their home institutions (Lockett &
Speicher, 2016; Mannana-Rodriguez & Giménez-Toledo, 2018). Al mismo tiempo, hay
drawbacks, if not challenges, associated with the opposite “multidisciplinary” publishing
acercarse, many of which relate to the editor’s role. When editors can successfully concentrate
on books from specialized research areas, they can “become more knowledgeable concerning
(new research) developments,” build reputations that will attract the best authors, and grow “more
effective in cultivating and nurturing personal acquaintance with scholars” from specialty areas
(Parsons, 1990, pag. 4).

To find out if university presses are indeed less specialized, or “multidisciplinary,” Mannana-
Rodriguez and Giménez-Toledo (2018) developed and applied to a set of Spanish publishers
(norte = 1952), what they call an “entropy-based indicator” (pag. 19). All publishers in the list, cual

Estudios de ciencias cuantitativas

592

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
5
8
8
1
9
3
0
6
9
3
q
s
s
_
a
_
0
0
1
3
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Performance-based publisher lists

was downloaded from the DILVE database (developed by the Federation of Publishers’ Guilds
of Spain), had been producing texts for the social sciences and humanities. Además, todo,
including commercial and university presses, were tested for their specialization on the basis
of the unevenness of their distribution of titles across various disciplines. The following expla-
nation was given:

let’s suppose that two publishers count, cada, with one hundred titles in ten disciplines.
Publisher ‘A’ published ten titles in each of the ten fields. Publisher ‘B’ published 91 títulos
in one field and 1 in each of the other 9 campos. Since ‘B’ has a greater concentration of titles
in one field, it is more specialized (in that field) than ‘A’. The unevenness of the distribution of
titles across disciplines is greater in ‘B’ than it is in ‘A’ and ‘B’ can be defined as more special-
ized than ‘A’. For the purposes of this article higher entropy values point to more multidisci-
plinary publishers; while lower entropy values indicate those that are more specialized.

The results of this study showed that many commercial publishers in Spain tended to have a
very high concentration of titles by field of knowledge. This was especially true for those pro-
ducing books for fields such as law and education. With other research fields, production pro-
cesses tended to be less concentrated (p.ej., fields such as Archaeology, Library and Information
Estudios, Antropología). As expected, Mannana-Rodriguez and Giménez-Toledo (2018) found
that many university presses from Spain were significantly more multidisciplinary than their
commercial counterparts. While the authors recognize that the “mission” of a university press
is to represent the intellectual range of interests of its home institution’s researchers, they also
suggest that with “efficient decision-making processes” many presses might “benefit from the
opportunities brought by specialization” (Mannana-Rodriguez & Giménez-Toledo, 2018, pag. 28).

2.1.3. Quality

The term quality when applied to books or monographs, is based on “the overall intellectual
and editorial quality of a publisher’s monographic offerings, reflecting the expertise of typical
autores; the persuasiveness of evidence; the intellectual level of discourse; the tendency of a
publisher’s titles to be influential in their fields; and the degree of editorial care” (Metz &
Stemmer, 1996). While “editorial care” has always been the sine qua non for book quality, él
is only recently that scholars have become more concerned with peer review, and isolating
review procedures as a standard of quality for book publishers. Para tal fin, there has been a
movement towards peer review stamping or labeling, given that “there is no straightforward
way to decide whether [books] have been subjected to [Revisión por pares]" ( Verleysen & Engels,
2013, pag. 428). The impetus for labeling books stems from the fact that “virtually all high-esteem
journals use some type of [Revisión por pares] to assure content quality of published articles” (pag. 428).
In this sense the production of a new book is likened to a research article, with all “double-blind,
single blind, abierto, and signed variants” that are “regarded by many as the quintessential mech-
anism to safeguard academic publishing standards” for journals (pag. 428).

In Flanders, which is the Northern Dutch-speaking region of Belgium, it was decided that to
achieve a comprehensive coverage of regional academic outputs in the Flemish Academic
Bibliographic Database for the Social Sciences and Humanities ( VABB-SHW), both books
and articles should be included. De este modo, with the inclusion of books it was considered necessary
to know which book publishers were adhering to the criteria set forth in the legislation, incluido
the use of peer review. A limited number of mostly international publishers are currently con-
sidered to use peer review, pero en 2010 the Publishers Association of Flanders “introduced a

Estudios de ciencias cuantitativas

593

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
5
8
8
1
9
3
0
6
9
3
q
s
s
_
a
_
0
0
1
3
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Performance-based publisher lists

quality label for individual books guaranteeing the peer-reviewed character of their contents, el
so-called “GPRC label” (Benelux Trademark No. 0916696)" ( Verleysen & Engels, 2013, pag. 429).
In Finland, the Federation of Finnish Learned Societies (TSV) has also developed a similar label,
whereby scholarly publishers indicate themselves which of their books and articles have been
peer reviewed (Kulczycki et al., 2019).

2.2. Current Evaluation Systems for Books

2.2.1. Visibility and impact indicators

Although individual-level indicators for articles and authors have been studied frequently
(Wildegaard, 2019), there is less certainty about the reliability of indicators at the individual
level of a book (Zuccala & Robinson-Garcia, 2019, páginas. 720–723). Even if citation counts ac-
crue for books, there has not been any formal measure of a citation window. We do not know
the time period in which books receive their peak citations, or if and when there is a normal
decline. Además, the same problem with correlating journal article citations and social me-
dia counts holds true for books (see Thelwall, Haustein et al., 2013). Social media mentions
occur more rapidly than citations, and thus each measures different phenomena: visibility with
el primero (Leonardi, 2014), and impact with the latter (waltman, 2016).

The growth and development of individual-level book indicators depends also on how well
data sources are compiled and updated, and how reliable they are for extracting “clean” counts
for producing valid statistics. If the databases are incomplete or inaccurate, an indicator is less
likely to be a sound proxy for the concept it measures. We need to test and retest counts or sta-
tistics for books at an individual level, with the hope that data sources are improving. Para en-
postura, it has been known for a long time that books, in comparison to journal articles,
represent a lower percentage of what is indexed on social media platforms. Sin embargo, “as of
Julio 2018, Altmetric.com has tracked attention for more than 829,000 books and 80,000 libro
chapters across a wide range of subjects” (Konkiel & Adie, 2018, pag. 2). Similar to journal articles,
statistics show that “more than 70% de [mentions to books] occur on Twitter” (pag. 3). At the time of
este estudio, we did not have access to Altmetric.com, and thus chose to use PlumX with the un-
derstanding that it is of similar value (Torres-Salinas et al., 2017a, 2017b).

2.2.2. Publisher level ratings in Denmark, Norway, and Finland

Dinamarca (en 2011), Finland (en 2012), and Norway (en 2005) have all introduced PRFS where
part of the annual core-funding from each state is allocated to universities based on research
producción (Hicks, 2012; Norwegian Association of Higher Education Institutions, 2004;
Uddannelses-og-Forskningsministeriet, 2018). In all of these countries, institutional publication
data are used at a national level, because unlike the commercial databases (es decir., Web of Science
y Scopus), national databases and Current Research Information systems (CRIS systems) sup-
port a more comprehensive coverage of outputs in all publication languages.

For all three countries, the PRFS relies on research communities to identify peer-reviewed
publications channels, and to indicate differences in their perceived quality, prestige, and im-
pact. Nonpeer-reviewed channels are either excluded from the authority list or indicated as
nivel 0 o [−], whereas channels approved to be peer-reviewed are categorized as level 1. El
leading international channels are categorized as level 2, and in Finland and Denmark a small
subset of top channels is also categorized as level 3. Sin embargo, in Denmark the level 3 cate-
gorization does not concern book publishers; only journals.

There are some minor differences, among the Nordic countries, in how the channels for
book publications are handled. All countries have an authority list of book publishers with

Estudios de ciencias cuantitativas

594

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
5
8
8
1
9
3
0
6
9
3
q
s
s
_
a
_
0
0
1
3
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Performance-based publisher lists

Mesa 1.
and Norway

Nivel
3 Top

2 Leading

1 Basic

0/− Not approved

Total

Level categories and number of classified book publishers in Denmark, Finland,

Dinamarca
Not used

1,089

Not used

1,163

Finland
13

1,229

1,905

3,240

Norway
Not used

1,583

1,514

3,183

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
5
8
8
1
9
3
0
6
9
3
q
s
s
_
a
_
0
0
1
3
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

level ratings, but the Danish list is slightly more exclusive, as it does not contain level 0/−,
whereas Finland and Norway do include a “0” category. Note that for Norway, this category
“0” is made up of publishers that have either not applied for inclusion on the publishers list or
have not been approved. También, for Denmark, the number of publishers assigned to levels 1 y
2 is smaller than in Finland and Norway (ver tabla 1), while Finland, unlike the other coun-
intentos, has chosen to include the registration of academic textbooks as part of its PRFS.

The level 2 list of book publishers was first produced in Norway, and it was also used as the
basis for the level 2 categorization of publishers in Denmark and Finland. Sin embargo, hay
some differences. Por ejemplo, Elsevier and Springer are rated as level 2 in Finland but have
been assigned to level 1 in Norway and Denmark. Only Finland has a small subset of level 2
publishers categorized as level 3.

Another difference is that Denmark also lists more than 1,500 book series of mainly inter-
national book publishers, while Finland lists book series mainly from Finnish book publishers.
Norway also lists book series, and this is done in two different ways, depending on the pres-
ence or absence of a unique ISSN. If a book series has a unique ISSN, the series is included in
the field-specific journal list. This also means that a book series with an ISSN may have a dif-
ferent level from its publisher. If a book series does not have a unique ISSN (only an ISBN), él
will have the same level as its publisher. If articles in a book are connected to a series with an
ISSN, the articles have the same weights as in journals. All of this means that in some countries
the weight of monographs and articles in books in the PRFS are determined on the basis of the
book series, and not only on the publisher. Además, there are differences in the treatment of
national languages. In Finland, four leading Finnish and Swedish language SSH publishers,
and a few national language book series, are categorized as level 2, while in Norway and
Denmark channels for national language books are placed at level 1.

There are some notable similarities and differences between the countries in the PRFS
weighting of book publications. Dinamarca, Norway, and Finland all give monographs a higher

Mesa 2.
and Norway

Publication points for monographs according to publisher levels in Denmark, Finland,

Nivel
3 Top

2 Leading

1 Basic

Dinamarca
Not used

0 Not approved

Not used

Finland
16

0.4

Norway
Not used

Not used

595

Estudios de ciencias cuantitativas

Performance-based publisher lists

weight than journal articles. Sin embargo, the difference between level 1 y nivel 2 is stronger in
Finland (3-fold, and 4-fold for level 3) than in Denmark and Norway (1.6-fold; ver tabla 2).
Por otro lado, the weight of level 1 monographs compared to level 1 journal articles is
slightly stronger in Denmark and Norway (5-fold) than in Finland (4-fold). In Finland, the PRFS
includes all publications that the universities report to the ministry as being peer-reviewed (es decir.,
self-reported by researchers and checked by data-collection personnel), irrespective of the pub-
lisher levels. Por lo tanto, monographs published with a level 0 publisher may generate points
(0.4) to a university, while in Denmark and Norway, monographs that have not been approved
at either level 1 or level 2 are excluded from their PRFS.

With respect to publisher evaluation, there are more differences. In all three countries the
categorization of journals/series as level 2 (o 3) is the responsibility of field-specific expert
panels. Sin embargo, for book publishers at level 2, the assignment processes in Norway and in
Denmark are carried out by an academic committee (es decir., a select group of appointed re-
searchers across all fields), not by the expert panels themselves. This is done because pub-
lishers in most contexts publish books in many different fields. Field-specific journal experts
are invited to suggest levels also for publishers, but the final decision is made with an inter-
disciplinary perspective. In Finland, all book publishers approved to a level 2 y nivel 3 como-
signment are done collectively by panel chairs, based on a preliminary proposal by SSH fields.

There are also some limitations relevant to the number of book publishers assigned to and eval-
uated at level 2 (o 3) in the three countries. In the case of journals and series, nivel 2 can only be
comprised of at most 20% of the world’s output of each panel’s channels. In the case of book pub-
lishers, nivel 2 is determined somewhat differently. In Norway, there is a balanced approach to
fields when a larger share of its publications comes as books. Because books are more important
to scholars from the SSH than the other fields, most of the suggestions to change level for publishers
are received from the panels in SSH. These are also fields that are not sufficiently covered by interna-
tional statistics, such as Scopus and Web of Science. In these fields, nivel 2 is calculated by including
book and journal publications from national output. The limitation of book publishers on level 2 is then
determined by a simulation across fields. For both Norway and Denmark, the primary rule is that no
national publisher is to be categorized at level 2. In Finland, it has been broadly stated that around
10% of level 1 publishers can be placed on level 2, of which 1% are assigned to level 3.

The publication channel ratings in Denmark, Finland, and Norway attempt to stimulate both
quality and quantity publishing at an institutional level. The purpose is to make it more rewarding
for a university to have publication activity that focuses on channels with “more stringent require-
ments related to the originality and quality of submitted manuscripts” (Norwegian Association of
Higher Education Institutions, 2004). To a smaller extent, they describe differences in the average
quality and impact of outputs at the level of publication channel, assuming that individual outputs
can have a higher or lower quality than this average. The presumption is—from the perspective of
a PRFS—that on average, output from level 2 publishers is of higher quality than that from level 1
publishers, but there are individual exceptions to both directions.

3. DEVELOPING AN INTERNATIONAL PUBLISHER REGISTRY

En 2016, the COST Action ENRESSH (European Network for Research Evaluation in the Social
Sciences and Humanities) was established, and throughout its 4-year period it has served as a
critical forum for European researchers with an interest in how the social sciences and human-
ities (SSH) generate and disseminate knowledge; how scientific and societal interactions char-
acterize different SSH disciplines; and the types of transparent, adapted methods needed for the
evaluation of these different disciplines. The Action is comprised of four working groups, con

Estudios de ciencias cuantitativas

596

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
5
8
8
1
9
3
0
6
9
3
q
s
s
_
a
_
0
0
1
3
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Performance-based publisher lists

one dedicated specifically to databases and the uses of data for understanding SSH research. En
light of this topic, several ENRESSH members have recently conferred about the development
of an international publisher registry.

While this registry has not been officially established, some principal ideas have been put
adelante; Por ejemplo, that it shall be dynamic and interactive, and that publisher entries would in-
clude those that support the research quality standards of SSH on the basis of their peer review
and publishing practices (ENRESSH, 2019; Giménez-Toledo et al., 2019). The information reg-
istered would come from the publishers themselves, from national bibliographic databases and
legal deposit libraries where publications from research institutions are recorded, and from the
scholarly community by feedback given either directly by the authors or through CRIS systems
(ENRESSH, 2019). Además, registered publishing firms can be nationally or internationally ori-
ented, albeit there is still a need to decide upon the types of information added and updated,
such as procedures for peer review and statistics based on bibliographic information.

3.1. Use of a Publisher Registry for Evaluating Books

This idea of a publisher registry has strong implications for a top-down approach to the eval-
uation of books, currently used by Nordic PRFS. It may also add value to bottom-up, individual
level book assessments.

One of the main issues linked to PRFS is that different countries have unique and often very
different publisher lists. Research by Mañana-Rodriguez and Pölönen (2018) has examined
this earlier, thus providing the groundwork for what they call “future efforts toward supra-
national combinations of publishers’ lists” (pag. 643). A preliminary test of how to “merge”
the national publisher lists of Spain and Finland showed that both were quite different, pero
one of the added problems that they found concerned the way that publishers may be rated
from a field-specific standpoint.

Despite the fact that some publisher lists are coordinated to be similar (p.ej., Norway’s and
Denmark’s), while others are disparate (es decir., Spain’s and Finland’s), we are not at the stage of
“supra-national” mergers. Por eso, the initial benefit of a publisher registry is to first provide
eruditos, administradores, and research evaluators, including publishers and their associations, con
a comprehensive resource comprised of international academic publishing houses. There is an
ongoing discussion pertaining to the type of publishers to be included, and the general consensus
is that they should be “scholarly,” with established and accepted peer review standards. For an
individual scholar, this would make for a valuable advisory system, with publishing standards and
transparent peer review processes clearly flagged.

Although more emphasis has been placed on the publisher registry’s benefit to PRFS (es decir.,
potentially unifying PRFS), it is still critical to understand what scholars expect at an individual
level after they select a publisher, and their book is available to both scholarly and lay com-
munities. By investigating the visibility and impact of an individual title, we are better able to
explain potential misuses of publisher levels and ratings, and point to reasonable uses.

4. METHOD

4.1. Recopilación de datos

This study focuses on 743 academic/scholarly books published in 2017 by researchers affili-
ated with Danish, Finnish, and Norwegian universities. We collected 263 book titles (mono-
graphs; books in series; textbooks) registered in the Danish National database of scholarly
outputs for the year 2017, 221 book titles from the Finnish registry, y 259 desde el

Estudios de ciencias cuantitativas

597

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
5
8
8
1
9
3
0
6
9
3
q
s
s
_
a
_
0
0
1
3
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Performance-based publisher lists

Norwegian registry for the same year. These data were chosen based on the fact that all pub-
lished titles can be matched to a national publisher list, and all receive performance points vis-
à-vis the different research evaluation systems. We have also limited the published book titles
to the year 2017 to ensure a similar time frame for the collection of broad impact indicators.

Data collection first involved a manual retrieval process for all Google Scholar citations and
PlumX indicators; for library holdings and country distribution counts, data were delivered
directly from the OCLC WorldCat. All of the individual level book indicators are outlined
and described in Table 3.

In addition to the statistics, each title was coded according to language of publication, sujeto
área, and type of publisher, with private firms categorized as “commercial” and university
presses, research institutes, the World Bank, museums, and foundations categorized as “non-
commercial.” As the subject area or discipline of each book title was not recorded in a uniform
way in the Finnish and Norwegian registries, and not at all in the Danish registry, we obtained
Library of Congress or Dewey Decimal Classification numbers for the PRFS from OCLC
WorldCat. Juntos, these two types of classifications accounted for only 49% of the titles.

PubLanguage

PublisherType

SubjectClass

Mesa 3.

Individual-level indicator descriptions

Publication descriptors

Coded as 1 = English / 0 = Non-English

Coded as 1 = Commercial / 0 = Noncommercial

Library of Congress (p.ej., ML423.B33); Dewey Decimal Classification

(p.ej., MARC format: $a302.35 $223)

SubjectArea

Coded as follows:

1 = Accounting/Business/Management; 2 = Arts/Architecture/Design;
3 = Computing/Maths/Science; 4 = Economics; 5 = Education; 6 = Geography;
7 = Geology; 8 = Health/Medicine; 9 = History; 10 = Law; 11 = Linguistics;
12 = Literary Studies; 13 = Media/Communication; 14 = Philosophy; 15 = Political
Ciencia; 16 = Psychology/Social Sciences; 17 = Religion; 18 = Rhetorics;
19 = “Other” (es decir., Research Methods; Interdisciplinary Studies; Digital Humanities)

WorldCat

A count of how often the book is held in international libraries.

A count of book holdings at the level of a specific country (p.ej., when a book is held
in five libraries in Denmark and two libraries from Sweden, the book has been
distributed in libraries across two countries).

Library holding

Country distribution

Citations

A count of citations per individual book

Google Scholar and Scopus

PlumX altmetrics

Usage counts

Captures

Mentions

Social Media

A signal for whether or not anyone is reading the book. Abstract Views; Full Text Views;

Link-outs (EBSCO). (Nota: WorldCat holdings not included—retrieved separately)

Bookmarks (p.ej., Mendeley); Added to Goodreads library

Amazon or Goodreads Reviews; Blog mentions; Wikipedia references

Tweets; Shares and comments (Facebook)

Estudios de ciencias cuantitativas

598

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
5
8
8
1
9
3
0
6
9
3
q
s
s
_
a
_
0
0
1
3
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Performance-based publisher lists

We therefore used a combination of information (es decir., book title, LC or Dewey Class; WorldCat) a
categorize each book title according to one primary subject area.

A date stamp (p.ej., 2019.05.22) was added to our data table to indicate when PlumX counts and
Google Scholar citation counts were retrieved manually, as well as URLs. En algunos casos, the PlumX
indicators for an individual book were obtained using a DOI, and in other cases with an ISBN (el
ISBN was only useful if the book’s DOI had already been traced by PlumX). If a Google Scholar or
PlumX link led to no record or an inactive page, we left the associated data cell null. If the URL was
active and the book title was “traceable” via the platform, we recorded a value of zero or greater.

5. DATA ANALYSES

5.1. Descriptive Statistics and Correlations

For the publication year of 2017 (Cifra 2), 167 (63%) books from the Danish National repos-
itory were published with a level 1 publisher, y 96 (37%) were published with a level 2
publisher. A similar distribution can be seen in Norway, con 184 (71%) published at level
1 y 75 (29%) published at level 2. By comparison, el 2017 set of books retrieved from
the Finnish National Repository, includes those produced by a level 0 (norte = 52; 24%), nivel
1 (norte = 59; 27%), nivel 2 (norte = 63; 28%) y nivel 3 (norte = 47; 21%) publisher.

Mesa 4 y figura 3 show that the WorldCat holdings indicator yielded the most counts overall
(norte = 692) with the lowest missing values (norte = 51). A large number of titles were traced on Google
Scholar, but citations per book were typically much lower (Mean = 8 and Median = 2) than
WorldCat holdings. Sin embargo, both low and missing values are likely to occur when collecting
citations for books within a 2-year citation window.

PlumX yielded relatively high usage counts (Mean = 85 and Median = 26) and captures
(Mean = 16 and Median = 4) for some individual books, but a particularly high percentage
of data values were missing from this platform (es decir., aproximadamente 63%). For journal articles,

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
5
8
8
1
9
3
0
6
9
3
q
s
s
_
a
_
0
0
1
3
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Cifra 2. Number of books and PRFS points per country publisher level (2017).

Estudios de ciencias cuantitativas

599

Performance-based publisher lists

Mesa 4.

Statistics for 743 book titles registered in the Danish, Norwegian, and Finnish National repositories

WorldCat
country
distribución
692

PlumX
captures
279

Google
Scholar
citas
741

norte

Valid

Missing

Significar

Median

Mínimo

Máximo

Percentiles

WorldCat
holdings
692

135

PlumX
usage
278

465

3445

1597

180

464

756

PlumX
social
media
279

464

Scopus
citas
279

PlumX
menciona
279

464

830

172

201

.19

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

social media counts tend to accrue more rapidly than citations, but in the case of books, nosotros
recognize the problem of DOIs. Results similar to ours can be seen in previous research (ver
Torres-Salinas et al., 2017a, 2017b; Blanco & Zuccala, 2018; Zuccala & Blanco, 2015), dónde
library holdings or “libcitations” were also found to be more prevalent than Google Scholar
and Scopus citations, with much lower PlumX usage counts, captures, menciona, and social
media counts. Except for the correlation analyses, shown below, we present no further anal-
yses pertaining to the PlumX indicators.

Mesa 5 presents the Spearman rho correlation coefficients for each of the indicators tested
in this study. The strongest correlation was found for WorldCat holding counts and WorldCat

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
5
8
8
1
9
3
0
6
9
3
q
s
s
_
a
_
0
0
1
3
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Cifra 3. Boxplot distributions for all eight indicators.

Estudios de ciencias cuantitativas

600

Performance-based publisher lists

Mesa 5.

Spearman’s rho correlations between the eight indicators

WorldCat
holdings

.246**

.959**

.426**

.432**

PlumX
usage
.246**

WorldCat
country
distribución
.959**

PlumX
captures
.426**

Google
Scholar
citas
.432**

.321**

.505**

.206**

Plum X
social
media
0.111

−0,002

Scopus
citas
.124*

PlumX
menciona
0.016

.235**

0.027

.321**

.505**

.206**

.400**

.435**

0.062

.126*

−0,007

.400**

.313**

.179**

.316**

0.108

.435**

.313**

0.104

.258**

0.092

WorldCat holdings

PlumX usage

WorldCat country
distribución

PlumX captures

Google Scholar citations

PlumX social media

0.111

−0,002

0.062

.179**

0.104

0.025

.183**

Scopus citations

PlumX mentions

.124*

.235**

.126*

.316**

.258**

0.025

0.033

0.016

0.027

−0,007

0.108

0.092

.183**

0.033

* Correlation is significant at the 0.05 nivel (2-cola); ** Correlation is significant at the 0.01 nivel (2-cola).

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
q
s
s
/
a
r
t
i
C
mi
–
pag
d

F
/

2
2
5
8
8
1
9
3
0
6
9
3
q
s
s
_
a
_
0
0
1
3
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

country distribution counts (rho = .96, pag < 0.01). PlumX usage counts and captures are also strongly correlated (rho = .51, p < 0.01), followed by WorldCat country distribution counts and Google Scholar citations (rho = .44, p < 0.01), as well as WorldCat holdings and Google Scholar citations (rho = .43, p < 0.01). 5.2. Publisher Levels and WorldCat Holdings 5.2.1. Cumulative frequency distributions Figure 4 presents the full frequency distributions of WorldCat holdings for the different PRFS system levels. Again, Finland is the only registry country that has level 0 and level 3 publisher categories. The cumulative relative distributions show that the highest percentage of WorldCat Figure 4. Distribution frequencies of WorldCat library holdings according to PRFS publisher level. Quantitative Science Studies 601 Performance-based publisher lists Figure 5. Denmark’s publisher levels, points, and WorldCat library holdings (2019). holding counts are concentrated at level 3, as well as at the tail end of level 2. For books published at level 1, there is a much broader distribution of holding counts. 5.2.2. Boxplots In Figures 5, 6, and 7, boxplots are shown for book titles assigned to two publisher levels for Denmark and Norway, and four publisher levels for Finland. For each level, we see the col- lected book titles’ WorldCat library holding counts (“libcitations” as per White, Boell et al., l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / e d u q s s / a r t i c e - p d l f / / / / 2 2 5 8 8 1 9 3 0 6 9 3 q s s _ a _ 0 0 1 3 4 p d / . f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 Figure 6. Norway’s publisher levels, points, and WorldCat library holdings (2019). Quantitative Science Studies 602 Performance-based publisher lists Figure 7. Finland’s publisher levels and WorldCat library holdings (2019). 2009). Here, the broader “perceived cultural relevance” and visibility of each title is indicated by how many libraries worldwide hold at least one copy. Starting from Figure 5 (Denmark), a benchmark has been added, based on the combined median for levels 1 and 2 (Denmark and Norway), and the combined median for levels 0 to 3 (Finland) for WorldCat holdings. These benchmarks are in some sense arbitrary, as alternative values may be selected, but they are useful for explaining our metaphor. Books produced with a level 1 publisher that have achieved greater than or equal to the visibility benchmark (≥ 39 holdings) are what we call bigger fish in the small pond of national/regional publishing. Likewise, the books that fall below the benchmark median (< 39) are smaller fish in the big pond of international publishing (see Figure 3). In Figure 6 and Figure 7 we apply the same description of bigger fish and smaller fish in the small or large ponds for Norway, with a bench- mark median of 32, and for Finland, with a benchmark median of 111. 5.3. Publisher Levels and Google Scholar Citations 5.3.1. Cumulative frequency distributions Figure 8 shows the full cumulative frequency distributions of Google Scholar citations for the different PRFS levels. The relative distributions show that for all levels, 20–80% are concentrated within a range of 0 to 10 citations. Books published by a level 3 publisher generally outperform those from level 1, where we see some with greater than 100 citations, but there is generally an equal distribution of books published at level 2 that perform just as well as level 3. In Figures 9, 10, and 11 boxplots and combined median benchmarks are shown for book titles produced and assigned to two publisher levels for Denmark and Norway, and four publisher levels for Finland. For each level, we can observe the collected book titles’ Google Scholar citations. Here, the “impact” of each title is indicated by its count of citations. Again, for each of the boxplot figures (Figures 5–7 and 9–11) a benchmark median is shown for WorldCat holdings, as well as Google Scholar citations. Conceptually, we use these median values to explain how a PRFS “rewards” a book at an individual level. Quantitative Science Studies 603 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / e d u q s s / a r t i c e - p d l f / / / / 2 2 5 8 8 1 9 3 0 6 9 3 q s s _ a _ 0 0 1 3 4 p d . / f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 Performance-based publisher lists l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . Figure 8. Distribution frequencies of Google Scholar citations according to PRFS publisher level. All book titles at level 1 that fall below the benchmark perform “as expected.” This means that they correspond sufficiently with the five reward points (or in Finland four points) earned as a result of their country’s PRFS. Moreover, all books at level 2 that are higher than the benchmark have also performed as “expected.” This means that these titles correspond suffi- ciently with the eight reward points earned as a result of their country’s PRFS. / e d u q s s / a r t i c e - p d l f / / / / 2 2 5 8 8 1 9 3 0 6 9 3 q s s _ a _ 0 0 1 3 4 p d . / f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 Figure 9. Denmark’s publisher levels, points, and Google Scholar citations (2019). Quantitative Science Studies 604 Performance-based publisher lists Figure 10. Norway’s publisher levels, points, and Google Scholar citations (2019). Then, there are some level 1 books that are higher than the benchmark (as bigger fish in the smaller pond ) and level 2 books that are lower than the benchmark median (as smaller fish in the bigger pond ). These individual books are of interest because they all represent reward point discrepancies. Consider the following example. There are two books from our data set: Census and census takers: A global history and Culture war: Affective cultural politics, tepid nationalism and art activism. The first title acquired five PRFS reward points, and the second acquired eight PRFS reward points, yet both have achieved the same degree of international visibility based on l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / e d u q s s / a r t i c e - p d l f / / / / 2 2 5 8 8 1 9 3 0 6 9 3 q s s _ a _ 0 0 1 3 4 p d / . f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 Figure 11. Finland’s publisher levels, points, and Google Scholar citations (2019). Quantitative Science Studies 605 Performance-based publisher lists Table 6. Percentages of bigger/smaller fish in the smaller versus bigger ponds of WorldCat library holdings WorldCat holdings Country and median benchmark Denmark Bigger fish ≥ 39 Norway Finland Smaller fish < 39 Bigger fish ≥ 32 Smaller fish < 32 Bigger fish ≥ 111 Smaller fish < 111 PRFS publisher levels Level 0 Level 1 Bigger fish − smaller pond Level 2 Smaller fish − bigger pond Level 3 25% (41/167) 30% (55/184) 5% (5/96) 3% (2/75) 0% (0/52) 27% (16/59) 44% (28/63) 11% (5/47) WorldCat holdings (n = 86). At an individual level of assessment there is no difference be- tween the two books based on this one indicator, yet the second title was “rewarded” with more points than the other, simply because of its publisher. There is also an example for two book titles based on citations: Expanding welfare in an age of austerity: Increasing protec- tion in an unprotected world was published at level 1, with five PRFS points and eight cita- tions, and Visible hands: Government regulation and international business responsibility was published at level 2 with eight PRFS points, also having received eight citations. Again, there are no differences between these titles, except for reward points. In Table 6 and Table 7, all data pertaining to “reward” discrepancies for both indicators are summarized as percentages. Note that for the Danish registry 25% of the books earned five PRFS points for having been produced by a level 1 publisher, yet have been individually and culturally more visible via WorldCat library holdings than 5% of the level 2 titles rewarded 8 points. Also, 32% of the level 1 titles from this same registry have had a higher citation impact within a 2-year period than 24% of the titles published in the same year at level 2. For Finland, the share of smaller fish for WorldCat holdings in the bigger pond of international publishing at level 2 (44%) is much larger than it is for Denmark and Norway. The most likely explanation for this is that Finland includes regional publishers at level 2, whereas the other countries only assign international publishers to level 1. Table 7. Percentages of bigger/smaller fish in the smaller versus bigger ponds of Google Scholar citations Google Scholar citations Country and median benchmark Denmark Bigger fish ≥ 2 Norway Finland Smaller fish < 2 Bigger fish ≥ 2 Smaller fish < 2 Bigger fish ≥ 2 Smaller fish < 2 PFRS publisher levels Level 0 Level 1 Bigger fish − smaller pond Level 2 Level 3 Smaller fish − bigger pond 32% (53/167) 53% (97/184) 24% (23/96) 23% (17/75) 29% (15/52) 41% (24/59) 37% (23/63) 21% (10/47) Quantitative Science Studies 606 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / e d u q s s / a r t i c e - p d l f / / / / 2 2 5 8 8 1 9 3 0 6 9 3 q s s _ a _ 0 0 1 3 4 p d / . f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 Performance-based publisher lists 5.4. Negative Binomial Regression Model A Negative Binomial regression model was used to determine if additional features of a pub- lisher or book, including the PRFS level, are predictors of WorldCat holding counts and Google Scholar Citations. With the boxplots, we only see the extent to which PRFS level rat- ings correspond with impact or visibility, but there may be other factors that put an individual book in a bigger fish − smaller pond or smaller fish − bigger pond category. For this analysis we use additional data pertaining to Publisher Type (Commercial/Noncommercial), Publisher Language (English/Not English), and Subject Area (see categories in Table 3). As a Poisson model often precedes the use of a Negative Binomial model, a Poisson regres- sion was used first to see if the variances in the distributions of our two predictor variables were equal to the means of their distributions (i.e., the mean counts of WorldCat holding counts and mean counts of Google Scholar citations). The variances were greater than the means, and in our case pointed to overdispersion in the data ( WorldCat mean = 135; Variance = 50,643), Google Scholar mean = 8; Variance = 1,463). The ratio of the deviance to the degrees of freedom also signaled overdispersion (df = 131 for WorldCat holdings; df = 25 for Google Scholar citations). Note that the further this value is away from 1, the more the data are overdispersed. A Negative Binomial regression essentially adjusts the Poisson model when there is a high amount of overdispersion. The predicted change for our outcome or dependent variables is in ex- pected log counts. A positive regression value conveys a positive predictor relationship between the predictor variable and the expected count outcome. A negative value reflects a negative rela- tionship between the predictor and the expected count. Results from the Negative Binomial model point to a degrees of freedom ratio that improved the overdispersion (df = 1.55 for WorldCat hold- ings; df = 2.38 for Google Scholar citations). The omnibus tests also indicated a significant im- provement in fit over a null model ( p < .001; Sig .000), which would include no predictors. Table 8 shows the parameter estimates for WorldCat holdings as the dependent variable. Here we see that the Publisher Language (English versus Non-English) is positive and signifi- cant (B = 2.30; p < .001). The Publisher Level (Level 0, 1, 2, or 3) is also positive, but only to a Table 8. Negative Binomial parameter estimates: WorldCat holdings 95% Wald confidence interval Lower 2.408 Hypothesis test Upper Wald chi-square df 1 3.053 275.918 −.092 2.105 .042 −.105 .261 2.495 .344 .048 .878 531.860 6.252 .531 1 1 1 1 95% Wald confidence interval for Exp(B) Upper Lower 21.171 11.115 .912 8.203 1.043 .900 1.298 12.127 1.410 1.049 Exp(B) Sig. .000 15.340 .349 1.088 .000 9.974 .012 1.212 .466 .972 Parameter (Intercept) PubType B 2.730 Std. error .1644 .084 .0901 PubLanguage 2.300 .0997 PubLevel SubjectArea (Scale) .193 .0771 −.029 .0392 1a (Negative binomial) 1a Dependent Variable: WorldCatHoldingsCount Model: (Intercept), PubType, PubLanguage, PubLevel, SubjectArea a Fixed at the displayed value. Quantitative Science Studies 607 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / e d u q s s / a r t i c e - p d l f / / / / 2 2 5 8 8 1 9 3 0 6 9 3 q s s _ a _ 0 0 1 3 4 p d . / f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 Performance-based publisher lists limited degree (B = .193; p < .05), while the Subject Area shows a negative but nonsignificant relationship to holding counts (B = −.029; p > .05). A PRFS publisher level therefore relates to
some degree to WorldCat holdings, but it is still not the strongest predictor of the log change in
outcome for these counts when other variables are held constant.

Mesa 9 shows the parameter estimates for Google Scholar citations as our dependent var-
iable. Aquí, the Publisher Language is the strongest predictor variable (B = .746; pag < .000) followed by the Publisher Level (B = .699; p < .000), which is also positive and significant. The Subject Area of a book is also a positive predictive variable in relation to citations (B = .196; p < .000), and more so when measuring WorldCat holdings. Again, a PRFS publisher level relates to some degree to Google Scholar citations, but it is still not the strongest predictor of the log change in outcome for these counts when other variables are held constant. 5.5. Publisher “Distribution Potential” When PRFS in Denmark, Norway, and Finland assign levels to publishers, suggestions and deci- sions are normally based on the overall reputation of the publisher in a research field, where rep- utation is closely connected to how the publisher performs in terms of peer review. Because our Negative Binomial model shows that type of publisher (i.e., commercial versus noncommercial) does not have any predictive value for WorldCat holdings, the present analysis is carried out to examine how a publisher functions as a worldwide library distributor. This can help to explain, in part, the bigger/smaller fish phenomenon, where books classified at certain level may have been produced by publishers with broader distribution potential than others. The scatterplots in Figures 12 and 13 show the relationship between mean publisher WorldCat holdings and mean country distributions, both of which are strongly and significantly correlated (rho =.96, p < 0.01). For both figures we have selected a set of level 1, 2, and 3 publishing houses/presses that have produced three or morebooks. Due to an overlap in publisher names added to Figures 12 and 13, we further specify in Table 10 the primary differences in level assignments for the three countries. The assignments Table 9. Negative Binomial parameter estimates: Google Scholar citations Parameter (Intercept) PubType B .340 Std. error .1689 95% Wald confidence interval Lower .009 .671 Hypothesis test df 1 4.062 95% Wald confidence interval for Exp(B) Upper Lower 1.957 1.009 Sig. .044 Exp(B) 1.405 Upper Wald chi-square −.228 .0942 −.413 −.044 PubLanguage .746 .1012 PubLevel .699 .0779 SubjectArea .196 .0398 .548 .547 .118 .945 .852 .274 (Scale) (Negative binomial) 1a 1a Dependent Variable: Google Scholar Citations Model: (Intercept), PubType, PubLanguage, PubLevel, SubjectArea a Fixed at the displayed value. Quantitative Science Studies 5.874 54.335 80.620 24.259 1 1 1 1 .015 .796 .662 .000 2.109 1.729 .000 2.012 1.727 .000 1.216 1.125 .957 2.572 2.344 1.315 608 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / e d u q s s / a r t i c e - p d l f / / / / 2 2 5 8 8 1 9 3 0 6 9 3 q s s _ a _ 0 0 1 3 4 p d / . f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 Performance-based publisher lists l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . Figure 12. Publishers (level 1) with the strongest international library distribution. noted in brackets indicate that there was no book title associated with that particular publisher for Denmark, and/or Norway in our 2017 data set. Again, publishers facilitate editorial, peer review, and printing processes but also serve as distribution centers to libraries worldwide. In Figure 12, where the scale is logarithmic, most of / e d u q s s / a r t i c e - p d l f / / / / 2 2 5 8 8 1 9 3 0 6 9 3 q s s _ a _ 0 0 1 3 4 p d . / f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 Figure 13. Publishers (levels 2 and 3) with the strongest international library distribution. Quantitative Science Studies 609 Performance-based publisher lists Table 10. Differences in publisher level assignments per country Publishers Amsterdam University Press Bloomsbury Brill Elsevier Emerald Publishing I.B. Tauris John Wiley & Sons Palgrave Suomalaisen Kirjallisuuden Seura Wolters Kluwer Publisher level assignments (2018) Norway (1)* Finland 2 Denmark 1 2 2 1 2 (2) (2) 2 (1) 1 1 1 or 2 2 1, 2 or 3 1 (1) (2) 1 2 (1) 1 2 1 1 2 3 0, 1, or 2 2 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . * Bracketed numbers indicate country’s publisher level where data are missing. the publishers at level 1 fit within the quadrant representing the lowest worldwide distribution. The same quadrant is outlined in Figure 13, which utilizes a normal scale. Still, there are a few publishing outlets at level 1 (Amsterdam University Press, Reclam Verlag, Bloomsbury) with a relatively strong distribution potential. Figure 13 shows that level 2 and 3 publishers tend to have a much higher distribution potential compared to those at level 1, with John Wiley & Sons, Manchester University Press, John Benjamins, and Wiley-Blackwell representing the highest from these two levels. / e d u q s s / a r t i c e - p d l f / / / / 2 2 5 8 8 1 9 3 0 6 9 3 q s s _ a _ 0 0 1 3 4 p d . / f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 6. CONCLUSIONS 6.1. Overall Findings The following conclusions may be drawn from this study. The first is that WorldCat yields sub- stantial library holding and country distribution counts for most book titles (i.e., 93% of n = 743 titles had at least one holding). Approximately 60% of the books in our data set also had at least one Google Scholar citation, but with many titles observed on the platform with a zero count, we might have obtained additional citations with a longer time period for data collec- tion. With PlumX as a data source, some capture and usage counts were found, but the men- tions and social media counts yielded the least amount of useful data. This is mainly because PlumX depends highly on the availability of DOIs for books. Overall, the study results point clearly to the reliability of the two WorldCat indicators. The second conclusion relates specifically to the PRFS publisher level assignments. As the main aim of the Nordic systems is to stimulate high “quality” publishing, the focus is top down, with an emphasis on aggregate publishing patterns, leading to rewards at an institutional level. Thus, at an institutional level the expectation is that there will be an “average” quality of out- puts. With all of our boxplot distributions (Figures 5–7 and Figures 9–11) we show that a lower publisher level generally corresponds with lower visibility ( WorldCat counts) and lower im- pact (Google Scholar citations), while a higher publisher level generally corresponds with Quantitative Science Studies 610 Performance-based publisher lists higher visibility and higher impact. But within each of these distributions there are a number of exceptions. Approximately 25% to 41% of the books from each of the Danish, Norwegian, and Finnish data sets (2017) have higher degrees of visibility or impact as bigger fish in smaller ponds. A broader range of 5% to 44% present a lower degree of visibility or impact as smaller fish in bigger ponds. Many other factors are therefore contributing to a book’s achievement relative to this metaphorical “big-to-small fish” continuum. Our Negative Binomial regression model, which focuses on a few additional variables, shows that publisher type (i.e., commercial ver- sus noncommercial) is not a strong predictor of either WorldCat holdings or Google Scholar citations, but that publication language is (i.e., English/Non-English). Clearly a book published in English is more likely to be cited and accrue broad international WorldCat holdings, and this makes sense, given the predominance of English in academia. There are also differences in the distribution potential of every publisher. Publishers clas- sified at level 1 tend to demonstrate a lower WorldCat distribution effect than those catego- rized at level 2; however, our study points to a few classification discrepancies (i.e., with Amsterdam University Press, Brill and Bloomsbury classified differently per PRFS). Still, authors may expect to have their book distributed more widely by publishers such as Oxford University Press, Princeton University Press, or Wiley-Blackwell. While categories for journals (e.g., the Web of Science Subject Classification) are well- established and used often in journal-based analyses, it is still problematic to assess books on the basis of subjects. Again, only 49% of the titles had been assigned a real subject classification (i.e., with an OCLC Library of Congress or Dewey Decimal Number), and this meant that we could only make a manual subject assignment to cover the whole data set. With each title “freely” classified according to a main subject area, we found that the subject of a book has predictive value only in terms of log changes in Google Scholar citation rates, but not WorldCat holdings. While books pertaining to many subjects can be held in libraries worldwide, certain research areas tend to cite new books more than others. l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / e d u q s s / a r t i c e - p d l f / / / / 2 2 5 8 8 1 9 3 0 6 9 3 q s s _ a _ 0 0 1 3 4 p d / . f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 6.2. PRFS and Individuals PRFS committees as well as individual authors might more or less agree upon a publisher’s degree of specialization, quality, or prestige. An author’s choice of publisher can for instance, depend on the degree to which it is highly specialized or relevant to a very small field of specialization. In this instance, it might be the book series itself, the editorial staff, and review process that the author finds attractive about the publisher, more than the publisher’s level. Still, what an individual author might also want is his or her work to be exceptionally “visible” and generate some “impact,” based on a high uptake in the scholarly communication system. If relevant, authors would hope to see this take place on an international scale. As White et al. (2009) suggest: “esteem goes to researchers for their perceived impacts on culture and the life of the mind” (i.e., for scholars as well as the lay public [p. 1086]). What we know from journals, and now also with book publishers, is that there is no guarantee that a choice of publishing outlet will lead to academic and/or broad cultural impact. Nevertheless, it is possible that PRFS in their current form may be having an influence on Nordic scholars’ publishing choices. Perhaps some authors are selecting a publisher merely because of its assigned level, without considering other factors. Some studies have already investigated the influence of PRFS on scholars’ journal publishing habits (Ingwersen & Larsen, 2014; Pölönen, Auranen et al., 2018), and more research is needed to understand the effect of PRFS levels on book authors. Quantitative Science Studies 611 Performance-based publisher lists 6.3. PRFS and Regional Publishers As noted earlier, national publishers in Denmark are always categorized as level 1, with many in- ternational presses assigned to level 2. This means that national or regional publishing houses are at a disadvantage, particularly on an international scale, where the publishing world is intensely com- petitive. It is a routine feature of the publishing world to compete both in a “market for content and market for customers” (Thomson, 2005, p. 35). Regional specificity is a characteristic that can “make it more difficult for [a] press to attract the very best international scholars in international disciplines” (Thomson, 2005, p. 158). If authors are too concerned about the internationalization of their research and focus too intently on the acquisition of level 2 points, regional publishers, including research institutions, learned societies, and commercial publishers, may suffer. Publishing more books with international publishers (in English) also makes it increasingly difficult to establish reliable indicators for books published at a local or national level (not in English). Thus far, national visibility can best be seen via social media platforms, such as LinkedIn, Twitter and/or other local news outlets. For example, a new book entitled Grov Konfækt. Tre vilde år med trykkefrihed 1770–73 (Horstbøll, Langen, & Stjernfelt, 2020), which examines the freedom of the press in Denmark, was confirmed a “masterpiece” by the Carlsberg Foundation. It was written and published in Danish with a Copenhagen-based press (Gyldendal. dk). An announcement on LinkedIn (Budtz-Pedersen, 2020) was made to indicate that a book reception was held in Copenhagen, where the Danish Prime Minister (Mette Frederiksen) and the Chairman and Editor in Chief of Information from the Carlsberg Foundation attended. Metric-type indicators have not (yet) been collected to determine its international visibility or impact, but there is clear evidence via social media that it is of high national importance. 6.4. Findings Relevant to the Proposed Publisher Registry As our study focuses on publisher assignments and what they do at a national level, certain findings have implications for the proposed international publisher registry (ENRESSH, 2019; Giménez-Toledo et al., 2019). Although there are many reasons for its development, we are specifically interested in what it can do to enhance PRFS. There may be two benefits. The first relates to the lack of uniformity in publisher assignments, and in this paper we show that this is an issue even when the PRFS, like that of Denmark, Norway, and Finland, are relatively similar (see Table 10). Other publisher systems, like the one used in Spain, present even more incongruities (see Mañana-Rodriguez & Pölönen, 2018); hence it is still not clear what constitutes a good publisher from a specific geographical region. How can this be established? By gathering and registering more detailed information about publishers in one data source, it can be easier to promote a more unified approach to their classification. This, in turn, could give other countries in Europe a stronger motivation to adopt and coordinate more comprehensive PRFS. The second benefit is that it will afford scholarly researchers not only more transparent, but also more detailed information about book publishers. So far, PRFS have not been designed to give authors opportunities to identify precisely what it is that they need from a publisher. PRFS committees are broadly classifying them, yet as we show from our individual level analyses, broad classifications can never account completely for what happens to a book at an individual level. This depends on a myriad of factors, such as peer review and editorial procedures, publishing/print language options, publisher specialization, and the distribution potential of the publisher itself. Transparency of information will not only be good for publishers, but valuable to authors. For instance, if scholars are given an opportunity to interact with one Quantitative Science Studies 612 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / e d u q s s / a r t i c e - p d l f / / / / 2 2 5 8 8 1 9 3 0 6 9 3 q s s _ a _ 0 0 1 3 4 p d / . f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 Performance-based publisher lists another via the system, and obtain recommendations about their publishing experiences (i.e., like a publisher Trip Advisor), this could help them to determine which press is best for their own publishing strategy. Recommendations would also allow authors to make more “organic” choices, particularly if they are writing about subjects pertaining to regional or national concerns (Kulczycki, Guns et al., 2020). Last but not least, because the Nordic PRFS were originally created to be used as indicators for funding allocation at a macroevel, it is first and foremost critical to state that level ratings should not be applied at the level of individual researchers (Aagaard, 2018; Pölönen et al., 2018; Sivertsen, 2018). As our findings show (Sections 5.2 and 5.3), PRFS publisher levels do not necessarily relate to higher rates of visibility or impact for individual books. Still, some universities and departments have been known to (mis)use PRFS indicators for their internal evaluation and funding procedures (Aagaard, 2015; Hicks, 2012; Krog Lind, 2019; Sivertsen & Schneider, 2012; Wahlfors & Pölönen, 2018). In Sweden, for example, there is no PRFS, yet many Swedish universities use the Norwegian indicator and publication channel list for their internal evaluation/assessment/funding procedures (Hammarfelt, Nelhans et al., 2016). Similarly, in Poland, a new model was introduced in 2018 that utilizes publisher lists in a scholar’s habilitation procedure (Kulczycki & Korytkowski, 2018). When a candidate presents a book as his or her main achievement, it has to be from a publisher on Poland’s new publisher list. In this case, potential misuse occurs if the publisher alone signals the candidate’s achievement, to the exclusion of assessing the book’s content, visibility, and cul- tural impact. Finally, we look to the near future, and beyond. Perhaps there may be an opportunity to develop interoperability between the international publisher registry and WorldCat. By this, we mean that perhaps holding and country distribution counts from WorldCat could be incor- porated into the registry, in much the same way that PlumX indicators are linked to the Scopus index. This would make it easier and more appropriate to conduct performance evaluations at an individual level if the visibility or impact of a book is significant for a scholar’s promotion and/or tenure. All of this depends also on the progress of Open Access publishing. WorldCat holdings and country distribution counts seem to be reliable indicators at the moment, but may not be as useful in the future if open access publishing means that libraries no longer “hold” books. Publishers may develop stronger marketing approaches for open access books via social media (e.g., Wang & Zuccala, 2021) and there could be new opportunities to rethink metric indicators for books related to capture, publisher downloads, and usage statistics. ACKNOWLEDGMENTS The authors would like to thank the reviewers and the editor Ludo Waltman in particular for his valuable comments, as well as Adrian Díaz-Faes for his informal reviews of earlier drafts of this paper. COMPETING INTERESTS The authors have no competing interests. FUNDING INFORMATION No funding has been received for this research. DATA AVAILABILITY Data for this project are available on Figshare (https://doi.org/10.6084/m9.figshare.14472645). Quantitative Science Studies 613 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / e d u q s s / a r t i c e - p d l f / / / / 2 2 5 8 8 1 9 3 0 6 9 3 q s s _ a _ 0 0 1 3 4 p d / . f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 Performance-based publisher lists REFERENCES Aagaard, K. (2015). How incentives trickle down: Local use of a national bibliometric indicator system. Science and Public Policy, 42(5), 725–737. https://doi.org/10.1093/scipol/scu087 Aagaard, K. (2018). Performance-based research funding in Denmark: The adoption and translation of the Norwegian model. Journal of Data and Information Science, 3(4), 20–30. https://doi .org/10.2478/jdis-2018-0018 Archambault, E., & Larivière, V. (2009). History of the journal impact factor: Contingencies and consequences. Scientometrics, 79(3), 635–649. https://doi.org/10.1007/s11192-007-2036-x Aksnes, D. (2017). Artikler i nivå 2-tidsskrifter blir mest sitter. Forskerforum. Available at: https://www.forskerforum.no/artikler -i-niva-2-tidsskrifter-blir-mest-sitert/ Auranen, O., & Pölönen, J. (2012). Classification of scientific publi- cation channels: Final report of the Publication Forum project (2010–2012). Helsinki: Federation of Finnish Learned Societies. Available at: https://www.julkaisufoorumi.fi/sites/julkaisufoorumi .fi/files/publication_forum_project_final_report_0.pdf Budtz-Pedersen, D. (2020). Yesterday was a big day in the life of Humanomics Research Centre. Our Co-Director Frederik Stjernfelt published a new 1000-page book on freedom of press in Denmark 1770–1773 (with Ulrik Langen and Henrik Horstbøll). Available at: https://www.linkedin.com/posts/david -budtz-pedersen-phd-267b535_yesterday-was-a-big-day-in-the -life-of-humanomics-activity-6711903885260324864-ohko ENRESSH. (2019). Academic book publishers (ABP): A global and multilingual register. Available at: https://enressh.eu/wp-content /uploads/2019/08/Academic-Book-Publishers-ABP-A-global-and -interactive-register.pdf Garand, J. C., & Giles, M. W. (2011). Ranking scholarly publishers in political science: An alternative approach. PS: Political Science and Politics, 44(2), 375–383. https://doi.org/10.1017 /S1049096511000229 Garfield, E. (2006). The history and meaning of the journal impact factor. JAMA – Journal of the American Medical Association, 295(1), 90–93. https://doi.org/10.1001/jama.295.1.90, PubMed: 16391221 Giménez-Toledo, E., Mañana-Rodríguez, J., Engels, T. C. E., Ingwersen, P., Polonen, J., … Zuccala, A. A. (2016). Taking scholarly books into account. Current developments in five European countries. Scientometrics, 107(2), 685–699. https:// doi.org/10.1007/s11192-016-1886-5 Giménez-Toledo, E., Mañana-Rodríguez, J., & Tejada-Artigas, C-M. (2015). Scholarly Publishers Indicators: Prestige, specialization and peer review of scholarly book publishers. El profesional de la información, 24(6), 855–860. https://doi.org/10.3145/epi.2015 .nov.18 Giménez-Toledo, E., & Rom(cid:1)an-Rom(cid:1)an, A. (2009). Assessment of humanities and social sciences monographs through their pub- lishers: A review and study towards a model of evaluation. Research Evaluation, 18(3), 201–213. https://doi.org/10.3152 /095820209X471986 Giménez-Toledo, E., Sivertsen, G., & Mañana-Rodríguez, J. (2019). International Register of Academic Book Publishers (IRAP): Overview, current state and future challenges. In H. F. Moed, C. Daraio, G. Catalano, & G. Ruocco (Eds.), Proceedings of the International Society for Scientometrics (ISSI) Conference (pp. 1752–1757). Rome, Italy: September 2–5. González-Pereira, B., Guerrero-Bote, V. P., & Moya-Anegón, F. (2010). A new approach to the metric of journals’ scientific prestige: The SJR indicator. Journal of Informetrics, 4(3), 379–391. https://doi .org/10.1016/j.joi.2010.03.002 Goodson, L. P., Dillman, B., & Hira, A. (1999). Ranking the presses: Political scientists’ evaluations of publisher quality. PS: Political Science and Politics, 32(2), 257–262. https://doi.org/10.2307/420561 Guns, R. (2018). Concentration of academic book publishers. STI 2018 Conference Proceedings (pp. 518–525). Center for Science and Technology Studies. Available at: https://hdl.handle.net/1887/65268 Hammarfelt, B., Nelhans, G., Eklund, P., & Åström, F. (2016). The heterogeneous landscape of bibliometric indicators. Evaluating models for allocating resources at Swedish universities. Research Evaluation, 25(3), 292–305. https://doi.org/10.1093/reseval/rvv040 Hegarty, P., & Walton, Z. (2012). The consequences of predicting scientific impact in psychology using journal impact factors. Perspectives on Psychological Science, 7(1), 72–78. https://doi .org/10.1177/1745691611429356, PubMed: 26168426 Hicks, D. (2012). Performance-based university research funding systems. Research Policy, 41(2), 251–261. https://doi.org/10 .1016/j.respol.2011.09.007 Horstbøll, H., Langen, U., & Stjernfelt, F. (2020). Grov Konfækt. Tre vilde år med trykkefrihed 1770–73. Copenhagen, Denmark: Gyldendal. Ingwersen, P., & Larsen, B. (2014). Influence of a performance indicator on Danish research production and citation impact 2000–12. Scientometrics, 101, 1325–1344. https://doi.org/10 .1007/s11192-014-1291-x Konkiel, S., & Adie, E. (2018). What altmetrics can tell us about the “real world” impacts of books. Available at: https://figshare.com /articles/online_resource/ What_altmetrics_can_tell_us_about _the_real_world_impacts_of_books/6940325 Krog Lind, J. (2019). The missing link: How university managers mediate the impact of a performance-based research funding system. Research Evaluation, 28(1), 84–93. https://doi.org/10 .1093/reseval/rvy038 Kulczycki, E., Guns, R., Polonen, J., Engels, T., Rozkosz, E. A., … Sivertsen, G. (2020). Multilingual publishing in the social sciences and humanities: A seven-country European study. Journal of the Association for Information Science and Technology, 71(11), 1371–1385. https://doi.org/10.1002/asi.24336, PubMed: 33288998 Kulczycki, E., & Korytkowski, P. (2018). Redesigning the model of book evaluation in the Polish performance-based research funding system. Journal of Data and Information Science, 3(4), 60–72. https://doi.org/10.2478/jdis-2018-0021 Kulczycki, E., Rozkosz, E. A., Engels, T. C. E., Guns, R. Hołowiecki, M., & Pölönen, J. (2019). How to identify peer-reviewed publica- tions: Open-identity labels in scholarly book publishing. PLOS ONE, 14(3), e0214423. https://doi.org/10.1371/journal.pone .0214423, PubMed: 30908515 Larivière, V., Haustein, S., & Mongeon, P. (2015). The oligopoly of academic publishers in the digital era. PLOS ONE, 10(6), e0127502. https://doi.org/10.1371/journal.pone.0127502, PubMed: 26061978 Leonardi, P. M. (2014). Social media, knowledge sharing, and innovation: Toward a theory of communication visibility. Information Systems Research, 25(4), 796–816. https://doi.org/10.1287/isre.2014.0536 Lockett, A., & Speicher, L. (2016). New university presses in the UK: Accessing a mission. Learned Publishing, 29(S1), 320–329. https://doi.org/10.1002/leap.1049 Mañana-Rodriguez, J., & Pölönen, J. (2018). Scholarly book publishers’ ratings and lists in Finland and Spain: Comparison and assessment of the evaluative potential of merged lists. Aslib Journal of Information Management, 70(6), 643–659. https://doi .org/10.1108/AJIM-05-2018-0111 Mannana-Rodriguez, J., & Giménez-Toledo, E. (2018). Specialization and multidisciplinarity of scholarly book publishers: Differences Quantitative Science Studies 614 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / e d u q s s / a r t i c e - p d l f / / / / 2 2 5 8 8 1 9 3 0 6 9 3 q s s _ a _ 0 0 1 3 4 p d / . f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 Performance-based publisher lists between Spanish university presses and other scholarly publishers. Scientometrics, 114(1), 19–30. https://doi.org/10.1007/s11192-017-2563-z Metz, P., & Stemmer, J. (1996). A reputational study of academic publishers. College & Research Libraries, 57(3), 234–247. https:// doi.org/10.5860/crl_57_03_234 Moed, H. F. (2011). The Source-Normalized Impact per Paper (SNIP) is a valid and sophisticated indicator of journal citation impact. Journal of the American Society for Information Science and Technology, 62(1), 211–213. https://doi.org/10.1002/asi.21424 Norwegian Association of Higher Education Institutions. (2004). A bibliometric model for performance-based budgeting of research institutions. Available at: https://npi.nsd.no/dok/ Vekt_pa _forskning_2004_in_english.pdf Parsons, P. (1990). Specialization by university presses. Book Research Quarterly, 6(2), 3–15. https://doi.org/10.1007/BF02683807 Pudovkin, A. I. (2018). Comments on the use of the journal impact factor for assessing the research contributions of individual authors. Frontiers in Research Metrics and Analytics, 3, 2. https://doi.org/10 .3389/frma.2018.00002 Pölönen, J., Auranen, O., Engels, T., & Kulczycki, E. (2018). Taking na- tional language publications into account: The case of the Finnish performance-based research funding system. STI 2018 Conference Proceedings (pp. 204–211). Centre for Science and Technology Studies (CWTS). Available at: https://hdl.handle.net/1887/65223 Sivertsen, G. (2016). Publication-based funding: The Norwegian model. In: M. Ochsner et al. (Eds.), Research Assessment in the Humanities: Towards Criteria and Procedures (pp. 71–90). Springer International Publishing. https://doi.org/10.1007/978-3-319-29016-4_7 Sivertsen, G. (2018). The Norwegian model in Norway. Journal of Data and Information Science, 3(4), 2–18. https://doi.org/10.2478 /jdis-2018-0017 Sivertsen, G., & Schneider, J. (2012). Evaluering av den bibliometriske forskningsindikator. Nordisk institutt for studier av innovasjon, forskning og utdanning. Rapport 17/2012. Available at: https://ufm .dk/forskning-og-innovation/statistik-og-analyser/den-bibliometriske -forskningsindikator/endelig-rapport-august-2012.pdf Sugimoto, C. R., & Lariviére, V. (2018). Measuring research: What everyone needs to know. New York: Oxford University Press. Thelwall, M., Haustein, S., Larivière, V., & Sugimoto, C. R. (2013). Do altmetrics work? Twitter and ten other social web services. PLOS ONE, 8(5), e64841. https://doi.org/10.1371/journal.pone .0064841, PubMed: 23724101 Thomson, J. B. (2005). Books in the digital age: The transformation of academic and higher education publishing in Britain and the United States. Cambridge, UK: Polity Press. Torres-Salinas, D., Robinson-Garcia, N., Campanario, J. M., & López-Cózar, E. D. (2014). Coverage, field specialisation and the impact of scientific publishers indexed in the Book Citation Index. Online Information Review, 38(1), 24–42. https://doi.org /10.1108/OIR-10-2012-0169 Torres-Salinas, D., Gumpenberger, C., & Gorraiz, J. (2017a). PlumX as a potential tool to assess the macroscopic multidimensional impact of books. Frontiers in Research Metrics and Analytics, 2(5), 1–11. https://doi.org/10.3389/frma.2017.00005 Torres-Salinas, D., Robinson-Garcia, N., & Gorraiz, J. (2017b). Filling the citation gap: Measuring the multidimensional impact of the academic book at institutional level with PlumX. Scientometrics, 113(3), 1371–1384. https://doi.org/10.1007/s11192-017-2539-z Uddannelses-og-Forskningsministeriet. (2018). Organisationsstruktur for Den Bibliometriske Forskningsindikator. Available at: https://ufm.dk /forskning-og-innovation/statistik-og-analyser/den-bibliometriske -forskningsindikator/organisering/organisationsstruktur_for_bfi.pdf Van Leeuwen, T. (2007). Modelling of bibliometric approaches and importance of output verification in research performance assess- ment. Research Evaluation, 16(2), 93–105. https://doi.org/10 .3152/095820207X227529 Verleysen, F. T., & Engels, T. C. E. (2013). Brief communication. A label for peer-reviewed books. Journal of the Association for Information Science and Technology, 64(2), 428–430. https:// doi.org/10.1002/asi.22836 Verleysen, F. T., & Engels, T. C. E. (2014). A label for peer-reviewed books. Journal of the Association for Information Science and Technology, 64(2), 428–430. https://doi.org/10.1002/asi.22836 Wahlfors, L., & Pölönen, J. (2018). Julkaisufoorumi-luokituksen käyttö yliopistoissa, Hallinnon Tutkimus, 37(1), 7–21. Waltman, L. (2016). A review of the literature on citation impact factors. Journal of Informetrics, 10(2), 365–391. https://doi.org /10.1016/j.joi.2016.02.007 Waltman, L., & Traag, V. A. (2021). Use of the Journal Impact Factor for assessing individual articles need not be wrong. F100 Research, 9, 366. https://doi.org/10.12688/f1000research .23418.2, PubMed: 33796272 Wang, Y., & Zuccala, A. (2021). Scholarly book publishers as pub- licity agents for SSH titles on Twitter. Scientometrics. https://doi .org/10.1007/s11192-021-03947-6 Wildegaard, L. (2019). An overview of author-level indicators of research performance. In W. Glanzel, H. Moed, U. Schmoch, & M. Thelwall (Eds.), Handbook of science and technology indicators (pp. 361–396). Springer. https://doi.org/10.1007/978-3-030-02511-3_14 White, H., Boell, S. K., Yu, H., Davis, M., Wilson, C. S., & Cole, F. T. H. (2009). Libcitations: A measure for comparative assessment of book publications in the humanities and social sciences. Journal of the Association for Information Science and Technology, 60(6), 1083–1096. https://doi.org/10.1002/asi.21045 White, H., & Zuccala, A. (2018). Libcitations, WorldCat, cultural impact, and fame. Journal of the Association for Information Science and Technology, 69(12), 1502–1512. https://doi.org/10 .1002/asi.24064 Zuccala, A., Costas, R., & van Leeuwen, T. N. (2010). Evaluating research departments using individual level bibliometrics. Eleventh International Conference on Science and Technology Indicators (pp. 314–316). CWTS- Leiden University, Leiden. Available at: https://www.cwts.nl/pdf/ BookofAbstracts2010 _version_15072010.pdf#page=314 Zuccala, A., Guns, R., Cornacchia, R., & Bod, R. (2015). Can we rank scholarly book publishers? A bibliometric experiment with the field of history. Journal of the Association for Information Science and Technology, 66(7), 1333–1347. https://doi.org/10 .1002/asi.23267 Zuccala, A. A. Giménez-Toledo, E., & Peruginelli, G. (2018). Scholarly books and their evaluation context in the social sciences and humanities. Aslib Journal of Information Management, 70(6), 586–591. https://doi.org/10.1108/AJIM-11-2018-271 Zuccala, A., & Robinson-Garcia, N. (2019). Reviewing, indicating and counting books for modern research evaluation systems. In W. Glanzel, H. Moed, U. Schmoch, & M. Thelwall (Eds.), Handbook of Science and Technology Indicators (pp. 715–728). Springer. https://doi.org/10.1007/978-3-030-02511-3_27 Zuccala, A., & White, H. (2015). Correlating libcitations and cita- tions in the humanities with WorldCat and Scopus Data. In A. A. Salah, Y. Tonta, A. A. Akdag Salah, C. Sugimoto, & U. Al (Eds.), Proceedings of the 15th International Society for Scientometrics and Informetrics Conference (pp. 305–316). Istanbul, Turkey: Bogazici University. Quantitative Science Studies 615 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / e d u q s s / a r t i c e - p d l f / / / / 2 2 5 8 8 1 9 3 0 6 9 3 q s s _ a _ 0 0 1 3 4 p d / . f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 ARTÍCULO DE INVESTIGACIÓN imagen

Descargar PDF