Citizen Science and - Specialized Research AI at MIT

Citizen Science and
Scientific Objectivity:
Mapping Out Epistemic
Risks and Benefits

Baptiste Bedessem
Université de Lyon, France

Stéphanie Ruphy
Ecole normale supeŕioure—
Université PSL, France

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

Given the importance of the issue of scientiﬁc objectivity in our democratic societies
and the signiﬁcant development of citizen science, it is crucial to investigate how
citizen science may either undermine or foster scientiﬁc objectivity. This paper iden-
tiﬁes a variety of epistemic risks and beneﬁts that participation of lay citizens in
scientiﬁc inquiries may bring. It also discusses concrete actions and pending issues
that should be addressed in order to foster objectivity in citizen science programs.

Introduction

1.
Nowadays, the issue of the nature and sources of scientiﬁc objectivity largely
exceeds the academic sphere. Indeed, objectivity is commonly considered to be
the very basis of the authority of science in our society and is seen as a key pre-
condition for public trust in science. As numerous political decisions call on
scientiﬁc expertise, questions concerning the very nature and limits of scientiﬁc
objectivity are becoming central to public debate. At the same time, calls for a
larger implication of lay citizens in the process of knowledge and expertise pro-
duction itself are on the rise. The general concept of “citizen science” refers in
the current literature to a large diversity of forms of participation of citizens
who are not professional scientists (individual citizens, NGOs, groups of pa-
tients, etc.) in the production of scientiﬁc knowledge, in ways that go beyond
the traditional roles that research subjects have had in the biomedical sciences

This work was supported ﬁnancially by the IDEXLYON Impulsion2018 project PartiSCiP,
Université de Lyon. www.partiscip.com

Perspectives on Science 2020, vol. 28, no. 5
©2020 by The Massachusetts Institute of Technology

https://doi.org/10.1162/posc_a_00353

630

e
d
u
p
o
s
c
/
a
r
t
i
c
e
–
p
d

f
/

2
8
5
6
3
0
1
8
8
9
5
2
6
p
o
s
c
_
a
_
0
0
3
5
3
p
d

b
y
g
u
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

Perspectives on Science

631

(e.g., in clinical trials) and in the social sciences (e.g., via interviews or ques-
tionnaires) (Cooper and Lewenstein 2016; Eitzel et al. 2017).1 Inclusiveness in
scientiﬁc research is more and more valued by scientiﬁc institutions, as shown
by their growing ﬁnancial support of citizen science and the numerous
commissioned reports (e.g., European Commission, 2013, 2016; Ofﬁce of
Science and Technology Policy 2019).

These reports emphasize that the development of citizen science should not
be seen as hype, as it corresponds to profound epistemological and political
trends affecting the way knowledge is produced in our democracies. On the
epistemological side, one of the major evolutions is the massive collection of
data, playing a key role in the development of many disciplines – from envi-
ronmental sciences to astronomy or bio-medicine (Leonelli 2014). And when
more and more data are on the scientiﬁc agenda, non-academic data collectors
are seen, not surprisingly, as a welcomed additional resource for scientiﬁc in-
quiry. On the political side, a larger involvement of citizens in the orientation of
research and/or in the production of knowledge is also valued, for the sake of
democracy, both in the philosophical literature (e.g., Kitcher 2011) and in
global scientiﬁc policy strategies: for instance, the notion of “Responsible
Research and Innovation” developed by the European commission implies that
“societal actors (researchers, citizens, policy makers […]) work together during
the whole research and innovation process.”2 The growing importance of
citizen science in scientiﬁc practices and in academic and political discourse
shows that science is no exception to the broader “participation imperative”
that is arising in our democracies (Godden 2017).

On the face of it, opening scientiﬁc inquiry to non-professionals may appear
quite challenging from an epistemological point of view, to the extent that
such opening may impact the internal self-regulatory mechanisms of scientiﬁc
communities that are commonly regarded as sources of objectivity (e.g., the
Mertonian norm of organized skepticism grounded on shared intellectual
attitude among professional scientists). Not surprisingly then—recall that
“among peers” is the rule in contemporary research—participatory practices
sometimes meet resistance from scientists themselves (Riesch and Potter
2014; Golumbic et al. 2017). Indeed, important epistemological concerns
have been raised, such those relating to the quality of the data collected by
crowd-sourcing programs (Resnik et al. 2015), but also those connected to
the lack of neutrality of the militant groups (for instance, NGOs) who engage

See Bedessem (2020) for a more systematic analysis of the various meanings of “citizen
science” as well as a state-of-the-art of the current literature on citizen science. In this paper, we
use the term “citizen” to refer to a non-professional inquirer.

https://ec.europa.eu/programmes/horizon2020/en/h2020-section/responsible-research-

2.
innovation

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
p
o
s
c
/
a
r
t
i
c
e
–
p
d

f
/

2
8
5
6
3
0
1
8
8
9
5
2
6
p
o
s
c
_
a
_
0
0
3
5
3
p
d

b
y
g
u
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

632

Citizen Science and Scientiﬁc Objectivity

in citizen science (Sarewitz 2000; Kinchy and Kleinman 2003). But, on the
other hand, inﬂuential works on scientiﬁc objectivity in contemporary philos-
ophy of science value the existence of a diversity of perspectives on a given issue
as a condition for well-founded objectivity (Harding 2015; Wylie 2015;
Longino 1990), thus departing from the idea of scientiﬁc objectivity as a “view
from nowhere.” These pluralist approaches to scientiﬁc objectivity are thus
more hospitable (at least prima facie) to the opening of scientiﬁc inquiries to
a variety of actors.

Given the importance of the issue of scientiﬁc objectivity in our democratic
societies and the signiﬁcant development of citizen science, it is then crucial to
investigate how citizen science may either undermine or foster scientiﬁc objec-
tivity and whether current resistance to more inclusiveness is well-founded. To
be successful and useful, especially to people engaged in participatory research
programs, such an investigation must take into account both the diversity of
participatory practices in science and the multiple dimensions of the very
notion of objectivity (Reiss and Sprenger 2017; Megill 1994). Consequently,
the ﬁrst aim of this paper is to develop a conceptual framework that allows us to
investigate the variety of epistemological challenges posed to scientiﬁc objec-
tivity by different forms of citizen participation, as well as the variety of epis-
temological beneﬁts that participation may bring.3 The second aim of this
paper is, in light of the previous analysis, to identify and discuss concrete
actions to be undertaken and pending issues that need to be addressed when
designing participatory research programs with a concern for fostering objec-
tivity in mind. We will proceed as follows: building on previous works by var-
ious authors, we will ﬁrst establish a typology of participatory practices in
contemporary science that are well adapted to our purpose, as well as a typology
of the different meanings of scientiﬁc objectivity based on an epistemic risk
account of the notion (Koskinen forthcoming; Biddle 2016). By coupling these
two typologies, we will establish a cartography of various speciﬁc challenges
and beneﬁts to objectivity in participatory research programs. In other words,
the intersection of the two typologies (objectivity-participation) will lead to
distinct objectivity/participation couples (o, p couples) raising speciﬁc issues,
and we will discuss in each successive case how the kind of participation
p may challenge or foster the dimension o of objectivity. For each couple
(o, p), we will also discuss concrete actions and pending issues that should be
addressed in order to foster objectivity.

3. A more complete approach would have to address ethics challenges in addition to
epistemological ones. Indeed, citizen science raises important speciﬁc ethics issues (Resnik,
Elliott, and Miller 2015; Rasmussen and Cooper 2019) that may eventually bear on the
overall epistemic quality of the inquiry. We thank one of our reviewers for mentioning this
line of development and leave it to another paper.

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
p
o
s
c
/
a
r
t
i
c
e
–
p
d

f
/

2
8
5
6
3
0
1
8
8
9
5
2
6
p
o
s
c
_
a
_
0
0
3
5
3
p
d

b
y
g
u
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

Perspectives on Science

633

The Many Faces of Citizen Science

2.
Participation of citizens to scientiﬁc inquiries today takes various forms, and
several classiﬁcations have been proposed in the last decade or so based on social,
institutional, or epistemic properties (Bucchi and Neresini 2008; Bonney et al.
2009; Roy et al. 2012; Haklay et al. 2013; King 2016; see Schrögel and
Kolleck (2019) for a review of the typologies currently available). In this paper,
we adopt the classiﬁcation proposed by Bonney et al. (2009). The justiﬁcation
of this choice is twofold. First, this classiﬁcation (or similar ones) is commonly
used in the academic and institutional literature on citizen science. Second, its
categories reﬂect different degrees of implication of non-professionals, corre-
sponding to various stages of scientiﬁc inquiry, thereby allowing the identiﬁ-
cation and differentiation of a variety of epistemological challenges. The
classiﬁcation distinguishes three main categories: “contributory” citizen
science, “collaborative” citizen science, and “co-created” citizen science.

When engaged in contributory citizen science, citizens are passive or active
data-collectors supervised by scientists. Numerous examples of this type of
citizen science program, also often called “crowd-sourcing” programs, can be
found in environmental science. Many countries have for instance developed
participative observatories to follow the evolution of biodiversity.4 The
“Zooniverse” platform5 brings together several research programs, enabling
people to participate in research in various ﬁelds, from astronomy (classifying
distant galaxies) to humanities (transcribing handwritten documents by
Shakespeare’s contemporaries).

When engaged in collaborative science, citizen participation goes beyond the
mere collection of data in two ways that are often intertwined in practice. First,
under the supervision of professional scientists, citizen may be involved in more
complex technical and cognitive tasks, such as the design of methods, of
research plans, and the interpretation of results. Second, they may also bring
their own speciﬁc expertise to the solution of a problem. The “fold-it” program
in fundamental biology, for instance, aims at studying the tridimensional struc-
ture of proteins by engaging volunteer citizens in a numerical serious game
(Kelly and Maddalena 2015). Various other programs exist which demand
strong commitment from a speciﬁc group of people, identiﬁed by scientists
as sharing some expertise or skills in a given matter. This is notably the case
in agronomic programs for developing countries. For instance, in the project
for participatory plant breeding run by the French center for research in
agriculture and development (CIRAD),6 farmers from developing countries

See for instance “Vigie-Nature” in France (Couvet and Prévot 2015). For a review

on this subject at the international scale, see Palacin-Silva et al. (2016).

5.
6.

https://www.zooniverse.org/
http://participatory-plant-breeding.cirad.fr/

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
p
o
s
c
/
a
r
t
i
c
e
–
p
d

f
/

2
8
5
6
3
0
1
8
8
9
5
2
6
p
o
s
c
_
a
_
0
0
3
5
3
p
d

b
y
g
u
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

634

Citizen Science and Scientiﬁc Objectivity

participate actively in research programs on plant genetic resources, drawing
on their own agricultural practices to elaborate knowledge in collaboration
with scientists. The World Bank also proposes various similar programs
aiming at improving the productivity of agriculture in southern countries,
by developing mutual epistemic relationships between citizens and scientists.7
In these ﬁrst two categories of citizen science, the initial formulation of the
research problems or questions is mainly made by scientists. In contrast, when
engaged in co-created citizen science (often also called “community-based
research”), citizens initiate a research program aiming at solving a problem that
they themselves have identiﬁed. In this case, the participants are better
described as stakeholders. The notion of stakeholders should be understood
here in the traditional sense it takes in the literature on public deliberation
(Kahane and Lopston 2013)—that is, as a group of people having a non-
cognitive interest in the problem under consideration because they are directly
affected by the problem or would be directly affected by its resolution. In these
cases, a group of citizens sharing a common concern is heavily involved in all
phases of scientiﬁc inquiry, from the co-construction of the research questions
to the collection of data, the establishment of interpretations, and the diffusion
of results. This type of citizen science often takes place in anthropology
(Rappaport 2008) and in the ﬁelds of public health and environmental sciences
(Irwin 1995; Den Broeder et al. 2016; Mielke et al. 2017), where it typically
involves a group of people facing an environmental risk (for instance pollution)
or affected by a speciﬁc disease (for instance, a rare genetic disease).

The Many Faces of Scientiﬁc Objectivity

3.
Objectivity is notoriously a multi-faceted and historically situated epistemo-
logical concept (Daston and Galison 2007). What is required for our project
in this paper are what Koskinen (forthcoming) has recently called “usable”
notions of objectivity, that is, notions that can actually be used to evaluate
research activities. And since citizen participation can concern various stages
of a scientiﬁc inquiry and comes in various shades of involvement, as brieﬂy
described in the previous section, we need notions of objectivity that allow
us to assess this variety of dimensions. The epistemic risk account of objectivity
developed by Koskinen (forthcoming) is particularly well-suited to our
purpose. In this perspective, objectivity is linked to the avoidance of an
epistemic risk, as deﬁned by Biddle and Kukla (2017, p. 218)—that is, a “risk
of epistemic error that arises anywhere during knowledge practices.” We say
that X (a researcher, a community, a research process, a knowledge claim) is

See the World Bank report dedicated to this issue: http://documents.worldbank.

org/curated/en/578241468765924957/pdf/multi0page.pdf

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
p
o
s
c
/
a
r
t
i
c
e
–
p
d

f
/

2
8
5
6
3
0
1
8
8
9
5
2
6
p
o
s
c
_
a
_
0
0
3
5
3
p
d

b
y
g
u
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

Perspectives on Science

635

objective when we believe that “important epistemic risks arising from our
imperfections as epistemic agents have been effectively averted” (Koskinen
forthcoming, p. 9). In other words, to a given domain of epistemic risks
corresponds a domain of objectivity, in the sense that avoiding this kind of
epistemic risks favor objectivity in the corresponding sense. Following
Koskinen’s approach, we investigate what kind of epistemic risks should be
envisaged in the case of citizen science programs. In light of the various kinds
of citizen involvement described earlier, we identify three main domains of
epistemic risks, hence three kinds of objectivity.

The ﬁrst domain concerns the reliability and epistemic quality of the exper-
imental and cognitive techniques and processes deployed in a scientiﬁc inquiry.
In this paper, we shall call the (traditional) kind of objectivity at stake in this
domain “Baconian objectivity.”8 The second domain of epistemic risks focuses
on issues of impartiality (Lacey 1999). Objectivity then refers to the absence of
distortion by explicit non-cognitive interests: it requires that, as epistemic
agents, we do not ignore or distort empirical evidence for the sake of explicit
contextual (i.e., political, cultural, social, or economic) preferences. The third
domain of epistemic risks relevant to citizen science concerns the quality of the
interactions between epistemic agents. Following Douglas’ (2004) and
Koskinen’s (forthcoming) terminology, we shall call the kind of objectivity
at stake here “interactive objectivity.” “Interactive objectivity” requires that
transparent processes of critical discussion between agents be ensured. This
third kind of objectivity can, to a certain extent, be considered as instrumental
to the two aforementioned kinds of objectivity. Indeed, critical discussions
between agents involved in a scientiﬁc inquiry may for instance allow the
revelation and solution of faults in an experimental protocol or the correction
of improper use of statistical tools. It may also permit identiﬁcation of situa-
tions in which individual preferences or interests over-ride empirical evidence.
But interactive objectivity also points to additional speciﬁc beneﬁts for critical
discussion, linked to the avoidance of biases by non-explicit, hidden back-
ground assumptions (Longino 1990), as we will discuss in greater details
below. This third domain of objectivity is thus clearly distinct from the second
one, which relates to the existence of explicit biases, whereas the third domain
relates to hidden, unexplicit ones.9 With this tripartite approach to scientiﬁc
objectivity, our ambition is not to cover exhaustively all kinds of epistemic
risks, but to distinguish between three domains of epistemic risks that are

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
p
o
s
c
/
a
r
t
i
c
e
–
p
d

f
/

2
8
5
6
3
0
1
8
8
9
5
2
6
p
o
s
c
_
a
_
0
0
3
5
3
p
d

b
y
g
u
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

8. This simply refers to Bacon’s caveat in The New Organon (Book I, aphorism 4)

against “discoloring the light of nature” in the course of a scientiﬁc inquiry.

9. Note also that Harding’s notion of strong objectivity (2015) is relevant here to the
extent that the existence of such background assumptions is related to the social, cultural,
and political location of the epistemic agents.

636

Citizen Science and Scientiﬁc Objectivity

relevant for citizen science and that each raise speciﬁc issues, both theoretical
and practical.

4. Mapping Out Epistemic Risks
By crossing our three-dimensional typology of objectivity (Baconian, value-
impartial, and interactive) and our three-dimensional classiﬁcation of citizen
participation in science (contributory, collaborative, and co-created) we iden-
tify and discuss in this section, for each couple (o, p), epistemic risks and
potential beneﬁts of participation, as well as theoretical and practical obsta-
cles or challenges still to be overcome in order to foster better citizen science
practices. Results of this analysis are summarized in Table 1. We start with
Baconian objectivity and investigate successively how each type of citizen
science may impact it.

4.1. Baconian Objectivity
As Baconian objectivity is dependent on the reliability of the protocols
(observational, experimental, inferential, etc.) followed by the inquirers,
epistemic risks and potential beneﬁts of citizen participation will vary with
the phase of the scientiﬁc inquiry and the difﬁculty of the associated tasks. Let
us turn ﬁrst to citizen participation restricted to the phase of data collection,
that is, contributory citizen science.

4.1.1. Contributory Science. When a research program mobilizes non-
professionals as data collectors, a ﬁrst, straightforward potential epistemic beneﬁt
is simply: more data. But its value evidently depends on the quality of the data
collected. The ability of the data collectors to appropriately follow protocols and
rules designed by professional scientists is thus the key source of epistemic risks.
At least three responses to this issue of data quality have not only been discussed
in the literature on citizen science but also implemented in citizen science
programs: training of the participants, interactive adaptation of the protocols,
and ex-post control of data quality. Let us brieﬂy discuss them in turn.

As noted by Kosmala et al. (2016), training is the most obvious approach to
improving data quality, and most existing citizen science programs include a
phase of training. It has been shown in different projects in ecology and envi-
ronmental sciences that effective training allows participants to produce data of
a quality similar to the quality of data produced by professional scientists
(Prysby and Oberhauser 2004; Danielsen et al. 2014). In some programs, this
training is associated with a measurement of the reliability of the participants,
by using skill tests. The result is then used by professional scientists to select
participants (see for instance Watson and Floridi (2016) on how this is achieved
by the Zooniverse numerical platform, the world largest citizen science portal).
Concerning the production of adapted protocols by scientists, a ﬁrst (obvi-
ous) challenge is to design easily understandable protocols to be used by lay

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
p
o
s
c
/
a
r
t
i
c
e
–
p
d

f
/

2
8
5
6
3
0
1
8
8
9
5
2
6
p
o
s
c
_
a
_
0
0
3
5
3
p
d

b
y
g
u
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

Perspectives on Science

637

citizens. Beyond this ex-ante production of accessible scientiﬁc guidelines,
interactive processes between citizens and scientists are also implemented to
adapt protocols in the course of the scientiﬁc inquiry. Some programs have
implemented such an iterative improvement of the tasks performed by volun-
teers, by designing, testing and comparing various simple and reliable data
collection protocols (Crall et al. 2010).

Ex-post control of data quality is by no means reserved to citizen science
programs, but it does take speciﬁc forms in this kind of scientiﬁc inquiry.
As noted by Haywood (2014), there already exists a large variety of techniques
used by scientists to manage and control the reliability of large data sets
collected by non-professionals. For instance, Watson and Floridi (2016)
describe how various biases are corrected in the Galaxy Zoo project: classiﬁca-
tion is weighted by considering the tendency of each user to divert from the
majority. In ecology and environmental sciences, Kosmala et al. (2016) explain
how systematic errors are modeled and corrected: these corrections can concern
biases also well-known in professional communities such as classiﬁcation biases,
or more speciﬁc ones, such as the high variability among volunteers in terms
ofability, effort, or commitment. In this latter case, statistical methods have
been developed to attenuate the effects of such variability. In other cases,
automatic ﬁlters are used in order to verify ex-post the internal consistency of
the data sets collected by each participant (Kelling et al. 2011). In ecology
and environmental sciences, citizens may be asked to regularly deliver some
samples (for instance, a photo, a video, or a physical specimen) justifying the
identiﬁcation and classiﬁcation they make (Delaney et al. 2008). These samples
are then used by professional scientists to measure the accuracy of the data col-
lected and classiﬁed by the participants.

When proper training and interactive processes of adaptation of protocols
are implemented, the current literature on data quality in citizen science pro-
grams suggests that scientists manage to successfully control the epistemic
risks raised by entrusting the task of data collection to non-professional
collectors (Chatzigeorgiou et al. 2016; Kosmala et al. 2016). As regards
Baconian objectivity, the main challenge for professional scientists in contrib-
utory citizen science programs thus turns out to concern the creation of efﬁ-
cient techniques and spaces of interaction with non-professional scientists. It
should be noted that this kind of communication and organizational skills are
not part of the set of technical and social skills usually expected from a good
scientist, since professional scientists are primarily trained to interact with
each other, rather than with non-professionals. Therefore, the main challenge
here is not so much epistemological than educational and social: expected
skills for professional scientists (or at least for a subset of professional scientists)
should also include the ability to interact efﬁciently with non-professionals
(who are not in the usual student-mentor relationship).

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
p
o
s
c
/
a
r
t
i
c
e
–
p
d

f
/

2
8
5
6
3
0
1
8
8
9
5
2
6
p
o
s
c
_
a
_
0
0
3
5
3
p
d

b
y
g
u
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

638

Citizen Science and Scientiﬁc Objectivity

4.1.2. Collaborative Science. In collaborative science, we have seen that
citizens may make two kinds of contributions, often intertwined in practice.
Under the supervision of professional scientists, citizens may be involved in
more complex technical and cognitive tasks than data collection, such as the
design of methods, of research plans, and the interpretation of results, and they
may also bring their own speciﬁc expertise to contribute to the solution of a
problem under the supervision of scientists. As regards the ﬁrst form of contri-
bution, here again, as in contributory science, the main challenge to meet is in
terms of control by professional scientists of the technical and epistemic quality
of the various tasks performed by the lay participants. Ex-post control proce-
dures and interactive training are also the main tools usually employed to
control this kind of epistemic risk associated with Baconian objectivity. Bonney
et al. (2009) provides various examples of such strategies. Consider for instance
the “salal harvest sustainability study” (2001–2004) that aimed at determining
the effect of harvest intensity on the plant’s growth (Bonney et al. 2009 p. 32).
This project enlisted the salal harvesters to select the sites where the data would
be collected, to design the methods and standards to measure the effects of
harvesting, and to collaborate on the interpretation of data. To ensure that this
collaboration led to reliable knowledge, professional scientists and harvesters
interacted before, during, and after the program. The harvesters ﬁrst followed
training on the basic elements to be taken into consideration when selecting
sites, designing a method of measure, and interpreting the results. The analysis
of the results was made in the context of a collaborative workshop where citizens
interacted with scientists to discuss the graphs obtained, and the recommenda-
tions for the future. After the program, the impact of the project in terms of
education and knowledge gain for the harvesters was assessed through an inter-
view by an exterior evaluator, in order to evaluate the efﬁciency of the technical
and cognitive learning attained by the participants. Overall, the current litera-
ture assessing the quality of the knowledge produced by this kind of
collaboration suggests, as for contributory science, that the epistemic risks
associated with Baconian objectivity may be successfully averted, provided that
adequate spaces and techniques of communication and collaboration are set up
(Parrish et al. 2018; Herman-Mercer et al. 2018)

As mentioned above, collaborative science is not only a matter of beneﬁtting
from a larger task force, so to speak, in every phase of a scientiﬁc inquiry, but is
also a matter of drawing on the lay expertise that the participants may bring.10
What then are the speciﬁc challenges and beneﬁts of this kind of citizen

10. The notion of lay expertise is often used in the literature to refer to various forms
of expertise developed by people who are not professional scientists. A lay person may ac-
quire some scientiﬁc knowledge and methods to be able to interact with professional sci-
entists and experts. She may also draw on her own life experience or professional experience
to acquire her own speciﬁc expertise. It is sometimes the case that lay expertise mixes both.

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
p
o
s
c
/
a
r
t
i
c
e
–
p
d

f
/

2
8
5
6
3
0
1
8
8
9
5
2
6
p
o
s
c
_
a
_
0
0
3
5
3
p
d

b
y
g
u
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

Perspectives on Science

639

contribution? First, concerning the potential beneﬁts, a growing literature points
to various robust epistemic gains, such as enriching the pool of data available to
scientists or enlarging the space of possible causes to be considered in the in-
terpretation of a given phenomenon. In the biomedical ﬁeld, for instance, a
well-documented case is the contribution of AIDS patients, who brought their
proper expertise on side-effects of their treatment to help develop better under-
standing and treatment of the disease (Epstein 1995; Godlee 2016). Another
well-known example of the successful contribution of lay expertise is provided
by Wynne (1998), with his now classical analysis of the “Cumbria sheep”: the
speciﬁc expertise of the shepherds on sheep grazing turned out to be essential to
scientists to understanding the effect of radioactivity on the natural environ-
ment of this region of England. In archeology, Wylie (2015) discusses how
collaborative practices between professional archeologists and descent commu-
nities (e.g., indigenous people) may favor, and have actually signiﬁcantly
improved archaeological science. Beyond their speciﬁcities, such cases have
in common the fact of showing that the participation of citizens who contribute
their own lay expertise can increase Baconian objectivity, to the extent that it
can improve the overall epistemic quality of the scientiﬁc inquiry process in
various ways: it can enrich the pool of available data by making visible to pro-
fessional scientists phenomena or data that were invisible to them, and it can
improve the interpretation of data and the understanding of a phenomenon by
providing interpretative perspectives inaccessible to professional scientists.11
Beyond the analysis of cases of successful contributions of lay expertise, it
would be interesting, on a more general note, to investigate the conditions
fostering the successful integration of lay expertise into a scientiﬁc inquiry.
From an epistemological point of view, this would take us to issues of the
integrability of different types of knowledge, to the extent that lay expertise
refers to some type of knowledge grounded in one’s speciﬁc experience. The
recent philosophical literature on scientiﬁc pluralism could be a resource here,
even if it has so far mainly focused on issues of the compatibility (or incompat-
ibility) of different models, approaches, and theoretical perspectives produced
within science, typically in different disciplines (e.g., Longino 2013; Mitchell
2003; Ruphy 2016).12 From a more practical perspective, it should ﬁrst
be noted that cases of successful contribution of lay expertise such as those

11.

Standpoint theories provide here a very useful theoretical background to aid in
grasping this kind of epistemic beneﬁt (see for instance Rolin (2016) for a recent overview
of standpoint theories), especially Harding’s notion of strong objectivity (2015). See also
the notion of “local epistemology” (Longino and Lennon 1997). In the sociological litera-
ture, Collins and Evans (2002) offers a now classical discussion of the epistemic contribu-
tion of lay expertise.

12. Koskinen and Mäki (2016) is a programmatic paper pleading for more involve-

ment of pluralist philosophers in the study of extra-academic transdisciplinarity.

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
p
o
s
c
/
a
r
t
i
c
e
–
p
d

f
/

2
8
5
6
3
0
1
8
8
9
5
2
6
p
o
s
c
_
a
_
0
0
3
5
3
p
d

b
y
g
u
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

640

Citizen Science and Scientiﬁc Objectivity

mentioned above are not without difﬁculties concerning the interaction between
professional scientists and lay citizens (Kullenberg 2015). This should not
come as a surprise. After all, as often emphasized in the literature on interdis-
ciplinarity, it is hard enough to get professional researchers from different dis-
ciplinary backgrounds to interact and work smoothly together. Issues of a lack
of mutual understanding, the lack of a common language, “imperialism” of one
discipline over another etc., often hamper the inquiry (Mäki et al. 2018). The
challenge is obviously even greater when it comes to collaborative science in-
volving professional scientists and lay experts coming from very different epi-
stemic (and sometimes cultural) backgrounds. Here again, as in the case of
contributory science, we suggest that the training of professional scientists
(or a subset of them) should be reconsidered, so that it also includes epistemo-
logical elements on lay expertise as well as practical training on how to deal
with non-professional experts.

4.1.3. Co-Created Science. When engaged in co-created science, citizens
(as stakeholders) collaborate with scientists to resolve speciﬁc problems that
they themselves have identiﬁed and are thus more autonomous in the inquiry
process when compared to the two previous types of citizen science. A straight-
forward beneﬁt of co-created citizen science is an enrichment of the pool of
questions addressed by scientiﬁc inquiry, resulting in an enrichment of the pool
of available data, to the extent that some data are collected directly in relation to
the new questions brought to light by citizens. But we would contend that the
most signiﬁcant link with Baconian objectivity results from a kind of second
order mechanism. In the case of co-created science where citizens are more
autonomous, professional scientists are not in a position to closely manage
and control every step of the inquiry in order to increase its reliability and
epistemic quality. However, there exists a strong incentive for the stakeholders
collaborating with professional scientists to ensure Baconian objectivity. Since
it is (still) widely considered as the basis of epistemic authority, Baconian
objectivity is valued and sought after by the stakeholders in order to get
recognition by political and scientiﬁc authorities of the reality of the problems
they are facing. In other words, it is in their interest, when collaborating with
professional scientists, to conform to the usual standards of reliability and
epistemic quality in place in scientiﬁc communities if the stakeholders want
their practical problems to be taken into serious consideration. Let us illustrate
this point with a well-known example of a co-created research program in the
environmental sciences. As they were living near a Shell plant, a group of
inhabitants of the Diamond subdivision in Norco (Louisiana) decided to assess
the air pollution that they were undergoing.13 The citizens collected the air

13. We draw here on the detailed presentation of this case given by Schrögel and

Kolleck 2019.

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
p
o
s
c
/
a
r
t
i
c
e
–
p
d

f
/

2
8
5
6
3
0
1
8
8
9
5
2
6
p
o
s
c
_
a
_
0
0
3
5
3
p
d

b
y
g
u
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

Perspectives on Science

641

samples using “buckets” and the analysis of the samples was performed in
professional laboratories. Their goal was to convince professional scientists of
the reality of the risks to their health posed by the Shell plant. They thus
had a strong incentive to strengthen the reliability and epistemic quality of
the data they were collecting, by conforming to usual scientiﬁc protocols
and standards of data collection. Citizens also interpreted and discussed the
results of the analysis of samples, in light of the standards ofﬁcially used by
authorities concerning acceptable levels of concentration. In addition, they
discussed the overall reliability of the methods (e.g., distribution of the mea-
surements points, frequency of the measurements) and standards used to assess
the environmental contamination due to the Shell plant, showing that some
revisions were required in order to improve the epistemic quality of the risk
assessment (Ottinger 2010).

4.2. Objectivity as Impartiality
Epistemic risks in the domain of impartiality are mainly linked (not surpris-
ingly) to issues of conﬂicts of interests and, more generally, to research miscon-
duct, but also, in the speciﬁc case of co-created science, to issues of unbalanced
production of expertise and fragmentation of the research agenda.

4.2.1. Contributory and Collaborative Science. When citizens involved in
various stages of a scientiﬁc inquiry have stakes in its outputs, a ﬁrst (obvious)
risk is that in order to reach a particular outcome, they may be tempted to
depart from standards of integrity in research. For instance, having stakes can
incite the modiﬁcation, fabrication, or falsiﬁcation of data, thus undermining
the overall reliability and epistemic quality of the inquiry. But is the risk
serious? To the best of our knowledge, there are no studies evaluating this
effect. As in the case of scientiﬁc misconduct (including conﬂicts of interests
by professional scientists), such misconduct is difﬁcult to document. An
interesting – and still pending – issue is whether the control mechanisms,
regulations, and institutional measures that are increasingly implemented in
scientiﬁc communities to foster research integrity could be successfully applied
to a larger community that includes non-professional inquirers. We suggest
that it is unlikely, if only because the reputational mechanisms and scientiﬁc
reward systems that play a central role in the auto-regulation of scientiﬁc com-
munities are not present in larger, non-professional communities. However, as
for professional scientists, a minimal safeguard that could be implemented in
citizen science programs is asking participants to report on possible conﬂicts of
interest, in order to foster transparency.

Interestingly, involving citizens having stakes in the outputs of a scientiﬁc
inquiry can also bring beneﬁts as regards objectivity as impartiality. As stake-
holders, participants may pay more attention to the existence of potential
conﬂicts of interest in professional researchers. Yamamoto (2012) gives the

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
p
o
s
c
/
a
r
t
i
c
e
–
p
d

f
/

2
8
5
6
3
0
1
8
8
9
5
2
6
p
o
s
c
_
a
_
0
0
3
5
3
p
d

b
y
g
u
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

642

Citizen Science and Scientiﬁc Objectivity

example, in environmental sciences, of militant groups having developed an
expertise on the credibility of professional scientists, by systematically
controlling biases due to potential conﬂicts of interest. Mutual checking by
professional and non-professional inquirers on possible conﬂicts of interests
may thus foster value impartiality objectivity (Elliott et al. 2017).

4.2.2. Co-Created Science. In addition to the previously-mentioned risks
that it shares with contributory and collaborative science, co-created science is
subject to a speciﬁc epistemic risk related to issues of unbalanced processes of
acquisition of scientiﬁc expertise on a given phenomenon or problem to be
solved. This is especially the case with multifaceted environmental or public
health issues. In such cases, various stakeholders with divergent political or
economic interests engage in co-created science programs in order to produce
scientiﬁc expertise concerning a limited dimension of the general issue, in
relation to their interests. Some authors have emphasized that this multipli-
cation of interest-related assessments often turns out to be detrimental both
epistemologically and politically (Sarewitz 2000, 2004; van der Vegt 2018).
On the epistemological side, some aspects of a phenomenon may be under-
studied because it is unlikely to serve the interests of a dominant group. For
example, to put it (too) brieﬂy, anti-GMOs will favor the production of
expertise on potential negative impacts on the environment, whereas the
food industry will favor studies on yield increase.14 Consequently, the overall
production of scientiﬁc expertise may be unbalanced: some aspects will be
thoroughly studied whereas others will be understudied, depending on power
relations. In other words, however unbiased and correct each individual
assessment may be, the overall picture resulting from their juxtaposition can
be biased, to the extent that it overemphasizes some aspects in regard to
other, understudied aspects. This kind of epistemic risk has political conse-
quences. In public debates, selective appraisals often appear as conﬂicting
appraisals (even when they do not address the same aspect of the multifac-
eted issue at stake) because they are put forward in support of different,
conﬂicting political actions. And this, in turn, complexiﬁes and slows down
processes of consensus formation, and hence political action.

Another risk worthy of being discussed here is the risk of fragmentation of
the research agenda. Since co-created science programs are mainly developed
by local communities to respond to speciﬁc interests or to solve speciﬁc prob-
lems that they face, the multiplication of such programs may lead to a jux-
taposition of research questions selected mainly on the basis of the political
visibility or activism of the stakeholder groups concerned. While how best to

14.

See Biddle (2018) for a much more detailed analysis of this controversy over

GMOs in terms of differences in values and interests, also Hicks (2015).

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
p
o
s
c
/
a
r
t
i
c
e
–
p
d

f
/

2
8
5
6
3
0
1
8
8
9
5
2
6
p
o
s
c
_
a
_
0
0
3
5
3
p
d

b
y
g
u
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

Perspectives on Science

643

ensure that every community is given a fair chance to develop the research
program it needs is a politically crucial issue, it is not the one that we will
address here. We will rather focus on the epistemological issue raised by the
potential effect of the fragmentation of research questions. Let us ﬁrst
rephrase this using the notion of “exogeneous” problem, as deﬁned in
Bedessem and Ruphy (2019). An exogeneous problem or research question
is simply a problem identiﬁed and formulated outside (or at least partly outside)
a scientiﬁc ﬁeld, incorporating various interests and expectations (not only
those of scientiﬁc communities), by contrast with an “endogenous” problem.
Research questions addressed in co-created science programs are typically
exogeneous. To what extent, then, may the juxtaposition of exogeneous
problems be problematic from an epistemological point of view? Our answer
to this question will be given in two stages. First, we will recall a general
point made in Bedessem and Ruphy (2019) about the epistemic fecundity of
integrating exogenous questions into the research agenda. Second, we will
discuss whether such epistemic beneﬁts may be gained in the speciﬁc case of
exogenous problems addressed in co-created science programs.

Drawing on various cases studies, especially in the ﬁeld of biomedicine,
Bedessem and Ruphy (2019) shows that the integration of exogeneous
problems in the development of a research program may lead to the
opening of new and fecund directions of endogenous inquiry. In other
words, the co-orientation of a research program by the integration of exoge-
neous problems within endogenous ones may be epistemologically virtuous
to the extent that it leads, in addition to the resolution of practical problems,
to an enrichment of the fundamental knowledge and the set of endogenous
research questions in the domain concerned.15 Let us turn now to the case of
exogeneous problems added to the scientiﬁc agenda by co-created science
programs. Can one expect the same kind of epistemic beneﬁts? By contrast
with the aforementioned “traditional” (i.e., non-participatory) research pro-
grams, we lack historical and epistemological studies on the impact of such exo-
geneous problems on the development of a given ﬁeld of academic research. But
our hunch is that their impact may not be particularly signiﬁcant for the
following reason: in the case of co-created science programs, exogenous
problems added to the scientiﬁc agenda often require a local solution, which
may not require the opening of any new lines of inquiry likely to increase
fundamental knowledge in the domain concerned. In other words, in the case

15. As discussed in more detail in Bedessem and Ruphy (2019), a good example of
this epistemologically virtuous integration is given by the history of molecular biology. The
practical search for new cancer treatments by targeting molecular regulation networks
(“rational drug design”) has strongly fostered the acquisition of new fundamental knowl-
edge in cell biology (Adam 2005).

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
p
o
s
c
/
a
r
t
i
c
e
–
p
d

f
/

2
8
5
6
3
0
1
8
8
9
5
2
6
p
o
s
c
_
a
_
0
0
3
5
3
p
d

b
y
g
u
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

644

Citizen Science and Scientiﬁc Objectivity

of co-created science, the selection of exogeneous problems is generally not
made in light of their potential interest for the development of the ﬁeld as
perceived by professional scientists but, rather, in light of their urgency, from
a practical point of view, of ﬁnding a solution. Consequently, the epistemolog-
ically virtuous effect of integrating exogenous to endogenous problems seems
less likely to occur. One might rather end up with a multiplicity of speciﬁc
exogeneous problems needing to be solved in isolation, without integration
in a common research agenda. This fragmentation of the research agenda
may not be deemed at all problematic from a practical (and political) point
of view, when what matters most is to be able to solve local problems. But from
an epistemological point of view, there is a risk is of weakening the virtuous
dynamics of integrating exogeneous problems within endogenous ones.

Interactive Objectivity

4.3.
Since interactive objectivity requires the implementation of transparent
processes of critical discussion between agents, epistemic risks in the domain
of such objectivity will generally turn out to be linked to issues of shared
standards and culture.

4.3.1. Contributory Science. When citizens are involved as collectors of
data, transparent processes of mutual criticism take the form of mechanisms
of mutual control and interactive learning between citizens, with the more
experienced guiding the less experienced, as described for instance by
Cosquer et al. (2012) in the case of contributory science programs developed
in biodiversity and conservation sciences. On a more general note, we suggest
that a better understanding of these interactive learning processes and of the
organizational conditions of their success is thus an important element for
fostering interactive objectivity.16

4.3.2. Collaborative Science and Co-Created Science. Well-known insights
from social epistemology suggest that proper processes of critical discussion
between epistemic agents allow the inﬂuence of hidden background assump-
tions to be made apparent at various stages of the scientiﬁc inquiry, thereby
limiting biases and increasing interactive objectivity. Such processes of “trans-
formative criticism” (Longino 1990) are all the more efﬁcient when the het-
erogeneity of perspectives of the agents involved is high: the less shared
background assumptions there are, the more likely the role they play will be-
come visible and can thus be challenged. As initially discussed by Longino,
interactive objectivity is enacted by transformative criticism taking place
within scientiﬁc communities. And in order to achieve “the transformative

16.

See Jennett et al. (2016) for an example of such studies in the case of online citizen

science programs.

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
p
o
s
c
/
a
r
t
i
c
e
–
p
d

f
/

2
8
5
6
3
0
1
8
8
9
5
2
6
p
o
s
c
_
a
_
0
0
3
5
3
p
d

b
y
g
u
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

Perspectives on Science

645

dimension of critical discourse”, scientiﬁc communities must satisfy four cri-
teria that are worth being recalled here: “(1) there must be recognized avenues
for the criticism of evidence, of methods, and of assumptions and reasoning;
(2) there must exist shared standards that critics can invoke; (3) the commu-
nity as a whole must be responsive to such criticism; (4) intellectual authority
must be shared equally among qualiﬁed practitioners” (Longino 1990, p. 76).
When it comes to assessing the potential epistemic beneﬁts and risks of
involving citizens in more complex technical and cognitive tasks than the
simple collection of data, the main question that needs to be addressed is as
follows: to what extent, and under what conditions, can these insights
from social epistemology be extended beyond the frontiers of scientiﬁc
communities?

On the face of it, the inclusion of non-professional inquirers in processes of
transformative criticism appears beneﬁcial since it is very likely to increase a
heterogeneity of perspectives, and thus the efﬁciency of the critical process.17
But this holds only when the four above-mentioned criteria are satisﬁed. Is it
likely to be the case with collaborative and co-created science? Let us here
make clear the potential difﬁculties for a more inclusive process of transfor-
mative criticism. As Longino explains when discussing the criterion of shared
standards: “in order for criticism to be relevant to a position it must appeal to
something accepted by those who hold the position criticized” (1990, p. 77).
In other words, agents involved in transformative criticism must feel bound
by the standards. Admittedly, it is not necessary that the different subcom-
munities share all the standards, but there must be some overlapping subsets
of standards that are shared by all. Shared professional training ensures that
this becomes the case for subcommunities within scientiﬁc communities
(e.g., empirical adequacy, at least, is expected to be a shared standard). In
principle, a ﬁrst difﬁculty for the extension of transformative criticism
beyond scientiﬁc communities is thus a possible asymmetry in responses to
mutual criticism: some subcommunities may not accept criticism because
they disagree on central norms of justiﬁcation (for instance a subcommunity
may not feel bound by the standard of empirical adequacy when it runs coun-
ter to authority or tradition). On a more practical side, another difﬁculty can
arise from a lack of shared practices and norms of interaction and communi-
cation. Here again, within scientiﬁc communities, common professional
training ensures that scientists share tacit knowledge concerning how to com-
municate with each other and when, for instance, another member of the
group can be trusted or not. When scientists come from different disciplines,
the price of communication and of a lack of mutual understanding rises

17.

See for instance Wylie (2015) for an epistemically successful example in archeology.

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
p
o
s
c
/
a
r
t
i
c
e
–
p
d

f
/

2
8
5
6
3
0
1
8
8
9
5
2
6
p
o
s
c
_
a
_
0
0
3
5
3
p
d

b
y
g
u
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

646

Citizen Science and Scientiﬁc Objectivity

quickly, as abundantly noted in the literature on interdisciplinarity (e.g.,
MacLeod 2018). On the face of it, difﬁculties can only be all the more serious
in the case of extra-academic interdisciplinarity (here co-created research).
And indeed, as Steel (2019) reminds us when discussing the epistemic effects
of diversity in relation to the elaboration of information, the effect of diversity
represents a “double-edged sword,” whose negative edge is the following:
“demographic diversity may generate obstacles to communication and trust,
which may impair group performance” (Steel et al. 2019, p. 2). Such difﬁcul-
ties are no doubt real and serious. And they are also still largely underana-
lyzed: empirical studies of the effects of demographic and cognitive
diversity within scientiﬁc communities (not to mention within co-created
science) are still very few and far between, and even if the philosophical lit-
erature on diversity in science—including citizen science—is growing fast, it
is too early to reach any deﬁnitive conclusions. The practical and organiza-
tional conditions under which the epistemic beneﬁts of transformative crit-
icism within scientiﬁc communities can be extended beyond the frontiers of
such communities is still very much an open research question, both empir-
ically and theoretically. In particular, empirical studies of actual cases of trans-
formative criticism involving citizens are greatly needed.

Table 1. Epistemic risks and beneﬁts, and pending issues for each couple
(o, p). o: type of objectiveity, p: type of participatory science.

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
p
o
s
c
/
a
r
t
i
c
e
–
p
d

f
/

2
8
5
6
3
0
1
8
8
9
5
2
6
p
o
s
c
_
a
_
0
0
3
5
3
p
d

b
y
g
u
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

Perspectives on Science

647

5. Discussion
Table 1 sums up the main epistemic beneﬁts, challenges, and pending issues
that we have identiﬁed with regard to objectivity in citizen science programs.
Without attempting to be exhaustive, our aim in this paper has been to provide
a cartography bringing to the fore the variety and speciﬁcities of the challenges
to be met in order to foster objectivity in citizen science, as well as a number of
open research questions calling for more empirical and theoretical work. Two
main (and related) lessons can be drawn from this cartography. First, from an
epistemological point of view, no obstacle has been identiﬁed that could not be
overcome, at least in principle. Most tools ensuring objectivity within profes-
sional scientiﬁc communities (e.g., control processes of data quality, transforma-
tive criticism) could in principle be extended to more inclusive communities of
inquirers. This is not to say that making these tools efﬁcient beyond the frontiers
of scientiﬁc communities is easy in real life, but we suggest that the difﬁculties
are not different in nature from the difﬁculties faced within professional
research communities engaged in interdisciplinary work. Second, it turns
out that the main challenges are thus not so much epistemological as institu-
tional and cultural. As many historical and sociological works have taught us,
scientiﬁc communities are characterized by a very high level of closure
(“among peers” is still the general rule). Communicating and interacting with
lay people in the context of a research inquiry and valuing their expertise is
not part and parcel of scientiﬁc training and culture. Such skills may be
acquired on the job, but they are not in general valued for career advancement
by most scientiﬁc institutions. Thus, and beyond ﬁnancial support, what is
needed from scientiﬁc institutions are also new incentives for professional
scientists (or a subset of them), especially as regards evaluation criteria and pro-
fessional training. In other words, what is needed is not only an increase (as is
usually claimed) of the scientiﬁc literacy of citizens, but also an increase of
professional scientists’ literacy concerning lay knowledge and expertise.18

References
Adam, Matthias. 2005. “Integrating Research and Development: The
Emergence of Rational Drug Design in the Pharmaceutical Industry.” Studies
in History and Philosophy of Biological and Biomedical Sciences 36: 513–537.
DOI: https://doi.org/10.1016/j.shpsc.2005.07.003, PMID: 16137601

18. When lay knowledge refers mainly to scientiﬁc knowledge and methods acquired
by people who are not professional scientists, literacy on lay knowledge amounts to make
professional scientists aware of such epistemic resources, thereby developing their capacity
to interact on scientiﬁc grounds with non-professionals. When lay knowledge refers mainly
to some knowledge grounded on one’s own speciﬁc life or professional experience, literacy
on lay knowledge amounts to inform professional scientists on the relevance of this type of
knowledge for scientiﬁc inquiry, thereby also developing their capacity to interact with
non-professionals and to draw epistemic beneﬁts from it.

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
p
o
s
c
/
a
r
t
i
c
e
–
p
d

f
/

2
8
5
6
3
0
1
8
8
9
5
2
6
p
o
s
c
_
a
_
0
0
3
5
3
p
d

b
y
g
u
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

648

Citizen Science and Scientiﬁc Objectivity

Bedessem, Baptiste, and Stéphanie Ruphy. 2019. “The Unpredictability of
Scientiﬁc Inquiry and Scientiﬁc Autonomy: The Unexpected Might
Not Be Where You Would Expect.” Studies in History and Philosophy
of Science 73: 1–7. DOI: https://doi.org/10.1016/j.shpsa.2018.08.002,
PMID: 30914120

Bedessem, Baptiste. 2020. “Sciences participatives: enjeux épistémologiques.”
Lato Sensu 7(1): 1–16. DOI: https://doi.org/10.20416/LSRSPS.V7I1.1
Biddle, Justin B. 2016. “Inductive Risk, Epistemic Risk, and Overdiag-
nosis of Disease.” Perspectives on Science 24: 192–205. DOI: https://doi.
org/10.1162/POSC_a_00200

Biddle, Justin B. 2018. “Antiscience Zealotry? Values, Epistemic Risk, and
the GMO Debate.” Philosophy of Science 85(3): 360–379. DOI: https://
doi.org/10.1086/697749

Biddle, Justin B., and Quill Kukla. 2017. The Geography of Epistemic
Risk. Pp. 215–237 in Exploring Inductive Risk: Case Studies of Values in
Science. Edited by K. C. Elliott and T. Richards. New York: Oxford
University Press.

Bonney, Rick, Hedi L. Ballard, Rebecca C. Jordan, Ellen McCallie, Tina
Phillips, Jennifer L. Shirk, and Candie C. Wilderman. 2009. Public Par-
ticipation in Scientiﬁc Research. Washington, DC: Center for Advancement
of Informal Science Education (CAISE).

Bucchi, Massimiano, and Federico Neresini. 2008. Science and Public
Participation. Pp. 449–472 in The Handbook of Science and Technology
Studies. Edited by E. J. Hackett, O. Amsterdamska, M. Lynch, and
J. Wajcman. Cambridge and London: The MIT Press.

Chatzigeorgiou, Giorgos, Sarah Faulwetter, Thanos Dailianis, Vincent S.
Smith, Panagiota Koulouri, Costas Dounas, and Christos Arvantidis.
2016. “Testing the Robustness of Citizen Science Projects: Evaluating the
Results of Pilot Project COMBER.” Biodiversity Data Journal 4: e10859.
DOI: https://doi.org/10.3897/BDJ.4.e10859, PMID: 28174507,
PMCID: PMC5267550

Collins, Harry M., and Robert Evans. 2002. “The Third Wave of Science
Studies: Studies of Expertise and Experience.” Social Studies of Science
32(2): 235–296. DOI: https://doi.org/10.1177/0306312702032002003
Cooper, Caren, and Bruce V. Lewenstein. 2016. Two Meanings of Citizen
Science. Pp. 51–61 in The Rightful Place of Science: Citizen Science. Edited
by D. Cavelier and E. B. Kennedy. Tempe, AZ: Consortium for Science,
Policy & Outcomes.

Cosquer, Alix, Richard Raymond, and Anne-Caroline Prévot-Julliard.
2012. “Observation of Everyday Biodiversity: a New Perspective for
Conservation?” Ecology and Society 17(4): 2. DOI: https://doi.org/10.5751/
ES-04955-170402

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
p
o
s
c
/
a
r
t
i
c
e
–
p
d

f
/

2
8
5
6
3
0
1
8
8
9
5
2
6
p
o
s
c
_
a
_
0
0
3
5
3
p
d

b
y
g
u
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

Perspectives on Science

649

Couvet, Denis, and Anne-Caroline Prévot. 2015. “Citizen-Science Pro-
grams: Towards Transformative Biodiversity Governance.” Environmental
Development 13: 39–45. DOI: https://doi.org/10.1016/j.envdev.2014.11.003
Crall, Alycia W., Gregory J. Newman, Catherine S. Jarnevich, Thomas J.
Stohlgren, Donald M. Waller, and Jim Graham. 2010. “Improving and
Integrating Data on Invasive Species Collected by Citizen Scientists.”
Biological Invasions 12: 3419–3428. DOI: https://doi.org/10.1007/
s10530-010-9740-9

Danielsen, Finn. et al. 2014. “A Multicountry Assessment of Tropical
Resource Monitoring by Local Communities.” BioScience 64: 236–251.
DOI: https://doi.org/10.1093/biosci/biu001

Daston, Lorraine, and Peter Galison. 2007. Objectivity. Cambridge: MIT

Press.

Delaney, Daved G., Corinne D. Sperling, Christiaan S. Adams, and Brian
Leung. 2008. “Marine Invasive Species: Validation of Citizen Science and
Implications for National Monitoring Networks.” Biological Invasions 10:
117–128. DOI: https://doi.org/10.1007/s10530-007-9114-0

Den Broeder, Lea, Jeroen Devilee, Hans Van Oers, A. Jantine Schuit, and
Annemarie Wagemakers. 2016. “Citizen Science for Public Health.”
Health Promotion International 33: 505–514. DOI: https://doi.org/
10.1093/heapro/daw086, PMID: 28011657, PMCID: PMC6005099
Douglas, Heather. 2004. “The Irreducible Complexity of Objectivity.”
Synthese 138: 453–473. DOI: https://doi.org/10.1023/B:SYNT.0000016451.
18182.91

Eitzel, Melissa V. et al. (2017). “Citizen Science Terminology Matters:
Exploring Key Terms.” Citizen Science: Theory and Practice 2(1): 1. DOI:
https://doi.org/10.5334/cstp.96

Elliott, Kevin, Aaron McCright, Allen Summer, and Thomas Dietz. 2017.
“Values in Environmental Research: Citizens’ Views of Scientists Who
Acknowledge Values.” PLoS One 12(10): e0186049. DOI: https://doi.
org/10.1371/journal.pone.0186049, PMID: 29069087, PMCID:
PMC5656313

Epstein, Steven. 1995. “The Construction of Lay Expertise: AIDS Activism
and the Forging of Credibility in the Reform of Clinical Trials.” Science,
Technology and Human Values 20(4): 408–437. DOI: https://doi.org/
10.1177/016224399502000402, PMID: 11653331

European Commission. 2013. Science for Environment Policy. In Depth-Report:
Environmental Citizen Science. https://ec.europa.eu/environment/integration/
research/newsalert/pdf/IR9_en.pdf

European Commission. 2016. Survey Report: Data Management in Citizen
Science Projects. Report number: EUR 27920 EN. DOI: https://doi.org/
10.2788/539115

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
p
o
s
c
/
a
r
t
i
c
e
–
p
d

f
/

2
8
5
6
3
0
1
8
8
9
5
2
6
p
o
s
c
_
a
_
0
0
3
5
3
p
d

b
y
g
u
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

650

Citizen Science and Scientiﬁc Objectivity

Godden, Naomi J. 2017. “The Participation Imperative in Co-Operative
Inquiry: Personal Reﬂections of an Initiating Researcher.” Systemic Practice
And Action Research 30(1): 1–18. DOI: https://doi.org/10.1007/s11213-
016-9387-2

Godlee, Fiona. 2016. “At Your Next Conference Ask Where the Patients
Are.” BMJ 354: i5123. DOI: https://doi.org/10.1136/bmj.i5123
Golumbic, Yaela, Daniela Orr, Ayelet Baram-Tsabari, and Barak Fishbain.
2017. “Between Vision and Reality: A Study of Scientists’ Views on
Citizen Science.” Citizen Science: Theory and Practice 2(1): 1–13. DOI:
https://doi.org/10.5334/cstp.53

Haklay, Muki. 2013. Citizen Science and Volunteered Geographic Infor-
mation Overview and Typology of Participation. Pp. 105–122 in
Crowdsourcing Geographic Knowledge: Volunteered Geographic Information
(VGI) in Theory and Practice. Edited by D. Z. Sui, S. Elwood, and M. F.
Goodchild. Haarlem: Springer. DOI: https://doi.org/10.1007/978-94-
007-4587-2_7

Harding, Sandra. 2015. Objectivity and Diversity: Another Logic of Scientiﬁc
Research. Chicago: Chicago University Press. DOI: https://doi.org/
10.7208/chicago/9780226241531.001.0001

Haywood, Benjamin K. 2014. “A Sense of Place in Public Participation in
Scientiﬁc Research.” Science Education 98(1): 64–83. DOI: https://doi.
org/10.1002/sce.21087

Herman-Mercer, Nicole, Ronald Antweiler, Nicole Wilson, Edda Mutter,
Ryan Toohey, and Paul Schuster. 2018. “Data Quality from a Community-
Based, Water-Quality Monitoring Project in the Yukon River Basin.”
Citizen Science: Theory and Practice 3(2): 113. DOI: https://doi.org/
10.5334/cstp.123

Hicks, Daniel. 2015. “Epistemological Depth in a GM Crops Controversy.”
Studies in the History and Philosophy of Biology and Biomedical Sciences 50: 1–12.
DOI: https://doi.org/10.1016/j.shpsc.2015.02.002, PMID: 25768981
Irwin, Alain. 1995. A Study of Citizen Expertise and Sustainable Development.

London: Routledge.

Jennett, Charlene, Laure Kloetzer, Daniel Schneider, Ioanna Iacovides,
Anna L. Cox, Margaret Gold, Brian Fuchs, Alexandra Eveleigh, Kathleen
Mathieu, Zoya Ajani, and Yasmin Talsi. 2016. “Motivations, Learning
and Creativity in Online Citizen Science.” Journal of Science Communication
15(3): A05. DOI: https://doi.org/10.22323/2.15030205

Kahane, David, and Kristjana Lopston. 2013. “Stakeholder and Citizen
Roles in Public Deliberation.” Journal of Public Deliberation 9(2): 1–37.
DOI: https://doi.org/10.16997/jdd.164

Kelling, Steve, Jun Yu, Jeff Gerbracht, and Weng-Keen Wong. 2011.
Emergent Filters: Automated Data Veriﬁcation in a Large-Scale Citizen

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
p
o
s
c
/
a
r
t
i
c
e
–
p
d

f
/

2
8
5
6
3
0
1
8
8
9
5
2
6
p
o
s
c
_
a
_
0
0
3
5
3
p
d

b
y
g
u
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

Perspectives on Science

651

Science Project. Pp. 20–27 in IEEE Seventh International Conference on e-Science,
Workshop Proceedings, 5–8 December. Stockholm. https://ieeexplore.ieee.
org/document/6130726; DOI: https://doi.org/10.1109/eScienceW.2011.13
Kelly, Ashley R., and Kate Maddalena. 2015. “Harnessing Agency for Efﬁcacy:
“Foldit” and Citizen Science.” Poroi 11(1): 1–20. DOI: https://doi.org/
10.13008/2151-2957.1184

Kinchy, Abbe J., and Daniel L. Kleinman. 2003. “Organizing Credibility:
Discursive and Organizational Orthodoxy on the Borders of Ecology and
Politics.” Social Studies of Science 33: 869–896. DOI: https://doi.org/
10.1177/0306312703336003

King, A. C. et al. 2016. “Leveraging Citizen Science and Information
Technology for Population Physical Activity Promotion.” Journal of
the American College of Sports Medicine 1: 30–44.

Kitcher, Philip. 2011. Science in a Democratic Society. Oxford: Oxford
University Press. DOI: https://doi.org/10.1163/9789401207355_003
Koskinen, Inkeri. Forthcoming. “Defending a Risk Account of Scientiﬁc
Objectivity.” The British Journal for the Philosophy of Science. DOI: https://
doi.org/10.1093/bjps/axy053

Koskinen, Inkeri, and Uskali Mäki. 2016. “Extra-Academic Transdiscipli-
narity and Scientiﬁc Pluralism: What Might They Learn from One
Another?” European Journal for Philosophy of Science 6(3): 419–444. DOI:
https://doi.org/10.1007/s13194-016-0141-5

Kosmala, Margaret, Andrea Wiggins, Alexandra Swanson, and Brooke
Simmons. 2016. “Assessing Data Quality in Citizen Science.” Frontiers
in Ecology and the Environment 14(10): 551–560. DOI: https://doi.org/
10.1002/fee.1436

Kullenberg, Christopher. 2015. “Citizen Science as Resistance: Crossing the
Boundary Between Reference and Representation.” Journal of Resistance
Studies 1(1): 50–76.

Lacey, Hugh. 1999. Is Science Value-Free? Values and Scientiﬁc Understanding.

London: Routledge.

Leonelli, Sabina. 2014. “What Differences Does Quantity Make? On the
Epistemology of Big Data in Biology.” Big Data & Society 1(1): 1–11.
DOI: https://doi.org/10.1177/2053951714534395, PMID: 25729586,
PMCID: PMC4340542

Longino, Helen. 1990. Science as Social Knowledge. Princeton: Princeton

University Press.

Longino, Helen. 2013. Studying Human Behavior. The University of Chicago
Press. DOI: https://doi.org/10.7208/chicago/9780226921822.001.0001
Longino, Helen, and Lennon, Kathleen. 1997. “Feminist Epistemology as a
Local Epistemology.” Proceedings of the Aristotelian Society, Supplementary
Volumes 71: 19–35. DOI: https://doi.org/10.1111/1467-8349.00017

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
p
o
s
c
/
a
r
t
i
c
e
–
p
d

f
/

2
8
5
6
3
0
1
8
8
9
5
2
6
p
o
s
c
_
a
_
0
0
3
5
3
p
d

b
y
g
u
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

652

Citizen Science and Scientiﬁc Objectivity

MacLeod, Miles. 2018. “What Makes Interdisciplinary Difﬁcult? Some
Consequences of Domain Speciﬁcity in Interdisciplinary Practice.” Synthese
195: 697–710. DOI: https://doi.org/10.1007/s11229-016-1236-4
Mäki, Uskali, Adrian Walsh, and Manuela Fernandez Pinto (Eds.). 2018.
Scientiﬁc Imperialism: Exploring the Boundaries of Interdisciplinarity. London:
Routledge. DOI: https://doi.org/10.4324/9781315163673

Megill, A. (1994). Introduction: Four Senses of Objectivity. Pp. 1–20 in
Rethinking Objectivity. Edited by A. Megill. Durham, NC: Duke University
Press.

Mielke, Jahel, Hannah Vermaben, and Saskia Ellenbeck. 2017. “Ideals,
Practices, and Future Prospects of Stakeholder Involvement in Sustainability
Science.” PNAS 114(50): E10648–E10657. DOI: https://doi.org/10.1073/
pnas.1706085114, PMID: 29162698, PMCID: PMC5740662

Mitchell, Sandra. 2003. Biological Complexity and Integrative Pluralism.
Cambridge: Cambridge University Press. DOI: https://doi.org/10.1017/
CBO9780511802683

Ofﬁce of Science and Technology Policy. 2019. Implementation of Federal
Prize and Citizen Science Authority: Fiscal Years 2017–2018. https://
www.whitehouse.gov/wp-content/uploads/2019/06/Federal-Prize-and-
Citizen-Science-Implementation-FY17-18-Report-June-2019.pdf

Ottinger, Gwen. 2010. “Buckets of Resistance: Standards and the Effectiveness
of Citizen Science.” Science, Technology & Human Values 35(2): 244–270.
DOI: https://doi.org/10.1177/0162243909337121

Palacin-Silva, Maria, Ahmed Seffah, Kari Heikkinen, Jari Porras, Timo
Pyhälahti, Yrjö Sucksdorff, Saku Anttila, Hanna Alasalmi, Eeva Bruun,
and Soﬁa Junttila. 2016. State-of-the Art Study in Citizen Observatories:
Technological Trends, Development Challenges and Research Avenues. Reports
of the Finnish Environment Institute, 28.

Parrish, J., Hillary Burgess, Jake F. Weltzin, Lucy Fortson, Andrea Wiggins,
and Brooke Simmons. 2018. “Exposing the Science in Citizen Science:
Fitness to Purpose and Intentional Design.” Integrative and Comparative
Biology 58(1): 150–160. DOI: https://doi.org/10.1093/icb/icy032, PMID:
29790942

Prysby, Michelle D., and Karen S. Oberhauser. 2004. Temporal and Geographic
Variation in Monarch Densities: Citizen Scientists Document Monarch Popu-
lation Patterns. Pp. 9–20 in Monarch Butterﬂy Biology & Conservation. Edited by
Karen S. Oberhauser and M. J. Solensky. Ithaca, NY: Cornell University Press.
Rappaport, Joanne. 2008. “Beyond Participant Observation: Collaborative
Ethnography as Theoretical Innovation.” Collaborative Anthropologies 1:
1–31. DOI: https://doi.org/10.1353/cla.0.0014

Rasmussen, Lisa M., and Caren Cooper. 2019. “Citizen Science Ethics.” Citizen
Science: Theory and Practice 4(1): 5–7. DOI: https://doi.org/10.5334/cstp.235

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
p
o
s
c
/
a
r
t
i
c
e
–
p
d

f
/

2
8
5
6
3
0
1
8
8
9
5
2
6
p
o
s
c
_
a
_
0
0
3
5
3
p
d

b
y
g
u
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

Perspectives on Science

653

Reiss, Julian, and Jan Sprenger. 2017. “Scientiﬁc Objectivity.” The Stanford
Encyclopedia of Philosophy (Winter 2017 edn.). Edited by Edward N.
Zalta. https://plato.stanford.edu/archives/win2017/entries/scientiﬁc-
objectivity/

Resnik, David B., Kevin Elliott, and Aubrey K. Miller. 2015. “A Framework
for Addressing Ethical Issues in Citizen Science.” Environmental Science
and Policy 54: 47581. DOI: https://doi.org/10.1016/j.envsci.2015.
05.008

Riesch, Hauke, and Potter, Clive. 2014. “Citizen Science as Seen by
Scientists: Methodological, Epistemological and Ethical Dimensions.”
Public Understanding of Science 23(1): 107–120. DOI: https://doi.org/
10.1177/0963662513497324, PMID: 23982281

Rolin, Kristina. 2016. “Values, Standpoints, and Scientiﬁc/Intellectual
Movements.” Studies in History and Philosophy of Science 56: 11–19.
DOI: https://doi.org/10.1016/j.shpsa.2015.10.008, PMID: 27083080
Roy, Helen, Michael Pocock, Chris D. Preston, David Roy, Joana Savage,
and John Tweddle. 2012. Understanding Citizen Science & Environmental
Monitoring. Final Report on behalf of UK-EOF. https://www.ceh.ac.uk/
sites/default/ﬁles/citizensciencereview.pdf

Ruphy, Stéphanie. 2016. Scientiﬁc Pluralism Reconsidered. Pittsburg: University

of Pittsburg Press. DOI: https://doi.org/10.2307/j.ctt1mtz6n9

Sarewitz, Daniel. 2000. Science and Environmental Policy: An Excess of
Objectivity. Pp. 79–98 in Earth Matters: The Earth Sciences, Philosophy,
and the Claims of Community. Edited by Robert Frodeman. Upper Saddle
River, NJ: Prentice Hall.

Sarewitz, Daniel. 2004. “How Science Makes Environmental Controversies
Worse.” Environmental Science and Policy 7(5): 385403. DOI: https://doi.
org/10.1016/j.envsci.2004.06.001

Schrögel, Philipp, and Alma Kolleck. 2019. “The Many Faces of Participa-
tion in Science: Literature Review and Proposal for a Three-Dimensional
Framework.” Science & Technology Studies 32(2): 77–99. DOI: https://doi.
org/10.23987/sts.59519

Steel, Daniel, Sina Fazelpour, Bianca Crewe, and Kinley Gillette. 2019.
“Information Elaboration and Epistemic Effects of Diversity.” Synthese.
DOI: https://doi.org/10.1007/s11229-019-02108-w

Van der Vegt, Rens. 2018. “A Literature Review on the Relationships
between Risk Governance and Public Engagement in Relation to Complex
Environmental Issues.” Journal of Risk Research 21(11): 1. DOI: https://doi.
org/10.1080/13669877.2017.1351466

Watson, David, and Floridi, Luciano. 2016. “Crowdsourced Science: Socio-
technical Epistemology in the E-Research Paradigm.” Synthese 195(2):
741–764. DOI: https://doi.org/10.1007/s11229-016-1238-2

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
p
o
s
c
/
a
r
t
i
c
e
–
p
d

f
/

2
8
5
6
3
0
1
8
8
9
5
2
6
p
o
s
c
_
a
_
0
0
3
5
3
p
d

b
y
g
u
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

654

Citizen Science and Scientiﬁc Objectivity

Wylie, Alison. 2015. A Plurality of Pluralisms: Collaborative Practice in
Archaeology. Pp. 189–210 in Objectivity in Science: New Perspectives from
Science and Technology Studies. Edited by Flavia Padovani, Alan Richardson,
and Jonathan Y. Tsou. Berlin: Springer. DOI: https://doi.org/10.1007/
978-3-319-14349-1_10

Wynne, Brian. 1998. May the Sheep Safely Graze? A Reﬂexive View Of
The Expert-Lay Knowledge Divide. Pp. 44–84 in Risk, Environment and
Modernity: Towards a New Ecology. Edited by Scott Lash, Bronislaw
Szerszynski, and Brian Wynne. Thousand Oaks: Sage Publication
Ltd. DOI: https://doi.org/10.4135/9781446221983.n3

Yamamoto, Yuri. 2012. “Values, Objectivity and Credibility of Scientists in
a Contentious Natural Resource Debate.” Public Understanding of Science
21(1): 101–125. DOI: https://doi.org/10.1177/0963662510371435,
PMID: 22530490

D
o
w
n
o
a
d
e
d

f
r
o
m
h

t
t

:
/
/

d
i
r
e
c
t
.

i
t
.

e
d
u
p
o
s
c
/
a
r
t
i
c
e
–
p
d

f
/

2
8
5
6
3
0
1
8
8
9
5
2
6
p
o
s
c
_
a
_
0
0
3
5
3
p
d

b
y
g
u
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3
Download pdf