Corpora Annotated with Negation: An
Overview

Salud Mar´ıa Jim´enez-Zafra
SINAI, Computer Science Department
CEATIC, Universidad de Ja´en
sjzafra@ujaen.es

Roser Morante
CLTL Lab, Linguistica computazionale
VU University Amsterdam
r.morantevallejo@vu.nl

Mar´ıa Teresa Mart´ın-Valdivia
SINAI, Computer Science Department
CEATIC, Universidad de Ja´en
maite@ujaen.es

l. Alfonso Ure ˜na-L ´opez
SINAI, Computer Science Department
CEATIC, Universidad de Ja´en
laurena@ujaen.es

l

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

P

:
/
/

D
io
R
e
C
T
.

M

io
T
.

e
D
tu
/
C
o

l
io
/

l

UN
R
T
io
C
e
–
P
D

F
/

/

/

/

4
6
1
1
1
8
4
7
7
6
9
/
C
o

Negation is a universal linguistic phenomenon with a great qualitative impact on natural lan-
guage processing applications. The availability of corpora annotated with negation is essential to
training negation processing systems. Currently, most corpora have been annotated for English,
but the presence of languages other than English on the Internet, such as Chinese or Spanish,
is greater every day. In this study, we present a review of the corpora annotated with negation
information in several languages with the goal of evaluating what aspects of negation have been
annotated and how compatible the corpora are. We conclude that it is very difficult to merge
the existing corpora because we found differences in the annotation schemes used, and most
importantly, in the annotation guidelines: the way in which each corpus was tokenized and the
negation elements that have been annotated. Differently than for other well established tasks
like semantic role labeling or parsing, for negation there is no standard annotation scheme nor
guidelines, which hampers progress in its treatment.

l
io

_
UN
_
0
0
3
7
1
P
D

.

F

B
sì
G
tu
e
S
T

T

o
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3

Invio ricevuto: 4 Dicembre 2018; revised version received: 23 ottobre 2019; accepted for publication:
17 novembre 2019.

https://doi.org/10.1162/COLI a 00371

© 2020 Associazione per la Linguistica Computazionale
Published under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 Internazionale
(CC BY-NC-ND 4.0) licenza

Linguistica computazionale

Volume 46, Numero 1

1. introduzione

Negation is a key universal phenomenon in language. All languages possess differ-
ent types of resources (morphological, lexical, syntactic) that allow speakers to speak
about properties that people or things do not hold or events that do not happen. IL
presence of a negation in a sentence can have enormous consequences in many real
world situations: A world in which Donald Trump was elected as president would
be very different from a world in which Donald Trump was not elected as president,
Per esempio. Così, the presence of a single particle modifying a proposition describes
a completely different situation. Negation is a main linguistic phenomenon and the
issue of its computational treatment has not been resolved yet due to its complexity,
the multiple linguistic forms in which it can appear, and the different ways it can act
on the words within its scope. If we want to develop systems that approach human
understanding, it is necessary to incorporate the treatment of one of the main linguistic
phenomena used by people in their daily communication.

Natural language processing (PNL) is a subfield of artificial intelligence that focuses
on the processing and generation of human language in order for computers to learn,
understand, and produce human language (Hirschberg and Manning 2015). Some lin-
guistic phenomena such as negation, speculation, irony, or sarcasm pose challenges for
computational natural language learning. One might think that, given the fact that nega-
tions are so crucial in language, most NLP pipelines incorporate negation modules and
that the computational linguistics community has already addressed this phenomenon.
Tuttavia, this is not the case. Work on processing negation has started relatively late
as compared to work on processing other linguistic phenomena and, as a matter fact,
there are no publicly available off-the-shelf tools that can be easily incorporated into
applications to detect negations.

Work on negation started in 2001 with the aim of processing clinical records
(Chapman et al. 2001UN; Mutalik, Deshpande, and Nadkarni 2001; Goldin and Chapman
2003). Some rule-based systems were developed based on lists of negations and stop
parole (Mitchell et al. 2004; Harkema et al. 2009; Mykowiecka, Marciniak, and Kup´s´c
2009; Uzuner, Zhang, and Sibanda 2009; Sohn, Wu, and Chute 2012). With the surge
of opinion mining, negation was studied as a marker of polarity change (Das and
Chen 2001; Wilson, Wiebe, and Hoffmann 2005; Polanyi and Zaenen 2006; Taboada
et al. 2011; Jim´enez-Zafra et al. 2017). Only with the release of the BioScope corpus
(Vincze et al. 2008) did the work on negation receive a boost. But even so, despite the
existence of several publications that focus on negation, it is difficult to find a negation
processor for languages other than English. For English, some systems are available for
processing clinical documents (NegEx [Chapman et al. 2001B], ConText [Harkema et al.
2009], Deepen [Mehrabi et al. 2015]) E, recently, a tool for detecting negation cues
and scopes in natural language texts has been published (Enger, Velldal, and Øvrelid
2017).

Four tasks are usually performed in relation to processing negation: (io) negation cue
detection, in order to find the words that express negation; (ii) scope identification, In
order to find which parts of the sentence are affected by the negation cues; (iii) negated
event recognition, to determine which events are affected by the negation cues; E (iv)
focus detection, in order to find the part of the scope that is most prominently negated.
Most of the works have modeled these tasks as token-level classification tasks, Dove
a token is classified as being at the beginning, inside, or outside a negation cue, scope,
event, or focus. Scope, event, and focus identification tasks are more complex because
they depend on negation cue detection. In this article we focus on reviewing existing

190

l

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

P

:
/
/

D
io
R
e
C
T
.

M

io
T
.

e
D
tu
/
C
o

l
io
/

l

UN
R
T
io
C
e
–
P
D

F
/

/

/

/

4
6
1
1
1
8
4
7
7
6
9
/
C
o

l
io

_
UN
_
0
0
3
7
1
P
D

.

F

B
sì
G
tu
e
S
T

T

o
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3

Jim´enez-Zafra et al.

Corpora Annotated with Negation: An Overview

corpora annotated with negation, without entering in the realm of reviewing negation
processing systems.

Most applications treat negation in an ad hoc manner by processing main negation
constructions, but processing negation is not as easy as using a list of negation markers
and applying look-up methods because negation cues do not always act as negators.
Per esempio, in the sentence “You bought the car to use it, didn’t you?” the cue “not”
is not used as a negation but it is used to reinforce the first part of the sentence. Noi
believe that there are three main reasons for which most applications treat negation
in an ad hoc manner: One is that negation is a complex phenomenon, which has not
been completely modeled yet. In this way it is similar to phenomena like factuality
for which it is necessary to read large amounts of theoretical literature in order to put
together a model, as shown by Saur´ı’s work on modeling factuality for its computational
treatment (Saur´ı and Pustejovsky 2009). A second reason is that, although negation is
a phenomenon of habitual use in language, it is difficult to measure its quantitative
impact in some tasks such as anaphora resolution or text simplification. The number of
sentences with negation in the English texts of the corpora analyzed is between 9.37%
E 32.16%, whereas in Spanish texts it is between 10.67% E 34.22%, depending on
the domain. In order to evaluate the improvement that processing negation produces, Esso
would be necessary to focus only on those parts of the text in which negation is present
and perform an evaluation before and after its treatment. Tuttavia, from a qualitative
perspective, its impact is very high—for example, when processing clinical records,
because the health of patients is at stake. A third reason is that there are no large corpora
exhaustively annotated with negation phenomena, which hinders the development of
machine learning systems.

Processing is relevant for a wide range of applications, such as information retrieval
(Liddy et al. 2000), information extraction (Savova et al. 2010), machine translation
(Baker et al. 2012), or sentiment analysis (Liu 2015). Information retrieval systems
aim to provide relevant documents from a collection, given a user query. Negation
has an important role because it is not the same to make a search (“recipes with milk
and cheese”) than to make the negated version of the search (“recipes without milk and
cheese”). The information retrieval system must return completely different documents
for both queries. In other tasks, such as information extraction, negation analysis is
also beneficial. Clinical texts often refer to negative findings, questo è, conditions that
are not present in the patient. Processing negation in these documents is crucial because
the health of patients is at stake. Per esempio, a diagnosis of a patient will be totally
different if negation is not detected in the sentence “No signs of DVT.” Translating a
negative sentence from one language into another is also challenging because negation
is not used in the same way. Per esempio, the Spanish sentence “No tiene ninguna
pretensi´on en la vida” is equivalent to the English sentence “He has no pretense in life”,
but in the first case two negation cues are used whereas in the second only one is
used. Sentiment analysis is also another task in which the presence of negation has a
great impact. A sentiment analysis system that does not process negation can extract a
completely different opinion than the one expressed by the opinion holder. Per esempio,
the polarity of the sentence “A fascinating film, I would repeat” should be the opposite of
its negation “A film nothing fascinating, I would not repeat.” Notwithstanding, negation
does not always imply polarity reversal, it can also increment, reduce, or have no effect
on sentiment expressions, which makes the task even more difficult.

Tuttavia, as we can see in some of the systems we use regularly, this phenomenon
is not being processed effectively. Per esempio, if we do the Google search in Spanish
“pel´ıculas que no sean de aventuras” (non-adventure movies), we obtain adventure movies,

191

l

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

P

:
/
/

D
io
R
e
C
T
.

M

io
T
.

e
D
tu
/
C
o

l
io
/

l

UN
R
T
io
C
e
–
P
D

F
/

/

/

/

4
6
1
1
1
8
4
7
7
6
9
/
C
o

l
io

_
UN
_
0
0
3
7
1
P
D

.

F

B
sì
G
tu
e
S
T

T

o
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3

Linguistica computazionale

Volume 46, Numero 1

which reflects that the engine is not taking into account negation. Other examples can
be found in online systems for sentiment analysis. If we analyze the Spanish sentence
“Jam´as recomendar´ıa comprar este producto.” (I would never recommend buying this product.)
with Mr. Tuit system1, we can see that the output returned by the system is positive but
the text clearly expresses a negative opinion. In the meaning cloud system2 we can find
another example. If we write the Spanish sentence “Este producto tiene fiabilidad cero.”
(This product has zero reliability.), the system indicates that it is a positive text, although
in fact it is negative.

One of the first steps when attempting to develop a machine learning negation
processing system is to check whether there are training data and to decide whether
their quality is good enough. Differently than for other well established tasks like
semantic role labeling or parsing, for negation there is no corpus of reference, but several
small corpora, E, ideally, a training corpus needs to be large for a system to be able
to learn. This motivates our main research questions: Is it possible to merge the existing
negation corpora in order to create a larger training corpus? What are the problems
that arise? In order to answer the questions we first review all existing corpora and
characterize them in terms of several factors: type of information about negation that
they contain, type of information about negation that is lacking, and type of application
they would be suitable for. Available corpora that contain a representation of negation
can be divided into two types (Fancellu et al. 2017): (io) those that represent negation in a
logical form, using quantifiers, predicates, and relations (per esempio., Groningen Meaning Bank
[Basile et al. 2012], DeepBank [Flickinger, Zhang, and Kordoni 2012]); E (ii) those that
use a string-level, where the negation operator and the elements (scope, event, focus) are
defined as spans of text (per esempio., BioScope [Vincze et al. 2008], ConanDoyle-neg [Morante
and Daelemans 2012]). It should be noted that we focus on corpora that deal with
string-level negation.

The rest of the article is organized as follows: In Section 2 previous overviews that
focus on negation are presented; in Section 3 the criteria used to review the existing
corpora annotated with negation are described; in Sections 4, 5, E 6 the existing
corpora for English, Spanish, and other languages are reviewed; in Section 7 we briefly
describe negation processing systems that have been developed using the corpora; In
Sezioni 8 E 9 the corpora are analyzed showing features of interest, applications
for which they can be used, and problems found for the development of negation
processing systems; and finally, conclusions are drawn in Section 10.

2. Related Work

To the best of our knowledge, there are currently no extensive reviews of corpora
annotated with negation, but there are overviews that focus on the role of negation.
An interesting overview on how modality and negation have been modeled in com-
putational linguistics was presented by Morante and Sporleder (2012). The authors
emphasize that most research in NLP has focused on propositional aspects of meaning,
but extra-propositional aspects, such as negation and modality, are also important to un-
derstanding language. They also observe a growing interest in the computational treat-
ment of these phenomena, evidenced by several annotations projects. In this overview,

1 http://www.mrtuit.com/.
2 https://www.meaningcloud.com/es/productos/analisis-de-sentimiento.

192

l

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

P

:
/
/

D
io
R
e
C
T
.

M

io
T
.

e
D
tu
/
C
o

l
io
/

l

UN
R
T
io
C
e
–
P
D

F
/

/

/

/

4
6
1
1
1
8
4
7
7
6
9
/
C
o

l
io

_
UN
_
0
0
3
7
1
P
D

.

F

B
sì
G
tu
e
S
T

T

o
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3

Jim´enez-Zafra et al.

Corpora Annotated with Negation: An Overview

modality and negation are defined in detail with some examples. Inoltre, details on
the linguistic resources annotated with modality and negation until then are provided
as well as an overview of automated methods for dealing with these phenomena. In
aggiunta, a summary of studies in the field of sentiment analysis that have modeled
negation and modality are shown. Some of the conclusions drawn by Morante and
Sporleder are that although work on the treatment of negation and modality has been
carried out in recent years, there is still much to do. Most research has been carried
out on the English language and on specific domains and genres (biomedical, recensioni,
newswire, eccetera.). At the time of this overview only corpora annotated with negation for
English had been developed, with the exception of one Swedish corpus (Dalianis and
Velupillai 2010). Therefore, the authors indicate that it would be interesting to look at
different languages and also distinct domains and genres, due to the fact that extra-
propositional meaning is susceptible to domain and genre effects. Another interesting
conclusion drawn from this study is that it would be a good idea to study which aspects
of extra-propositional meaning need to be modeled for which applications, and the
appropriate modeling of modality and negation.

In relation to the modeling of negation, we can reference one survey about the
role of negation in sentiment analysis (Wiegand et al. 2010). In this survey, several
papers with novel approaches to modeling negation in sentiment analysis are presented.
Sentiment analysis focuses on the automatic detection and classification of opinions
expressed in texts; and negation can affect the polarity of a word (usually positive,
negative, or neutral) because it can change, increment, or reduce the polarity value,
hence the importance of dealing with this phenomenon in this area. The authors study
the level of representation used for sentiment analysis, negation word detection, E
scope of negation. In relation to the representation of negation, the usual way to
incorporate negation in supervised machine learning is to use a bag-of-words model
adding a new feature NOT x. Così, if a word x is preceded by a negation marker (per esempio.,
non, never), it would be represented as NOT x and as x in any other case. Pang, Lee,
e Vaithyanathan (2002) followed a similar approach but they added the tag NOT to
every word between a negation cue and the first punctuation mark. They found that the
effect of adding negation was relatively small, probably because the introduction of the
feature NOT x increased the feature space. Later, negation was modeled as a polarity
shifter and not only negation was considered, but also intensifiers and diminishers.
Negation was incorporated into models including knowledge of polar expressions by
changing the polarity of an expression (Polanyi and Zaenen 2004; Kennedy and Inkpen
2006) or encoding negation as features using polar expressions (negation features,
shifter features, and polarity modification features) (Wilson, Wiebe, and Hoffmann
2005). The results obtained with these models led to a significant improvement over
the bag-of-words model. The conclusion drawn by the authors of this survey is that
negation is highly relevant to sentiment analysis and that for a negation model to be
effective in this area, knowledge of polar expressions is required. Inoltre, they state
that negation markers do not always function as negators and, consequently, need to
be disambiguated. Another interesting remark is that, despite the existence of several
approaches to modeling negation for sentiment analysis, to make affirmations of the
effectiveness of the methods it is necessary to carry out comparative analysis with
regard to classification type, text granularity, target domain, lingua, and so forth.
The papers presented in this study are the pioneering studies of negation modeling in
sentiment analysis for English texts. In recent studies researchers have been developing
rule-based systems using syntactic dependency trees (Jia, Yu, and Meng 2009), applying
more complex calculations in order to obtain polarity (Taboada et al. 2011), using

193

l

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

P

:
/
/

D
io
R
e
C
T
.

M

io
T
.

e
D
tu
/
C
o

l
io
/

l

UN
R
T
io
C
e
–
P
D

F
/

/

/

/

4
6
1
1
1
8
4
7
7
6
9
/
C
o

l
io

_
UN
_
0
0
3
7
1
P
D

.

F

B
sì
G
tu
e
S
T

T

o
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3

Linguistica computazionale

Volume 46, Numero 1

deep-learning (Socher et al. 2013), and using machine-learning with lexical and syntactic
caratteristiche (Cruz, Taboada, and Mitkov 2016a).

The studies analyzed above were carried out on English texts, but interest in pro-
cessing negation in languages other than English has been increasing in recent years.
Jim´enez-Zafra et al. (2018UN) recently presented a review of Spanish corpora annotated
with negation. The authors consulted the main catalogs and platforms that provide
information about resources and/or access to them (LDC catalog,3 ELRA catalog,4 LRE
Map,5 META-SHARE,6 and ReTeLe7) with the aim of developing a negation processing
system for Spanish. Because of the difficulty in finding corpora annotated with negation
in Spanish, they conducted an exhaustive search of these resources. Di conseguenza, Essi
provided a description of the corpora found as well as the direct links for accessing the
data where possible. Inoltre, the main features of the corpora were analyzed in order
to determine whether the existing annotation schemes account for the complexity of
negation in Spanish, questo è, whether the typology of negation patterns in this language
(Marti et al. 2016) was taken into account in the existing annotation guidelines. IL
conclusions drawn from this analysis were that the Spanish corpora are very different
in several aspects: the genres, the annotation guidelines, and the aspects of negation
that have been annotated. As a consequence, it would not be possible to merge all of
them to training a negation processing system.

3. Criteria for Corpus Review

An essential requirement for developing machine learning systems is the availability
of annotated corpora, and also that the corpora be large enough and the annotations
consistent. In order to gain insight into the available data sets, we reviewed all existing
corpora annotated with negation, based on several criteria of analysis that we present
in this section. To the best of our knowledge, there are corpora annotated for English,
Spanish, Swedish, Chinese, Dutch, German, and Italian. For each corpus we collected in-
formation about the source of the texts, and the size and the percentage of sentences that
contain negation. Inoltre, we indicate what type of information has been annotated,
whether the annotation has been thought of for a specific task, and whether negation
is the main focus of the annotation. In relation to negation, we specify what types of
negation have been annotated (syntactic, lexical, morphological), what elements have
been annotated (cue, scope, event, focus), and what guidelines have been followed for
the annotation. Inoltre, we include information on the number of annotators, their
background, and how the inter-annotator agreement was measured. Finalmente, we also
provide information on the availability of the corpora and their format. Prossimo, we define
the criteria that have been applied to review the corpora:

•

Language: The language(S) of the texts included in the corpus. Questo
characteristic should always be specified in the description of any corpus,
as it conditions its use.

3 https://catalog.ldc.upenn.edu/.
4 http://catalog.elra.info/en-us/.
5 http://lremap.elra.info/.
6 http://www.meta-share.org/.
7 http://linguistic.linkeddata.es/retele-share/sparql-editor/.

194

l

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

P

:
/
/

D
io
R
e
C
T
.

M

io
T
.

e
D
tu
/
C
o

l
io
/

l

UN
R
T
io
C
e
–
P
D

F
/

/

/

/

4
6
1
1
1
8
4
7
7
6
9
/
C
o

l
io

_
UN
_
0
0
3
7
1
P
D

.

F

B
sì
G
tu
e
S
T

T

o
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3

Jim´enez-Zafra et al.

Corpora Annotated with Negation: An Overview

•

•

•

•

•

•

•

Domain: Field to which the texts belong. Although cross-domain
methodologies are being used for many tasks (Li et al. 2012; Szarvas et al.
2012; Bollegala, Mu, and Goulermas 2016), the domain of a corpus partly
determines its area of application since different areas have different
vocabularies.

Availability: Accessibility of the corpora. We indicate whether the corpus
is publicly available and we provide the links for obtaining the data when
possible. Corpora annotation is time-consuming and expensive, so it is not
only necessary that corpora exist, but also that they be publicly available
for the research community to use.

Guidelines: We study the guidelines used for the annotation showing
similarities and differences between corpora. The definition of guidelines
for the annotation of any phenomenon is fundamental because the
generation of quality data will depend on it. The goal of annotation
guidelines can be formulated as follows: given a theoretically described
phenomenon or concept, describe it as generically as possible but as
precisely as necessary so that human annotators can annotate the concept
or phenomenon in any text without running into problems or ambiguity
issues (Ide 2017).

Sentences: Corpus size is measured in sentences. The number of sentences
is the information that is usually provided in the statistics of a corpus to
give an idea of its extension, although the important thing is not the
number of sentences but the information contained in them.

Annotated elements: This aspect refers to the elements on which the
annotation has been performed, such as sentences, events, relationships,
and so forth.

Elements with negation: Total number of elements that have been
annotated with negation. As has been mentioned before, the number of
annotated sentences is not important, but rather the information annotated
in them. The annotation should cover all the relevant cases that algorithms
need to process in order to allow for a rich processing of negation.

Negation types: Refers to the types of negation that have been annotated.
There are different types of negation depending on the type of negation
cue used (Jim´enez-Zafra et al. 2018B):

–

–

Syntactic negation, if a syntactically independent negation marker
is used to express negation (per esempio., NO [‘no/not’], nunca [‘never’]).
Lexical negation, if the cue is a word whose meaning has a
negative component (per esempio., negar [‘deny’], desistir [‘desist’]).

– Morphological negation, if a morpheme is used to express

negation (per esempio., io- in ilegal [‘illegal’], in in incoherente [‘incoherent’]). It
is also known as affixal negation.

•

Negation components: Components of negation that have been annotated:

–

Cues: lexical items that modify the truth value of the propositions
that are within their scope (Morante 2010), questo è, they are words
that express negation. Negation cues can be adverbs (per esempio., I have
never been to Los Angeles), pronouns (per esempio., His decisions have nothing

l

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

P

:
/
/

D
io
R
e
C
T
.

M

io
T
.

e
D
tu
/
C
o

l
io
/

l

UN
R
T
io
C
e
–
P
D

F
/

/

/

/

4
6
1
1
1
8
4
7
7
6
9
/
C
o

l
io

_
UN
_
0
0
3
7
1
P
D

.

F

B
sì
G
tu
e
S
T

T

o
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3

195

Linguistica computazionale

Volume 46, Numero 1

to do with me), verbs (per esempio., The magazine desisted from published false
stories about the celebrity), and words with negative prefixes (per esempio.,
What you’ve done is illegal). They may consist of a single token (per esempio.,
I do not like the food of this restaurant), a sequence of two or more
contiguous tokens (per esempio., He has not even tried it), or two or more
non-contiguous tokens (per esempio., I am not going back at all). IL
annotation of cues in corpora is very important because they are
the elements that act as triggers of negation. The identification of
negation cues is usually the first task that a negation processing
system needs to perform, hence the importance of the annotation of
corpora with this information.
Scope: part of the sentence affected by the negation cue (Vincze
et al. 2008), questo è, all elements whose individual falsity would
make the negated statement strictly true (Blanco and Moldovan
2011B). Per esempio, consider the sentence (UN) My children do not like
meat and its positive counterpart (B) My children like meat. In order
for (B) to be true the following conditions must be satisfied: (io)
somebody likes something, (ii) my children are the ones who like it,
E (iii) meat is what is liked. The falsity of any of them would
make (UN) VERO. Therefore, all these elements are the scope of
negation: My children do not like meat. The words identified as scope
are those on which the negation acts and on which it will be
necessary to make certain decisions based on the objective of the
final system. Per esempio, in a sentiment analysis system, these
words could see their polarity modified.
Negated event: the event that is directly negated by the negation
cue, usually a verb, a noun, or an adjective (Kim, Ohta, and Tsujii
2008). The negated event or property is always within the scope of
a cue, and it is usually the head of the phrase in which the negation
cue appears. Per esempio, in the sentence “Technical assistance did
not arrive on time,” the event is the verbal form “arrive,” which is the
head of the sentence. There are some domains in which the
identification of the negated events is crucial. Per esempio, in the
clinical domain it is relevant for the correct processing of diagnoses
and for the analysis of clinical records.
Focus: part of the scope that is most prominently or explicitly
negated (Blanco and Moldovan 2011a). It can also be defined as the
part of the scope that is intended to be interpreted as false or whose
intensity is modified. It is one of the most difficult aspects of
negation to identify, especially without knowing the stress or
intonation. Per esempio, in the sentence “I’m not going to the concert
with you,” the focus is “with you” because what is false is not the
fact of going to the concert, but the fact of going with a specific
persona (with you). Detecting the focus of negation is useful for
retrieving the numerous words that contribute to implicit positive
meanings within a negation (Morante and Blanco 2012).

–

–

–

Esempio (1) shows a sentence with the last four elements, which have been
explained above. The negation cue appears in bold, the event in italics, the focus
underlined, and the scope between [brackets]. The adverb “no”/no is the negation cue

196

l

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

P

:
/
/

D
io
R
e
C
T
.

M

io
T
.

e
D
tu
/
C
o

l
io
/

l

UN
R
T
io
C
e
–
P
D

F
/

/

/

/

4
6
1
1
1
8
4
7
7
6
9
/
C
o

l
io

_
UN
_
0
0
3
7
1
P
D

.

F

B
sì
G
tu
e
S
T

T

o
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3

Jim´enez-Zafra et al.

Corpora Annotated with Negation: An Overview

because it is used to change the meaning of the words that are within its scope. IL
negated event is the verbal form “tiene”/has and the focus is the noun “l´ımites”/limits,
because it is the part that is intended to be false, it is equivalent to saying “cero
l´ımites”/zero limits. The scope goes from the negation cue8 to the end of the verb phrase,
although this is not always the case, or else it would be very easy to detect the words
affected by the negation. In Example (2) we show a sentence in which the scope of
negation is the whole sentence and, in Example (3), a sentence with two coordinated
structures with independent negation cues and predicates in which a scope is annotated
for each coordinated negation marker.

1. Es una persona que [no tiene l´ımites], aunque a veces puede controlarse.

He is a person who has no limits, although sometimes he can control himself.

2. [El objetivo de la c´amara nunca ha funcionado bien].

The camera lens has never worked well.

3. [No soy alta] aunque [tampoco soy un pitufo].

I’m not tall, but I’m not a smurf either.

In this section we have presented the aspects that we have described for each
corpus. In Sections 4, 5, E 6, we present the existing corpora annotated with negation
grouped by language. In Section 9 we provide an analysis of all the factors and we
summarize them in different tables than can be found in Appendix A.

4. English Corpora

As we already indicated, our analysis focuses on corpora with string-level annotations.
We are aware of two corpora that do not follow this annotation approach: Groningen
Meaning Bank (Basile et al. 2012) and DeepBank (Flickinger, Zhang, and Kordoni 2012).
The Groningen Meaning Bank9 corpus is a collection of semantically annotated English
texts with formal meaning representations rather than shallow semantics. It is com-
posed of newswire texts from Voice of America, country descriptions from the CIA Fact-
book, a collection of texts from the open ANC (Ide et al. 2010), and Aesop’s fables. It was
automatically annotated using C&C tools and Boxer (Curran, Clark, and Bos 2007) E
then manually corrected. The DeepBank corpus10 contains rich syntactic and semantic
annotations for the 25 Wall Street Journal sections included in the Penn Treebank (Taylor,
Marcus, and Santorini 2003). The annotations are for the most part produced by manual
disambiguation of parses licensed by the English Resource Grammar (Flickinger 2000).
It is available in a variety of representation formats.

To the best of our knowledge, the following are corpora that contain texts in English

and string-level annotations.

4.1 BioInfer

The first corpus annotated with negation was BioInfer (Pyysalo et al. 2007). It focuses
on the development of Information Extraction systems for extracting relationships be-
tween genes, proteins, and RNAs. Therefore, only entities relevant to this focus were
annotated. It consists of 1,100 sentences extracted from the abstracts of biomedical

8 There are authors that do not include the negation cue within the scope.
9 The Groningen Meaning Bank is available at http://gmb.let.rug.nl.
10 DeepBank is available at http://moin.delph-in.net/DeepBank.

197

l

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

P

:
/
/

D
io
R
e
C
T
.

M

io
T
.

e
D
tu
/
C
o

l
io
/

l

UN
R
T
io
C
e
–
P
D

F
/

/

/

/

4
6
1
1
1
8
4
7
7
6
9
/
C
o

l
io

_
UN
_
0
0
3
7
1
P
D

.

F

B
sì
G
tu
e
S
T

T

o
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3

Linguistica computazionale

Volume 46, Numero 1

research articles that were annotated with named entities and their relationships, E
with syntactic dependencies including negation predicates. Out of 2,662 relationships,
163 (6%) are negated using the predicate NOT. The predicate NOT was used to annotate
any explicit statements of the non-existence of a relationship. For this purpose, the three
types of negation were considered: syntactic, morphological, and lexical. The scope of
negation was not annotated as such, but the absence of a relationship between entities,
such as not affected by or unable to, was annotated with the predicate NOT:

4. Abundance of actin is not affected by calreticulin expression. (See Figure 1.)

NOT(affected by:AFFECT(abundance of actin, calreticulin expression))

Figura 1
Annotated example from the BioInfer corpus (not affected by).

5. N-WASP mutant unable to interact with profilin. (See Figure 2.)

NOT(interact with:BIND(N-WASP mutant, profilin))

Figura 2
Annotated example from the BioInfer corpus (unable to).

In relation to the annotation process, this was divided into two parts. On the one
hand, the dependency annotations were created by six annotators who worked in
rotating pairs to reduce variation and avoid systematic errors. Two of the annotators
were biology experts and the other four had the possibility of consulting with an expert.
D'altra parte, the entity and relationship annotations were created based on a
previously unpublished annotation of the corpus and were carried out by a biology
expert, with difficult cases and annotation rules being discussed with two Information
Extraction researchers. The inter-annotator agreement was not measured in this corpus
because the authors considered that there were some difficulties in calculating the kappa
statistic for many of the annotation types. They said that they intended to measure
agreement separately for the different annotation types, applying the most informative
measures for each type but, to the best of our knowledge, this information was not
published. The annotation manual used for producing the annotation can be found at
http://tucs.fi/publications/view/?pub_id=tGiPyBjHeSa07a.

The BioInfer corpus is in XML format, licensed under a Creative Commons
Attribution-ShareAlike 3.0 Unported License and can be downloaded at http://mars.
cs.utu.fi/BioInfer/.

198

l

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

P

:
/
/

D
io
R
e
C
T
.

M

io
T
.

e
D
tu
/
C
o

l
io
/

l

UN
R
T
io
C
e
–
P
D

F
/

/

/

/

4
6
1
1
1
8
4
7
7
6
9
/
C
o

l
io

_
UN
_
0
0
3
7
1
P
D

.

F

B
sì
G
tu
e
S
T

T

o
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3

Jim´enez-Zafra et al.

Corpora Annotated with Negation: An Overview

4.2 Genia Event

The Genia Event corpus (Kim, Ohta, and Tsujii 2008) is composed of 9,372 sentences
from Medline abstracts that were annotated with biological events and with negation
and uncertainty. It is an extension of the Genia corpus (Ohta, Tateisi, and Kim 2002; Kim
et al. 2003), which was annotated with the Part Of Speech (POS), syntactic trees, E
terms (biological entities).

As for negation, it was annotated whether events were explicitly negated or not,
using the label non-exists or exists, rispettivamente. The three types of negation were consid-
ered, but linguistic cues were not annotated.

6. This pathway involves the Rac1 and Cdc42 GTPases, two enzymes that are not
required for NF-kappaB activation by IL-1beta in epithelial cells. (See Figure 3.)

This pathway involves the Rac1 and Cdc42 GTPases,
two enzymes which are not
required for NF-kappaB activation by IL-1beta
in epithelial cells.

Figura 3
Annotated example from the Genia Event corpus.

l

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

P

:
/
/

D
io
R
e
C
T
.

M

io
T
.

e
D
tu
/
C
o

l
io
/

l

UN
R
T
io
C
e
–
P
D

F
/

/

/

/

4
6
1
1
1
8
4
7
7
6
9
/
C
o

Out of a total of 36,858 tagged events, 2,351 events were annotated as explicitly
negated. The annotation process was carried out by a biologist and three graduate
students in molecular biology following the annotation guidelines defined.11 However,
there is no information about inter-annotator agreement.

The corpus is provided as a set of XML files, and it can be downloaded at http://
www.geniaproject.org/genia-corpus/event-corpus under the terms of the Creative
Commons Public License.

4.3 BioScope

The BioScope corpus (Vincze et al. 2008) is one of the largest corpora and is the first
in which negation and speculation markers have been annotated with their scopes. It
contains 6,383 sentences from clinical free-texts (radiology reports), 11,871 sentences
from full biological papers, E 2,670 sentences from biological paper abstracts from
the GENIA corpus (Collier et al. 1999). In total, it has 20,924 sentences, out of which
2,720 contains negations.

Negation is understood as the implication of the non-existence of something.
The strategy for annotating keywords was to mark the minimal unit possible (only
lexical and syntactic negations were considered). The largest syntactic unit possible

l
io

_
UN
_
0
0
3
7
1
P
D

.

F

B
sì
G
tu
e
S
T

T

o
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3

11 http://www.nactem.ac.uk/meta-knowledge/Annotation_Guidelines.pdf.

199

Linguistica computazionale

Volume 46, Numero 1

should be annotated as scope. Inoltre, negation cues were also included within the
scope.

7. PMA treatment, and not retinoic acid treatment of the U937 cells, acts in inducing

NF-KB expression in the nuclei. (See Figure 4.)

PMA treatment, E

non
retinoic acid treatment of the U937 cells

acts in inducing NF-KB expression in the nuclei.

Figura 4
Annotated example from the BioScope corpus.

The corpus was annotated by two independent linguist annotators and a chief
linguist following annotation guidelines.12 The consistency level of the annotation was
measured using the inter-annotator agreement rate defined as the Fβ − 1 measure of one
annotazione, considering the second one as the gold standard. The average agreement of
negation keywords annotation was 93.69, 93.74, E 85.97 for clinical records, abstracts,
and full articles, rispettivamente, and the average agreement of scope identification for the
three corpora was 83.65, 94.98, E 78.47, rispettivamente.

The BioScope corpus is in XML format and is freely available for academic purposes
at http://rgai.inf.u-szeged.hu/index.php?lang=en&page=bioscope. This corpus
was also used in the CoNLL-2010 Shared Task: Learning to detect hedges and their scope
in natural language text (Farkas et al. 2010).

4.4 Product Review Corpus

the Product Review corpus was presented (Councill, McDonald, E
In 2010,
Velikovich 2010b). It is composed of 2,111 sentences from 268 product reviews extracted
from Google Product Search. This corpus was annotated with the scope of syntactic
negation cues and 679 sentences were found to contain negation. Each review was
manually annotated with the scope of negation by a single person, after achieving inter-
annotator agreement of 91% with a second person on a smaller subset of 20 recensioni
containing negation. Inter-annotator agreement was calculated using a strict exact span
criteria where both the existence and the left/right boundaries of a negation span were
required to match. In questo caso, negation cues were not included within the scope. IL
guidelines used for the annotation are described in the work in which the corpus was
presented.

The format of the corpus is not mentioned by the authors and is not publicly
available. Tuttavia, we contacted the authors and they sent us the corpus. In this way
we were able to see that it is in XML format and extract an example of it:

8. I am a soft seller, If you don’t want or need the services offered that’s cool with

me. (See Figure 5.)

12 The annotation guidelines can be downloaded at http://rgai.inf.u-szeged.hu/project/nlp/

bioscope/Annotation%20guidelines2.1.pdf and a discussion of them can be found in Vincze (2010).

200

l

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

P

:
/
/

D
io
R
e
C
T
.

M

io
T
.

e
D
tu
/
C
o

l
io
/

l

UN
R
T
io
C
e
–
P
D

F
/

/

/

/

4
6
1
1
1
8
4
7
7
6
9
/
C
o

l
io

_
UN
_
0
0
3
7
1
P
D

.

F

B
sì
G
tu
e
S
T

T

o
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3

Jim´enez-Zafra et al.

Corpora Annotated with Negation: An Overview

I am a soft seller, If you don’t

want or need the services offered

that’s cool with me.

Figura 5
Annotated example from the Product Review corpus.

4.5 PropBank Focus (PB-FOC)

In 2011, the PropBank Focus (PB-FOC) corpus was presented. It introduced a new
element for the annotation of negation, the focus. Blanco and Moldovan (2011UN) selected
3,993 verbal negations contained in 3,779 sentences from the WSJ section of the Penn
TreeBank marked with MNEG in the PropBank corpus (Palmer, Gildea, and Kingsbury
2005), and performed annotations of negation focus. They reduced the task to selecting
the semantic role most likely to be the focus.

Fifty percent of the instances were annotated twice by two graduate students in
computational linguistics and an inter-annotator agreement of 72% percent was ob-
tained (it was calculated as the percentage of annotations that were a perfect match).
Later, disagreements were examined and resolved by giving annotators clearer instruc-
zioni. Finalmente, the remaining instances were annotated once. The annotation guidelines
defined are described in the paper in which the corpus was presented.

This corpus was used in Task 2, focus detection, at the *SEM 2012 Shared Task
(Resolving the scope and focus of negation) (Morante and Blanco 2012). It is in CoNLL
format (Farkas et al. 2010) and can be downloaded at http://www.clips.ua.ac.be/
sem2012-st-neg/data.html. Figura 6 shows the annotations for Example (4.5). IL
columns provide the following information: token (1), token number (2), POS tag (3),
named entities (4), chunk (5), parse tree (6), syntactic head (7), dependency relation (8),
semantic roles (9 to previous to last, with one column per verb), negated predicates
(previous to last), focus (last).

PB-FOC is distributed as standalone annotations on top of the Penn TreeBank. IL
distribution must be completed with the actual words from the the Penn TreeBank,
which is subject to an LDC license.

9. Marketers believe most Americans won’t make the convenience trade-off. (Vedere

Figura 6.)

4.6 ConanDoyle-neg

The ConanDoyle-neg (Morante and Daelemans 2012) is a corpus of Conan Doyle stories
annotated with negation cues and their scopes, as well as the event or property that is
negated. It is composed of 3,640 sentences from The Hound of the Baskervilles story, fuori
of which 850 contain negations, E 783 sentences from The Adventure of Wisteria Lodge
story, out of which 145 contain negations. In questo caso, the three types of negation cues
(lexical, syntactic, and morphological) were taken into account.

The corpus was annotated by two annotators, a master’s student and a researcher,
both with a background in linguistics. The inter-annotator agreement in terms of F1 was
Di 94.88% E 92.77% for negation cues in The Hound of the Baskervilles story and The
Adventure of Wisteria Lodge story, rispettivamente, and of 85.04% E 77.31% for scopes. IL

201

l

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

P

:
/
/

D
io
R
e
C
T
.

M

io
T
.

e
D
tu
/
C
o

l
io
/

l

UN
R
T
io
C
e
–
P
D

F
/

/

/

/

4
6
1
1
1
8
4
7
7
6
9
/
C
o

l
io

_
UN
_
0
0
3
7
1
P
D

.

F

B
sì
G
tu
e
S
T

T

o
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3

Linguistica computazionale

Volume 46, Numero 1

Figura 6
Annotated example from the PropBank Focus (PB-FOC) corpus.

Figura 7
Annotated example from the ConanDoyle-neg corpus.

annotation guidelines13 are based on those of the BioScope corpus, but there are some
differences. The most important differences are that in the ConanDoyle-neg corpus the
cue is not considered to be part of the scope, the scope can be discontinuous, and all the
arguments of the event being negated are considered to be within the scope, including
the subject, which is kept out of the scope in the BioScope corpus.

The ConanDoyle-neg corpus was prepared with the aim of using it at the *SEM 2012
Shared Task14 (Morante and Blanco 2012), which was dedicated to resolving the scope
and focus of negation. It is in CoNLL format (Farkas et al. 2010) and can be downloaded
at http://www.clips.ua.ac.be/sem2012-st-neg/data.html. In Figure 7 it can be seen
how Example (4.6) is represented in the corpus. The content of the columns is as follows:
chapter name (1), sentence number within chapter (2), token number within sentence
(3), token (4), lemma (5), POS tag (6), parse tree information (7). If the sentence has
no negations, column (8) has a “***” value and there are no more columns, but if the
sentence has negations, the annotation for each negation is provided in three columns.
The first column contains the word that belongs to the negation cue, the second the
word that belongs to the scope of the negation cue, and the third the word that is the
negated event or property.

10. After his habit he said nothing, and after mine I asked no questions. (See Figure 7.)

No license is needed to download the corpus.

13 The annotation guidelines are described in Morante, Schrauwen, and Daelemans (2011).
14 www.clips.ua.ac.be/sem2012-st-neg/.

202

l

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

P

:
/
/

D
io
R
e
C
T
.

M

io
T
.

e
D
tu
/
C
o

l
io
/

l

UN
R
T
io
C
e
–
P
D

F
/

/

/

/

4
6
1
1
1
8
4
7
7
6
9
/
C
o

l
io

_
UN
_
0
0
3
7
1
P
D

.

F

B
sì
G
tu
e
S
T

T

o
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3

Jim´enez-Zafra et al.

Corpora Annotated with Negation: An Overview

4.7 SFU ReviewEN

Konstantinova et al. (2012) annotated the SFU ReviewEN corpus (Taboada,
Anthony, and Voll 2006) with information about negation and speculation. This corpus
is composed of 400 reviews extracted from the Web site Epinions.com that belong to 8
different domains: books, cars, computers, cookware, hotels, films, music, and phones.
It was annotated with negation and speculation markers and their scopes. Out of
the total 17,263 sentences, 18% contain negation cues (3,017 sentences). In this corpus
syntactic negation was annotated, but not lexical nor morphological negation.

The annotation process was carried out by two linguists. The entire corpus was
annotated by one of them and 10% of the documents (randomly selected in a stratified
modo) were annotated by the second one in order to measure inter-annotator agreement.
The kappa agreement was a value of 0.927 for negation cues and 0.872 for the scope.
The guidelines of the BioScope corpus were taken into consideration with some modifi-
cations. The min-max strategy of BioScope corpus was used but negation cues were not
included within the scope. A complete description of the annotation guidelines can be
found in Konstantinova, De Sousa, and Sheila (2011).

This corpus is in XML format and publicly available at https://www.sfu.ca/
~mtaboada/SFU_Review_Corpus.html, under the terms of the GNU General Public Li-
cense as published by the Free Software Foundation. Figura 8 shows how Example (4.7)
is annotated in the corpus:

11. I have never liked the much taller instrument panel found in BMWs and Audis.

IO
Avere

never

liked
IL
much
taller
instrument
panel
found
In
BMWs

E

Audis

.

Figura 8
Annotated example from the SFU ReviewEN corpus.

4.8 NEG-DrugDDI

In the biomedical domain, the DrugDDI 2011 corpus (Segura Bedmar, Martinez, E
de Pablo S´anchez 2011) was also tagged with negation cues and their scopes, producing

203

l

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

P

:
/
/

D
io
R
e
C
T
.

M

io
T
.

e
D
tu
/
C
o

l
io
/

l

UN
R
T
io
C
e
–
P
D

F
/

/

/

/

4
6
1
1
1
8
4
7
7
6
9
/
C
o

l
io

_
UN
_
0
0
3
7
1
P
D

.

F

B
sì
G
tu
e
S
T

T

o
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3

Linguistica computazionale

Volume 46, Numero 1

the NEG-DrugDDI corpus (Bokharaeian, D´ıaz Esteban, and Ballesteros Mart´ınez 2013).
It contains 579 documents extracted from the DrugBank database and it is composed of
5,806 sentences, out of which 1,399 sentences (24%) contain negation. Figura 9 expands
Esempio (12), a corpus sentence containing two negations.

12. Repeating the study with 6 healthy male volunteers in the absence of gliben-

clamide did not detect an effect of acitretin on glucose tolerance.

Repeating the study with 6 healthy male
volunteers in the absence Di
glibenclamide did non detect
an effect of acitretin on glucose tolerance.

Figura 9
Annotated example from the NEG-Drug DDI corpus.

This corpus was automatically annotated with a subsequent manual revision. IL
first annotation was performed using a rule-based system (Ballesteros et al. 2012),
which is publicly available and works on biomedical literature following the BioScope
guidelines to annotate sentences with negation. After applying the system, a set of 1,340
sentences were annotated with negation. Then, the outcome was manually checked,
correcting annotations when needed. In order to do so, the annotated corpus was
divided into three different sets that were assigned to three different evaluators. IL
evaluators checked all the sentences contained in each set and corrected the annotation
errors. After this revision, a different evaluator revised all the annotations produced
by the first three evaluators. Prossimo, sentences were explored in order to annotate some
negation cues that were not detected by the system, such as unaffected, unchanged, O
non-significant. Finalmente, 1,399 sentences of the corpus were annotated with the scope of
negation.

The NEG-DrugDDI corpus is in XML format and can be downloaded at http://

nil.fdi.ucm.es/sites/default/files/NegDrugDDI.zip.

4.9 NegDDI-DrugBank

A new corpus, which included the DrugDDI 2011 corpus as well as Medline abstracts,
was developed and it was named the DDI-DrugBank 2013 corpus (Herrero Zazo et al.
2013). This corpus was also annotated with negation markers and their scopes and it is
known as the NegDDI-DrugBank corpus (Bokharaeian et al. 2014). It consists of 6,648
sentences from 730 files and it has 1,448 sentences with at least one negation scope,
which corresponds to 21.78% of the sentences. The same approach as the one used for
the annotation of the NEG-DrugDDI corpus was followed.

204

l

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

P

:
/
/

D
io
R
e
C
T
.

M

io
T
.

e
D
tu
/
C
o

l
io
/

l

UN
R
T
io
C
e
–
P
D

F
/

/

/

/

4
6
1
1
1
8
4
7
7
6
9
/
C
o

l
io

_
UN
_
0
0
3
7
1
P
D

.

F

B
sì
G
tu
e
S
T

T

o
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3

Jim´enez-Zafra et al.

Corpora Annotated with Negation: An Overview

This corpus is in XML format and is freely available at http://nil.fdi.ucm.es/
sites/default/files/NegDDI_DrugBank.zip. In Figure 10, we show the annotations
from Example (13). It can be seen that the annotation scheme is the same as the one
used in the corpus NEG-DrugDDI.

13. Drug-Drug Interactions: The pharmacokinetic and pharmacodynamic interactions
between UROXATRAL and other alpha-blockers have not been determined.

Drug-Drug Interactions: IL
pharmacokinetic and pharmacodynamic interactions
between UROXATRAL and other alpha-blockers have
non been determined.

Figura 10
Annotated example from the NEGDDI-DrugBank corpus.

4.10 Deep Tutor Negation

l

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

P

:
/
/

D
io
R
e
C
T
.

M

io
T
.

e
D
tu
/
C
o

l
io
/

l

UN
R
T
io
C
e
–
P
D

F
/

/

/

/

4
6
1
1
1
8
4
7
7
6
9
/
C
o

The Deep Tutor Negation corpus (DT-Neg) (Banjade and Rus 2016) consists of texts
extracted from tutorial dialogues where students interacted with an Intelligent Tutoring
System to solve conceptual physics problems. It contains annotations about negation
cues, and the scope and focus of negation. From a total of 27,785 student responses,
2,603 responses (9.36%) contain at least one explicit negation marker. In this corpus,
syntactic and lexical negation were taken into account but not morphological negation.
In relation to the annotation process, the corpus was first automatically annotated
based on a list of cue words that the authors compiled from different research reports
(Morante, Schrauwen, and Daelemans 2011; Vincze et al. 2008). After this, annotators
validated the automatically detected negation cues and annotated the corresponding
negation scope and focus. The annotation was carried out by a total of five graduate
students and researchers following an annotation manual that was inspired by the
guidelines of Morante, Schrauwen, and Daelemans (2011). In order to measure inter-
annotator agreement, a subset of 500 instances was randomly selected. It was equally
divided into five subsets and each of them was annotated by two annotators. IL
averaged agreement for scope and focus detection was 89.43% E 94.20%, rispettivamente
(the agreement for negation cue detection was not reported).

This corpus is in TXT format and it is available for research-only, non-commercial,
and internal use at http://deeptutor.memphis.edu/resources.htm. Figura 11 is an
example of an annotated response.

l
io

_
UN
_
0
0
3
7
1
P
D

.

F

B
sì
G
tu
e
S
T

T

o
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3

14. They will not hit the water at the same time. (See Figure 11.)

205

Linguistica computazionale

Volume 46, Numero 1

ID: APR2639A
METAINFO: SpeechAct:
Contribution Corpus: April2013CollegeStudents
AnswerId: 2639 Strand: VM_LV02_PR00.FCI-38.vMHK
QUESTION: If initial velocity and the rate of change in velocity,
which the acceleration, are the same vertically what can you say
about the time it takes for the two girls to travel the same
distance vertically?
ANSWER: They will not hit the water at the same time.
CUE: non
ANNOTATEDANSWER: [They will] <t>>
[hit the water {at the same time}] .
TAG: 0
WATCH: 0

Figura 11
Annotated example from the Deep Tutor Negation corpus.

4.11 SOCC

Finalmente, the last English corpus we are aware of is the SFU Opinion and Comments
Corpus (SOCC) (Kolhatkar et al. 2019) that was presented at the beginning of 2018.
The original corpus contains 10,339 opinion articles (editorials, columns, and op-eds)
together with their 663,173 comments from 303,665 comment threads, from the main
Canadian daily newspaper in English, The Globe and Mail, for a five-year period (from
Gennaio 2012 to December 2016). The corpus is organized into three subcorpora: IL
articles corpus, the comments corpus, and the comment-threads corpus. The corpus
description and download links are publicly available.15

SOCC was recollected to study different aspects of on-line comments such as the
connections between articles and comments; the connections of comments to each other;
the types of topics discussed in comments; the nice (constructive) or mean (toxic) ways
in which commenters respond to each other; and how language is used to convey very
specific types of evaluation. Tuttavia, the main focus of the annotation is oriented
toward the study of the constructiveness and evaluation in the comments. Così, a subset
of SOCC with 1,043 comments was selected to be annotated with three different layers:
constructiveness, appraisal, and negation.

The primary intention of the research and annotation was to examine the relation-
ship between negation, negativity, and appraisal. In the annotation process up to two
individuals participated. Specific guidelines were developed to assist the annotators
throughout the annotation process, and to ensure that annotations were standardized.
These guidelines are publicly available through the GitHub page for the corpus.16 The
1,043 comments were annotated for negation using Webanno (de Castilho et al. 2016)
and the elements to consider were the negation cue or keyword, focus, and scope.
Syntactic negation was taken into account, as well as some verbs and adjectives that
indicate negation. The negation cue is excluded from the scope. In cases of elision or
question and response, a special annotation label, xscope, was created to indicate the
implied content of a non explicit scope. For the 1,043 comments there were 1,397 nega-
tion cues, 1,349 instances of scope, 34 instances of xscope, E 1,480 instances of focus.

15 https://github.com/sfu-discourse-lab/SOCC.
16 https://github.com/sfu-discourse-lab/SOCC/tree/master/guidelines.

206

l

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

P

:
/
/

D
io
R
e
C
T
.

M

io
T
.

e
D
tu
/
C
o

l
io
/

l

UN
R
T
io
C
e
–
P
D

F
/

/

/

/

4
6
1
1
1
8
4
7
7
6
9
/
C
o

l
io

_
UN
_
0
0
3
7
1
P
D

.

F

B
sì
G
tu
e
S
T

T

o
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3

Jim´enez-Zafra et al.

Corpora Annotated with Negation: An Overview

Regarding the agreement, two annotators performed the annotation, a graduate
student in computer science and an expert in computational linguistics. The expert was
in charge of overseeing the process and training the research assistant. The research
assistant annotated the entire corpus. The senior annotator then refined and resolved
any disagreements. To calculate agreement, 50 comments from the beginning of the
annotation process and 50 comments from the conclusion of the annotation process
were compared. Agreement between the annotators was calculated individually based
on the label and the span for the keyword, scope, and focus. Agreement was calculated
using percentage agreement for nominal data, with annotations regarded as either
agreeing or disagreeing. A percentage indicating agreement was measured for both
label and span, then combined to yield an average agreement for the tag. The agreement
for the first 50 comments was 99.0% for keyword, 98.0% for scope, E 85.3% for focus.
For the last 50 comments the agreement was 96.4% for keyword, 94.2% for scope, E
75.8% for focus.

The annotated corpus is in TSV format and it can be downloaded at https://
researchdata.sfu.ca/islandora/object/islandora%3A9109 under a Creative Com-
mons Attribution-NonCommercial-ShareAlike 4.0 International License. Prossimo, we show
an annotated example in Figure 12.

15. Because if nobody is suggesting that then this is just another murder where

someone was at the WRONG PLACE at the WRONG TIME.

2-1 186-193 Because _
2-2 194-196 if _
2-3 197-203 nobody NEG
2-4 204-206 is SCOPE[2]
2-5 207-217 suggesting SCOPE[2]
2-6 218-222 that SCOPE[2]|FOCUS[3]
2-7 223-227 then _
2-8 228-232 this _
2-9 233-235 is _
2-10 236-240 just _
2-11 241-248 another _
2-12 249-255 murder _
2-13 256-261 where _
2-14 262-269 someone _
2-15 270-273 was _
2-16 274-276 at _
2-17 277-280 the _
2-18 281-286 WRONG _
2-19 287-292 PLACE _
2-20 293-295 at _
2-21 296-299 the _
2-22 300-305 WRONG _
2-23 306-310 TIME _
2-24 310-311 . _

Figura 12
Annotated example from the SOCC corpus.

5. Spanish Corpora

In this section we present the Spanish corpora annotated with negation. To the best of
our knowledge, five corpora exist from different domains, although the clinical domain
is the predominant one.

207

l

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

P

:
/
/

D
io
R
e
C
T
.

M

io
T
.

e
D
tu
/
C
o

l
io
/

l

UN
R
T
io
C
e
–
P
D

F
/

/

/

/

4
6
1
1
1
8
4
7
7
6
9
/
C
o

l
io

_
UN
_
0
0
3
7
1
P
D

.

F

B
sì
G
tu
e
S
T

T

o
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3

Linguistica computazionale

Volume 46, Numero 1

5.1 UAM Spanish Treebank

The first Spanish corpus annotated with negation that we are aware of is the UAM
Spanish Treebank (Moreno et al. 2003), which was enriched with the annotation of
negation cues and their scopes (Sandoval and Salazar 2013).

The initial UAM Spanish Treebank consisted of 1,500 sentences extracted from
newspaper articles (El Pa´ıs Digital and Compra Maestra) that were annotated syntacti-
cally. Trees were encoded in a nested structure, including syntactic category, syntactic
and semantic features, and constituent nodes, following the Penn Treebank model.
Later, this version of the corpus was extended with the annotation of negation and
10.67% of the sentences were found to contain negations (160 sentences).

In this corpus, syntactic negation was annotated but not lexical nor morpholog-
ical negation. It was annotated by two experts in corpus linguistics, who followed
similar guidelines to those of the Bioscope corpus (Szarvas et al. 2008; Vincze 2010).
They included negation cues within the scope as in Bioscope and NegDDI-DrugBank
(Bokharaeian et al. 2014). All the arguments of the negated events were also included in
the scope of negation, including the subject (as in ConanDoyle-neg corpus [Morante and
Daelemans 2012]), which was excluded from the scope in active sentences in Bioscope.
There is no information about inter-annotator agreement.

The UAM Spanish Treebank corpus is freely available for research purposes at
http://www.lllf.uam.es/ESP/Treebank.html, but it is necessary to accept the license
agreement for non-commercial use and send it to the authors. It is in XML format,
negation cues are tagged with the label Type=“NEG”, and the scope of negation is tagged
with the label Neg=“YES” in the syntactic constituent on which negation acts. If negation
affects the complete sentence, the label is included as an attribute of the tag
O, by contrast, if negation only affects part of the sentence, Per esempio, an adjectival
syntagma represented as , the label Neg=“YES” is included in the corresponding
tag. In Figure 13, we present an example extracted from the corpus in which negation
affects the complete sentence.

16. No juega a ser un magnate.

He doesn’t play at being a tycoon.

No
juega

UN

ser

un
magnate

Figura 13
Annotated example from the UAM Spanish Treebank corpus.

208

l

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

P

:
/
/

D
io
R
e
C
T
.

M

io
T
.

e
D
tu
/
C
o

l
io
/

l

UN
R
T
io
C
e
–
P
D

F
/

/

/

/

4
6
1
1
1
8
4
7
7
6
9
/
C
o

l
io

_
UN
_
0
0
3
7
1
P
D

.

F

B
sì
G
tu
e
S
T

T

o
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3

Jim´enez-Zafra et al.

Corpora Annotated with Negation: An Overview

5.2 IxaMed-GS

The IxaMed-GS corpus (Oronoz et al. 2015) is composed of 75 real electronic health
records from the outpatient consultations of the Galdakao-Usansolo Hospital in Biscay
(Spain). It was annotated by two experts in pharmacology and pharmacovigilance
with entities related to diseases and drugs, and with the relationships between entities
indicating adverse drug reaction events. They defined their own annotation guidelines,
taking into consideration the issues that should be considered for the design of a corpus
according to Ananiadou and McNaught (2006).

The objective of this corpus was not the annotation of negation but the identification
of entities and events in clinical reports. Tuttavia, negation and speculation were taken
into account in the annotation process. In the corpus, four entity types were annotated:
diseases, allergies, drugs, and procedures. For diseases and allergies, they distinguished
between negated entity, speculated entity, and entity (for non-speculative and non-
negated entities). On the one hand, 2,362 diseases were annotated, out of which 490
(20.75%) were tagged as negated diseases and 40 (1.69%) as speculated diseases. On the
other hand, 404 allergy entities were identified, from which 273 (67.57%) were negated
allergies and 13 (3.22%) speculated allergies. The quality of the annotation process was
assessed by measuring the inter-annotator agreement, which was 90.53% for entities
E 82.86% for events.

The corpus might be possible to acquire via the EXTRECM project17 following a
procedure of some conditions that include a confidentiality agreement, and its format is
not specified.

5.3 SFU ReviewSP-NEG

The SFU ReviewSP-NEG18 (Jim´enez-Zafra et al. 2018B) is the first Spanish corpus that
includes the event in the annotation of negation and that takes into account discontin-
uous negation markers. Inoltre, it is the first corpus in which it is annotated how
negation affects the words that are within its scope—that is, whether there is a change
in the polarity or an increment or reduction of its value. It is an extension of the Spanish
part of the SFU Review corpus (Taboada, Anthony, and Voll 2006) and it could be
considered the counterpart of the SFU Review Corpus with negation and speculation
annotations19 (Konstantinova et al. 2012).

The Spanish SFU Review corpus consists of 400 reviews extracted from the Web site
Ciao.es that belong to 8 different domains: cars, hotels, washing machines, books, cell
phones, music, computers, and movies. For each domain there are 50 positive and 50
negative reviews, defined as positive or negative based on the number of stars given by
the reviewer (1–2 = negative; 4 –5 = positive; 3-star reviews were not included). Later, Esso
was extended to the SFU ReviewSP-NEG corpus in which each review was automatically
annotated at the token level with POS-tags and lemmas using Freeling (Padro and
Stanilovsky 2012), and manually annotated at the sentence level with negation cues and
their corresponding scopes and events. It is composed of 9,455 sentences, out of which
3,022 sentences (31.97%) contain at least one negation marker.

17 http://ixa.si.ehu.eus/extrecm.
18 First Online: 22 May 2017 https://doi.org/10.1007/s10579-017-9391-x.
19 https://www.sfu.ca/~mtaboada/SFU_Review_Corpus.html.

209

l

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

P

:
/
/

D
io
R
e
C
T
.

M

io
T
.

e
D
tu
/
C
o

l
io
/

l

UN
R
T
io
C
e
–
P
D

F
/

/

/

/

4
6
1
1
1
8
4
7
7
6
9
/
C
o

l
io

_
UN
_
0
0
3
7
1
P
D

.

F

B
sì
G
tu
e
S
T

T

o
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3

Linguistica computazionale

Volume 46, Numero 1

In this corpus, syntactic negation was annotated but not lexical nor morphological
negation, as in the UAM Spanish Treebank corpus. Unlike this one, annotations on
the event and on how negation affects the polarity of the words within its scope were
included. It was annotated by two senior researchers with in-depth experience in corpus
annotation who supervised the whole process and two trained annotators who carried
out the annotation task. The kappa coefficient for inter-annotator agreement was 0.97
for negation cues, 0.95 for negated events, E 0.94 for scopes.20 A detailed discussion
of the main sources of disagreements can be found in Jim´enez-Zafra et al. (2016).

The guidelines of the Bioscope corpus were taken into account, but after a thorough
analysis of negation in Spanish, a typology of negation patterns in Spanish (Marti et al.
2016) was defined. As in Bioscope, NegDDI-DrugBank, and UAM Spanish Treebank,
negation markers were included within the scope. Inoltre, the subject was also
included within the scope when the word directly affected by negation is the verb
of the sentence. The event was also included within the scope of negation as in the
ConanDoyle-neg corpus.

The SFU ReviewSP-NEG is in XML format. It is publicly available and can be down-
loaded at http://sinai.ujaen.es/sfu-review-sp-neg-2/ under a Creative Com-
mons Attribution-NonCommercial-ShareAlike 4.0 International License. In Figure 14,
we present an example of a sentence containing negation annotated in this corpus:

17. El 307 es muy bonito, pero no os lo recomiendo.
IL 307 is very nice, but I don’t recommend it.