Reﬂections on the Penn Discourse - IA de Investigación especializada en el MIT

Reﬂections on the Penn Discourse
TreeBank, Comparable Corpora,
and Complementary Annotation

Rashmi Prasad∗
University of Wisconsin–Milwaukee

Bonnie Webber∗∗
University of Edinburgh

Aravind Joshi†
Universidad de Pennsylvania

The Penn Discourse Treebank (PDTB) was released to the public in 2008. It remains the largest
manually annotated corpus of discourse relations to date. Its focus on discourse relations that
are either lexically grounded in explicit discourse connectives or associated with sentential
adjacency has not only facilitated its use in language technology and psycholinguistics but also
has spawned the annotation of comparable corpora in other languages and genres.

Given this situation, this paper has four aims: (1) to provide a comprehensive introduction
to the PDTB for those who are unfamiliar with it; (2) to correct some wrong (or perhaps
inadvertido) assumptions about the PDTB and its annotation that may have weakened previous
results or the performance of decision procedures induced from the data; (3) to explain variations
seen in the annotation of comparable resources in other languages and genres, which should
allow developers of future comparable resources to recognize whether the variations are relevant
to them; y (4) to enumerate and explain relationships between PDTB annotation and comple-
mentary annotation of other linguistic phenomena. The paper draws on work done by ourselves
and others since the corpus was released.

1. Introducción

The Penn Discourse TreeBank, or PDTB (Prasad et al. 2008; PDTB-Group 2008) es el
largest manually annotated resource of discourse relations. This annotation has been
added to the million-word Wall Street Journal portion of the Penn Treebank (PTB) cuerpo

∗ Department of Health Informatics and Administration, University of Wisconsin-Milwaukee, 2025 mi.

Newport Ave (NWQB), Milwaukee WI 53211. Correo electrónico: prasadr@uwm.edu.

∗∗ School of Informatics, University of Edinburgh, 10 Crichton Street (IF4.29), Edinburgh UK EH8 9AB.

Correo electrónico: bonnie.webber@ed.ac.uk.

† Institute for Research in Cognitive Science, Universidad de Pennsylvania, 3401 Walnut Street (Suite 400A),

Philadelphia PA 19104-6228. Correo electrónico: joshi@seas.upenn.edu.

Envío recibido: 17 Junio 2013; revised submission received: 14 Febrero 2014; accepted for publication:
18 Abril 2014.

doi:10.1162/COLI a 00204

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
/
C
oh

yo
i
/

a
r
t
i
C
mi
–
pag
d

F
/

4
0
4
9
2
1
1
8
0
4
6
9
5
/
C
oh

yo
i

_
a
_
0
0
2
0
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Ligüística computacional

Volumen 40, Número 4

(marco, Santorini, and Marcinkiewicz 1993), indicating relations between the events,
hechos, estados, and propositions conveyed in the text—relations that are essential to its
comprensión. Some relations are signalled explicitly, as in Example (1), donde el
underlined phrase as a result signals a causal relation between the situation described in
the ﬁrst two sentences (called Arg1 in the PDTB, formatted here in italics) and a situation
described in the third sentence (called Arg2, formatted here in bold). Other relations
lack an explicit signal, as in Example (2), where there is no explicit signal of the causal
relation between the situation described in the ﬁrst sentence and that described in the
segundo. Sin embargo, there is no change in meaning if the relation is made explicit—for
ejemplo, using the same phrase as a result (Martín 1992).

(1)

(2)

Jewelry displays in department stores were often cluttered and uninspired.
And the merchandise was, Bueno, fake.
Como resultado, marketers of faux gems steadily lost space in department
stores to more fashionable rivals—cosmetics makers. [wsj 0280]

In July, the Environmental Protection Agency imposed a gradual ban on virtually
all uses of asbestos. (implicit=as a result) Por 1997, almost all remaining uses
of cancer-causing asbestos will be outlawed. [wsj 0003]

Over 18K explicitly signalled relations and over 16K implicit forms have been
annotated in the PDTB 2.0 (cf. Sección 3.2, Mesa 1), which was released in February 2008,
through the Linguistic Data Consortium (LDC).1 Researchers since then, in both lan-
guage technology and psycholinguistics, have begun to use the PDTB in their research,
developing methods and tools for automatically annotating discourse relations (Wellner
and Pustejovsky 2007; Elwell and Baldridge 2008; Pitler et al. 2008; Pitler and Nenkova
2009; Wellner 2009; Prasad et al. 2010a, 2011; Zhou et al. 2010; Ghosh et al. 2011a, 2011b,
2012; lin, Ng, and Kan 2012; Ramesh et al. 2012), generating questions (Prasad and
Joshi 2008; agarwal, Shah, and Mannem 2011), ensuring an appropriate realization of
discourse relations in the output of statistical machine translation (Meyer 2011; Meyer
and Popescu-Belis 2012; Meyer and Webber 2013), and testing hypotheses about human
discourse processing (Asr and Demberg 2012a, 2012b, 2013; Jiang 2013; Patterson and
Kehler 2013). Other researchers have adapted the PDTB style of annotation to create
comparable resources in other languages and genres (Sección 4).

What then are the aims of this paper? Primero, for those researchers who are unaware
of the PDTB, Sección 2 of the paper lays out the key ideas behind the PDTB annotation
methodology, y Sección 3 describes the corpus in more detail than previous papers
(Prasad et al. 2008; PDTB-Group 2008) and presents what we have learned since release
of the corpus in 2008. En segundo lugar, for those researchers who have used the PDTB, Sección 3
aims to point out signiﬁcant features of its annotation that have either been ignored
or taken to be intrinsic when they are simply accidental. We hope that this will enable
researchers to derive more from the corpus in the future and recognize the value of
having it more completely annotated. Thirdly, annotation of comparable resources in
other languages and genres has turned out to vary from PDTB annotation in ways that
may be of interest to people contemplating the development of comparable resources
in other languages and genres. Sección 4 summarizes and explains the sources of this
variación. Fourthly, Sección 5 aims to show how PDTB annotation complements Time-
Bank (Pustejovsky et al. 2003a) and PropBank (Palmer, Gildea, and Kingsbury 2005)

1 LDC Catalog ID LDC2008T05. http://www.seas.upenn.edu/~pdtb provides more information on the

PDTB, including a complete list of publications.

922

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
/
C
oh

yo
i
/

a
r
t
i
C
mi
–
pag
d

F
/

4
0
4
9
2
1
1
8
0
4
6
9
5
/
C
oh

yo
i

_
a
_
0
0
2
0
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Prasad, Webber, and Joshi

Reﬂections on the PDTB

annotation over the same Penn TreeBank corpus. Sección 6 closes with a summary of
the key points.

Although extensive documentation can be found on the PDTB Web site, a lo largo de
with discussions of various aspects of PDTB annotation, there has not as yet been as
comprehensive and quantiﬁed a discussion of issues as presented here, especially con-
cerning comparable corpora and complementary annotation. Providing this discussion
is a major goal of this paper. We ourselves will be addressing many of these issues in
the next few years.

2. Key Ideas Underlying PDTB Annotation

Two key ideas underlie the methodology used in annotating the PDTB, setting it apart
from other efforts to annotate discourse relations (p.ej., Carlson, marco, and Okurowski
2001; Polanyi et al. 2004; Baldridge, Asher, and Hunter 2007). Primero, it makes no com-
mitment to any kind of higher-level discourse structure over the discourse relations
annotated between individual text spans. De este modo, while theory-neutral itself with respect to
higher-level discourse structure, the PDTB invites experimentation with approaches to
high-level topic and functional structuring (Stede 2012; Webber, Egg, and Kordoni 2012)
or to hierarchical structuring (Mann y Thompson 1988; Asher and Lascarides 2003),
as a resource for research aimed at a “data-driven and emergent theory of discourse
structure” (Bunt, Prasad, & Joshi 2012, página 61).

En segundo lugar, the annotation of discourse relations is lexically grounded. Rather than
asking annotators to directly classify the sense of relations, which is a difﬁcult task
(Stede 2008), annotators were asked to look at lexical items that can signal discourse
relaciones, such as the expression As a result in Example (1). When they did signal
discourse relations, their arguments and senses were then annotated. Annotators were
also asked to look at adjacent sentences that lacked one of these explicit signals. Where
they inferred a discourse relation, they ﬁrst labeled it with a lexical item that could
serve as its explicit signal (such as As a result in Example (2)), before going on to classify
its sense. In both cases, this lexical grounding was aimed at making the annotation more
confiable, but it can also serve as a feature in the automated identiﬁcation of discourse
relations mentioned in Section 1.

A more detailed introduction to the PDTB can be found in the PDTB-2.0 overview
paper (Prasad et al. 2008) and the PDTB-2.0 annotation manual (PDTB-Group 2008).
Other papers describe speciﬁc aspects of the annotation such as the senses used in
annotating relations (Miltsakaki et al. 2008), alternative lexicalizations (Prasad, Joshi,
and Webber 2010b), and attribution (Prasad et al. 2007).

3. Key Features of PDTB Annotation

Here we discuss four key aspects of PDTB annotation that have been partially ignored
or misunderstood: Explicitly signalled discourse relations; implicit discourse relations;
properties of the arguments to discourse relations; and several issues concerning the
senses of discourse relations and their annotation. These discussions extend the descrip-
tion of these features in the original PDTB overview paper and annotation manual.

3.1 Explicitly Signaled Discourse Relations

As Patterson and Kehler (2013) nota, the inference of discourse relations may draw
heavily upon world knowledge, but may also be facilitated by speciﬁc linguistic signals.

923

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
/
C
oh

yo
i
/

a
r
t
i
C
mi
–
pag
d

F
/

4
0
4
9
2
1
1
8
0
4
6
9
5
/
C
oh

yo
i

_
a
_
0
0
2
0
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Ligüística computacional

Volumen 40, Número 4

It is these signals that we discuss here, distinguishing between (1) the linguistic expres-
sions that can explicitly signal a discourse relation; (2) the resource-limited subset of
these expressions that were annotated as such in the PDTB; y (3) the consequences of
this resource limit on annotation for using the PDTB.

We have taken the view that discourse relations hold between two and only two
(possibly discontinuous) spans of text that can be interpreted as propositions, eventual-
ities, creencias, etc.. (what Asher [1993] has called abstract objects). Tal como, the spans are
primarily one or more sentences or clauses, and the expressions that can signal relations
between them come from four well-deﬁned syntactic classes:

(cid:114)
(cid:114)

(cid:114)

Subordinating conjunctions: porque, a pesar de, cuando, si, como, etc..
Coordinating conjunctions: y, pero, entonces, nor, o (and paired versions of the
latter — neither. . . nor, either. . . o)
Prepositional phrases: como resultado, in comparison, Por un lado. . . sobre el
other hand, etc..
Adverbs: entonces, sin embargo, en cambio, todavía, likewise, subsequently, etc..

These we have called discourse connectives, or explicit connectives.

During the pilot phase of PDTB annotation, we took as explicit signals of discourse
relaciones, linguistic expressions suggested by previous researchers (Halliday and Hasan
1976; Martín 1992; nudo 1996; Forbes-Riley, Webber, and Joshi 2006). This set was then
enlarged as new connectives were found in the WSJ corpus itself. Also identiﬁed during
this phase were productive modiﬁers of explicit connectives such as apparently, al menos
partly, in large part, incluso, solo, etcétera, which were then annotated as connective
modiﬁers.2

What were not taken to be discourse connectives were adverbial cue phrases, incluir-
ing sentence-initial Now (Ejemplo (3)), Well (Ejemplo (4)), So (Ejemplo (5)), and OK
(Ejemplo (6)), because they signal topic changes such as the beginning of a subtopic or
a return to a previous topic (Hirschberg and Litman 1993), rather than relating particular
discourse elements.

(3) Now why, you have to ask yourself, would intelligent beings haul a bunch

of rocks around the universe? [wsj 0550]

(4) Well, mankind can rest easier for now. [wsj 1272]

(5)

So, OK kids, everybody on stage for “Carry On Trading.” [wsj 2402]

(6) When Mr. Jacobson walked into the ofﬁce at 7:30 soy. EDT, he announced:

“OK, buckle up.” [wsj 1171]

We did not intend to annotate as discourse connectives pragmatic markers such
as actually and in fact, which serve to signal the conversational role of the speaker’s
matrix utterance—speciﬁcally, that it is “either aligned with or contrary to something
previously said by another speaker, by the speaker on a previous occasion or to what
people in general say” (Aijmer and Simon-Vandenbergen 2004). But in fact was anno-
tated in the PDTB as a discourse connective, whereas actually was not. Sin embargo,

2 We have not yet seen much use made of these modiﬁers, even though it is clear that they can, para
ejemplo, be used to disambiguate connectives. (E.g., all tokens of modiﬁed ever since convey a
TEMPORAL sense, while only 51% of unmodiﬁed since do. Similarmente, 76% of modiﬁed even though convey
the sense CONCESSION, whereas only 37.3% of unmodiﬁed though do.)

924

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
/
C
oh

yo
i
/

a
r
t
i
C
mi
–
pag
d

F
/

4
0
4
9
2
1
1
8
0
4
6
9
5
/
C
oh

yo
i

_
a
_
0
0
2
0
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Prasad, Webber, and Joshi

Reﬂections on the PDTB

this accidental annotation provides interesting information on what discourse relations
pragmatic markers are associated with, which seems worth further study.

Resources then limited which types of explicit linguistic signals of discourse re-
lations were actually annotated as such. En particular, sentence-initial prepositional
phrases with an overt deictic argument (p.ej., for that reason, by then) were not included in
the set of explicit discourse connectives and hence not systematically annotated because
it was felt this could be put off until deictic coreference was annotated more generally.
The consequences of limiting a priori what were taken as possible signals for a discourse
relation3 meant that adjacent sentences lacking one of these expressions might contain
a different sort of evidence for a discourse relation between them. The consequence for
annotating implicit discourse relations is described in the next section.

3.2 Implicit Relations

The PDTB calls discourse relations that lack an explicit discourse connective between
their arguments implicit discourse relations. Users of the PDTB thus need to under-
stand (1) where and how implicit relations were annotated and (2) what was done in
their absence.

As to the ﬁrst point, the PDTB did not mandate unconstrained search for im-
plicit discourse relations. Bastante, annotators were asked to consider implicit discourse
relations only between adjacent sentences within a paragraph, in the absence of an
explicit connective relating them. The procedure involved (1) identifying one or more
connectives that could be inserted between the two sentences without changing the
discourse relation(s) between them, y luego (2) specifying the sense of those relations.
This had several consequences, each of which is discussed further in this section:

A sentence might bear no relation to its left-adjacent neighbor, incluso
though a wider search might ﬁnd some earlier text to which it was related.

Paragraph-initial sentences were taken to have no left-adjacent neighbor
and were thus not examined as an argument to a discourse relation
unless they contained an explicit discourse connective.

Implicit discourse relations were not annotated within a sentence except
between clauses connected by a semicolon.

There were cases where annotators could not insert a connective between
sentences because to do so appeared redundant.

There were cases where annotators could not insert a connective between
sentences because they did not infer a discourse relation between them.
Bastante, the later sentence simply provided more information about an
entity mentioned in the previous one.

Despite there being common patterns of multiple explicit connectives,
annotators were not asked whether an implicit discourse relation might
hold concurrently with a relation signalled with an explicit connective.

3 “Possible” because many expressions on the list have non-discourse functions as well—e.g., in addition

to functioning as a discourse connective expressing a “result” relation, so can also function as an
intensiﬁer (so short) or as part of the verb phrase anaphor do so. Part of the annotation process involved
excluding tokens that did not function as discourse connectives.

925

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
/
C
oh

yo
i
/

a
r
t
i
C
mi
–
pag
d

F
/

4
0
4
9
2
1
1
8
0
4
6
9
5
/
C
oh

yo
i

_
a
_
0
0
2
0
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Ligüística computacional

Volumen 40, Número 4

Mesa 1
Total relations annotated in the PDTB.

PDTB Relations No. of tokens

Explicit
Implicit
AltLex
EntRel
NoRel

Total

18,459
16,224
624
5,210
254

40,600

Punto (1) has been addressed in the BioDRB (Prasad et al. 2011), which adheres to
most of the PDTB annotation conventions but allows an implicit discourse relation to
hold between non-adjacent sentences within the same paragraph (cf. Sección 4.5). Este
has reduced the proportion of potential implicit relations that were marked NoRel from
1.15% in the PDTB (254/22141, cf. Mesa 1) a 0.9% (29/3223) in the BioDRB (Prasad et al.
2011). The same choice was made in the Hindi DRB (Kolachina et al. 2012).

Points (2) y (3) remain gaps in PDTB annotation that we plan to address in the
future. Ejemplo (7) illustrates point (3): One could insert a connective such as afterwards
or thereafter before the free adjunct (es decir., afterwards returning . . . ), making explicit the
relation of temporal PRECEDENCE between the event expressed in the main clause
and that in the free adjunct. (Arg1 and Arg2 are not indicated in italics and bold in
Ejemplo (7) because free adjuncts have not yet been annotated in the PDTB.)

(7) He ﬂew to Fort Bragg, N.C., in September of that year for a course in
psychological operations, returning to the School of the Americas in
Panama for a two-month course called “military intelligence for
ofﬁcers.” [wsj 2013]

En la sección 5.2, we discuss how PropBank ArgM annotation can be used in addressing
gaps in the annotation of sentence-internal implicit relations in the PDTB.

With respect to point (4), cases where inserting connectives seemed redundant were
taken to arise from the relation being signalled by an expression from outside the set
of explicit connectives. These expressions were annotated as Alternative Lexicalizations
of evidence for discourse relations and their Arg1 and Arg2 annotated accordingly. Nosotros
have counted them under AltLex relations in Table 1 rather than as Implicit discourse
relaciones (also shown there). Por ejemplo, in Example (8), inserting a connective like be-
cause between the sentences was felt to be redundant. Aquí, One reason is was annotated
as an alternative lexicalization of the causal relation between them (indicated in SMALL
CAPITALS adjacent to Arg2).

(8)

Ahora, GM appears to be stepping up the pace of its factory consolidation to get in
shape for the 1990s. (CONTINGENCY.CAUSE.REASON) One reason is mounting
competition from new Japanese car plants in the U.S. that are pouring
out more than one million vehicles a year at costs lower than GM can
match. [wsj 2338]

Some AltLex expressions are the deictic PPs which were not annotated as explicit
discourse connectives due to resource limitations (cf. Sección 3.1). Other expressions

926

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
/
C
oh

yo
i
/

a
r
t
i
C
mi
–
pag
d

F
/

4
0
4
9
2
1
1
8
0
4
6
9
5
/
C
oh

yo
i

_
a
_
0
0
2
0
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Prasad, Webber, and Joshi

Reﬂections on the PDTB

such as quite the contrary, eventually, and thereafter (cerca de 15% of alternative lexical-
izations) meet all the criteria for explicit connectives, even though they had not been
included earlier. Otro 9% of expressions were found to be phrases such as What’s
más (Ejemplo (9)), which suggests that the range of discourse connectives should be
widened to include other syntactic classes.

(9) Marketers themselves are partly to blame: They’ve increased spending for

coupons and other short-term promotions at the expense of image-building
advertising. (EXPANSION.CONJUNCTION) What’s more, a ﬂood of new
products has given consumers a dizzying choice of brands, many of
which are virtually carbon copies of one other. [wsj 1856]

AltLex expressions are under-annotated in the PDTB because they were only an-
notated when annotators found it redundant to insert an implicit connective between
adjacent sentences. Por ejemplo, mientras 15 tokens of that means were noticed and
annotated as AltLex, otro 18 in the corpus were not examined, such as the one
following and in Example (10).

(10)

“I see a lot of evidence indicating a slower economy, and that means my
interest-rate outlook has a downward tilt,” said Garnett L. Keith Jr.
. . . [wsj 1694]

Como resultado, AltLex expressions cannot be exploited in machine learning—for example,
for inducing a model of discourse relation annotation—because no individual AltLex
expression can be guaranteed to be fully annotated in the corpus.

Everyone who has attempted to annotate or catalogue discourse connectives has
commented on the lack of a complete list of words and phrases serving this role (Versley
2010; Rysov´a 2012; Meyer and Webber 2013). Rather than provide annotators with an
incomplete list of connectives and allowing them to identify alternative lexicalizations
during annotation), one might consider giving them complete freedom as to what to
annotate as grounding for discourse relations. Although such a process has its own
problemas (Sección 4.1), expert annotators and good training may make it a plausible
basis for effective discourse annotation.

As for point (5), if annotators were unable to insert a connective between sentences
because they were not able to infer a discourse relation between them, they were asked
to check whether the second sentence provided more information about one or more
entities mentioned in the previous sentence, as in

(11)

Pierre Vinken, 61 years old, will join the board as a nonexecutive director
Nov. 29. Señor. Vinken is chairman of Elsevier N.V., the Dutch publishing
grupo. [wsj 0001]

If it did, annotators were told to annotate the relation type as EntRel. EntRel captures
entity-based coherence between sentences (Knott y otros. 2001) realized either directly
(es decir., via an anaphoric pronoun or NP in the second sentence) or indirectly (es decir., a través de
a bridging inference). Annotators were not asked to annotate their evidence for EntRel.
Although it might be possible to use a resource such as the coreference annotation in
OntoNotes (Weischedel et al. 2012) to recover what the annotators had in mind, in cases
such as Example (12) the entity-based coherence is less obvious.

(12)

This ﬁnancing system was created in the new law in order to keep the bailout
spending from swelling the budget deﬁcit. Otro $20 billion would be raised through Treasury bonds, which pay lower interest rates. [wsj 2200] 927 l D o w n o a d e desde h t t p : / / directo . mi t . e d u / c o l i / lartice – pdf / / / / 4 0 4 9 2 1 1 8 0 4 6 9 5 / c o l i _ a _ 0 0 2 0 4 pd . f por invitado 0 7 septiembre 2 0 2 3 Computational Linguistics Volume 40, Número 4 Such cases would beneﬁt from the entity or entities that ground this relation being an- notated explicitly, as well as provide valuable data for studies of entity-based coherence. Mesa 1 also notes the 254 cases where the annotators did not see either an Alter- native Lexicalization or an Entity-relation between adjacent sentences (within the same paragraph). These they annotated as NoRel. Punto (6) involves the possibility of an explicit connective or AltLex expression occurring concurrently with an implicit connective. Eso es, a recent unpublished pilot study carried out at the University of Edinburgh (Jiang 2013) used MTurk to show that readers presented with text containing a discourse adverbial also infer the sense asso- ciated with a conjunction (coordinating or subordinating), even when no conjunction is present in the text. The study involved 80 items taken from the freely available Corpus of American English, each consisting of a clause, followed by a gap, followed by a clause containing one of four discourse adverbials (después de todo, En realidad, en general, en cambio). Each HIT (Human Intelligence Task) consisted of one item and six conjunctions (y, porque, antes, pero, o, entonces, or none). The Turker was asked to insert into the gap, the conjunction (or none) that seemed most natural between the clauses. Por ejemplo, in Example (13), 50/52 Turkers inserted but into the gap, showing that they interpreted the relation between the clauses as being the same as if what was explicit was but instead: (13) Logically, she should be dead. En cambio, she feels ﬁne, caring for her daughters and walking a pedometer-measured two miles a day. In Example (14), 49/52 Turkers inserted so into the gap, showing that they interpreted the relation as being the same as if what was explicit was so instead: (14) He suspected he shouldn’t say that. Instead he lied. Notice how different Example (13) would be if it were realized with so instead, or Example (14) with but instead: Neither is what the writer intended. The effect was less strong in Example (15), dónde 33/52 Turkers chose because, showing that they interpreted the relation as being the same as because instead, whereas the other Turkers chose differently among the other options: (15) If he’d expected her to be upset, he was disappointed. En cambio, she laughed, clapping her hands. Although it is not yet clear which discourse adverbials are compatible with one or more concurrent implicit relations, it is nevertheless likely that such discourse relations are under-annotated in the PDTB and should be addressed. 3.3 Arguments The two arguments to discourse relations contribute to the senses of the relations taken to hold between them. The PDTB gives annotators two ways to restrict these arguments to only the material needed for these relations. As already noted, it is the events, estados, proposiciones, claims, etc., in a text that participate in discourse relations. In English, such abstract entities tend to be conveyed through sentences, clausulas, nominalizations, and verb phrases. Por eso, these are what can be annotated in the PDTB as arguments.4 Also because discourse deictics (p.ej., este, 4 Because neither and nor or was annotated as a discourse connective when it conjoined VPs, so-conjoined VPs were not annotated as arguments. 928 l D o w n o a d e desde h t t p : / / directo . mi t . e d u / c o l i / lartice – pdf / / / / 4 0 4 9 2 1 1 8 0 4 6 9 5 / c o l i _ a _ 0 0 2 0 4 pd . f por invitado 0 7 septiembre 2 0 2 3 Prasad, Webber, and Joshi Reﬂections on the PDTB that, entonces) can refer back to such interpretations (Ejemplo (16)), as can particles such as yes and no that function as responses to questions (Ejemplo (17)), these can also be annotated as arguments. (16) (17) Evaluations suggest that good ones are—especially so if the effects on participants are counted. [wsj 2412] Underclass youth are a special concern. [Sup1 Are such expenditures worthwhile, entonces]? Sí, if targeted. [wsj 2412] One way to limit arguments to only the minimal text needed for a given discourse relation (a minimality principle), was to allow annotators to specify that other text ap- peared relevant but not necessary to that interpretation. Específicamente, they could annotate as Sup1, material supplementary to Arg1, as in Example (17), where the preceding ques- tion was annotated as relevant to interpreting the question response particle, and as Sup2, material supplementary to Arg2, as in Example (18), where the material enclosed in square brackets was annotated as relevant but not necessary to interpreting the temporal relation expressed with then. (18) It acquired Thomas Edison’s microphone patent and then immediately sued the Bell Co. [Sup2 claiming that the microphone invented by my grandfather, Emile Berliner, which had been sold to Bell for a princely $50,000, infringed
upon Western Union’s Edison patent.] [wsj 0091]

Supplementary information (both Sup1 and Sup2) appears to have been under-
annotated in the PDTB, mainly because annotaters were only invited, and not required,
to check whether any text should be so annotated. This shows when one compares
the number of Sup1 or Sup2 annotations on explicit discourse relations, which were
annotated ﬁrst, with the number of such annotations on implicit discourse relations,
which were annotated on a subsequent pass: 1,571 explicit relations were annotated
with supplementary information, whereas only 126 implicit relations were, a pesar de
nearly equal numbers of both. Before considering the existence of Sup1 or Sup2 as a
feature indicative of the likely use of an explicit connective to signal a discourse relation
(Patterson and Kehler 2013), it must be assessed whether this is an accidental feature of
the PDTB’s annotation or an intrinsic feature of the discourse relations themselves.

A second way of limiting arguments to only the minimal text needed to complete
a given discourse relation involves the separate annotation of attribution (Prasad et al.
2007). This allows the attribution holding between an agent and an abstract object to
be included or excluded from the discourse relation as appropriate. Por ejemplo, en
Ejemplo (19), annotators could exclude the attribution phrase “said Howard Rubel, un
analyst with C.J. lorenzo, Morgan Grenfell Inc. in New York” from Arg1, as unnec-
essary for the discourse relation, while including the attribution phrase “Mr. Asman is
also annoyed” as necessary for the discourse relation in Example (20).5

(19)

Defense contractors “cannot continue to get contracts on that basis,” said
Howard Rubel, an analyst with C.J. lorenzo, Morgan Grenfell Inc.
in New York. (implicit=because) “The pain is too great.” [wsj 0673]

5 The PDTB also annotates attribution relations, capturing their textual signal and semantic features over

each discourse relation and each of its arguments. For a full description of attribution and its annotation,
the reader is referred to Prasad et al. (2007). Attribution is now being annotated as a separate layer over
the WSJ (Pareti 2012), building on the PDTB attribution scheme, but aiming to capture the phenomena
more comprehensively than in the PDTB.

929

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
/
C
oh

yo
i
/

a
r
t
i
C
mi
–
pag
d

F
/

4
0
4
9
2
1
1
8
0
4
6
9
5
/
C
oh

yo
i

_
a
_
0
0
2
0
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Ligüística computacional

Volumen 40, Número 4

(20) Señor. Asman is also annoyed that Mr. Castro has resisted collaboration with U.S.

ofﬁcials, even though by his own account that collaboration has been
devised essentially as a mechanism for acts directly hostile to the
Cuban regime, such as facilitating defections. [wsj 1416]

Attribution differs from supplementary information in that, when its polarity is
negative, it can interact with discourse relations. (Sup has no such interaction.) Este
can be seen by contrasting Example (21), dónde (negative) denying is part of Arg2, y
Ejemplo (22), dónde (negative) denying is not part of Arg1, but is rather its attribution.

(21)

(22)

Estados Unidos. wants the removal of what it perceives as barriers to investment;
(COMPARISON.CONTRAST) Japan denies there are real barriers. [wsj 0082]

Viacom denies it’s using pressure tactics. (EXPANSION.RESTATEMENT.
SPECIFICATION) “We’re willing to negotiate,” says Dennis Gillespie,
executive vice president of marketing. [wsj 0060]

In Example (21), the wanting in Arg1 is taken to contrast with the denying in Arg2. Pero
in Example (22), the negative polarity of denying as the attribution of Arg1 means that
being willing to negotiate is taken to further specify not using pressure tactics.

These techniques are concerned with excluding material unnecessary to concluding
the existence of a particular discourse relation. There is no comparable attempt to
ensure that spans annotated as arguments to discourse relations include all the features
that motivate a given relation (Sección 3.4). This can be seen with discourse relations
associated with the connective instead. Its Arg1 must convey an alternative that does not
hold (Webber 2013): In Example (23), Arg1 conveys that “a price for the new shares has
been set” is an alternative that does not hold.

(23)

No price for the new shares has been set. En cambio, the companies will leave
it up to the marketplace to decide. [wsj 0018]

But the features that allow an argument to convey an alternative that does not hold
may not be present in the argument itself. Por ejemplo, the PDTB annotators agreed
that the clause “to be any silver lining” was Arg1 of instead in Example (24), Residencia en
the minimality principle mentioned at the start of this section. But there is nothing in
this argument that conveys that this alternative does not hold. That would require Arg1
to be “there isn’t likely to be any silver lining.” However, the annotators did not take
such an argument to be minimal.

(24)

In China, sin embargo, there isn’t likely to be any silver lining because the
economy remains guided primarily by the state.
En cambio, China is likely to shell out ever-greater subsidies to its
coddled state-run enterprises, which ate up $18 billion in bailouts last year. [wsj 1646] Although in the majority of cases, minimal argument spans do contain all the features needed to license the annotated sense, this was not required by the PDTB guidelines. This point has been missed in efforts to use the PDTB in training automated sense recognition. 3.4 Senses and their Annotation A well-known feature of the PDTB is its three-level hierarchy of senses (Cifra 1). The express purpose of this hierarchy was to allow back-off to a more general sense if (1) un 930 l D o w n o a d e desde h t t p : / / directo . mi t . e d u / c o l i / lartice – pdf / / / / 4 0 4 9 2 1 1 8 0 4 6 9 5 / c o l i _ a _ 0 0 2 0 4 pd . f por invitado 0 7 septiembre 2 0 2 3 Prasad, Webber, and Joshi Reﬂections on the PDTB l D o w n o a d e d f r o m h t t p : / / directo . mi t . e d u / c o l i / lartice – pdf / / / / 4 0 4 9 2 1 1 8 0 4 6 9 5 / c o l i _ a _ 0 0 2 0 4 pd . f por invitado 0 7 septiembre 2 0 2 3 Cifra 1 PDTB sense hierarchy. individual annotator could not decide among its more speciﬁc senses or if (2) pairs of annotators disagreed as to a more speciﬁc sense. Sin embargo, many researchers interested in inducing automated classiﬁers for explicit and/or implicit discourse relations have used the four top-level (level-1) sense classes for their research (p.ej., Pitler et al. 2008; Pitler and Nenkova 2009; Zhou et al. 2010) because of the relatively large number of tokens in each class at this level of speciﬁcity (Mesa 2). Our goal here is not to defend the hierarchy or its sense inventory, but rather to discuss three issues in sense labeling, which should help readers to better understand both the PDTB and the comparable corpora described in Section 4. The issues are: (1) senses found to be missing from the sense inventory; (2) disagreements between annotators; y (3) annotation of multiple concurrent discourse relations. 3.4.1 Missing Senses. We have long realized that there are gaps in the set of available senses. Some of these gaps are noted in the PDTB Annotation Manual (PDTB-Group 2008), such as the absense of a SIMILARITY sense for labeling explicit relations headed by as if and the absense of a PURPOSE sense for labeling explicit relations headed by 931 Computational Linguistics Volume 40, Número 4 Mesa 2 Total explicit and implicit relations that fall under each level-1 sense. level-1 PDTB Senses No. of explicits No. of implicits CONTINGENCY COMPARISON EXPANSION TEMPORAL 3,741 5,589 6,423 3,696 4,255 2,503 8,861 950 de modo que. Cases of the latter, as well as relations conveying the sense that one argument was the GOAL of the other, were simply annotated with the overloaded label CON- TINGENCY.CAUSE.RESULT. El 34 cases of the subordinating conjunction just as, which can also signal SIMILARITY, were found on subsequent analysis of the corpus to be annotated as either TEMPORAL.SYNCHRONY (13 tokens) or EXPANSION.CONJUNCTION (1 simbólico), or to have been left unannotated (20 tokens). Some of these and other missing senses have been added to the sense inventories used in annotating corpora comparable to the PDTB (Sección 4.2). They should also be added to the PDTB, and senses known to be overloaded should be split appropriately. Although this will eliminate already noted sense gaps, the potential remains for addi- tional senses to be identiﬁed, and hence this remains a problem. 3.4.2 Disagreements Between Annotators. Sense annotation was done by two annotators. Disagreement at level-1 occurred when the two annotators picked senses that belonged at or under different level-1 classes. Disagreement at level-2 occurred when the an- notators picked senses within the same level-1 class but different level-2 classes (p.ej., CONTRAST versus CONCESSION). Similarmente, disagreement at level-3 occurred when the annotators picked different senses under the same level-2 sense class (p.ej., JUXTAPOSI- TION versus OPPOSITION). Adjudication for disagreement at level-1 was done manually, by a team of experts, and disagreements at level-2 or level-3 were handled through automatic back-off to the next higher level. Por ejemplo, a JUXTAPOSITION versus OPPOSITION disagreement would lead to a relation being automatically assigned their level-2 parent, a saber, CONTRAST. Annotation associated with automated back-off has contributed to there being only a level-1 sense annotation for 444 explicit and 257 implicit relations in the PDTB, almost all of which are either COMPARISON or EXPANSION. Although neither label is very informative, one might interpret such a label simply as under-speciﬁed with respect to its more speciﬁc level-2 daughters. 3.4.3 Multiple Concurrent Discourse Relations. Researchers using the PDTB for automated sense labeling of discourse relations have, en general, assumed that its four level-1 senses (Cifra 1) are disjoint. That is incorrect. Particular level-3 senses may be dis- joint because they are deﬁned as each other’s inverse and hence can’t both hold— for example, REASON and RESULT, PRECEDENCE and SUCCESSION, EXPECTATION and CONTRA-EXPECTATION. Other senses may be disjoint because their deﬁning inferences contradict one another: Por ejemplo, REASON requires Arg2 to precede or coincide with Arg1, whereas PRECEDENCE requires that Arg1 precede Arg2. They cannot both hold. Sin embargo, most senses are compatible. 932 l D o w n o a d e desde h t t p : / / directo . mi t . e d u / c o l i / lartice – pdf / / / / 4 0 4 9 2 1 1 8 0 4 6 9 5 / c o l i _ a _ 0 0 2 0 4 pd . f por invitado 0 7 septiembre 2 0 2 3 Prasad, Webber, and Joshi Reﬂections on the PDTB Table 3 Most common (≥10) pairs of level-2 sense labels on the 999 multiply-labeled explicit discourse relations. In pairs above the double line, one of the senses is temporal. count connective senses 50 30 145 30 92 10 66 41 65 12 59 21 138 13 10 11 Contingency.Cause Expansion.Conjunction Contingency.Cause after and as meanwhile Comparison.Contrast meanwhile since when when when when while while Expansion.Conjunction Contingency.Cause Contingency.Cause Contingency.Cause Contingency.Condition Contingency.Condition Comparison.Contrast Expansion.Conjunction Temporal.Asynchronous Temporal.Asynchronous Temporal.Synchrony Temporal.Synchrony Temporal.Synchrony Temporal.Asynchronous Temporal.Asynchronous Temporal.Synchrony Temporal.Synchrony Temporal.Asynchronous Temporal.Synchrony Temporal.Synchrony and but if while Contingency.Cause Expansion.Conjunction Comparison.Concession Contingency.Condition Comparison.Contrast Expansion.Conjunction Comparison.Pragmatic contrast Expansion.List This is evident in the fact that annotators were allowed to assign up to two sense la- bels to each explicit or implicit connective, representing concurrent discourse relations. With explicit connectives, 999 del 18,459 tokens (5.4%) were assigned two concurrent sense labels, with the most common pairs shown in Table 3. Es 5.4% an accurate indicator of the frequency of multiple concurrent discourse rela- tions between two arguments when they are linked by an explicit discourse connective? Evidence for a higher ﬁgure comes from an early experiment with two connectives, since and when (Miltsakaki et al. 2005). Allá, two annotators were given the option of labeling relations linked by one of these connectives in the WSJ corpus as either TEM- PORAL or CAUSAL or TEMPORAL/CAUSAL, to indicate that both senses were conveyed. (The experiment was done on the 184 relations in the corpus headed by since and the ﬁrst 100 relations headed by when, out of a total of 989.6) Those headed by since were annotated TEMPORAL/CAUSAL 21 times by one of the annotators (11.3%) y 16 times by the other (8.6%). Those headed by when were annotated TEMPORAL/CAUSAL even more frequently: 22% by one annotator and 28% by the other. Compare this with the counts for multiply-labeled since and when in Table 3: Solo 10/184 tokens of since (5.4%) were annotated with both a TEMPORAL and a CAUSAL sense, y solo 184/989 tokens of when (18.6%) were annotated with both some kind of TEMPORAL and some kind of CAUSAL sense. In both cases, this is signiﬁcantly less frequent than in the earlier experiment, suggesting that if annotators are not given explicit joint-sense options (such as TEMPORAL/CAUSAL or more speciﬁc pairs) and only invited to use multiple concurrent sense labels if they take multiple discourse relations to hold, their use of multiple labels may be intermittent at best. This is a loss to both language technology and theoretical and psycholinguistic understanding of discourse relations, and a situation that deserves to be ﬁxed. 6 Miltsakaki et al. (2005) reported 186 tokens of since as discourse connectives; PDTB-Group (2008) subsequently reported 184 tokens. Most likely, two were later found not to be connectives. 933 l D o w n o a d e desde h t t p : / / directo . mi t . e d u / c o l i / lartice – pdf / / / / 4 0 4 9 2 1 1 8 0 4 6 9 5 / c o l i _ a _ 0 0 2 0 4 pd . f por invitado 0 7 septiembre 2 0 2 3 Computational Linguistics Volume 40, Número 4 In the case of implicit discourse relations, annotators could assign more than one sense label to a single implicit connective or they could insert more than one implicit connective, which were then individually sense-labeled. Both options indicated that concurrent discourse relations could be taken to hold between the speciﬁed arguments. Del 16,224 implicit relations, 359 (2.2%) were annotated with a single implicit connec- tive with multiple senses, y 171 (1.1%) were annotated with two implicit connectives, each taken to have a single sense. Both of these are very small numbers, so no hard conclusions can be drawn. Sin embargo, one might sample whether more of the implicit relations annotated with some CAUSAL sense might be more accurately annotated with some TEMPORAL sense as well. We close by noting that of the 171 cases annotated with two implicit connectives, with each assigned a single sense, over half (93/171 = 54.4%) involved a connective paired with for example, por ejemplo, or for one thing (p.ej., desde, Por ejemplo; como, por ejemplo; porque, for one thing); 13 más (7.5%) were paired with in particular or speciﬁcally (p.ej., En particular, porque; speciﬁcally, porque) and another 13 (7.5%) were paired with in fact (p.ej., a pesar de, En realidad; entonces, En realidad). All but in fact are really connective modiﬁers (Sección 3.1), even though they can also appear separately as connectives in their right. Such cases deserve further analysis, in connection with getting a better understanding of modiﬁed connectives, their prevalence, and their semantics. 4. Annotated Corpora Comparable to the PDTB We noted in Section 1 that release of the PDTB has spawned similar efforts to annotate resources in other languages and genres following a lexically grounded approach to discourse relations. We also noted that these efforts vary in interesting ways from that of the PDTB. Here we describe both the nature and the sources of this variation, so that people contemplating development of comparable resources in additional languages and/or genres will recognize variation that is appropriate to their situation, while avoiding unnecessary variation that prevents inter-operability of these comparable corpora (Bunt, Prasad, and Joshi 2012). Mesa 4 identiﬁes the corpora we will discuss and the extent of their current annotation: the BioDRB (Prasad et al. 2011), the Leeds Arabic Discourse TreeBank, or LADTB (Al-Saif and Markert 2010, 2011; Al-Saif 2012), the Chinese Discourse TreeBank (Xue 2005; Zhou and Xue 2012; Zhou and Xue (in press)), the Turkish Discourse Bank or TDB (Zeyrek et al. 2008, 2009; Aktas¸, Bozs¸ahin, and Zeyrek 2010; Zeyrek et al. 2010; Demirsahin et al. 2013; Zeyrek et al. 2013), the Hindi Discourse Relation Bank (Oza et al. 2009; Kolachina et al. 2012; Sharma et al. 2013), and the Prague Discourse TreeBank, or PDiT (Mladov´a, Zik´anov´a, and Hajiˇcov´a 2008; J´ınov´a, M´ırovsk ´y, and Pol´akov´a 2012; Rysov´a 2012; Pol´akov´a et al. 2013), now part of the Prague Dependency TreeBank, versión 3.0, PDT 3.0 (Bejˇcek et al. 2013). (A comparable discourse treebank is being developed for French (Danlos et al. 2012), but it has not yet been released and the information needed to compare it to the other corpora in Table 4 is not available.) Although these comparable corpora differ in ways to be discussed subsequently, they all adhere to the key ideas of PDTB annotation (Sección 2) in being neutral to any discourse structure beyond the argument structure of individual discourse relations and in grounding discourse relations in lexical expressions. Where they annotate implicit discourse relations (Mesa 4), these comparable corpora follow the PDTB in annotating an inferred lexical grounding. All of the corpora also follow the PDTB in taking discourse relations to hold be- tween two and only two abstract objects, called Arg1 and Arg2, each associated with 934 l D o w n o a d e desde h t t p : / / directo . mi t . e d u / c o l i / lartice – pdf / / / / 4 0 4 9 2 1 1 8 0 4 6 9 5 / c o l i _ a _ 0 0 2 0 4 pd . f por invitado 0 7 septiembre 2 0 2 3 Prasad, Webber, and Joshi Reﬂections on the PDTB Table 4 Comparison of the PDTB and comparably annotated corpora. Count is the number of annotated relations; Coverage is the text genre(s) in the corpus; Mods=Y if connective modiﬁers are annotated. Impl=Y if implicit connectives are annotated. EntR=Y if Entity Relations are annotated. AltL=Y if Alternative Lexicalizations are annotated. Attr=Y if attribution is annotated. Supp=Y if arguments can have supplementary text. Sens=Y if senses have been annotated. Mult=Y if multiple sense relations can be annotated for a single connective. Name PDTB BioDRB LADTB Chinese DTB Turkish DB Hindi DRB PDT 3.0 (PDiT 1.0) Coverage Count Mods Impl EntR AltL Attr Supp Sens Mult WSJ news, essays Biomed papers 40,600 5,859 Arabic news Xinhua news novels, noticias, etc.. news news 6,328 3,951 8,484 ∼5K 20,542 Y Y Y Y Y Y Y Y Y N1 Y2 N Y N Y N N N N Y Y3 Y Y N Y N Y Y Y N N N N N N Y N N N Y N N Y Y Y Y N Y Y Y Y Y N N Y Y 1∼70% of adjacent sentences in the LADTB are linked by an explicit connective, compared with ∼12% in the PDTB. 2En 20 randomly selected ﬁles, encima 80% of DRels were found to be implicit, compared with around 54.5% in the PDTB (Zhou and Xue 2012). 3Included in coreference annotation. a possibly discontinuous text span. Although not every corpus annotates atttribution, where it is annotated, it is separate from the annotation of discourse relations. l D o w n o a d e desde h t t p : / / directo . mi t . e d u / c o l i / lartice – pdf / / / / 4 0 4 9 2 1 1 8 0 4 6 9 5 / c o l i _ a _ 0 0 2 0 4 pd . f por invitado 0 7 septiembre 2 0 2 3 4.1 Annotation Workﬂow Because one purpose of this section is to inform people considering the development of similar resources in other languages and genres, we will brieﬂy mention how workﬂow has varied in the development of comparable corpora and how it has affected annotator effort and inter-annotator agreement. Workﬂow on the PDTB itself was based on the idea of using easier tasks to pave the way for more difﬁcult ones. En la práctica, this meant separating the annotation of explicit and implicit relations, as explicits were perceived as easier to annotate. Explicit discourse relations were annotated one connective at a time throughout the corpus, before moving on to the next one on the list. The rationale for this was to improve the annotators’ ability to annotate a particular connective by focusing their attention on that connective. Después, implicit discourse relations were annotated document by document, analyzing each pair of adjacent sentences within each paragraph, as described in Section 3.2. Más tarde, senses were annotated for explicit and implicit discourse relations—explicits by connective, and implicits by document. Even for annotating explicit relations, this is not the only possible workﬂow. In annotating the LADTB (Al-Saif 2012; Al-Saif and Markert 2011), the nature of Modern Standard Arabic (MSA) demanded a different workﬂow. In MSA, as in English, words that can function as discourse connectives also have non-discourse functions. Tal como, conﬁrming that a potential connective has a discourse function is directly related to identifying its arguments. One common form of argument to discourse relations in 935 Computational Linguistics Volume 40, Número 4 Arabic news texts is an Al-maSdar noun, which is a tense-less expression of an event.7 Their frequency affected annotation workﬂow in the LADTB. The LADTB used a workﬂow for annotating explicit discourse relations that in- volved highlighting for the annotators all potential discourse connectives (including word-initial clitics), based on a pre-compiled list. An annotator ﬁrst read through the entire text to achieve an overall understanding, before stepping through the highlighted items one by one. In order to tell if a potential connective has a discourse function, the annotator would see whether it had arguments, including strings interpretable as al-maSdar nouns. Workﬂow thus involved simultaneous conﬁrmation of potential discourse connectives and identiﬁcation of their arguments. Después, the annotator would add the one or more senses that a relation expresses. If a potential connective did not express a discourse function, the annotator would note it and go on to the next highlighted item. Workﬂow for the BioDRB (Prasad et al. 2011) was designed to address the difﬁculty perceived in annotating inter-sentential relations in scientiﬁc text. On encountering a new sentence, the annotator had to ﬁrst mark its inter-sentential relation(s) with the prior discourse, and only then annotate any intra-sentential relations within it. In this way, annotators were made to ﬁrst attend to relations that were harder to pin down, as they progressed in their sequential reading and annotation of the text. Ongoing annotation of the Chinese Discourse TreeBank (Zhou and Xue 2012) fol- lows a fully sequential annotation strategy, largely for a language-speciﬁc reason—the customary writing style of Chinese, which often does not bother to distinguish the end of a sentence (marked with a full stop) from the end of a clause (marked with a comma). This has two major consequences: No rigid distinction can be made between inter- and intra-sentential connectives, and annotators must consider implicit relations both between full-stop delimited sentences and comma-delimited clauses. (The latter have not been annotated in the current PDTB.) Annotating implicit relations between comma-delimited clauses results in many more implicit relations. Zhou and Xue (2012) report a 18–82% split in their data between explicit and implicit discourse relations, compared with a 46–54% split in the PDTB. Tal como, having a separate task to cover 18% of the data was disfavored. Although the particular style of annotation should have no effect on the content of annotation, it can affect inter-annotator agreement. Para tal fin, researchers developing the Hindi DRB (Oza et al. 2009; Kolachina et al. 2012) carried out a systematic study of three workﬂow strategies (Sharma et al. 2013). The ﬁrst strategy modeled the task exactly as in PDTB. In the second, explicits and implicits were annotated in exactly the order in which they were encountered on a sequential reading of the text. The third strategy operated per text, with annotators ﬁrst marking all of its explicit connectives, and then its implicit relations, before moving on to another text. The latter two strategies were designed to ensure that annotators were aware of the coherence and ﬂow of the discourse when carrying out the task. Sharma’s ﬁndings show that better agreement is obtained when the annotators’ attention is held to the text, but with no clear preference for a fully sequential approach (as in the second strategy), or an approach that separated the tasks on a text-by-text basis (as in the third). Although ultimately, the choice of workﬂow may be language- or genre-speciﬁc, as noted for Chinese, the ﬁnal choice, entonces, should be driven by considerations of 7 Although the PDTB admits nominalizations as arguments to explicit discourse connectives, they constitute only a small fraction of its arguments. 936 l D o w n o a d e desde h t t p : / / directo . mi t . e d u / c o l i / lartice – pdf / / / / 4 0 4 9 2 1 1 8 0 4 6 9 5 / c o l i _ a _ 0 0 2 0 4 pd . f por invitado 0 7 septiembre 2 0 2 3 Prasad, Webber, and Joshi Reﬂections on the PDTB annotation reliability, which seems to be enhanced by the annotators attending to the coherence and ﬂow of the discourse. Interoperability among these resources is not an issue here, as long as whatever strategy is used yields highly consistent annotation. 4.2 Inventory and Organization of Senses The senses of discourse relations used in the PDTB and the hierarchy in which they are organized (Miltsakaki et al. 2008) drew on both in-house experiments (Miltsakaki et al. 2004, 2005) and previous work on the semantics of discourse relations (Lakoff 1971; Moens and Steedman 1988; Sweetser 1990; Jayez and Rossari 1998; Kehler 2002), among others. Neither has been adopted without some change in the comparable corpora: Additional senses have been introduced, while other senses have been eliminated or modiﬁed; the sense hierarchy has been modiﬁed, and in one case, abandoned. Por ejemplo, Oza et al. (2009) propose a more general and uniform treatment of those dis- course relations that are pragmatic, relating the speech act of one argument to either the content or speech act of the other. Al-Saif and Markert (2011) do the same. Almost every corpus includes at least one additional sense class, including SIMILARITY, PURPOSE, BACKGROUND, and GRADATION, among others, motivated more by the genre of the texts being annotated than by their language. Changes in the PDTB sense hierarchy have been either at its root or its leaves: The BioDRB has eliminated the four top-level classes, adopting a two-level hierarchy. The Prague Discourse TreeBank (Pol´akov´a et al. 2012) now part of the PDT 3.0 and the LADTB (Al-Saif and Markert 2011) have also adopted a two-level hierarchy, but they preserve the top-level classes, collapsing the second and third levels of annotation. The Chinese Discourse TreeBank (Zhou and Xue 2012) has eliminated the hierarchy entirely, using a ﬂat classiﬁcation of just twelve sense categories. Sense annotation has not yet begun on the Turkish Discourse Bank. From the standpoint of interoperability, a shared assumption about the meaning and classiﬁcation of discourse relation senses is of utmost importance, because conﬂicts in the assumed meaning of labels would preclude any kind of comparative studies of the annotated resources, both within and across languages and domains. With the growing number of variations in sense annotation schema, we believe it is critical to collect the insights and ﬁndings from these studies and to ﬁnd common threads, since we believe there is much that is common between them. En efecto, some recent work has usefully provided a mapping between their classiﬁcation schemes and the PDTB classiﬁcation (Prasad et al. 2011; Zhou and Xue 2012). 4.3 Annotation of Explicit Connectives An obvious way in which the corpora vary is in the choice of explicit connectives to be annotated. Because of the rich morphology of Turkish, explicit connectives in Turk- ish include morphological sufﬁxes attached to verb roots and complex subordinators consisting of a connective and a nominalizing sufﬁx. The former have not yet been annotated in the Turkish Discourse Bank (TDB), although the latter have been (Zeyrek et al. 2013). Counterparts of the latter, called phrasal expressions in the TDB, appear as AltLex in the PDTB. As in the LADTB (Al-Saif 2012), nominalizations are commonly annotated as arguments in the TDB. As well as being a morphologically rich language, with preﬁx clitics (as well as separate words and phrases) serving as explicit connectives, Arabic writing tends 937 l D o w n o a d e desde h t t p : / / directo . mi t . e d u / c o l i / lartice – pdf / / / / 4 0 4 9 2 1 1 8 0 4 6 9 5 / c o l i _ a _ 0 0 2 0 4 pd . f por invitado 0 7 septiembre 2 0 2 3 Computational Linguistics Volume 40, Número 4 towards long sentences conjoined with coordinating conjunctions (Ostler 1987), with the equivalent of and commonly found at the beginning of sentences and paragraphs (Al-Saif 2012). It was so common at the beginning of paragraphs in the newswire text annotated in the LADTB that all such tokens were simply assigned a CONJUNCTION relation to the closest proposition, unless a clearer discourse relation was explicitly indicated. 4.4 Lexical Grounding for Implicit Discourse Relations The approach used in lexically grounding implicit discourse relations seems to be language-speciﬁc. Para ingles, the PDTB’s lexically grounded approach led to guide- lines for annotating implicit relations in which annotators were asked to identify one or more connectives that could be inserted between proposed arguments to express the discourse relation(s) they took to hold between them (cf. Sección 3.2). This was meant to serve as explicit evidence for their decisions. For the Chinese Discourse TreeBank, Zhou and Xue (2012) adopt a different strategy, effectively using paraphrase rather than insertion. This is because, in a majority of cases, the wording rejects insertion of a connective even if it expresses the underlying discourse relation exactly (or sometimes, maybe the wording itself is the reason for not having a connective). (Zhou and Xue 2012, página 73) This suggests that Chinese may use particular syntactic constructions to indicate intra- sentential discourse relations even more than English and German do (Meyer and Webber 2013). De este modo, instead of having their annotators insert explicit connectives, Zhou and Xue (2012) have them paraphrase the relation between proposed arguments in terms of explicit connectives that typically express each discourse relation. These prototypical connectives then serve as the lexical grounding for the relation. Although this is the only case we are aware of that has used a different approach to lexically grounding implicit relations, it is something that corpus developers should keep in mind, especially when considering the annotation of discourse relations within sentences. 4.5 Locus of Implicit Relations The corpora differ in where they look for implicit discourse relations. As noted in Section 3.2, implicit relations were only considered in the PDTB between adjacent sentences within the same paragraph. Although a sentence might have an implicit relation to a sentence further aﬁeld, we decided that it would add too much to an already costly effort to have annotators seek them out. With respect to implicit discourse relations within a single sentence, we are aware of having deliberately ignored (for lack of resources) discourse relations that we know are there. Where to look for implicit discourse relations is, en parte, language-speciﬁc. We have already noted (Sección 4.1) that the structure of Chinese sentences is such that a much larger proportion of discourse relations in Chinese occur intra-sententially. Hence the greater need to look for them there. As for looking for discourse relations further aﬁeld, the comparable corpora vary, but not for language-speciﬁc reasons. Bastante, it follows from the cost–coverage decision that all annotation efforts face. En este caso, in both the Hindi Discourse Resource and the 938 l D o w n o a d e desde h t t p : / / directo . mi t . e d u / c o l i / lartice – pdf / / / / 4 0 4 9 2 1 1 8 0 4 6 9 5 / c o l i _ a _ 0 0 2 0 4 pd . f por invitado 0 7 septiembre 2 0 2 3 Prasad, Webber, and Joshi Reﬂections on the PDTB BioDRB, implicit discourse relations have been sought more widely, allowing a sentence to be related to a non-adjacent sentence within the same paragraph. 4.6 Naming Convention for Arg1 and Arg2 One ﬁnal way in which corpora comparable to the PDTB vary is with respect to the naming convention for Arg1 and Arg2. In the PDTB, the choice follows syntactic criteria: With explicit discourse connectives, Arg2 is the argument syntactically bound to the connective, and Arg1 is the other argument. With implicit connectives, Arg1 is the left- adjacent sentence, and Arg2, the right-adjacent one. Although the same convention is followed in the BioDRB and Turkish Discourse Bank, the Chinese Discourse TreeBank, the Prague Discourse TreeBank, and the Hindi Discourse Resource have followed a semantically driven convention, in which argu- ments that play the same semantic role have the same label. This then eliminates level-3 senses in the PDTB sense hierarchy (p.ej., reason/result, expectation/contra-expectation, precedence/succession) whose only purpose is to reﬂect the different linear order of the arguments. De nuevo, while these differences are admissible without impacting the annotation scheme in any major way, comparative studies using these corpora need to be sensitive to these differences. We note, sin embargo, that in an experiment using this strategy for the Hindi annotation, Kolachina et al. (2012) report poor agreement for arguments of relations, and speculate that it was harder for annotators to use the semantic labeling convention. 4.7 Summary Discourse annotation efforts that have followed the PDTB in adopting a lexically grounded (or adjacency-based) approach to annotation nevertheless differ from the PDTB in ways discussed earlier. Still, it appears that none of these differences is so great as to affect their interoperability with the PDTB or each other, or their use in multi- lingual language technology or machine translation (Meyer 2011; Meyer and Popescu- Belis 2012; Meyer and Webber 2013). 5. Complementary Annotations Some of the linguistic phenomena annotated in the PDTB have also been annotated in connection with other levels of linguistic annotation—in particular, the temporal annotation of the Wall Street Journal portion of the Penn Treebank corpus found in TimeBank 1.2 (Pustejovsky et al. 2003a) and the verb-argument annotation found in PropBank (Palmer, Gildea, and Kingsbury 2005). Here we describe how these annota- tions are related. We had both practical and theoretical motivation for carrying out the work described here. From a practical perspective, it might allow for future merging of annotation layers (Pustejovsky et al. 2005), future seeding of one annotation layer with another, and/or future consistency checking based on constraints between annotation at different levels. From a deeper, theoretical perspective, the work has the potential to lower, or even remove, barriers that have long existed between linguistic research at the sentence level and at the discourse level—barriers that have been equally obstructive to research in computational linguistics. This work can thus be seen as a small step towards “the transition from sentence to discourse.” 939 l D o w n o a d e desde h t t p : / / directo . mi t . e d u / c o l i / lartice – pdf / / / / 4 0 4 9 2 1 1 8 0 4 6 9 5 / c o l i _ a _ 0 0 2 0 4 pd . f por invitado 0 7 septiembre 2 0 2 3 Computational Linguistics Volume 40, Número 4 5.1 TimeML and TimeBank The TimeML temporal/event annotation (Pustejovsky et al. 2003b) of texts from the Penn TreeBank forms part of the TimeBank 1.2 cuerpo (Pustejovsky et al. 2003a). TimeML supports the annotation of events, time periods, and temporal relations, expressed either explicitly or implicitly in a text. The information that TimeML makes explicit includes temporal expressions such as the date 10/26/1989 (tagged as TIMEX3), temporal events such as Nigel Lawson resigning as Chancellor of the Exchequer (tagged as EVENT), temporal signals such as after, durante, and in (tagged as SIGNAL), and temporal relations between pairs of temporal expressions or event instances, or between a tem- poral expression and an event instance (tagged as TLINK). The set of temporal relations comes from Allen (1984). When a temporal relation is explicitly indicated by a temporal signal, the signal is included in the TLINK. This enables a clear correspondence with the PDTB. A temporal event is annotated on the head of the syntactic construction that expresses it—the verb, in the case of a clause, as in the annotation of resume and warrant in Example (25), where the SIGNAL until is asserted to signal the temporal relation between resume and warrant. (25) Él dicho construction wouldn’t resume until market conditions warrant él. [wsj 0610] This corresponds to the PDTB annotation: (26) He said construction wouldn’t resume until (TEMPORAL.ASYNCHRONOUS. PRECEDENCE) market conditions warrant it. But TimeML also annotates events expressed as nominalizations (p.ej., construction in Example (25)) and simple nouns (p.ej., tax in Example (27)). (27) And while there was no proﬁt this year from discontinued operations, last year they contributed $34 millón,
antestax. [wsj 0127]

Because temporal events are not limited to clauses, signals of temporal relations are not lim-
ited to clausal coordinators or subordinators or discourse adverbials, but also include
prepositions such as before in Example (27). This is not annotated in the PDTB.

TimeML also allows for the annotation of certain non-temporal relations between
events, including conditional, evidential, non-evidential, and factive relations. These are
tagged SLINK (for Subordination Link). As with temporal relations, when these non-
temporal relations are indicated with a SIGNAL (such as if for a conditional relation),
the SIGNAL is included in the SLINK.

Although many of the same linguistic elements have been annotated in both
the PDTB and TimeBank, the annotation itself can be quite different. The sim-
plest difference relates to the sense of temporal relations: TimeML allows more
speciﬁc relations between events than the PDTB’s three broad relations TEMPO-
RAL.ASYNCHRONOUS.PRECEDENCE (antes), TEMPORAL.ASYNCHRONOUS.SUCCESSION (después),
and TEMPORAL.SYNCHRONY (mismo tiempo). Por ejemplo, TimeML annotators can indicate
that one event is immediately before or immediately after another, although TimeBank
annotators have not used this when annotating relations signalled by before or after.

A more signiﬁcant difference lies in where temporal relations are inferred in the
PDTB and TimeBank. As noted earlier, the PDTB aims to annotate every discourse

940

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
/
C
oh

yo
i
/

a
r
t
i
C
mi
–
pag
d

F
/

4
0
4
9
2
1
1
8
0
4
6
9
5
/
C
oh

yo
i

_
a
_
0
0
2
0
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Prasad, Webber, and Joshi

Reﬂections on the PDTB

relation—including temporal relations—that holds between abstract objects (principalmente
clausal or sentential interpretations) that are signaled by an explicit discourse connec-
tivo (or some alternative lexicalization of a connective) or by the fact of sentence adjacency.
In the latter case, either an implicit discourse relation will be inferred between them, o
a relation involving some entity mentioned in the ﬁrst sentence (EntRel), or no relation
en absoluto (NoRel). En todos los casos, something will be annotated.

A diferencia de, TimeML guidelines specify that if a temporal relation is explicitly sig-
naled in the text, then events and/or time periods speciﬁed in different sentences may
be linked through signals such as previously, earlier, at the same time, entonces, or meanwhile. Si
no temporal relation is explicitly signaled, then temporal elements in different sentences
are not linked, so there are no TLINKS in TimeBank corresponding to the PDTB’s 950
implicit temporal relations between adjacent sentences.

Por otro lado, the TimeML guidelines allow a temporal relation to be in-
ferred from discourse relations that are not primarily temporal. Por ejemplo, alguno
discourse relations annotated in the PDTB as causal (es decir., CONTINGENCY.CAUSE.REASON or
CONTINGENCY.CAUSE.RESULT) are annotated as temporal relations in TimeBank because
both arguments express temporal events and because a cause event starts before its
resultado. This is the case in Example (28), where TimeBank annotates the holding event
as occurring before the adjusting event, which has a negative polarity attribute.8 In the
PDTB, only the explicitly signaled causal relation is annotated.

(28)

Previously, Columbia didn’t have to adjust the book value of its junk-bond holdings
to reﬂect declines in market prices, porque (CONTINGENCY.CAUSE.REASON)
it held the bonds as long-term investments. [wsj 1013]

Sin embargo, not all relations annotated in the PDTB as CONTINGENCY.CAUSE.REASON have
a corresponding temporal annotation in TimeBank: Those corresponding to generic
statements (p.ej., Ejemplo (29)) do not, because generic statements are not taken to
express temporal relations.

(29)

It’s harder to sell stocks when the sell programs come in because (CONTINGENCY.
CAUSE.REASON) some market makers don’t want to {take the orders}.
[wsj 0585]

Finalmente, there is some correspondence between PDTB annotation and the non-
temporal relations that TimeBank annotates as SLINK. What TimeBank annotates as a
conditional SLINK overlaps with the explicit discourse relations annotated in the PDTB as
CONTINGENCY.CONDITION. Other types of SLINK (modal, factive, evidential, and negative
evidential) are related to the properties of Attribution in the PDTB (cf. Sección 3.3; Prasad
et al. 2007).

Enriching TimeBank relations based on annotation in the PDTB, y viceversa,
would require a more detailed study of both the annotation frameworks and annotation
practice in the two corpora. The same would go for using the annotation in one as a
consistency check on the other. Sin embargo, both would potentially be of great value
to delivering more usefully annotated resources.

8 http://www.timeml.org/site/publications/timeMLdocs/annguide_1.2.1.pdf, página 44.

941

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
/
C
oh

yo
i
/

a
r
t
i
C
mi
–
pag
d

F
/

4
0
4
9
2
1
1
8
0
4
6
9
5
/
C
oh

yo
i

_
a
_
0
0
2
0
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Ligüística computacional

Volumen 40, Número 4

a. ( (S

(SBAR-TMP:ARGM-TMP (IN Until)

(NP-SBJ-1 (DT the) (NN building) )
(vicepresidente (VBZ is)

(vicepresidente (VBN completed)

(notario público (-NONE- *-1) )))))

(, ,)
(NP-SBJ:ARG0 (NNP Exxon) )
(vicepresidente (Maryland:ARGM-MOD will)

(vicepresidente (VB rent)
(notario público:ARG1

(notario público (NN part) )
(PÁGINAS (IN of)

(notario público (DT an) (VBG existing) (NN office) (NN tower) )))))

(. .) ))

b. Until (TEMPORAL.ASYNCHRONOUS.PRECEDENCE) the building is completed, Exxon will
rent part of an existing ofﬁce tower. [wsj 0784]

Cifra 2
(a) PropBank annotation of the verb rent; (b) PDTB annotation of the sentence that rent heads.

5.2 PropBank

More interesting is the relation between PDTB annotation of discourse relations and
PropBank annotation (Palmer, Gildea, and Kingsbury 2005) of the sentence-internal ar-
gument structure. PropBank provides, for each verb predicate in the Penn TreeBank, es
sense and the semantic role of each of its arguments. An argument can be either required
by the verb through its valency and assigned an index such as Arg0, Arg1, etcétera,
or accepted as a modiﬁer (annotated with an ArgM label). ArgM arguments are further
assigned functional tags such as MNR (manner), MOD (modal), TMP (temporal), CAU
(causal), DIS (discourse), Etcétera. Por ejemplo, the PropBank annotation of one
instance of the verb rent is shown in Figure 2a. Besides its subject and object (Arg0 and
Arg1), the modal auxiliary will is annotated as ArgM-MOD and the subordinate clause
headed by until is annotated ArgM-TMP.

Many of the ArgM arguments in PropBank are either clauses or nominalizations
that denote events. Many of these align with discourse relations in the PDTB. Para
ejemplo, the ArgM annotation of the subordinate clause in Figure 2a corresponds to
Arg2 of the PDTB annotation of the discourse relation associated with until (Figure 2b).
We can quantify the extent and nature of this correspondence between the two
annotation layers, and in doing so consider two related questions: (1) How many and
to what extent are the intra-sentential relations in PDTB also accounted for by the
dependencies annotated in PropBank, y (2) Are there gaps in the discourse-level
annotation that can be identiﬁed from the PropBank layer? Our analysis here is in
terms of the annotation in PropBank-1.9 Although the scope of this annotation has been
extended within the Ontonotes project (Weischedel et al. 2012), this does not affect our
general points.

5.2.1 Correspondence of PDTB Intra-sentential Relations with PropBank. We ﬁrst assess
whether the intra-sentential relations in the PDTB can be fully accounted for by the verb-
ArgM dependencies in PropBank in terms of quantity, contenido, and consistency. En ese caso,
annotating them again at the discourse level would have involved needless repetition.

9 http://catalog.ldc.upenn.edu/LDC2004T14.

942

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
/
C
oh

yo
i
/

a
r
t
i
C
mi
–
pag
d

F
/

4
0
4
9
2
1
1
8
0
4
6
9
5
/
C
oh

yo
i

_
a
_
0
0
2
0
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Prasad, Webber, and Joshi

Reﬂections on the PDTB

Hay 11,830 intra-sentential relations annotated in the PDTB, accounting for
28% of the discourse relations annotated in the corpus. De estos 11,830 relaciones (todo
which are candidates for overlap with PropBank), 11,236 involve explicit connectives
y 594 do not. The latter primarily hold between independent clauses separated by
a semicolon. As the clauses so-linked are independent, neither being a modiﬁer of the
otro, these relations are not covered in PropBank.

El otro 11,236 intra-sentential relations include relations between clauses linked
by an explicit coordinating conjunction (such as and and but). Like the semicolon-
separated clauses, these are independent and so also outside the scope of PropBank. El
set also includes relations between clauses in the same sentence signaled by a discourse
adverbial. In PropBank, discourse adverbials are generally annotated as discourse-
linking modiﬁers (ARGM-DIS), leaving unspeciﬁed what they link to. Por ejemplo,
in Example (30), the discourse adverbial instead conveys a relation between the two
“throwing” propositions. While PropBank annotates instead as ARGM-DIS of the main
clause throw, it does not explicitly link instead to its other argument.

(30) When the champ has lost his stuff, the great mystery novelist wrote, when he

can no longer throw the high hard one, he throws his heart instead. [wsj 1649]

PropBank-1 did not annotate arguments to copula verbs, so subordinating clauses
attached to these verbs were not covered. Sin embargo, copula verbs have subsequently
been included in extensions covering over 75% of PropBank-1 and released as part of
Ontonotes-5.0, so these subordinating clauses are now marked as arguments.

Besides differing in terms of intra-sentential coverage, PropBank and the PDTB also
differ in their semantics. Específicamente, even those PropBank ArgM roles that are closest
to discourse relations—ArgM-CAU (causal), ArgM-TMP (temporal), ArgM-PNC (pur-
pose), ArgM-MNR (manner), and ArgM-ADV (adverbial)—differ from the semantics of
PDTB senses in several ways.

(cid:114)

Speciﬁcity: ARG-TMP is annotated where the PDTB annotates a more
speciﬁc sense (Synchrony, Precedence, and Succession) of its top-level
class TEMPORAL.

Heterogeneity: Subordinate clauses that are labeled as ArgM-ADV
correspond to the full range of PDTB senses.

Multiplicity: The PDTB allows more than one sense label to be associated
with a single discourse connective to indicate that multiple sense relations
hold concurrently (p.ej., a token of since may be labeled with both a
temporal and causal sense). A diferencia de, PropBank only permits a
constituent to ﬁll a single functional role. Sin embargo, the seven cases
where subordinate clauses are annotated as causal (ArgM-CAU) en
PropBank and some form of TEMPORAL sense in the PDTB reveal
additional cases of the earlier-mentioned under-annotation of multiple
concurrent senses in the PDTB (Sección 3.4.3).

Coverage: The PDTB’s sense inventory currently lacks senses
corresponding to PropBank’s ArgM-MNR and ArgM-PNC roles.
This will be discussed in Section 5.2.2.

In terms of consistency, there are also some mismatches in alignment between
PDTB arguments and PropBank’s semantic role structure, due to the fact that PropBank

943

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
/
C
oh

yo
i
/

a
r
t
i
C
mi
–
pag
d

F
/

4
0
4
9
2
1
1
8
0
4
6
9
5
/
C
oh

yo
i

_
a
_
0
0
2
0
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Ligüística computacional

Volumen 40, Número 4

annotation is tied directly to the syntactic trees in the PTB. Cifra 3(b) shows the
PropBank annotation of the verb say over the PTB parse tree shown in Figure 3(a), con
its initial when-clause parsed as a temporal modiﬁer (ArgM-TMP) of say. A diferencia de,
PDTB annotation has been done over the raw text, with discontinuous spans permitted
as arguments. This allows attribution to be included or excluded from a discourse
relation (Sección 3.3). En este caso, Cifra 3(C) shows attribution is excluded: The tem-
poral relation (SUCCESSION) is annotated between winning and awarding, implying that
Señor. Green’s winning of the verdict was followed by the judge giving him the additional
award. Given the difference in annotation practice, the extent of such mismatches
between PDTB and PropBank is expected to be the same as that between PDTB and
PTB (Dinesh et al. 2005).

En esta sección, we have considered whether the intra-sentential relations in the
PDTB (es decir., those with both arguments in the same sentence) can be fully accounted for
by the verb-ArgM dependencies in PropBank. We have shown that the account is only
partial, due in part to the signiﬁcantly different goals of the two annotation projects
and in part to differences in methodological choices. Tal como, even intra-sententially, a
separate layer of discourse relation annotation is motivated.

(a) ( (S

(SBAR-TMP

(WHADVP-1 (WRB When) )
(S

(NP-SBJ (NNP Mr.) (NNP Green) )
(vicepresidente (VBD won)

(notario público

(notario público (DT a)

(ADJP ($ $) (CD 240,000) (-NONE- *U*) )
(NN verdict) )

(PÁGINAS (IN in)

(notario público

(notario público (DT a) (NN land) (NN condemnation) (NN case) )
(PÁGINAS (IN against)

(notario público (DT the) (NN state) )))))

(ADVP-TMP (-NONE- *T*-1) )
(PP-TMP (IN in)

(notario público (NNP June) (CD 1983) )))))

(, ,)
(NP-SBJ (PRP he) )
(vicepresidente (VBZ says)

(SBAR (-NONE- 0)

(NP-SBJ (NNP Judge) (NNP O’Kicki) )
(vicepresidente

(ADVP (RB unexpectedly) )
(VBD awarded)
(notario público (PRP him) )
(notario público (DT an) (JJ additional) ($ $) (CD 100,000) (-NONE- *U*) )))))

(. .) ))

(b) rel= say; Arg0= he;
Arg1=”Judge O’Kicki unexpectedly awarded him an additional $100,000.”; ArgM-TMP=”When Mr. Green won a $240,000 verdict in a land condemnation case against the
state in June 1983”

(C) Cuando (TEMPORAL.ASYNCHRONOUS.SUCCESSION) Señor. Green won a $240,000 verdict in a land condemnation case against the state in June 1983, he says Judge O’Kicki unexpectedly awarded him an additional $100,000. [wsj 0267]

Cifra 3
Comparison of (a) Penn TreeBank; (b) PropBank; (C) PDTB annotation.

944

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
/
C
oh

yo
i
/

a
r
t
i
C
mi
–
pag
d

F
/

4
0
4
9
2
1
1
8
0
4
6
9
5
/
C
oh

yo
i

_
a
_
0
0
2
0
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Prasad, Webber, and Joshi

Reﬂections on the PDTB

5.2.2 Potential of Seeding New Discourse Relations from PropBank. Próximo, we assess whether
any of the verb-ArgM dependencies in PropBank could potentially correspond to dis-
course relations that are not yet annotated in the PDTB. If the type and number of such
relations is signiﬁcant, then PropBank annotations could be used to seed the PDTB with
new relations, which could then be corrected and/or annotated manually.

Para hacer esto, we aligned the PropBank annotations with the PTB and the PDTB. Nosotros
started by considering only the ﬁve ArgM types mentioned earlier, which gave us a
total of 43,432 verb–ArgM dependencies. For ease of analysis, we ignored the tokens of
split ArgMs (es decir., ArgMs that are not spanned by a single node). We also ignored tokens
from WSJ texts that were not included in the PDTB distribution because of problems
with conversion of the parsed ﬁles to stand off annotation format (PDTB-Group 2008).
From the total of 43,432 ArgMs, we identiﬁed 11,538 ArgMs as clausal, using the
PropBank alignment with the PTB. 4116 of these are free adjuncts (Ejemplo (31)), todo
which are new potential discourse relations to consider for the PDTB (cf. Sección 3.2).
El restante 7,422 clauses either start with a subordinator or subordinating con-
junction, including both ﬁnite (Ejemplo (32)) and non-ﬁnite (Ejemplo (33)) adverbial
clausulas, or are reduced clauses (Ejemplo (34)). For these explicitly subordinated clauses,
PropBank alignment with the PDTB shows that explicit subordinators/subordinating
conjunctions were annotated as connectives in 5,471 del 7,422 ArgMs, leaving the
remaining 1,951 ArgMs as new potential discourse relations for the PDTB. Cada uno de
estos 6,067 new potential relations identiﬁed from the PropBank (i,e., el 4,116 gratis
adjuncts and the 1,951 subordinated clauses) would still have to be reviewed manually
to determine whether it does in fact fulﬁll a discourse function or not.

(31)

(32)

(33)

(34)

They say greedy market manipulators have made a shambles of the
nation’s free-enterprise system [ArgM−ADV turning the stock market
into a big gambling den, with the odds heavily stacked against the
small investor].

Eso $130 millón, Señor. Sherwood said, “gives us some ﬂexibility
[ArgM−CAU in case Temple raises its bid].
Those dividend bulls argue that corporations are in the unusual position
of having plenty of cash left over [ArgM−TMP after paying dividends
and making capital expenditures].

[ArgM−ADV If not for a 59.6% surge in orders for capital goods by defense
contractors], factory orders would have fallen 2.1%.

New potential relations identiﬁed through PropBank would allow for expanding
not just the number of PDTB relations, but also the repertoire of connectives (semejante
as in case in Example (32)) and sense categories, in particular a MANNER relation,
corresponding to ArgM-MNR, and a PURPOSE relation, corresponding to ArgM-PNC,
that are not currently covered in the PDTB. It is important to note here that manner
and purpose arguments annotated in PropBank will only be considered arguments to
discourse relations when they denote events, hechos, estados, or propositions, since these
are what are taken to be arguments to discourse relations in the PDTB.

5.3 Summary

Linguistic annotation invariably involves a considerable amount of time and effort.
When linguistic analysis at multiple levels is encoded on the same source corpus as
different layers of annotation, there is potential value in assessing how the annotation

945

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
/
C
oh

yo
i
/

a
r
t
i
C
mi
–
pag
d

F
/

4
0
4
9
2
1
1
8
0
4
6
9
5
/
C
oh

yo
i

_
a
_
0
0
2
0
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Ligüística computacional

Volumen 40, Número 4

content of the different layers differ from each other, and in exploring how annotations
from one layer can be exploited usefully for annotation in other layers. This section has
compared the annotation content of the PDTB with that of TimeBank and PropBank,
showing that while some of the linguistic phenomena annotated in PDTB have also
been annotated in TimeBank and PropBank, there are signiﬁcant differences in both the
extent and the content of the annotation. This section has also discussed some of the
ways in which annotations from one layer can enrich and/or improve the consistency
of annotation in other layers.

6. Conclusión

Our goals in this paper have been to

(cid:114)

give a thoughtful description of the PDTB that reﬂects what we have
learned since release of the corpus in 2008;

correct some assumptions about the PDTB that show that researchers
may either be ignoring signiﬁcant features of its annotation of discourse
relations or taking accidental properties of its annotation to be intrinsic
properties of the discourse relations themselves;

describe and place in context the ways in which annotation of comparable
resources in other languages and genres has varied from that of the PDTB;
y

provide an analysis of the relation of PDTB annotation to that of TimeBank
and PropBank over the same Penn TreeBank corpus and show how they
son, in large part, complementary.

We hope that the number of researchers able to make use of the PDTB will continue
to grow, as will the number of similarly annotated corpora. We ourselves hope to
be able to enrich the PDTB in the future—widening the scope of discourse relations
that are annotated, improving the recording of evidence for annotation decisions, y
expanding the annotation to include additional textual genres, especially ones that are
less formal than news texts, such as public talks and consumer health advice. El
results should be of further beneﬁt to a growing community of scholars and developers
considering the challenges of extended text.

Expresiones de gratitud
This work was partially supported by NSF
grants EIA–02024417, RI–0705671, y
CNS–1059353. We would like to thank other
members of the team involved in developing
the PDTB: Nikhil Dinesh, Alan Lee, Eleni
Miltsakaki, Livio Robaldo, and Geraud
Campion. We are also grateful to Katja
Markert, Jiˇr´ı M´ırovsk ´y, Dipti Misra Sharma,
Himanshu Sharma, Nianwen Xue, and Deniz
Zeyrek for providing valuable clariﬁcation
and information about their discourse
annotation projects, and to Sameer Pradhan
for clariﬁcation of Ontonotes and PropBank.

We would also like to thank our three
anonymous reviewers for helping us
deliver as clear and informative a paper
as possible.

Referencias
agarwal, METRO., R. Shah, y P. Mannem. 2011.

Automatic question generation using
discourse cues. In Proceedings of the ACL
HLT 2011 Workshop on Innovative Use of
NLP for Building Educational Applications,
pages 1–9, Portland, O.

946

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
/
C
oh

yo
i
/

a
r
t
i
C
mi
–
pag
d

F
/

4
0
4
9
2
1
1
8
0
4
6
9
5
/
C
oh

yo
i

_
a
_
0
0
2
0
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Prasad, Webber, and Joshi

Reﬂections on the PDTB

Aijmer, k. and A.-M. Simon-Vandenbergen.
2004. A model and a methodology for the
study of pragmatic markers. Diario de
Pragmatics, 36:1781–1805.

Aktas¸, B., C. Bozs¸ahin, y D. Zeyrek.

2010. Discourse relation conﬁgurations in
Turkish and an annotation environment.
In Proceedings of the 4th Linguistic
Annotation Workshop, pages 202–206,
Uppsala.

Al-Saif, A. 2012. Human and automatic

annotation of discourse relations for Arabic.
Doctor. tesis, University of Leeds.

Al-Saif, A. and K. Markert. 2010. The Leeds
Arabic Discourse Treebank: Annotating
discourse connectives for Arabic. En
Proceedings of the 7th International
Conference on Language Resources and
Evaluation (LREC-2010), pages 2,046–2,053,
Valletta.

Al-Saif, A. and K. Markert. 2011. Modelling

discourse relations for Arabic. En
Proceedings of the Conference on Empirical
Métodos en el procesamiento del lenguaje natural,
pages 736–747, Edimburgo.

allen, j. 1984. Towards a general theory of
action and time. Artiﬁcial Intelligence,
23:123–154.

Asher, norte. 1993. Reference to Abstract Objects.

Desorden, Dordrecht.

Asher, norte. y un. Lascarides. 2003. Logics of
conversation. Prensa de la Universidad de Cambridge.
Asr, F. t. and V. Demberg. 2012a. Implicitness

of discourse relations. En procedimientos de
COLECCIONAR, pages 2,669–2,684, Mumbai.
Asr, F. t. and V. Demberg. 2012b. Measuring
the strength of linguistic cues for discourse
relaciones. In Proceedings of the Workshop on
Advances in Discourse Analysis and its
Computational Aspects (ADACA),
pages 33–42, Mumbai.

Asr, F. t. and V. Demberg. 2013. Sobre el
information conveyed by discourse
markers. In Proceedings of the 4th Annual
Workshop on Cognitive Modeling and
Ligüística computacional (CMCL),
pages 84–93, Soﬁa.

Baldridge, J., norte. Asher, y j. Cazador. 2007.
Annotation for and robust parsing of
discourse structure on unrestricted
textos. Zeitschrift fur Sprachwissenschaft,
26:213–239.

Bejˇcek, MI., mi. Hajiˇcov´a, j. Hajiˇc, PAG. J´ınov´a,

V. Kettnerov´a, V. Kol´aˇrov´a, METRO. Mikulov´a,
j. M´ırovsk ´y, A. Nedoluzhko, j. Panevov´a,
l. Pol´akov´a, METRO. ˇSevˇc´ıkov´a, j. ˇStˇep´anek,
and ˇS´arka Zik´anov´a. 2013. Prague
Dependency Treebank 3.0 data/
software. Technical report, Univerzita

Karlova v Praze, MFF, FAL, Prague.
http://ufal.mff.cuni.cz/pdt3.0/.
Bunt, h., R. Prasad, y un. Joshi. 2012.
First steps towards an ISO standard
for annotating discourse relations.
In Proceedings of the Joint ISA-7, SRSL-3,
and I2MRT Workshop on Semantic
Annotation and the Integration and
Interoperability of Multimodal Resources
and Tools, pages 60–69, Istanbul.

Carlson, l., D. marco, y M. mi. Okurowski.

2001. Building a discourse-tagged
corpus in the framework of rhetorical
structure theory. En Actas de la
2nd SIGDIAL Workshop on Discourse and
Dialogue, Eurospeech 2001, pages 1–10,
Aalborg.

Danlos, l., D. Antolinos-Basso, C. Braud,
and C. Roze. 2012. Vers le FDTB: Francés
Discourse Tree Bank. En Actas de la
Joint Conference JEP-TALN-RECITAL,
pages 471–479, Grenoble.

Demirsahin, I., A. Ozturel, C. Bozsahin, y
D. Zeyrek. 2013. Applicative structures
and immediate discourse in the Turkish
Discourse Bank. En Actas de la
7th Linguistic Annotation Workshop and
Interoperability with Discourse,
pages 122–130, Soﬁa.

Dinesh, NORTE., A. Sotavento, mi. Miltsakaki, R. Prasad,
A. Joshi, y B. Webber. 2005. Atribución
y el (non)-alignment of syntactic and
discourse arguments of connectives.
In Proceedings of the ACL Workshop on
Frontiers in Corpus Annotation II: Pie in
the Sky, pages 29–36, ann-arbor, MI.
Elwell, R. y j. Baldridge. 2008. Discurso

connective argument identiﬁcation
with connective speciﬁc rankers.
In Proceedings of ICSC-2008, pages 198–205,
Santa Clara, California.

Forbes-Riley, K., B. Webber, y un. Joshi.
2006. Computing discourse semantics:
The predicate-argument semantics of
discourse connectives in D-LTAG. Diario
of Semantics, 23:55–106.

Ghosh, S., R. Johansson, GRAMO. Riccardi, y
S. Tonelli. 2011a. Shallow discourse
parsing with conditional random ﬁelds.
In Proceedings of the International Joint
Conferencia sobre procesamiento del lenguaje natural,
pages 1,071–1,079, Chiang Mai.

Ghosh, S., R. Johansson, GRAMO. Riccardi, y

S. Tonelli. 2012. Improving the recall of a
discourse parser by constraint-based
postprocessing. En Actas del Octavo
International Conference on Language
Resources and Evaluation, pages 2,791–2,794,
Istanbul.

947

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
/
C
oh

yo
i
/

a
r
t
i
C
mi
–
pag
d

F
/

4
0
4
9
2
1
1
8
0
4
6
9
5
/
C
oh

yo
i

_
a
_
0
0
2
0
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Ligüística computacional

Volumen 40, Número 4

Ghosh, S., S. Tonelli, GRAMO. Riccardi, y

R. Johansson. 2011b. End-to-end discourse
parser evaluation. In Proceedings of the Fifth
IEEE International Conference on Semantic
Informática, pages 169–172, Palo Alto, California.

Halliday, METRO. A. k. y r. Hasan. 1976.

Cohesion in English. Longman, Londres.
Hirschberg, j. y D. Litman. 1993. Empirical

studies on the disambiguation of cue
phrases. Ligüística computacional,
19(3):501–530.

Jayez, j. and C. Rossari. 1998. Pragmatic
connectives as predicates: the case of
inferential connectives. En P. St Dizier, ed.,
Predicative forms in natural language and in
lexical knowledge bases, pages 285–319.
Saltador, Dordrecht.

Jiang, X. 2013. Predicting the use and

interpretation of implicit and explicit discourse
conectivos. Doctor. tesis, M.Sc. Thesis,
escuela de psicologia, Philosophy and
Language Sciences (PPLS), Universidad de
Edimburgo.

J´ınov´a, PAG., j. M´ırovsk ´y, y yo. Pol´akov´a.
2012. Semi-automatic annotation of
intra-sentential discourse relations in PDT.
In Proceedings of the Workshop on Advances
in Discourse Analysis and its Computational
Aspects (ADACA), pages 43–58, Mumbai.
Kehler, A. 2002. Coherence, Reference, y el
Theory of Grammar. CSLI Publications,
Palo Alto, California.

nudo, A. 1996. A Data-Driven Methodology for

Motivating a Set of Coherence Relations.
Doctor. tesis, University of Edinburgh.

nudo, A., j. Oberlander, METRO. O'Donnell,

and C. Mellish. 2001. Más allá de la elaboración:
The interaction of relations and
focus in coherent text. In T. Lijadoras,
j. Schilperoord, and W. Pistas,
editores, Representación de texto: Linguistic and
Psycholinguistic Aspects, páginas 181–196.
John Benjamins Publishing.

Kolachina, S., R. Prasad, D. METRO. sharma, y
A. Joshi. 2012. Evaluation of discourse
relation annotation in the Hindi Discourse
Relation Bank. En Actas del Octavo
International Conference on Language
Resources and Evaluation, pages 823–828,
Istanbul.

Lakoff, R. 1971. Ifs, ands and buts about

conjunction. Studies in Linguistic Semantics,
3:114–149.

lin, Z., h. t. Ng, and M.-Y. Kan. 2012.

A PDTB-styled end-to-end discourse
parser. Natural Language Engineering,
20:151–184.

Mann, W.. C. and S. A. Thompson. 1988.
Teoría de la estructura retórica: Toward a

948

functional theory of text organization.
Texto, 8(3):243–281.

marco, METRO. PAG., B. Santorini, y

METRO. A. Marcinkiewicz. 1993. Building a
large annotated corpus of English: El
Penn Treebank. Ligüística computacional,
19(2):313–330.

Martín, j. R. 1992. English Text: System and

Estructura. Benjamín, Ámsterdam.

Meyer, t. 2011. Disambiguating

temporal-contrastive connectives for
machine translation. En Actas de la
LCA 2011 Student Session, pages 46–51,
Portland, O.

Meyer, t. y un. Popescu-Belis. 2012.

Using sense-labeled discourse connectives
for statistical machine translation. En
Proceedings of the Workshop on Hybrid
Approaches to Machine Translation (HyTra),
pages 129–138, Avignon.

Meyer, t. y B. Webber. 2013. Implicitation
of discourse connectives in (machine)
traducción. In Proceedings of the ACL
Workshop on Discourse in Machine
Translation, pages 19–26, Soﬁa.

Miltsakaki, MI., norte. Dinesh, R. Prasad, A. Joshi,
y B. Webber. 2005. Experiments on sense
annotation and sense disambiguation of
discourse connectives. En Actas de la
Fourth Workshop on Treebanks and Linguistic
Theories (TLT), pages 1–12, Barcelona.
Miltsakaki, MI., R. Prasad, A. Joshi, y

B. Webber. 2004. Annotating discourse
connectives and their arguments. En
Proceedings of the Workshop on Frontiers in
Corpus Annotation (Human Language
Technology Conference and the Conference
of the North American Association of
Ligüística computacional), pages 9–16,
Bostón, MAMÁ.

Miltsakaki, MI., l. Robaldo, A. Sotavento, y

A. Joshi. 2008. Sense annotation in the
Penn Discourse Treebank. computacional
Linguistics and Intelligent Text Processing,
Lecture Notes in Computer Science,
4919:275–286.

Mladov´a, l., ˇS´arka Zik´anov´a, and E.
Hajiˇcov´a. 2008. From sentence to
discourse: Building an annotation
scheme for discourse based on Prague
Dependency Treebank. En Actas de la
Sixth International Language Resources and
Evaluation (LREC’08), pages 2,564–2,570,
Marrakech.

Moens, METRO. y M. Steedman. 1988. Temporal

ontology and temporal reference.
Ligüística computacional, 14(2):15–28.

Ostler, S. 1987. Academic and ethnic

background as factors affecting writing

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
/
C
oh

yo
i
/

a
r
t
i
C
mi
–
pag
d

F
/

4
0
4
9
2
1
1
8
0
4
6
9
5
/
C
oh

yo
i

_
a
_
0
0
2
0
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Prasad, Webber, and Joshi

Reﬂections on the PDTB

actuación. In A. Purves, editor, Writing
across Languages and Cultures: Issues in
Contrastive Rhetoric, pages 261–272. Sage.
Oza, Ud., R. Prasad, S. Kolachina, S. Meena,

D. METRO. sharma, y un. Joshi. 2009.
Experiments with annotating discourse
relations in the Hindi Discourse Relation
Bank. In Proceedings of the 7th International
Conferencia sobre procesamiento del lenguaje natural
(ICON), pages 1–10, Hyderabad.

Palmer, METRO., D. Gildea, y P. Kingsbury.

2005. El banco de propuestas: An annotated
corpus of semantic roles. computacional
Lingüística, 31(1):71–106.

Pareti, S. 2012. A database of attribution

relaciones. In Proceedings of the 8th Conference
on International Language Resources and
Evaluation (LREC12), pages 3,213–3,217,
Istanbul.

Patterson, GRAMO. y un. Kehler. 2013. Predicting
the presence of discourse connectives. En
Actas de la 2013 Conferencia sobre
Empirical Methods in Natural Language
Procesando, pages 914–923, seattle, Washington.
PDTB-Group. 2008. The Penn Discourse

TreeBank 2.0 Annotation Manual.
Technical report IRCS-08-01, Institute for
Research in Cognitive Science, Universidad
of Pennsylvania.

Pitler, mi. y un. Nenkova. 2009. Using syntax

to disambiguate explicit discourse
connectives in text. En Actas de la
Joint Conference of the 47th Meeting of the
Association for Computational Linguistics and
the 4th International Joint Conference on
Natural Language Processing, pages 13–16,
Singapur.

Pitler, MI., METRO. Raghupathy, h. Mehta,

A. Nenkova, A. Sotavento, y un. Joshi. 2008.
Easily identiﬁable discourse relations.
In Proceedings of COLING, pages 87–90,
Manchester.

Pol´akov´a, l., PAG. J´ınov´a, ˇS´arka Zik´anov´a,

z. Bedˇrichov´a, j. M´ırovsk ´y, METRO. Rysov´a,
j. Zde ˇnkov´a, V. Pavl´ıkov´a, y
mi. Hajiˇcov´a. 2012. Manual for annotation
of discourse relations in the Prague
Dependency Treebank. Technical report
TR-2012/47, Institute of Formal and
Applied Linguistics, Charles University
in Prague, Prague, Czech Republic.

Pol´akov´a, l., j. M´ırovsk ´y, A. Nedoluzhko,
PAG. J´ınov´a, V. Zik´anov´a, and E. Hajiˇcov´a.
2013. Introducing the Prague Discourse
Treebank 1.0. In Proceedings of the 6th
International Joint Conference on Natural
Procesamiento del lenguaje, pages 91–99, Nagoya.

Polanyi, l., C. Culy, METRO. van den berg,

GRAMO. l. Thione, y D. Ahn. 2004. Sentential

structure and discourse parsing. In ACL
2004 Workshop on Discourse Annotation,
pages 80–87, Barcelona.

Prasad, r., norte. Dinesh, A. Sotavento, A. Joshi,

y B. Webber. 2007. Attribution and its
annotation in the Penn Discourse
TreeBank. Traitement Automatique des
Langues, Special Issue on Computational
Approaches to Document and Discourse,
47(2):43–64.

Prasad, r., norte. Dinesh, A. Sotavento, mi. Miltsakaki,
l. Robaldo, A. Joshi, y B. Webber. 2008.
The Penn Discourse TreeBank 2.0. En
Proceedings of LREC, pages 2,961–2,968,
Marrakesh.

Prasad, R. y un. Joshi. 2008. A

discourse-based approach to generating
why-questions from texts. En procedimientos
of the Workshop on the Question Generation
Shared Task and Evaluation Challenge,
pages 1–3, Arlington, Virginia.

Prasad, r., A. Joshi, y B. Webber. 2010a.
Exploiting scope for shallow discourse
analizando. In Proceedings of the Seventh
International Conference on Language
Resources and their Evaluation,
pages 2,076–2,083, Valletta.

Prasad, r., A. Joshi, y B. Webber. 2010b.
Realization of discourse relations by
other means: Alternative lexicalizations.
In Proceedings of the 23rd International
Congreso sobre Lingüística Computacional,
pages 1,023–1,031, Beijing.

Prasad, r., S. McRoy, norte. Frid, A. Joshi, y
h. Yu. 2011. The Biomedical Discourse
Relation Bank. BMC Bioinformatics,
12(188):1–18.

Pustejovsky, J., PAG. Hanks, R. Sauri,

A. Ver, R. Gaizauskas, A. Setzer, y
D. Radev. 2003a. The Timebank corpus.
In Proceedings of the Corpus Linguistics
Meeting, pages 647–656, Lancaster.
Pustejovsky, J., A. Meyers, METRO. Palmer,

y M. Poesio. 2005. Merging PropBank,
NomBank, TimeBank, Penn Discourse
Treebank and Coreference. En procedimientos
of the Workshop on Frontiers in Corpus
Annotations II: Pie in the Sky, pages 5–12,
ann-arbor, MI.

Pustejovsky, J., j. Casta ˜no, R. Ingria, R. Sauri,

R. Gaizauskas, A. Setzer, y G. katz.
2003b. TimeML: Robust speciﬁcation of
event and temporal expressions in text.
New Directions in Question Answering,
3:28–34.

Ramesh, B., R. Prasad, t. Molinero, B.

Harrington, and H. Yu. 2012. Automatic
discourse connective detection in
biomedical text. Journal of the American

949

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
/
C
oh

yo
i
/

a
r
t
i
C
mi
–
pag
d

F
/

4
0
4
9
2
1
1
8
0
4
6
9
5
/
C
oh

yo
i

_
a
_
0
0
2
0
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Ligüística computacional

Volumen 40, Número 4

Medical Informatics Association,
19(5):800–808.

Rysov´a, METRO. 2012. Alternative lexicalizations

of discourse connectives in Czech.
In Proceedings of the 8th International
Conference on Language Resources and
Evaluation, pages 2,800–2,807, Istanbul.

sharma, h., PAG. Dakwale, D. sharma,

R. Prasad, y un. Joshi. 2013. Evaluación
of different workﬂow strategies for
annotating discourse relations: A case
study with HDRB. In A. Gelbukh, editor,
Computational Linguistics and Intelligent Text
Procesando, LNCS 7816, pages 523–532.
Saltador.

Stede, METRO. 2008. RST revisited: Disentangling
nuclearidad. Cª. Fabricius-Hansen and
W.. Ramm, editores, ‘Subordination’ versus
‘coordination’ in sentence and text–from a
cross-linguistic perspective, pages 33–58.
Juan Benjamín, Ámsterdam.

Stede, METRO. 2012. Discourse Processing. morgan

& Claypool Publishers.

Sweetser, mi. 1990. From Etymology to

Pragmatics: Metaphorical and Cultural
Aspects of Semantics. Cambridge
Prensa universitaria.

Versley, Y. 2010. Discovery of ambiguous and
unambiguous discourse connectives via
annotation projection. En Actas de la
Workshop on the Annotation and Exploitation
of Parallel Corpora (AEPC), pages 83–92,
Tartu.

Webber, B. 2013. What excludes an

alternative in coherence relations? En
Proceedings of the 10th International
Conference on Computational Semantics
(IWCS 2013), pages 276–287, Potsdam.
Webber, B., METRO. Egg, and V. Kordoni. 2012.

Discourse structure and language
tecnología. Natural Language Engineering,
18(4):437–490.

Weischedel, r., METRO. Palmer, METRO. marco,

mi. Azul, S. Pradhan, l. Ramshaw, norte. Xue,
A. taylor, j. Kaufman, METRO. Franchini,
METRO. El-Bachouti, R. Belvin, y un. houston.
2012. Ontonotes release 5.0. Técnico
informe, Linguistic Data Consortium.

Wellner, B. 2009. Sequence Models and

Re-ranking Methods for Discourse Parsing.
Doctor. tesis, Brandeis University,
Bostón, MAMÁ.

Wellner, B. y j. Pustejovsky. 2007.

Automatically identiﬁying the arguments
of discourse connectives. En procedimientos de
EMNLP-CoNLL, pages 92–101.
Xue, norte. 2005. Annotating discourse

connectives in the Chinese Treebank.
In Proceedings of the ACL Workshop on
Frontiers in Corpus Annotation II: Pie in
the Sky, pages 84–91, ann-arbor, MI.

Zeyrek, D., ¨Umit Deniz Turan, y

I. Demirs¸ahin. 2008. Structural and
presuppositional connectives in Turkish.
In A. Benz, PAG. K ¨uhnlein, y M. Stede,
editores, Constraints in Discourse 3,
pages 131–137. University of Potsdam,
Alemania.

Zeyrek, D., ¨Umit Deniz Turan, C. Bozs¸ahin,
R. C¸ akıcı, A. Sevdik-C¸ allı, I. Demirs¸ahin,
B. Aktas¸, ˙Ihsan Yalc¸ınkaya, and H. ¨Ogel.
2009. Annotating subordinators in the
Turkish Discourse Bank. En procedimientos de
the Third Linguistic Annotation Workshop
(LAW III), ACL-IJCNLP-2009, pages 44–48,
Singapur.

Zeyrek, D., I. Demirs¸ahin, A. Sevdik-C¸ allı,
h. ¨Ogel, ˙Ihsan Yalc¸ınkaya, and ¨Umit
Deniz Turan. 2010. The annotation scheme
of the Turkish Discourse Bank and an
evaluation of inconsistent annotations.
In Proceedings of the Fourth Linguistic
Annotation Workshop (LAW-IV), LCA 2010,
pages 282–289, Uppsala.

Zeyrek, D., I. Demirs¸ahin, A. Sevdik-C¸ allı,
y r. C¸ akıcı. 2013. Turkish Discourse
Bank: Porting a discourse annotation
style to a morphologically rich language.
Dialogue and Discourse, 4(2):174–184.

zhou, Y. y N. Xue. (in press). The Chinese
Discourse TreeBank: A Chinese corpus
annotated with discourse relations. Diario
of Language Resources and Evaluation.
zhou, Y. y N. Xue. 2012. PDTB-style
discourse annotation of Chinese text.
In Proceedings of the 50th Annual Meeting
of the ACL, pages 69–77, Jeju Island.

zhou, Z.-M., METRO. Lan, Y. Xu, Z.-Y. Niu, j. Su,
and C. l. Broncearse. 2010. Predicting discourse
connectives for implicit discourse relation
recognition. In Proceedings of the 23rd
International Conference on Computational
Lingüística (COLECCIONAR), pages 1,507–1,514,
Beijing.

950

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
/
C
oh

yo
i
/

a
r
t
i
C
mi
–
pag
d

F
/

4
0
4
9
2
1
1
8
0
4
6
9
5
/
C
oh

yo
i

_
a
_
0
0
2
0
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3 Reﬂections on the Penn Discourse image

Descargar PDF