Transactions of the Association for Computational Linguistics, Bd. 3, S. 475–488, 2015. Action Editor: Diana McCarthy.
Submission batch: 3/2015; Revision batch 6/2015; Published 8/2015.
2015 Verein für Computerlinguistik. Distributed under a CC-BY 4.0 Lizenz.
C
(cid:13)
SemanticProto-RolesDrewReisingerRachelRudingerFrancisFerraroCraigHarmanKyleRawlins∗BenjaminVanDurme∗{rawlins@cogsci,vandurme@cs}.jhu.eduJohnsHopkinsUniversityAbstractWepresentthefirstlarge-scale,corpusbasedverificationofDowty’sseminaltheoryofproto-roles.Ourresultsdemonstrateboththeneedforandthefeasibilityofaproperty-basedannotationschemeofsemanticrelationships,asopposedtothecurrentlydominantnotionofcategoricalroles.1IntroductionFordecadesresearchershavedebatedthenumberandcharacterofthematicrolesrequiredforatheoryofthesyntax/semanticsinterface.AGENTandPA-TIENTarecanonicalexamples,butquestionsemergesuchas:shouldwehaveadistinctroleforBENE-FICIARY?WhataboutRECIPIENT?Whataretheboundariesbetweentheseroles?Andsoon.Dowty(1991),inaseminalarticle,respondedtothisdebatebyconstructingthenotionofaProto-AgentandProto-Patient,basedonentailmentsthatcanbemappedtoquestions,suchas:“Didthear-gumentchangestate?”,or“Didtheargumenthavevolitionalinvolvementintheevent?”.Dowtyarguedthatthesepropertiesgrouptogetherinthelexiconnon-categorically,inawaythatalignswithclas-sicAgent/Patientintuitions.Forinstance,aProto-Patientoftenbothchangesstate(butmightnot),andofteniscausallyaffectedbyanotherparticipant.Variousresourceshavebeendevelopedforcom-putationallinguistsworkingon‘SemanticRoleLa-beling’(SRL),largelyundertheclassical,categor-icalnotionofrole.HerewerevisitDowty’sre-∗Correspondingauthors.searchascomputationallinguistsdesiringdataforanewtask,SemanticProto-RoleLabeling(SPRL),inwhichexistingcoarse-grainedcategoricalrolesarereplacedbyscalarjudgementsofDowty-inspiredproperties.Astheavailabilityofsupportingdataisacriticalcomponentofsuchatask,muchofouref-fortsherearefocusedonshowingthateverydayEn-glishspeakers(untrainedannotators)areabletoan-swerbasicquestionsaboutsemanticrelationships.Inthisworkweconsiderthefollowingquestions:(ich)cancrowdsourcingmethodsbeusedtoempiri-callyvalidatetheformallinguistictheoryofDowty,followingpriorworkinpsycholinguistics(Kako,2006B)?(ii)Howmightexistingsemanticanno-tationeffortsbeusedinsuchapursuit?(iii)CanthepursuitofDowty’ssemanticpropertiesbeturnedintoapracticalandscalableannotationtask?(iv)Dotheresultsofsuchanannotationtask(atvariousscales,includingoververylargecorpora)continuetoconfirmDowty’sproto-rolehypothesis?Andfi-nally,(v)howdotheresultingconfigurationsoffine-grainedrolepropertiescomparetocoarserannotatedrolesinresourcessuchasVerbNet?1Wefirstderiveasetofbasicsemanticques-tionspertainingtoDowty-inspiredproperties.ThesequestionsareusedintwoMechanicalTurkHITsthataddresstheaboveissuess.InthefirstHIT,webuildonpsycholinguisticwork(Kako,2006B)todirectlyaccess‘type-level’intuitionsaboutalexicalitem,byaskingsubjectsproperty-questionsusingmade-up(“nonce”)wordsinargumentpositions.Ourresults1Tobeclear,Dowtyhimselfdoesnotmakedirectpredic-tionsaboutthedistributionofproto-rolepropertieswithinacor-pus,exceptinsofarasacorpusisrepresentativeofthelexicon.
l
D
Ö
w
N
Ö
A
D
e
D
F
R
Ö
M
H
T
T
P
:
/
/
D
ich
R
e
C
T
.
M
ich
T
.
e
D
u
/
T
A
C
l
/
l
A
R
T
ich
C
e
–
P
D
F
/
D
Ö
ich
/
.
1
0
1
1
6
2
/
T
l
A
C
_
A
_
0
0
1
5
2
1
5
6
6
8
1
0
/
/
T
l
A
C
_
A
_
0
0
1
5
2
P
D
.
F
B
j
G
u
e
S
T
T
Ö
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3
476
replicatethesepreviousexperiments,anddemon-stratethatwhatcanbedoneinthisdomaininacontrolledlabexperimentcanbedoneviacrowd-sourcing.Weextendthistoalarge-scaleMTurkan-notationtaskusingcorpusdata.Thistaskpresentsanannotatorwithaparticular(‘token-level’)sen-tencefromPropBank(Palmeretal.,2005)andahighlightedargument,andasksthemforalikelihoodjudgmentaboutaproperty;forexample,“HowlikelyorunlikelyisitthatARGissentient?”.Bylookingacrossmanytoken-levelinstancesofaverb,wecantheninfertype-levelinformationabouttheverb.Wediscussresultsfromthistaskover11rolepropertiesannotatedbyasingle(trusted)annota-toronapproximately5000verbtokens.Ourresultsrepresentthefirstlarge-scalecorpusstudyexplicitlyaimedatconfirmingDowty’sproto-rolehypothesis:Proto-Agentpropertiespredictthemappingofse-manticargumentstosubjectandobject.Weshowthatthisallowsustobothcaptureanddiscoverfine-graineddetailsofsemanticrolesthatcoarseranno-tationschemessuchasVerbNetdonot:empirically,thisdatasetshowsagreatdegreeofrolefragmen-tation,muchgreaterthananyexistingannotationschemeallows.Theresultsofthistaskrepresentanewlarge-scaleannotatedresource,involvingcloseto345hoursofhumaneffort.22Background2.1RolesinlinguisticsThematicroleshavebeenakeyanalyticalcompo-nentinmodernlinguistictheory.3Despitethevastliterature,thereissurprisinglylittleconsensusoverwhatathematicroleis,orhowtoidentifyorpre-ciselycharacterizethem.A‘textbook’approach,in-fluentialinlinguisticsandcomputerscience,isthatthereisa(short)listofcoreGeneralizedThematicRoles,suchasAGENT,PATIENT,EXPERIENCER,2AvailablethroughtheJHUDecompositionalSemanticsIni-tiative(Decomp):http://decomp.net.3Afullaccountingofthehistoryofthematicrolesisbe-yondthescopeavailablehere(Blake,1930;Gruber,1965;Fill-more,1966;1976;1982;Casta˜neda,1967;Jackendoff,1972;1987;Cruse,1973;Talmy,1978;Chomsky,1981;Carlson,1984;CarlsonandTanenhaus,1988;RappaportandLevin,1988;RappaportHovavandLevin,1998;LevinandRappaportHovav,2005;Dowty,1989;1991;Parsons,1990;Croft,1991;DavisandKoenig,2000,amongothers).etc.thatverbsassigntoarguments.However,ithasbeenknownforsometimethatthisviewisprob-lematic(seeLevinandRappaportHovav(2005)foranoverview).PerhapsthebestknownargumentsemergefromtheworkofDavidDowty.Proto-rolesDowty(1991),inanexhaustivesur-veyofresearchonthematicrolesuptothatpoint,identifiesanumberofproblemswithgeneralizedthematicroles.Firstandforemost,iftheinven-toryofroletypesissmall,thenitprovesimpos-sibletoclearlydelineatetheboundariesbetweenroletypes.Thissituationpushesresearcherswhowantcleanroleboundariestowardsaverylargein-ventoryofspecialized,fine-grainedthematicroles–whatDowtytermedrolefragmentation.Alarge,fragmentedsetofrole-typesmaybeusefulformanypurposes,butnotforexpressinggeneraliza-tionsthatshouldbestatedintermsofthematicroles.Dowty(1991)focusesongeneralizationsrelatedtothemappingproblem:howaresyntacticargumentsmappedtosemanticarguments?Themappingprob-lemisnotjustalinguisticpuzzle,butacentralprob-lemfortaskssuchasSRL,semanticparsing,etc.Dowtyoffersasolutiontothemappingproblemcouchednotintermsoffine-grainedfragmentedthematicroles,butintermsofwhatDowtyanalo-gizesto‘prototype’conceptsconstructedoverfine-grainedroleproperties.Inparticular,therole-propertiesarefeaturessuchaswhetherthepartic-ipantinquestioncausestheeventtohappen,orwhethertheparticipantchangesstate.Dowtygroupspropertiesintotwoclasses:Proto-Agentproperties,andProto-Patientproperties.AsemanticargumentismoreAGENT-likethemoreProto-Agentproper-tiesithas,andmorePATIENT-likethemoreProto-Patientpropertiesithas.Thesetwosetsofpropertiesareincompetition,andanargumentcanhavesomeofeach,orevennoneoftheproperties.Dowty’sroleproperties(slightlymodified)areshowninTable1;weusetheseasastartingpointforourownchoiceoffine-grainedfeaturesin§3.4Classicroletypesfalloutfromwhatwewill4Dowty’sArgumentSelectionPrinciple:“Inpredicateswithgrammaticalsubjectandobject,theargumentforwhichthepredicateentailsthegreatestnumberofProto-Agentpropertieswillbelexicalizedasthesubjectofthepredicate;theargumenthavingthegreatestnumberofProto-Patiententailmentswillbelexicalizedasthedirectobject.”(Dowty1991:31)
l
D
Ö
w
N
Ö
A
D
e
D
F
R
Ö
M
H
T
T
P
:
/
/
D
ich
R
e
C
T
.
M
ich
T
.
e
D
u
/
T
A
C
l
/
l
A
R
T
ich
C
e
–
P
D
F
/
D
Ö
ich
/
.
1
0
1
1
6
2
/
T
l
A
C
_
A
_
0
0
1
5
2
1
5
6
6
8
1
0
/
/
T
l
A
C
_
A
_
0
0
1
5
2
P
D
.
F
B
j
G
u
e
S
T
T
Ö
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3
477
Proto-AgentpropertiesProto-Patientpropertiesa.volitionalinvolvementf.changesstateb.sentience(/perception)g.incrementalthemec.causeschangeofstateh.causallyaffectedd.movement(relative)i.stationary(relative)e.independentexistencej.noindep.existenceTable1:Proto-roleproperties(Dowty1991:27–28).termconfigurationsoftheseproperties.A‘core’AGENT,forexample,wouldhavealloftheProto-Agentproperties.AnEXPERIENCERwouldhaveProto-Agentproperties(B)Und(e),andProto-Patientproperty(H),andsowouldbelessAGENT-likethanacoreAGENT.ThisideaisfurtherdevelopedbyGrimm(2005;2011),whopointsoutthatwhencombinationsofproto-rolepropertiesarelookedatasalatticestructure,generalizedthematicrolescanbeidentifiedwithparticularpartsofthelattice.IfDowty’sproposalisright,thelexiconwillinstanti-ateaverylargenumberofpropertyconfigurations,ratherthanasmallandconstrainedset.AkeyresultofthistheoryisexplanationofthecontrastbetweenwhatDowtytermsstableandun-stablepredicates.Astablepredicateisonelikekillwhosemappingbehaviorissimilaracrosslanguages–theKILLERismappedtosubject,andtheVIC-TIMtoobject.Anunstablepredicateisonewherethisisnotso.Instabilitycanalsomanifestwithinalanguage,intheformoflexicaldoubletssuchasbuyandsell.TheProto-Patientargumentfortheseverbsisstable,butthesubjectalternates:forbuyitistheGOALargumentthatappearsassubjectwhileforsellitistheSOURCE.Dowty’sexplanationisthatfortransactionevents,SOURCEandGOALareverysimilarintheirProto-Agentproperties,andsocompeteequallyforsubjectposition.Dowty’slinguisticproposal,ifcorrect,hassub-stantialimplicationsforhumanlanguagetechnol-ogy(seealsodiscussioninPalmeretal.(2005)).Itsuggestsanapproachtosemanticannotation,se-manticparsing,andrelatedtasksthatfocusesonthisfine-grainedlevelofproto-roleproperties,withanymoregeneralizedthematicrolesasemergentpropertyconfigurations.Iflexicalargumentstruc-tureisorganizedaroundproto-roles,thenwepredictthatwewillfindthisorganizationreflectedincor-pora,andthattoken-levelannotationsofverbmean-ingswouldbenefitfromobservingthisorganiza-Proto−AgentProto−Patientstationarychangedcreatedmovedexistedchosecaused docaused changeaware−6−3036Mean difference (subject − object)Figure1:Proto-rolepropertiesinKako2006exp.1(re-productionofKako’sFig.1).Errorbarsinallfiguresshow95%t-testCIs.tion.Inparticular,anannotationstrategythattakestheproto-rolehypothesisseriouslywouldannotateverbsforpropertiessuchthoseshowninTable1.ExperimentalworkCantheproto-rolehypothe-sisbeoperationalized?Astartingpointisexperi-mentalworkbyKako(2006A,B),whotooktheproto-rolehypothesisintothelab.Kakodevelopedseveralexperimentalversionsofthehypothesis,wherebyparticipantswereaskedsimplifiedquestion-basedversionsofDowty’sproto-rolepropertiesaboutsen-tencesofEnglish.KakodidnotuseactualorattestedsentencesofEnglish,butratherfocusedon‘nonce’-basedtasks.Thatis,heconstructedstimulibytakingconstructedsentencesofEnglishcontainingthetar-getverbs,andreplacingnounpositionswithnoncewordslikedax.Subjectswerethenpresentedwiththesenoncesentencesandaskedquestionssuchas,“Howlikelyisitthatthedaxmoved?”.Thenonce-methodisdesignedtoaccess‘type-level’judgmentsaboutverbsacrossframes.Acrossallexperiments,Kakoconfirmsaversionoftheproto-rolehypothesis:subjectargumentsacrosstheverbsheexamineshavesignificantlymoreProto-AgentthanProto-Patientproperties,andviceversaforobjects.Fine-grainedresultsforindividualproto-rolepropertiesfromoneofhisexperimentsareshowninFigure1:thispresentsanaggregatemeasureofthesuccessoftheproto-rolehypothesis,showingthemeandifferencebetweenpropertyrat-ingsforsubjectvs.objectarguments.Dowty’smap-pinghypothesispredictsthatsubjectsshouldskewtowardsProto-Agentproperties,andobjectstowards
l
D
Ö
w
N
Ö
A
D
e
D
F
R
Ö
M
H
T
T
P
:
/
/
D
ich
R
e
C
T
.
M
ich
T
.
e
D
u
/
T
A
C
l
/
l
A
R
T
ich
C
e
–
P
D
F
/
D
Ö
ich
/
.
1
0
1
1
6
2
/
T
l
A
C
_
A
_
0
0
1
5
2
1
5
6
6
8
1
0
/
/
T
l
A
C
_
A
_
0
0
1
5
2
P
D
.
F
B
j
G
u
e
S
T
T
Ö
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3
478
Proto-Patientproperties,exactlyKako’sfinding.Kako’sworkhelpleadAlishahiandStevenson(2010)toannotateasmallcollectionofchilddi-rectedspeechwithDowty-inspiredproperties,usedtoevaluateaBayesianmodelforinducingwhattheytermedsemanticprofiles.52.2RolesincomputationallinguisticsPropBank6PropBank(Palmeretal.,2005)lay-erspredicate/argumentannotationsontheEnglishportionofthePennTreebank(PTB)(Marcusetal.,1993),treatingsemanticroleannotationasasortofslot-fillingexercise:aframesetdefinesasetofse-manticrolesthataparticulartypeofpredicatemayuse.Everyverbisassignedaframeset(grob,averbsense),andargumentsoftheverb(potentiallyanon-contiguousspan)arelabeledwithaparticu-larrole.Coarsecategoricallabels,suchasARG0andARG1,allowPropBanktobothcapturesomeofLevin(1993)’ssyntacticvariations,andimbuethissyntacticinformationwithshallowsemantics.An-notationsdonotcrosssentenceboundaries.AseveryverbinthePTBwasannotated,Prop-Bankhasgoodcoverage:4,500framesetscoveraround3,300verbtypes.AdditionalresourceshaveadoptedandextendedPropBank,z.B.(Weischedeletal.,2013,etc.),andtherehavebeenmultiplesharedtaskscenteredaroundPropBank-styleSRL(CarrerasandM`arquez,2005).Jedoch,atthreedays(Palmeretal.,2005),thetrainingtimeforanannotatorissignificantlyhigherthanthecrowd-sourcingsolutionwepursuehere.VerbNetandSemLinkVerbNet(Schuler,2005)providesaclass-basedviewofverbs.ItappliesLevin’sverbclasses(Levin,1993)tomorethanfivethousand(English)verbs,categorizingthemaccord-5Probabilitydistributionsoverobservedconfigurationsthatcaptureageneralizednotionofsemantic(proto-)role.6Thissectionisnotafullyexhaustivelistofresources,andweomitdiscussionofseveralimportantonesthatarecomple-mentarytoourefforts.Forexample,resourcessuchasthePatternDictionaryofEnglishVerbs(Hanks,2013),currentlyinprogress,couldbesupplementedbyourSPRLannotations.(ThePDEVwillcontainvalencypatternsforthousandsofverbsalongwithrestrictionsonthesemantictypesoftheirargumentsbasedon(Pustejovskyetal.,2004)’sontology.)Alsoimportantisearlyconnectionistwork,whichproposed“semanticmicro-features”tomodelsemanticrolegeneralizations;seee.g.Hin-ton(1981;1986)andMcClellandandKawamoto(1986).ingtotheirsyntacticbehaviors.Beyondthisgroup-ing,whichincludesashallowsemanticparseframe,VerbNetprovidesitsownsemanticrolelabels,andaneo-Davidsonian-inspiredlogicalform.Allinfor-mationwithinVerbNetisclass-specific;theframesandrolesapplyequallytoallverbswithinaclass.7Further,VerbNet’slexicalentriesallowforassign-ingselectionalrestrictionsonthematicroles,e.g.re-quiringaparticipantbeCONCRETE,orANIMATE.Whiletheserestrictionstaketheformofproperties,thethematicrolesthemselvesareleftcategorical.Bonialetal.(2011)unitedVerbNet’ssemanticroleswiththoseofLIRICS8,astandardizationef-forttofacilitatemultilingualNLP.MotivatedinpartbythepropertiesofDowty,theyconstructedahi-erarchyof35rolesinterrelatedthroughtheirprop-ertyrequirements,implicitintheorganizationofthehierarchypairedwithnaturallanguageroledefini-tions.Thepropertiesbundledintotheserolesarethentakentakentobetype-level,hardconstraints:theycannotreflectsemanticnuanceswithinindivid-ualsentences,andarestrictlyboolean(apropertycannotholdtoadegree,orwithsomeuncertainty).TheSemLinkproject(Loperetal.,2007)providesamappingbetweenVerbNet,PropBank,FrameNet(seebelow)andWordNet(Fellbaum,1998).Cru-ciallyforourwork(see§6),SemLinkprovidesamappingfromtherolehierarchyofBonialetal.(2011)totheargumentannotationsofPropBank.VerbCornerVerbCorner(Hartstoneetal.,2013;Hartshorneetal.,2014)isanon-goingefforttoval-idateVerbNet’ssemanticannotations,focusingatafiner-grainedlevelofroleinformation.Forapartic-ularverbandsemanticfeatures,annotatorsarepro-videdcontextthroughasmall,made-upstory.An-notatorsthenreadexamplesentencespulledfromVerbNetanddeterminewhetherthosesentencesvi-olatethecontextualexpectations.Aswiththepresentwork,VerbCornercrowd-sourcestheanno-7Forinstance,thelemmasbreakandshatterarebothmem-bersofthesameclass(BREAK-45.1),capturingthecausativealternation.Bothsensescanbeusedtransitively(“Johnbroke/shatteredthemirror”)orintransitively(“Themirrorbroke/shattered.”),whilesemanticrolesassignJohntoAGENTandthemirrortoPATIENTinbothsyntacticframes,capturingthelogicalentailmentofaresultingdegradedphysicalform.8LinguisticInfRastructureforInteroperableResourCesandSystems(LIRICS):http://lirics.loria.fr/
l
D
Ö
w
N
Ö
A
D
e
D
F
R
Ö
M
H
T
T
P
:
/
/
D
ich
R
e
C
T
.
M
ich
T
.
e
D
u
/
T
A
C
l
/
l
A
R
T
ich
C
e
–
P
D
F
/
D
Ö
ich
/
.
1
0
1
1
6
2
/
T
l
A
C
_
A
_
0
0
1
5
2
1
5
6
6
8
1
0
/
/
T
l
A
C
_
A
_
0
0
1
5
2
P
D
.
F
B
j
G
u
e
S
T
T
Ö
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3
479
Station,thoughtherearekeydifferences:Hartstoneetal.(2013)arefocusedonlogicalentailments(whatmustbetrue)whereaswearefocusedonstronglysuggestedimplications(whatislikelytobetrue).FrameNetTheBerkeleyFrameNetProject(Bakeretal.,1998)isaninstantiationofFillmore’sframesemantictheory(Fillmore,1982).FrameNetde-scribeseventsviaaframe,consistingoflexicaltriggersandsemanticrolesthatareexpectedtobefilled.ThisissimilartoPropBank’stakeonpredi-cate/argumentstructure,thoughtherearesignificantdifferences:(1)FrameNettriggersmaybemul-tiword,verbalornominalexpressions;(2)unlikePropBank,FrameNetdefinesinterframerelations;(3)FrameNetisextremelyfine-grained(embracesrole-fragmentation),optingforsemanticcomplete-nessratherthanannotatorease.FrameNethasin-spiredsemanticrolelabeling(GildeaandJurafsky,2002;Litkowski,2004),inadditiontoframeseman-ticparsing(Bakeretal.,2007;Dasetal.,2010).3ExperimentalSetupTheliteraturereviewmakesclearthatunderstandingandannotatingfine-grainedrolepropertiesisvalu-ableinbothlinguistictheoryandincomputationallinguistics:undermanysetsofassumptions,suchpropertiesgroundoutthetheoryofcoarse-grainedroles.WefollowHartstoneetal.(2013)indirectlyaddressingfine-grainedproperties,hereinthecon-textoftheproto-roletheory.Theproto-roleap-proachgivesusasetoftestablequestionstoas-sessonacorpus.Wefocusontwomainissues:(ich)whethertheproto-rolesolutiontothemappingprob-lemscalesuptoverylargesetsofdata,Und(ii)thepredictionthattherewillbeaverylargesetofprop-ertyconfigurationsattestedasrolesinalargedataset.Ifthepredictionsfromtheproto-roletheoryaretrue,thenweconcludethatalargedatasetannotatedwithfine-grainedrolepropertiesmaybevaluableintasksrelatedtosemanticrolesandeventdetection.Toassessthesepredictions,webroadlyfollowKako(2006B)inoperationalizingproto-rolesusinglikelihoodquestionstargetingspecificroleproper-tiesinsentencesofEnglish.Thispaperpresentstwoexperimentsthatimplementthisstrategy.Inthere-mainderofthissectionwedescribethegeneralsetupoftheexperiments.Inparticular,wedescribeapro-RolepropertyQ:Howlikelyorunlikelyisitthat…instigatedArgcausedthePredtohappen?volitionalArgchosetobeinvolvedinthePred?awarenessArgwas/wereawareofbeinginvolvedinthePred?sentientArgwassentient?movedArgchangeslocationduringthePred?physexistedArgexistedasaphysicalobject?existedbeforeArgexistedbeforethePredbegan?existedduringArgexistedduringthePred?existedafterArgexistedafterthePredstopped?changedpossArgchangedpossessionduringthePred?changedstateTheArgwas/werealteredorsomehowchangedduringorbytheendofthePred?stationaryArgwasstationaryduringthePred?Table2:Questionsposedtoannotators.cessforarrivingatthespecificfine-grainedpropertyquestionsweask,thecreationofthedatasetthatweaskthequestionsabout,thetaskthatMechanicalTurkersarepresentedwith,andthemannerinwhichweanalyzeanddisplaytheresults.WefirstinspectedtherolehierarchyofBonialetal.(2011)alongwiththeassociatedtextualdefini-tions:theseweremanuallydecomposedintoasetofexplictbinaryproperties.Forexample,wedefinetheSemLinkACTORroleasaparticipantthathasthebinarypropertyofINSTIGATION.Fromthesepropertieswesubselectedthosethatweremostsim-ilartotheoriginalquestionsproposedbyDowty(seeTable1).Foreachsuchpropertywethengeneratedaquestioninnaturallanguagetobeposedtoanno-tatorsgivenanexamplesentence(seeTable2).Thesetwereportonhererepresentsasubsetoftheques-tionswehavetested;inongoingworkweareeval-uatingwhetherwecanexpandDowty’ssetofques-tions,e.g.tocapturerolessuchasINSTRUMENT.MethodsBecauseweareinterestedinthepoten-tialimpactofDowty’sproto-rolestheoryonhumanlanguagetechnologies,weperformanumberofre-latedcrowdsourcingexperiments,withthedualaimofvalidatingtheexisting(psycho-)linguisticlitera-tureonproto-rolesaswellaspilotingthishighlyscalableframeworkforfuturedecompositionalse-manticannotationefforts.Allofthecrowdsourcingexperimentsinthispa-perarerunusingAmazonMechanicalTurk,Und(ex-
l
D
Ö
w
N
Ö
A
D
e
D
F
R
Ö
M
H
T
T
P
:
/
/
D
ich
R
e
C
T
.
M
ich
T
.
e
D
u
/
T
A
C
l
/
l
A
R
T
ich
C
e
–
P
D
F
/
D
Ö
ich
/
.
1
0
1
1
6
2
/
T
l
A
C
_
A
_
0
0
1
5
2
1
5
6
6
8
1
0
/
/
T
l
A
C
_
A
_
0
0
1
5
2
P
D
.
F
B
j
G
u
e
S
T
T
Ö
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3
480
ceptforthekappascoresreportedforexperiment2)allworkerswererecruitedfromtheMTurkworkerpool.ThebasicsetupoftheexperimentsinSec-tions4and5isthesame.TheMechanicalTurkworkerispresentedwithasinglesentencewithahighlightedverbandonehighlightedargumentofthatverb.Thentheworkeranswersalloftheques-tionsinTable2forthatverb-argumentpairusingaLikertscalefrom1to5,withtheresponselabels:veryunlikely,somewhatunlikely,notenoughinfor-mation,somewhatlikely,andverylikely(SeeFigure2).EachMechanicalTurkHITyieldsresponsesforallthequestionsinTable2appliedtoasingleverb-argumentpair.TheMechanicalTurkexperimentsarerunwithtwotypesofsentences:thosewithrealverbsandnonsense(“nonce”)arguments,andthosewithentirelyrealEnglishsentences.Section4dis-cussestheformer“type-level”HITwithnonceargu-ments,whileSection5discussesthelatter“token-level”annotationtaskwithrealarguments.Figure2:ExampleHITquestionwithnoncearguments.DataToobtainverb-argumentpairsforthetaskdescribedhere,wedrewsentencesfromthesubsetofPropBankthatSemLinkannotatesforVerbNetroles.Fromthese,weremovedverbsannotatedasparticiples,verbswithtracearguments,verbsundernegationormodalauxiliaries,andverbsinembed-dedclausestoensurethatannotatorsonlysawverbsinveridicalcontexts–contextswherelogicalop-erationssuchasnegationdonotinterferewithdi-rectjudgmentsabouttheverbs.Forexample,inJohndidn’tdie,negationreversesthechange-of-statejudgmentforthewholesentence,despitethatbeingpartofthemeaningoftheverbdie.Wealsore-movedclausalarguments,asmostofthequestionsinTable2donotmakesensewhenappliedtoclauses;inongoingworkweareconsideringhowtoextendthisapproachtosucharguments.Atotalof7,045verbtokenswith11,913argumentspansfrom6,808sentencesremainedafterapplyingthesefilters.AnalysisToevaluatewhethertheresultsofthefollowingexperimentsaccordwithDowty’spro-posal,wefollowKako(2006B)intakingthemeandifferencebetweenthepropertyratingsofthesub-jectandobjectacrosssentences;see§2.1.WepresentthesedifferencesinthesameformatasinFigure1.HerewestickwithKako’sevaluationoftheresults,inordertodemonstratetheconvergenceofthelinguisticandpsycholinguisticevidencewithcomputationallinguisticapproaches;ourimmedi-ategoalinthepresentworkisnottoadvancethemethodology,buttoshowthatthesetechniquescanbepursuedthroughlarge-scalecrowdsourcing.WeperformtwoMechanicalTurkexperimentsonverbs:onewithnoncearguments,andonewithrealdatainSection5.Becausenonceargumentshavenomeaningintheirownright,weassumethatthepropertiesthatannotatorsassigntheseargumentsareafunctionoftheverbandrole,nottheargumentit-self.Hence,weassumethattheseannotationsareattheverb-roletypelevel.Conversely,theexperi-mentinSection5areatthetokenlevel,becauseallargumentshaverealEnglishinstantiations.4Experiment1:Nonce-basedThefirstexperimentwerunwithnonceargumentsisanattempttoreplicatetheresultsofKako(2006B).RecallthatKako(2006B)upholdsthepsychologicalvalidityofDowty(1991)’sArgumentSelectionPrin-ciple,bydemonstratingthathumansubjectsassignProto-AgentandProto-Patientpropertiestogram-maticalsubjectandobjectargumentsaccordingtoDowty’sprediction.(SeeFigure1.)Inthisexperiment,wegeneratesimpletransitivesentenceswithasmallsetofrealverbsandnoncearguments.Thesetofverbsarepreciselythosese-lectedbyKako(2006B)inhisfirstexperiment:add,deny,discover,finish,find,help,maintain,mention,pass,remove,show,write.Thequestionsweaskworkerstoanswercomefromaslightlyexpandedsetofproto-roleproperties.9Therewere16partic-9Aspointedoutbyareviewer,averbinanoncesentenceispotentiallyambiguous.BecauseweconstructedthenoncesentencesfromactualframesinPropBankexamples,ananno-tatorwillhaveatleastcoarsecuestotheintendedsense.InthisrespectwefollowKako,andestablishedprotocolinnonceex-perimentsingeneral.Weleavetheeffectofsenseambiguityonnoncepropertyjudgmentsforfuturework.
l
D
Ö
w
N
Ö
A
D
e
D
F
R
Ö
M
H
T
T
P
:
/
/
D
ich
R
e
C
T
.
M
ich
T
.
e
D
u
/
T
A
C
l
/
l
A
R
T
ich
C
e
–
P
D
F
/
D
Ö
ich
/
.
1
0
1
1
6
2
/
T
l
A
C
_
A
_
0
0
1
5
2
1
5
6
6
8
1
0
/
/
T
l
A
C
_
A
_
0
0
1
5
2
P
D
.
F
B
j
G
u
e
S
T
T
Ö
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3
481
ipantsintheexperiment,recruitedfromtheMTurkworkerpool,eachcompleting7.5HITsonaverage.Theresultsofthisexperiment,broadly,repli-cateKako(2006B)’searlierfindings:humananno-tatorsonaverageindicatethat,withinthesamesen-tence,thesubject-positionargumentismorelikelytohaveProto-Agentpropertiesthantheobject-positionargument,andtheobject-positionargumentismorelikelytohaveProto-Patientpropertiesthanthesubject-positionargument.Thisfindingisillus-tratedinFigure3.Inaddition,thebasicfactsmatchKako’soriginalfinding;compareFigures3and1.(OurINSTIGATIONpropertyisequivalenttoKako’sCAUSEDCHANGEproperty,andwedonothaveananalogueofhisCAUSEDDOproperty.)Proto-AgentpropertieshaveagreatereffectthanProto-Patientproperties,andCAUSATION,VOLITION,andAWARENESSareallstrongProto-Agentproperties.CREATIONandSTATIONARYareallweaker,butnon-zero,Proto-Patientpropertiesfortheseverbs.Therearesomedifferencesthatareapparent.Firstofall,whereKakodidnot(inthisparticularexperi-ment)findaneffectofCHANGEOFSTATE,wedid;thisisbroadlyconsistentwithKako’soverallfind-ings.WedidnotgetaneffectforMOVEMENTorforPHYSICALEXISTENCEinthisexperiment,incon-trasttoKako’sresults.Proto−AgentProto−Patientstationarychanged_statechanged_possessiondestroyedcreatedphysically_existedmovedsentientawarenessvolitionalinstigated−2−1012Mean difference (subject − object)Figure3:MechanicalTurkresultsforthenonceexper-iment.Apositivevalueforapropertyindicatesthat,onaverage,subject-positionargumentsreceivedahigherscoreforthatpropertythanobject-positionarguments.OurabilitytoreplicateKako(2006B)issignifi-cantfortworeason:(ich)itlendsfurthercredencetotheproto-rolehypothesis,Und(ii)itestablishesthatcrowd-sourcingwithnon-expertsinalesscontrolledsituationthanaformalexperimentresultsinreason-ableannotationsforthistaskwithminimaltraining.5Experiment2:Corpus-basedCanthisresultextendtorealcorpusdata?Ifso,theproto-roletheorycanleadtoavaluablesourceofannotationinformationaboutthematicroles.Toas-sessthis,wemovedfromasyntheticnoncetasktoamuchlargerscaleversionofthetaskusingdatafromPropBank(Palmeretal.,2005).EachiteminthistaskpresentstheannotatorwithaPropBanksen-tencewiththepredicateandargumenthighlighted,andasksthemthesamequestionsaboutthatactualsentence.ThesentencesweresampledfromProp-Bankasdescribedin§3.Ourprimarygoalinthiscollectioneffortwastoobtaininternallyconsistent,broad-coverageannota-tions.Thusweworkedthroughanumberofpilotannotationeffortstodeterminecross-annotatorreli-abilitybetweenannotatorsandwithourownjudge-ments.Fromthefinalversionofourpilot10wese-lectedasingleannotatorwithstrong-pairwiseagree-mentamongsttheothermostprolificannotators.Comparedtothefiveothermostprolificannota-torsinourfinalpilot,thepair-wiseaverageCohen’sKappawithsquaredmetriconanordinalinterpreta-tionoftheLikertscalewas0.576.11Inourlarge-scaleannotationtask,wehavecol-lectedpropertyjudgmentsonover9,000argumentsofnear5,000verbtokens,spanning1,610PropBankrolesets.Thisrepresentscloseto350hoursofanno-tationeffort.TheresultsareshowninFigure4.Be-causesomeargumentsinPropBankareabstract,forwhichmanyofthequestionsinTable2donotmakesense,weaddedanadditionalresponsefieldthatasks“Doesthisquestionmakesense”iftheworkergivesaresponselowerthan3(Figure6).Figure5showstheresultswithN/Aresponsesremoved.Forpresentationpurposes,weconvertthetemporalexis-tencepropertiestoCREATIONandDESTRUCTION.DiscussionTheoverallresultssubstantiallyre-semblebothKako’soriginalresultsandourexperi-10Basedonasetof10verbsselectedbasedonfrequencyintheCHILDEScorpus,filteringforverbsthathadenoughto-kensinPropBank;want.01,put.01,think.01,sehen.01,know.01,look.02,sagen.01,take.01,tell.01,andgive.01.11Oneofthosefiveannotatorshadlessstablejudgementsthantherest,whichweidentifiedbasedonapair-wiseKappascoreofonly0.383withourfinalannotator.Ifremovingthatannotatortheaveragepair-wisescorewiththeremainingfourannotatorsthenroseto0.625.
l
D
Ö
w
N
Ö
A
D
e
D
F
R
Ö
M
H
T
T
P
:
/
/
D
ich
R
e
C
T
.
M
ich
T
.
e
D
u
/
T
A
C
l
/
l
A
R
T
ich
C
e
–
P
D
F
/
D
Ö
ich
/
.
1
0
1
1
6
2
/
T
l
A
C
_
A
_
0
0
1
5
2
1
5
6
6
8
1
0
/
/
T
l
A
C
_
A
_
0
0
1
5
2
P
D
.
F
B
j
G
u
e
S
T
T
Ö
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3
482
Proto−AgentProto−Patientstationarychanged_statechanged_possessiondestroyedcreatedphysically_existedmovedsentientawarenessvolitionalinstigated−4−2024Mean difference (subject − object)Figure4:MechanicalTurkresultsforexperiment2.Proto−AgentProto−Patientstationarychanged_statechanged_possessiondestroyedcreatedphysically_existedmovedsentientawarenessvolitionalinstigated−4−2024Mean difference (subject − object)Figure5:Experiment2withN/Aremoved.Figure6:Anadditionalresponsefieldappearsforques-tionsthatmightnotbeapplicable.(Examplesentenceforthisquestion:“Theantibodythenkillsthecell.”)ment1.Aspredicted,theProto-Agentpropertiesarepredictorsofwhetheranargumentwillbemappedtosubjectposition,andtheProto-Patientproper-tiessimilarlypredictobjecthood.Notallproper-tiesareequal,andonthismuchlargerdatasetwecanclearlyseethatINSTIGATION(causation)isthestrongestproperty.Becausewehavemanydatapointsandareliableannotator,thevarianceonthisdataismuchsmaller.Thisgraphconfirmsthetheproto-rolehypothesisoveralargecorpus:fine-grainedrolepropertiespredictthemappingofse-manticrolestoargumentposition.Thisdatasetputsusinapositiontoaskawiderangeoffollowupquestionsaboutthenatureofthematicroles,manyofwhichwecannotadddressinthepresentpaper.Thecentralquestionwedoaddresshereisaboutpropertyconfigurations:sinceeachpropertycon-ExampleRtg(A)Heearnedamaster’sdegreeinarchitecturefromYale.N/A(B)Thebridgenormallycarries250,000commutersaday.1(C)Basketsofrosesandpottedpalmsadornedhisbench.5Table3:STATIONARYexamplesfromexperiment2.figurationrepresentsacoarse-grainedrole,wecanaskwhatthedistributionofpropertyconfigurationsisoverthiscorpus.Dowty’spredictionisthatweshouldseesomeclusteringaroundcommonconfig-urations,butalongtailrepresentingrolefragmenta-tion.Thepredictionofclassicalapproachesisthatweshouldseeonlythecommonconfigurationsasclusters,withnolongtail.Weturntothisissueinthefollowingsections,comparingourroleannota-tionsalsotorolesinVerbNetandFrameNet(usingSemLinkasthemappingamongthethreedatasets).OnekeydifferencefromDowty’spredictionsisthatSTATIONARYappearstoactasaProto-Agentproperty.First,weareusingaslightlydifferentno-tionofstationarytoDowty,whoproposedthatitberelativetotheevent–inthiswefollowKako.Second,ourMOVEMENTpropertyisreallyaboutchangeoflocation(seeTable2)andsoisnotthenegationofSTATIONARY.Third,ourcorpusisheav-ilybiasedtonon-physicaleventsandstates,wherethenotionofmotiondoesnotapply,andsointhisrespectmaynotbefullyrepresentativeofamorenaturalisticcorpus.Withintherelativelysmallpro-portionofdatathatisleft,wefindthatobjectsdonottendtobestationary,andsoifthisiscorrect,itmaysimplybewrongtoclassifytheabsoluteversionofSTATIONARYasaProto-Patientproperty.Threeex-amplesfromthedatasetareshownTable3,foreachcase–theresultisthatonceN/Aresponsesareex-cluded,examplessuchas(B)arestillmorethenormthanexamplessuch(C).AnnotationqualityToassessannotationqualitywebeganwithastratifiedsamplebasedoneachPropBankargumentIDintheset{0,1,2,3,4,5}.12Localresearcherseachthenanswered209questionsoverthissample.Oneoftheauthorsparticipated,12WhileargumentIDsaremeanttobemeaningfulwhencon-ditionedonaroleset,thevaluesstillcorrelatewiththe“core-ness”ofanargumentevenwhenindependentofroleset(e.g.,argumentIDs0and1aremostlikelytobeAGENTandPA-TIENT):ourstratificationaimedtosurveyacrossboth“core”and“peripheral”argumentroletypes.
l
D
Ö
w
N
Ö
A
D
e
D
F
R
Ö
M
H
T
T
P
:
/
/
D
ich
R
e
C
T
.
M
ich
T
.
e
D
u
/
T
A
C
l
/
l
A
R
T
ich
C
e
–
P
D
F
/
D
Ö
ich
/
.
1
0
1
1
6
2
/
T
l
A
C
_
A
_
0
0
1
5
2
1
5
6
6
8
1
0
/
/
T
l
A
C
_
A
_
0
0
1
5
2
P
D
.
F
B
j
G
u
e
S
T
T
Ö
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3
483
01230200400600800Ranklog10(Frequency)Figure7:Distributionofpropertyconfigurationsinex-periment2.Toobtaincategoricalrolesforpurposesofcomparison,responsesof2/4weremappedto1/5,giv-ingconfigurationson11propertiesoverwhatwemightcoarselyconsider:{False(1),Unknown(3),True(5)}.achievingaKappascoreof0.619withtheannota-tor.Twocolleaguesgenerallyfamiliarwiththematicrolesbutwithoutpriorexperiencewiththeprotocolorourgoalsachievedscoresof0.594and0.642.Fi-nally,acolleaguewhospeaksEnglishasasecondlanguageacheivedaKappascoreof0.479.Thesecorrelations,alongwithourinitialselectioncriteriafortheannotator,andthencombinedwiththosecor-relationsobservedinTable6(discussedbelow),sug-gestsourprocessresultedinausefulresourcewhichwewillreleasetothecommunity.Insection6weadditionallyprovideaqualitativeindicatorofannotationquality,intheformofanalignmenttoVerbNetroles.6ComparisontoOtherRolesetsApredictionemergingfromtheproto-rolehypoth-esisisthat,whenasetofrole-relevantpropertiessuchasthoseinTable2aretestedonalargescale,weshouldnotfindcleanrole-clusters.Wedoex-pecttofindcertaincommonrole-typesappearingfrequently,butwealsoexpecttheretobealongtailofcombinationsofproperties.Thisisexactlywhatwefindwhenexaminingourresults.Figure6showsthefrequencyofpropertyconfigurationsinthedataset.Around800configurationsareattested,withnearly75%ofthosemakingupthetail.Theproto-rolehypothesispredictsthattherearenaturalsentencesinwhichanargumentcanbeAGENT/PATIENT-like,yetbemissingoneormoreProto-agent/patientproperties.Thisiswhatgivesrisetotheobservedlongtailofpropertyconfigu-rations:casesthatwouldotherwisebelumpedto-getheras,e.g.,AGENT,areinsteadplacedinamorediversesetofbins.WhileDowty’stheoryisreallyaboutrolesatthetype-level,thesebinsarealsouse-fulforunderstandingroleannotationsatthetokenlevel,i.e.capturingexactlythosepropertiesthatholdofthegivenargumentincontext.Table4showsthreereal-worldsentencestakenfromtheWallStreetJournalinvolvingtheverbkill.EachsentencehaswhatPropBankwouldcallaKILL.01,ARG0-PAG,orthefirstargumentoftherolesetKILL.01,aparticularsenseofthewordkill.13Further,eachoftheseargumentsarelabeledasaVerbNetAGENTandFrameNetKILLER/CAUSEthroughSemLink.ThesesentenceswereselectedpurelybecausetheyweretheonlyinstancesofkillinourdatasetwithSemLinkroleannotations.Then,whenexaminingourannotationsfortheseargu-ments,wefindthatourmotivationsfrom§3forthisenterprisearejustified.Atthetokenlevel,therearerobustinferencesleadingtodifferentresultsoneachexampleforkeyproto-roleproperties,butineachcasethesubjectisstillabetterProto-agentthantheobject.Fromthistriplet,welearnthatthesubjectofkillneedn’tbevolitionallyinvolved(asintheacci-dentaldeathinA),needn’tbeawareofthekilling,andevenneednotbesentient.Thepresentanno-tationscheme,incontrasttothecoarselabelpro-videdtotheseexamplesinVerbNet,capturesthisvariationwhilestillallowinginferencetotype-levelpropertiesoftheverbkill.(Theseexamplesalsoclearlyillustratethedegreetowhichnounseman-ticscaninfluencethematicrole-relatedjudgmentswhencarriedoutonnaturaldata,somethingthefine-grainedapproachallowsustoexploredirectly.)WecanalsoclearlyseefromthistripletthatINSTIGA-TIONisconstantacrossexamples,asisPHYSICALEXISTENCE.Interestingly,theexample(B)showsthatkillingdoesnotevenprecludethecontinuation13PAGisarecentadditiontoPropBanksemantics,standingforProto-Agentbutinterpretedasanunweighteddisjunctionoffeatures:“itactsvolitionally,issentient,orperceives,causesachangesofstate,ormoves”(K¨ublerandZinsmeister,2015).Anotheraddition,PPT,standsforProto-Patient.Whilemoti-vatedbyDowty’sterminology,theseadditionsdonotcapturetheindividualproperty-basednotionweadvocateforhere.
l
D
Ö
w
N
Ö
A
D
e
D
F
R
Ö
M
H
T
T
P
:
/
/
D
ich
R
e
C
T
.
M
ich
T
.
e
D
u
/
T
A
C
l
/
l
A
R
T
ich
C
e
–
P
D
F
/
D
Ö
ich
/
.
1
0
1
1
6
2
/
T
l
A
C
_
A
_
0
0
1
5
2
1
5
6
6
8
1
0
/
/
T
l
A
C
_
A
_
0
0
1
5
2
P
D
.
F
B
j
G
u
e
S
T
T
Ö
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3
484
SentencesProperty(A)(B)(C)(A)Shewasuntrainedand,inonebotchedjobkilledaclient.instigated555(B)Theantibodythenkillsthecell.volitional215(C)AnassassininColombiakilledafederaljudgeonaMedellinstreet.awareness315PropBankKILL.01,ARG0-PAG:killersentient515VerbNetMURDER-42.1-1,AGENT:ACTORinaneventwhoinitiatesandcarriesouttheeventintentionallyorconsciously,andwhoexistsindependentlyoftheeventmoved333physexisted555created111destroyed131FrameNetKILLING,KILLER/CAUSE:(Thepersonorsentiententity)/(Aninanimateentityorprocess)thatcausesthedeathoftheVICTIM.changedposs111changedstate333stationary333Table4:Comparisonofroleannotationsforkillacrossresources.Ratings:1=veryunlikely,5=verylikely.SentencesProperty(A)(B)(C)(A)Thestocksplitfour-for-oneonOct.10.instigated111(B)“In1979,thepairsplitthecompanyinhalf,withWalterandhisson,Sam,agreeingtooperateundertheMerksamerJeweleryname.”volitional111awareness151(C)ThecompanydownplayedthelossofMr.Leskandsplithismerchandisingresponsibilitiesamongacommitteeoffourpeople.sentient111moved111PropBankSPLIT.01,ARG1-PPT:thingbeingdividedphysexisted111VerbNetSPLIT-23.2,PATIENT:UNDERGOERinaneventthatexperiencesachangeofstate,locationorcondition,thatiscausallyinvolvedordirectlyaffectedbyotherparticipants,andexistsindependentlyoftheevent.created111destroyed151changedposs155FrameNetCAUSETOFRAGMENT,WHOLEPATIENT:TheentitywhichisdestroyedbytheAGENTandthatendsupbrokenintoPIECES.changedstate554stationary111Table5:Comparisonofroleannotationsforsplitacrossresources.ofexistenceaftertheevent,sotheEXISTENCEprop-ertymaynotbefullyindependent.Table5makesasimilarpointusingtheverbsplit.Thesethreeinstancesofsplit,labeledwiththesamerole(andverbsense)inPropBank/VerbNet,showcleardifferencesintermsoffine-grainedroleprop-erties.(Notealsothatin(A),aPropBankARG1ap-pearsinsubjectposition.)WhilethereisconsensusonCHANGEOFSTATE,thereisvariationinwhethertheargumentisDESTROYED,CHANGESPOSSES-SION,andisAWAREofitsinvolvementintheevent.AlignmentwithVerbNetInwhatfollowsweex-ploreanon-exactmappingwherewehavetakensen-tencesinSemLinkannotatedwithVerbNetcoarse-grainroles,andsimplyprojectedthemean1-5proto-roleratings(subtractingN/A)ontoeachrole.Thisservestwopurposes:(1)thequalityofthismappingservestoverifythequalityoftheproto-roleannotations,Und(2)thisalignmenthelpscom-parebetweencoarseandfine-grainedroleannota-tions.Thisalignmentisaproof-of-concept,andweleaveadeeperexplorationofwaysofdoingthissortofalignmentforthefuture.Table6showsthefullalignment.Avalueof5indicatesthattheroletendstodeterminetheproto-propertypositively,i.e.AGENTSareextremelylikelytobejudgedasinsti-gators.Avaluecloseto3indicatesthattheroleisneutralwithrespecttotheproto-property,e.g.AGENTSmayormaynotmove.Avaluecloseto1indicatesthattheargumentswiththatrolearelikelytohavethenegativeversionoftheproto-property,e.g.AGENTStendnottoCHANGEPOSSESSION.Atabroadleveltheresultsarestrong,thoughwewillnotbeabletodiscusseverydetailhere.InthisalignmentthejudgmentsofN/Ahavebeenremoved.14Inthecaseofe.g.theINSTIGATIONvalueforTHEME,thissupportsinterpretingtheroleasassigningnovaluetoinstigationatall;similarlyforsomeoftheothervaluesforTHEME.Insome14ThisisnottheonlywaytotreatN/Aratings,andwewillleaveafullexplorationtofuturework.
l
D
Ö
w
N
Ö
A
D
e
D
F
R
Ö
M
H
T
T
P
:
/
/
D
ich
R
e
C
T
.
M
ich
T
.
e
D
u
/
T
A
C
l
/
l
A
R
T
ich
C
e
–
P
D
F
/
D
Ö
ich
/
.
1
0
1
1
6
2
/
T
l
A
C
_
A
_
0
0
1
5
2
1
5
6
6
8
1
0
/
/
T
l
A
C
_
A
_
0
0
1
5
2
P
D
.
F
B
j
G
u
e
S
T
T
Ö
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3
485
RoleFreqinstigatedvolitionalawarenesssentientmovedexistedcreateddestroyedchgposschgstatestationaryAgent15464.9(1355)4.8(1273)4.9(1275)4.8(810)3.1(897)4.7(947)1.1(1413)1.1(1508)1.7(432)3.3(1489)2.8(874)Theme11533.4(214)3.9(215)4.6(226)4.7(147)3.6(335)4.3(412)1.9(986)1.3(1037)3.3(339)3.9(999)2.5(300)Patient3123.1(77)3.2(80)4.5(85)4.4(47)3.3(75)4.6(100)1.1(285)1.6(293)3.4(99)4.5(294)2.7(69)Experiencer2104.4(161)4.3(167)4.9(169)4.8(128)3.1(135)4.7(139)1.0(195)1.1(204)1.4(41)3.6(204)2.8(137)Stimulus1294.3(64)4.0(33)4.1(35)4.2(26)2.9(35)3.7(42)1.7(107)1.1(114)1.8(20)3.1(115)2.9(32)Topic1144.0(2)2.3(3)2.5(4)3.5(4)3.0(4)3.0(7)2.0(92)1.1(91)2.6(18)3.4(74)3.0(3)Destination911.6(5)2.9(15)4.5(16)4.8(8)2.3(24)4.9(48)1.5(74)1.2(75)2.2(22)4.2(75)4.1(39)Recipient881.4(37)3.6(58)4.8(60)4.9(35)3.0(46)4.5(52)1.5(84)1.0(85)2.3(30)3.7(82)3.0(40)Extent87—(0)—(0)—(0)—(0)—(0)—(0)1.0(1)1.0(2)—(0)3.0(1)—(0)…12rolesomitted…Instrument164.4(9)4.5(8)4.5(8)5.0(4)3.8(5)4.3(7)1.3(15)1.0(15)1.3(6)3.3(13)2.0(4)InitialLoc.152.2(5)2.3(7)4.2(8)3.5(2)2.5(4)3.0(4)1.0(14)2.1(14)1.8(6)4.2(13)2.3(3)Beneficiary133.6(5)5.0(5)5.0(5)5.0(1)3.0(1)3.5(4)2.2(10)1.0(13)3.0(4)3.7(13)3.0(1)Material95.0(1)5.0(1)5.0(2)5.0(1)3.7(3)5.0(3)1.0(7)1.2(8)5.0(1)3.7(7)1.7(3)Predicate8—(0)5.0(1)5.0(1)—(0)—(0)—(0)1.0(8)2.2(8)—(0)3.4(5)—(0)Asset7—(0)—(0)—(0)—(0)3.0(1)3.0(1)1.0(5)1.0(6)5.0(1)4.3(3)—(0)Table6:HighandlowfrequencyVerbNetroles(viaSemLink)alignedwithmeanpropertyratingswhenexcludingN/Ajudgments.Freqprovidesthenumberofannotationsthatoverlappedwitharole.Inparenthesisisthenumberofcasesforthatpropertywhichwerejudgedapplicable(notN/A).E.g.weannotated1,546argumentsthatSemLinkcallsAGENT,where1,355ofthoseweredeemedapplicablefortheinstigationproperty,withameanresponseof4.9.12mid-frequencyrolesareomittedhereforspacereasons;thefullalignmentisprovidedwiththedatasetforthispaper.caseswithlargenumbersofN/Aresponses,e.g.theawarenessandsentientpropertiesforTHEME,theprovidedmeanishigh,suggestingtherolemaybemoreheterogeneousthanwouldotherwiseappear.Inlieuofanexhaustivediscussion,wewillmoti-vatethealignmentwithseveralinterestingexamplesofAWARENESS.AWARENESStendedtoberatedhighly.Table7givesarangeofexamplesillustrat-ingparticulartokensofjudgementsrelativetoVerb-Netroles.In(A-C)wegivethreestraightforwardexampleswheretheboldedargumentwasjudgedtobeawareofinvolvementintheevent.Abstractenti-tiessuchascompanieswereconsistentlyannotatedashavingthepotentialtobeaware.Consequently,inBforexample,FordisannotatedasbeingawarethatMazdamakestheTracerforthecompany.Inthesecases,itisintuitivelyrightatthetokenlevelthattheparticipantislikelytobeawareoftheirinvolvementintheevent,butthisdoesnotmeanthatwecancon-cludeanythingabouttherole;forexample,forBEN-EFICIARIESandINSTRUMENTSwehaveonlyafewexamplesoftheAWARENESSproperty.InC-E,wehavegiventhreeexamplesofdifferentratingsforAWARENESSfocusingontheDESTINA-TIONrole.Allthreeratingsareintuitivelystraight-forwardatthetokenlevel;inDtherecipientofthecases(thecourt)maynotyetbeawareofthedeci-sion.InEtherecipientofthesprinklingwasababyandthereforewasquiteunlikelytobeawareofherRtg(Role)ExampleA5(AGENT)Theyworryabouttheircareers,drinktoomuchandsuffer[…]B5(BENEFICIARY)MazdamakestheTracerforFord.C5(DESTINATION)Commercialfishermenandfishprocessorsfiledsuitinfederalcourt[…]D3(DESTINATION)Butthecourt[…]sentthecasesbacktofed-eraldistrictcourtinDallas.E1(DESTINATION)Whenthegoodfairy[…]hoveredoverthecradleofEdita[…],shesprinkledherwithhighEflats,[…]F5(INSTRUMENT*)Guestsbringmoviesontape,andshowtheirfavoritethree-to-fiveminutesegmentsonthescreen[…]Table7:ExamplesofAWARENESS:howlikelyisitthattheboldargumentisawareofbeinginvolvedintheevent?potentialfuturesingingcareer(afactaboutthecon-textandargumentmorethantheverb).Fhelpsillus-tratethequalityofourannotations:personalcom-municationwithSemLinkresearchersverifiedthatwediscoveredararebugviaourprocess.157SemanticProto-RoleLabelingSRLsystemsaretrainedtopredicteither:(ich)apredicateorframespecificnotionofrole(e.g.,FrameNet),oder(ii)across-predicate,sharednotionofrole(e.g.,PropBank).(ich)allowsforfine-graindistictionsspecifictoasinglepredicate,butrisksdatasparsity(needingmanyexamplesperpredi-cate).(ii)allowsforsharingstatisticsacrosspred-icates,butrequirescareful,manualcross-predicate15TheroleviaSemLinkshouldbeAGENT.
l
D
Ö
w
N
Ö
A
D
e
D
F
R
Ö
M
H
T
T
P
:
/
/
D
ich
R
e
C
T
.
M
ich
T
.
e
D
u
/
T
A
C
l
/
l
A
R
T
ich
C
e
–
P
D
F
/
D
Ö
ich
/
.
1
0
1
1
6
2
/
T
l
A
C
_
A
_
0
0
1
5
2
1
5
6
6
8
1
0
/
/
T
l
A
C
_
A
_
0
0
1
5
2
P
D
.
F
B
j
G
u
e
S
T
T
Ö
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3
486
instigatedvolitionalawarenesssentientmovedNull57.459.058.474.064.1Pos79.074.975.574.164.1Full82.974.975.575.267.1existedcreateddestroyedchgposschgstatestationary57.869.480.472.445.965.059.769.680.772.446.565.164.872.082.372.358.069.0Table8:Testclassificationaccuraciesforeachproperty.analysistoensureequivalentrole-semantics(Loperetal.,2007),andasinseenTables4and5itmaynotbefeasibletoensureexactequivalence.Ourapproachaddressesthischallengebydrop-pingthenotionofcategoricalroleentirely,replacingitwithresponsestoproto-rolequestionsthatcanbesharedacrossallarguments16andpredicates.17Fur-ther,aslikelihoodjudgementsmaybeinterpretedasscalars,thenthismayprovideasmootherrepresen-tationforpredictionanddownstreamuse,akintotherecentpushtoreplacecategorical“1-hot”wordrep-resentationswithvector-spacemodels.AsanexampleSPRLmodel,wetrainedseparatelog-linearclassifierswithL2regularizationonthejudgmentsofeachpropertyintheresultsfromEx-periment2.AsinFig.6wecollapsedratingstoacategorical{1,3,5},andincludedN/A,foraresul-tant4-wayclassificationproblem.18The9,778ar-gumentsthatappearinthedatasetweredividedintotraining(7,823),Entwicklung(978),andtest(977).Wetrainedthreemodels:Null,withonlyaninterceptfeature19;Pos,whichaddsasafeaturethelinearoffsetoftheargumentrelativetotheverb(asacoarseproxyforsyntacticstructure);andFull,whichaddedavectorembeddingoftheverb(Rastogietal.,2015).20Evenwiththisbasicmodelweseeevidenceoflearningproperty-specificdistributionsacrossverbalpredicates,suchasforCHANGEDSTATE.16E.g.,thenotionsofACTOR,AGENT,andevenPATIENTmayoverlapintheirunderlyingproperties.17E.g.,theproto-Agentofbuildwillberelatedbutnotidenti-caltothatofkill:wherecommonalitiesexist,predictivemodelscanbenefitfromtheoverlap.18Futureworkonpredictionmayexplorealternativeformu-lations,suchasa2-stepprocessoffirstpredictingN/A,thenperformingregressiononlikelihood.19TheNullclassifierpredictsaratingof1forCREATEDandDESTROYEDandN/Aforalltheotherproperties.20http://cs.jhu.edu/˜prastog3/mvlsa/8ConclusionsandFutureWorkInthispaperwehaveadoptedfromtheoreticallin-guisticstheideathatthematicrolesshouldbede-composedintomorefine-grainedpropertiesthathaveaprototypestructure–theProto-roleHypoth-esis.Wedevelopedanannotationtaskbasedonthisidea,andtesteditbothinasmalescalenonce-basedversionandaverylargescaleversionusingrealdatafromPropBank(/WSJ).Onemainresultisthattheproto-rolehypothesisholdstrueatthisverylargescale.Asecondresultisthat,atthisscalewegainevidenceforasubstantialamountof‘rolefragmen-tation’inthelexiconofEnglish:wefindapprox-imately800discretepropertyconfigurations.Theproto-roleapproachallowsustocopewithfragmen-tationbyfocusingonthefine-grainedpropertiesthatmakeuproles.Weshowedthisallowsagreaterde-greeofaccuracyinroleannotations,forexamplehandlingvariabilityinfine-grainedpropertiesacrosstokensofaverbinacorpusthatleadtocoarse-grainedcategorizationchallenges.Finally,wehaveshownitpracticaltodirectlyannotateacorpuswithfine-grainedpropertiesandproducedalargecollec-tionofsuchannotations,whichwereleasetothecommunity.Wearecurrentlyexpandingtheannota-tionsetbeyondWSJ,andbeyondEnglish,aswellasapplyingittotheoreticalquestionsaboutverbclassandargumentstructure(DavisandKoenig,2000;Kako,2006B),alongwithwordsense.Finally,wearebuildingonthebaselinemodelin§7tomorebroadlyinvestigatehowdecompositionalsemanticannotationscanguidelinguisticallymotivatedrep-resentationlearningofmeaning.AcknowledgmentsGreatthankstoMarthaPalmer,TimO’Gorman,ScottGrimm,andthereviewersfortheirfeedback;RobertBusbyforannotations;JulianGroveforworkonapredecessorprojectwithKyleRawlins.ThankstoSanjeevKhudanpur,JohnJ.Godrey,andJanHaji˘c,aswellasJHUandCharlesUniversityforcoor-dinatingtheFredJelinekMemorialWorkshopin2014,supportedbyNSFPIRE(0530118).SupportcamefromanNSFGraduateResearchFellowship,DARPADEFTFA8750-13-2-001(LargeScaleParaphrasingforNaturalLanguageUnderstanding),theJHUHLTCOE,thePaulAllenInstituteofArtificialIntelligence(AcquisitionandUseofParaphrasesinaKnowledge-RichSetting),andNSFBCS-1344269(GradientSymbolicComputation).
l
D
Ö
w
N
Ö
A
D
e
D
F
R
Ö
M
H
T
T
P
:
/
/
D
ich
R
e
C
T
.
M
ich
T
.
e
D
u
/
T
A
C
l
/
l
A
R
T
ich
C
e
–
P
D
F
/
D
Ö
ich
/
.
1
0
1
1
6
2
/
T
l
A
C
_
A
_
0
0
1
5
2
1
5
6
6
8
1
0
/
/
T
l
A
C
_
A
_
0
0
1
5
2
P
D
.
F
B
j
G
u
e
S
T
T
Ö
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3
487
ReferencesAfraAlishahiandSuzanneStevenson.2010.Acompu-tationalmodeloflearningsemanticrolesfromchild-directedlanguage.LanguageandCognitivePro-cesses,25(1):50–93.CollinF.Baker,CharlesJ.Fillmore,andJohnB.Lowe.1998.TheBerkeleyFrameNetproject.InProceed-ingsoftheInternationalConferenceonComputationalLinguistics,pages86–90.CollinBaker,MichaelEllsworth,andKatrinErk.2007.Semeval’07task19:Framesemanticstructureextrac-tion.InProceedingsofthe4thInternationalWorkshoponSemanticEvaluations,pages99–104.AssociationforComputationalLinguistics.FrankR.Blake.1930.Asemanticanalysisofcase.InJamesT.Hatfield,WernerLeopold,andA.J.FriedrichZiglschmid,editors,Curmevolumeoflinguisticstud-ies,pages34–49.LinguisticSocietyofAmerica.ClaireBonial,WilliamCorvey,MarthaPalmer,VolhaV.Petukhova,andHarryBunt.2011.Ahierarchicaluni-ficationofLIRICSandVerbNetsemanticroles.InSe-manticComputing(ICSC),2011FifthIEEEInterna-tionalConferenceon,pages483–489,September.GregN.CarlsonandMichaelK.Tanenhaus.1988.Thematicrolesandlanguagecomprehension.InW.Wilkins,editor,SyntaxandSemantics:ThematicRelations.AcademicPress.GregN.Carlson.1984.Thematicrolesandtheirroleinsemanticinterpretation.Linguistics,pages259–279.XavierCarrerasandLlu´ısM`arquez.2005.IntroductiontotheCoNLL-2005sharedtask:Semanticrolelabel-ing.InProceedingsoftheNinthConferenceonCom-putationalNaturalLanguageLearning,pages152–164.AssociationforComputationalLinguistics.Hector-NeriCasta˜neda.1967.CommentsonDonaldDavidson’s‘Thelogicalformofactionsentences’.InNicholasRescher,editor,TheLogicofDecisionandAction.UniversityofPittsburghPress.NoamChomsky.1981.Lecturesongovernmentandbinding.ForisPublications.W.Croft.1991.Syntacticcategoriesandgrammaticalrelations.UniversityofChicagoPress.D.Cruse.1973.Somethoughtsonagentivity.JournalofLinguistics,9:1–204.DipanjanDas,NathanSchneider,DesaiChen,andNoahA.Smith.2010.Probabilisticframe-semanticparsing.InProceedingsofHumanLanguageTech-nologiesConferenceoftheNorthAmericanChapteroftheAssociationforComputationalLinguistics,pages948–956.A.R.DavisandJ.-P.Koenig.2000.Linkingascon-straintsonwordclassesinahierarchicallexicon.Lan-guage,76:56–91.DavidDowty.1989.Onthesemanticcontentofthenotion‘thematicrole’.InBarbaraPartee,GennaroChierchia,andRayTurner,editors,Properties,typesandmeanings,volII.Kluwer.DavidDowty.1991.Thematicproto-rolesandargumentselection.Language,67(3):547–619.ChristianeFellbaum.1998.WordNet.WileyOnlineLi-brary.CharlesFillmore.1966.Towardsamoderntheoryofcase.TheOhioStateUniversityprojectonlinguisticanalysisreport13,TheOhioStateUniversity.CharlesJ.Fillmore.1976.Framesemanticsandthena-tureoflanguage.InAnnalsoftheNewYorkAcademyofSciences:ConferenceontheOriginandDevelop-mentofLanguageandSpeech,volume280,pages20–32.CharlesFillmore.1982.Framesemantics.Linguisticsinthemorningcalm,pages111–137.DanielGildeaandDanielJurafsky.2002.Automaticla-belingofsemanticroles.ComputationalLinguistics,28(3):245–288.ScottGrimm.2005.Thelatticeofcaseandagentivity.MScthesis,UniversityofAmsterdam.ScottGrimm.2011.Semanticsofcase.Morphology,21:515–544.JeffreyS.Gruber.1965.Studiesinlexicalrelations.Ph.D.dissertation,MassachusettsInstituteofTechnol-ogy.PatrickHanks.2013.LexicalAnalysis:NormsandEx-ploitations.MITPress.JoshuaK.Hartshorne,ClaireBonial,andMarthaPalmer.2014.TheVerbCornerProject:FindingsfromPhase1ofcrowd-sourcingasemanticdecompositionofverbs.InProceedingsoftheAnnualMeetingoftheAssoci-ationforComputationalLinguistics,pages397–402.AssociationforComputationalLinguistics.JoshuaHartstone,ClaireBonial,andMarthaPalmer.2013.TheVerbCornerproject:Towardanempirically-basedsemanticdecompositionofverbs.InProceedingsoftheConferenceonEmpiricalMeth-odsinNaturalLanguageProcessing,pages1438–1442.GeoffreyE.Hinton.1981.Implementingsemanticnet-worksinparallelhardware.InGeoffreyE.HintonandJohnA.Anderson,editors,ParallelModelsofAsso-ciativeMemory.Erlbaum,Hillsdale,NJ.GeoffreyE.Hinton.1986.Learningdistributedrepre-sentationsofconcepts.InProceedingsoftheEighthAnnualConferenceoftheCognitiveScienceSociety,pages1–12.Amherst,MA.RayS.Jackendoff.1972.SemanticInterpretationinGenerativeGrammar.TheMITPress.RayS.Jackendoff.1987.Thestatusofthematicrelationsinlinguistictheory.LinguisticInquiry,18:369–411.
l
D
Ö
w
N
Ö
A
D
e
D
F
R
Ö
M
H
T
T
P
:
/
/
D
ich
R
e
C
T
.
M
ich
T
.
e
D
u
/
T
A
C
l
/
l
A
R
T
ich
C
e
–
P
D
F
/
D
Ö
ich
/
.
1
0
1
1
6
2
/
T
l
A
C
_
A
_
0
0
1
5
2
1
5
6
6
8
1
0
/
/
T
l
A
C
_
A
_
0
0
1
5
2
P
D
.
F
B
j
G
u
e
S
T
T
Ö
N
0
8
S
e
P
e
M
B
e
R
2
0
2
3
488
EdwardKako.2006a.Thesemanticsofsyntacticframes.LanguageandCognitiveProcesses,21(5):562–575.EdwardKako.2006b.Thematicrolepropertiesofsub-jectsandobjects.Cognition,101:1–42.SandraK¨ublerandHeikeZinsmeister.2015.Cor-pusLinguisticsandLinguisticallyAnnotatedCorpora.BloomsburyPublishing.BethLevinandMalkaRappaportHovav.2005.Argu-mentRealization.CambridgeUniversityPress.BethLevin.1993.EnglishVerbClassesandAlter-nations:APreliminaryInvestigation.UniversityofChicagoPress.KenLitkowski.2004.Senseval-3task:Automaticlabel-ingofsemanticroles.InSenseval-3:ThirdInterna-tionalWorkshopontheEvaluationofSystemsfortheSemanticAnalysisofText,pages141–146.EdwardLoper,Szu-TingYi,andMarthaPalmer.2007.Combininglexicalresources:Mappingbetweenprop-bankandverbnet.InProceedingsofthe7thInterna-tionalWorkshoponComputationalLinguistics.MitchellPMarcus,MaryAnnMarcinkiewicz,andBeat-riceSantorini.1993.Buildingalargeannotatedcor-pusofEnglish:ThePennTreebank.ComputationalLinguistics,19(2):313–330.JamesL.McClellandandAlanH.Kawamoto.1986.Mechanismsofsentenceprocessing:Assigningrolestoconstituents.InDavidE.RumelhartandJamesL.McClelland,editors,ParallelDistributedProcessing:ExplorationsintheMicrostructureofCognition,Vol.2.MITPress,Cambridge,MA.MarthaPalmer,DanGildea,andPaulKingsbury.2005.Thepropositionbank:Acorpusannotatedwithse-manticroles.ComputationalLinguistics,31:71–106.TerenceParsons.1990.EventsintheSemanticsofEn-glish.MITPress,Cambridge,MA.JamesPustejovsky,PatrickHanks,andAnnaRumshisky.2004.Automatedinductionofsenseincontext.InProceedingsoftheInternationalConferenceonCom-putationalLinguistics.MalkaRappaportandBethLevin.1988.Whattodowiththetaroles.InW.Wilkins,editor,Syntaxandseman-tics21:ThematicRelations,pages7–36.AcademicPress.MalkaRappaportHovavandBethLevin.1998.Buildingverbmeanings.InM.ButtsandW.Geuder,editors,TheProjectionofArguments:LexicalandComposi-tionalFactors,pages97–134.CSLIPublications.PushpendreRastogi,BenjaminVanDurme,andRamanArora.2015.MultiviewLSA:Representationlearn-ingviageneralizedCCA.InProceedingsoftheAn-nualMeetingoftheNorthAmericanChapteroftheAssociationforComputationalLinguistics.KarinKipperSchuler.2005.VerbNet:Abroadcover-age,comprehensiveverblexicon.Ph.D.dissertation,UniversityofPennsylvania.LeonardTalmy.1978.Figureandgroundincomplexsentences.InJosephGreenberg,editor,UniversalsofHumanLanguage,pages625–649.StanfordUniver-sityPress.RalphWeischedel,MarthaPalmer,MitchellMarcus,Ed-uardHovy,SameerPradhan,LanceRamshaw,Ni-anwenXue,AnnTaylor,JeffKaufman,MichelleFranchini,MohammedEl-Bachouti,RobertBelvin,andAnnHouston.2013.Ontonotesrelease5.0LDC2013T19.LinguisticDataConsortium,Philadel-phia,PA.
