Transactions of the Association for Computational Linguistics, 2 (2014) 143–154. Action Editor: Ellen Riloff.
Submitted 9/2013; Revised 2/2014; Published 4/2014. c(cid:13)2014 Association for Computational Linguistics.
TemporalAnnotationintheClinicalDomainWilliamF.StylerIV1,StevenBethard2,SeanFinan3,MarthaPalmer1,SameerPradhan3,PietCdeGroen4,BradErickson4,TimothyMiller3,ChenLin3,GuerganaSavova3andJamesPustejovsky51DepartmentofLinguistics,UniversityofColoradoatBoulder2DepartmentofComputerandInformationSciences,UniversityofAlabamaatBirmingham3Children’sHospitalBostonInformaticsProgramandHarvardMedicalSchool4MayoClinicCollegeofMedicine,MayoClinic,Rochester,MN5DepartmentofComputerScience,BrandeisUniversityAbstractThisarticlediscussestherequirementsofaformalspecificationfortheannotationoftemporalinformationinclinicalnarratives.WediscusstheimplementationandextensionofISO-TimeMLforannotatingacorpusofclinicalnotes,knownastheTHYMEcor-pus.Toreflecttheinformationtaskandtheheavilyinference-basedreasoningdemandsinthedomain,anewannotationguidelinehasbeendeveloped,“theTHYMEGuidelinestoISO-TimeML(THYME-TimeML)”.Toclarifywhatrelationsmeritannotation,wedistinguishbetweenlinguistically-derivedandinferentially-derivedtemporalorderingsinthetext.WealsoapplyatopperformingTemp-Eval2013systemagainstthisnewresourcetomeasurethedifficultyofadaptingsystemstotheclinicaldomain.ThecorpusisavailabletothecommunityandhasbeenproposedforuseinaSemEval2015task.1IntroductionThereisalong-standinginterestintemporalreason-ingwithinthebiomedicalcommunity(Savovaetal.,2009;Hripcsaketal.,2009;Meystreetal.,2008;Bramsenetal.,2006;Combietal.,1997;Keravnou,1997;Dolin,1995;Irvineetal.,2008;Sullivanetal.,2008).Thisinterestextendstotheautomaticex-tractionandinterpretationoftemporalinformationfrommedicaltexts,suchaselectronicdischargesum-mariesandpatientcasesummaries.Makingeffectiveuseoftemporalinformationfromsuchnarrativesisacrucialstepintheintelligentanalysisofinformat-icsformedicalresearchers,whileanawarenessoftemporalinformation(bothimplicitandexplicit)inatextisalsonecessaryformanydataminingtasks.Ithasalsobeendemonstratedthatthetemporalin-formationinclinicalnarrativescanbeusefullyminedtoprovideinformationforsomehigher-leveltempo-ralreasoning(Zhaoetal.,2005).Robusttemporalunderstandingofsuchnarratives,cependant,hasbeendifficulttoachieve,duetothecomplexityofdeter-miningtemporalrelationsamongevents,thediver-sityoftemporalexpressions,andtheinteractionwithbroadercomputationallinguisticissues.RecentworkonElectronicHealthRecords(EHRs)pointstonewwaystoexploitandminetheinforma-tioncontainedtherein(Savovaetal.,2009;Robertsetal.,2009;Zhengetal.,2011;Turchinetal.,2009).Wetargettwomainusecasesforextracteddata.First,wehopetoenableinteractivedisplaysandsummariesofthepatient’srecordstothephysicianatthetimeofvisit,makingacomprehensivereviewofthepatient’shistorybothfasterandlesspronetooversights.Sec-ond,wehopetoenabletemporally-awaresecondaryresearchacrosslargedatabasesofmedicalrecords(e.g.,“Whatpercentageofpatientswhoundergopro-cedureXdevelopside-effectYwithinZmonths?»).Bothoftheseapplicationsrequiretheextractionoftimeanddateassociationsforcriticaleventsandtherelativeorderingofeventsduringthepatient’speriodofcare,allfromthevariousrecordswhichmakeupapatient’sEHR.Althoughwehavethesetwospecificapplicationsinmind,theschemawehavedevelopedisgeneralizableandcouldpotentiallybeembeddedinawidevarietyofbiomedicalusecases.NarrativetextsinEHRsaretemporallyrichdoc-umentsthatfrequentlycontainassertionsaboutthetimingofmedicalevents,suchasvisits,laboratoryvalues,symptoms,signes,diagnoses,andprocedures(Bramsenetal.,2006;Hripcsaketal.,2009;Zhouetal.,2008).Temporalrepresentationandreason-inginthemedicalrecordaredifficultdueto:(1)thediversityoftimeexpressions;(2)thecomplexityofdeterminingtemporalrelationsamongevents(whichareoftenlefttoinference);(3)thedifficultyofhan-dlingthetemporalgranularityofanevent;et(4)
je
D
o
w
n
o
un
d
e
d
F
r
o
m
h
t
t
p
:
/
/
d
je
r
e
c
t
.
m
je
t
.
e
d
toi
/
t
un
c
je
/
je
un
r
t
je
c
e
–
p
d
F
/
d
o
je
/
.
1
0
1
1
6
2
/
t
je
un
c
_
un
_
0
0
1
7
2
1
5
6
6
8
2
2
/
/
t
je
un
c
_
un
_
0
0
1
7
2
p
d
.
F
b
oui
g
toi
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
144
generalissuesinnaturallanguageprocessing(e.g.,ambiguity,anaphora,ellipsis,conjunction).Asare-sult,thesignalsusedforreconstructingatimelinecanbebothdomain-specificandcomplex,andareoftenleftimplicit,requiringsignificantdomainknowledgetoaccuratelydetectandinterpret.Inthispaper,wediscussthedemandsonaccuratelyannotatingsuchtemporalinformationinclinicalnotes.WedescribeanimplementationandextensionofISO-TimeML(Pustejovskyetal.,2010),devel-opedspecificallyfortheclinicaldomain,whichwerefertoasthe“THYMEGuidelinestoISO-TimeML”(“THYME-TimeML”),whereTHYMEstandsfor“TemporalHistoriesofYourMedicalEvents”.Asim-plifiedversionoftheseguidelinesformedthebasisforthe2012i2b2medical-domaintemporalrelationchallenge(Sunetal.,2013a).ThisisbeingdevelopedinthecontextoftheTHYMEproject,whosegoalistobothcreatero-bustgoldstandardsforsemanticinformationinclini-calnotes,aswellastodevelopstate-of-the-artalgo-rithmstotrainandtestonthisdataset.Derivingtimelinesfromnewstextrequiresthecon-creterealizationofcontext-dependentassumptionsabouttemporalintervals,orderingsandorganization,underlyingtheexplicitsignalsmarkedinthetext(PustejovskyandStubbs,2011).Derivingpatienthistorytimelinesfromclinicalnotesalsoinvolvesthesetypesofassumptions,buttherearespecialde-mandsimposedbythecharacteristicsoftheclinicalnarrative.Duetobothmedicalshorthandpracticesandgeneraldomainknowledge,manyevent-eventrelationsarenotsignaledinthetextatall,andrelyonasharedunderstandingandcommonconceptualmodelsoftheprogressionsofmedicalproceduresavailableonlytoreadersfamiliarwithlanguageuseinthemedicalcommunity.Identifyingtheseimplicitrelationsandtemporalpropertiesputsaheavyburdenontheannotationprocess.Assuch,intheTHYME-TimeMLguideline,considerableefforthasgoneintobothdescribingandproscribingtheannotationoftemporalorderingsthatareinferableonlythroughdomain-specifictemporalknowledge.AlthoughtheTHYMEguidelinesdescribeanum-berofdeparturesfromtheISO-TimeMLstandardforexpediencyandeaseofannotation,thispaperwillfocusonthosedifferencesspecificallymotivatedbytheneedsoftheclinicaldomain,andontheconse-quencesforsystemsbuilttoextracttemporaldatainboththeclinicalandgeneraldomain.2TheNatureofClinicalDocumentsIntheTHYMEcorpus,wehavebeenexamining1,254de-identified1notesfromalargehealthcarepractice(theMayoClinic),representingtwodistinctfieldswithinoncology:braincancer,andcoloncan-cer.Todate,wehaveprincipallyexaminedtwodif-ferentgeneraltypesofclinicalnarrativeinourEHRs:clinicalnotesandpathologyreports.Clinicalnotesarerecordsofphysicianinteractionswithapatient,andoftenincludemultiple,clearlydelineatedsectionsdetailingdifferentaspectsofthepatient’scareandpresentillness.Thesenotesarefairlygenericacrossinstitutionsandspecialities,andalthoughsometermsandinferencesmaybespecifictoaparticulartypeofpractice(suchasoncology),theyshareauniformstructureandpattern.The‘His-toryofPresentIllness’,forexample,summarizesthecourseofthepatient’schiefcomplaint,aswellastheinterventionsanddiagnosticswhichhavebeenthusfarattempted.Inothersections,thedoctormayout-linehercurrentplanforthepatient’streatment,thenlaterdescribethepatient’sspecificmedicalhistory,allergies,caredirectives,andsoforth.Mostcriticallyfortemporalreasoning,eachclin-icalnotereflectsasingletimeinthepatient’streat-menthistoryatwhichallofthedoctor’sstatementsareaccurate(theDOCTIME),andeachsectiontendstodescribeeventsofaparticulartimeframe.Forexample,‘HistoryofPresentillness’predominantlydescribeseventsoccuringbeforeDOCTIME,whereas‘Medications’providesasnapshotatDOCTIMEand‘OngoingCareOrders’discusseseventswhichhavenotyetoccurred.2Clinicalnotescontainrichtemporalinformationandbackground,movingfluidlyfrompriortreat-mentsandsymptomstopresentconditionstofutureinterventions.Theyarealsooftenrichwithhypo-theticalstatements(“ifthetumorrecurs,wecan…»),eachofwhichcanformitsownseparatetimeline.Byconstrast,pathologynotesarequitedifferent.Suchnotesaregeneratedbyamedicalpathologist1Althoughmostpatientinformationwasremoved,datesandtemporalinformationwerenotmodifiedaccordingtothisproject’sspecificdatauseagreement.2Onecomplicationisthepropensityofdoctorsandautomatedsystemstolaterupdatesectionsinanotewithoutchangingthetimestampormetadata.WehaveaddedaSECTIONTIMEtokeeptheseupdatedsectionsfromaffectingouroveralltimeline.
je
D
o
w
n
o
un
d
e
d
F
r
o
m
h
t
t
p
:
/
/
d
je
r
e
c
t
.
m
je
t
.
e
d
toi
/
t
un
c
je
/
je
un
r
t
je
c
e
–
p
d
F
/
d
o
je
/
.
1
0
1
1
6
2
/
t
je
un
c
_
un
_
0
0
1
7
2
1
5
6
6
8
2
2
/
/
t
je
un
c
_
un
_
0
0
1
7
2
p
d
.
F
b
oui
g
toi
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
145
uponreceiptandanalysisofspecimens(rangingfromtissuesamplesfrombiopsytoexcisedportionsoftumorororgans).Pathologynotesprovidecrucialinformationtothepatient’sdoctorconfirmingthemalignancy(cancer)insamples,describingsurgi-calmargins(whichindicatewhetheratumorwascompletelyexcised),andclassifyingand‘staging’atumor,describingtheseverityandspreadofthecan-cer.Becausetheinformationinsuchnotespertainstosamplestakenatasinglemomentintime,theyaretemporallysparse,seldomreferringtoeventsbeforeoraftertheexaminationofthespecimen.However,theycontaincriticalinformationaboutthestateofthepatient’sillnessandaboutthecanceritself,andmustbeinterpretedtounderstandthehistoryofthepatient’sillness.Mostimportantly,inallEHRs,wemustcontendwiththeresultsofafundamentaltensioninmod-ernmedicalrecords:hyper-detailedrecordsprovideacrucialdefenseagainstmalpracticelitigation,butincludingsuchdetailtakesenormoustime,whichdoctorsseldomhave.Giventhatthesenotesarewrit-tenbyandformedicalprofessionals(whoformarelativelyinsularspeechcommunity),agreatmanynon-standardexpressions,abbreviations,andassump-tionsofsharedknowledgeareused,whicharesimul-taneouslyconciseanddetail-richforotherswhohavesimilarbackgrounds.Thesetime-savingdevicescanrangefromtempo-rallyloadedacronyms(e.g.,‘qid’,Latinforquaterindie,‘fourtimesdaily’),toassumedorderings(adiag-nostictestforadisorderisassumedtocomebeforetheprocedurewhichtreatsit),andeventocompletelyimpliciteventsandtemporaldetails.Forexample,considerthesentencein(1).(1)Colonoscopy3/12/10,nodulebiopsiesnegativeWemustunderstandthatduringthecolonoscopy,thedoctorobtainedbiopsiesofnodules,whichwerepackagedandsenttoapathologist,whoreviewedthemanddeterminedthemtobe‘negative’(non-cancerous).Insuchdocuments,wemustrecoverasmuchtem-poraldetailaspossible,eventhoughitmaybeex-pressedinawaywhichisnoteasilyunderstoodout-sideofthemedicalcommunity,letalonebylinguistsorautomatedsystems.Wemustalsobeawareofthelegalrelevanceofsomeevents(e.g.,“Wediscussedthepossiblesideeffects”),evenwhentheymaynotseemrelevanttothepatient’sactualcare.Finally,eachspecialtyandnotetypehasseparateconventions.Withincoloncancernotes,theAmer-icanJointCommitteeonCancer(AJCC)StagingCodes(e.g.,T4N1,indicatingthenatureofthetumor,lymphnodeandmetastasisinvolvement)aremetic-ulouslyrecorded,butarelargelyabsentinthebraincancernoteswhichmakeupthesecondcorpusinourproject.So,althoughclinicalnotessharemanysimilarities,annotatorswithoutsufficientdomainex-pertisemayrequireadditionaltrainingtoadapttotheinferencesandnuancesofanewclinicalsubdomain.3Interpreting‘Event’andTemporalExpressionsintheClinicalDomainMuchpriorworkhasbeendoneonstandardizingtheannotationofeventsandtemporalexpressionsintext.ThemostwidelyusedapproachistheISO-TimeMLspecification(Pustejovskyetal.,2010),anISOstandardthatprovidesacommonframeworkforannotatingandanalyzingtime,events,andeventrela-tions.AsdefinedbyISO-TimeML,anEVENTreferstoanythingthatcanbesaid“toobtainorholdtrue,tohappenortooccur”.Thisisabroadnotionofevent,consistentwithBach’suseoftheterm“eventuality”(Bach,1986)aswellasthenotionoffluentsinAI(McCarthy,2002).BecausethegoalsoftheTHYMEprojectinvolveautomaticallyidentifyingtheclinicaltimelineforapatientfromclincalrecords,thescopeofwhatshouldbeadmittedintothedomainofeventsisinter-pretedmorebroadlythaninISO-TimeML3.WithintheTHYME-TimeMLguideline,anEVENTisany-thingrelevanttotheclinicaltimeline,i.e.,anythingthatwouldshowuponadetailedtimelineofthepa-tient’scareorlife.Thebestsingle-wordsyntacticheadfortheEVENTisthenusedasitsspan.Forexample,adiagnosiswouldcertainlyappearonsuchatimeline,aswouldatumor,illness,orprocedure.Ontheotherhand,entitiesthatpersistthroughouttherelevanttemporalperiodoftheclinicaltimeline(endurantsinontologicalcircles)wouldnotbecon-sideredasevent-like.Thisincludesthepatient,otherhumansmentioned(thepatient’smother-in-laworthedoctor),organizations(theemergencyroom),non-anatomicalobjects(thepatient’scar),orindi-vidualpartsofthepatient’sanatomy(anarmisnotanEVENTunlessmissingorotherwisenotable).Tomeetourexplicitgoals,theTHYME-TimeMLguidelineintroducestwoadditionallevelsofinterpre-3Ouruseoftheterm‘EVENT’correspondswiththelessspecificISO-TimeMLterm‘Eventuality’
je
D
o
w
n
o
un
d
e
d
F
r
o
m
h
t
t
p
:
/
/
d
je
r
e
c
t
.
m
je
t
.
e
d
toi
/
t
un
c
je
/
je
un
r
t
je
c
e
–
p
d
F
/
d
o
je
/
.
1
0
1
1
6
2
/
t
je
un
c
_
un
_
0
0
1
7
2
1
5
6
6
8
2
2
/
/
t
je
un
c
_
un
_
0
0
1
7
2
p
d
.
F
b
oui
g
toi
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
146
tationbeyondthatspecifiedbyISO-TimeML:(je)awell-definedtask;et(ii)aclearlyidentifieddomain.Byfocusingonthecreationofaclinicaltimelinefromclinicalnarrative,theguidelineimposescon-straintsthatcannotbeassumedforabroadlydefinedanddomainindependentannotationschema.SomeEVENTsannotatedunderourguidelineareconsideredmeaningfulandeventivemostlybyvirtueofaspecificclinicalorlegalvalue.Forexample,AJCCStagingCodes(discussedinSection2)areeventiveonlyinthesenseofthecodebeingassignedtoatumoratagivenmomentinthepatient’scare.However,theyareofsuchcriticalimportanceandinformativevaluetodoctorsthatwehavechosentoannotatethemspecificallysothattheywillshowuponthepatient’stimelineinaclinicalsetting.Similarly,becauseoflegalpressurestoestablishin-formedconsentandpatientknowledgeofrisk,entireparagraphsofclinicalnotesarededicatedtodocu-mentingthedoctor’sdiscussionofrisks,plans,andalternativestrategies.Assuch,weannotateverbsofdiscussion(“Wetalkedabouttherisksofthisdrug”),consent(“Sheagreedwiththecurrentplan”),andcomprehension(“Mrs.Larsenrepeatedthepotentialsideeffectsbacktome”),eventhoughtheyaremorerelevanttolegaldefensethanmedicaltreatment.Itisalsobecauseofthisgroundinginclinicallan-guagethatentitiesandothernon-eventsareofteninterpretedintermsoftheirassociatedeventiveprop-erties.Therearetwomajortypesforwhichthisisasignificantshiftinsemanticinterpretation:(2)aMedicationasEvent:Orders:Lariamtwicedaily.bDisorderasEvent:Tumoroftheleftlung.Inboththesecases,entitieswhicharenottypicallymarkedaseventsareidentifiedassuch,becausetheycontributesignificantinformationtotheclinicaltime-linebeingconstructed.In(2un),forexample,theTIMEX3“twicedaily”isinterpretedasscopingovertheeventualityofthepatienttakingthemedication,nottheprescriptionevent.Insentence(2b),the“tu-mor”isinterpretedasastativeeventualityofthepatienthavingatumorlocatedwithinananatomicalregion,ratherthananentitywithinanentity.Withinthemedicaldomain,theseeventiveinter-pretationsofmedications,growthsandstatuscodesareunambiguousandconsistent.Doctorsinclini-calnotes(unlikeinbiomedicalresearchtexts)donotdiscussmedicationswithoutanassociated(im-plicit)administeringEVENT(thoughsomementionsmaybehypothetical,genericornegated).De la même manière,mentionsofsymptomsordisordersreflectoccur-rencesinapatient’slife,ratherthanabstractentities.Withtheseinterpretationsinmind,wecansafelyin-fer,forinstance,thatallUMLS(UnifiedMedicalLanguageSystem,(Bodenreider,2004))entitiesofthetypesDisorder,Chemical/Drug,ProcedureandSign/SymptomwillbeEVENTs.Ingeneral,inthemedicaldomain,itisessentialtoread“betweenthelines”oftheshorthandexpressionsusedbythedoctors,andrecognizeimpliciteventsthatarebeingreferredtobyspecificanatomicalsitesormedications.4ModificationstoISO-TimeMLfortheClinicalDomainOverall,wehavefoundthatthespecificationrequiredfortemporalannotationintheclinicaldomaindoesnotrequiresubstantialmodificationfromexistingspecificationsforthegeneraldomain.Theclinicaldomainincludesnoshortageofinferences,short-hands,andunusualuseoflanguage,butthestructureoftheunderlyingtimelineisnotunique.Asaresultofthis,wehavebeenabletoadoptmostoftheframeworkfromISO-TimeML,adaptingtheguidelineswhereneeded,aswellasreframingthefocusofwhatgetsannotated.Thisisreflectedinacomprehensiveguideline,incorporatingthespecificpatternsandusesofeventsandtemporalexpressionsasseeninclinicaldata.Thisapproachallowstheresultingannotationstobeinteroperablewithexist-ingsolutions,whilestillaccommodatingthemajordifferencesinthenatureofthetexts.Ourguide-lines,aswellastheannotateddata,areavailableathttp://thyme.healthnlp.org4OurextensionsoftheISO-TimeMLspecificationtotheclinicaldomainareintendedtoaddressspecificconstructions,meanings,andphenomenainmedicaltexts.OurschemadiffersfromISO-TimeMLinafewnotableways.EVENTPropertiesWehavebothsimplifiedtheISO-TimeMLcodingofEVENTs,andextendedittomeettheneedsoftheclinicaldomainandthespecificlanguagegoalsoftheclinicalnarrative.4Accesstothecorpuswillrequireadatauseagreement.Moreinformationaboutthisprocessisavailablefromthecorpuswebsite.
je
D
o
w
n
o
un
d
e
d
F
r
o
m
h
t
t
p
:
/
/
d
je
r
e
c
t
.
m
je
t
.
e
d
toi
/
t
un
c
je
/
je
un
r
t
je
c
e
–
p
d
F
/
d
o
je
/
.
1
0
1
1
6
2
/
t
je
un
c
_
un
_
0
0
1
7
2
1
5
6
6
8
2
2
/
/
t
je
un
c
_
un
_
0
0
1
7
2
p
d
.
F
b
oui
g
toi
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
147
Consider,forexample,howmodalsubordinationishandledinISO-TimeML.Thisinvolvesthesemanticcharacterizationofaneventas“likely”,“possible”,oraspresentedbyobservation,evidence,orhearsay.AlloftheseareaccountedforcompositionallyinISO-TimeMLwithintheSLINK(SubordinatingLink)relation(Pustejovskyetal.,2005).Whileaccept-ingISO-TimeML’sdefinitionofeventmodality,wehavesimplifiedtheannotationtaskwithinthecur-rentguideline,sothatEVENTsnowcarryattributesfor“contextualmodality”,“contextualaspect”and“permanence”.ContextualmodalityallowsthevaluesACTUAL,HYPOTHETICAL,HEDGED,andGENERIC.ACTUALcoversEVENTswhichhaveactuallyhappened,e.g.,“We’venotedatumor”.HYPOTHETICALcoverscon-ditionalsandpossibilities,e.g.,“Ifshedevelopsatumor”.HEDGEDisforsituationswheredoctorsprofferadiagnosis,butdosocautiously,toavoidlegalliabilityforanincorrectdiagnosisorforover-lookingacorrectone.Forexample:(3)a.ThesignalintheMRIisnotinconsistentwithatumorinthespleen.b.Therashappearstobemeasles,awaitingantibodytesttoconfirm.TheseHEDGEDEVENTsaremorerealthanahypo-theticaldiagnosis,andlikelymeritinclusiononatimelineaspartofthediagnostichistory,butmustnotbeconflatedwithconfirmedfact.These(andotherformsofuncertaintyinthemedicaldomain)arediscussedextensivelyin(Vinczeetal.,2008).Incontrast,GENERICEVENTsdonotrefertothepa-tient’sillnessortreatment,butinsteaddiscussillnessortreatmentingeneral(ofteninthepatient’sspecificdemographic).Forexample:(4)Inotherpatientswithoutsignificantcomor-biditythatcantolerateadjuvantchemother-apy,thereisabenefittosystemicadjuvantchemotherapy.Thesesectionswouldbetrueifpastedintoanypa-tient’snote,andareoftenidenticalchunksoftextrepeatedlyusedtojustifyacourseofactionortreat-mentaswellastodefendagainstliability.ContextualAspect(todistinguishfromgrammati-calaspect),allowstheclinically-necessarycategory,INTERMITTENT.Thisservestodistinguishintermit-tentEVENTs(suchasvomitingorseizures)fromconstant,morestativeEVENTs(suchasfeverorsore-ness).Forexample,theboldedEVENTin(5un)wouldbemarkedasINTERMITTENT,whilethatin(5b)wouldnot:(5)aShehasbeenvomitingsinceJune.bShehashadswellingsinceJune.Inthefirstcase,weassumethathervomitinghasbeenintermittent,i.e.,therewereseveralpointssinceJuneinwhichshewasnotactivelyvomiting.Inthesecondcase,unlessmadeotherwiseexplicit(“shehashadoccasionalswelling”),weassumethatswellingwasaconstantstate.ThispropertyisalsousedwhenaparticularinstanceofanEVENTisintermittent,eventhoughitgenerallywouldnotbe:(6)Sincestartinghernewregime,shehashadocca-sionalboutsoffever,butisfeelingmuchbetter.Thepermanenceattributehastwovalues,FINITEandPERMANENT.Permanenceisapropertyofdis-easesthemselves,roughlycorrespondingtothemed-icalconceptof“chronic”vs.“acute”disease,whichmarkswhetheradiseaseispersistentfollowingdiag-nosis.Forexample,un(currently)uncurablediseaselikeMultipleSclerosiswouldbeclassedasPERMA-NENT,andthus,oncementionedinapatient’snote,willbeassumedtopersistthroughtheendofthepatient’stimeline.ThisiscomparedwithFINITEdisorderslike“Influenza”or“fever”,lequel,ifnotmentionedinsubsequentnotes,shouldbeconsideredcuredandnolongerbelongsonthepatient’stime-line.Becauseitrequiresdomain-specificknowledge,althoughpresentinthespecification,Permanenceisnotcurrentlyannotated.However,annotatorsaretrainedonthebasicideaandtoldaboutsubsequentaxiomaticassignment.Theadditionofthispropertytoourschemaisdesignedtorelieveannotatorsofanyfeelingofobligationtoexpressthisinferredinforma-tioninsomeotherway.TIMEX3TypesTemporalexpressions(TIMEX3s)intheclinicaldomainfunctionthesameasinthegen-erallinguisticcommunity,withtwonotableexcep-tions.ISO-TimeMLSETs(statementsoffrequency)occurquitefrequentlyinthemedicaldomain,par-ticularlywithregardtomedicationsandtreatments.Medicationsectionswithinnotesoftencontainlonglistsofmedications,eachwithaparticularassociatedset(“Claritin30mgtwicedaily”),andfurthertempo-ralspecificationisnotuncommon(e.g.,“threetimesperdayatmeals”,“onceaweekatbedtime”).ThesecondmajorchangeforthemedicaldomainisanewtypeofTIMEX3whichwecallPREPOS-TEXP.Thiscoverstemporallycomplextermslike
je
D
o
w
n
o
un
d
e
d
F
r
o
m
h
t
t
p
:
/
/
d
je
r
e
c
t
.
m
je
t
.
e
d
toi
/
t
un
c
je
/
je
un
r
t
je
c
e
–
p
d
F
/
d
o
je
/
.
1
0
1
1
6
2
/
t
je
un
c
_
un
_
0
0
1
7
2
1
5
6
6
8
2
2
/
/
t
je
un
c
_
un
_
0
0
1
7
2
p
d
.
F
b
oui
g
toi
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
148
“preoperative”,“postoperative”,and“intraoperative”.Thesetemporalexpressionsdesignateaspanoftimebordered,usuallyonlyononeside,bytheincorpo-ratedevent(anoperation,inthepreviousEVENTs).Inmanycases,thereferentisclear:(7)Sheunderwenthemicolectomylastweek,andhadsomepostoperativebleeding.Hereweunderstandthat“postoperative”refersto“theperiodoftimefollowingthehemicolectomy”.Inthesecases,thePREPOSTEXPmakesexplicitatempo-rallinkbetweenthebleedingandthehemicolectomy.Inothercases,noclearreferentispresent:(8)Patientshowssomepost-procedurescarring.Inthesesituations,wherenoprocedureismentioned(orthereferenceisneverexplicitlyresolved),wetreatthePREPOSTEXPasanarrativecontainer(seeSection5),coveringthespanoftimefollowingtheunnamedprocedure.Finally,itisworthnotingthattheprocessofnor-malizingthoseTIMEX3sissignificantlymorecom-plexrelativetothegeneraldomain,becausemanytemporalexpressionsareanchorednottodatesortimes,buttootherEVENTs(whosedatesareoftennotmentionedornotknownbythephysician).Aswemovetowardsacompletesystem,weareworkingtoexpandtheISO-TimeMLsystemforTIMEX3nor-malizationtoallowsomevaluetobeassignedtoaphraselike“inthemonthsafterherhemicolectomy”whennoreferentdateispresent.ISO-TimeML,indiscussionwithISOTC37SC4,planstoreferencetosuchTIMEX3sinafuturereleaseofthestandard.5TemporalOrderingandNarrativeContainersThesemanticcontentandinformationalimpactofatimelineisencodedintheorderingrelationsthatareidentifiedbetweenthetemporalandeventexpres-sionspresentinclinicalnotes.ISO-TimeMLspeci-fiesthestandardthirteen“Allenrelations”fromtheintervalcalculus(Allen,1983),whichitreferstoasTLINKvalues.Forunguided,general-purposeannota-tion,thenumberofrelationsthatcouldbeannotatedgrowsquadraticallywiththenumberofeventsandtimes,andthetaskquicklybecomesunmanageable.Thereare,cependant,strategiesthatwecanadopttomakethislabelingtaskmoretractable.Temporalorderingrelationsintextareofthreekinds:1.Relationsbetweentwoevents2.Relationsbetweentwotimes3.Relationsbetweenatimeandanevent.ISO-TimeML,asaformalspecificationofthetem-poralinformationconveyedinlanguage,makesnodistinctionbetweentheseorderingtypes.Humans,cependant,domakedistinctions,basedonlocaltempo-ralmarkersandthediscourserelationsestablishedinanarrative(Miltsakakietal.,2004;Poesio,2004).Becauseofthedifficultyofhumanscapturingev-eryrelationshippresentinthenote(andthedisagree-mentwhichariseswhenannotatorsattempttodoso),itisvitalthattheannotationguidelinesdescribeanapproachthatreducesthenumberofrelationsthatmustbeconsidered,butstillresultsinmaximallyin-formativetemporallinks.Wehavefoundthatmanyoftheweaknessesinpriorannotationapproachesstemfrominteractionbetweentwocompetinggoals:•Theguidelineshouldspecifycertaintypesofan-notationsthatshouldbeperformed;•Theguidelineshouldnotforceannotationstobeperformedwhentheyneednotbe.Failinginthefirstgoalwillresultinunder-annotationandtheneglectofrelationswhichprovidenecessaryinformationforinferenceandanalysis.Failureinthesecondgoalresultsinover-annotation,creatingcom-plexwebsoftemporalrelationswhichyieldmostlyinferableinformation,butwhichcomplicateannota-tionandadjudicationconsiderably.Ourmethodofaddressingbothgoalsintempo-ralrelationsannotationisthatofthenarrativecon-tainer,discussedinPustejovskyandStubbs(2011).AnarrativecontainercanbethoughtofasatemporalbucketintowhichanEVENTorseriesofEVENTsmayfall,oranaturalclusterofEVENTsaroundagiventimeorsituation.Thesenarrativecontainersareoftenrepresented(or“anchored”)bydatesorothertemporalexpressions(withinwhichavarietyofdifferentEVENTsoccur),althoughtheycanalsobeanchoredtomoreabstractconcepts(“recovery”whichmightinvolveavarietyofEVENTs)orevendurativeEVENTs(manyotherEVENTscanoccurdur-ingasurgery).RatherthanmarkingeverypossibleTLINKbetweeneachEVENT,weinsteadtrytolinkallEVENTstotheirnarrativecontainers,andthenlinkthosecontainerssothatthecontainedEVENTscanbelinkedbyinference.First,annotatorsassigneacheventtooneoffourbroadnarrativecontainers:beforetheDOCTIME,be-foreandoverlappingtheDOCTIME,justoverlappingtheDOCTIMEoraftertheDOCTIME.Thisnarrative
je
D
o
w
n
o
un
d
e
d
F
r
o
m
h
t
t
p
:
/
/
d
je
r
e
c
t
.
m
je
t
.
e
d
toi
/
t
un
c
je
/
je
un
r
t
je
c
e
–
p
d
F
/
d
o
je
/
.
1
0
1
1
6
2
/
t
je
un
c
_
un
_
0
0
1
7
2
1
5
6
6
8
2
2
/
/
t
je
un
c
_
un
_
0
0
1
7
2
p
d
.
F
b
oui
g
toi
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
149
containerisidentifiedbytheEVENTattributeDoc-TimeRel.AftertheassignmentofDocTimeRel,theremainderofthenarrativecontainerrelationsmustbespecifiedusingtemporallinks(TLINKs).TherearefivedifferenttemporalrelationsusedforsuchTLINKs:BEFORE,OVERLAP,BEGINS-ON,ENDS-ONandCONTAINS5.Duetoournarrativecontainerap-proach,CONTAINSisthemostfrequentrelationbyalargemargin.EVENTsservingasnarrativecontaineranchorsarenottaggedascontainersper-se.Instead,annotatorsusethenarrativecontainerideatohelpthemvisu-alizethetemporalrelationswithinadocument,andthenmakeaseriesofCONTAINSTLINKannotationswhichestablishEVENTsandTIMEX3sasanchors,andspecifytheircontents.Iftheannotatorsdotheirjobscorrectly,properlyimplementingDocTimeRelandcreatingaccurateTLINKs,agoodunderstandingofthenarrativecontainerspresentinadocumentwillnaturallyemergefromtheannotatedtext.Themajoradvantageintroducedwithnarrativecontainersisthis:anarrativeeventisplacedwithinaboundingtemporalintervalwhichisexplicitlymen-tionedinthetext.ThisallowsEVENTswithinsep-aratecontainerstobelinkedbypost-hocinference,temporalreasoning,anddomainknowledge,ratherthanbyexplicit(andtime-consuming)one-by-onetemporalrelationsannotation.Asecondaryadvantageisthatthisapproachworksnicelywiththegeneralstructureofstory-tellinginboththegeneralandclinicaldomains,andprovidesacompellingandusefulmetaphorforinterpretingtime-lines.Often,especiallyinclinicalhistories,doctorswillclusterdiscussionsofsymptoms,interventionsanddiagnosesaroundagivendate(e.g.awholepara-graphstarting“June2009:»),aspecifichospitaliza-tion(“DuringherJanuarystayatMercy”),oragivenillnessortreatment(“WhilesheunderwentChemo”).EvenwhenspecificEVENTsarenotexplicitlyor-deredwithinacluster(oftenbecausetheordercanbeeasilyinferredwithdomainknowledge),itisoftenquiteeasytoplacetheEVENTsintocontainers,andjustafewTLINKscanorderthecontainersrelativetooneanotherwithenoughdetailtocreateaclinicallyusefulunderstandingoftheoveralltimeline.Narrativecontainersalsoallowtheinferenceofre-lationsbetweensub-eventswithinnestedcontainers:5ThisisasubsetoftheISO-TimeMLTLINKtypes,excludingthoseseldomoccurringinmedicalrecords,like‘simultaneous’aswellasinverserelationslike‘during’or‘after’.(9)December19th:ThepatientunderwentanMRIandEKGaswellasemergencysurgery.Dur-ingthesurgery,thepatientexperiencedmildtachycardia,andshealsobledsignificantlyduringtheinitialincision.1.December19thCONTAINSMRI2.December19thCONTAINSEKG3.December19thCONTAINSsurgerya.surgeryCONTAINStachycardiab.surgeryCONTAINSincisionc.incisionCONTAINSbledThroughourcontainernesting,wecanautomaticallyinferthat‘bled’occurredonDecember19th(because‘19th’CONTAINS‘surgery’whichCONTAINS‘inci-sion’whichCONTAINS‘bled’).ThisalsoallowsthecaptureofEVENT/sub-eventrelations,andtherapidexpressionofcomplextemporalinteractions.6Explicitvs.InferableAnnotationGivenaspecificationlanguage,thereareessentiallytwowaysofintroducingtheelementsintothedocu-ment(datasource)beingannotated:6•Manualannotation:Elementsareintroducedintothedocumentdirectlybythehumanannotatorfol-lowingtheguideline.•Automatic(inferred)annotation:Elementsarecre-atedbyapplyinganautomatedprocedurethatin-troducesnewelementsthatarederivablefromthehumanannotations.Assuch,thereisacomplexinteractionbetweenspec-ificationandguideline,andwefocusonhowtheclinicalannotationtaskhashelpedshapeandrefinetheannotationguidelines.Itisimportanttonotethatanannotationguidelinedoesnotnecessarilyforcethemarkupofcertainelementsinatext,eventhoughthespecificationlanguage(andtheeventualgoaloftheproject)mightrequirethoseannotationstoexist.Insomecases,theseaddedannotationsarederivedlogicallyfromhumanannotations.Explicitlymarkedtemporalrelationscanbeusedtoinferothersthatarenotmarkedbutexistimplicitlythroughclosure.Forinstance,givenEVENTsA,BandCandTLINKs‘ABEFOREB’and‘BBEFOREC’,theTLINK‘ABE-FOREC’canbeautomaticallyinferred.Repeatedlyapplyingsuchinferencerulesallowsallinferable6Weignoretheapplicationofautomatictechniques,suchasclassifierstrainedonexternaldatasets,asourfocushereisonthepreparationofthegoldstandardusedforsuchclassifiers.
je
D
o
w
n
o
un
d
e
d
F
r
o
m
h
t
t
p
:
/
/
d
je
r
e
c
t
.
m
je
t
.
e
d
toi
/
t
un
c
je
/
je
un
r
t
je
c
e
–
p
d
F
/
d
o
je
/
.
1
0
1
1
6
2
/
t
je
un
c
_
un
_
0
0
1
7
2
1
5
6
6
8
2
2
/
/
t
je
un
c
_
un
_
0
0
1
7
2
p
d
.
F
b
oui
g
toi
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
150
TLINKstobegenerated(Verhagen,2005).Wecanusethisideaofclosuretoshowourannotatorswhichannotationsneednotbemarkedexplicitly,savingtimeandeffort.Wehavealsoincorporatedtheseclo-surerulesintoourinter-annotatoragreement(IAA)calculationfortemporalrelations,describedfurtherinSection7.2.TheautomaticapplicationofrulesfollowingtheannotationofthetextisnotlimitedtothemarkingoflogicallyinferablerelationsorEVENTs.Intheclinicaldomain,thecombinationofwithin-groupsharedknowledgeandpressuretowardsconcisewrit-ingleadstoanumberofcommon,inferredrelations.Take,forexample,thesentence:(10)Jan2013:Colonoscopy,biopsies.Pathologyshowedadenocarcinoma,resectedatMercy.DiagnosisT3N1Adenocarcinoma.Inthissentence,onlytheCONTAINSrelationsbe-tween“Jan2013”andtheEVENTs(inbold)areexplicitlystated.However,basedontheknownprogression-of-careforcoloncancer,wecaninferthatthecolonoscopyoccursfirst,biopsiesoccurdur-ingthecolonoscopy,pathologyhappensafterwards,adiagnosis(ici,adenocarcinoma)isreturnedafterpathology,andresectionofthetumoroccursafterdiagnosis.ThepresenceoftheAJCCstaginginfor-mationinthefinalsentence(alongwiththeconfir-mationoftheadenocarcinomadiagnosis)impliesapost-surgicalpathologyexamoftheresectedspec-imen,astheAJCCstaginginformationcannotbedeterminedwithoutthisadditionalexamination.Theseinferencescomenaturallytodomainex-pertsbutarelargelyinaccessibletopeopleoutsidethemedicalcommunitywithoutconsiderableanno-tatortraining.Makingexplicitourunderstandingofthese“understoodorderings”iscrucial;althoughtheyarenotmarkedbyhumanannotatorsinourschema,theannotatorsoftenfounditinitiallyfrustratingtoleavethese(purelyinferential)relationsunstated.Al-thoughmanyofour(primarilylinguisticallytrained)annotatorslearnedtoseethesepatterns,wechosetoexcludethemfromthemanualtasksinceneweran-notatorswithvaryingdegreesofdomainknowledgemaystruggleifaskedtomanuallyannotatethem.Similarunspoken-but-understoodorderingsarefoundthroughouttheclinicaldomain.AsmentionedinSection3,bothPermanenceandContextualAs-pect:Intermittentarepropertiesofsymptomsanddis-easesthemselves,ratherthanofthepatient’sparticu-larsituation.Assuch,thesepropertiescouldeasilyAnnotationTypeRawCountEVENT15,769TIMEX31,426LINK7935Total25,130Table1:RawFrequencyofAnnotationTypesTLINKTypeRawCount%ofTLINKsCONTAINS5,11264.42%OVERLAP1,20515.19%BEFORE1,00412.65%BEGINS-ON4886.15%ENDS-ON1261.59%Total7,935100.00%Table2:RelativeFrequencyofTLINKtypesbeidentifiedandmarkedacrossamedicalontology,andthenbeautomaticallyassignedtoEVENTsrec-ognizedasspecificmedicalnamedentities.Finally,duetothepeculiaritiesofEHRsystems,someannotationsmustbedoneprogramatically.Ex-actdatesofpatientvisit(orofpathology/radiologyconsult)areoftenrecordedasmetadataontheEHRitself,ratherthanwithinthetext,makingthecanoni-calDOCTIME(ortimeofautomaticsectionmodifi-cations)difficulttoaccessinde-identifiedplaintextdata,buteasytofindautomatically.7ResultsWereportresultsontheannotationsfromthehere-releasedsubsetoftheTHYMEcoloncancercorpus,whichincludesclinicalnotesandpathologyreportsfor35patientsdiagnosedwithcoloncancerforatotalof107documents.EachnotewasannotatedbyapairofgraduateorundergraduatestudentsinLinguisticsattheUniversityofColorado,thenadju-dicatedbyadomainexpert.TheseclinicalnarrativesweresampledfromtheEHRsofamajorhealthcarecenter(theMayoClinic).Theyweredeidentifiedforallpatient-sensitiveinformation;cependant,originaldateswereretained.7.1DescriptiveStatisticsTable1presentstherawcountsforevents,temporalexpressionsandlinksintheadjudicatedgoldanno-tations.Table2presentsthenumberandpercentageofTLINKsbytypeintheadjudicatedrelationsgoldannotations.
je
D
o
w
n
o
un
d
e
d
F
r
o
m
h
t
t
p
:
/
/
d
je
r
e
c
t
.
m
je
t
.
e
d
toi
/
t
un
c
je
/
je
un
r
t
je
c
e
–
p
d
F
/
d
o
je
/
.
1
0
1
1
6
2
/
t
je
un
c
_
un
_
0
0
1
7
2
1
5
6
6
8
2
2
/
/
t
je
un
c
_
un
_
0
0
1
7
2
p
d
.
F
b
oui
g
toi
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
151
AnnotationTypeF1-ScoreAlphaEVENT0.80380.7899TIMEX30.80470.6705LINK:Participantsonly0.50120.4999LINK:Participants+type0.45060.4503LINK:CONTAINS0.56300.5626Table3:IAA(F1-ScoreandAlpha)byannotationtypeEVENTPropertyF1-ScoreAlphaDocTimeRel0.71890.6889Cont.Aspect0.99470.9930Cont.Modality0.95470.9420Table4:IAA(F1-ScoreandAlpha)forEVENTproperties7.2Inter-annotatorAgreementWereportinter-annotatoragreement(IAA)resultsontheTHYMEcorpus.Eachnotewasannotatedbytwoindependentannotators.Thefinalgoldstandardwasproducedafterdisagreementadjudicationbyathirdannotatorwasperformed.WecomputedtheIAAasF1-scoreandKrippen-dorff’sAlpha(Krippendorff,2012)byapplyingclo-sure,usingexplicitlymarkedtemporalrelationstoidentifyothersthatarenotmarkedbutexistimplicitly.InthecomputationoftheIAA,inferred-onlyTLINKsdonotcontributetothescore,matchedorunmatched.Forinstance,ifbothannotatorsmarkABEFOREBandBBEFOREC,topreventartificiallyinflatingtheagreementscore,theinferredABEFORECisignored.Likewise,ifoneannotatormarkedABEFOREBandBBEFORECandtheotherannotatordidnot,theinferredABEFORECisnotcounted.However,ifoneannotatordidexplicitlymarkABEFOREC,thenanequivalentinferredTLINKwouldbeusedtomatchit.EVENTandTIMEX3IAAwasgeneratedbasedonexactandoverlappingspans,respectively.TheseresultsarereportedinTable3.TheTHYMEcorpusalsodiffersfromISO-TimeMLintermsofEVENTproperties,withtheadditionofDocTimeRel,ContextualModalityandContextualAspect.IAAforthesepropertiesisinTable4.7.3BaselineSystemsTogetanideaofhowmuchworkwillbeneces-sarytoadaptexistingtemporalinformationextrac-tionsystemstotheclinicaldomain,wetookthefreelyavailableClearTK-TimeMLsystem(Bethard,2013),TempEval2013THYMECorpusPRF1PRF1TIMEX383.271.777.059.342.849.7EVENT81.476.478.878.923.936.6DocTimeRel—47.447.447.4LINK728.630.926.622.718.620.4EVENT-TIMEX3—32.360.742.1EVENT-EVENT—7.03.04.2Table5:PerformanceofClearTK-TimeMLmodels,asreportedintheTempEval2013competition,andasappliedtotheTHYMECorpusdevelopmentset.whichwasamongthetopperformingsystemsinTempEval2013(UzZamanetal.,2013),andeval-uateditsperformanceontheTHYMEcorpus.ClearTK-TimeMLusessupportvectormachineclassifierstrainedontheTempEval2013trainingdata,employingasmallsetoffeaturesincludingcharacterpatterns,tokens,stems,part-of-speechtags,nearbynodesintheconstituencytree,andasmalltimewordgazetteer.ForEVENTsandTIMEX3s,theClearTK-TimeMLsystemcouldbeapplieddi-rectlytotheTHYMEcorpus.ForDocTimeRels,therelationforanEVENTwastakenfromtheTLINKbetweenthatEVENTandthedocumentcreationtime,aftermappingINCLUDEStoOVERLAP.EVENTswithnosuchTLINKwereassumedtohaveaDoc-TimeRelofOVERLAP.Forothertemporalrelations,INCLUDESwasmappedtoCONTAINS.ResultsofthissystemonTempEval2013andtheTHYMEcorpusareshowninTable5.Fortimeex-pressions,performancewhenmovingtotheclinicaldatadegradesabout25%,fromF1of77.0to49.7.Forevents,thedegradationismuchlarger,about40%,from78.8to36.6,mostlikelybecauseofthelargenumberofclinicalsymptoms,diseases,disor-ders,etc.whichhaveneverbeenobservedbythesystemduringtraining.TemporalrelationsareabitmoredifficulttocomparebecauseTempEvallumpedDocTimeRelandothertemporalrelationstogetherandhadseveraldifferencesintheirevaluationmet-ric7.However,weatleastcanseethatperformanceoftheClearTK-TimeMLsystemontemporalrela-tionsislowonclinicaltext,achievingonlyF1of20.4.Theseresultssuggestthatclinicalnarrativesdo7TheTempEval2013evaluationmetricpenalizedsystemsforpartsofthetextthatwerenotexaminedbyannotators,anduseddifferentvariantsofclosure-basedprecisionandrecall.
je
D
o
w
n
o
un
d
e
d
F
r
o
m
h
t
t
p
:
/
/
d
je
r
e
c
t
.
m
je
t
.
e
d
toi
/
t
un
c
je
/
je
un
r
t
je
c
e
–
p
d
F
/
d
o
je
/
.
1
0
1
1
6
2
/
t
je
un
c
_
un
_
0
0
1
7
2
1
5
6
6
8
2
2
/
/
t
je
un
c
_
un
_
0
0
1
7
2
p
d
.
F
b
oui
g
toi
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
152
indeedpresentnewchallengesfortemporalinforma-tionextractionsystems,andthathavingaccesstodomainspecifictrainingdatawillbecrucialforac-curateextractionintheclinicaldomain.Atthesametime,itisencouragingthatwewereabletoapplyexistingISO-TimeML-basedsystemstoourcorpus,despitetheseveralextensionstoISO-TimeMLthatwerenecessaryforclinicalnarratives.8DiscussionCONTAINSplaysalargeroleintheTHYMEcor-pus,representing66%ofTLINKannotationsmade,comparedwithonly14.6%forOVERLAP,thesecondmostfrequenttype.WealsoseethatBEFORElinksarerelativelylesscommonthanOVERLAPandCON-TAINS,illustratingthatmuchofthetemporalorderingonthetimelineisaccomplishedbyusingmanyver-ticallinks(CONTAINS,OVERLAP)tobuildcontain-ers,andfewhorizontallinks(BEFORE,BEGINS-ON,ENDS-ON)toorderthem.IAAonEVENTsandTemporalExpressionsisstrong,althoughdifferentiatingimplicitEVENTs(whichshouldnotbemarked)fromexplicit,mark-ableEVENTsremainsoneofthebiggestsourcesofdisagreement.Whencomparedtothedatafromthe2012i2b2challenge(Sunetal.,2013b),ourIAAfiguresarequitesimilar.Evenwithourmorecom-plexschema,weachievedanF1-scoreof0.8038forEVENTs(comparedtothei2b2scoreof0.87forpar-tialmatch).ForTIMEX3s,ourF1-scorewas0.8047,comparedtoanF1-scoreof0.89fori2b2.TLINKingmedicalEVENTsremainsaverydiffi-culttask.Byusingournarrativecontainerapproachtoconstrainthenumberofnecessaryannotationsandbyeliminatingoften-confusinginverserelations(like‘after’and‘during’)(neitherofwhichweredoneforthei2b2data),wewereabletosignificantlyimproveonthei2b2TLINKspanagreementF1-scoreof0.39,achievinganagreementscoreof0.5012forallLINKsacrossourcorpus.Themajorityofremainingan-notatordisagreementcomesfromdifferentopinionsaboutwhetheranytwoEVENTsrequireanexplicitTLINKbetweenthemoraninferredone,ratherthanwhattypeofTLINKitwouldbe(e.g.BEFOREvs.CONTAINS).Althoughourresultsarestillsignifi-cantlyhigherthantheresultsreportedfori2b2,andinlinewithpreviouslyreportedgeneralnewsfigures,wearenotsatisfied.ImprovingIAAisanimportantgoalforfuturework,andwithfurthertraining,speci-fication,experience,andstandardization,wehopetoclarifycontextsforexplicitTLINKS.News-trainedtemporalinformationextractionsys-temsseeasignificantdropinperformancewhenap-pliedtotheclinicaltextsoftheTHYMEcorpus.ButasthecorpusisanextensionofISO-TimeML,futureworkwillbeabletotrainISO-TimeMLcompliantsystemsontheannotationsoftheTHYMEcorpustoreduceoreliminatethisperformancegap.Someapplicationsthatourworkmayenablein-clude(1)betterunderstandingofeventsemantics,suchaswhetheradiseaseischronicoracuteanditsusualnaturalhistory,(2)typicaleventdurationfortheseevents,(3)theinteractionofgeneralanddomain-specificeventsandtheirimportanceinthefi-naltimeline,et,moregenerally,(4)theimportanceofroughtemporalityandnarrativecontainersasasteptowardsfiner-grainedtimelines.Wehaveseveralavenuesofongoingandfuturework.First,weareworkingtodemonstratetheutilityoftheTHYMEcorpusfortrainingmachinelearningmodels.WehavedesignedsupportvectormachinemodelswithconstituencytreekernelsthatwereabletoreachanF1-scoreof0.737onanEVENT-TIMEX3narrativecontaineridentificationtask(Milleretal.,2013),andweareworkingontrainingmodelstoidentifyevents,timesandtheremainingtypesoftemporalrelations.Second,asperourmotivatingusecases,weareworkingtointegratethisannotationdatawithtimelinevisualizationtoolsandtousetheseannotationsinquality-of-careresearch.Forexample,weareusingtemporalreasoningbuiltonthisworktoinvestigatethelivertoxicityofmethotrexateacrossalargecorpusofEHRs(Linetal.,underreview)].Enfin,weplantoexploretheapplicationofournotionofanevent(anythingthatshouldbevisibleonadomain-appropriatetimeline)tootherdomains.Itshouldtransfernaturallytoclinicalnotesaboutother(non-cancer)conditions,andeventoothertypesofclinicalnotes,ascertainbasiceventsshouldalwaysbeincludedinapatient’stimeline.Applyingournotionofeventtomoredistantdomains,suchaslegalopinions,wouldrequirefirstidentifyingaconsensuswithinthedomainaboutwhicheventsmustappearonatimeline.9ConclusionMuchoftheinformationinclinicalnotescriticaltotheconstructionofadetailedtimelineisleftimplicitbytheconciseshorthandusedbydoctors.Manyeventsarereferredtoonlybyatermsuchas“tu-
je
D
o
w
n
o
un
d
e
d
F
r
o
m
h
t
t
p
:
/
/
d
je
r
e
c
t
.
m
je
t
.
e
d
toi
/
t
un
c
je
/
je
un
r
t
je
c
e
–
p
d
F
/
d
o
je
/
.
1
0
1
1
6
2
/
t
je
un
c
_
un
_
0
0
1
7
2
1
5
6
6
8
2
2
/
/
t
je
un
c
_
un
_
0
0
1
7
2
p
d
.
F
b
oui
g
toi
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
153
mor”,whilepropertiesoftheeventitself,suchas“intermittent”,maynotbespecified.Inaddition,theorderingofeventsonatimelineisoftenlefttothereadertoinfer,basedondomain-specificknowledge.Itisincumbentupontheannotationguidelinetoin-dicatethatonlyinformativeeventorderingsshouldbeannotated,whileleavingdomain-specificorder-ingstopost-annotationinference.ThisdocumenthasdetailedourapproachtoadaptingtheexistingISO-TimeMLstandardtothisrecoveryofimplicitinformation,anddefiningguidelinesthatsupportan-notationwithinthiscomplexdomain.Ourguide-lines,aswellastheannotateddata,areavailableathttp://thyme.healthnlp.org,andthefullcorpushasbeenproposedforuseinaSemEval2015sharedtask.AcknowledgmentsTheprojectdescribedissupportedbyGrantNum-berR01LM010090andU54LM008748fromtheNa-tionalLibraryOfMedicine.ThecontentissolelytheresponsibilityoftheauthorsanddoesnotnecessarilyrepresenttheofficialviewsoftheNationalLibraryOfMedicineortheNationalInstitutesofHealth.WewouldalsoliketothankDr.PietC.deGroenandDr.BradEricksonattheMayoClinic,aswellasDr.WilliamF.StylerIII,fortheircontributionstotheschemaandtoourunderstandingoftheintricaciesofclinicallanguage.ReferencesJamesFAllen.1983.Maintainingknowledgeabouttemporalintervals.CommunicationsoftheACM,26(11):832–843.EmmonBach.1986.Thealgebraofevents.Linguisticsandphilosophy,9(1):5–16.StevenBethard.2013.Cleartk-timeml:Aminimalistap-proachtotempeval2013.InSecondJointConferenceonLexicalandComputationalSemantics(*SEM),Vol-ume2:ProceedingsoftheSeventhInternationalWork-shoponSemanticEvaluation(SemEval2013),pages10–14,Atlanta,Georgia,Etats-Unis,June.AssociationforComputationalLinguistics.OlivierBodenreider.2004.TheUnifiedMedicalLanguageSystem(UMLS):integratingbiomedicalterminology.Nucleicacidsresearch,32(Databaseissue):D267–D270,January.PhilipBramsen,PawanDeshpande,YoongKeokLee,andReginaBarzilay.2006.Findingtemporalorderindischargesummaries.InAMIAAnnualSymposiumProceedings,volume2006,page81.AmericanMedicalInformaticsAssociation.CarloCombi,YuvalShahar,etal.1997.Temporalreason-ingandtemporaldatamaintenanceinmedicine:issuesandchallenges.Computersinbiologyandmedicine,27(5):353–368.RobertHDolin.1995.Modelingthetemporalcomplex-itiesofsymptoms.JournaloftheAmericanMedicalInformaticsAssociation,2(5):323–331.GeorgeHripcsak,NicholasDSoulakis,LiLi,FrancesPMorrison,AlbertMLai,CarolFriedman,NeilSCal-man,andFarzadMostashari.2009.Syndromicsurveil-lanceusingambulatoryelectronichealthrecords.Jour-naloftheAmericanMedicalInformaticsAssociation,16(3):354–361.AnnKIrvine,StephanieWHaas,andTessaSullivan.2008.Tn-ties:Asystemforextractingtemporalinfor-mationfromemergencydepartmenttriagenotes.InAMIAAnnualSymposiumproceedings,volume2008,page328.AmericanMedicalInformaticsAssociation.ElpidaTKeravnou.1997.Temporalabstractionofmed-icaldata:Derivingperiodicity.InIntelligentDataAnalysisinMedicineandPharmacology,pages61–79.Springer.KlausH.Krippendorff.2012.ContentAnalysis:AnIntroductiontoItsMethodology.SAGEPublications,Inc,thirdeditionedition,April.ChenLin,ElizabethKarlson,DmitriyDligach,Mon-icaRamirez,TimothyMiller,HuanMo,NatalieBraggs,AndrewCagan,JoshuaDenny,andGuer-gana.Savova.underreview.Automaticidentificationofmethotrexade-inducedlivertoxicityinrheumatoidarthritispatientsfromtheelectronicmedicalrecords.JournaloftheMedicalInformaticsAssociation.JohnMcCarthy.2002.Actionsandothereventsinsit-uationcalculus.InProceedingsoftheInternationalconferenceonPrinciplesofKnowledgeRepresentationandReasoning,pages615–628.MorganKaufmannPublishers;1998.St´ephaneMMeystre,GuerganaKSavova,KarinCKipper-Schuler,JohnFHurdle,etal.2008.Extractinginfor-mationfromtextualdocumentsintheelectronichealthrecord:areviewofrecentresearch.YearbMedInform,35:128–44.TimothyMiller,StevenBethard,DmitriyDligach,SameerPradhan,ChenLin,andGuerganaSavova.2013.Dis-coveringtemporalnarrativecontainersinclinicaltext.InProceedingsofthe2013WorkshoponBiomedicalNaturalLanguageProcessing,pages18–26,Sofia,Bulgaria,August.AssociationforComputationalLin-guistics.
je
D
o
w
n
o
un
d
e
d
F
r
o
m
h
t
t
p
:
/
/
d
je
r
e
c
t
.
m
je
t
.
e
d
toi
/
t
un
c
je
/
je
un
r
t
je
c
e
–
p
d
F
/
d
o
je
/
.
1
0
1
1
6
2
/
t
je
un
c
_
un
_
0
0
1
7
2
1
5
6
6
8
2
2
/
/
t
je
un
c
_
un
_
0
0
1
7
2
p
d
.
F
b
oui
g
toi
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
154
EleniMiltsakaki,RashmiPrasad,AravindJoshi,andBon-nieWebber.2004.Thepenndiscoursetreebank.InInProceedingsofLREC2004.MassimoPoesio.2004.Discourseannotationandseman-ticannotationinthegnomecorpus.InInProceedingsoftheACLWorkshoponDiscourseAnnotation.JamesPustejovskyandAmberStubbs.2011.Increasinginformativenessintemporalannotation.InProceedingsofthe5thLinguisticAnnotationWorkshop,pages152–160.AssociationforComputationalLinguistics.JamesPustejovsky,RobertKnippen,JessicaLittman,andRoserSauri.2005.Temporalandeventinformationinnaturallanguagetext.LanguageResourcesandEvalu-ation,39(2-3):123–164.JamesPustejovsky,KiyongLee,HarryBunt,andLaurentRomary.2010.Iso-timeml:Aninternationalstandardforsemanticannotation.InProceedingsoftheSeventhInternationalConferenceonLanguageResourcesandEvaluation(LREC2010),Valletta,Malta.AngusRoberts,RobertGaizauskas,MarkHepple,GeorgeDemetriou,YikunGuo,andIanRoberts.2009.Build-ingasemanticallyannotatedcorpusofclinicaltexts.Journalofbiomedicalinformatics,42(5):950–966.GuerganaSavova,StevenBethard,WillStyler,JamesMar-tin,MarthaPalmer,JamesMasanz,andWayneWard.2009.Towardstemporalrelationdiscoveryfromtheclinicalnarrative.InAMIAAnnualSymposiumPro-ceedings,volume2009,page568.AmericanMedicalInformaticsAssociation.TessaSullivan,AnnIrvine,andStephanieWHaas.2008.It’sallrelative:usageofrelativetemporalexpressionsintriagenotes.ProceedingsoftheAmericanSocietyforInformationScienceandTechnology,45(1):1–8.WeiyiSun,AnnaRumshisky,andOzlemUzuner.2013a.Evaluatingtemporalrelationsinclinicaltext:2012i2b2challenge.JournaloftheAmericanMedicalInformat-icsAssociation.WeiyiSun,AnnaRumshisky,andOzlemUzuner.2013b.Evaluatingtemporalrelationsinclinicaltext:2012i2b2challenge.JournaloftheAmericanMedicalInformat-icsAssociation,20(5):806–813.AlexanderTurchin,MariaShubina,EugeneBreydo,MerriLPendergrass,andJonathanSEinbinder.2009.Comparisonofinformationcontentofstructuredandnarrativetextdatasourcesontheexampleofmedica-tionintensification.JournaloftheAmericanMedicalInformaticsAssociation,16(3):362–370.NaushadUzZaman,HectorLlorens,LeonDerczynski,JamesAllen,MarcVerhagen,andJamesPustejovsky.2013.Semeval-2013task1:Tempeval-3:Evaluatingtimeexpressions,events,andtemporalrelations.InSec-ondJointConferenceonLexicalandComputationalSemantics(*SEM),Volume2:ProceedingsoftheSev-enthInternationalWorkshoponSemanticEvaluation(SemEval2013),pages1–9,Atlanta,Georgia,Etats-Unis,June.AssociationforComputationalLinguistics.MarcVerhagen.2005.TemporalClosureinanAnnota-tionEnvironment.LanguageResourcesandEvalua-tion,39(2):211–241.VeronikaVincze,GyrgySzarvas,RichrdFarkas,GyrgyMra,andJnosCsirik.2008.Thebioscopecorpus:biomedicaltextsannotatedforuncertainty,negationandtheirscopes.BMCBioinformatics,9(Suppl11):1–9.YingZhao,GeorgeKarypis,andUsamaM.Fayyad.2005.Hierarchicalclusteringalgorithmsfordocu-mentdatasets.DataMiningandKnowledgeDiscovery,10:141–168.JiapingZheng,WendyWChapman,RebeccaSCrowley,andGuerganaKSavova.2011.Coreferenceresolution:Areviewofgeneralmethodologiesandapplicationsintheclinicaldomain.Journalofbiomedicalinformatics,44(6):1113–1122.LiZhou,SimonParsons,andGeorgeHripcsak.2008.Theevaluationofatemporalreasoningsysteminprocessingclinicaldischargesummaries.JournaloftheAmericanMedicalInformaticsAssociation,15(1):99–106.
Télécharger le PDF