Transactions of the Association for Computational Linguistics, vol. 3, pp. 131–143, 2015. Action Editor: Masaaki Nagata.
Submission batch: 10/2014; Revision batch 1/2015; Published 3/2015. c
2015 Association for Computational Linguistics.
(cid:13)
131
UnsupervisedDeclarativeKnowledgeInductionforConstraint-BasedLearningofInformationStructureinScientificDocumentsYufanGuoDTALUniversityofCambridge,UKyg244@cam.ac.ukRoiReichartTechnion-IITHaifa,Israelroiri@ie.technion.ac.ilAnnaKorhonenDTALUniversityofCambridge,UKalk23@cam.ac.ukAbstractInferringtheinformationstructureofscien-tificdocumentsisusefulformanyNLPappli-cations.Existingapproachestothistaskre-quiresubstantialhumaneffort.Weproposeaframeworkforconstraintlearningthatre-duceshumaninvolvementconsiderably.Ourmodelusestopicmodelstoidentifylatenttop-icsandtheirkeylinguisticfeaturesininputdocuments,inducesconstraintsfromthisin-formationandmapssentencestotheirdomi-nantinformationstructurecategoriesthroughaconstrainedunsupervisedmodel.Whentheinducedconstraintsarecombinedwithafullyunsupervisedmodel,theresultingmodelchallengesexistinglightlysupervisedfeature-basedmodelsaswellasunsupervisedmod-elsthatusemanuallyconstructeddeclarativeknowledge.Ourresultsdemonstratethatuse-fuldeclarativeknowledgecanbelearnedfromdatawithverylimitedhumaninvolvement.1IntroductionAutomaticanalysisofscientifictextcanhelpscien-tistsfindinformationfromliteraturefaster,savingvaluableresearchtime.Inthispaperwefocusontheanalysisoftheinformationstructure(IS)ofsci-entificarticleswheretheaimistoassigneachunitofanarticle(typicallyasentence)intoacategorythatrepresentstheinformationtypeitconveys.Byinfor-mationstructurewerefertoaparticulartypeofdis-coursestructurethatfocusesonthefunctionalroleofaunitinthediscourse(Webberetal.,2011).Forinstance,inthescientificliterature,thefunctionalroleofasentencecouldbethebackgroundormoti-vationoftheresearch,themethodsused,theexperi-mentscarriedout,theobservationsontheresults,ortheauthor’sconclusions.ReadersofscientificliteraturefindinformationinIS-annotatedarticlesmuchfasterthaninunanno-tatedarticles(Guoetal.,2011b).ArgumentativeZoning(AZ)–aninformationstructureschemethathasbeenappliedsuccessfullytomanyscientificdo-mains(Teufeletal.,2009)–hasimprovedtaskssuchassummarizationandinformationextractionandretrieval(TeufelandMoens,2002;Tbahritietal.,2006;Ruchetal.,2007;Liakataetal.,2012;Contractoretal.,2012).Existingapproachestoinformationstructureanal-ysisrequiresubstantialhumaneffort.Mostusefeature-basedmachinelearning,suchasSVMsandCRFs(par exemple.(TeufelandMoens,2002;Linetal.,2006;Hirohataetal.,2008;Shatkayetal.,2008;Guoetal.,2010;Liakataetal.,2012))whichrelyonthousandsofmanuallyannotatedtrainingsen-tences.Alsotheperformanceofsuchmethodsisratherlimited:Liakataetal.(2012)reportedper-classF-scoresrangingfrom.53to.76inthebio-chemistryandchemistrydomainsandGuoetal.(2013un)reportedsubstantiallylowernumbersforthechallengingIntroductionandDiscussionsectionsinbiomedicaldomain.Guoetal.(2013un)recentlyappliedtheGeneral-izedExpectation(GE)criterion(MannandMcCal-lum,2007)toinformationstructureanalysisusingexpertknowledgeintheformofdiscourseandlexi-calconstraints.Theirmodelproducespromisingre-sults,especiallyforsectionsandcategorieswhere
je
D
o
w
n
o
un
d
e
d
F
r
o
m
h
t
t
p
:
/
/
d
je
r
e
c
t
.
m
je
t
.
e
d
toi
/
t
un
c
je
/
je
un
r
t
je
c
e
–
p
d
F
/
d
o
je
/
.
1
0
1
1
6
2
/
t
je
un
c
_
un
_
0
0
1
2
8
1
5
6
6
7
6
0
/
/
t
je
un
c
_
un
_
0
0
1
2
8
p
d
.
F
b
oui
g
toi
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
132
feature-basedmodelsperformpoorly.Eventheunsupervisedversionwhichusesconstraintsunderamaximum-entropycriterionwithoutanyfeature-basedmodel,outperformsfully-supervisedfeature-basedmodelsindetectingchallenginglowfre-quencycategoriesacrosssections.However,thisap-proachstillrequiressubstantialhumaneffortincon-straintgeneration.Particularly,lexicalconstraintswereconstructedbycreatingadetailedwordlistforeachinformationstructurecategory.Forexample,wordssuchas“assay”werecarefullyselectedandusedasastrongindicatorofthe“Method”category:p(Method|assay)wasconstrainedtobehigh(above0.9).Suchaconstraint(developedforthebiomedi-caldomain)maynotbeapplicabletoanewdomain(e.g.computerscience)withadifferentvocabularyandwritingstyle.Infact,mostexistingworksonlearningwithdeclarativeknowledgerelyonmanuallyconstructedconstraints.Littleworkexistsonautomaticdeclar-ativeknowledgeinduction.Anotableexceptionis(McCloskyandManning,2012)thatproposedaconstraintlearningmodelfortimelineextraction.Thisapproach,cependant,requireshumansupervi-sioninseveralformsincludingtaskspecificcon-strainttemplates(seeSection2).Wepresentanovelframeworkforlearningdeclar-ativeknowledgewhichrequiresverylimitedhumaninvolvement.Weapplyittoinformationstructureanalysis,basedontwokeyobservations:1)Eachinformationstructurecategorydefinesadistributionoverasection-specificandanarticle-levelsetoflin-guisticfeatures.2)Eachsentenceinascientificdoc-ument,whilehavingadominantcategory,maycon-sistoffeaturesmostlyrelatedtoothercategories.Thisflexibleviewenablesustomakeuseoftopicmodelswhichhavenotprovedusefulinpreviousre-latedworks(Vargaetal.,2012;ReichartandKorho-nen,2012).Weconstructtopicmodelsatboththeindividualsectionandarticlelevelandapplythesemodelstodata,identifyinglatenttopicsandtheirkeylinguis-ticfeatures.Thisinformationisusedtoconstrainorbiasunsupervisedmodelsforthetaskinastraight-forwardway:weautomaticallygenerateconstraintsforaGEmodelandabiastermforagraphclus-teringobjective,suchthattheresultingmodelsas-signeachoftheinputsentencestooneinformationZoneDefinitionBackground(BKG)thebackgroundofthestudyProblem(PROB)theresearchproblemMethod(METH)themethodsusedResult(RES)theresultsachievedConclusion(CON)theauthors’conclusionsConnection(CN)workconsistentwiththecurrentworkDifference(DIFF)workinconsistentwiththecurrentworkFuturework(FUT)thepotentialfuturedirectionoftheresearchTable1:TheAZcategorizationschemeofthispaperstructurecategory.Bothmodelsprovidehighqual-itysentence-basedclassification,demonstratingthegeneralityofourapproach.WeexperimentwiththeAZschemefortheanal-ysisofthelogicalstructure,scientificargumenta-tionandintellectualattributionofscientificpapers(TeufelandMoens,2002),usinganeight-categoryversionofthisschemeforbiomedicine((Mizutaetal.,2006;Guoetal.,2013b),Table1).Inevalu-ationagainstgoldstandardannotations,ourmodelrivalsthemodelofGuoetal.(2013un)whichreliesonmanuallyconstructedconstraints,aswellasastrongsupervisedfeature-basedmodeltrainedwithupto2000sentences.Intask-basedevaluationwemeasuretheusefulnessoftheinducedcategoriesforcustomizedsummarization(Contractoretal.,2012)fromspecifictypesofinformationinanarticle.TheAZcategoriesinducedbyourmodelprovemorevaluablethanthoseof(Guoetal.,2013a)andthoseinthegoldstandard.Ourworkdemonstratesthegreatpotentialofautomaticallyinduceddeclarativeknowledgeinbothimprovingtheperformanceofin-formationstructureanalysisandreducingrelianceofhumansupervision.2PreviousWorkAutomaticDeclarativeKnowledgeInductionLearningwithdeclarativeknowledgeofferseffectivemeansofreducinghumansupervisionandimprov-ingperformance.Thisframeworkaugmentsfeature-basedmodelswithdomainandexpertknowledgeintheformof,e.g.,linearconstraints,posteriorprobabilitiesandlogicalformulas(par exemple.(Changetal.,2007;MannandMcCallum,2007;MannandMcCallum,2008;Ganchevetal.,2010)).IthasprovenusefulformanyNLPtasksincludingunsu-pervisedandsemi-supervisedPOStagging,parsing(Drucketal.,2008;Ganchevetal.,2010;Rushetal.,2012)andinformationextraction(Changetal.,
je
D
o
w
n
o
un
d
e
d
F
r
o
m
h
t
t
p
:
/
/
d
je
r
e
c
t
.
m
je
t
.
e
d
toi
/
t
un
c
je
/
je
un
r
t
je
c
e
–
p
d
F
/
d
o
je
/
.
1
0
1
1
6
2
/
t
je
un
c
_
un
_
0
0
1
2
8
1
5
6
6
7
6
0
/
/
t
je
un
c
_
un
_
0
0
1
2
8
p
d
.
F
b
oui
g
toi
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
133
2007;MannandMcCallum,2008;ReichartandKo-rhonen,2012;ReichartandBarzilay,2012).Cependant,declarativeknowledgeisstillcreatedinacostlymanualprocess.Weproposeinducingsuchknowledgedirectlyfromtextwithminimalhumaninvolvement.ThisideacouldbeappliedtoalmostanyNLPtask.Weapplyitheretoinformationstruc-tureanalysisofscientificdocuments.Littlepriorworkexistsonautomaticconstraintlearning.Recently,(McCloskyandManning,2012)investigatedtheapproachfortimelineextraction.Theyusedasetofgoldrelationsandtheirtemporalspansandapplieddistantlearningtofindapproxi-mateinstancesforclassifiertraining.Asetofcon-strainttemplatesspecifictotemporallearningwerealsospecified.Incontrast,wedonotusemanuallyspecifiedguidanceinconstraintlearning.Particu-larly,weconstructconstraintsfromlatentvariables(topicsintopicmodeling)estimatedfromrawtextratherthanapplyingmaximumlikelihoodestimationoverobservedvariables(fluentsandtemporalex-pressions)inlabeleddata.Ourmethodisthereforelessdependentonhumansupervision.Evenmorerecently,(Anzarootetal.,2014)presentedasuper-viseddual-decompositionbasedmethod,inthecon-textofcitationfieldextraction,whichautomaticallygenerateslargefamiliesofconstraintsandlearntheircostswithaconvexoptimizationobjectiveduringtraining.Ourworkisunsupervised,asopposedtotheirmodelwhichrequiresamanuallyannotatedtrainingcorpusforconstraintlearning.InformationStructureAnalysisVariousschemeshavebeenproposedforanalysingtheinformationstructureofscientificdocuments,inparticularthepatternsoftopics,functionsandre-lationsatsentencelevel.Existingschemesincludeargumentativezones(TeufelandMoens,2002;Mizutaetal.,2006;Teufeletal.,2009),discoursestructure(Bursteinetal.,2003;Webberetal.,2011),qualitativedimensions(Shatkayetal.,2008),scientificclaims(Blake,2009),scientificconcepts(Liakataetal.,2010),andinformationstatus(Mark-ertetal.,2012),amongothers.Mostpreviousworkonautomaticanalysisofinformationstructurereliesonsupervisedlearning(TeufelandMoens,2002;Bursteinetal.,2003;Mizutaetal.,2006;Shatkayetal.,2008;Guoetal.,2010;Liakataetal.,2012;Markertetal.,2012).Giventheprohibitivecostofmanualannotation,unsupervisedandminimallysupervisedtechniquessuchasclustering(Kielaetal.,2014)andtopicmodeling(Vargaetal.,2012;´OS´eaghdhaandTeufel,2014)arehighlyimportant.However,theperformanceofsuchapproachesshowsalargeroomforimprovement.Ourworkisspecificallyaimedataddressingthisproblem.InformationStructureLearningwithDeclar-ativeKnowledgeRecently,ReichartandKorhonen(2012)andGuoetal.(2013un)developedconstrainedmodelsthatintegraterichlinguisticknowledge(e.g.discoursepatterns,syntacticfeaturesandsentencesimilarityinformation)formorereliableunsuper-visedortransductivelearningofinformationcate-goriesinscientificabstractsandarticles.Guoetal.(2013un)useddetailedlexicalconstraintsdevelopedviahumansupervision.Whetherautomaticallyin-duceddeclarativeknowledgecanrivalsuchmanualconstraintsisaquestionweaddressinthiswork.WhileReichartandKorhonen(2012)usedmoregeneralconstraints,theirmosteffectivediscourseconstraintsweretailoredtoscientificabstractsandarelessrelevanttofullpapers.3ModelWeintroduceatopic-modelbasedapproachtodeclarativeknowledge(DK)acquisitionanddescribehowthisknowledgecanbeappliedtotwounsuper-visedmodelsforourtask.Section3.1describeshowtopicmodelsareusedtoinducetopicsthatserveasthemainbuildingblocksofourDK.Section3.2ex-plainshowtheresultingtopicsandtheirkeyfeaturesaretransformedintoDK–constraintsinthegeneral-izedexpectation(GE)modelandbiasfunctionsinagraphclusteringalgorithm.3.1InducingInformationStructureCategorieswithLatentDirichletAllocationLatentDirichletAllocation(LDA)LDAisagener-ativeprocesswidelyusedfordiscoveringlatenttop-icsintextdocuments(Bleietal.,2003).Itassumesthefollowinggenerativeprocessforeachdocument:1.Chooseθi∼Dirichlet(un),i∈{1,…,M.}2.Chooseφk∼Dirichlet(β),k∈{1,…,K}3.Foreachwordwij,j∈{1,…,Ni}(un)Chooseatopiczij∼Multinomial(θi)(b)Chooseawordwij∼Multinomial(φzij),
je
D
o
w
n
o
un
d
e
d
F
r
o
m
h
t
t
p
:
/
/
d
je
r
e
c
t
.
m
je
t
.
e
d
toi
/
t
un
c
je
/
je
un
r
t
je
c
e
–
p
d
F
/
d
o
je
/
.
1
0
1
1
6
2
/
t
je
un
c
_
un
_
0
0
1
2
8
1
5
6
6
7
6
0
/
/
t
je
un
c
_
un
_
0
0
1
2
8
p
d
.
F
b
oui
g
toi
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
134
whereθiisthedistributionoftopicsindocumenti,φkisthedistributionofobservedfeatures(usuallywords)fortopick,zijisthetopicofthej-thwordindocumenti,andwijisthej-thwordindocumenti.Anumberofinferencetechniqueshavebeenpro-posedfortheparameterestimationofthisprocess,e.g.variationalBayes(Bleietal.,2003)andGibbssampling(GriffithsandSteyvers,2004)whichweuseinthiswork.TopicsandInformationStructureCategoriesAkeychallengeintheapplicationofLDAtoin-formationstructureanalysisisdefiningtheobservedfeaturesgeneratedbythemodel.Topicsareusuallydefinedtobedistributionsoverallthewordsinadocument,butinourtaskthiscanleadtoundesiredtopics.Consider,forexample,thefollowingsen-tencesfromtheIntroductionsectionofanarticle:-D'abord,exposuretoBD-diolviainhalationcausesanincreaseinHprtmutationfrequencyinbothmiceandrats(25).-Troisième,BD-diolisaprecursortoMI,animportanturinarymetaboliteinhumansexposedtoBD(19).Inaword-basedtopicmodelwecanexpectthatmostofthecontentwordsinthesesentenceswillbegen-eratedbyasingletopicthatcanbetitledas“BD-diol”,orbytwodifferenttopicsrelatedto“micerat”and“human”.However,informationstructurecategoriesshouldreflecttheroleofthesentenceine.g.thediscourseorargumentstructureofthepa-per.Forexample,giventheAZschemebothsen-tencesshouldbelongtothebackground(BKG)cate-gory(Table1).Thesamerequirementappliestothetopicsinducedbythetopicmodels.FeaturesInapplyingLDAtoAZ,wedefinetop-icsasdistributionsover:(un)wordsofparticularsyn-tacticcategories;(b)syntactic(POStag)motifs;et(c)discoursemarkers(citations,tablesandfig-ures).Belowwelistourfeatures,amongwhichPro-noun,Conjunction,AdjectiveandAdverbarenovelandtherestareadaptedfrom(Guoetal.,2013a):CitationAsinglefeaturethataggregatestogetherthevariouscitationformatsinscientificarticles(par exemple.[20]ou(Tudek2007)).Tableau,FigureAsinglefeaturerepresentinganyref-erencestotablesorfiguresinasentence.VerbVerbsarecentraltothemeaningofasentence.Eachofthebaseformsoftheverbsinthecorpusisauniquefeature.PronounPersonal(e.g.“we”)andpossessivepro-nouns(e.g.“our”)andthefollowingadjectives(asine.g.“ourrecent”or“ourprevious”)mayindicatetheownershipofthework(e.g.theauthor’sownvs.otherpeople’swork),whichisimportantforourtask.Eachoftheabovewordsorwordcombinationsisauniquefeature.ConjunctionConjunctionsindicatetherelationshipbetweendifferentsentencesintext.Weconsidertwotypesofconjunctions:(1)coordinatingconjunctions(indicatedbythePOStag“CC”intheoutputoftheC&CPOStagger);et(2)saturatedclausalmodi-fiers(indicatedbythePOStag“IN”andthecorre-spondinggrammaticalrelation“cmod”intheoutputoftheC&Cparser).Eachwordthatformsacon-junctionaccordingtothisdefinitionisauniquefea-ture.AdjectiveandAdverbAdjectivesprovidedescrip-tiveinformationaboutobjects,whileadverbsmaychangeorqualifythemeaningofverbsoradjectives.Eachadverbandadjectivethatappearsinmorethan5articlesinthecorpusisauniquefeature.1Modal,Tense,VoicePreviousworkhasdemon-stratedastrongcorrelationbetweentense,voice,modalsandinformationcategories(par exemple.(Guoetal.,2011a;Liakataetal.,2012)).Thesefeaturesarein-dicatedbythepart-of-speech(POS)tagofverbs.Forexample,thephrase“mayhavebeeninvesti-gated”isrepresentedas“may-MDhave-VBZbe-VBNverb-VBN”.Asapre-processingstep,eachsentenceinthein-putcorpuswasrepresentedwiththelistoffeaturesitconsistsof.Consider,forexample,thefollowingsentencefromaDiscussionsectioninourdata-set:-Inapreviouspreliminarystudywereportedthattheresultsofalimitedproofofconcepthumanclinicaltrialusingsulin-dac(1-5%)andhydrogenperoxide(25%)gelsapplieddailyforthreeweeksonactinickeratoses(AK)involvingtheupperex-tremities[27].BeforerunningtheDiscussionsectiontopicmodel(seebelowforthefeaturesconsideredbythismodel),thissentenceisconvertedtothefol-lowingrepresentation:[cite]previouspreliminarywelimitedThetopicmodelsweconstructareassumedtogen-1Wecollapsedadverbsendingwith-lyintothecorrespond-ingadjectivestoreducedatasparsity.Verbsweresparedthefrequencycut-offbecauserarelyoccurringverbsarelikelytocorrespondtodomain-specificactionsthatareprobablyindica-tiveoftheMETHcategory.
je
D
o
w
n
o
un
d
e
d
F
r
o
m
h
t
t
p
:
/
/
d
je
r
e
c
t
.
m
je
t
.
e
d
toi
/
t
un
c
je
/
je
un
r
t
je
c
e
–
p
d
F
/
d
o
je
/
.
1
0
1
1
6
2
/
t
je
un
c
_
un
_
0
0
1
2
8
1
5
6
6
7
6
0
/
/
t
je
un
c
_
un
_
0
0
1
2
8
p
d
.
F
b
oui
g
toi
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
135
ModelFeaturesArticleVerb,Tableau,Chiffre,Modal,Tense,VoiceIntroductionCitation,Pronoun,Verb,Modal,Tense,VoiceDiscussionCitation,Pronoun,Conjunction,Adjective,AdverbTable2:Thefeaturesusedinthearticle-levelandthesection-specifictopicmodelsinthispapereratethesefeaturesratherthanbag-of-words.TopicModelsConstructionLookingatthecate-goriesinTable1itiseasytoseethatdifferentcom-binationsofthefeaturesintopicmodelgenerationwillberelevantfordifferentcategorydistinctions.Forexample,personalpronounsareparticularlyrel-evantfordistinguishingbetweencategoriesrelatedtocurrentvs.previousworks.Somedistinctionsbetweencategoriesare,inturn,morerelevantforsomesectionsthanforothers.Forexample,thedistinctionbetweenthebackground(BKG)andthedefinitionoftheresearchproblem(PROB)isimportantfortheIntroductionsection,butlessimportantfortheresultssection.Similarlythedistinctionbetweenconclusions(CON)anddiffer-encefrompreviouswork(DIFF)ismorerelevantfortheDiscussionsectionthanothersections.Wethereforeconstructedtwotypesoftopicmod-els:section-specificandarticle-levelmodels,rea-soningthatsomedistinctionsapplygloballyatthearticlelevelwhilesomeapplymorelocallyatthesectionlevel.Section-specificmodelswerecon-structedfortheIntroductionsectionandfortheDis-cussionsection.2Table2presentsthefeaturesthatareusedwitheachtopicmodel.Akeyissueintheapplicationoftopicmodelstoourtaskisthedefinitionoftheunitoftextforwhichθi,thedistributionovertopics,isdrawnfromtheDirichletdistribution(step1ofthealgorithm).Thischoiceisdatadependent,andthestandardchoiceisthedocumentlevel.However,forscientificarti-clestheparagraphlevelisabetterchoice,becauseaparagraphcontainsonlyasmallsubsetofinforma-tionstructurecategorieswhileinafullarticlecat-egoriesaremoreevenlydistributed.Wethereforeadoptedtheparagraphasourbasicunitoftext.Thesection-levelandthearticle-levelmodelsareapplied2TheMethodssectionislesssuitableforasection-leveltopicmodelas97.5%ofitssentencesbelongtoitsdominantcategory(METH)(Table3).Preliminaryexperimentswithsection-leveltopicmodelsfortheMethodsandResultssectionsdidnotleadtoimprovedperformance.tothecollectionofparagraphsinthespecificsectionacrossthetestsetarticlesorintheentiresetoftestarticles,respectively.3.2DeclarativeKnowledgeInductionMostsentence-basedinformationstructureanalysisapproachesassociateeachsentencewithauniquecategory.However,sincetheMAPassignmentoftopicstofeaturesassociateseachsentencewithmul-tipletopics,wecannotdirectlyinterprettheresultingtopicsascategoriesofinputsentences.3Inthissectionwepresenttwomethodsforin-corporatingtheinformationconveyedbythetopicmodels(seeSection3.1)inunsupervisedmodels.ThefirstmethodbiasesagraphclusteringalgorithmwhilethesecondgeneratesconstraintsthatcanbeusedwithaGEcriterion.GraphClusteringWeusethegraphclusteringobjectiveofDhillonetal.(2007)whichcanbeopti-mizedefficiently,withouteigenvaluescalculations:max˜Ytrace(˜YTW−1/2AW−1/2˜Y)whereAisasimilaritymatrix,Wisadiagonalmatrixoftheweightofeachcluster,and˜Yisanorthonormalmatrix,indicatingclustermembership,whichisproportionaltothesquarerootofW.Tomakeuseoftopicstobiasthegraphclusteringtowardsthedesiredsolution,wedefinethesimilaritymatrixA,dont(je,j)−thentrycorrespondstothei-thandj-thtestsetsentencesasfollows:UN(je,j)=f(Si,Sj)+γg(Si,Sj,T),whereSi={Allthefeaturesextractedfromsentencei}T={Tk|Tk={topNfeaturesassociatedwithtopick}}F(Si,Sj)=|Si∩Sj|g(Si,Sj,T)=(cid:26)1∃x∈Si∃y∈Sj∃kx∈Tk∧y∈Tk0OtherwisewhereTkconsistsoftheNfeaturesthatareas-signedthemaximumprobabilityaccordingtothek-thtopic.Underthisformulation,thetopicmodeltermg(·)isdefinedtobetheindicatorofwhethertwosentencessharefeaturesassociatedwiththesametopic.Ifthisistrue,thealgorithmisencour-agedtoassignthesesentencestothesamecluster.GeneralizedExpectationAgeneralizedexpecta-tion(GE)criterionisaterminanobjectivefunction3Ourpreliminaryexperimentsdemonstratedthatassigningthelearnedtopicstothetestsentencesperformspoorly.
je
D
o
w
n
o
un
d
e
d
F
r
o
m
h
t
t
p
:
/
/
d
je
r
e
c
t
.
m
je
t
.
e
d
toi
/
t
un
c
je
/
je
un
r
t
je
c
e
–
p
d
F
/
d
o
je
/
.
1
0
1
1
6
2
/
t
je
un
c
_
un
_
0
0
1
2
8
1
5
6
6
7
6
0
/
/
t
je
un
c
_
un
_
0
0
1
2
8
p
d
.
F
b
oui
g
toi
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
136
thatassignsascoretomodelexpectations(MannandMcCallum,2008;Drucketal.,2008;Bellareetal.,2009).Givenascorefunctiong(·),adiscrim-inativemodelpλ(oui|X),avectoroffeaturefunctionsf∗(·),andanempiricaldistribution˜p(X),thevalueofaGEcriterionis:g(E˜p(X)[Epλ(oui|X)[f∗(X,oui)]])Apopularchoiceofg(·)isameasureofdistance(e.g.L2norm)betweenmodelandreferenceexpec-tations.Thefeaturefunctionsf∗(·)andtherefer-enceexpectationsoff∗(·)aretraditionallyspecifiedbyexperts,whichprovidesawaytointegratedeclar-ativeknowledgeintomachinelearning.ConsideraMaximumEntropy(MaxEnt)modelpλ(oui|X)=1Zλexp(λ·f(X,oui)),wheref(·)isavec-toroffeaturefunctions,λthefeatureweights,andZλthepartitionfunction.ThefollowingobjectivefunctioncanbeusedfortrainingMaxEntwithGEcriteriaonunlabeleddata:maxλ−g(E˜p(X)[Epλ(oui|X)[f∗(X,oui)]])−Xjλ2j2σ2wherethesecondtermisazero-meanσ2-varianceGaussianprioronparameters.Letthek-thfeaturefunctionf∗k(·)beanindicatorfunction:f∗k(X,oui)=1{xik=1∧y=yk}(X,oui)wherexikistheik-thelement/featureinthefeaturevectorx.Themodelexpectationoff∗k(·)becomes:E˜p(X)[Epλ(oui|X)[f∗k(X,oui)]]=˜p(xik=1)pλ(yk|xik=1)Tocalculateg(·),areferenceexpectationoff∗k(·)canbeobtainedafterspecifying(theupperandlowerlimitsof)p(yk|xik=1):lk≤p(yk|xik=1)≤ukThistypeofconstraints,forexample,0.9≤p(CON|suggérer)≤1,havebeensuccessfullyap-pliedtoGE-basedinformationstructureanalysisbyGuoetal.(2013un).Herewebuildontheirframe-workandourcontributionistheautomaticinductionofsuchconstraintsbytopicmodeling.Theassociationbetweenfeaturesandtopicscanbetransformedintoconstraintsasfollows.LetWzbeasetoftopNkeyfeaturesassociatedwithtopicz–theNfeaturesthatareassignedthemaximumprobabilityaccordingtothetopic.Wecomputethefollowingtopic-specificfeaturesets:Az={w|w∈Wz∧∀t6=zw6∈Wt}–thesetoffeaturesassociatedwithtopiczbutnotwithanyoftheothertopics;Bz=St6=zWt–thesetoffeaturesassociatedwithatleastonetopicotherthanz.Foreverytopic-featurepair(zk,wk)wethereforewritethefollowingconstraint:lk≤p(zk|wk=1)≤ukWesettheprobabilityrangeforthek-thpairasfol-lows:Ifwk∈Azkthenlk=0.9,uk=1,Ifwk∈Bzkthenlk=0,uk=0.1,Inanyothercaselk=0,uk=1.Thevaluesoflkandukwereselectedsuchthattheyreflectthestrongassociationbetweenthekeyfea-turesandtheirtopics.Ourbasicreasoningisthatifasentenceisrepresentedbyoneofthekeyuniquefeaturesofagiventopic,itishighlylikelytobeas-sociatedwiththattopic.Likewise,asentenceisun-likelytobeassociatedwiththetopicofinterestifithasakeyfeatureforanyothertopics.3.3SummaryofContributionLearningwithdeclarativeknowledgeisanactivere-centresearchavenueintheNLPcommunity.Inthisframeworkfeature-basedmodelsareaugmentedwithdomainandexpertknowledgeencodedmostoftenbyconstraintsofvarioustypes.Thehumaneffortinvolvedwiththisframeworkisthemanualspecificationofthedeclarativeknowledge.Thisre-quiresdeepunderstandingofthedomainandtaskinquestion.Theresultingconstraintstypicallyspec-ifydetailedassociationsbetweenlexical,grammat-icalanddiscourseelementsandtheinformationtobelearned(voir,e.g.,tables2and3of(Guoetal.,2013a)andtable1of(Changetal.,2007)).Ourkeycontributionistheautomaticinductionofdeclarativeknowledgethatcanbeeasilyintegratedintounsupervisedmodelsintheformofconstraintsandbiasfunctions.Ourmodelrequiresminimaldo-mainandtaskknowledge.Wedonotspecifylistsofwordsordiscoursemarkers(asin(Guoetal.,2013a))mais,instead,ourmodelautomaticallyasso-ciateslatentvariablesbothwithlinguisticfeatures,takenfromaverybroadandgeneralfeatureset(e.g.
je
D
o
w
n
o
un
d
e
d
F
r
o
m
h
t
t
p
:
/
/
d
je
r
e
c
t
.
m
je
t
.
e
d
toi
/
t
un
c
je
/
je
un
r
t
je
c
e
–
p
d
F
/
d
o
je
/
.
1
0
1
1
6
2
/
t
je
un
c
_
un
_
0
0
1
2
8
1
5
6
6
7
6
0
/
/
t
je
un
c
_
un
_
0
0
1
2
8
p
d
.
F
b
oui
g
toi
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
137
BKGPROBMETHRESCONCNDIFFFUTArticle16.92.834.817.922.34.30.80.2(8171)Introduction74.813.25.40.65.90.1–(1160)Methods0.50.297.51.40.20.20.1-(2557)Results4.02.111.768.912.11.10.1-(2054)Discussion16.91.10.71.563.513.32.40.7(2400)Table3:Distributionofsentences(showninpercentages)inarticlesandindividualsectionsintheAZ-annotatedcorpus.Thetotalnumberofsentencesineachsectionappearsinparenthesesbelowthesectionname.allthewordsthatbelongtoagivensetofPOStags),andwithsentencesintheinputtext.Inthenextsec-tionwepresentourexperimentswhichdemonstratetheusefulnessofthisdeclarativeknowledge.4ExperimentsDataandModelsWeusedthefullpapercor-pusearlieremployedin(Guoetal.,2013a)whichincludes8171annotatedsentences(withreportedinter-annotatoragreement:κ=.83)from50biomedicaljournalarticlesfromthecancerriskas-sessmentdomain.Onethirdofthiscorpuswassavedforadevelopmentsetonwhichourmodelwasde-signedanditshyperparametersweretuned(seebe-low).ThecorpusisannotatedaccordingtotheArgu-mentativeZoning(AZ)scheme(TeufelandMoens,2002;Mizutaetal.,2006)describedinTable1.Ta-ble3showsthedistributionofAZcategoriesandthetotalnumberofsentencesineachindividualsection.Sincesectionnamesvaryacrossarticles,wegroupedsimilarsectionsbeforecalculatingthestatistics(e.g.MaterialsandMethodssectionsweregroupedunderMethod).Thetabledemonstratesthatalthoughthereisadominantcategoryineachsection(e.g.BKGinIntroduction),upto36.5%ofthesentencesineachsectionfallintoothercategories.FeatureExtractionWeusedtheC&CPOStag-gerandparsertrainedonbiomedicalliterature(Cur-ranetal.,2007;RimellandClark,2009)inthefea-tureextractionprocess.LemmatizationwasdonewithMorpha(Minnenetal.,2001).BaselinesWecomparedourmodels(TopicGCandTopicGE)againstthefollowingbaselines:(un)anunconstrainedunsupervisedmodel–theunbiasedversionofthegraphclusteringweuseforTopicGC(i.e.whereg(·)isomitted,GC);(b)theunsuper-visedconstrainedGEmethodof(Guoetal.,2013a)wheretheconstraintswerecreatedbyexperts(Ex-pertGE);(c)supervisedunconstrainedMaximumEntropymodels,eachtrainedtopredictcategoriesinaparticularsectionusing150sentencesfromthatsection,asinthelightlysupervisedcasein(Guoetal.,2013a)(MaxEnt);et(d)abaselinethatassignsallthesentencesinagivensectiontothemostfre-quentgold-standardcategoryofthatsection(Table3).Thisbaselineemulatestheuseofsectionnamesforinformationstructureclassification.Ourconstraints,whichweuseintheTopicGEandTopicGCmodels,arebasedontopicsthatarelearnedonthetestcorpus.Whilehavingaccesstotherawtesttextattrainingtimeisastandardas-sumptioninmanyunsupervisedNLPworks(par exemple.(KleinandManning,2004;GoldwaterandGrif-fiths,2007;LangandLapata,2014)),itisimpor-tanttoquantifytheextenttowhichourmethodde-pendsonitsaccesstothetestset.Wethereforecon-structedtheTopicGE*modelwhichisidenticaltoTopicGEexceptthatthetopicsarelearnedfroman-othercollectionof47biomedicalarticlescontain-ing9352sentences.Likeourtestset,thesearticlesarefromthecancerriskassessmentdomain-allofthemwerepublishedintheToxicol.Sci.journalintheyears2009-2012andwereretrievedusingthePubMedsearchenginewiththekeywords“cancerriskassessment”.Thereisnooverlapbetweenthisnewdatasetandourtestset(Guoetal.,2013a).ModelsandParametersForgraphclustering,weusedtheGraclussoftware(Dhillonetal.,2007).ForGEandMaxEnt,weusedtheMalletsoftware(McCallum,2002).Theγparameterinthegraphclusteringwassetto10usingthedevelopmentdata.Severalvaluesofthisparameterintherangeof[10,1000]yieldedverysimilarperformance.Thenumberofkeyfeaturesconsideredforeachtopic,N,wassetto40,20and15forthearticle,Introduc-tionsection,andDiscussionsectiontopicmodels,respectively.Thisdifferencereflectsthenumberoffeaturetypes(Table2)andthetextvolume(Table3)oftherespectivemodels.EvaluationWeevaluatedtheoverallaccuracyaswellasthecategory-levelprecision,recallandF-scoreforeachsection.TopicGC,TopicGE,Top-icGE*andthebaselineGCmethodsareunsuper-
je
D
o
w
n
o
un
d
e
d
F
r
o
m
h
t
t
p
:
/
/
d
je
r
e
c
t
.
m
je
t
.
e
d
toi
/
t
un
c
je
/
je
un
r
t
je
c
e
–
p
d
F
/
d
o
je
/
.
1
0
1
1
6
2
/
t
je
un
c
_
un
_
0
0
1
2
8
1
5
6
6
7
6
0
/
/
t
je
un
c
_
un
_
0
0
1
2
8
p
d
.
F
b
oui
g
toi
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
138
IntroductionMethodResultDiscussionGCTGCTGETGE*EGEMFCGCTGCTGETGE*EGEMFCGCTGCTGETGE*EGEMFCGCTGCTGETGE*EGEMFCF1BKG.78.83.89.86.87.86—-.07—–.46-.47.47.45.49.46-PROB.34.16.31.19.24—–.33—–.04—–.32-METH-.16.12.16.35-.98.98.98.98.93.99.29-.25.32.29—–.14-RES—-.07—–.27-.67.82.81.77.80.82—-.14-CON-.10.26.03.28——-.39.28.27.29.42-.82.83.82.82.71.78CN—————-.25–.21.23.11.20-DIFF———————-.12-FUT———————-.36-Acc..61.68.77.74.72.75.97.97.97.97.87.97.51.68.67.62.64.69.66.67.67.67.56.63Table4:Performance(classbasedF1-scoreandoverallaccuracy(Acc.))ofunbiasedGraphClustering(GC),GraphClusteringwithdeclarativeknowledgelearnedfromtopicmodeling(TopicGCmodel,TGCcolumn),GeneralizedExpectationusingconstraintslearnedfromtopicmodeling(TopicGE,TGE)andthesamemodelwhereconstraintsarelearnedusinganexternalsetofarticles(TopicGE*,TGE*),GEwithconstraintscreatedbyexperts(ExpertGE,EGE-areplicationof(Guoetal.,2013a))andthemostfrequentgoldstandardcategoryofthesection(MFC)visedandthereforeinduceunlabeledcategories.ToevaluatetheiroutputagainstthegoldstandardAZannotationwefirstapplyastandardgreedymany-to-onemapping(naming)schemeinwhicheachin-ducedcategoryismappedtothegoldcategorythatsharesthehighestnumberofelements(sentence)withit(ReichartandRappoport,2009).Theto-talnumberofinducedtopicswas9witheachtopicmodelinducingthreetopics.4Forlightsupervision,aten-foldcross-validationschemewasapplied.Inaddition,wecomparethequalityoftheauto-maticallyinducedandmanuallyconstructeddeclar-ativeknowledgeinthecontextofcustomizedsum-marization(Contractoretal.,2012)wheresum-mariesofspecifictypesofinformationinanarticlearetobegenerated(wefocusedonthearticle’scon-clusions).WhileanintuitivesolutionwouldbetosummarizetheDiscussionsectionofapaper,only63.5%ofitssentencesbelongtothegoldstandardConclusioncategory(Table3).Forourexperiment,wefirstgeneratedfivesetsofsentences.Thefirstfoursetsconsistofthear-ticlesentencesannotatedwiththeCONcategoryac-cordingto:TopicGEorTopicGCorExpertGEorthegoldstandardannotation.ThefifthsetistheDiscus-sionsection.WethenusedMicrosoftAutoSumma-rize(Microsoft,2007)toselectsentencesfromeachofthefivesetssuchthatthenumberofwordsineachsummaryamountsfor10%ofthewordsintheinput.4ThenumberofgoldstandardAZcategoriesis8.However,wewantedeachofourtopicmodelstoinducethesamenumberoftopicsinordertoreducethenumberofparameterstotherequiredminimum.Forevaluation,weaskedanexperttosummarizetheconclusionsofeacharticleinthecorpus.Wethenevaluatedthefivesummariesagainstthegold-standardsummarieswrittenbytheexpertintermsofvariousROUGEscores(Lin,2004).5ResultsWereportheretheresultsforourconstrainedunsu-pervisedmodelscomparedtothebaselines.Westartwithquantitativeevaluationandcontinuewithqual-itativedemonstrationofthetopicslearnedbythetopicmodelsandtheirkeyfeatureswhichprovidethesubstancefortheconstraintsandbiasfunctionsusedinourinformationstructuremodels.UnsupervisedLearningResultsTable4presentstheperformanceofthefourmainunsupervisedlearningmodelsdiscussedinthispaper:GC,Top-icGC,TopicGE,andExpertGEof(Guoetal.,2013a).Ourmodels(TopicGCandTopicGE)out-performtheExpertGEwhenconsideringcategorybasedF-scoreforthedominantcategoriesofeachsection.ExpertGEismostusefulinidentifyingthelessfrequentcategoriesofeachsection(Table3),whichisinlinewith(Guoetal.,2013a).Theoverallsentence-basedaccuracyofTopicGEissignificantlyhigherthanthatofExpertGEforallfoursections(bottomlineofthetable).En outre,forallfoursectionsitisoneofourmodels(TopicGCorTop-icGE)thatprovidesthebestresultunderthismea-sure,amongtheunsupervisedmodels.Thetablefurtherprovidesacomparisonoftheun-supervisedmodelstotheMFCbaselinewhichas-signsallthesentencesofasectiontoitsmostfre-
je
D
o
w
n
o
un
d
e
d
F
r
o
m
h
t
t
p
:
/
/
d
je
r
e
c
t
.
m
je
t
.
e
d
toi
/
t
un
c
je
/
je
un
r
t
je
c
e
–
p
d
F
/
d
o
je
/
.
1
0
1
1
6
2
/
t
je
un
c
_
un
_
0
0
1
2
8
1
5
6
6
7
6
0
/
/
t
je
un
c
_
un
_
0
0
1
2
8
p
d
.
F
b
oui
g
toi
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
139
IntroductionMethodResultDiscussionTopicGELightTopicGELightTopicGELightTopicGELightPRFPRFPRFPRFPRFPRFPRFPRFBKG.84.95.89.78.99.87————.41.51.45.38.19.25PROB.33.30.31.57.11.18———.25.02.04——METH.40.07.12.50.21.30.971.98.971.98.34.20.25.62.14.23——RES————.74.90.81.71.98.82——CON.44.18.26.80.06.11——.30.25.27.57.16.25.78.87.82.69.96.80CN——————.32.18.23.35.06.10DIFF————————FUT————————Acc.0.770.770.970.970.670.700.670.66Table5:Performance(classbasedPrecision,RecallandF-scoreaswellasoverallaccuracy(Acc.))oftheTopicGEmodelandofanunconstrainedMaxEntmodeltrainedwithLightsupervision(totalof600sentences-150trainingsentencesforeachsection-levelmodel).ThesamepatternofresultsholdswhentheMaxEntistrainedwithupto2000sentences(500sentencesforeachsection-levelmodel).TopicGETopicGCExpertGESectionGoldRPFRPFRPFRPFRPFROUGE-145.254.046.843.555.146.143.749.143.846.743.842.643.355.446.2ROUGE-230.035.830.828.435.729.825.528.225.228.626.325.827.835.129.3ROUGE-L43.351.644.841.652.644.141.346.241.344.241.340.341,152.343.7Table6:ROUGEscoresofzone(TopicGE,TopicGC,ExpertGEorgoldstandard)andDiscussionsectionbasedsum-maries.TopicGEprovidesthebestsummaries.TopicGCoutperformsExpertGEandtheDiscussionsectionsystemsandintwomeasuresthegoldcategorizationbasedsystemaswell.ResultpatternswithROUGE(3,4,W-1.2,S*andSU*)areverysimilartothoseofthetable.ThedifferencesbetweenTopicGEandExpertGEarestatisticallysignificantusingt-testwithp<0.05.ThedifferencesbetweenTopicGEandgold,aswellasbetweenExpertGEandgoldarenotstatisticallysignificant.quentcategoryaccordingtothegoldstandard.Thisbaselineshedslightontheusefulnessofsectionnamesforourtask.Asisevidentfromthetable,whilethisbaselineiscompetitivewiththeunsuper-visedmodelsintermsofaccuracy,itsclass-basedF-scoreperformanceisquitepoor.NotonlydoesitlagbehindtheunsupervisedmodelsintermsoftheF-scoreofthemostfrequentclassesoftheIntroduc-tionandDiscussionsections,butitdoesnotiden-tifyanyoftheclassesexceptfromthemostfrequentonesinanyofthesections-atasktheunsupervisedmodelsoftenperformwithreasonablequality.Finally,thetablealsopresentstheperformanceoftheTopicGE*modelforwhichconstraintsareleanedfromanexternaldataset-differentfromthetestset.TheresultsshowthatthereisnosubstantialdifferencebetweentheperformanceoftheTopicGEandTopicGE*models.WhileTopicGEachievesbetterF-scoresinfiveofthecasesinthetable,Top-icGE*isbetterinfourcasesandtheperformanceisidenticalintwocases.SectionlevelaccuraciesarebetterforTopicGEintwoofthefoursections,butthedifferenceisonly3-5%.ComparisonwithSupervisedLearningTable5comparesthequalityofunsupervisedconstrained-basedlearningwiththatoflightlysupervisedfeature-basedlearning.Sinceourmodels,TopicGCandTopicGE,performquitesimilarly,weincludedonlyTopicGEinthisevaluation.Thelightlysu-pervisedmodels(MaxEntclassifiers)weretrainedwithatotalof600sentences-150foreachsection-specificclassifier.ThetabledemonstratesthatTop-icGEoutperformsMaxEntwithlightsupervisionintermsofclassbasedF-scoresintheIntroductionandDiscussionsections.IntheMethodssection,where97.5%ofthesentencesbelongtothemostfrequentcategory,andintheResultssection,themodelsper-formquitesimilarly.OverallaccuracynumbersarequitesimilarforbothmodelswithMaxEntdoingbetterfortheResultssectionandTopicGEfortheDiscussionsection.Theseresultsfurtherdemon-stratethatunsupervisedconstrainedlearningpro-videsapracticalsolutiontoinformationstructureanalysisofscientificarticles.ExtractiveSummarizationEvaluationTable6presentstheaverageROUGEscoresforzone-based(TopicGE,TopicGC,ExpertGEandgold)andsection-basedsummariesacrossourtestsetarticles.
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
e
d
u
/
t
a
c
l
/
l
a
r
t
i
c
e
-
p
d
f
/
d
o
i
/
.
1
0
1
1
6
2
/
t
l
a
c
_
a
_
0
0
1
2
8
1
5
6
6
7
6
0
/
/
t
l
a
c
_
a
_
0
0
1
2
8
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
140
TopicFeatures1{do}be{done}{doing}{bedone}{havebeendone}induce{maydo}{todo}showhave{havedone}increase{did}suggestindicatereportcauseincludeinhibitfindobserveinvolveassociateactivatedemonstrateresultuseleadplay{coulddo}know{dodo}formcontribute{cando}{woulddo}promotereduce2{weredone}{done}{doing}{did}usebedescribecontainperformincubate{do}determineanalyzefollowaddisolatepurchasewashaccord{todo}treatcollectremoveprepareobtainmeasurestorestaincentrifugetransferdetectpurifyassesssupplementcarrydissolveplatereceivekill3{did}{done}be{doing}{weredone}[tabfig]{do}showincreaseobservecompare{todo}exposeusehavefind{diddo}treat{bedone}reportfollowdrinkreduceresultadministerdecreasedeterminemeasureincludeevaluateaffectdetectinduceindicateassociateproviderevealsuggestoccurTable7:Topicsandkeyfeaturesextractedbythearticle-levelmodel(includingmodal,tenseandvoicemarkedincurlybrackets,referencetotablesorfiguresmarkedinsquarebrackets,andverbsinthebaseform)TopicFeatures1[nocite]{did}(we){done}{do}{doing}use{weredone}(present){todo}investigatebe(mammary)determineprovide(our)treatcompareexamine2{did}{done}[cite]{doing}{weredone}beexposefind[nocite]drinkincreasereport(recent)(previous)admin-ister{do}containevaluate(early)3{do}[cite]be{done}[nocite]{doing}{bedone}{havebeendone}induce{havedone}(it)show{maydo}have{todo}includeincrease(their)associateTable8:Topicsandkeyfeaturesextractedbythesection-specifictopicmodeloftheIntroductionsection(includingcitationsmarkedinsquarebrackets,pronounsandthefollow-upadjectivemodifiersmarkedinparentheses,modal,tenseandvoicemarkedincurlybrackets,andverbsintheirbaseform)TopicFeatures1(we)[nocite](our)higher(mammary)asbecause(first)significantpossiblehigh(early)(positive)most2[cite]present(present)(previous)similardifferent(its)althoughconsistentfurthermoregreaterduemostwhereas3[nocite]notalso(it)buthowevermore(their)boththereforeonlythussignificantlowerTable9:Topicsandkeyfeaturesextractedbythesection-specifictopicmodeloftheDiscussionsection(includingcitationsmarkedinsquarebrackets,pronounsandthefollow-upadjectivemodifiersmarkedinparentheses,andcon-junctions,adjectivesandadverbs)TopicGEandTopicGCbasedsummariesoutperformtheothersystems,eventheonethatusesgoldstan-dardinformationstructurecategorization.Apoten-tialexplanationforthebetterperformanceofourmodelscomparedtoExpertGEisthattherelativestrengthofourmodelsisinidentifyingthemajorcategoryofeachsectionwhileExpertGEisbetteratidentifyinglowormediumfrequencycategories.QualitativeAnalysisWenextprovideaqualita-tiveanalysisofthetopicsinducedbyourtopicmod-els—thearticle-levelmodelaswellasthesection-levelmodels—andtheirkeyfeatures.Notethatbothourmodels,TopicGEandTopicGC,assumethattheinducedtopicsprovideagoodapproxima-tionoftheinformationstructurecategoriesandbuildtheirconstraints(expertknowledge)fromthesetop-icsaccordingly.Belowweexaminethisassumption.Table7presentsthetopicsandkeyfeaturesob-tainedfromglobaltopicmodelingappliedtofullar-ticles.Thetablerevealsastrongcorrelationbetweenpresent/futuretenseandtopic1,andbetweenpasttenseandtopics2and3(Modal,TenseandVoicefeatures).Thetablefurtherdemonstratesthattop-ics1and3arelinkedtoverbsthatdescriberesearchfindings,suchas“show”and“demonstrate”intopic1,and“report”and“indicate”intopic3,whereastopic2seemsrelatedtoverbsthatdescribemethodsandexperimentssuchas“use”and“prepare“.Thefeaturecorrespondingtotablesandfigures[tabfig]isonlyseenintopic3.Basedontheseobservations,topics1,2and3seemtoberelatedtoAZcategoriesCON,METHandRESrespectively.Tables8and9presentthetopicsandthekeyfea-turesobtainedfromthesection-specifictopicmod-
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
e
d
u
/
t
a
c
l
/
l
a
r
t
i
c
e
-
p
d
f
/
d
o
i
/
.
1
0
1
1
6
2
/
t
l
a
c
_
a
_
0
0
1
2
8
1
5
6
6
7
6
0
/
/
t
l
a
c
_
a
_
0
0
1
2
8
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
141
elingfortheIntroductionandDiscussionsections.Duetospacelimitationswecannotprovideade-tailedanalysisoftheinformationincludedinthesetables,butitiseasytoseethattheyprovideevi-denceforthecorrelationbetweentopicsinthesec-tionspecificmodelsandAZcategories.Table8demonstratesthatfortheIntroductionsectiontopic1correlateswiththeauthor’sworkandtopics2and3withpreviouswork.Table9showsthatfortheDiscussionsectiontopics1and3wellcorrelatewiththeAZCONcategoryandtopic2withtheBKG,CNandDIFFcategories.Ouranalysisthereforedemon-stratesthattheinducedtopicsarewellalignedwiththeactualcategoriesoftheAZclassificationschemeorwithdistinctions(e.g.theauthor’sownworkvs.worksofothers)thatareveryrelevantforthisscheme.Notethatwehavenotseededourmodelswithword-listsandtheinducedtopicsarethereforepurelydata-driven.6DiscussionWepresentedanewframeworkforautomaticin-ductionofdeclarativeknowledgeandappliedittoconstraint-basedmodelingoftheinformationstruc-tureanalysisofscientificdocuments.Ourmaincon-tributionisatopic-modelbasedmethodforunsuper-visedacquisitionoflexical,syntacticanddiscourseknowledgeguidedbythenotionoftopicsandtheirkeyfeatures.Wedemonstratedthattheinducedtop-icsandkeyfeaturescanbeusedwithtwodiffer-entunsupervisedlearningmethods–aconstrainedunsupervisedgeneralizedexpectationmodelandagraphclusteringformulation.Ourresultsshowthatthisnovelframeworkrivalsmoresupervisedalterna-tives.Ourworkthereforecontributestotheimpor-tantchallengeofautomaticallyinducingdeclarativeknowledgethatcanreducethedependenceofMLalgorithmsonmanuallyannotateddata.Thenextnaturalstepinthisresearchisgeneraliz-ingourframeworkandmakeitapplicabletomoreapplications,domainsandmachinelearningmod-els.Wearecurrentlyinvestigatinganumberofideaswhichwillhopefullyleadtobetternaturallanguagelearningwithreducedhumansupervision.ReferencesSamAnzaroot,AlexandrePassos,DavidBelanger,andAndrewMcCallum.2014.Learningsoftlinearcon-straintswithapplicationtocitationfieldextraction.InACL,pages593–602.KedarBellare,GregoryDruck,andAndrewMcCallum.2009.Alternatingprojectionsforlearningwithexpec-tationconstraints.InProceedingsofthe25thCon-ferenceonUncertaintyinArtificialIntelligence,pages43–50.CatherineBlake.2009.Beyondgenes,proteins,andabstracts:Identifyingscientificclaimsfromfull-textbiomedicalarticles.JournalofBiomedicalInformat-ics,43(2):173–189.DavidM.Blei,AndrewY.Ng,andMichaelI.Jordan.2003.Latentdirichletallocation.JournalofMachineLearningResearch,3:993–1022.JillBurstein,DanielMarcu,andKevinKnight.2003.Findingthewritestuff:Automaticidentificationofdiscoursestructureinstudentessays.IEEEIntelligentSystems,18(1):32–39.Ming-WeiChang,LevRatinov,andDanRoth.2007.Guidingsemi-supervisionwithconstraint-drivenlearning.InACL,pages280–287.DanishContractor,YufanGuo,andAnnaKorhonen.2012.Usingargumentativezonesforextractivesum-marizationofscientificarticles.InCOLING,pages663–678.JamesCurran,StephenClark,andJohanBos.2007.Linguisticallymotivatedlarge-scalenlpwithc&candboxer.InProceedingsoftheACL2007DemoandPosterSessions,pages33–36.InderjitS.Dhillon,YuqiangGuan,andBrianKulis.2007.Weightedgraphcutswithouteigenvectors:Amultilevelapproach.IEEETransactionsonPat-ternAnalysisandMachineIntelligence,29(11):1944–1957.GregoryDruck,GideonMann,andAndrewMcCallum.2008.Learningfromlabeledfeaturesusinggener-alizedexpectationcriteria.InProceedingsofthe31stannualinternationalACMSIGIRconferenceonResearchanddevelopmentininformationretrieval,pages595–602.KuzmanGanchev,Jo˜aoGrac¸a,JenniferGillenwater,andBenTaskar.2010.Posteriorregularizationforstruc-turedlatentvariablemodels.JournalofMachineLearningResearch,11:2001–2049.SharonGoldwaterandTomGriffiths.2007.Afullybayesianapproachtounsupervisedpart-of-speechtag-ging.InACL,pages744–751.ThomasLGriffithsandMarkSteyvers.2004.Find-ingscientifictopics.ProceedingsoftheNationalAcademyofSciences,101(suppl1):5228–5235.
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
e
d
u
/
t
a
c
l
/
l
a
r
t
i
c
e
-
p
d
f
/
d
o
i
/
.
1
0
1
1
6
2
/
t
l
a
c
_
a
_
0
0
1
2
8
1
5
6
6
7
6
0
/
/
t
l
a
c
_
a
_
0
0
1
2
8
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
142
YufanGuo,AnnaKorhonen,MariaLiakata,IlonaSilinsKarolinska,LinSun,andUllaStenius.2010.Identify-ingtheinformationstructureofscientificabstracts:aninvestigationofthreedifferentschemes.InBioNLP,pages99–107.YufanGuo,AnnaKorhonen,andThierryPoibeau.2011a.Aweakly-supervisedapproachtoargumenta-tivezoningofscientificdocuments.InEMNLP,pages273–283.YufanGuo,AnnaKorhonen,IlonaSilins,andUllaSte-nius.2011b.Weakly-supervisedlearningofinfor-mationstructureofscientificabstracts–isitaccurateenoughtobenefitreal-worldtasksinbiomedicine?Bioinformatics,27(22):3179–3185.YufanGuo,RoiReichart,andAnnaKorhonen.2013a.Improvedinformationstructureanalysisofscientificdocumentsthroughdiscourseandlexicalconstraints.InNAACLHLT,pages928–937.YufanGuo,IlonaSilins,UllaStenius,andAnnaKorho-nen.2013b.Activelearning-basedinformationstruc-tureanalysisoffullscientificarticlesandtwoapplica-tionsforbiomedicalliteraturereview.Bioinformatics,29(11):1440–1447.KenjiHirohata,NaoakiOkazaki,SophiaAnaniadou,andMitsuruIshizuka.2008.Identifyingsectionsinsci-entificabstractsusingconditionalrandomfields.InIJCNLP,pages381–388.DouweKiela,YufanGuo,UllaStenius,andAnnaKo-rhonen.2014.Unsuperviseddiscoveryofinforma-tionstructureinbiomedicaldocuments.Bioinformat-ics,pagebtu758.DanKleinandChristopherManning.2004.Corpus-basedinductionofsyntacticstructure:Modelsofde-pendencyandconstituency.InACL,pages478–485.JoelLangandMirellaLapata.2014.Similarity-drivensemanticroleinductionviagraphpartitioning.Com-putationalLinguistics,40(3):633–669.MariaLiakata,SimoneTeufel,AdvaithSiddharthan,andColinBatchelor.2010.Corporaforconceptualizationandzoningofscientificpapers.InLREC,pages2054–2061.MariaLiakata,ShyamasreeSaha,SimonDobnik,ColinBatchelor,andDietrichRebholz-Schuhmann.2012.Automaticrecognitionofconceptualizationzonesinscientificarticlesandtwolifescienceapplications.Bioinformatics,28(7):991–1000.JimmyLin,DamianosKarakos,DinaDemner-Fushman,andSanjeevKhudanpur.2006.Generativecontentmodelsforstructuralanalysisofmedicalabstracts.InBioNLP,pages65–72.Chin-YewLin.2004.ROUGE:Apackageforauto-maticevaluationofsummaries.InTextSummarizationBranchesOut:ProceedingsoftheACL-04Workshop,pages74–81.GideonSMannandAndrewMcCallum.2007.Simple,robust,scalablesemi-supervisedlearningviaexpecta-tionregularization.InICML,pages593–600.GideonS.MannandAndrewMcCallum.2008.General-izedexpectationcriteriaforsemi-supervisedlearningofconditionalrandomfields.InACL,pages870–878.KatjaMarkert,YufangHou,andMichaelStrube.2012.Collectiveclassificationforfine-grainedinformationstatus.InACL,pages795–804.AndrewKachitesMcCallum.2002.MAL-LET:Amachinelearningforlanguagetoolkit.http://mallet.cs.umass.edu.DavidMcCloskyandChristopherD.Manning.2012.Learningconstraintsforconsistenttimelineextraction.InEMNLP-CoNLL,pages873–882.Microsoft.2007.AutoSummarize:Automaticallysummarizeadocument.https://support.office.com/en-us/article/Automatically-summarize-a-document-b43f20ae-ec4b-41cc-b40a-753eed6d7424.GuidoMinnen,JohnCarroll,andDarrenPearce.2001.Appliedmorphologicalprocessingofenglish.NaturalLanguageEngineering,7(3):207–223.YokoMizuta,AnnaKorhonen,TonyMullen,andNigelCollier.2006.Zoneanalysisinbiologyarticlesasabasisforinformationextraction.InternationalJournalofMedicalInformaticsonNaturalLanguageProcess-inginBiomedicineandItsApplications,75(6):468–487.Diarmuid´OS´eaghdhaandSimoneTeufel.2014.Unsu-pervisedlearningofrhetoricalstructurewithun-topicmodels.InProceedingsofCOLING2014:TechnicalPapers,pages2–13.RoiReichartandReginaBarzilay.2012.Multi-eventex-tractionguidedbyglobalconstraints.InNAACLHLT,pages70–79.RoiReichartandAnnaKorhonen.2012.Documentandcorpuslevelinferenceforunsupervisedandtransduc-tivelearningofinformationstructureofscientificdoc-uments.InCOLING,pages995–1006.RoiReichartandAriRappoport.2009.Thenvicluster-ingevaluationmeasure.InCoNLL,pages165–173.LauraRimellandStephenClark.2009.Portingalexicalized-grammarparsertothebiomedicaldomain.JournalofBiomedicalInformatics,42(5):852–865.PatrickRuch,CliaBoyer,ChristineChichester,ImadTbahriti,AntoineGeissbhler,PaulFabry,JulienGob-eill,ViolainePillet,DietrichRebholz-Schuhmann,ChristianLovis,andAnne-LiseVeuthey.2007.Usingargumentationtoextractkeysentencesfrombiomedi-calabstracts.InternationalJournalofMedicalInfor-matics,76(2-3):195–200.
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
e
d
u
/
t
a
c
l
/
l
a
r
t
i
c
e
-
p
d
f
/
d
o
i
/
.
1
0
1
1
6
2
/
t
l
a
c
_
a
_
0
0
1
2
8
1
5
6
6
7
6
0
/
/
t
l
a
c
_
a
_
0
0
1
2
8
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
143
AlexanderRush,RoiReichart,MichaelCollins,andAmirGloberson.2012.Improvedparsingandpostag-gingusinginter-sentenceconsistencyconstraints.InEMNLP-CoNLL,pages1434–1444.HagitShatkay,FengxiaPan,AndreyRzhetsky,andWJohnWilbur.2008.Multi-dimensionalclassifica-tionofbiomedicaltext:Towardautomated,practicalprovisionofhigh-utilitytexttodiverseusers.Bioin-formatics,24(18):2086–2093.ImadTbahriti,ChristineChichester,Fr´ed´eriqueLisacek,andPatrickRuch.2006.Usingargumentationtoretrievearticleswithsimilarcitations.InternationalJournalofMedicalInformatics,75(6):488–495.SimoneTeufelandMarcMoens.2002.Summariz-ingscientificarticles:Experimentswithrelevanceandrhetoricalstatus.ComputationalLinguistics,28(4):409–445.SimoneTeufel,AdvaithSiddharthan,andColinBatche-lor.2009.Towardsdomain-independentargumenta-tivezoning:Evidencefromchemistryandcomputa-tionallinguistics.InEMNLP,pages1493–1502.AndreaVarga,DanielPreotiuc-Pietro,andFabioCiravegna.2012.Unsuperviseddocumentzoneiden-tificationusingprobabilisticgraphicalmodels.InLREC,pages1610–1617.BonnieWebber,MarkusEgg,andValiaKordoni.2011.Discoursestructureandlanguagetechnology.NaturalLanguageEngineering,18(4):437–490.
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
e
d
u
/
t
a
c
l
/
l
a
r
t
i
c
e
-
p
d
f
/
d
o
i
/
.
1
0
1
1
6
2
/
t
l
a
c
_
a
_
0
0
1
2
8
1
5
6
6
7
6
0
/
/
t
l
a
c
_
a
_
0
0
1
2
8
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
144
Télécharger le PDF