计算语言学协会会刊, 2 (2014) 351–362. 动作编辑器: Hal Daume III.

Submitted 2/2014; 修改 5/2014; 已发表 10/2014. C
(西德:13)

2014 计算语言学协会.

TREETALK:CompositionandCompressionofTreesforImageDescriptionsPolinaKuznetsova††StonyBrookUniversityStonyBrook,NYpkuznetsova@cs.stonybrook.eduVicenteOrdonez‡TamaraL.Berg‡‡UNCChapelHillChapelHill,NC{vicente,tlberg}@cs.unc.eduYejinChoi††††UniversityofWashingtonSeattle,WAyejin@cs.washington.eduAbstractWepresentanewtreebasedapproachtocomposingexpressiveimagedescriptionsthatmakesuseofnaturallyoccuringwebimageswithcaptions.Weinvestigatetworelatedtasks:imagecaptiongeneralizationandgen-eration,wheretheformerisanoptionalsub-taskofthelatter.Thehigh-levelideaofourapproachistoharvestexpressivephrases(astreefragments)fromexistingimagedescrip-tions,thentocomposeanewdescriptionbyselectivelycombiningtheextracted(andop-tionallypruned)treefragments.Keyalgo-rithmiccomponentsaretreecompositionandcompression,bothintegratingtreestructurewithsequencestructure.Ourproposedsystemattainssigniﬁcantlybetterperformancethanpreviousapproachesforbothimagecaptiongeneralizationandgeneration.Inaddition,ourworkistheﬁrsttoshowtheempiricalben-eﬁtofautomaticallygeneralizedcaptionsforcomposingnaturalimagedescriptions.1IntroductionThewebisincreasinglyvisual,withhundredsofbil-lionsofusercontributedphotographshostedonline.Asubstantialportionoftheseimageshavesomesortofaccompanyingtext,rangingfromkeywords,tofreetextonwebpages,totextualdescriptionsdi-rectlydescribingdepictedimagecontent(i.e.cap-tions).Wetapintothelastkindoftext,usingnatu-rallyoccuringpairsofimageswithnaturallanguagedescriptionstocomposeexpressivedescriptionsforqueryimagesviatreecompositionandcompression.Suchautomaticimagecaptioningeffortscouldpotentiallybeusefulformanyapplications:fromautomaticorganizationofphotocollections,tofacil-itatingimagesearchwithcomplexnaturallanguagequeries,toenhancingwebaccessibilityforthevi-suallyimpaired.Ontheintellectualside,bylearn-ingtodescribethevisualworldfromnaturallyexist-ingwebdata,ourstudyextendsthedomainsoflan-guagegroundingtothehighlyexpressivelanguagethatpeopleuseintheireverydayonlineactivities.Therehasbeenarecentspikeineffortstoau-tomaticallydescribevisualcontentinnaturallan-guage(Yangetal.,2011;Kulkarnietal.,2011;Lietal.,2011;Farhadietal.,2010;Krishnamoorthyetal.,2013;ElliottandKeller,2013;YuandSiskind,2013;Socheretal.,2014).Thisreﬂectsthelongstandingunderstandingthatencodingthecomplex-itiesandsubtletiesofimagecontentoftenrequiresmoreexpressivelanguageconstructsthanasetoftags.Nowthatvisualrecognitionalgorithmsarebe-ginningtoproducereliableestimatesofimagecon-tent(Perronninetal.,2012;Dengetal.,2012a;Dengetal.,2010;Krizhevskyetal.,2012),thetimeseemsripetobeginexploringhigherlevelsemantictasks.Therehavebeentwomaincomplementarydirec-tionsexploredforautomaticimagecaptioning.Theﬁrstfocusesondescribingexactlythoseitems(e.g.,objects,属性)thataredetectedbyvisionrecog-nition,whichsubsequentlyconﬁneswhatshouldbedescribedandhow(Yaoetal.,2010;Kulkarnietal.,2011;Kojimaetal.,2002).Approachesinthisdirec-tioncouldbeidealforvariouspracticalapplicationssuchasimagedescriptionforthevisuallyimpaired.However,itisnotclearwhetherthesemanticexpres-sivenessoftheseapproachescaneventuallyscaleuptothecasual,buthighlyexpressivelanguagepeo-

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你

/
t

A
C
我
/

我

A
r
t
我
C
e
–
p
d

F
/

d
哦

我
/

1
0
1
1
6
2

/
t

我

A
C
_
A
_
0
0
1
8
8
1
5
6
6
9
0
5

/
t

我

A
C
_
A
_
0
0
1
8
8
p
d

乙
y
G
你
e
s
t

哦
n
0
9
S
e
p
e
米
乙
e
r
2
0
2
3

352

Target’Image’A”cow!常设!在!这!水!我!no/ced!那!这!funny!cow!曾是”staring”在”我”A!bird!hovering!在”这”grass”你!能!看!这些!beau/ful!hills!仅有的!在”这”农村”Object’Ac/on’Stuﬀ’Scene’Figure1:Harvestingphrases(astreefragments)forthetargetimagebasedon(部分的)visualmatch.plenaturallyuseintheironlineactivities.InFig-ure1,forexample,itwouldbehardtocompose“Inoticedthatthisfunnycowwasstaringatme”or“Youcanseethesebeautifulhillsonlyinthecoun-tryside”inapurelybottom-upmannerbasedontheexactcontentdetected.Thekeytechnicalbottleneckisthattherangeofdescribablecontent(i.e.,objects,属性,行动)isultimatelyconﬁnedbythesetofitemsthatcanbereliablyrecognizedbystate-of-the-artvisiontechniques.Theseconddirection,inacomplementaryavenuetotheﬁrst,hasexploredwaystomakeuseoftherichspectrumofvisualdescriptionscontributedbyonlinecitizens(Kuznetsovaetal.,2012;FengandLapata,2013;石匠,2013;Ordonezetal.,2011).Intheseapproaches,thesetofwhatcanbedescribedcanbesubstantiallylargerthanthesetofwhatcanberecognized,wheretheformerisshapedanddeﬁnedbythedata,ratherthanbyhumans.Thisallowstheresultingdescriptionstobesubstantiallymoreex-pressive,elaborate,andinterestingthanwhatwouldbepossibleinapurelybottom-upmanner.Ourworkcontributestothissecondlineofresearch.Onechallengeinutilizingnaturallyexistingmul-timodaldata,然而,isthenoisysemanticalign-mentbetweenimagesandtext(Dodgeetal.,2012;Bergetal.,2010).所以,wealsoinvesti-gatearelatedtaskofimagecaptiongeneralization(Kuznetsovaetal.,2013),whichaimstoimprovethesemanticimage-textalignmentbyremovingbitsoftextfromexistingcaptionsthatarelesslikelytobetransferabletootherimages.Thehigh-levelideaofoursystemistoharvestusefulbitsoftext(astreefragments)fromexist-ingimagedescriptionsusingdetectedvisualcontentsimilarity,andthentocomposeanewdescriptionbyselectivelycombiningtheseextracted(andop-tionallypruned)treefragments.Thisoverallideaofcompositionbasedonextractedphrasesisnotnewinitself(Kuznetsovaetal.,2012),然而,wemakeseveraltechnicalandempiricalcontributions.First,weproposeanovelstochastictreecompo-sitionalgorithmbasedonextractedtreefragmentsthatintegratesbothtreestructureandsequenceco-hesionintostructuralinference.Ouralgorithmper-mitsasubstantiallyhigherleveloflinguisticexpres-siveness,ﬂexibility,andcreativitythanthosebasedonrulesortemplates(Kulkarnietal.,2011;Yangetal.,2011;Mitchelletal.,2012),whilealsoaddress-inglong-distancegrammaticalrelationsinamoreprincipledwaythanthosebasedonhand-codedcon-straints(Kuznetsovaetal.,2012).第二,weaddressimagecaptiongeneralizationasanoptionalsubtaskofimagecaptiongeneration,andproposeatreecompressionalgorithmthatper-formsalight-weightparsingtosearchfortheop-timalsetoftreebranchestoprune.Ourworkistheﬁrsttoreportempiricalbeneﬁtsofautomaticallycompressedcaptionsforimagecaptioning.Theproposedapproachesattainsigniﬁcantlybet-terperformanceforbothimagecaptiongeneraliza-tionandgenerationtasksovercompetitivebaselinesandpreviousapproaches.Ourworkresultsinanim-provedimagecaptioncorpuswithautomaticgener-alization,whichispubliclyavailable.12HarvestingTreeFragmentsGivenaqueryimage,weretrieveimagesthatarevi-suallysimilartothequeryimage,thenextractpo-tentiallyusefulsegments(i.e.,phrases)fromtheircorrespondingimagedescriptions.Wethencom-poseanewimagedescriptionusingtheseretrievedtextfragments(§3).Extractionofusefulphrasesisguidedbybothvisualsimilarityandthesyn-tacticparseofthecorrespondingtextualdescrip-1http://ilp-cky.appspot.com/

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你

/
t

A
C
我
/

我

A
r
t
我
C
e
–
p
d

F
/

d
哦

我
/

1
0
1
1
6
2

/
t

我

A
C
_
A
_
0
0
1
8
8
1
5
6
6
9
0
5

/
t

我

A
C
_
A
_
0
0
1
8
8
p
d

乙
y
G
你
e
s
t

哦
n
0
9
S
e
p
e
米
乙
e
r
2
0
2
3

353

tion.Thisextractionstrategy,originallyproposedbyKuznetsovaetal.(2012),attemptstomakethebestuseoflinguisticregularitieswithrespecttoobjects,行动,andscenes,makingitpossibletoobtainrichertextualdescriptionsthanwhatcur-rentstate-of-the-artvisiontechniquescanprovideinisolation.InallofourexperimentsweusethecaptionedimagecorpusofOrdonezetal.(2011),ﬁrstpre-processingthecorpusforrelevantcontentbyrunningdeformablepartmodelobjectdetec-tors(Felzenszwalbetal.,2010).Forourstudy,werundetectorsfor89objectclassessetahighconﬁ-dencethresholdfordetection.AsillustratedinFigure1,foraqueryimagede-tection,weextractfourtypesofphrases(astreefragments).第一的,weretrieverelevantnounphrasesfromimageswithvisuallysimilarobjectdetections.Weusecolor,texture(LeungandMalik,1999),andshape(DalalandTriggs,2005;Lowe,2004)basedfeaturesencodedinahistogramofvectorquantizedresponsestomeasurevisualsimilarity.Second,weextractverbphrasesforwhichthecorrespondingnounphrasetakesthesubjectrole.Third,fromthoseimageswith“stuff”detections,e.g.“water”,or“sky”(typicallymassnouns),weextractpreposi-tionalphrasesbasedonsimilarityofbothvisualap-pearanceandrelativespatialrelationshipsbetweendetectedobjectsand“stuff”.Finally,weuseglobal“scene”similarity2toextractprepositionalphrasesreferringtotheoverallscene,e.g.,“attheconfer-ence,”or“inthemarket”.Weperformthisphraseretrievalprocessforeachdetectedobjectinthequeryimageandgenerateonesentenceforeachobject.Allsentencesarethencombinedtogethertoproducetheﬁnaldescription.Optionally,weapplyimagecaptiongeneralization(viacompression)(§4)toallcaptionsinthecorpuspriortothephraseextractionandcomposition.3TreeCompositionWemodeltreecompositionasconstraintoptimiza-tion.Theinputtoouralgorithmisthesetofre-trievedphrases(i.e.,treefragments),asillustratedin§2.LetP={p0,…,pL−1}bethesetofallphrasesacrossthefourphrasetypes(物体,ac-tions,stuffandscene).Weassumeamappingfunc-2L2distancebetweenclassiﬁcationscorevectors(Xiaoetal.,2010)tionpt:[0,L)→T,whereTisthesetofphrasetypes,sothatthephrasetypeofpiispt(我).Inad-dition,letRbethesetofPCFGproductionrulesandNTbethesetofnonterminalsymbolsofthePCFG.Thegoalistoﬁndandcombineagoodse-quenceofphrasesG,|G|≤|时间|=N=4,drawnfromP,intoaﬁnalsentence.Moreconcretely,wewanttoselectandorderasubsetofphrases(atmostonephraseofeachphrasetype)whileconsideringboththeparsestructureandn-gramcohesionacrossphrasalboundaries.Figure2showsasimpliﬁedexampleofacom-posedsentencewithitscorrespondingparsestruc-ture.Forbrevity,theﬁgureshowsonlyonephraseforeachphrasetype,butinactualitytherewouldbeasetofcandidatephrasesforeachtype.Figure3showstheCKY-stylerepresentationoftheinternalmechanicsofconstraintoptimizationfortheexam-plecompositionfromFigure2.EachcellijoftheCKYmatrixcorrespondstoGij,asubsequenceofGstartingatpositioniandendingatpositionj.IfacellintheCKYmatrixislabeledwithanontermi-nalsymbols,itmeansthatthecorrespondingtreeofGijhassasitsroot.AlthoughwevisualizetheoperationusingaCKY-stylerepresentationinFigure3,notethatcomposi-tionrequiresmorecomplexcombinatorialdecisionsthanCKYparsingduetotwoadditionalconsidera-tions.Weare:(1)selectingasubsetofcandidatephrases,和(2)re-orderingtheselectedphrases(hencemakingtheproblemNP-hard).所以,weencodeourproblemusingIntegerLinearPro-gramming(ILP)(RothandtauYih,2004;ClarkeandLapata,2008)andusetheCPLEX(ILOG,Inc,2006)solver.3.1ILPVariablesVariablesforSequenceStructure:Variablesαen-codephraseselectionandordering:αik=1iffphrasei∈Pisselected(1)forpositionk∈[0,氮)WherekisoneoftheN=4positionsinasentence.3Additionally,wedeﬁnevariablesforeachpairofad-jacentphrasestocapturesequencecohesion:3Thenumberofpositionsisequaltothenumberofphrasetypes,sinceweselectatmostonefromeachtype.

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你

/
t

A
C
我
/

我

A
r
t
我
C
e
–
p
d

F
/

d
哦

我
/

1
0
1
1
6
2

/
t

我

A
C
_
A
_
0
0
1
8
8
1
5
6
6
9
0
5

/
t

我

A
C
_
A
_
0
0
1
8
8
p
d

乙
y
G
你
e
s
t

哦
n
0
9
S
e
p
e
米
乙
e
r
2
0
2
3

354

A”cow in”这”countryside was”staring”在”me in#the#grass NP PP VP PP NP S i=0$j=2$k=1$0123levelandeachnodeofthatlevel,algorithmhastodecide,whichparsetagtochoose.Thisprocessisrepresentedbyassignmentofaparticulartagtoamatrixcell.Thechosentagmustbeaheadofarule,ﬁexamplecell12isassignedtagVP,correspond-ingtoruleVP!VPPP.Thisruleconnectsleafs“goingouttosea”and“intheocean”.Theprob-lemistoﬁndtagassignmentforeachcellofthema-trix,givensomecellscanbeempty,iftheydonotconnectchildrencells.lattercorrespondtochildrenbranchesofthetreeandbelongtothepreviousdiag-onalintheleft-to-rightorder.Alsowedonottryallpossiblepairs5ofchildrenfrompreviousdiagonal.WeusetechniquesimilartotheoneusedinCKYparsingapproach.Matrixcellpairscorrespondingtochildrenpairsare,wherek2[我,j).Hereandfortheremainderofthepaper,notation[我,j)方法{我,i+1,…,j1}andrish pqunlessotherwisestated.Theproblemofchoosingphraseordertogetherwiththebestparsetreeofthedescriptionisacom-plexoptimizationproblem,whichwesolveusingIntegerLinearProgramming(ILP).Weuseasepa-rateILPformulationforforsentencereorderingandsalientobjectselection,whichweomitforbrevity.Asmentionedearlier,overallforeachobjectwehavefourtypesofphrases.WeuseCKY-drivenILPformulationtocombinethemtogetherintoaplausi-bledescriptionswhichobeysPCFGrules.FortheremainderofthepaperwewillcallourILPformu-lationILP-TREE.WeexploitCplex(ILOG,Inc,2006)tosolveILPproblem.Todo:[mentioncplexparameters.Forinstance,30seclimitongeneration]3.0.2ILPvariablesPhraseIndicatorVariables:Wedeﬁnevariables↵whichindicatephraseselectionandphraseordering.↵ijk=1iffphraseioftypej(1)isselectedforpositionk5ThereisonlytwochildrenasweuseChomskyNormalForm↵ij0=1↵ij1pq2=102S=1010(NP!NPPP)=1021=1Wherek2[0,氮)Todo:[checkforthewholepa-perifkrangesfrom0]indexesoneofN=4positionsinasentence6.Phraseorderingiscapturedbyindicatorvariablesforadjacentpairsofphrases:↵ijkpq(k+1)=1iff↵ijk=↵pq(k+1)=1(2)AnexampleofILP-CKYatFigure3showsselec-tionofphrasesandtheirordering:“Thelittleboat”,“goingouttosea”and“intheocean”.TreeIndicatorVariables:Wealsodeﬁnevariables,whichareindicatorsofCKYmatrixcontent(Fig-ure3).ijs=1iffcellijofthematrixisassigned(3)parsetreesymbolsTodo:[Renamesymbolstotagsthroughoutthepa-per]Wherei2[0,氮)indexesCKYmatrixdiagonalsandj2[0,Ni)indexeselementsofdiagonali.InordertomodelruleselectionateachCKYstep,wedeﬁnevariables,whichcorrespondtoaPCFGruleusedatthegivencellijofCKYmatrix:ijkr=1iffijh=ikp(4)=(k+1)jq=1,Wherer=h pq2Randk2[我,j).Valuekcorrespondstothechoiceofchildrenforthecurrentcell.6ThenumberofpositionsisequaltothenumberofphrasetypesFigure2:Anexamplescenariooftreecomposition.Onlytheﬁrstthreephrasesarechosenforthecomposition.αijk=1iffαik=αj(k+1)=1(2)VariablesforTreeStructure:Variablesβencodetheparsestructure:βijs=1iffthephrasesequenceGij(3)mapstothenonterminalsymbols∈NTWherei∈[0,氮)andj∈[我,氮)indexrowsandcolumnsoftheCKY-stylematrixinFigure3.Acor-respondingexampletreeisshowninFigure2,wherethephrasesequenceG02correspondstothecellla-beledwithS.WealsodeﬁnevariablestoindicateselectedPCFGrulesintheresultingparse:βijkr=1iffβijh=βikp(4)=β(k+1)jq=1,Wherer=h→pq∈Randk∈[我,j).IndexkpointstotheboundaryofsplitbetweentwochildrenasshowninFigure2forthesequenceG02.AuxiliaryVariables:Fornotationalconvenience,wealsoinclude:γijk=1iffPs∈NTβijs(5)=Ps∈NTβiks=Ps∈NTβ(k+1)js=13.2ILPObjectiveFunctionWemodeltreecompositionasmaximizationofthefollowingobjectivefunction:F=XiFi×N−1Xk=0αik(6)+XijFij×N−2Xk=0αijk+Xijj−1Xk=iXr∈RFr×βijkrNP NP S A”cow PP PP-VP in”这”countryside VP VP was”staring”在”me PP in#the#grass 00″01″02″03″11″12″13″33″22″23″k=1$k=0$levelandeachnodeofthatlevel,algorithmhastodecide,whichparsetagtochoose.Thisprocessisrepresentedbyassignmentofaparticulartagtoamatrixcell.Thechosentagmustbeaheadofarule,ﬁexamplecell12isassignedtagVP,correspond-ingtoruleVP!VPPP.Thisruleconnectsleafs“goingouttosea”and“intheocean”.Theprob-lemistoﬁndtagassignmentforeachcellofthema-trix,givensomecellscanbeempty,iftheydonotconnectchildrencells.lattercorrespondtochildrenbranchesofthetreeandbelongtothepreviousdiag-onalintheleft-to-rightorder.Alsowedonottryallpossiblepairs5ofchildrenfrompreviousdiagonal.WeusetechniquesimilartotheoneusedinCKYparsingapproach.Matrixcellpairscorrespondingtochildrenpairsare,wherek2[我,j).Hereandfortheremainderofthepaper,notation[我,j)方法{我,i+1,…,j1}andrish pqunlessotherwisestated.Theproblemofchoosingphraseordertogetherwiththebestparsetreeofthedescriptionisacom-plexoptimizationproblem,whichwesolveusingIntegerLinearProgramming(ILP).Weuseasepa-rateILPformulationforforsentencereorderingandsalientobjectselection,whichweomitforbrevity.Asmentionedearlier,overallforeachobjectwehavefourtypesofphrases.WeuseCKY-drivenILPformulationtocombinethemtogetherintoaplausi-bledescriptionswhichobeysPCFGrules.FortheremainderofthepaperwewillcallourILPformu-lationILP-TREE.WeexploitCplex(ILOG,Inc,2006)tosolveILPproblem.Todo:[mentioncplexparameters.Forinstance,30seclimitongeneration]3.0.2ILPvariablesPhraseIndicatorVariables:Wedeﬁnevariables↵whichindicatephraseselectionandphraseordering.↵ijk=1iffphraseioftypej(1)isselectedforpositionk5ThereisonlytwochildrenasweuseChomskyNormalForm↵ij0=1↵ij1pq2=102S=1010(NP!NPPP)=1021=1Wherek2[0,氮)Todo:[checkforthewholepa-perifkrangesfrom0]indexesoneofN=4positionsinasentence6.Phraseorderingiscapturedbyindicatorvariablesforadjacentpairsofphrases:↵ijkpq(k+1)=1iff↵ijk=↵pq(k+1)=1(2)AnexampleofILP-CKYatFigure3showsselec-tionofphrasesandtheirordering:“Thelittleboat”,“goingouttosea”and“intheocean”.TreeIndicatorVariables:Wealsodeﬁnevariables,whichareindicatorsofCKYmatrixcontent(Fig-ure3).ijs=1iffcellijofthematrixisassigned(3)parsetreesymbolsTodo:[Renamesymbolstotagsthroughoutthepa-per]Wherei2[0,氮)indexesCKYmatrixdiagonalsandj2[0,Ni)indexeselementsofdiagonali.InordertomodelruleselectionateachCKYstep,wedeﬁnevariables,whichcorrespondtoaPCFGruleusedatthegivencellijofCKYmatrix:ijkr=1iffijh=ikp(4)=(k+1)jq=1,Wherer=h pq2Randk2[我,j).Valuekcorrespondstothechoiceofchildrenforthecurrentcell.6Thenumberofpositionsisequaltothenumberofphrasetypeslevelandeachnodeofthatlevel,algorithmhastodecide,whichparsetagtochoose.Thisprocessisrepresentedbyassignmentofaparticulartagtoamatrixcell.Thechosentagmustbeaheadofarule,ﬁexamplecell12isassignedtagVP,correspond-ingtoruleVP!VPPP.Thisruleconnectsleafs“goingouttosea”and“intheocean”.Theprob-lemistoﬁndtagassignmentforeachcellofthema-trix,givensomecellscanbeempty,iftheydonotconnectchildrencells.lattercorrespondtochildrenbranchesofthetreeandbelongtothepreviousdiag-onalintheleft-to-rightorder.Alsowedonottryallpossiblepairs5ofchildrenfrompreviousdiagonal.WeusetechniquesimilartotheoneusedinCKYparsingapproach.Matrixcellpairscorrespondingtochildrenpairsare,wherek2[我,j).Hereandfortheremainderofthepaper,notation[我,j)方法{我,i+1,…,j1}andrish pqunlessotherwisestated.Theproblemofchoosingphraseordertogetherwiththebestparsetreeofthedescriptionisacom-plexoptimizationproblem,whichwesolveusingIntegerLinearProgramming(ILP).Weuseasepa-rateILPformulationforforsentencereorderingandsalientobjectselection,whichweomitforbrevity.Asmentionedearlier,overallforeachobjectwehavefourtypesofphrases.WeuseCKY-drivenILPformulationtocombinethemtogetherintoaplausi-bledescriptionswhichobeysPCFGrules.FortheremainderofthepaperwewillcallourILPformu-lationILP-TREE.WeexploitCplex(ILOG,Inc,2006)tosolveILPproblem.Todo:[mentioncplexparameters.Forinstance,30seclimitongeneration]3.0.2ILPvariablesPhraseIndicatorVariables:Wedeﬁnevariables↵whichindicatephraseselectionandphraseordering.↵ijk=1iffphraseioftypej(1)isselectedforpositionk5ThereisonlytwochildrenasweuseChomskyNormalForm↵ij0=1↵ij1pq2=102S=1010(NP!NPPP)=1021=1Wherek2[0,氮)Todo:[checkforthewholepa-perifkrangesfrom0]indexesoneofN=4positionsinasentence6.Phraseorderingiscapturedbyindicatorvariablesforadjacentpairsofphrases:↵ijkpq(k+1)=1iff↵ijk=↵pq(k+1)=1(2)AnexampleofILP-CKYatFigure3showsselec-tionofphrasesandtheirordering:“Thelittleboat”,“goingouttosea”and“intheocean”.TreeIndicatorVariables:Wealsodeﬁnevariables,whichareindicatorsofCKYmatrixcontent(Fig-ure3).ijs=1iffcellijofthematrixisassigned(3)parsetreesymbolsTodo:[Renamesymbolstotagsthroughoutthepa-per]Wherei2[0,氮)indexesCKYmatrixdiagonalsandj2[0,Ni)indexeselementsofdiagonali.InordertomodelruleselectionateachCKYstep,wedeﬁnevariables,whichcorrespondtoaPCFGruleusedatthegivencellijofCKYmatrix:ijkr=1iffijh=ikp(4)=(k+1)jq=1,Wherer=h pq2Randk2[我,j).Valuekcorrespondstothechoiceofchildrenforthecurrentcell.6Thenumberofpositionsisequaltothenumberofphrasetypeslevelandeachnodeofthatlevel,algorithmhastodecide,whichparsetagtochoose.Thisprocessisrepresentedbyassignmentofaparticulartagtoamatrixcell.Thechosentagmustbeaheadofarule,ﬁexamplecell12isassignedtagVP,correspond-ingtoruleVP!VPPP.Thisruleconnectsleafs“goingouttosea”and“intheocean”.Theprob-lemistoﬁndtagassignmentforeachcellofthema-trix,givensomecellscanbeempty,iftheydonotconnectchildrencells.lattercorrespondtochildrenbranchesofthetreeandbelongtothepreviousdiag-onalintheleft-to-rightorder.Alsowedonottryallpossiblepairs5ofchildrenfrompreviousdiagonal.WeusetechniquesimilartotheoneusedinCKYparsingapproach.Matrixcellpairscorrespondingtochildrenpairsare,wherek2[我,j).Hereandfortheremainderofthepaper,notation[我,j)方法{我,i+1,…,j1}andrish pqunlessotherwisestated.Theproblemofchoosingphraseordertogetherwiththebestparsetreeofthedescriptionisacom-plexoptimizationproblem,whichwesolveusingIntegerLinearProgramming(ILP).Weuseasepa-rateILPformulationforforsentencereorderingandsalientobjectselection,whichweomitforbrevity.Asmentionedearlier,overallforeachobjectwehavefourtypesofphrases.WeuseCKY-drivenILPformulationtocombinethemtogetherintoaplausi-bledescriptionswhichobeysPCFGrules.FortheremainderofthepaperwewillcallourILPformu-lationILP-TREE.WeexploitCplex(ILOG,Inc,2006)tosolveILPproblem.Todo:[mentioncplexparameters.Forinstance,30seclimitongeneration]3.0.2ILPvariablesPhraseIndicatorVariables:Wedeﬁnevariables↵whichindicatephraseselectionandphraseordering.↵ijk=1iffphraseioftypej(1)isselectedforpositionk5ThereisonlytwochildrenasweuseChomskyNormalForm↵ij0=1↵ij1pq2=102S=1010(NP!NPPP)=1021=1Wherek2[0,氮)Todo:[checkforthewholepa-perifkrangesfrom0]indexesoneofN=4positionsinasentence6.Phraseorderingiscapturedbyindicatorvariablesforadjacentpairsofphrases:↵ijkpq(k+1)=1iff↵ijk=↵pq(k+1)=1(2)AnexampleofILP-CKYatFigure3showsselec-tionofphrasesandtheirordering:“Thelittleboat”,“goingouttosea”and“intheocean”.TreeIndicatorVariables:Wealsodeﬁnevariables,whichareindicatorsofCKYmatrixcontent(Fig-ure3).ijs=1iffcellijofthematrixisassigned(3)parsetreesymbolsTodo:[Renamesymbolstotagsthroughoutthepa-per]Wherei2[0,氮)indexesCKYmatrixdiagonalsandj2[0,Ni)indexeselementsofdiagonali.InordertomodelruleselectionateachCKYstep,wedeﬁnevariables,whichcorrespondtoaPCFGruleusedatthegivencellijofCKYmatrix:ijkr=1iffijh=ikp(4)=(k+1)jq=1,Wherer=h pq2Randk2[我,j).Valuekcorrespondstothechoiceofchildrenforthecurrentcell.6Thenumberofpositionsisequaltothenumberofphrasetypesk=0$oftwovariableshavebeendiscussedbyClarkeandLapata(2008).ForEquation2,weaddthefollow-ingconstraints(similarconstraintsarealsoaddedforEquations4,5).8ijkpqm,↵ijk↵ik(7)↵ijk↵j(k+1)↵ijk+(1↵ik)+(1↵j(k+1))1ConsistencybetweenTreeLeafsandSequences:Theorderingofphrasesimpliedby↵ijkmustbeconsistentwiththeorderingofphrasesimpliedbythevariables.Thiscanbeachievedbyaligningtheleafcells(i.e.,kks)intheCKY-stylematrixwith↵variablesasfollows:8ik,↵ikXs2Sikks(8)8k,Xi↵ik=Xs2Skks(9)WhereSireferstothesetofPCFGnonterminalsthatarecompatiblewiththephrasetypeofpi.Forexample,Si={NN,NP,…}ifpicorrespondstoan“object”(noun-phrase).因此,Equation8en-forcesthecorrespondencebetweenphrasetypesandnonterminalsymbolsatthetreeleafs.Equation9enforcestheconstraintthatthenumberofselectedphrasesandinstantiatedtreeleafsmustbethesame.TreeCongruenceConstraints:ToensurethateachCKYcellhasatmostonesymbolwerequire8ij,Xs2Sijs1(10)Wealsorequirethat8i,j>i,H,ijh=j1Xk=iXr2Rhijkr(11)WhereRh={r2R:r=h!pq}.Weenforcetheseconstraintsonlyfornon-leafs.Thisconstraintforbidsinstantiationswhereanonterminalsymbolhisselectedforcellijwithoutselectingacorrespond-ingPCFGrule.Wealsoensurethatweproduceavalidtreestruc-ture.Forinstance,ifweselect3phrasesasshowninFigure3,wemusthavetherootofthetreeatthecorrespondingcell02.8k2[1,氮),Xs2SkksN1Xt=kXs2S0ts(12)Wealsorequirecellsthatarenotselectedfortheresultingparsestructuretobeempty:8ijXkijk1(13)↵i0=1(14)↵ij1=1(15)此外,wepenalizesolutionswithouttheStagattheparserootasasoft-constraint.MiscellaneousConstraints:最后,weincludeseveralconstraintstoavoiddegeneratesolutionsorotherwisetoenhancethecomposedoutput:(1)en-forcethatanoun-phraseisselected(toensurese-manticrelevancetotheimagecontent),(2)allowatmostonephraseofeachtype,(3)donotallowmul-tiplephraseswithidenticalheadwords(toavoidre-dundancy),(4)allowatmostonescenephraseforallsentencesinthedescription.Weﬁndthathan-dlingofsentenceboundariesisimportantiftheILPformulationisbasedonlyonsequencestructure,butwiththeintegrationoftree-basedstructure,weneednothandlesentenceboundaries.3.4DiscussionAninterestingaspectofdescriptiongenerationex-ploredinthispaperisthatbuildingblocksofcom-positionaretreefragments,ratherthanindividualwords.Therearethreepracticalbeneﬁts:(1)syn-tacticandsemanticexpressiveness,(2)correctness,和(3)computationalefﬁciency.Becauseweex-tractnicesegmentsfromhumanwrittencaptions,weareabletouseexpressivelanguage,andlesslikelytomakesyntacticorsemanticerrors.Ourphraseextractionprocesscanbeviewedatahighlevelasvisually-groundedorvisually-situatedparaphrasing.Also,becausetheunitofoperationistreefrag-ments,theILPformulationencodedinthisworkiscomputationallylightweight.Iftheunitofcompo-sitionwaswords,theILPinstanceswouldbesig-niﬁcantlymorecomputationallyintensive,andmorelikelytosufferfromgrammaticalandsemanticer-rors.oftwovariableshavebeendiscussedbyClarkeandLapata(2008).ForEquation2,weaddthefollow-ingconstraints(similarconstraintsarealsoaddedforEquations4,5).8ijkpqm,↵ijk↵ik(7)↵ijk↵j(k+1)↵ijk+(1↵ik)+(1↵j(k+1))1ConsistencybetweenTreeLeafsandSequences:Theorderingofphrasesimpliedby↵ijkmustbeconsistentwiththeorderingofphrasesimpliedbythevariables.Thiscanbeachievedbyaligningtheleafcells(i.e.,kks)intheCKY-stylematrixwith↵variablesasfollows:8ik,↵ikXs2Sikks(8)8k,Xi↵ik=Xs2Skks(9)WhereSireferstothesetofPCFGnonterminalsthatarecompatiblewiththephrasetypeofpi.Forexample,Si={NN,NP,…}ifpicorrespondstoan“object”(noun-phrase).因此,Equation8en-forcesthecorrespondencebetweenphrasetypesandnonterminalsymbolsatthetreeleafs.Equation9enforcestheconstraintthatthenumberofselectedphrasesandinstantiatedtreeleafsmustbethesame.TreeCongruenceConstraints:ToensurethateachCKYcellhasatmostonesymbolwerequire8ij,Xs2Sijs1(10)Wealsorequirethat8i,j>i,H,ijh=j1Xk=iXr2Rhijkr(11)WhereRh={r2R:r=h!pq}.Weenforcetheseconstraintsonlyfornon-leafs.Thisconstraintforbidsinstantiationswhereanonterminalsymbolhisselectedforcellijwithoutselectingacorrespond-ingPCFGrule.Wealsoensurethatweproduceavalidtreestruc-ture.Forinstance,ifweselect3phrasesasshowninFigure3,wemusthavetherootofthetreeatthecorrespondingcell02.8k2[1,氮),Xs2SkksN1Xt=kXs2S0ts(12)Wealsorequirecellsthatarenotselectedfortheresultingparsestructuretobeempty:8ijXkijk1(13)↵i0=1(14)↵ij1=1(15)此外,wepenalizesolutionswithouttheStagattheparserootasasoft-constraint.MiscellaneousConstraints:最后,weincludeseveralconstraintstoavoiddegeneratesolutionsorotherwisetoenhancethecomposedoutput:(1)en-forcethatanoun-phraseisselected(toensurese-manticrelevancetotheimagecontent),(2)allowatmostonephraseofeachtype,(3)donotallowmul-tiplephraseswithidenticalheadwords(toavoidre-dundancy),(4)allowatmostonescenephraseforallsentencesinthedescription.Weﬁndthathan-dlingofsentenceboundariesisimportantiftheILPformulationisbasedonlyonsequencestructure,butwiththeintegrationoftree-basedstructure,weneednothandlesentenceboundaries.3.4DiscussionAninterestingaspectofdescriptiongenerationex-ploredinthispaperisthatbuildingblocksofcom-positionaretreefragments,ratherthanindividualwords.Therearethreepracticalbeneﬁts:(1)syn-tacticandsemanticexpressiveness,(2)correctness,和(3)computationalefﬁciency.Becauseweex-tractnicesegmentsfromhumanwrittencaptions,weareabletouseexpressivelanguage,andlesslikelytomakesyntacticorsemanticerrors.Ourphraseextractionprocesscanbeviewedatahighlevelasvisually-groundedorvisually-situatedparaphrasing.Also,becausetheunitofoperationistreefrag-ments,theILPformulationencodedinthisworkiscomputationallylightweight.Iftheunitofcompo-sitionwaswords,theILPinstanceswouldbesig-niﬁcantlymorecomputationallyintensive,andmorelikelytosufferfromgrammaticalandsemanticer-rors.oftwovariableshavebeendiscussedbyClarkeandLapata(2008).ForEquation2,weaddthefollow-ingconstraints(similarconstraintsarealsoaddedforEquations4,5).8ijkpqm,↵ijk↵ik(7)↵ijk↵j(k+1)↵ijk+(1↵ik)+(1↵j(k+1))1ConsistencybetweenTreeLeafsandSequences:Theorderingofphrasesimpliedby↵ijkmustbeconsistentwiththeorderingofphrasesimpliedbythevariables.Thiscanbeachievedbyaligningtheleafcells(i.e.,kks)intheCKY-stylematrixwith↵variablesasfollows:8ik,↵ikXs2Sikks(8)8k,Xi↵ik=Xs2Skks(9)WhereSireferstothesetofPCFGnonterminalsthatarecompatiblewiththephrasetypeofpi.Forexample,Si={NN,NP,…}ifpicorrespondstoan“object”(noun-phrase).因此,Equation8en-forcesthecorrespondencebetweenphrasetypesandnonterminalsymbolsatthetreeleafs.Equation9enforcestheconstraintthatthenumberofselectedphrasesandinstantiatedtreeleafsmustbethesame.TreeCongruenceConstraints:ToensurethateachCKYcellhasatmostonesymbolwerequire8ij,Xs2Sijs1(10)Wealsorequirethat8i,j>i,H,ijh=j1Xk=iXr2Rhijkr(11)WhereRh={r2R:r=h!pq}.Weenforcetheseconstraintsonlyfornon-leafs.Thisconstraintforbidsinstantiationswhereanonterminalsymbolhisselectedforcellijwithoutselectingacorrespond-ingPCFGrule.Wealsoensurethatweproduceavalidtreestruc-ture.Forinstance,ifweselect3phrasesasshowninFigure3,wemusthavetherootofthetreeatthecorrespondingcell02.8k2[1,氮),Xs2SkksN1Xt=kXs2S0ts(12)Wealsorequirecellsthatarenotselectedfortheresultingparsestructuretobeempty:8ijXkijk1(13)Fi(14)Fij(15)此外,wepenalizesolutionswithouttheStagattheparserootasasoft-constraint.MiscellaneousConstraints:最后,weincludeseveralconstraintstoavoiddegeneratesolutionsorotherwisetoenhancethecomposedoutput:(1)en-forcethatanoun-phraseisselected(toensurese-manticrelevancetotheimagecontent),(2)allowatmostonephraseofeachtype,(3)donotallowmul-tiplephraseswithidenticalheadwords(toavoidre-dundancy),(4)allowatmostonescenephraseforallsentencesinthedescription.Weﬁndthathan-dlingofsentenceboundariesisimportantiftheILPformulationisbasedonlyonsequencestructure,butwiththeintegrationoftree-basedstructure,weneednothandlesentenceboundaries.3.4DiscussionAninterestingaspectofdescriptiongenerationex-ploredinthispaperisthatbuildingblocksofcom-positionaretreefragments,ratherthanindividualwords.Therearethreepracticalbeneﬁts:(1)syn-tacticandsemanticexpressiveness,(2)correctness,和(3)computationalefﬁciency.Becauseweex-tractnicesegmentsfromhumanwrittencaptions,weareabletouseexpressivelanguage,andlesslikelytomakesyntacticorsemanticerrors.Ourphraseextractionprocesscanbeviewedatahighlevelasvisually-groundedorvisually-situatedparaphrasing.Also,becausetheunitofoperationistreefrag-ments,theILPformulationencodedinthisworkiscomputationallylightweight.Iftheunitofcompo-sitionwaswords,theILPinstanceswouldbesig-niﬁcantlymorecomputationallyintensive,andmorelikelytosufferfromgrammaticalandsemanticer-rors.oftwovariableshavebeendiscussedbyClarkeandLapata(2008).ForEquation2,weaddthefollow-ingconstraints(similarconstraintsarealsoaddedforEquations4,5).8ijkpqm,↵ijk↵ik(7)↵ijk↵j(k+1)↵ijk+(1↵ik)+(1↵j(k+1))1ConsistencybetweenTreeLeafsandSequences:Theorderingofphrasesimpliedby↵ijkmustbeconsistentwiththeorderingofphrasesimpliedbythevariables.Thiscanbeachievedbyaligningtheleafcells(i.e.,kks)intheCKY-stylematrixwith↵variablesasfollows:8ik,↵ikXs2Sikks(8)8k,Xi↵ik=Xs2Skks(9)WhereSireferstothesetofPCFGnonterminalsthatarecompatiblewiththephrasetypeofpi.Forexample,Si={NN,NP,…}ifpicorrespondstoan“object”(noun-phrase).因此,Equation8en-forcesthecorrespondencebetweenphrasetypesandnonterminalsymbolsatthetreeleafs.Equation9enforcestheconstraintthatthenumberofselectedphrasesandinstantiatedtreeleafsmustbethesame.TreeCongruenceConstraints:ToensurethateachCKYcellhasatmostonesymbolwerequire8ij,Xs2Sijs1(10)Wealsorequirethat8i,j>i,H,ijh=j1Xk=iXr2Rhijkr(11)WhereRh={r2R:r=h!pq}.Weenforcetheseconstraintsonlyfornon-leafs.Thisconstraintforbidsinstantiationswhereanonterminalsymbolhisselectedforcellijwithoutselectingacorrespond-ingPCFGrule.Wealsoensurethatweproduceavalidtreestruc-ture.Forinstance,ifweselect3phrasesasshowninFigure3,wemusthavetherootofthetreeatthecorrespondingcell02.8k2[1,氮),Xs2SkksN1Xt=kXs2S0ts(12)Wealsorequirecellsthatarenotselectedfortheresultingparsestructuretobeempty:8ijXkijk1(13)Fi(14)Fij(15)此外,wepenalizesolutionswithouttheStagattheparserootasasoft-constraint.MiscellaneousConstraints:最后,weincludeseveralconstraintstoavoiddegeneratesolutionsorotherwisetoenhancethecomposedoutput:(1)en-forcethatanoun-phraseisselected(toensurese-manticrelevancetotheimagecontent),(2)allowatmostonephraseofeachtype,(3)donotallowmul-tiplephraseswithidenticalheadwords(toavoidre-dundancy),(4)allowatmostonescenephraseforallsentencesinthedescription.Weﬁndthathan-dlingofsentenceboundariesisimportantiftheILPformulationisbasedonlyonsequencestructure,butwiththeintegrationoftree-basedstructure,weneednothandlesentenceboundaries.3.4DiscussionAninterestingaspectofdescriptiongenerationex-ploredinthispaperisthatbuildingblocksofcom-positionaretreefragments,ratherthanindividualwords.Therearethreepracticalbeneﬁts:(1)syn-tacticandsemanticexpressiveness,(2)correctness,和(3)computationalefﬁciency.Becauseweex-tractnicesegmentsfromhumanwrittencaptions,weareabletouseexpressivelanguage,andlesslikelytomakesyntacticorsemanticerrors.Ourphraseextractionprocesscanbeviewedatahighlevelasvisually-groundedorvisually-situatedparaphrasing.Also,becausetheunitofoperationistreefrag-ments,theILPformulationencodedinthisworkiscomputationallylightweight.Iftheunitofcompo-sitionwaswords,theILPinstanceswouldbesig-niﬁcantlymorecomputationallyintensive,andmorelikelytosufferfromgrammaticalandsemanticer-rors.Figure3:CKY-stylerepresentationofdecisionvariablesasdeﬁnedin§3.1forthetreeexampleinFig2.Non-terminalsymbolsinboldface(inblue)andsolidarrows(alsoinblue)representthechosenPCFGrulestocom-binetheselectedsetofphrases.Nonterminalsymbolsinsmallerfont(inred)anddottedarrows(alsoinred)rep-resentpossibleotherchoicesthatarenotselected.Thisobjectiveiscomprisedofthreetypesofweights(conﬁdencescores):Fi,Fij,Fr.4Firepresentsthephraseselectionscorebasedonvisualsimilarity,de-scribedin§2.Fijquantiﬁesthesequencecohe-sionacrossphraseboundaries.Forthis,weusen-gramscores(n∈[2,5])betweenadjacentphrasescomputedusingtheGoogleWeb1-Tcorpus(BrantsandFranz.,2006).最后,FrquantiﬁesPCFGrulescores(logprobabilities)estimatedfromthe1Mim-agecaptioncorpus(Ordonezetal.,2011)parsedus-ingtheStanfordparser(KleinandManning,2003).OnecanviewFiasacontentselectionscore,whileFijandFrcorrespondtolinguisticﬂuencyscorescapturingsequenceandtreestructurerespec-tively.Ifwesetpositivevaluesforalloftheseweights,theoptimizationfunctionwouldbebiasedtowardverboseproduction,sinceselectinganaddi-tionalphrasewillincreasetheobjectivefunction.Tocontrolforverbosity,wesetscorescorrespondingtolinguisticﬂuency,i.e.,FijandFrusingnegativevalues(smallerabsolutevaluesforhigherﬂuency),tobalancedynamicsbetweencontentselectionandlinguisticﬂuency.3.3ILPConstraintsSoundnessConstraints:Weneedconstraintstoenforceconsistencybetweendifferenttypesofvari-4Allweightsarenormalizedusingz-score.

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你

/
t

A
C
我
/

我

A
r
t
我
C
e
–
p
d

F
/

d
哦

我
/

1
0
1
1
6
2

/
t

我

A
C
_
A
_
0
0
1
8
8
1
5
6
6
9
0
5

/
t

我

A
C
_
A
_
0
0
1
8
8
p
d

乙
y
G
你
e
s
t

哦
n
0
9
S
e
p
e
米
乙
e
r
2
0
2
3

355

埃布尔斯(Equations2,4,5).ConstraintsforaproductoftwovariableshavebeendiscussedbyClarkeandLapata(2008).ForEquation2,weaddthefollow-ingconstraints(similarconstraintsarealsoaddedforEquations4,5).∀ijk,αijk≤αik(7)αijk≤αj(k+1)αijk+(1−αik)+(1−αj(k+1))≥1ConsistencybetweenTreeLeafsandSequences:Theorderingofphrasesimpliedbyαijkmustbeconsistentwiththeorderingofphrasesimpliedbytheβvariables.Thiscanbeachievedbyaligningtheleafcells(i.e.,βkks)intheCKY-stylematrixwithαvariablesasfollows:∀ik,αik≤Xs∈NTiβkks(8)∀k,Xiαik=Xs∈NTβkks(9)WhereNTireferstothesetofPCFGnonterminalsthatarecompatiblewithaphrasetypept(我)ofpi.Forexample,NTi={NN,NP,…}ifpicorrespondstoan“object”(noun-phrase).因此,Equation8en-forcesthecorrespondencebetweenphrasetypesandnonterminalsymbolsatthetreeleafs.Equation9enforcestheconstraintthatthenumberofselectedphrasesandinstantiatedtreeleafsmustbethesame.TreeCongruenceConstraints:ToensurethateachCKYcellhasatmostonesymbolwerequire∀ij,Xs∈NTβijs≤1(10)Wealsorequirethat∀i,j>i,H,βijh=j−1Xk=iXr∈Rhβijkr(11)WhereRh={r∈R:r=h→pq}.Weenforcetheseconstraintsonlyfornon-leafs.Thisconstraintforbidsinstantiationswhereanonterminalsymbolhisselectedforcellijwithoutselectingacorrespond-ingPCFGrule.Wealsoensurethatweproduceavalidtreestruc-ture.Forinstance,ifweselect3phrasesasshowninFigure3,wemusthavetherootofthetreeatthecorrespondingcell02.∀k∈[1,氮),Xs∈NTβkks≤N−1Xt=kXs∈NTβ0ts(12)Wealsorequirecellsthatarenotselectedfortheresultingparsestructuretobeempty:∀ijXkγijk≤1(13)此外,wepenalizesolutionswithouttheStagattheparserootasasoft-constraint.MiscellaneousConstraints:最后,weincludeseveralconstraintstoavoiddegeneratesolutionsortootherwiseenhancethecomposedoutput.We:(1)enforcethatanoun-phraseisselected(toensurese-manticrelevancetotheimagecontent),(2)allowatmostonephraseofeachtype,(3)donotallowmul-tiplephraseswithidenticalheadwords(toavoidre-dundancy),(4)allowatmostonescenephraseforallsentencesinthedescription.Weﬁndthathan-dlingofsentenceboundariesisimportantiftheILPformulationisbasedonlyonsequencestructure,butwiththeintegrationoftree-basedstructure,wedonotneedtospeciﬁcallyhandlesentenceboundaries.3.4DiscussionAninterestingaspectofdescriptiongenerationex-ploredinthispaperisusingtreefragmentsasthebuildingblocksofcompositionratherthanindivid-ualwords.Therearethreepracticalbeneﬁts:(1)syntacticandsemanticexpressiveness,(2)correct-ness,和(3)computationalefﬁciency.Becauseweextractphrasesfromhumanwrittencaptions,weareabletouseexpressivelanguage,andlesslikelytomakesyntacticorsemanticerrors.Ourphraseex-tractionprocesscanbeviewedatahighlevelasvisually-groundedorvisually-situatedparaphrasing.Also,becausetheunitofoperationistreefragments,theILPformulationencodedinthisworkiscom-putationallylightweight.Iftheunitofcompositionwaswords,theILPinstanceswouldbesigniﬁcantlymorecomputationallyintensive,andmorelikelytosufferfromgrammaticalandsemanticerrors.4TreeCompressionAsnotedbyrecentstudies(MasonandCharniak,2013;Kuznetsovaetal.,2013;Jamiesonetal.,2010),naturallyexistingimagecaptionsoftenin-cludecontextualinformationthatdoesnotdirectlydescribevisualcontent,whichultimatelyhinderstheirusefulnessfordescribingotherimages.There-fore,toimprovetheﬁdelityofthegenerateddescrip-tions,weexploreimagecaptiongeneralizationasan

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你

/
t

A
C
我
/

我

A
r
t
我
C
e
–
p
d

F
/

d
哦

我
/

1
0
1
1
6
2

/
t

我

A
C
_
A
_
0
0
1
8
8
1
5
6
6
9
0
5

/
t

我

A
C
_
A
_
0
0
1
8
8
p
d

乙
y
G
你
e
s
t

哦
n
0
9
S
e
p
e
米
乙
e
r
2
0
2
3

356

Late%in%the%day,%A,er%my%sunset%shot%a2empts,%my%cat%strolled%along%the%fence%and%posed%for%this%classic%proﬁle%Late%in%the%day%%%cat%%%posed%for%this%proﬁle%Generaliza)on+This%bridge%stands%late%in%the%day,%A,er%my%sunset%shot%a2empts%A%cat%strolled%along%the%fence%and%posed%for%this%classic%proﬁle%Figure4:Compressedcaptions(ontheleft)aremoreap-plicablefordescribingnewimages(ontheright).optionalpre-processingstep.Figure4illustratesaconcreteexampleofimagecaptiongeneralizationinthecontextofimagecaptiongeneration.Wecastcaptiongeneralizationassentencecom-pression.WeencodetheproblemastreepruningvialightweightCKYparsing,whilealsoincorporatingseveralotherconsiderationssuchasleaf-levelngramcohesionscoresandvisuallyinformedcontentselec-tion.Figure5showsanexamplecompression,andFigure6showsthecorrespondingCKYmatrix.Atahighlevel,thecompressionoperationresem-blesbottom-upCKYparsing,butinadditiontopars-ing,wealsoconsiderdeletionofpartsofthetrees.Whendeletingpartsoftheoriginaltree,wemightneedtore-parsetheremainderofthetree.Notethatweconsiderre-parsingonlywithrespecttotheorig-inalparsetreeproducedbyastate-of-the-artparser,henceitisonlyalight-weightparsing.54.1DynamicProgrammingInputtothealgorithmisasentence,representedasavectorx=x0…xn−1=x[0:n−1],anditsPCFGparseπ(X)obtainedfromtheStanfordparser.Forsimplicityofnotation,weassumethatboththeparsetreeandthewordsequenceareencodedinx.Then,thecompressioncanbeformalizedas:5Integratingfullparsingintotheoriginalsentencewouldbeastraightforwardextensionconceptually,butmaynotbeanem-piricallybetterchoicewhenparsingforcompressionisbasedonvanillaunlexicalizedparsing.ˆy=argmaxyYiφi(X,y)(14)Whereeachφiisapotentialfunction,correspondingtoacriteriaofthedesiredcompression:φi(X,y)=exp(θi·fi(X,y))(15)Whereθiistheweightforaparticularcriteria(de-scribedin§4.2),whosescoringfunctionisfi.Wesolvethedecodingproblem(Equation14)us-ingdynamicprogramming.Forthis,weneedtosolvethecompressionsub-problemsforsequencesx[我:j],whichcanbeviewedasbranchesˆy[我,j]oftheﬁnaltreeˆy[0:n−1].Forexample,inFigure5,theﬁnalsolutionisˆy[0:7],whileasub-solutionofx[4:7]correspondstoatreebranchPP.Noticethatsub-solutionˆy[3:7]representsthesamebranchasˆy[4:7]duetobranchdeletion.Somecomputedsub-solutions,e.g.,ˆy[1:4],getdroppedfromtheﬁnalcompressedtree.WedeﬁneamatrixofscoresD[我,j,H](Equa-tion17),wherehisoneofthenonterminalsymbolsbeingconsideredforacellindexedbyi,j,i.e.acan-didatefortherootsymbolofabranchˆy[我:j].WhenallvaluesD[我,j,H]arecomputed,wetakeˆh=argmaxhD[0,n−1,h](16)andbacktracktoreconstructtheﬁnalcompression(theexactsolutiontoequation14).D[我,j,H]=maxk∈[我,j)r∈Rh(1)D[我,k,p]+D[k+1,j,q]+∆φ[r,ij](2)D[我,k,p]+∆φ[r,ij](3)D[k+1,j,p]+∆φ[r,ij](17)WhereRh={r∈R:r=h→pq∨r=h→p}.Indexkdeterminesasplitpointforchildbranchesofasubtreeˆy[我:j].Forexample,intheFigure5thesplitpointforchildrenofthesubtreeˆy[0:7]isk=2.Thethreecases((1)–(3))oftheaboveequationcorrespondtothefollowingtreepruningcases:PruningCase(1):Noneofthechildrenofthecur-rentnodeisdeleted.Forexample,inFigures5and6,thePCFGrulePP→INPP,correspondingtothesequence“inblackandwhite”,isretained.Anothersituationthatcanbeencounteredistreere-parsing.

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你

/
t

A
C
我
/

我

A
r
t
我
C
e
–
p
d

F
/

d
哦

我
/

1
0
1
1
6
2

/
t

我

A
C
_
A
_
0
0
1
8
8
1
5
6
6
9
0
5

/
t

我

A
C
_
A
_
0
0
1
8
8
p
d

乙
y
G
你
e
s
t

哦
n
0
9
S
e
p
e
米
乙
e
r
2
0
2
3

357

优质的!motorcycle!shot!done!在!黑色的!和!白色的!JJ!NN!NN!VBN!在!JJ!JJ!CC!NP, NN!NP!CC-JJ VP, PP NP!PP S Dele%on!probability!规则!probability!Vision!信心!Ngram!cohesion!(Dele%on,)案件)2))(Dele%on,)案件)1))01234567k=2$Figure5:CKYcompression.Boththechosenrulesandphrases(blueboldfontandbluesolidarrows)andnotchosenrulesandphrases(reditalicsmallerfontandreddashedlines)areshown.PruningCase(2)/(3):Deletionoftheleft/rightchildrespectively.Therearetwotypesofdeletion,asillustratedinFigures5and6.Theﬁrstcorre-spondstodeletionofachildnode.Forexample,thesecondchildNNofruleNP→NPNNisdeleted,whichyieldsdeletionof“shot”.Thesec-ondtypeisaspecialcaseofpropagatinganodetoahigher-levelofthetree.InFigure6,thissit-uationoccurswhendeletingJJ“Vintage”,whichcausesthepropagationofNNfromcell11tocell01.Forthispurpose,weexpandthesetofrulesRwithadditionalspecialrulesoftheformh→h,e.g.,NN→NN,whichallowspropagationoftreenodestohigherlevelsofthecompressedtree.64.2ModelingCompressionCriteriaThe∆φterm7inEquation17denotesthesumoflogofpotentialfunctionsforeachcriteriaq:∆φ[r,ij]=Xqθ·∆fq(r,ij)(18)Notethat∆φdependsonthecurrentruler,alongwiththehistoricalinformationbeforethecurrentstepij,suchastheoriginalrulerij,andngramsontheborderbetweenleftandrightchildbranchesofrulerij.Weusethefollowingfourcriteriafqinourmodel,whicharedemonstratedinFigures5and6.I.TreeStructure:WecapturePCFGruleprob-abilitiesestimatedfromthecorpusas∆fpcfg=logPpcfg(r).6Weassignprobabilitiesofthesespecialpropagationrulesto1sothattheywillnotaffecttheﬁnalparsetreescore.TurnerandCharniak(2005)handledpropagationcasessimilarly.7Weuse∆todistinguishthepotentialvalueforthewholesentencefromthegainofthepotentialduringasinglestepofthealgorithm.JJ NP, NN NP S Vintage NN motorcycle NN shot VBN VP, PP done IN PP in JJ NP black CC CC-JJ and JJ white 00″11″01″Rule%probability%Ngram%cohesion%Dele6on%probability%Vision%Conﬁdence%i”j”Figure6:CKYcompression.Boththechosenrulesandphrases(blueboldfontandbluesolidarrows)andnotchosenrulesandphrases(reditalicsmallerfontandreddashedlines)areshown.II.SequenceStructure:Weincorporatengramcohesionscoresonlyacrosstheborderbetweentwobranchesofasubtree.III.BranchDeletionProbabilities:Wecomputeprobabilitiesofdeletionforchildrenas:∆fdel=logP(rt|rij)=logcount(rt,rij)数数(rij)(19)Wherecount(rt,rij)isthefrequencyinwhichrijistransformedtortbydeletionofoneofthechildren.Weestimatethisprobabilityfromatrainingcorpus,describedin§4.3.count(rij)isthecountofrijinuncompressedsentences.IV.VisionDetection(ContentSelection):Wewanttokeepwordsreferringtoactualobjectsintheimage.Thus,weuseV(xj),avisualsimilarityscore,asourconﬁdenceofanobjectcorrespondingtowordxj.Thissimilarityisobtainedfromthevi-sualrecognitionpredictionsof(Dengetal.,2012b).Notethatsometestinstancesincluderulesthatwehavenotobservedduringtraining.Wedefaulttotheoriginalcaptioninthosecases.Theweightsθiaresetusingatuningdataset.Wecontrolover-compressionbysettingtheweightforfdeltoasmallvaluerelativetotheotherweights.4.3HumanCompressedCaptionsAlthoughwemodelimagecaptiongeneralizationassentencecompression,inpracticalapplicationswemaywanttheoutputsofthesetwotaskstobediffer-ent.Forexample,theremaybedifferencesinwhatshouldbedeleted(namedentitiesinnewswiresum-mariescouldbeimportanttokeep,whiletheymay

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你

/
t

A
C
我
/

我

A
r
t
我
C
e
–
p
d

F
/

d
哦

我
/

1
0
1
1
6
2

/
t

我

A
C
_
A
_
0
0
1
8
8
1
5
6
6
9
0
5

/
t

我

A
C
_
A
_
0
0
1
8
8
p
d

乙
y
G
你
e
s
t

哦
n
0
9
S
e
p
e
米
乙
e
r
2
0
2
3

358

Orig:”笔记”这”pillows,”他们”匹配”这”chair”那”去”和”它,”加”这”桌子”在”这”picture”是”included.%SeqCompression:%这”桌子”在”这”picture.””TreePruning:”这”chair”和”这”桌子”在”这”picture.”Orig:”仅有的”在”winter;我”我们”看”这些”birds”这里”在”这”river.”%SeqCompression:”看”这些”birds”在”这”river.””TreePruning:”这些”birds”在”这”river.””Orig:”这”world’s”最多”强大的”lighthouse”si@ng”beside”这”房子”和”这”world’s”thickest”curtains.”SeqCompression:%Si@ng”beside”这”房子””TreePruning:”Powerful”lighthouse”beside”这”房子”和”这”curtains.””Orig:”Orange”云”在”street”光”C”near”Lanakila”街道”(电话”camera).””SeqCompression:%Orange”street””TreePruning:”Phone”camera.%Relevance(问题(Orig:”There’s”something”关于”拥有”5″卡车”parked”在”正面”的”我的”房子”那”使”我”feel”全部”importantClike.”SeqCompression:%Front”的”我的”house.””TreePruning:”Trucks”在”正面”我的”house.%Grammar(mistakes(Figure7:Captiongeneralization:good/badexamples.beextraneousforimagecaptiongeneralization).Tolearnthesyntacticpatternsforcaptiongeneraliza-tion,wecollectasmallsetofexamplecompressedcaptions(380intotal)usingAmazonMechanicalTurk(AMT)(Snowetal.,2008).Foreachimage,weasked3turkerstoﬁrstlistallvisibleobjectsinanimageandthentowriteacompressedcaptionbyremovingnotvisuallyveriﬁablebitsoftext.Wethenaligntheoriginalandcompressedcaptionstomea-sureruledeletionprobabilities,excludingmisalign-ments,similartoKnightandMarcu(2000).Notethatweremovethisdatasetfromthe1Mcaptioncor-puswhenweperformdescriptiongeneration.5ExperimentsWeusethe1McaptionedimagecorpusofOrdonezetal.(2011).Wereserve1Kimagesasatestset,andusetherestofthecorpusforphraseextraction.Weexperimentwiththefollowingapproaches:ProposedApproaches:•TREEPRUNING:Ourtreecompressionap-proachasdescribedin§4.•SEQ+TREE:Ourtreecompositionapproachasdescribedin§3.•SEQ+TREE+PRUNING:SEQ+TREEusingcompressedcaptionsofTREEPRUNINGasbuildingblocks.BaselinesforComposition:•SEQ+LINGRULE:Themostequivalenttotheoldersequence-drivensystem(Kuznetsovaetal.,2012).Usesafewminorenhancements,suchassentence-boundarystatistics,toim-provegrammaticality.•SEQ:The§3systemwithouttreemodelsandmentionedenhancementsofSEQ+LINGRULE.MethodBleuMeteorw/(w/o)penaltyPRMSEQ+LINGRULE0.152(0.152)0.130.170.095SEQ0.138(0.138)0.120.180.094SEQ+TREE0.149(0.149)0.130.140.082SEQ+PRUNING0.177(0.177)0.150.160.101SEQ+TREE+PRUNING0.140(0.189)0.160.120.088Table1:AutomaticEvaluation•SEQ+PRUNING:SEQusingcompressedcap-tionsofTREEPRUNINGasbuildingblocks.Wealsoexperimentwiththecompressionofhumanwrittencaptions,whichareusedtogenerateimagedescriptionsforthenewtargetimages.BaselinesforCompression:•SEQCOMPRESSION(Kuznetsovaetal.,2013):Inferenceoperatesoverthesequencestructure.Althoughoptimizationissubjecttoconstraintsderivedfromdependencyparse,parsingisnotanexplicitpartoftheinferencestructure.Ex-ampleoutputsareshowninFigure7.5.1AutomaticEvaluationWeperformautomaticevaluationusingtwomea-sureswidelyusedinmachinetranslation:蓝线(Pa-pinenietal.,2002)8andMETEOR(DenkowskiandLavie,2011).9Weremoveallpunctuationandcon-vertcaptionstolowercase.Weuse1Ktestim-agesfromthecaptionedimagecorpus,10andas-sumetheoriginalcaptionsasthegoldstandardcap-tionstocompareagainst.TheresultsinTable18WeusetheunigramNISTimplementation:ftp://jaguar.ncsl.nist.gov/mt/resources/mteval-v13a-20091001.tar.gz9WithequalweightbetweenprecisionandrecallinTable1.10ExceptforthoseforwhichimageURLsarebroken,orCPLEXdidnotreturnasolution.

我

D
哦
w
n
哦
A
d
e
d

F
r
哦
米
H

t
t

:
/
/

d
我
r
e
C
t
.

米

我
t
.

e
d
你

/
t

A
C
我
/

我

A
r
t
我
C
e
–
p
d

F
/

d
哦

我
/

1
0
1
1
6
2

/
t

我

A
C
_
A
_
0
0
1
8
8
1
5
6
6
9
0
5

/
t

我

A
C
_
A
_
0
0
1
8
8
p
d

乙
y
G
你
e
s
t

哦
n
0
9
S
e
p
e
米
乙
e
r
2
0
2
3

359

Method-1Method-2CriteriaMethod-1preferredoverMethod-2(%)allturkersturkersw/κ>0.55turkersw/κ>0.6ImageDescriptionGenerationSEQ+TREESEQRel727272SEQ+TREESEQGmar838383SEQ+TREESEQAll686966SEQ+TREE+PRUNINGSEQ+TREERel687272SEQ+TREE+PRUNINGSEQ+TREEGmar413841SEQ+TREE+PRUNINGSEQ+TREEAll636466SEQ+TREESEQ+LINGRULEAll626462SEQ+TREE+PRUNINGSEQ+LINGRULEAll677577SEQ+TREE+PRUNINGSEQ+PRUNINGAll737575SEQ+TREE+PRUNINGHUMANAll241919ImageCaptionGeneralizationTREEPRUNINGSEQCOMPRESSION∗Rel656566Table2:HumanEvaluation:posedasabinaryquestion“whichofthetwooptionsisbetter?”withrespecttoRelevance(Rel),语法(Gmar),andOverall(全部).AccordingtoPearson’sχ2test,allresultsarestatisticallysigniﬁcant.showthatboththeintegrationofthetreestructure(+TREE)andthegeneralizationofcaptionsusingtreecompression(+PRUNING)improvetheBLEUscorewithoutbrevitypenaltysigniﬁcantly,11whileimprovingMETEORonlymoderately(duetoanim-provementonprecisionwithadecreaseinrecall.)5.2HumanEvaluationNeitherBLEUnorMETEORdirectlymeasuregrammaticalcorrectnessoverlongdistancesandmaynotcorrespondperfectlytohumanjudgments.Therefore,wesupplementautomaticevaluationwithhumanevaluation.Forhumanevaluations,wepresenttwooptionsgeneratedfromtwocompet-ingsystems,andaskturkerstochoosetheonethatisbetterwithrespectto:关联,grammar,andoverall.ResultsareshowninTable2with3turkerratingsperimage.Weﬁlteroutturkersbasedonacontrolquestion.Wethencomputetheselec-tionrate(%)ofpreferringmethod-1overmethod-2.Theagreementamongturkersisafrequentconcern.Therefore,wevarythesetofdependableusersbasedontheirCohen’skappascore(κ)againstotherusers.Itturnsout,ﬁlteringusersbasedonκdoesnotmakeabigdifferenceindeterminingthewinningmethod.Asexpected,tree-basedsystemssigniﬁcantlyout-performsequence-basedcounterparts.Forexample,11While4-gramBLEUwithbrevitypenaltyisfoundtocor-relatebetterwithhumanjudgesbyrecentstudies(ElliottandKeller,2014),wefoundthatthisisnotthecaseforourtask.Thismaybeduetothedifferencesinthegoldstandardcap-tions.Weusenaturallyexistingones,whichincludeawiderrangeofcontentandstylethancrowd-sourcedcaptions.Seq:”A”bu&erﬂy”到”这”car”曾是”spo&编辑”经过”我的”九”年”老的”cousin.”Seq+Pruning:”这”bu&erﬂies”是”A&racted”到”这”colourful”ﬂowers”到”这”car.+Seq+Tree:”这”bu&erﬂies”是”A&racted”到”这”colourful”ﬂowers”在”Hope”Gardens.””Seq+Tree+Pruning:”这”bu&erﬂies”是”A&racted”到”这”colourful”ﬂowers.”Orig:”这”bu&erﬂies”是”A&racted”到”这”colourful”ﬂowers”在”Hope”Gardens.””SeqCompression:”这”colourful”ﬂowers.”””TreePruning:”这”bu&erﬂies”是”A&racted”到”这”colourful”ﬂowers.”””Cap>on”Generaliza>on”图像”Descrip>on”Genera>on”Figure8:Anexampleofadescriptionpreferredoverhu-mangoldstandard.Imagedescriptionisimprovedduetocaptiongeneralization.SEQ+TREEisstronglypreferredoverSEQ,withaselectionrateof83%.Somewhatsurprisingly,im-provedgrammaticalityalsoseemstoimproverele-vancescores(72%),possiblybecauseitishardertoappreciatethesemanticrelevanceofautomaticcap-tionswhentheyarelesscomprehensible.Alsoasexpected,compositionsbasedonprunedtreefrag-mentssigniﬁcantlyimproverelevance(68–72%),whileslightlydeterioratinggrammar(38–41%).尤其,thecaptionsgeneratedbyoursystemarepreferredovertheoriginal(ownergenerated)cap-tions19–24%ofthetime.Onesuchexampleisin-cludedinFigure8:“Thebutterﬂiesareattractedtothecolorfulﬂowers.”Additionalexamples(goodandbad)arepro-videdinFigures9and10.Manyofthesecaptionsarehighlyexpressivewhileremainingsemantically