Transactions of the Association for Computational Linguistics, vol. 4, pp. 61–74, 2016. Action Editor: Janyce Wiebe and Kristina Toutanova.
Submission batch: 10/2015; Revision batch: 12/2015; Published 3/2016.
2016 Association for Computational Linguistics. Distributed under a CC-BY 4.0 Licence.
c
(cid:13)
61
AnEmpiricalAnalysisofFormalityinOnlineCommunicationElliePavlickUniversityofPennsylvania∗epavlick@seas.upenn.eduJoelTetreaultYahooLabstetreaul@yahoo-inc.comAbstractThispaperpresentsanempiricalstudyoflinguisticformality.Weperformananaly-sisofhumans’perceptionsofformalityinfourdifferentgenres.Thesefindingsareusedtodevelopastatisticalmodelforpre-dictingformality,whichisevaluatedun-derdifferentfeaturesettingsandgenres.Weapplyourmodeltoaninvestigationofformalityinonlinediscussionforums,andpresentfindingsconsistentwiththeoriesofformalityandlinguisticcoordination.1IntroductionLanguageconsistsofmuchmorethanjustcon-tent.Considerthefollowingtwosentences:1.Thoserecommendationswereunsolicitedandundesirable.2.that’sthestupidestsuggestionEVER.Bothsentencescommunicatethesameidea,butthefirstissubstantiallymoreformal.Suchstylisticdifferencesoftenhavealargerimpactonhowthehearerunderstandsthesentencethantheliteralmeaningdoes(Hovy,1987).Fullnaturallanguageunderstandingrequirescomprehendingthisstylisticaspectofmeaning.Toenablerealadvancementsindialogsystems,informationextraction,andhuman-computerinteraction,computersneedtounderstandtheentiretyofwhathumanssay,boththeliteralandthenon-literal.Inthispaper,wefocusonthe∗ResearchperformedwhileatYahooLabs.particularstylisticdimensionillustratedabove:formality.Formalityhaslongbeenofinteresttolinguistsandsociolinguists,whohaveobservedthatitsubsumesarangeofdimensionsofstylein-cludingserious-trivial,polite-casual,andlevelofsharedknowledge(Irvine,1979;BrownandFraser,1979).Theformal-informaldimensionhasevenbeencalledthe“mostimportantdi-mensionofvariationbetweenstyles”(HeylighenandDewaele,1999).Aspeaker’slevelofformal-itycanrevealinformationabouttheirfamiliar-itywithaperson,opinionsofatopic,andgoalsforaninteraction(Hovy,1987;Endrassetal.,2011).Asaresult,theabilitytorecognizefor-malityisanintegralpartofdialoguesystems(Mairesse,2008;MairesseandWalker,2011;BattaglinoandBickmore,2015),sociolinguisticanalyses(Danescu-Niculescu-Miziletal.,2012;Justoetal.,2014;KrishnanandEisenstein,2015),human-computerinteraction(Johnsonetal.,2005;KhosmoodandWalker,2010),summa-rization(SidhayeandCheung,2015),andau-tomaticwritingassessment(FeliceandDeane,2012).Formalitycanalsoindicatecontext-independent,universalstatements(HeylighenandDewaele,1999),makingformalitydetectionrelevantfortaskssuchasknowledgebasepopu-lation(Suhetal.,2006;ReiterandFrank,2010)andtextualentailment(Daganetal.,2006).Thispaperinvestigatesformalityinonlinewrittencommunication.Thecontributionsareasfollows:1)Weprovideananalysisofhumans’subjectiveperceptionsofformalityinfourdif-ferentgenres.Wehighlightareasofhighandlowagreementandextractpatternsthatconsis-
je
D
o
w
n
o
un
d
e
d
F
r
o
m
h
t
t
p
:
/
/
d
je
r
e
c
t
.
m
je
t
.
e
d
toi
/
t
un
c
je
/
je
un
r
t
je
c
e
–
p
d
F
/
d
o
je
/
.
1
0
1
1
6
2
/
t
je
un
c
_
un
_
0
0
0
8
3
1
5
6
7
3
6
0
/
/
t
je
un
c
_
un
_
0
0
0
8
3
p
d
.
F
b
oui
g
toi
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
62
tentlydifferentiateformalfrominformaltext.2)Wedevelopastate-of-the-artstatisticalmodelforpredictingformalityatthesentencelevel,evaluatethemodel’sperformanceagainsthu-manjudgments,andcomparedifferencesintheeffectivenessoffeaturesacrossgenres.3)Weapplyourmodeltoanalyzelanguageuseinon-linedebateforums.Ourresultsprovidenewev-idenceinsupportoftheoriesoflinguisticcoordi-nation,underliningtheimportanceofformalityforlanguagegenerationsystems.4)Wereleaseournewdatasetof6,574sentencesannotatedforformalitylevel.2RelatedWorkThereisnogenerallyagreedupondefinitionastowhatconstitutesformallanguage.Somede-fineformalityintermsofsituationalfactors,suchassocialdistanceandsharedknowledge(Sigley,1997;Hovy,1987;Lahirietal.,2011).Otherrecentworkadoptsalessabstractdefi-nitionwhichissimilartothenotionof“noisytext”–e.g.useofslangandpoorgrammar(MosqueraandMoreda,2012un;Petersonetal.,2011).Asaresult,manyruleshavebeenex-ploredforrecognizingandgeneratinginformallanguage.Someoftheserulesareabstract,suchasthelevelofimplicature(HeylighenandDe-waele,1999;Lahiri,2015)orthedegreeofsub-jectivity(MosqueraandMoreda,2012un),whileothersaremuchmoreconcrete,suchasthenum-berofadjectives(FangandCao,2009)oruseofcontractions(AbuSheikhaandInkpen,2011).Muchpriorworkondetectingformalityhasfocusedonthelexicallevel(Brookeetal.,2010;BrookeandHirst,2014;PavlickandNenkova,2015).Forlargerunitsoftext,perhapsthebest-knownmethodformeasuringformalityistheF-score1(HeylighenandDewaele,1999),whichisbasedonrelativepart-of-speechfre-quencies.F-scoreanditsmorerecentvariants(Lietal.,2013)provideacoarsemeasureoffor-mality,butaredesignedtoworkatthegenre-level,makingthemlessreliableforshorterunitsoftextsuchassentences(Lahiri,2015).Exist-1WeusespecialfonttodenoteHeylighenandDe-waele’sF-scoretoavoidconfusionwithF1measure.ingstatisticalapproachestodetectingformal-ity(AbuSheikhaandInkpen,2010;Petersonetal.,2011;MosqueraandMoreda,2012b)havetreatedtheproblemasabinaryclassificationtaskandreliedheavilyonwordliststodifferen-tiatethetwoclasses.Linguisticsliteraturesup-portstreatingformalityasacontinuum(Irvine,1979;HeylighenandDewaele,1999),ashasbeendoneinstudiesofotherpragmaticdimensionssuchaspoliteness(Danescu-Niculescu-Miziletal.,2013)andemotiveness(Walkeretal.,2012).Lahirietal.(2011)providedapreliminaryin-vestigationofannotatingformalityonanordi-nalscaleandreleasedadatasetofsentence-levelformalityannotations(Lahiri,2015),butdidnotusetheirdatainanycomputationaltasks.Thispaperextendspriorworkby(je)introducingastatisticalregressionmodelofformalitywhichisbasedonanempiricalanalysisofhumanper-ceptionsratherthanonheuristicsand(ii)byapplyingthatmodeltoalinguisticanalysisofonlinediscussions.3HumanperceptionsofformalityBeforewecanautomaticallyrecognizeformal-ity,weneedanunderstandingofwhatitmeansforlanguagetobeformalorinformal.AswediscussedinSection2,anumberoftheoriesex-istwithnoclearconsensus.Inthiswork,wedonotattempttodevelopaconcretedefinitionofformality,butinsteadtakeabottom-upap-proachinwhichweassumethateachindividualhastheirowndefinitionofformality.Thisap-proachofusingunguidedhumanjudgmentshasbeensuggestedbySigley(1997)asoneofthemostreliablewaystogetagold-standardmea-sureofformality,andhasbeenappliedinpriorcomputationallinguisticsstudiesofpragmatics(Danescu-Niculescu-Miziletal.,2013;Lahiri,2015).Weaimtoanswer:dohumans’individualintuitionscollectivelyprovideacoherentnotionofformality(§3.2)?Et,ifso,whichlinguisticfactorscontributetothisnotion(§3.3)?3.1DataandAnnotationSinceformalityvariessubstantiallyacrossgen-res(Lietal.,2013),welookattextfromfourdifferentgenres:News,Blogs,Emails,andcom-
je
D
o
w
n
o
un
d
e
d
F
r
o
m
h
t
t
p
:
/
/
d
je
r
e
c
t
.
m
je
t
.
e
d
toi
/
t
un
c
je
/
je
un
r
t
je
c
e
–
p
d
F
/
d
o
je
/
.
1
0
1
1
6
2
/
t
je
un
c
_
un
_
0
0
0
8
3
1
5
6
7
3
6
0
/
/
t
je
un
c
_
un
_
0
0
0
8
3
p
d
.
F
b
oui
g
toi
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
63
(un)Answers(µ=-0.7,σ=1.3)(b)Blogs(µ=0.2,σ=1.1)(c)Emails(µ=0.5,σ=1.4)(d)News(µ=0.7,σ=0.86)Figure1:Distributionofsentence-levelformalityscoresbygenre.Answers2.8Thatisinadditiontoanycustomsdutiesthatmaybeassessed.Answers-3.0(LOL)juskidding…theanswertoyourquestionisGASPRICES!!!News2.6Baghdadisacityofsurprisingtopiarysculptures:leafyficustreesarecarvedingeometricspirals,balls,archesandsquares,asiftoimposeorderonachaoticsprawl.News-2.2Heboughtandboughtandneverstopped.Table1:Examplesofformal(positive)andinformal(negative)sentencesindifferentgenres.Scoresaretakenasthemeanof5humanjudgmentsonascalefrom-3to3.munityquestionansweringforums(henceforth“Answers”).Lahiri(2015)releasedacorpusofsentence-levelformalityannotations,whichcontains2,775newsand1,821blogsentences.Inadditionwetakearandomsampleof1,701sentencesfromprofessionalemails2and4,977sentencesfromYahooAnswers.3WefollowtheprotocolusedinLahiri(2015)inordertogatherjudgmentsonAmazonMechanicalTurkfortheEmailandAnswersdata.Specifically,weusea7-pointLikertscale,withlabelsfrom-3(VeryInformal)to3(VeryFormal).Soasnottobiastheannotatorswithourownno-tionsofformality,weprovideonlyabriefde-scriptionofformallanguageandencouragean-notatorstofollowtheirinstinctswhenmakingtheirjudgments.Weusethemeanof5anno-tators’scoresastheoverallformalityscoreforeachsentence.4Ournewlycollectedannotationshavebeenmadepublic.5Formoreinformationontheannotation,pleaserefertothesupple-2http://americanbridgepac.org/jeb-bushs-gubernatorial-email-archive/3https://answers.yahoo.com/4Intotal,wehad301annotators,meaningeachanno-tatorlabeled22sentencesonaverage.5http://www.seas.upenn.edu/~nlp/resources/formality-corpus.tgzmentarymaterialtothispaper.63.2AnalysisFigure1showsthedistributionofmeanformal-ityscoresforthesentencesineachofourgenres.WeseethatNewsisthemostformalofourdo-mainsandAnswersistheleast.However,wecanseeanecdotally(Table1)thatthestandardofwhatconstitutes“informal”dependsheavilyonthegenre:aninformalsentencefromNewsismuchmoreformalthanonefromAnswers.Wecanalsoseecleardifferencesinthevarianceofsentenceformalitieswithineachgenre.Ingen-eral,theinteractivegenres(EmailandAnswers)showamuchflatterdistributionthandothein-formationalgenres(NewsandBlogs).Inter-annotatoragreement.Wewanttoknowwhetherindividuals’intuitionsaboutfor-mallanguageresultinacoherentcollectiveno-tionofformality.Toquantifythis,wemeasurewhetherannotators’ordinalratingsofformalityarewellcorrelatedandwhethertheircategor-icaljudgmentsareinagreement.Forthefor-mer,weuseintraclasscorrelation7(ICC)which6http://www.seas.upenn.edu/~epavlick/papers/formality_supplement.pdf7Wereporttheaverageratersabsoluteagreement(ICC1k)usingthepsychpackageinR:https://cran.
je
D
o
w
n
o
un
d
e
d
F
r
o
m
h
t
t
p
:
/
/
d
je
r
e
c
t
.
m
je
t
.
e
d
toi
/
t
un
c
je
/
je
un
r
t
je
c
e
–
p
d
F
/
d
o
je
/
.
1
0
1
1
6
2
/
t
je
un
c
_
un
_
0
0
0
8
3
1
5
6
7
3
6
0
/
/
t
je
un
c
_
un
_
0
0
0
8
3
p
d
.
F
b
oui
g
toi
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
64
3,3,3,3,3FormalIwouldtrustthesocialworkerstomaketheappropriatecasebycasedetermination.-3,-3,-3,-3,-3Informal*whattheworldneedsisonlymoreofU&URsmile!!-3,-2,0,-1,3MixedGovernor,ifthiswasintentionallydone,whoeverdidithasatleastonevotetogotohell.-1,0,0,0,1NeutralYoushouldtryherbalpepperminttea.Table2:Examplesofsentenceswithdifferentpatternsofagreement.Numbersshowthelistofscoresassignedbythe5annotators.Somesentencesexhibit“mixed”formality,i.e.workersweresplitonwhethertocallthesentencegenerallyinformalorgenerallyformal,whileothersare“neutral,”i.e.workersagreedthesentencewasneitherformalnorinformal.issimilartoPearsonρbutaccountsforthefactthatwehavedifferentgroupsofannotatorsforeachsentence.Forthelatter,weuseaquadraticweightedκ,whichisavariationofCohen’sκbetterfitformeasuringagreementonordinalscales.8Whenusingcrowdsourcedlabels,com-putingreliablemeasuresofκisdifficultsince,foragivenpairofannotators,thenumberofitemsforwhichbothprovidedalabelislikelysmall.Wethereforesimulatetwoannotatorsasfollows.Foreachsentence,werandomlychooseoneannotator’slabeltobethelabelofAnnota-tor1andwetakethemeanlabeloftheother4annotators,roundedtothenearestinteger,tobethelabelofAnnotator2.Wethencomputeκforthesetwosimulatedannotators.Werepeatthisprocess1,000times,andreportthemedianand95%confidenceinterval(Table3).NICCWeightedκAnswers4,9770.79±0.010.54±0.05Blog1,8210.58±0.030.31±0.05Email1,7010.83±0.020.59±0.04News2,7750.39±0.050.17±0.06Table3:Annotatoragreementmeasuredbyin-traclasscorrelation(ICC)andcategoricalagree-ment(quadraticweightedκ)foreachgenre.Agreementisreasonablystrongacrossgenres,withtheexceptionofNews,whichappearstobethemostdifficulttojudge.Table2shedslightonthetypesofsentencesthatreceivehighandlowlevelsofagreement.Attheextremeendsr-project.org/web/packages/psych/psych.pdf8Weightedκpenalizeslargedisagreementsmorethansmalldisagreements.E.g.ifAnnotator1labelsasen-tenceas−2andAnnotator2labelsit−3,thisispenal-izedlessthanifAnnotator1chooses−2andAnnotator2chooses+3.ofthespectrumwhereagreementisveryhigh(meanscoresnear−3and+3),weseesentenceswhichareunambiguouslyformalorinformal.However,inthemiddle(meanscoresnear0)weseebothhighandlowagreementsentences.Highagreementsentencestendtobe“neutral,”i.e.annotatorsagreetheyareneitherformalnorinformal,whilethelow-agreementsentencestendtoexhibit“mixed”formality,i.e.theycon-tainbothformalandinformalsub-sententialele-ments.Weleavethetopicofsub-sententialfor-malityforfuturework,andinsteadallowouruseofthemeanscoretoconflatemixedformal-itywithneutralformality.Thisfitsnaturallyintoourtreatmentofformalityasacontinuousasopposedtoabinaryattribute.3.3FactorsaffectingformalityFromtheaboveanalysis,weconcludethathu-manshaveareasonablycoherentconceptoffor-mality.However,itisdifficulttoteaseapartperceivedformalitydifferencesthatarisefromtheliteralmeaningofthetext(e.g.whetherthetopicisseriousortrivial)asopposedtoarisingfromthestyleinwhichthoseideasareexpressed.Togetabetterunderstandingofthestylisticchoicesthatdifferentiateformalfrominformal,weranasecondexperimentinwhichweaskedannotatorstorewriteinformalsentencesinordertomakethemmoreformal.Thegoalistoisolatesomeofthelinguisticfactorsthatcontributetoperceivedformalitywhileconstrainingtheliteralcontentofthetexttobethesame.Weusethisdataforanalysisinthissection,aswellasforevaluationinSection4.2.Forthistask,wechose1,000sentencesfromtheAnswersdataset,sinceitdisplaysthewidestvarietyoftopicsandstyles.Weattemptto
je
D
o
w
n
o
un
d
e
d
F
r
o
m
h
t
t
p
:
/
/
d
je
r
e
c
t
.
m
je
t
.
e
d
toi
/
t
un
c
je
/
je
un
r
t
je
c
e
–
p
d
F
/
d
o
je
/
.
1
0
1
1
6
2
/
t
je
un
c
_
un
_
0
0
0
8
3
1
5
6
7
3
6
0
/
/
t
je
un
c
_
un
_
0
0
0
8
3
p
d
.
F
b
oui
g
toi
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
65
Capitalization50%idonotlikewalmart.IdonotlikeWalmart.Punctuation39%She’s40,butsheseemsmorelikea30!!!!!Sheis40,butsheseemsmorelike30!Paraphrase33%Lexuscarsareawesome!Lexusbrandcarsareverynice.Deletefillers19%wellitdependsonthatgirl.Itdependsonthegirl.Completion17%looksgoodonyourrecord.Itlooksgoodonyourrecord.Addcontext16%alive-ihaveseenthatguyworkingata7-11behndthecounterMyopinionisthatOsamaBinLadenisaliveasIhaveencounteredhimworkingata7-11store.Contractions16%Ireallydon’tlikethem.Ireallydonotlikethem.Spelling10%ilovedancingiwthmychickfriends.Ienjoydancingwithmygirlfriends.Normalization8%juztrytoputurheartintoit.Justtrytoputyourheartintoit.Slang/idioms8%that’sabigno.Idonotagree.Politeness7%uh,moredetails?Couldyouprovidemoredetails,please?Splitsentences4%[…]notastough…likehighschool[…]notastough.It’slikehighschool.Relativizers3%sorryi’mnotmuchhelphehSorrythatIamnotmuchhelp.Table4:Frequencyoftypesofedits/changesmadeinrewritingexperiment,andexamplesofeach.Notethecategoriesarenotmutuallyexclusive.choosesentencesthatareinformalenoughtopermitformalizing,whilecoveringallrangesofinformality,fromhighlyinformal(“yep…lovethepiclol”)toonlyslightlyinformal(“Aslongasyoufeelgood.”).Eachsentenceisshowninthecontextofthequestionandthefullan-swerpostinwhichitappeared.Wecollectonerewritepersentence,andmanuallyremovespammers.Peoplemakealargevarietyofedits,whichcoverthe“noisytext”senseofformality(e.g.punctuationfixes,lexicalnormalization)aswellasthemoresituationalsense(e.g.insertingpoliteness,providingcontext).Tocharacter-izethesedifferentedittypes,wemanuallyre-viewedasampleof100rewritesandcategorizedthetypesofchangesthatweremade.Table4givestheresultsofthisanalysis.Overhalfoftherewritesinvolvedchangestocapitalizationandpunctuation.Aquarterinvolvedsomesortoflexicalorphrasalparaphrase(e.g“awesome”→“verynice”).In16%ofcases,therewrittensen-tenceincorporatedadditionalinformationthatwasapparentfromthelargercontext,butnotpresentintheoriginalsentence.ThisaccordswithHeylighenandDewaele(1999)’sdefinitionof“deep”formality,whichsaysthatformallan-guagestrivestobelesscontext-dependent.4RecognizingformalityautomaticallyIntheabovesection,weaskedwhetherhumanscanrecognizeformalityandwhatcontributestotheirperceptionofformalorinformal.Wenowask:howwellcancomputersautomaticallydis-tinguishformalfrominformalandwhichlinguis-tictriggersareimportantfordoingso?4.1SetupWeusethedatadescribedinSection3.1fortraining,usingthemeanoftheannotators’scoresasthegoldstandardlabels.Wetrainaridgeregression9modelwiththemodelparame-terstunedusingcrossvalidationonthetrainingdata.Unlessotherwisespecified,wekeepgen-resseparate,sothatmodelsaretrainedonlyondatafromthegenreinwhichtheyaretested.Features.Weexplore11differentfeaturegroups,describedinTable5.Tothebestofourknowledge,5ofthesefeaturegroups(ngrams,wordembeddings,parsetreepro-ductions,dependencytuples,andnameden-tities)havenotbeenexploredinpriorworkonformalityrecognition.Theremainingfea-tures(e.g.length,POStags,case,punctua-tion,formal/informallexicons,andsubjectiv-ity/emotiveness)largelysubsumethefeaturesexploredbypreviouslypublishedclassifiers.WeuseStanfordCoreNLP10forallofourlinguisticprocessing,exceptforsubjectivityfeatures,forwhichweuseTextBlob.119http://scikit-learn.org/10http://nlp.stanford.edu/software/corenlp11https://textblob.readthedocs.org
je
D
o
w
n
o
un
d
e
d
F
r
o
m
h
t
t
p
:
/
/
d
je
r
e
c
t
.
m
je
t
.
e
d
toi
/
t
un
c
je
/
je
un
r
t
je
c
e
–
p
d
F
/
d
o
je
/
.
1
0
1
1
6
2
/
t
je
un
c
_
un
_
0
0
0
8
3
1
5
6
7
3
6
0
/
/
t
je
un
c
_
un
_
0
0
0
8
3
p
d
.
F
b
oui
g
toi
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
66
caseNumberofentirely-capitalizedwords;binaryindicatorforwhethersentenceislowercase;binaryindi-catorforwhetherthefirstwordiscapitalized.*dependencyOne-hotfeaturesforthefollowingdependencytuples,withlexicalitemsbackedofftoPOStag:(gov,typ,dep),(gov,typ),(typ,dep),(gov,dep).*entityOne-hotfeaturesforentitytypes(e.g.PERSON,LOCATION)occurringinthesentence;averagelength,incharacters,ofPERSONmentions.lexicalNumberofcontractionsinthesentence,normalizedbylength;averagewordlength;averagewordlog-frequencyaccordingtoGoogleNgramcorpus;averageformalityscoreascomputedbyPavlickandNenkova(2015).*ngramOne-hotfeaturesfortheunigrams,bigrams,andtrigramsappearinginthesentence.*parseDepthofconstituencyparsetree,normalizedbysentencelength;numberoftimeseachproductionruleappearsinthesentence,normalizedbysentencelength,andnotincludingproductionswithterminalsymbols(i.e.lexicalitems).POSNumberofoccurrencesofeachPOStag,normalizedbythesentencelength.punctuationNumberof‘?',‘…',and‘!’inthesentence.readabilityLengthofthesentence,inwordsandcharacters;Flesch-KincaidGradeLevelscore.subjectivityNumberofpassiveconstructions;numberofhedgewords,accordingtoawordlist;numberof1stpersonpronouns;numberof3rdpersonpronouns;subjectivityaccordingtotheTextBlobsentimentmodule;binaryindicatorforwhetherthesentimentispositiveornegative,accordingtotheTextBlobsentimentmodule.Allofthecountsarenormalizedbythesentencelength.*word2vecAverageofwordvectorsusingpre-trainedword2vecembeddings,skippingOOVwords.Table5:Summaryoffeaturegroupsusedinourmodel.Tothebestofourknowledge,thosemarkedwith(*)havenotbeenpreviouslystudiedinthecontextofdetectinglinguisticformality.Baselines.WemeasuretheperformanceofourmodelusingSpearmanρwithhumanlabels.Wecompareagainstthefollowingbaselines:•Sentencelength:Wemeasurelengthincharacters,asthisperformedslightlybetterthanlengthinwords.•Flesch-Kincaidgradelevel:FKgradelevel(Kincaidetal.,1975)isafunctionofwordcountandsyllablecount,designedtomeasurereadability.Weexpecthighergradelevelstocorrespondtomoreformaltext.•F-score:HeylighenandDewaele(1999)’sformalityscore(F-score)isafunctionofPOStagfrequencywhichisdesignedtomeasureformalityatthedocument-andgenre-level.WeexpecthigherF-scoretocorrespondtomoreformaltext.•LMperplexity:Wereporttheperplex-ityaccordingtoa3-gramlanguagemodeltrainedontheEnglishGigawordwithavo-cabularyof64Kwords.Wehypothesizethatsentenceswithlowerperplexity(i.e.sentenceswhichlookmoresimilartoeditednewstext)willtendtobemoreformal.Wealsoexploredusingtheratiooftheper-plexityaccordingtoan“informal”languagemodelovertheperplexityaccordingtoa“formal”languagemodelasabaseline,buttheresultsofthisbaselinewerenotcompet-itive,andso,forbrevity,wedonotincludethemhere.•Formalitylexicons:WecompareagainsttheaveragewordformalityscoreaccordingtotheformalitylexiconreleasedbyBrookeandHirst(2014).WecomputethisscoreinthesamewayasSidhayeandCheung(2015),whousedittomeasuretheformal-ityoftweets.•Ngramclassifier:Asourfinalbaseline,wetrainaridgeregressionmodelwhichusesonlyngrams(unigrams,bigrams,andtri-grams)asfeatures.Comparisonagainstpreviouslypublishedmodels.Notethatwearenotabletomakeameaningfulcomparisonagainstagainstanyofthepreviouslypublishedstatisticalmodelsforformalitydetection.Toourknowledge,therearethreerelevantpreviouspublicationsthatpro-ducedstatisticalmodelsfordetectingformality:AbuSheikhaandInkpen(2010),Petersonetal.(2011),andMosqueraandMoreda(2012b).Allthreeofthesemodelsperformedabinaryclas-sification(asopposedtoregression)andoper-
je
D
o
w
n
o
un
d
e
d
F
r
o
m
h
t
t
p
:
/
/
d
je
r
e
c
t
.
m
je
t
.
e
d
toi
/
t
un
c
je
/
je
un
r
t
je
c
e
–
p
d
F
/
d
o
je
/
.
1
0
1
1
6
2
/
t
je
un
c
_
un
_
0
0
0
8
3
1
5
6
7
3
6
0
/
/
t
je
un
c
_
un
_
0
0
0
8
3
p
d
.
F
b
oui
g
toi
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
67
atedatthedocument(asopposedtosentencelevel).WewereabletocloselyreimplementthemodelofPetersonetal.(2011),butwechoosenottoincludetheresultsheresincetheirmodelwasdesignedforbinaryemail-levelclassificationandthusreliesondomain-specificfeatures(e.g.casinginthesubjectline),thatarenotavailableinourreal-valued,sentence-leveldatasets.Theothermodelsandthedata/lexiconsonwhichtheyreliedarenotreadilyavailable.Forthisreason,wedonotcomparedirectlyagainstthepreviouslypublishedstatisticalmodels,butac-knowledgethatseveralofourfeaturesoverlapwithpriorwork(seeSection4.1andTable5).4.2PerformanceTable6reportsourresultson10-foldcrossval-idation.Usingourfullsuiteoffeatures,weareabletoachievesignificantperformancegainsinallgenres,improvingbyasmuchas11pointsoverourstrongestbaseline(thengrammodel).AnswersBlogsEmailNewsLMppl0.00-0.010.14-0.08F-score0.160.350.210.27Length0.230.510.530.34F-Klevel0.450.540.630.41B&Hlexicon0.470.410.550.30Ngrammodel0.600.550.650.43Classifier0.700.660.750.48Table6:Spearmanρwithhumanjudgmentsforourmodelandseveralbaselines.Notethat,whilethebasicLMperplexitycor-relatesveryweaklywithformalityoverall,theEmailgenreactuallyexhibitsatrendoppositeofthatwhichweexpected:inEmail,sentenceswhichlooklesslikeGigawordtext(higherper-plexity)tendtobemoreformal.Oninspec-tion,weseethatmanyofthesentenceswhichhavelowperplexitybutwhichhumanslabelasinformalincludesentencescontainingnamesandgreeting/signaturelines,aswellassentenceswhichareentirelycapitalized(capitalizationisnotconsideredbytheLM).Contributionsoffeaturegroups.Inordertogainbetterinsightintohowformalitydiffersacrossgenres,welookmorecloselyattheperfor-manceofeachfeaturegroupinisolation.Table7showstheperformanceofeachfeaturegrouprelativetotheperformanceofthefullclassifier,foreachgenre.Afewinterestingresultsstandout.Ngramandwordembeddingfeaturesper-formwellacrosstheboard,achievingover80%oftheperformanceofthefullclassifierinallcases.Casingandpunctuationfeaturesaresig-nificantlymoreimportantintheAnswersdo-mainthanintheotherdomains.ConstituencyparsefeaturesandentityfeaturescarrynotablymoresignalintheBlogandNewsdomainsthanintheEmailandAnswersdomains.AnswersBlogsEmailNewsngram0.840.850.840.91word2vec0.830.830.840.87parse0.700.890.740.89readability0.690.750.840.83dependency0.640.890.840.85lexical0.560.550.590.70case0.500.280.240.37POS0.490.740.670.74punctuation0.470.380.370.20subjectivity0.290.310.250.37entity0.140.630.340.72Table7:Relativeperformanceofeachfeaturegroupacrossgenres.Numbersreflecttheperfor-mance(Spearmanρ)oftheclassifierwhenusingonlythespecifiedfeaturegroup,relativetotheperformancewhenusingallfeaturegroups.train\testAnswersBlogsEmailNewsAnswers0-5-5-6Blogs-170-9-2Email-13-40-4News-23-4-130Table8:Dropinperformance(Spearmanρ×100)whenmodelistrainedonsentencesfromonedomain(row)andtestedonsentencesfromanother(column).Changesarerelativetotheperformancewhentrainedonlyonsentencesfromthetestdomain(representedbyzerosalongthediagonal).Allmodelsweretrainedonanequalamountofdata.Observingthesedifferencesbetweendatasetsraisesthequestion:howwelldoesknowledgeofformalitytransferacrossdomains?Toanswerthis,wemeasureclassifierperformancewhentrainedinonedomain12andtestedinanother(Table8).Inourexperiments,themodeltrained12Allmodelsweretrainedonanequalamountofdata.
je
D
o
w
n
o
un
d
e
d
F
r
o
m
h
t
t
p
:
/
/
d
je
r
e
c
t
.
m
je
t
.
e
d
toi
/
t
un
c
je
/
je
un
r
t
je
c
e
–
p
d
F
/
d
o
je
/
.
1
0
1
1
6
2
/
t
je
un
c
_
un
_
0
0
0
8
3
1
5
6
7
3
6
0
/
/
t
je
un
c
_
un
_
0
0
0
8
3
p
d
.
F
b
oui
g
toi
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
68
onAnswersconsistentlyprovidedthebestper-formanceoutofdomain,resultinginperfor-mancedegradationsofroughly5points(Spear-manρ)comparedtomodelstrainedontargetdomaindata.TrainingonNewsandtestingonAnswerscausedthelargestdrop(23pointscom-paredtotrainingonAnswers).Pairwiseclassification.Asafinalevalua-tion,weusethe1,000rewrittensentencesfromSection3.3asaheld-outtestset.Thisallowsustotestthatourclassifierislearningrealstyledifferences,notjusttopicdifferences.Weas-sumethatworkers’rewritesindeedresultedinmoreformalsentences,andweframethetaskasapairwiseclassificationinwhichthegoalistodeterminewhichofthetwosentences(theoriginalortherewrite)ismoreformal.Aran-dombaselineachieves50%accuracy.IfweusetheF-Kreadabilitymeasure,andassumethesentencewiththehighergradelevelisthemoreformalofthetwo,weachieveonly57%accuracy.Byrunningoursupervisedregressionmodelandchoosingthesentencewiththehigherpredictedformalityscoreasthemoreformalsentence,weachieve88%accuracy,providingevidencethatthemodelpicksupsubtlestylistic,notjusttopic,differences.5FormalityinonlinediscussionsSofarwehavefocusedonbuildingamodelthatcanautomaticallydistinguishbetweenformalandinformalsentences.Wenowusethatmodeltoanalyzeformalityinpractice,inthecontextofonlinediscussionforums.Welooktoexist-ingtheoriesofformalityandoflinguisticstylematchingtoguideouranalysis.Inparticular:•Formalityishigherwhentheamountofsharedcontextbetweenspeakersislow(HeylighenandDewaele,1999).•Formalityishigherwhenspeakersdislikeoneanother(FieldingandFraser,1978).•Speakersadapttheirlanguageinordertomatchthelinguisticstyleofthosewithwhomtheyareinteracting(Danescu-Niculescu-Miziletal.,2011).LadywolfIwascheckingoutthiswebsiteforExodusInternationalandIunderstandtheirmissionistoprovideanalternativeforpeoplewhochoosetobeheterosexual.[…]Ijustfindithardtobelievethattheydon’tsomehowmanipulatethesituationinalessthanfairway.joebrummerIstartedathreadearlieraboutjustthis!ThesegroupsaredangerousLadywolf,Thereissomuchevidencetosupportthat[…]LadywolfIthoughtso[…]Ialsoseethattheyarerunningmajornewspaperads…hmmm,howunbiasedcananewspaperadlikethisbe?[…]I’msogladIwasn’traisedaChristianbecausefromthetoneofsomeofthereplies,somemembersofthiscultcanbeprettymeanhuh?joebrummerYes,Thearemeanfunnyenoughinthenameofgod.Iwasraisedchristian,catholicnoless.Istudiedthebible,IwasraisedbelievingIwouldgotohell.Thatwastough.LadywolfIbetthatwastough[…]IwasraisedJewish[…]It’slikesowierdbecauseI’veneverhadtodealwiththesetypesofpeoplebefore.Figure2:Exampleofathreadfromourdata.[…]indicatestexthasbeenleftouttosavespace.Withthesehypothesesinmind,weexplorehowformalitychangesacrosstopicsandusers(§5.2),howitrelatestootherpragmaticdimensions(§5.3),andhowitchangesoverthelifetimeofathread(§5.4).Understandingthesepatternsisanimportantfirststeptowardbuildingsystemsthatcaninteractwithpeopleinapragmaticallycompetentway.5.1DiscussionDataOurdatacomesfromtheInternetArgumentCorpus(IAC)dataset(Walkeretal.,2012),acorpusofthreadeddiscussionsfromonlinede-bateforums.Thedatasetconsistsof388Kpostscovering64differenttopics,fromEconomicstoEntertainment.Wefocusonthreadsinouranal-ysis,definedaschainsofpostsinwhicheachisanexplicitreplytothepreviouspost(Figure2).Whenthesameusermakesmultipleconsecutivepostsinathread(i.e.repliestotheirownpost),wecollapsetheseandtreatthemasasinglepost.Intotal,ourdatacovers104,625threads.AutomaticClassification.First,weassignaformalityscoretoeachpostinourdataus-ingtheAnswersmodelinSection4.Sincethismodelisdesignedforsentence-levelprediction,wedefinethescoreofaposttobethemeanscoreofthesentencesinthatpost.Weacknowl-edgethatthisapproximationisnotideal;tocon-firmthatitwillbesufficientforouranalyses,wecollecthumanjudgmentsfor1,000randompostsusingthesametasksetupasweusedforthesentence-leveljudgmentsinSection3.1.The
je
D
o
w
n
o
un
d
e
d
F
r
o
m
h
t
t
p
:
/
/
d
je
r
e
c
t
.
m
je
t
.
e
d
toi
/
t
un
c
je
/
je
un
r
t
je
c
e
–
p
d
F
/
d
o
je
/
.
1
0
1
1
6
2
/
t
je
un
c
_
un
_
0
0
0
8
3
1
5
6
7
3
6
0
/
/
t
je
un
c
_
un
_
0
0
0
8
3
p
d
.
F
b
oui
g
toi
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
69
correlationofourpredictedscorewiththemeanhumanscoreis0.58,whichiswithintherangeofinter-annotatoragreementforlabelingpostformality(0.34≤ρ≤0.64).13Wetakethisasconfirmationthatthemeanpredictedsentencescoreisadecentapproximationofhumanfor-malityjudgmentsforourpurposes.Figure3:Formalitydistributionofpostsin20mostpopulartopicsindiscussiondata.The10mostpopulartopics(*)areusedinourotheranalyses.5.2Howdotopicanduseraffectformality?Asformalityisintertwinedwithmanycontent-specificstyledimensionssuchas“serious-trivial”(Irvine,1979),weexpecttheoverallfor-malityleveltodifferacrosstopics.Figure3confirmsthis–manytopicsareclearlyskewedtowardbeingformal14(e.g.Economics)whileothersareskewedtowardinformal(e.g.Fun).Cependant,everytopicincludesbothformalandinformalposts:thereareinformalpostsinEco-nomics(“Ohmy!Apoorperson….howcouldthishavehappened!»)andformalpostsinFun(“Difficulttoconsidereitherone,ortheirvari-13Thisrangematchestheagreementrangeobservedforpost-levelpolitenessannotations(Danescu-Niculescu-Miziletal.,2013).Noteagreementismorevariedatthepostlevelthanatthesentencelevel.Thismakessensegiventhe“mixedformality”phenomenon:i.e.forlongposts,arangeofformalitycanbeexpressed,makingthechoiceofasinglescoremoresubjective.14Therangeofpostformalitiesisgenerallynarrowerthanwastherangeofsentenceformalities.Whilesentence-levelscoresrangebetween-3and3,wefindthat80%ofpostscoresfallbetween-1and1.ations,asaviablebeveragewhenbeerisavail-able.”).Weseeasimilarpatternwhenwelookatpostformalitylevelsbyuser:whilemostpeoplespeakgenerallyformallyorgenerallyinformally(84%ofusershaveameanformalitylevelthatissignificantlydifferentfrom0atp<0.01),nearlyeveryuser(91%)producesbothformalandin-formalposts.15Thisistrueevenwhenwelookatuserswithinonetopic.Theseresultsarein-teresting:theysuggestthatwhiletheformalityofapostisrelatedtothetopicofdiscussionandtotheindividualspeaker,thesealonedonotexplainformalityentirely.Rather,astheaforementionedtheoriessuggest,thesameper-sondiscussingthesametopicmaybecomemoreorlessformalinresponsetopragmaticfactors.5.3Howdoesformalityrelatetootherpragmaticstyles?Formalityisoftenconsideredtobehighlyre-latedwith,andeventosubsume,severalotherstylisticdimensionsincludingpoliteness,impar-tiality,andintimacy.HeylighenandDewaele(1999)suggestthatformalityishigherwhensharedsocialcontextislower,andthuslan-guageshouldbemoreformalwhendirectedatlargeraudiencesorspeakingaboutabstractcon-cepts.FieldingandFraser(1978)furthersug-gestthatinformalityisanimportantwayofex-pressingclosenesswithsomeone,andthusfor-malityshouldbehigherwhenspeakersdislikeoneanother.Toinvestigatetheseideasfurther,welookathowformalitycorrelateswithhumanjudgmentsofseveralotherpragmaticdimensions.Weusethemanualstyleannotationsthatarereleasedforasubsetofpost-replypairs(3Ktotal)intheIACdataset(Walkeretal.,2012).Theseanno-tationsinclude,forexample,theextenttowhichthereplyagrees/disagreeswiththepostandtheextenttowhichthereplyisinsulting/respectfulofthepost.EachofthesedimensionshasbeenratedbyhumanannotatorsonaLikertscale,similartoourownformalityannotations.Addi-tionally,toinvestigatehowformalitycorrelates15Weconsiderpostswithscores>0.25as“formal”andthosewithscores<−0.25as“informal.”
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
e
d
u
/
t
a
c
l
/
l
a
r
t
i
c
e
-
p
d
f
/
d
o
i
/
.
1
0
1
1
6
2
/
t
l
a
c
_
a
_
0
0
0
8
3
1
5
6
7
3
6
0
/
/
t
l
a
c
_
a
_
0
0
0
8
3
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
70
EmotionalThemaincauseofsomuchhateanddisrespectisthephonywarwe’refightingandourtacticsinviolationofinternationallaw,ourattitudeofsuperiorityintheworld,andourbullyingofothers.ImpoliteAsaformeradministrator,andthereforeaveteraneditorwhoknowshowwikipediareallyworks,Iamactuallysurprisedyouwouldevenasksuchaquestionwithsuchanobviousanswer.InsultingAndhereladiesandgentlemenwehavetheevidenceofwhyIamjustifiedincallingthelikesof‘stormboy’anidiot.SarcasticThankyouforbringingtomyattentionthatatoms,neutronsandprotonsaremerelyscientificassumptions.NowasIgazeatthenightskywithallitsbitsandpiecesspinningaroundeachotherIcansleephappilyknowingthatoursolarsystemisnotpartofahousebrickafterall.Table9:Formalpostsexhibitingstylepropertiesoftenthoughtnottoco-occurwithformality.withpoliteness,weusethetheStanfordPo-litenessCorpus(Danescu-Niculescu-Miziletal.,2013),whichconsistsof11KshortpostsfromWikipediadiscussionforumswhichagainhavebeenmanuallyannotatedonanordinalscale.Ourresultsaregenerallyconsistentwithwhattheoriessuggest.Wefindthatpostswhicharetargetedtowardmoregeneralaudiences(asop-posedtospecificpeople)andwhichmakefact-based(asopposedtoemotion-based)argumentsaregenerallymoreformal(ρ=0.32and0.17,respectively),andthatformalityissignificantlypositivelycorrelatedwithpoliteness(ρ=0.14).Wefindsignificantnegativecorrelationsbe-tweenformalityandtheextenttowhichthepostisseenassarcastic(ρ=−0.25)orinsulting(ρ=−0.22).Interestingly,wedonotfindasig-nificantcorrelationbetweenformalityandthedegreeofexpressedagreement/disagreement.Whilethedirectionsoftheserelationshipsmatchpriortheoriesandourintuitions,thestrengthofthecorrelationinmanycasesisweakerthanweexpectedtosee.Table9pro-videsexamplesofsomeofthelessintuitiveco-occurencesofstyle,e.g.impolitebutformalposts.Theseexamplesillustratethecomplex-ityofthenotionofformality,andhowformallanguagecanbeusedtogivetheimpressionofsocialdistancewhilestillallowingthespeaker’semotionsandpersonalitytobeveryapparent.5.4Howdoesformalitychangethroughoutadiscussion?Priorworkhasrevealedthatspeakersoftenadapttheirlanguagetomatchthelanguageofthosewithwhomtheyareinteracting(Danescu-Niculescu-Miziletal.,2011).Wethereforeinves-tigatehowformalitychangesoverthelifetimeofathread.Dodiscussionsbecomemoreorlessformalovertime?Dospeakers’levelsofformal-ityinteractwithoneanother?Fortheseanalyses,wefocusonthreadsfrom5to20postsinlength.Becausethreadscanbranch,multiplethreadsmightshareaprefixsequenceofposts.Toavoiddoublecounting,wegrouptogetherthreadswhichstemfromthesamepostandrandomlychoseonethreadfromeachsuchgroup,throwingawaytherest.Followingthetheorythatformalityisdeter-minedbythelevelofsharedcontext,HeylighenandDewaele(1999)hypothesizethatformalityshouldbehighestatthebeginningofaconversa-tion,whennocontexthasbeenestablished.Weobservethat,infact,thefirstpostshavesignif-icantlyhigherformalitylevelsonaveragethandotheremainingpostsinthethread(Figure4).Onceacontextisestablishedandadiscus-sionbegins,thetheoryoflinguisticstylematch-ingsuggeststhatpeoplechangetheirlanguagetomatchthatofothersintheconversation(NiederhofferandPennebaker,2002;Danescu-Niculescu-Miziletal.,2011).Isthisphe-nomenontrueofformality?Doesaperson’slevelofformalityreflecttheformalityofthosewithwhomtheyarespeaking?Figure2showsanexamplethreadinwhichthespeakerstogethermovetowardmoreinfor-maltoneastheconversationbecomesmoreper-sonal.Toseeifthiskindofformalitymatchingisthecaseingeneral,weusealinearmixedef-fectsmodel.16Briefly,amixedeffectsmodelis16Weusethemixedeffectsmodelwithrandominterceptsprovidedbythestatsmodelspythonpackage:http://statsmodels.sourceforge.net/devel/mixed_
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
e
d
u
/
t
a
c
l
/
l
a
r
t
i
c
e
-
p
d
f
/
d
o
i
/
.
1
0
1
1
6
2
/
t
l
a
c
_
a
_
0
0
0
8
3
1
5
6
7
3
6
0
/
/
t
l
a
c
_
a
_
0
0
0
8
3
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
71
InitialIwishtohaveaformaldebateintheDebateTournamentssectiononglobalwarming.Iproposethesubjecttitleof”GlobalWarmingisbothoccuringandhasbeenshowntobeatleastinpartcausedbyhumanactivity”Iwilltaketheafirmativeposition.Anyonewanttoarguetheopposite?ReplyGlobalwarmingisacontroversy.PersonallyIamlikehundredofmaybethousandsifnotmillionsofpeoplethatthinkitisliberal###.Theholeintheozonelayerisfalse,andIamsurethisistoo.InitialTheUSmilitarysaysthatSaddamHussein’sbriefcasecontainedtranscriptsofmeetingswithterrorists,contactinformationforthoseterrorists,andinformationonfinancialtransactionsthathecarriedout.[...]Iwonderwhatelsewasinthebriefcase.[...]ReplyTranscripts?Strange.Iwouldbecurioustoo.Figure4:Onaverage,firstpostsaresignificantlymoreformalthanlaterposts.Left:meanformalityofpostsbypositioninthread.Right:someexampleswhereformalinitialpostsarefollowedbylessformalreplies.(Note:4forums.comreplacesexpletiveswith#s.)aregressionanalysisthatallowsustomeasuretheinfluenceofvarious“fixedeffects”(e.g.theformalityofthepriorpost)onapost’sformality,whilecontrollingforthe“randomeffects”whichpreventusfromtreatingeverypostasaninde-pendentobservation.Inourcase,wetreatthetopicandtheauthorasrandomeffects,i.e.weacknowledgethattheformalitylevelsofpostsinthesametopicbythesameauthorarenotinde-pendent,andwewanttocontrolforthiswhenmeasuringtheeffectsofothervariables.Weinclude7fixedeffectsinourmodelofapost’sformality:theformalityofthepreviouspost,thenumberofpriorpostsinthethread(position),thenumberofpriorpostsbythisau-thorinthethread(veteranlevel),thelengthoftheentirethread,thetotalnumberofpartici-pantsintheentirethread,andthelengthsofthecurrentandpriorposts.Wealsoincludethepairwiseinteractionsbetweenthesefixedeffects.Weincludethetopicandauthorasarandomef-fect.Fortheseanalyses,weomitthefirstpostineverythread,asprioranalysissuggeststhatthefunctionofthefirstpost,anditsformality,ismarkedlydifferentfromthatoflaterposts.Table10givesthemostsignificantresultsfromourregression.Weobserveseveralinter-estingsignificanteffects,suchasanegativere-lationshipbetweenthenumberoftimesanau-thorhaspostedinthethreadandtheirformal-itylevel:i.e.peoplearemoreinformalthemoretheypost.However,thesinglebestpredictoroftheformalityofapostistheformalityoftheposttowhichitisreplying.Theestimatedef-linear.html.CoefficientPreviousscore0.219Veteranlevel−0.078Threadlength0.020Numberofparticipants−0.010Previousscore×position0.009Position0.008Table10:Estimatedcoefficientsofvariablesstronglyrelatedtotheformalityofapost,con-trollingfortopic-andauthor-specificrandomef-fects.Alleffectsaresignificantatp<0.0001.×signifiesaninteractionbetweenvariables.fectsizeis0.22,meaning,allelsebeingequal,weexpectanincreaseof1inthepriorpost’sformalitytocorrespondtoanincreaseof0.22intheformalityofthecurrentpost.Thissug-geststhataperson’sformalitydoesdependontheformalityofothersintheconversation.Perhapsmoreinterestingly,weseeasignifi-cantpositiveeffectoftheinteractionbetweenpreviousscoreandposition.Thatis,theeffectofpriorpostformalityoncurrentpostformalitybecomesstrongerlaterinathreadcomparedtoatthebeginningofathread.Figure5showshowtheestimatedcoefficientforpriorpostformalityoncurrentpostformalitychangeswhenwelookonlyatpostsataparticularindexinathread(e.g.onlysecondposts,onlytenthposts).Wecanseethatthecoefficientismorethantwiceaslargeforthetenthpostofathreadthanitisforthesecondpostinthatthread.Onecouldimag-ineseveralexplanationsforthis:i.e.userswithsimilarformalitylevelsmayengageinlongerdis-cussions,oruserswhoengageinlongerdiscus-
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
e
d
u
/
t
a
c
l
/
l
a
r
t
i
c
e
-
p
d
f
/
d
o
i
/
.
1
0
1
1
6
2
/
t
l
a
c
_
a
_
0
0
0
8
3
1
5
6
7
3
6
0
/
/
t
l
a
c
_
a
_
0
0
0
8
3
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
72
246810Position of post in thread0.00.10.20.30.40.50.60.7Estimated coefficient for prior post's formalityFigure5:Effectsizeofpriorposts’sformalityoncurrentpost’sformalityforpostsatdifferentpositionsinathread.Effectsizecanbeinter-pretedastheexpectedincreaseinapost’sfor-malitycorrespondingtoanincreaseof1inthepriorpost’sformality,allelsebeingequal.sionsmaytendtoadaptbettertooneanotherasthediscussionprogresses.Weleavefurtherinvestigationforfuturework.6ConclusionLanguagecontainsmorethanitsliteralcontent:stylisticvariationaccountsforalargepartofthemeaningthatiscommunicated.Formalityisoneofthemostbasicdimensionsofstylisticvaria-tioninlanguage,andtheabilitytorecognizeandrespondtodifferencesinformalityisanecessarypartoffulllanguageunderstanding.Thispaperhasprovidedananalysisofformalityinwrittencommunication.Wepresentedastudyofhumanperceptionsofformalityacrossmultiplegenres,andusedourfindingstobuildastatisticalmodelwhichapproximateshumanperceptionsoffor-malitywithhighaccuracy.Thismodelenabledustoinvestigatetrendsinformalityinonlinede-bateforums,revealingnewevidenceinsupportofexistingtheoriesaboutformalityandaboutlinguisticcoordination.Thesefindingsprovideimportantstepstowardbuildingpragmaticallycompetentautomatedsystems.Acknowledgements.WewouldliketothankMartinChodorowforvaluablediscussionandin-put,andMarilynWalker,ShereenOraby,andShibamouliLahiriforsharingandfacilitatingtheuseoftheirresources.WewouldalsoliketothankAmandaStent,DragomirRadev,ChrisCallison-Burch,andtheanonymousreviewersfortheirthoughtfulsuggestions.ReferencesFadiAbuSheikhaandDianaInkpen.2010.Auto-maticclassificationofdocumentsbyformality.InInterntionalConferenceonNaturalLanguagePro-cessingandKnowledgeEngineering(NLP-KE),pages1–5.IEEE.FadiAbuSheikhaandDianaInkpen.2011.Gen-erationofformalandinformalsentences.InPro-ceedingsofthe13thEuropeanWorkshoponNatu-ralLanguageGeneration,pages187–193,Nancy,France,September.AssociationforComputa-tionalLinguistics.CristinaBattaglinoandTimothyBickmore.2015.Increasingtheengagementofconversationalagentsthroughco-constructedstorytelling.EighthWorkshoponIntelligentNarrativeTechnologies.JulianBrookeandGraemeHirst.2014.Supervisedrankingofco-occurrenceprofilesforacquisitionofcontinuouslexicalattributes.InProceedingsofThe25thInternationalConferenceonComputa-tionalLinguistics.JulianBrooke,TongWang,andGraemeHirst.2010.Automaticacquisitionoflexicalformality.InCol-ing2010:Posters,pages90–98,Beijing,China,August.Coling2010OrganizingCommittee.PenelopeBrownandColinFraser.1979.Speechasamarkerofsituation.InSocialMarkersinSpeech,pages33–62.CambridgeUniversityPress.IdoDagan,OrenGlickman,andBernardoMagnini.2006.ThePASCALrecognisingtextualentail-mentchallenge.InMachineLearningChallenges.EvaluatingPredictiveUncertainty,VisualObjectClassification,andRecognisingTextualEntail-ment,pages177–190.Springer.CristianDanescu-Niculescu-Mizil,MichaelGamon,andSusanDumais.2011.Markmywords!:Lin-guisticstyleaccommodationinsocialmedia.InProceedingsofthe20thInternationalConferenceonWorldWideWeb,pages745–754.ACM.CristianDanescu-Niculescu-Mizil,LillianLee,BoPang,andJonKleinberg.2012.Echoesofpower:Languageeffectsandpowerdifferencesinsocialinteraction.InProceedingsofthe21stInternationalConferenceonWorldWideWeb,pages699–708.ACM.CristianDanescu-Niculescu-Mizil,MoritzSudhof,DanJurafsky,JureLeskovec,andChristopher
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
e
d
u
/
t
a
c
l
/
l
a
r
t
i
c
e
-
p
d
f
/
d
o
i
/
.
1
0
1
1
6
2
/
t
l
a
c
_
a
_
0
0
0
8
3
1
5
6
7
3
6
0
/
/
t
l
a
c
_
a
_
0
0
0
8
3
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
73
Potts.2013.Acomputationalapproachtopolite-nesswithapplicationtosocialfactors.Proceedingsofthe51stAnnualMeetingoftheAssociationforComputationalLinguistics(Volume1:LongPa-pers),pages250–259,August.BirgitEndrass,MatthiasRehm,andElisabethAndr´e.2011.Planningsmalltalkbehaviorwithculturalinfluencesformultiagentsystems.Com-puterSpeech&Language,25(2):158–174.AlexChengyuFangandJingCao.2009.Adjec-tivedensityasatextformalitycharacteristicforautomatictextclassification:Astudybasedonthebritishnationalcorpus.InProceedingsofthe23rdPacificAsiaConferenceonLanguage,Infor-mationandComputation,pages130–139,HongKong,December.CityUniversityofHongKong.RacheleDeFeliceandPaulDeane.2012.Identifyingspeechactsine-mails:TowardautomatedscoringoftheTOEICR(cid:13)e-mailtask.ETSResearchReportSeries,2012(2):i–62.GuyFieldingandColinFraser.1978.Languageandinterpersonalrelations.Thesocialcontextoflan-guage,pages217–232.FrancisHeylighenandJean-MarcDewaele.1999.Formalityoflanguage:Definition,measurementandbehavioraldeterminants.InternerBericht,Center“LeoApostel”,VrijeUniversiteitBr¨ussel.EduardHovy.1987.Generatingnaturallanguageunderpragmaticconstraints.JournalofPragmat-ics,11(6):689–719.JudithT.Irvine.1979.Formalityandinformalityincommunicativeevents.AmericanAnthropologist,81(4):773–790.W.LewisJohnson,RichardE.Mayer,ElisabethAndr´e,andMatthiasRehm.2005.Cross-culturalevaluationofpolitenessintacticsforpedagogicalagents.InAIED,volume5,pages298–305.RaquelJusto,ThomasCorcoran,StephanieM.Lukin,MarilynWalker,andM.In´esTorres.2014.Extractingrelevantknowledgeforthede-tectionofsarcasmandnastinessinthesocialweb.Knowledge-BasedSystems,69:124–133.FoaadKhosmoodandMarilynWalker.2010.Grapevine:Agossipgenerationsystem.InPro-ceedingsoftheFifthInternationalConferenceontheFoundationsofDigitalGames,pages92–99.ACM.J.PeterKincaid,RobertP.FishburneJr.,RichardL.Rogers,andBradS.Chissom.1975.Derivationofnewreadabilityformulas(automatedreadabil-ityindex,fogcountandFleschreadingeasefor-mula)fornavyenlistedpersonnel.Technicalre-port,DTICDocument.VinodhKrishnanandJacobEisenstein.2015.You’reMr.Lebowski,I’mtheDude:Inducingaddresstermformalityinsignedsocialnetworks.Proceedingsofthe2015ConferenceoftheNorthAmericanChapteroftheAssociationforCompu-tationalLinguistics:HumanLanguageTechnolo-gies,pages1616–1626,May–June.ShibamouliLahiri,PrasenjitMitra,andXiaofeiLu.2011.Informalityjudgmentatsentencelevelandexperimentswithformalityscore.InComputa-tionalLinguisticsandIntelligentTextProcessing,pages446–457.Springer.ShibamouliLahiri.2015.SQUINKY!Acorpusofsentence-levelformality,informativeness,andim-plicature.arXivpreprintarXiv:1506.02306.HaiyingLi,ZhiqiangCai,andArthurC.Graesser.2013.Comparingtwomeasuresforformality.InTheTwenty-SixthInternationalFLAIRSConfer-ence.Fran¸coisMairesseandMarilynA.Walker.2011.Controllinguserperceptionsoflinguisticstyle:Trainablegenerationofpersonalitytraits.Com-putationalLinguistics,37(3):455–488.Fran¸coisMairesse.2008.Learningtoadaptindia-loguesystems:Data-drivenmodelsforpersonalityrecognitionandgeneration.Ph.D.thesis,Univer-sityofSheffield,UnitedKingdom.AlejandroMosqueraandPalomaMoreda.2012a.Aqualitativeanalysisofinformalitylevelsinweb2.0texts:Thefacebookcasestudy.InProceedingsoftheLRECworkshop:@NLPcanutag#usergeneratedcontent,pages23–29.AlejandroMosqueraandPalomaMoreda.2012b.Smile:Aninformalityclassificationtoolforhelp-ingtoassessqualityandcredibilityinweb2.0texts.InProceedingsoftheICWSMworkshop:Real-TimeAnalysisandMiningofSocialStreams(RAMSS).KateG.NiederhofferandJamesW.Pennebaker.2002.Linguisticstylematchinginsocialinterac-tion.JournalofLanguageandSocialPsychology,21(4):337–360.ElliePavlickandAniNenkova.2015.Inducinglex-icalstylepropertiesforparaphraseandgenredif-ferentiation.InProceedingsofthe2015Confer-enceoftheNorthAmericanChapteroftheAssoci-ationforComputationalLinguistics:HumanLan-guageTechnologies,pages218–224,Denver,Col-orado,May–June.AssociationforComputationalLinguistics.KellyPeterson,MattHohensee,andFeiXia.2011.Emailformalityintheworkplace:AcasestudyontheEnroncorpus.InProceedingsoftheWork-shoponLanguageinSocialMedia(LSM2011),
l
D
o
w
n
o
a
d
e
d
f
r
o
m
h
t
t
p
:
/
/
d
i
r
e
c
t
.
m
i
t
.
e
d
u
/
t
a
c
l
/
l
a
r
t
i
c
e
-
p
d
f
/
d
o
i
/
.
1
0
1
1
6
2
/
t
l
a
c
_
a
_
0
0
0
8
3
1
5
6
7
3
6
0
/
/
t
l
a
c
_
a
_
0
0
0
8
3
p
d
.
f
b
y
g
u
e
s
t
t
o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3
74
pages86–95,Portland,Oregon,June.AssociationforComputationalLinguistics.NilsReiterandAnetteFrank.2010.Identifyinggenericnounphrases.InProceedingsofthe48thAnnualMeetingoftheAssociationforComputa-tionalLinguistics,pages40–49,Uppsala,Sweden,July.AssociationforComputationalLinguistics.PriyaSidhayeandJackieChiKitCheung.2015.In-dicativetweetgeneration:Anextractivesumma-rizationproblem?ProceedingsoftheConferenceonEmpiricalMethodsinNaturalLanguagePro-cessing.RobertJ.Sigley.1997.Textcategoriesandwhereyoucanstickthem:Acrudeformalityin-dex.InternationalJournalofCorpusLinguistics,2(2):199–237.SangweonSuh,HarryHalpin,andEwanKlein.2006.Extractingcommonsenseknowledgefromwikipedia.InProceedingsoftheWorkshoponWebContentMiningwithHumanLanguageTech-nologiesatISWC,volume6.MarilynA.Walker,JeanE.FoxTree,PranavAnand,RobAbbott,andJosephKing.2012.Acorpusforresearchondeliberationanddebate.TheInter-nationalConferenceonLanguageResourcesandEvaluation,pages812–817.
Télécharger le PDF