Transacciones de la Asociación de Lingüística Computacional, 2 (2014) 79–92. Editor de acciones: Mirella Lapata.
Submitted 12/2013; Publicado 2/2014. C
(cid:13)
2014 Asociación de Lingüística Computacional.
TheLanguageDemographicsofAmazonMechanicalTurkElliePavlick1MattPost2AnnIrvine2DmitryKachaev2ChrisCallison-Burch1,21ComputerandInformationScienceDepartment,UniversityofPennsylvania2HumanLanguageTechnologyCenterofExcellence,JohnsHopkinsUniversityAbstractWepresentalargescalestudyofthelanguagesspokenbybilingualworkersonMechanicalTurk(MTurk).Weestablishamethodologyfordeterminingthelanguageskillsofanony-mouscrowdworkersthatismorerobustthansimplesurveying.Wevalidateworkers’self-reportedlanguageskillclaimsbymeasuringtheirabilitytocorrectlytranslatewords,andbygeolocatingworkerstoseeiftheyresideincountrieswherethelanguagesarelikelytobespoken.Ratherthanpostingaone-offsurvey,wepostedpaidtasksconsistingof1,000as-signmentstotranslateatotalof10,000wordsineachof100languages.Ourstudyranforseveralmonths,andwashighlyvisibleontheMTurkcrowdsourcingplatform,increas-ingthechancesthatbilingualworkerswouldcompleteit.Ourstudywasusefulbothtocre-atebilingualdictionariesandtoactascen-susofthebilingualspeakersonMTurk.Weusethisdatatorecommendlanguageswiththelargestspeakerpopulationsasgoodcandidatesforotherresearcherswhowanttodevelopcrowdsourced,multilingualtechnologies.Tofurtherdemonstratethevalueofcreatingdataviacrowdsourcing,wehireworkerstocreatebilingualparallelcorporainsixIndianlan-guages,andusethemtotrainstatisticalma-chinetranslationsystems.1OverviewCrowdsourcingisapromisingnewmechanismforcollectingdatafornaturallanguageprocessingre-search.Accesstoafast,cheap,andflexiblework-forceallowsustocollectnewtypesofdata,poten-tiallyenablingnewlanguagetechnologies.BecausecrowdsourcingplatformslikeAmazonMechanicalTurk(MTurk)giveresearchersaccesstoaworld-wideworkforce,oneobviousapplicationofcrowd-sourcingisthecreationofmultilingualtechnologies.WithanincreasingnumberofactivecrowdworkerslocatedoutsideoftheUnitedStates,thereiseventhepotentialtoreachfluentspeakersoflowerresourcelanguages.Inthispaper,weinvestigatethefeasi-bilityofhiringlanguageinformantsonMTurkbyconductingthefirstlarge-scaledemographicstudyofthelanguagesspokenbyworkersontheplatform.Thereareseveralcomplicatingfactorswhentry-ingtotakeacensusofworkersonMTurk.Theworkers’identitiesareanonymized,andAmazonprovidesnoinformationabouttheircountriesofori-ginortheirlanguageabilities.Postingasimplesur-veytohaveworkersreportthisinformationmaybeinadequate,desde(a)manyworkersmayneverseethesurvey,(b)manyoptnottodoone-offsurveyssincepotentialpaymentislow,y(C)validatingtheanswersofrespondentsisnotstraightforward.Ourstudyestablishesamethodologyfordeter-miningthelanguagedemographicsofanonymouscrowdworkersthatismorerobustthansimplesur-veying.Weaskworkerswhatlanguagestheyspeakandwhatcountrytheylivein,andvalidatetheirclaimsbymeasuringtheirabilitytocorrectlytrans-latewordsandbyrecordingtheirgeolocation.Toincreasethevisibilityandthedesirabilityofourtasks,wepost1,000assignmentsineachof100lan-guages.Thesetaskseachconsistoftranslating10foreignwordsintoEnglish.Twoofthe10wordshaveknowntranslations,allowingustovalidatethattheworkers’translationsareaccurate.Weconstructbilingualdictionarieswithupto10,000entries,withthemajorityofentriesbeingnew.Surveyingthousandsofworkersallowsustoana-lyzecurrentspeakerpopulationsfor100languages.
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
yo
yo
i
t
.
mi
d
tu
/
t
a
C
yo
/
a
r
t
i
C
mi
–
pag
d
F
/
d
oh
i
/
1
0
.
1
1
6
2
/
t
a
C
_
a
_
0
0
1
6
7
/
1
5
6
6
8
7
9
/
t
a
C
_
a
_
0
0
1
6
7
.
pag
d
F
yo
b
y
gramo
tu
mi
s
t
oh
norte
0
7
S
mi
pag
t
mi
metro
b
mi
r
2
0
2
3
80
11/26/13turkermap.htmlfile:///Users/ellie/Documents/Research/turker-demographics/code/src/20130905/paper-rewrite/turkermap.html1/11111,9981,9981,998Figure1:Thenumberofworkerspercountry.ThismapwasgeneratedbasedongeolocatingtheIPaddressof4,983workersinourstudy.Omittedare60workerswhowerelocatedinmorethanonecountryduringthestudy,and238workerswhocouldnotbegeolocated.Thesizeofthecirclesrepresentsthenumberofworkersfromeachcountry.ThetwolargestareIndia(1,998workers)andtheUnitedStates(866).Tocalibratethesizes:thePhilippineshas142workers,Egypthas25,Russiahas10,andSriLankahas4.Thedataalsoallowsustoanswerquestionslike:Howquicklyisworkcompletedinagivenlanguage?Arecrowdsourcedtranslationsreliablygood?Howoftendoworkersmisrepresenttheirlanguageabili-tiestoobtainfinancialrewards?2BackgroundandRelatedWorkAmazon’sMechanicalTurk(MTurk)isanon-linemarketplaceforworkthatgivesemployersandresearchersaccesstoalarge,low-costwork-force.MTurkallowsemployerstoprovidemicro-paymentsinreturnforworkerscompletingmicro-tasks.ThebasicunitsofworkonMTurkarecalled‘HumanIntelligenceTasks’(HITs).MTurkwasde-signedtoaccommodatetasksthataredifficultforcomputers,butsimpleforpeople.Thisfacilitatesresearchintohumancomputation,wherepeoplecanbetreatedasafunctioncall(vonAhn,2005;Littleetal.,2009;QuinnandBederson,2011).Ithasappli-cationtoresearchareaslikehuman-computerinter-action(Bighametal.,2010;Bernsteinetal.,2010),computervision(SorokinandForsyth,2008;Dengetal.,2010;Rashtchianetal.,2010),speechpro-cessing(Margeetal.,2010;Laneetal.,2010;ParentandEskenazi,2011;Eskenazietal.,2013),andnatu-rallanguageprocessing(Snowetal.,2008;Callison-BurchandDredze,2010;Lawsetal.,2011).OnMTurk,researcherswhoneedworkcompletedarecalled‘Requesters’,andworkersareoftenre-ferredtoas‘Turkers’.MTurkisatruemarket,mean-ingthatTurkersarefreetochoosetocompletetheHITswhichinterestthem,andRequesterscanpricetheirtaskscompetitivelytotrytoattractworkersandhavetheirtasksdonequickly(Faridanietal.,2011;SingerandMittal,2011).Turkersremainanony-moustoRequesters,andallpaymentoccursthroughAmazon.Requestersareabletoacceptsubmittedworkorrejectworkthatdoesnotmeettheirstan-dards.TurkersareonlypaidifaRequesteracceptstheirwork.SeveralreportsexamineMechanicalTurkasaneconomicmarket(Ipeirotis,2010a;LehdonvirtaandErnkvist,2011).WhenAmazonintroducedMTurk,itfirstofferedpaymentonlyinAmazoncredits,andlateroffereddirectpaymentinUSdollars.Morere-cently,ithasexpandedtoincludeoneforeigncur-rency,theIndianrupee.Despiteitspaymentsbe-inglimitedtotwocurrenciesorAmazoncredits,MTurkclaimsoverhalfamillionworkersfrom190countries(Amazonas,2013).Thissuggeststhatitsworkerpopulationshouldrepresentadiversesetoflanguages.
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
yo
yo
i
t
.
mi
d
tu
/
t
a
C
yo
/
a
r
t
i
C
mi
–
pag
d
F
/
d
oh
i
/
1
0
.
1
1
6
2
/
t
a
C
_
a
_
0
0
1
6
7
/
1
5
6
6
8
7
9
/
t
a
C
_
a
_
0
0
1
6
7
.
pag
d
F
yo
b
y
gramo
tu
mi
s
t
oh
norte
0
7
S
mi
pag
t
mi
metro
b
mi
r
2
0
2
3
81
AdemographicstudybyIpeirotis(2010b)fo-cusedonage,género,martialstatus,incomelev-els,motivationforworkingonMTurk,andwhetherworkersuseditasaprimaryorsupplementalformofincome.ThestudycontrastedIndianandUSworkers.Rossetal.(2010)completedalongitudi-nalfollow-onstudy.AnumberofotherstudieshaveinformallyinvestigatedTurkers’languageabilities.MunroandTily(2011)compiledsurveyresponsesof2,000Turkers,revealingthatfourofthesixmostrepresentedlanguagescomefromIndia(thetopsixbeingHindi,Malayalam,Tamil,Español,Francés,andTelugu).IrvineandKlementiev(2010)hadTurkersevaluatetheaccuracyoftranslationsthathadbeenautomaticallyinductedfrommonolingualtexts.Theyexaminedtranslationsof100wordsin42low-resourcelanguages,andreportedgeolocatedcountriesfortheirworkers(India,theUS,Romania,Pakistán,Macedonia,Latvia,BangladeshandthePhilippines).IrvineandKlementievdiscussedthedifficultyofqualitycontrolandassessingtheplausi-bilityofworkers’languageskillsforrarelanguages,whichweaddressinthispaper.SeveralresearchershaveinvestigatedusingMTurktobuildbilingualparallelcorporaforma-chinetranslation,ataskwhichstandstobenefitlowcost,highvolumetranslationondemand(Ger-mann,2001).Ambatietal.(2010)conductedapilotstudybyposting25sentencestoMTurkforSpan-ish,Chino,Hindi,Telugu,Urdu,andHaitianCre-ole.Inastudyof2000Urdusentences,ZaidanandCallison-Burch(2011)presentedmethodsforachievingprofessional-leveltranslationqualityfromTurkersbysolicitingmultipleEnglishtranslationsofeachforeignsentence.Zbibetal.(2012)usedcrowdsourcingtoconstructa1.5millionwordpar-allelcorpusofdialectArabicandEnglish,train-ingastatisticalmachinetranslationsystemthatpro-ducedhigherqualitytranslationsofdialectArabicthanasystematrainedon100timesmoreMod-ernStandardArabic-Englishparalleldata.Zbibetal.(2013)conductedasystematicstudythatshowedthattraininganMTsystemoncrowdsourcedtrans-lationsresultedinthesameperformanceastrainingonprofessionaltranslations,at15thecost.Huetal.(2010;Huetal.(2011)performedcrowdsourcedtranslationbyhavingmonolingualspeakerscollab-orateanditerativelyimproveMToutput.English689Tamil253Malayalam219Hindi149Spanish131Telugu87Chinese86Romanian85Portuguese82Arabic74Kannada72German66French63Polish61Urdu56Tagalog54Marathi48Russian44Italian43Bengali41Gujarati39Hebrew38Dutch37Turkish35Vietnamese34Macedonian31Cebuano29Swedish26Bulgarian25Swahili23Hungarian23Catalan22Thai22Lithuanian21Punjabi21Others≤20Table1:Self-reportednativelanguageof3,216bilingualTurkers.Notshownare49languageswith≤20speakers.Weomit1,801Turkerswhodidnotreporttheirnativelanguage,243whoreported2na-tivelanguages,and83with≥3nativelanguages.Severalresearchershaveexaminedcostoptimiza-tionusingactivelearningtechniquestoselectthemostusefulsentencesorfragmentstotranslate(Am-batiandVogel,2010;BloodgoodandCallison-Burch,2010;Ambati,2012).Tocontrastourresearchwithpreviouswork,themaincontributionsofthispaperare:(1)arobustmethodologyforassessingthebilingualskillsofanonymousworkers,(2)thelargest-scalecensustodateoflanguageskillsofworkersonMTurk,y(3)adetailedanalysisofthedatagatheredinourstudy.3ExperimentalDesignThecentraltaskinthisstudywastoinvestigateMe-chanicalTurk’sbilingualpopulation.Weaccom-plishedthisthroughself-reportedsurveyscombinedwithaHITtotranslateindividualwordsfor100languages.Weevaluatetheaccuracyofthework-ers’translationsagainstknowntranslations.Incaseswherethesewerenotexactmatches,weusedasec-ondpassmonolingualHIT,whichaskedEnglishspeakerstoevaluateifaworker-providedtranslationwasasynonymoftheknowntranslation.DemographicquestionnaireAtthestartofeachHIT,Turkerswereaskedtocompleteabriefsurveyabouttheirlanguageabilities.Thesurveyaskedthefollowingquestions:•Is[idioma]yournativelanguage?•Howmanyyearshaveyouspoken[idioma]?
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
yo
yo
i
t
.
mi
d
tu
/
t
a
C
yo
/
a
r
t
i
C
mi
–
pag
d
F
/
d
oh
i
/
1
0
.
1
1
6
2
/
t
a
C
_
a
_
0
0
1
6
7
/
1
5
6
6
8
7
9
/
t
a
C
_
a
_
0
0
1
6
7
.
pag
d
F
yo
b
y
gramo
tu
mi
s
t
oh
norte
0
7
S
mi
pag
t
mi
metro
b
mi
r
2
0
2
3
82
•IsEnglishyournativelanguage?•HowmanyyearshaveyouspokenEnglish?•Whatcountrydoyoulivein?Weautomaticallycollectedeachworker’scurrentlo-cationbygeolocatingtheirIPaddress.Atotalof5,281uniqueworkerscompletedourHITs.Ofthese,3,625providedanswerstooursurveyquestions,andwewereabletogeolocate5,043.Figure1plotsthelocationofworkersacross106countries.Table1givesthemostcommonself-reportednativelan-guages.SelectionoflanguagesWedrewourdatafromthedifferentlanguageversionsofWikipedia.Wese-lectedthe100languageswiththelargestnumberofarticles1(Table2).Foreachlanguage,wechosethe1,000mostviewedarticlesovera1yearperiod,2andextractedthe10,000mostfrequentwordsfromthem.TheresultingvocabulariesservedastheinputtoourtranslationHIT.TranslationHITForthetranslationtask,weaskedTurkerstotranslateindividualwords.WeshowedeachwordinthecontextofthreesentencesthatweredrawnfromWikipedia.Turkerswereal-lowedtomarkthattheywereunabletotranslateaword.Eachtaskcontained10words,8ofwhichwerewordswithunknowntranslations,and2ofwhichwerequalitycontrolwordswithknowntrans-lations.Wegavespecialinstructionfortranslat-ingnamesofpeopleandplaces,givingexamplesofhowtohandle‘BarackObama’and‘Australia’usingtheirinterlanguagelinks.Forlanguageswithnon-Latinalphabets,namesweretransliterated.Thetaskpaid$0.15forthetranslationof10words.Eachsetof10wordswasindependentlytranslatedbythreeseparateworkers.5,281workerscompleted256,604translationassignments,totalingmorethan3millionwords,overaperiodofthreeandahalfmonths.GoldstandardtranslationsAsetofgoldstan-dardtranslationswereautomaticallyharvestedfrom1http://meta.wikimedia.org/wiki/List_of_Wikipedias2http://dumps.wikimedia.org/other/pagecounts-raw/500K+ARTICLES:Alemán(de),Inglés(en),Español(es),Francés(fr),italiano(él),Japanese(ja),Dutch(nl),Polish(pl),Portuguese(pt),Russian(ru)100K-500KARTICLES:Arábica(ar),Bulgarian(bg),Catalan(ca),checo(cs),Danish(da),Esperanto(eo),Basque(eu),persa(fa),Finnish(fi),hebreo(él),Hindi(hi),Croatian(hr),Hungarian(hu),Indonesian(id),Korean(ko),Lithuanian(lt),Malay(EM),Norwe-gian(Bokmal)(No),Romanian(ro),Slovak(sk),Slovenian(sl),Ser-bian(sr),Swedish(sv),Turkish(tr),UKrainian(Reino Unido),Vietnamese(vi),Waray-Waray(guerra),Chino(zh)10K-100KARTICLES:Afrikaans(af)Amharic(am)Asturian(ast)Azerbaijani(az)Belarusian(ser)Bengali(bn)BishnupriyaManipuri(bpy)Breton(br)Bosnian(bs)Cebuano(ceb)galés(cy)Zazaki(diq)Griego(el)WestFrisian(fy)Irish(ga)Galician(gl)Gujarati(gu)Haitian(ht)Armenian(hy)Icelandic(es)Javanese(jv)Geor-gian(ka)canarés(kn)Kurdish(ku)Luxembourgish(lb)Latvian(lv)Malagasy(mg)Macedonian(mk)Malayalam(ml)Marathi(mr)Neapolitan(nap)LowSaxon(nds)Nepali(ne)Newar/NepalBhasa(nuevo)Norwegian(Nynorsk)(nn)Piedmontese(pms)Sicil-ian(scn)Serbo-Croatian(sh)Albanian(sq)Sundanese(su)Swahili(sw)Tamil(frente a)Telugu(te)Thai(th)Tagalog(tl)Urdu(ur)Yoruba(yo)<10KARTICLES:CentralBicolano(bcl)Tibetan(bo)Ilokano(ilo)Punjabi(pa)Kapampangan(pam)Pashto(ps)Sindhi(sd)Somali(so)Uzbek(uz)Wolof(wo)Table2:Alistofthelanguagesthatwereusedinourstudy,groupedbythenumberofWikipediaarticlesinthelanguage.Eachlanguage’scodeisgiveninparentheses.Theselanguagecodesareusedinotherfiguresthroughoutthispaper.Wikipediaforeverylanguagetouseasembeddedcontrols.WeusedWikipedia’sinter-languagelinkstopairtitlesofEnglisharticleswiththeircorre-spondingforeignarticle’stitle.Togetamoretrans-latablesetofpairs,weexcludedanypairswhere:(1)theEnglishwordwasnotpresentintheWordNetontology(Miller,1995),(2)eitherarticletitlewaslongerthanasingleword,(3)theEnglishWikipediapagewasasubcategoryofpersonorplace,or(4)theEnglishandtheforeigntitleswereidenticalorasubstringoftheother.Manualevaluationofnon-identicaltranslationsWecountedalltranslationsthatexactlymatchedthegoldstandardtranslationascorrect.Fornon-exactmatcheswecreatedasecond-passqualityas-suranceHIT.TurkerswereshownapairofEn-glishwords,oneofwhichwasaTurker’stransla-tionoftheforeignwordusedforqualitycontrol,andtheotherofwhichwasthegold-standardtrans-lationoftheforeignword.Evaluatorswereaskedwhetherthetwowordshadthesamemeaning,andchosebetweenthreeanswers:‘Yes’,‘No’,or‘Re- l D o w n o a d e d f r o m h t t p : / / d i r e c t . m l l i t . e d u / t a c l / a r t i c e - p d f / d o i / 1 0 . 1 1 6 2 / t a c _ a _ 0 0 1 6 7 / 1 5 6 6 8 7 9 / t a c _ a _ 0 0 1 6 7 . p d f l b y g u e s t o n 0 7 S e p t e m b e r 2 0 2 3 83 Figure2:DaystocompletethetranslationHITsfor40ofthelanguages.Tickmarksrepresentthecom-pletionofindividualassignments.latedbutnotsynonymous.’Examplesofmean-ingequivalentpairsinclude:
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
yo
yo
i
t
.
mi
d
tu
/
t
a
C
yo
/
a
r
t
i
C
mi
–
pag
d
F
/
d
oh
i
/
1
0
.
1
1
6
2
/
t
a
C
_
a
_
0
0
1
6
7
/
1
5
6
6
8
7
9
/
t
a
C
_
a
_
0
0
1
6
7
.
pag
d
F
yo
b
y
gramo
tu
mi
s
t
oh
norte
0
7
S
mi
pag
t
mi
metro
b
mi
r
2
0
2
3
86
Avg.Turkerquality(#Ts)PrimarylocationsPrimarylocationsInregionOutofregionofTurkersinregionofTurkersoutofregionHindi0.63(296)0.69(7)India(284)UAE(5)Reino Unido(3)SaudiArabia(2)Russia(1)Oman(1)Tamil0.65(273)**0.25(2)India(266)US(3)Canada(2)Tunisia(1)Egypt(1)Malayalam0.76(234)0.83(2)India(223)UAE(6)US(3)SaudiArabia(1)Maldives(1)Spanish0.81(191)0.84(18)US(122)México(16)España(14)India(15)NewZealand(1)Brasil(1)French0.75(170)0.82(11)India(62)US(45)Francia(23)Greece(2)Países Bajos(1)Japón(1)Chinese0.60(116)0.55(21)US(75)Singapur(13)Porcelana(9)Hong Kong(6)Australia(3)Alemania(2)German0.82(91)0.77(41)Alemania(48)US(25)Austria(7)India(34)Países Bajos(1)Greece(1)Italian0.86(90)*0.80(42)Italia(42)US(29)Romania(7)India(33)Irlanda(2)España(2)Amharic0.14(16)**0.01(99)US(14)Ethiopia(2)India(70)Georgia(9)Macedonia(5)Kannada0.70(105)NA(0)India(105)Arabic0.74(60)**0.60(45)Egypt(19)Jordán(16)Morocco(9)US(19)India(11)Canada(3)Sindhi0.19(96)0.06(9)India(58)Pakistán(37)US(1)Macedonia(4)Georgia(2)Indonesia(2)Portuguese0.87(101)0.96(3)Brasil(44)Portugal(31)US(15)Romania(1)Japón(1)Israel(1)Turkish0.76(76)0.80(27)Pavo(38)US(18)Macedonia(8)India(19)Pakistán(4)Taiwán(1)Telugu0.80(102)0.50(1)India(98)US(3)UAE(1)SaudiArabia(1)Irish0.74(54)0.71(47)US(39)Irlanda(13)Reino Unido(2)India(36)Romania(5)Macedonia(2)Swedish0.73(54)0.71(45)US(25)Suecia(22)Finland(3)India(23)Macedonia(6)Croatia(2)Czech0.71(45)*0.61(50)US(17)CzechRepublic(14)Serbia(5)Macedonia(22)India(10)Reino Unido(5)Russian0.15(67)*0.12(27)US(36)Moldova(7)Russia(6)India(14)Macedonia(4)Reino Unido(3)Breton0.17(3)0.18(89)US(3)India(83)Macedonia(2)Porcelana(1)Table3:Translationqualitywhenpartitioningthetranslationsintotwogroups,onecontainingtranslationssubmittedbyTurkerswhoselocationiswithinregionsthatplausiblyspeaktheforeignlanguage,andtheothercontainingtranslationsfromTurkersoutsidethoseregions.Ingeneral,in-regionTurkersprovidehigherqualitytranslations.(**)indicatesdifferencessignificantatp=0.05,(*)atp=0.10.etal.,2013)tocompilethelistofcountrieswhereeachlanguageisspoken.Table3comparestheav-eragetranslationqualityofassignmentscompletedwithintheregionofeachlanguage,andcomparesittothequalityofassignmentscompletedoutsidethatregion.Ourworkersreportedspeaking95languagesna-tively.USworkersalonereported61nativelan-guages.Overall,4,297workerswerelocatedinaregionlikelytospeakthelanguagefromwhichtheyweretranslating,and2,778workerswerelocatedincountriesconsideredoutofregion(meaningthataboutathirdofour5,281TurkerscompletedHITsinmultiplelanguages).Table3showsthedifferencesintranslationqual-itywhencomputedusingin-regionversusout-of-regionTurkers,forthelanguageswiththegreatestnumberofworkers.Withinregionworkerstypi-callyproducedhigherqualitytranslations.GiventhenumberofIndianworkersonMechanicalTurk,itisunsurprisingthattheyrepresentmajorityofout-of-regionworkers.Forthelanguagesthathadmorethan75outofregionworkers(Malay,Amharic,Ice-landic,Sicilian,Wolof,andBreton),Indianworkersrepresentedatleast70%oftheoutofregionworkersineachlanguage.Afewlanguagesstandoutforhavingsuspiciouslystrongperformancebyoutofregionworkers,no-tablyIrishandSwedish,forwhichoutofregionworkersaccountforanearequivalentvolumeandqualityoftranslationstotheinregionworkers.Thisisadmittedlyimplausible,consideringtherelativelysmallnumberofIrishspeakersworldwide,andtheverylownumberlivinginthecountriesinwhichourTurkerswerebased(primarilyIndia).Suchresultshighlightthefactthatcheatingusingonlinetransla-tionresourcesisarealproblem,anddespiteourbesteffortstoremoveworkersusingGoogleTranslate,somecheatingisstillevident.Restrictingtowithinregionworkersisaneffectivewaytoreducetheprevalenceofcheating.Wediscussthelanguageswhicharebestsupportedbytruenativespeakersinsection6.SpeedoftranslationFigure2givesthecomple-tiontimesfor40languages.The10languagestofinishintheshortestamountoftimewere:Tamil,Malayalam,Telugu,Hindi,Macedonian,Español,Serbian,Romanian,Gujarati,andMarathi.SevenofthetenfastestlanguagesarefromIndia,whichisun-
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
yo
yo
i
t
.
mi
d
tu
/
t
a
C
yo
/
a
r
t
i
C
mi
–
pag
d
F
/
d
oh
i
/
1
0
.
1
1
6
2
/
t
a
C
_
a
_
0
0
1
6
7
/
1
5
6
6
8
7
9
/
t
a
C
_
a
_
0
0
1
6
7
.
pag
d
F
yo
b
y
gramo
tu
mi
s
t
oh
norte
0
7
S
mi
pag
t
mi
metro
b
mi
r
2
0
2
3
87
32024681012141618202224262830800,0000100,000200,000300,000400,000500,000600,000700,000MalayalamTamilTeluguHindiUrduBengaliFigure6:Thetotalvolumeoftranslations(measuredinEnglishwords)asafunctionofelapseddays.sentenceEnglish+dictionarylanguagepairsforeignwordsentriesBengali22k732k22kHindi40k1,488k22kMalayalam32k863k23kTamil38k916k25kTelugu46k1,097k21kUrdu35k1,356k20kTable4:Sizeofparallelcorporaandbilingualdic-tionariescollectedforeachlanguage.surprisinggiventhegeographicdistributionofwork-ers.Somelanguagesfollowthepatternofhavingasmatteringofassignmentscompletedearly,withtheratepickinguplater.Figure6givesthethroughputofthefull-sentencetranslationtaskforthesixIndianlanguages.ThefastestlanguagewasMalayalam,forwhichwecol-lectedhalfamillionwordsoftranslationsinjustun-deraweek.Table4givesthesizeofthedatasetthatwecreatedforeachoftheselanguages.TrainingSMTsystemsWetrainedstatisticaltranslationmodelsfromtheparallelcorporathatwecreatedforthesixIndianlanguagesusingtheJoshuamachinetranslationsystem(Postetal.,2012).Table5showsthetranslationperformancewhentrainedonthebitextsalone,andwhenincorporatingthebilingualdictionariescreatedinourearlierHIT.Thescoresreflecttheperformancewhentestedonheldoutsentencesfromthetrainingdata.Addingthedic-trainedonbitext+BLEUlanguagebitextsalonedictionaries∆Bengali12.0317.295.26Hindi16.1918.101.91Malayalam6.659.723.07Tamil8.089.661.58Telugu11.9413.701.76Urdu19.2221.982.76Table5:BLEUscoresfortranslatingintoEnglishusingbilingualparallelcorporabythemselves,andwiththeadditionofsingle-worddictionaries.ScoresarecalculatedusingfourreferencetranslationsandrepresentthemeanofthreeMERTruns.tionariestothetrainingsetproducesconsistentper-formancegains,rangingfrom1to5BLEUpoints.Thisrepresentsasubstantialimprovement.Itisworthnoting,sin embargo,thatwhilethesourcedoc-umentsforthefullsentencesusedfortestingwerekeptdisjointfromthoseusedfortraining,thereisoverlapbetweenthesourcematerialsforthedictio-nariesandthosefromthetestset,sinceboththedic-tionariesandthebitextsourcesentencesweredrawnfromWikipedia.6DiscussionCrowdsourcingplatformslikeMechanicalTurkgiveresearchersinstantaccesstoadiversesetofbilin-gualworkers.Thisopensupexcitingnewavenuesforresearcherstodevelopnewmultilingualsystems.Thedemographicsreportedinthisstudyarelikelytoshiftovertime.Amazonmayexpanditspaymentstonewcurrencies.Postinglong-runningHITsinotherlanguagesmayrecruitmorespeakersofthoselan-guages.Newcrowdsourcingplatformsmayemerge.Thedatapresentedhereprovidesavaluablesnap-shotofthecurrentstateofMTurk,andthemethodsusedcanbeappliedgenerallyinfutureresearch.Basedonourstudy,wecanconfidentlyrecom-mend13languagesasgoodcandidatesforresearchnow:Dutch,Francés,Alemán,Gujarati,italiano,Kan-nada,Malayalam,Portuguese,Romanian,Serbian,Español,Tagalog,andTelugu.TheselanguageshavelargeTurkerpopulationswhocompletetasksquicklyandaccurately.Table6summarizesthestrengthsandweaknessesofall100languagescov-eredinourstudy.Severalotherlanguagesareviable
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
yo
yo
i
t
.
mi
d
tu
/
t
a
C
yo
/
a
r
t
i
C
mi
–
pag
d
F
/
d
oh
i
/
1
0
.
1
1
6
2
/
t
a
C
_
a
_
0
0
1
6
7
/
1
5
6
6
8
7
9
/
t
a
C
_
a
_
0
0
1
6
7
.
pag
d
F
yo
b
y
gramo
tu
mi
s
t
oh
norte
0
7
S
mi
pag
t
mi
metro
b
mi
r
2
0
2
3
88
workersqualityspeedmanyhighfastDutch,Francés,Alemán,Gu-jarati,italiano,canarés,Malay-alam,Portuguese,Romanian,Serbian,Español,Tagalog,Tel-uguslowArabic,hebreo,Irish,Punjabi,Swedish,TurkishlowfastHindi,Marathi,Tamil,UrduormediumslowBengali,BishnupriyaMa-nipuri,Cebuano,Chino,Nepali,Newar,Polish,Russian,Sindhi,TibetanfewhighfastBosnia,Croatian,Macedonian,Malay,Serbo-CroatianslowAfrikaans,Albanian,Aragonese,Asturian,Basque,Belarusian,Bulgarian,CentralBicolano,checo,Danish,Finnish,Galacian,Griego,Haitian,Hungarian,Icelandic,Ilokano,Indonesian,Japanese,Javanese,Kapampangan,Kazakh,Korean,Lithuanian,LowSaxon,Malagasy,Nor-wegian(Bokmal),Sicilian,Slovak,Slovenian,Thai,UKra-nian,Uzbek,Waray-Waray,WestFrisian,Yorubalowfast–ormediumslowAmharic,Armenian,Azer-baijani,Breton,Catalan,Georgian,Latvian,Luxembour-gish,Neapolitian,Norwegian(Nynorsk),Pashto,Pied-montese,Somali,Sudanese,Swahili,Tatar,Vietnamese,Walloon,WelshnonelowormediumslowEsperanto,Ido,Kurdish,Per-sian,Quechua,Wolof,ZazakiTable6:ThegreenboxshowsthebestlanguagestotargetonMTurk.Theselanguageshavemanywork-erswhogeneratehighqualityresultsquickly.Wedefinedmanyworkersas50ormoreactivein-regionworkers,highqualityas≥70%accuracyonthegoldstandardcontrols,andfastifallofthe10,000wordswerecompletedwithintwoweeks.candidatesprovidedadequatequalitycontrolmech-anismsareusedtoselectgoodworkers.SinceMechanicalTurkprovidesfinancialincen-tivesforparticipation,manyworkersattempttocompletetaskseveniftheydonothavethelan-guageskillsnecessarytodoso.SinceMTurkdoesnotprovideanyinformationaboutworkersdemo-graphics,includingtheirlanguagecompetencies,itcanbehardtoexcludesuchworkers.AsaresultnaivedatacollectiononMTurkmayresultinnoisydata.Avarietyoftechniquesshouldbeincorporatedintocrowdsourcingpipelinestoensurehighqualitydata.Asabestpractice,wesuggest:(1)restrictingworkerstocountriesthatplausiblyspeaktheforeignlanguageofinterest,(2)embeddinggoldstandardcontrolsoradministeringlanguagepretests,ratherthanrelyingsolelyonself-reportedlanguageskills,y(3)excludingworkerswhosetranslationshavehighoverlapwithonlinemachinetranslationsys-temslikeGoogletranslate.Ifcheatingusingexter-nalresourcesislikely,thenalsoconsider(4)record-inginformationliketimespentonaHIT(cumulativeandonindividualitems),patternsinkeystrokelogs,tab/windowfocus,etc.AlthoughourstudytargetedbilingualworkersonMechanicalTurk,andneglectedmonolingualwork-ers,webelieveourresultsreliablyrepresentthecur-rentspeakerpopulations,sincethevastmajorityoftheworkavailableonthecrowdsourcedplatformiscurrentlyEnglish-only.Wethereforeassumethenumberofnon-Englishspeakersissmall.Inthefu-ture,itmaybedesirabletorecruitmonolingualfor-eignworkers.Insuchcases,werecommendotherteststovalidatetheirlanguageabilitiesinplaceofourtranslationtest.Thesecouldincludeperform-ingnarrativecloze,orlisteningtoaudiofilescon-tainingspeechindifferentlanguageandidentifyingtheirlanguage.7DatareleaseWiththepublicationofthispaper,wearereleasingalldataandcodeusedinthisstudy.Ourdatareleaseincludestherawdata,alongwithbilingualdictionar-iesthatarefilteredtobehighquality.Itwillinclude256,604translationassignmentsfrom5,281Turkersand20,952synonymassignmentsfrom1,005Turk-ers,alongwithmetainformationlikegeolocation
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
yo
yo
i
t
.
mi
d
tu
/
t
a
C
yo
/
a
r
t
i
C
mi
–
pag
d
F
/
d
oh
i
/
1
0
.
1
1
6
2
/
t
a
C
_
a
_
0
0
1
6
7
/
1
5
6
6
8
7
9
/
t
a
C
_
a
_
0
0
1
6
7
.
pag
d
F
yo
b
y
gramo
tu
mi
s
t
oh
norte
0
7
S
mi
pag
t
mi
metro
b
mi
r
2
0
2
3
89
andtimesubmitted,plusexternaldictionariesusedforvalidation.Thedictionarieswillcontain1.5Mtotaltranslatedwordsin100languages,alongwithcodetofilterthedictionariesbasedondifferentcri-teria.ThedataalsoincludesparallelcorporaforsixIndianlanguages,ranginginsizebetween700,000to1.5millionwords.8AcknowledgementsThismaterialisbasedonresearchsponsoredbyaDARPAComputerScienceStudyPanelphase3awardentitled“CrowdsourcingTranslation”(con-tractD12PC00368).Theviewsandconclusionscontainedinthispublicationarethoseoftheauthorsandshouldnotbeinterpretedasrepresentingoffi-cialpoliciesorendorsementsbyDARPAortheU.S.Government.ThisresearchwassupportedbytheJohnsHopkinsUniversityHumanLanguageTech-nologyCenterofExcellenceandthroughgiftsfromMicrosoftandGoogle.Theauthorswouldliketothanktheanonymousreviewersfortheirthoughtfulcomments,whichsub-stantiallyimprovedthispaper.ReferencesAmazon.2013.Servicesummarytourforre-questersonAmazonMechanicalTurk.https://requester.mturk.com/tour.VamshiAmbatiandStephanVogel.2010.Cancrowdsbuildparallelcorporaformachinetranslationsystems?InProceedingsoftheNAACLHLT2010WorkshoponCreatingSpeechandLanguageDatawithAmazon’sMechanicalTurk.AssociationforComputationalLin-guistics.VamshiAmbati,StephanVogel,andJaimeCarbonell.2010.Activelearningandcrowd-sourcingforma-chinetranslation.InProceedingsofthe7thInterna-tionalConferenceonLanguageResourcesandEvalu-ation(LREC).VamshiAmbati.2012.ActiveLearningandCrowd-sourcingforMachineTranslationinLowResourceScenarios.Ph.D.thesis,LanguageTechnologiesIn-stitute,SchoolofComputerScience,CarnegieMellonUniversity,pittsburgh,PA.MichaelS.Bernstein,GregLittle,RobertC.Miller,BjrnHartmann,MarkS.Ackerman,DavidR.Karger,DavidCrowell,andKatrinaPanovich.2010.Soylent:awordprocessorwithacrowdinside.InProceed-ingsoftheACMSymposiumonUserInterfaceSoft-wareandTechnology(UIST).JeffreyP.Bigham,ChandrikaJayant,HanjieJi,GregLit-tle,AndrewMiller,RobertC.Miller,RobinMiller,AubreyTatarowicz,BrandynWhite,SamualWhite,andTomYeh.2010.VizWiz:nearlyreal-timean-swerstovisualquestions.InProceedingsoftheACMSymposiumonUserInterfaceSoftwareandTechnol-ogy(UIST).MichaelBloodgoodandChrisCallison-Burch.2010.Large-scalecost-focusedactivelearningforstatisti-calmachinetranslation.InProceedingsofthe48thAnnualMeetingoftheAssociationforComputationalLinguistics.ChrisCallison-BurchandMarkDredze.2010.CreatingspeechandlanguagedatawithAmazon’sMechanicalTurk.InProceedingsoftheNAACLHLT2010Work-shoponCreatingSpeechandLanguageDatawithAmazon’sMechanicalTurk,pages1–12,LosAngeles,June.AssociationforComputationalLinguistics.JiaDeng,AlexanderBerg,KaiLi,andLiFei-Fei.2010.Whatdoesclassifyingmorethan10,000imagecate-goriestellus?InProceedingsofthe12thEuropeanConferenceofComputerVision(ECCV,pages71–84.MaxineEskenazi,Gina-AnneLevow,HelenMeng,GabrielParent,andDavidSuendermann.2013.CrowdsourcingforSpeechProcessing,ApplicationstoDataCollection,TranscriptionandAssessment.Wi-ley.SiamakFaridani,Bj¨ornHartmann,andPanagiotisG.Ipeirotis.2011.What’stherightprice?pricingtasksforfinishingontime.InThirdAAAIHumanCompu-tationWorkshop(HCOMP’11).UlrichGermann.2001.Buildingastatisticalmachinetranslationsystemfromscratch:Howmuchbangforthebuckcanweexpect?InACL2001WorkshoponData-DrivenMachineTranslation,Tolosa,France.ChangHu,BenjaminB.Bederson,andPhilipResnik.2010.Translationbyiterativecollaborationbetweenmonolingualusers.InProceedingsofACMSIGKDDWorkshoponHumanComputation(HCOMP).ChangHu,PhilipResnik,YakovKronrod,VladimirEi-delman,OliviaBuzek,andBenjaminB.Bederson.2011.Thevalueofmonolingualcrowdsourcinginareal-worldtranslationscenario:Simulationusinghaitiancreoleemergencysmsmessages.InPro-ceedingsoftheSixthWorkshoponStatisticalMa-chineTranslation,pages399–404,Edinburgh,Scot-land,July.AssociationforComputationalLinguistics.PanagiotisG.Ipeirotis.2010a.Analyzingthemechani-calturkmarketplace.InACMXRDS,December.PanagiotisG.Ipeirotis.2010b.DemographicsofMechanicalTurk.TechnicalReportWorkingpaper
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
yo
yo
i
t
.
mi
d
tu
/
t
a
C
yo
/
a
r
t
i
C
mi
–
pag
d
F
/
d
oh
i
/
1
0
.
1
1
6
2
/
t
a
C
_
a
_
0
0
1
6
7
/
1
5
6
6
8
7
9
/
t
a
C
_
a
_
0
0
1
6
7
.
pag
d
F
yo
b
y
gramo
tu
mi
s
t
oh
norte
0
7
S
mi
pag
t
mi
metro
b
mi
r
2
0
2
3
90
CeDER-10-01,NewYorkUniversity,SternSchoolofBusiness.AnnIrvineandAlexandreKlementiev.2010.UsingMe-chanicalTurktoannotatelexiconsforlesscommonlyusedlanguages.InWorkshoponCreatingSpeechandLanguageDatawithMTurk.IanLane,MatthiasEck,KayRottmann,andAlexWaibel.2010.Toolsforcollectingspeechcorporaviamechanical-turk.InProceedingsoftheNAACLHLT2010WorkshoponCreatingSpeechandLan-guageDatawithAmazon’sMechanicalTurk,LosAn-geles.FlorianLaws,ChristianScheible,andHinrichSch¨utze.2011.Activelearningwithamazonmechanicalturk.InProceedingsofthe2011ConferenceonEmpiricalMethodsinNaturalLanguageProcessing,Edimburgo,Scotland.MatthewLease,JessicaHullman,JeffreyP.Bigham,JuhoKimMichaelS.Bernsteinand,WalterLasecki,SaeidehBakhshi,TanushreeMitra,andRobertC.Miller.2013.MechanicalTurkisnotanony-mous.http://dx.doi.org/10.2139/ssrn.2228728.ViliLehdonvirtaandMirkoErnkvist.2011.Knowl-edgemapofthevirtualeconomy:Convertingthevirtualeconomyintodevelopmentpotential.http://www.infodev.org/en/Document.1056.pdf,April.AnInfoDevPublication.M.PaulLewis,GaryF.Simons,andCharlesD.Fennig(editores.).2013.Ethnologue:Languagesoftheworld,seventeenthedition.http://www.ethnologue.com.GregLittle,LydiaB.Chilton,RobMiller,andMaxGold-man.2009.Turkit:Toolsforiterativetasksonme-chanicalturk.InProceedingsoftheWorkshoponHumanComputationattheInternationalConferenceonKnowledgeDiscoveryandDataMining(KDD-HCOMP’09),Paris.MatthewMarge,SatanjeevBanerjee,andAlexanderRudnicky.2010.UsingtheAmazonMechanicalTurktotranscribeandannotatemeetingspeechforextrac-tivesummarization.InWorkshoponCreatingSpeechandLanguageDatawithMTurk.GeorgeA.Miller.1995.WordNet:alexicaldatabaseforenglish.CommunicationsoftheACM,38(11):39–41.RobertMunroandHalTily.2011.Thestartoftheart:Introductiontotheworkshoponcrowdsourcingtechnologiesforlanguageandcognitionstudies.InCrowdsourcingTechnologiesforLanguageandCog-nitionStudies,Boulder.ScottNovotneyandChrisCallison-Burch.2010.Cheap,fastandgoodenough:Automaticspeechrecognitionwithnon-experttranscription.InHumanLanguageTechnologies:The2010AnnualConferenceoftheNorthAmericanChapteroftheAssociationforCom-putationalLinguistics,pages207–215.AssociationforComputationalLinguistics.GabrielParentandMaxineEskenazi.2011.Speakingtothecrowd:lookingatpastachievementsinusingcrowdsourcingforspeechandpredictingfuturechal-lenges.InProceedingsInterspeech2011,SpecialSes-siononCrowdsourcing.MattPost,ChrisCallison-Burch,andMilesOsborne.2012.Constructingparallelcorporaforsixindianlanguagesviacrowdsourcing.InProceedingsoftheSeventhWorkshoponStatisticalMachineTranslation,pages401–409,Montr´eal,Canada,June.AssociationforComputationalLinguistics.AlexanderJ.QuinnandBenjaminB.Bederson.2011.Humancomputation:Asurveyandtaxonomyofagrowingfield.InComputerHumanInteraction(CHI).CyrusRashtchian,PeterYoung,MicahHodosh,andJu-liaHockenmaier.2010.CollectingimageannotationsusingAmazon’sMechanicalTurk.InWorkshoponCreatingSpeechandLanguageDatawithMTurk.JoelRoss,LillyIrani,M.SixSilberman,AndrewZal-divar,andBillTomlinson.2010.Whoarethecrowd-workers?:ShiftingdemographicsinAmazonMechan-icalTurk.Inalt.CHIsessionofCHI2010extendedabstractsonhumanfactorsincomputingsystems,At-lanta,Georgia.YaronSingerandManasMittal.2011.Pricingmecha-nismsforonlinelabormarkets.InThirdAAAIHumanComputationWorkshop(HCOMP’11).RionSnow,BrendanO’Connor,DanielJurafsky,andAndrewY.Ng.2008.Cheapandfast-butisitgood?Evaluatingnon-expertannotationsfornaturallanguagetasks.InProceedingsofEMNLP.AlexanderSorokinandDavidForsyth.2008.Utilitydataannotationwithamazonmechanicalturk.InFirstIEEEWorkshoponInternetVisionatCVPR.LuisvonAhn.2005.HumanComputation.Ph.D.thesis,SchoolofComputerScience,CarnegieMellonUni-versity,pittsburgh,PA.OmarF.ZaidanandChrisCallison-Burch.2011.Crowd-sourcingtranslation:Professionalqualityfromnon-professionals.InProceedingsofthe49thAnnualMeetingoftheAssociationforComputationalLin-guistics:HumanLanguageTechnologies,pages1220–1229.AssociationforComputationalLinguistics.RabihZbib,ErikaMalchiodi,JacobDevlin,DavidStallard,SpyrosMatsoukas,RichardSchwartz,JohnMakhoul,OmarF.Zaidan,andChrisCallison-Burch.2012.MachinetranslationofArabicdialects.InThe2012ConferenceoftheNorthAmericanChapteroftheAssociationforComputationalLinguistics.Asso-ciationforComputationalLinguistics.
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
yo
yo
i
t
.
mi
d
tu
/
t
a
C
yo
/
a
r
t
i
C
mi
–
pag
d
F
/
d
oh
i
/
1
0
.
1
1
6
2
/
t
a
C
_
a
_
0
0
1
6
7
/
1
5
6
6
8
7
9
/
t
a
C
_
a
_
0
0
1
6
7
.
pag
d
F
yo
b
y
gramo
tu
mi
s
t
oh
norte
0
7
S
mi
pag
t
mi
metro
b
mi
r
2
0
2
3
91
RabihZbib,GretchenMarkiewicz,SpyrosMatsoukas,RichardSchwartz,andJohnMakhoul.2013.Sys-tematiccomparisonofprofessionalandcrowdsourcedreferencetranslationsformachinetranslation.InPro-ceedingsofthe2013ConferenceoftheNorthAmer-icanChapteroftheAssociationforComputationalLinguistics:HumanLanguageTechnologies,Atlanta,Georgia.
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
yo
yo
i
t
.
mi
d
tu
/
t
a
C
yo
/
a
r
t
i
C
mi
–
pag
d
F
/
d
oh
i
/
1
0
.
1
1
6
2
/
t
a
C
_
a
_
0
0
1
6
7
/
1
5
6
6
8
7
9
/
t
a
C
_
a
_
0
0
1
6
7
.
pag
d
F
yo
b
y
gramo
tu
mi
s
t
oh
norte
0
7
S
mi
pag
t
mi
metro
b
mi
r
2
0
2
3