MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages Xinyu Zhang1∗, Nandan Thakur1∗, Odunayo Ogundepo1, Ehsan Kamalloo1†, David Alfonso-Hermelo2, Xiaoguang Li3, Qun Liu3, Mehdi Rezagholizadeh2, Jimmy Lin1 1David R. Cheriton School of Computer Science, University of Waterloo, Canada 2Huawei Noah’s Ark Lab, Canada 3Huawei Noah’s Ark Lab, China Abstract MIRACL is a multilingual dataset for ad hoc retrieval across 18 languages that collectively encompass over

Exploring Contrast Consistency of Open-Domain Question Answering Systems on Minimally Edited Questions Zhihan Zhang, Wenhao Yu, Zheng Ning, Mingxuan Ju, Meng Jiang University of Notre Dame, Notre Dame, IN, USA {zzhang23, wyu1, zning, mju2, mjiang2}@nd.edu Abstract Contrast consistency, the ability of a model to make consistently correct predictions in the presence of perturbations, is an essential aspect in NLP. While studied in tasks such as

Cross-functional Analysis of Generalization in Behavioral Learning Pedro Henrique Luz de Araujo1,2 and Benjamin Roth1,3 1Faculty of Computer Science, University of Vienna, Vienna, Austria 2UniVie Doctoral School Computer Science, Vienna, Austria 3Faculty of Philological and Cultural Studies, University of Vienna, Vienna, Austria {pedro.henrique.luz.de.araujo, benjamin.roth}@univie.ac.at Abstract In behavioral testing, system functionalities underrepresented in the standard evaluation setting (with a held-out test set) are validated through controlled

A Cross-Linguistic Pressure for Uniform Information Density in Word Order Thomas Hikaru Clark1 Clara Meister2 Tiago Pimentel3 Michael Hahn4 Ryan Cotterell2 Richard Futrell5 Roger Levy1 1MIT, USA 2ETH Z¨urich, Switzerland 3University of Cambridge, UK 4Saarland University, Germany 5UC Irvine, USA thclark@mit.edu meistecl@inf.ethz.ch tp472@cam.ac.uk mhahn@lst.uni-saarland.de ryan.cotterell@inf.ethz.ch rfutrell@uci.edu rplevy@mit.edu Abstract While natural languages differ widely in both canonical word order and word order flex- ibility, their word

Time-and-Space-Efficient Weighted Deduction Jason Eisner Department of Computer Science Johns Hopkins University jason@cs.jhu.edu Abstract Many NLP algorithms have been described in terms of deduction systems. Unweighted deduction allows a generic forward-chaining execution strategy. For weighted deduction, however, efficient execution should propa- gate the weight of each item only after it has converged. This means visiting the items in topologically sorted order (as in dynamic programming).

Communication Drives the Emergence of Language Universals in Neural Agents: Evidence from the Word-order/Case-marking Trade-off Yuchen Lian(cid:2) † Arianna Bisazza‡∗ (cid:2)Faculty of Electronic and Information Engineering, Xi’an Jiaotong University, China †Leiden Institute of Advanced Computer Science, Leiden University, The Netherlands {y.lian, t.verhoef}@liacs.leidenuniv.nl ‡Center for Language and Cognition, University of Groningen, The Netherlands a.bisazza@rug.nl Tessa Verhoef †∗ Abstract Artificial learners often behave differently from human learners

Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design Valentina Pyatkin1 Frances Yung2 Merel C. J. Scholman2,3 Reut Tsarfaty1 Ido Dagan1 Vera Demberg2 1Bar Ilan University, Ramat Gan, Israel 2Saarland University, Saarbr¨ucken, Germany 3Utrecht University, Utrecht, Netherlands {pyatkiv,reut.tsarfaty}@biu.ac.il; dagan@cs.biu.ac.il {frances,m.c.j.scholman,vera}@coli.uni-saarland.de Abstract Disagreement in natural language annotation has mostly been studied from a perspective of biases introduced by the annotators and

Collective Human Opinions in Semantic Textual Similarity Yuxia Wang♠ Shimin Tao♣ Timothy Baldwin♠♥ Ning Xie♣ Karin Verspoor♠♦ Hao Yang♣ ♠ The University of Melbourne, Melbourne, Victoria, Australia ♣ Huawei TSC, Beijing, China ♥ MBZUAI, Abu Dhabi, UAE ♦RMIT University, Melbourne, Victoria, Australia yuxiaw@student.unimelb.edu.au karin.verspoor@rmit.edu.au {taoshimin,nicolas.xie,yanghao30}@huawei.com tb@ldwin.net l D o w n o a d e d f r o m h t t p :

Conditional Generation with a Question-Answering Blueprint Shashi Narayan1, Joshua Maynez1, Reinald Kim Amplayo1, Kuzman Ganchev1, Annie Louis2, Fantine Huot1, Anders Sandholm2, Dipanjan Das1, Mirella Lapata1 1Google DeepMind, UK 2Google Research shashinarayan@google.com, joshuahm@google.com, reinald@google.com, kuzman@google.com, annielouis@google.com, fantinehuot@google.com, sandholm@google.com, dipanjand@google.com, lapata@google.com Abstract The ability to convey relevant and faithful in- formation is critical for many tasks in con- ditional generation and yet remains elusive for neural seq-to-seq

Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generation Fei Huang Pei Ke Minlie Huang∗ The CoAI group, Tsinghua University, Beijing, China Institute for Artificial Intelligence, State Key Lab of Intelligent Technology and Systems, Beijing National Research Center for Information Science and Technology, Department of Computer Science and Technology, Tsinghua University, Beijing, China f-huang18@mails.tsinghua.edu.cn, kepei1106@outlook.com, aihuang@tsinghua.edu.cn Abstract Non-AutoRegressive (NAR) text generation models have drawn much

Reasoning over Public and Private Data in Retrieval-Based Systems Simran Arora3, Patrick Lewis4∗∗, Angela Fan1, Jacob Kahn2∗, and Christopher R´e3∗ 1Facebook AI Research, France 2Facebook AI Research, USA {angelafan, jacobkahn}@fb.com 3Stanford University, USA {simran, chrismre}@cs.stanford.edu 4Cohere, USA Patrick@cohere.ai Abstract Users an organizations are generating ever- increasing amounts of private data from a wide range of sources. Incorporating private con- text is important to personalize open-domain