Let’s Play Mono-Poly: BERT Can Reveal Words’ Polysemy Level
and Partitionability into Senses

Aina Gar´ı Soler
Universit´e Paris-Saclay
CNRS, LISN
91400, Orsay, France
aina.gari@limsi.fr

Marianna Apidianaki
Department of Digital Humanities
University of Helsinki
Helsinki, Finlande
marianna.apidianaki@helsinki.fi

Abstrait

Pre-trained language models (LMs) encode
rich information about
linguistic structure
but their knowledge about lexical polysemy
remains unclear. We propose a novel exper-
imental setup for analyzing this knowledge
in LMs specifically trained for different lan-
guages (English, French, Spanish, and Greek)
and in multilingual BERT. We perform
our analysis on datasets carefully designed
to reflect different sense distributions, et
control for parameters that are highly cor-
related with polysemy such as frequency and
grammatical category. We demonstrate that
BERT-derived representations reflect words’
polysemy level and their partitionability into
senses. Polysemy-related information is more
clearly present in English BERT embeddings,
but models in other languages also man-
age to establish relevant distinctions between
words at different polysemy levels. Our results
contribute to a better understanding of the
knowledge encoded in contextualized rep-
resentations and open up new avenues for
multilingual lexical semantics research.

1

Introduction

Pre-trained contextual language models have ad-
vanced the state of the art in numerous natural
language understanding tasks (Devlin et al., 2019;
Peters et al., 2018). Their success has motivated
a large number of studies exploring what these
models actually learn about language (Voita et al.,
2019un; Clark et al., 2019; Voita et al., 2019b;
Tenney et al., 2019). The bulk of this interpreta-
tion work relies on probing tasks that serve to
predict linguistic properties from the represen-
tations generated by the models (Linzen, 2018;
Rogers et al., 2020). The focus was initially put

825

on linguistic aspects pertaining to grammar and
syntax (Linzen et al., 2016; Hewitt and Manning,
2019; Hewitt and Liang, 2019). The first prob-
ing tasks addressing semantic knowledge explored
phenomena in the syntax-semantics interface, tel
as semantic role labeling and coreference (Tenney
et coll., 2019; Kovaleva et al., 2019), and the sym-
bolic reasoning potential of LM representations
(Talmor et al., 2020).

Lexical meaning was largely overlooked in
early interpretation work, but is now attracting
increasing attention. Pre-trained LMs have been
shown to successfully leverage sense annotated
data for disambiguation (Wiedemann et al., 2019;
Reif et al., 2019). The interplay between word
type and token-level information in the hidden
representations of LSTM LMs has also been
explored (Aina et al., 2019), as well as the
similarity estimates that can be drawn from
contextualized representations without directly
addressing word meaning (Ethayarajh, 2019). Dans
recent work, Vuli´c et al. (2020) probe BERT
representations for lexical semantics, addressing
out-of-context word similarity. Whether these
models encode knowledge about lexical polysemy
and sense distinctions is, cependant, still an open
question. Our work aims to fill this gap.

We propose methodology for exploring the
knowledge about word senses in contextualized
representations. Our approach follows a rigorous
experimental protocol proper to lexical semantic
analyse, which involves the use of datasets
carefully designed to reflect different sense
distributions. This allows us to investigate the
knowledge models acquire during training, et
the influence of context variation on token
representations. We account
the strong
correlation between word frequency and number
of senses (Zipf, 1945), and for the relationship
between grammatical category and polysemy,
by balancing the frequency and part of speech

pour

Transactions of the Association for Computational Linguistics, vol. 9, pp. 825–844, 2021. https://doi.org/10.1162/tacl a 00400
Action Editor: Walter Daelemans. Submission batch: 11/2020; Revision batch: 3/2021; Published 8/2021.
c(cid:2) 2021 Association for Computational Linguistics. Distributed under a CC-BY 4.0 Licence.

je

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

p

:
/
/

d
je
r
e
c
t
.

m

je
t
.

e
d
toi

/
t

un
c
je
/

je

un
r
t
je
c
e
–
p
d

F
/

d
o

je
/

.

1
0
1
1
6
2

/
t

je

un
c
_
un
_
0
0
4
0
0
1
9
5
5
2
0
4

/

/
t

je

un
c
_
un
_
0
0
4
0
0
p
d

.

F

b
oui
g
toi
e
s
t

t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Camacho-Collados, 2020); determine the needs
in terms of context size for disambiguation (par exemple.,
in queries, chatbots); help lexicographers define
the number of entries for a word to be present in
a resource, and plan the time and effort needed
in semantic annotation tasks (McCarthy et al.,
2016). It could also guide cross-lingual transfer,
serving to identify polysemous words for which
transfer might be harder. Enfin, analyzing words’
semantic space can be highly useful for the study
of lexical semantic change (Rosenfeld and Erk,
2018; Dubossarsky et al., 2019; Giulianelli et al.,
2020; Schlechtweg et al., 2020). We make our
code and datasets available to enable comparison
across studies and to promote further research in
these directions.1

2 Related Work

The knowledge pre-trained contextual LMs
encode about lexical semantics has only recently
started being explored. Works by Reif et al. (2019)
and Wiedemann et al. (2019) propose experiments
using representations built from Wikipedia and
the SemCor corpus (Miller et al., 1993), et
show that BERT can organize word usages in
the semantic space in a way that reflects the
meaning distinctions present in the data. C'est
also shown that BERT can perform well
dans
the word sense disambiguation (WSD) task by
leveraging the sense-related information available
in these resources. These works address the
disambiguation capabilities of the model but do
not show what BERT actually knows about words’
polysemy, which is the main axis of our work.
In our experiments, sense annotations are not
used to guide the models into establishing sense
distinctions, but rather for creating controlled
conditions that allow us to analyze BERT’s
inherent knowledge of lexical polysemy.

Probing has also been proposed for lexical
semantics analysis, but addressing different
questions than the ones posed in our work. Aina
et coll., (2019) probe the hidden representations of
a bidirectional (bi-LSTM) LM for lexical (type-
level) and contextual (token-level) information.
They specifically train diagnostic classifiers on
the tasks of retrieving the input embedding of
a word and a representation of its contextual

1Our code and data are available at https://github

.com/ainagari/monopoly.

(poly) words

Chiffre 1: BERT distinguishes monosemous (mono)
from polysemous
layers.
Representations for a poly word are obtained from
sentences reflecting up to ten different senses (poly-
the same sense (poly-same), or natural
bal),
occurrence in a corpus (poly-rand).

in all

(PoS) distributions in our datasets and applying a
frequency-based model to polysemy prediction.

Surtout, notre

investigation encompasses
monolingual models in different languages (En-
glish, French, Spanish, and Greek) and multi-
lingual BERT (mBERT). We demonstrate that
BERT contextualized representations encode an
impressive amount of knowledge about poly-
semy, and are able to distinguish monosemous
(mono) from polysemous (poly) words in a vari-
ety of settings and configurations (cf. Chiffre 1).
Surtout, we demonstrate that representations
derived from contextual LMs encode knowl-
edge about words’ polysemy acquired through
pre-training which is combined with informa-
tion from new contexts of use (Sections 3–6).
En plus, we show that BERT representa-
tions can serve to determine how easy it is to
partition a word’s semantic space into senses
(Section 7).

Our methodology can serve for the analy-
sis of words and datasets from different topics,
domains and languages. Knowledge about words’
polysemy and sense partitionability has numerous
practical
implications: It can guide decisions
towards a sense clustering or a per-instance
approach in applications (Reisinger and Mooney,
2010; Neelakantan et al., 2014; Camacho-
Collados and Pilehvar, 2018); indiquer
to words
with stable semantics which can be safe cues
for disambiguation in running text (Leacock et al.,
1998; Agirre and Martinez, 2004; Loureiro and

826

je

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

p

:
/
/

d
je
r
e
c
t
.

m

je
t
.

e
d
toi

/
t

un
c
je
/

je

un
r
t
je
c
e
–
p
d

F
/

d
o

je
/

.

1
0
1
1
6
2

/
t

je

un
c
_
un
_
0
0
4
0
0
1
9
5
5
2
0
4

/

/
t

je

un
c
_
un
_
0
0
4
0
0
p
d

.

F

b
oui
g
toi
e
s
t

t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

it

is given a central

meaning (as reflected in its lexical substitutes).
The results show that the information about the
input word that is present in LSTM representations
is not
lost after contextualization; cependant,
the quality of the information available for a
word is assessed through the model’s ability
to identify the corresponding embedding, as in
Adi et al. (2017) and Conneau et al. (2018).
Aussi, lexical ambiguity is only viewed through
In our work,
the lens of contextualization.
on the contrary,
role:
We explicitly address the knowledge BERT
encodes about words’ degree of polysemy and
partitionability into senses. Vuli´c et al. (2020) aussi
propose to probe contextualized models for lexical
semantics, but they do so using ‘‘static’’ word
embeddings obtained through pooling over several
contexts, or extracting representations for words
in isolation and from BERT’s embedding layer,
before contextualization. These representations
are evaluated on tasks traditionally used for
assessing the quality of static embeddings, tel que
out-of-context similarity and word analogy, lequel
are not tailored for addressing lexical polysemy.
Other contemporaneous work explores lexical
polysemy in static embeddings
(Jakubowski
et coll., 2020), and the relation of ambiguity
and context uncertainty as approximated in the
space constructed by mBERT using information-
theoretic measures (Pimentel et al., 2020). Enfin,
work by Ethayarajh (2019) provides useful
observations regarding the impact of context on
the representations, without explicitly addressing
the semantic knowledge encoded by the models.
Through an exploration of BERT, ELMo, et
GPT-2 (Radford et al., 2019), the author highlights
the highly distorted similarity of the obtained
contextualized representations which is due to
the vector space built by
the anisotropy of
each model.2 The question of meaning is not
addressed in this work, making it hard to draw any
conclusions about lexical polysemy.

Our proposed experimental setup is aimed at
investigating the polysemy information encoded
in the representations built at different layers of
deep pre-trained LMs. Our approach basically
relies on the similarity of contextualized repre-
sentations, which amounts to word usage similar-

ville (Usim) estimation, a classical task in lexical
semantics (Erk et al., 2009; Huang et al., 2012;
Erk et al., 2013). The Usim task precisely involves
predicting the similarity of word instances in con-
text without use of sense annotations. BERT has
been shown to be particularly good at this task
(Gar´ı Soler et al., 2019; Pilehvar and Camacho-
Collados, 2019). Our experiments allow us to
explore and understand what this ability is due to.

3 Lexical Polysemy Detection

3.1 Dataset Creation

is important

We build our English dataset using SemCor
3.0 (Miller et al., 1993), a corpus manually
annotated with WordNet senses (Fellbaum, 1998).
to note that we do not use
Il
the annotations for training or evaluating any
of the models. These only serve to control the
composition of the sentence pools that are used
for generating contextualized representations, et
to analyze the results. We form sentence pools
for monosemous (mono) and polysemous (poly)
words that occur at least ten times in SemCor.3
For each mono word, we randomly sample ten of
its instances in the corpus. For each poly word,
we form three sentence pools of size ten reflecting
different sense distributions:

• Balanced (poly-bal). We sample a
sentence for each sense of
the word in
SemCor until a pool of ten sentences is
formed.

• Random (poly-rand). We randomly
sample ten poly word instances from
SemCor. We expect this pool to be highly
biased towards a specific sense due to the
skewed frequency distribution of word senses
(Kilgarriff, 2004; McCarthy et al., 2004).
This configuration is closer to the expected
natural occurrence of senses in a corpus, it
thus serves to estimate the behaviour of the
models in a real-world setting.

• Same sense (poly-same). We sample ten
sentences illustrating only one sense of the
poly word. Although the composition of this
pool is similar to that of the mono pool (c'est à dire. tous
instances describe the same sense) we call it

2This issue affects all tested models and is particularly
present in the last layers of GPT-2, resulting in highly similar
representations even for random words.

3We find the number of senses for a word of a specific part
of speech (PoS) in WordNet 3.0, which we access through
the NLTK interface (Bird et al., 2009).

827

je

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

p

:
/
/

d
je
r
e
c
t
.

m

je
t
.

e
d
toi

/
t

un
c
je
/

je

un
r
t
je
c
e
–
p
d

F
/

d
o

je
/

.

1
0
1
1
6
2

/
t

je

un
c
_
un
_
0
0
4
0
0
1
9
5
5
2
0
4

/

/
t

je

un
c
_
un
_
0
0
4
0
0
p
d

.

F

b
oui
g
toi
e
s
t

t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Setting

Word

Sense

Sentences

mono

hotel.n

poly-same room.n

poly-bal

room.n

INN

INN

CHAMBER

CHAMBER

CHAMBER

The walk ended, inevitably, right in front of his hotel building.
Maybe he’s at the hotel.
The room vibrated as if a giant hand had rocked it.
(. . .) Tell her to come to Adam’s room (. . .)
(. . .) he left the room, walked down the hall (. . .)
It gives them room to play and plenty of fresh air.

SPACE
OPPORTUNITY Even here there is room for some variation, for metal surfaces vary (. . .)

Tableau 1: Example sentences for the monosemous noun hotel and the polysemous noun room.

poly-same because it describes one sense
of a polysemous word.4 Specifically, we want
to explore whether BERT representations
derived from these instances can serve to
distinguish mono from poly words.

The controlled composition of the poly sentence
pools allows us to investigate the behavior of
the models when they are exposed to instances
of polysemous words describing the same or
different senses. Il y a 1,765 poly words
in SemCor with at least 10 sentences available.5
We randomly subsample 418 from these in or-
der to balance the mono and poly classes. Notre
English dataset is composed of 836 mono and
poly words, and their instances in 8,195 unique
phrases. Tableau 1 shows a sample of the sentences
in different pools. For French, Spanish, and Greek,
we retrieve sentences from the Eurosense corpus
(Delli Bovi et al., 2017) which contains texts from
Europarl automatically annotated with BabelNet
word senses (Navigli and Ponzetto, 2012).6 Nous
extract sentences from the high-precision version7
of Eurosense, and create sentence pools in the
same way as in English, balancing the number of
monosemous and polysemous words (418). Nous
determine the number of senses for a word as the
number of its Babelnet senses that are mapped to
a WordNet sense.8

4The polysemous words are the same as in poly-bal

and poly-rand.

5We use sentences of up to 100 words.
6BabelNet is a multilingual semantic network built from
multiple lexicographic and encyclopedic resources, tel que
WordNet and Wikipedia.

7The high coverage version of Eurosense is larger than

the high-precision one, but disambiguation is less accurate.

8This filtering serves to exclude BabelNet senses that
correspond to named entities and are not useful for our
purposes (such as movie or album titles), and to run these
experiments under similar conditions across languages.

3.2 Contextualized Word Representations

We experiment with representations generated by
three English models: BERT (Devlin et al., 2019),9
ELMo (Peters et al., 2018), and context2vec
(Melamud et al., 2016). BERT is a Transformer
architecture (Vaswani et al., 2017) that is jointly
trained for a masked LM and a next sentence
prediction task. Masked LM inolves a Cloze-style
task, where the model needs to guess randomly
masked words by jointly conditioning on their
left and right context. We use the bert-base-
uncased and bert-base-cased models,
pre-trained on the BooksCorpus (Zhu et al., 2015)
and English Wikipedia. ELMo is a bi-LSTM LM
trained on Wikipedia and news crawl data from
WMT 2008-2012. We use 1024-d representations
from the 5.5B model.10 Context2vec is a neural
model based on word2vec’s CBOW architecture
(Mikolov et al., 2013) which learns embeddings
of wide sentential contexts using a bi-LSTM. Le
model produces representations for words and
their context. We use the context representations
from a 600-d context2vec model pre-trained on
the ukWaC corpus (Baroni et al., 2009).11

For French, Spanish, and Greek, we use BERT
models specifically trained for each language:
Flaubert (flaubert base uncased) (Le et al.,
2020), BETO (bert-base-spanish-wwm-
uncased) (Cañete et al., 2020), and Greek
BERT (bert-base-greek-uncased-v1)
(Koutsikakis et al., 2020). We also use the bert-
base-multilingual-cased model for each
of the four languages. mBERT was trained on

9We use Huggingface transformers (Wolf et al.,

2020).

10https://allennlp.org/elmo.
11https://github.com/orenmel/context2vec.

828

je

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

p

:
/
/

d
je
r
e
c
t
.

m

je
t
.

e
d
toi

/
t

un
c
je
/

je

un
r
t
je
c
e
–
p
d

F
/

d
o

je
/

.

1
0
1
1
6
2

/
t

je

un
c
_
un
_
0
0
4
0
0
1
9
5
5
2
0
4

/

/
t

je

un
c
_
un
_
0
0
4
0
0
p
d

.

F

b
oui
g
toi
e
s
t

t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Wikipedia data of 104 languages.12 All BERT
models generate 768-d representations.

presented in the next section provide additional
evidence in this respect.

3.3 The Self-Similarity Metric

All models produce representations that describe
word meaning in specific contexts of use. For each
instance i of a target word w in a sentence, nous
extract its representation from: (je) each of the 12
layers of a BERT model;13 (ii) each of the three
ELMo layers; et (iii) context2vec. We calculate
self-similarity (Self Sim) (Ethayarajh, 2019) pour
w in a sentence pool p and a layer l, by taking the
average of the pairwise cosine similarities of the
representations of its instances in l:
(cid:2)

(cid:2)

SelfSiml(w) =

1
|je|2 − |je|

cos(xwli, xwlj)

i∈I

j∈I
j(cid:4)=i

(1)
In formula 1, |je| is the number of instances for
w (ten in our experiments); xwli and xwlj are the
representations for instances i and j of w in layer
je. The Self Sim score is in the range [−1, 1]. Nous
report the average Self Sim for all w’s in a pool p.
We expect it to be higher for monosemous words
and words with low polysemy than for highly
polysemous words. We also expect the poly-
same pool to have a higher average Self Sim
score than the other poly pools which contain
instances of different senses.

Contextualization has a strong impact on
Self Sim since it
introduces variation in the
token-level representations, making them more
dissimilar. The Self Sim value for a word
would be 1 with non-contextualized (or static)
embeddings, as all its instances would be assigned
the same vector. In contextual models, Self Sim
is lower in layers where the impact of the context
is stronger (Ethayarajh, 2019). C'est, cependant,
important to note that contextualization in BERT
models is not monotonic, as shown by previous
studies of the models’ internal workings (Voita
et coll., 2019un; Ethayarajh, 2019). Our experiments

12The mBERT model developers recommend using the
cased version of the model rather than the uncased one,
especially for languages with non-Latin alphabets, because
it fixes normalization issues. More details about this model
can be found here: https://github.com/google
-research/bert/blob/master/multilingual
.md.

13We also tried different combinations of the last four lay-
ers, but this did not improve the results. When a word is split
into multiple wordpieces (WPs), we obtain its representation
by averaging the WPs.

829

3.4 Results and Discussion

3.4.1 Mono-Poly in English

Chiffre 2 shows the average Self Sim value
obtained for each sentence pool with representa-
tions produced by BERT models. The thin lines in
the first plot illustrate the average Self Sim score
calculated for mono and poly words using repre-
sentations from each layer of the uncased English
BERT model. We observe a clear distinction of
words according to their polysemy: Self Sim is
higher for mono than for poly words across
all layers and sentence pools. BERT establishes
a clear distinction even between the mono and
poly-same pools, which contain instances of
only one sense. This distinction is important; it
suggests that BERT encodes information about
a word’s monosemous or polysemous nature
regardless of the sentences that are used to derive
the contextualized representations. Spécifiquement,
BERT produces less similar representations for
word instances in the poly-same pool com-
pared to mono, reflecting that poly words can
have different meanings.

We also observe a clear ordering of the three
poly sentence pools: Average Self Sim is higher
in poly-same, which only contains instances
of one sense, followed by mid-range values in
poly-rand, and gets its lowest values in the
balanced setting (poly-bal). This is noteworthy
given that poly-rand contains a mix of senses
but with a stronger representation of w’s most
frequent sense than poly-bal (71% vs. 47%).14
Our results demonstrate that BERT represen-
tations encode two types of lexical semantic
the polysemous
connaissance:
nature of words acquired through pre-training
(as reflected in the distinction between mono
and poly-same), and information from the par-
ticular instances of a word used to create the
contextualized representations (as shown by the
finer-grained distinctions between different poly
settings). BERT’s knowledge about polysemy can
be due to differences in the types of context where
words of different polysemy levels occur. Nous
expect poly words to be seen in more varied
contexts than mono words, reflecting their differ-
ent senses. BERT encodes this variation with the

information about

14Numbers are macro-averages for words in the pools.

je

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

p

:
/
/

d
je
r
e
c
t
.

m

je
t
.

e
d
toi

/
t

un
c
je
/

je

un
r
t
je
c
e
–
p
d

F
/

d
o

je
/

.

1
0
1
1
6
2

/
t

je

un
c
_
un
_
0
0
4
0
0
1
9
5
5
2
0
4

/

/
t

je

un
c
_
un
_
0
0
4
0
0
p
d

.

F

b
oui
g
toi
e
s
t

t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

je

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

p

:
/
/

d
je
r
e
c
t
.

m

je
t
.

e
d
toi

/
t

un
c
je
/

je

un
r
t
je
c
e
–
p
d

F
/

d
o

je
/

.

1
0
1
1
6
2

/
t

je

un
c
_
un
_
0
0
4
0
0
1
9
5
5
2
0
4

/

/
t

je

un
c
_
un
_
0
0
4
0
0
p
d

.

F

b
oui
g
toi
e
s
t

t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Chiffre 2: Average Self Sim obtained with monolingual BERT models (top row) and mBERT (bottom row) across
all layers (horizontal axis). In the first plot, thick lines correspond to the cased model.

LM objective through exposure to large amounts
of data, and this is reflected in the represen-
tations. The same ordering pattern is observed
with mBERT (lower part of Figure 2) and with
ELMo (Chiffre 3(un)). With context2vec, average
Self Sim in mono is 0.40, 0.38 in poly-same,
0.37 in poly-rand, et 0.35 in poly-bal.
This suggests that these models also have some
inherent knowledge about lexical polysemy, mais
differences are less clearly marked than in BERT.
Using the cased model leads to an overall
increase in Self Sim and to smaller differences
between bands, as shown by the thick lines in the
first plot of Figure 2. Our explanation for the lower
distinction ability of the bert-base-cased
model is that it encodes sparser information about
words than the uncased model. It was trained
on a more diverse set of strings, so many WPs
are present in both their capitalized and non-
capitalized form in the vocabulary. In spite of
que, it has a smaller vocabulary size (29K WPs)
than the uncased model (30.5K). Aussi, a higher
number of WPs correspond to word parts than in
the uncased model (6,478 vs 5,829).

le

We test

t-tests when

the statistical significance of

le
mono/poly-rand distinction using unpaired
two-samples
normality
assumption is met (as determined with Shapiro
Wilk’s tests). Otherwise, we run a Mann Whitney
U test, the non-parametrical alternative of this
t-test. In order to lower the probability of type
increases when
I errors (false positives) que
performing multiple tests, we correct p-values
using the Benjamini–Hochberg False Discovery

Chiffre 3: Comparison of average Self Sim obtained
for mono and poly words using ELMo representations
(un), and for words in different polysemy bands in the
poly-rand sentence pool (b).

Rate (FDR) adjustment (Benjamini and Hochberg,
1995). Our results show that differences are
significant across all embedding types and layers
(α = 0.01).

The decreasing trend in Self Sim observed for
BERT in Figure 2, and the peak in layer 11,
confirm the phases of context encoding and token
reconstruction observed by Voita et al. (2019un).15
In earlier layers, context variation makes re-
presentations more dissimilar and Self Sim
decreases. In the last layers, information about the
input token is recovered for LM prediction and
similarity scores are boosted. Our results show
clear distinctions across all BERT and ELMo
layers. This suggests that lexical information is
spread throughout the layers of the models, et
contributes new evidence to the discussion on the
localization of semantic information inside the
models (Rogers et al., 2020; Vuli´c et al., 2020).

15They study the information flow in the Transformer
estimating the MI between representations at different layers.

830

je

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

p

:
/
/

d
je
r
e
c
t
.

m

je
t
.

e
d
toi

/
t

un
c
je
/

je

un
r
t
je
c
e
–
p
d

F
/

d
o

je
/

.

1
0
1
1
6
2

/
t

je

un
c
_
un
_
0
0
4
0
0
1
9
5
5
2
0
4

/

/
t

je

un
c
_
un
_
0
0
4
0
0
p
d

.

F

b
oui
g
toi
e
s
t

t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Chiffre 4: Average Self Sim obtained with monolingual BERT models (top row) and mBERT (bottom row)
for mono and poly words in different polysemy bands. Representations are derived from sentences in the
poly-rand pool.

3.4.2 Mono-Poly in Other Languages
The top row of Figure 2 shows the average
Self Sim obtained for French, Spanish, et
Greek words using monolingual models. Flaubert,
BETO, and Greek BERT representations clearly
distinguish mono and poly words, but average
Self Sim values for different poly pools are
much closer than in English. BETO seems to
capture these fine-grained distinctions slightly
better than the French and Greek models. Le
second row of the figure shows results obtained
with mBERT representations. We observe the
highly similar average Self Sim values assigned
to different poly pools, which show that
distinction is harder than in monolingual models.
Statistical tests show that the difference between
Self Sim values in mono and poly-rand is
significant in all layers of BETO, Flaubert, Greek
BERT, and mBERT for Spanish and French.16
The magnitude of the difference in Greek BERT
est, cependant, smaller compared to the other models
(0.03 vs. 0.09 in BETO at the layers with the
biggest difference in average Self Sim).

4 Polysemy Level Prediction

4.1 SelfSim-based Ranking

In this set of experiments, we explore the impact of
words’ degree of polysemy on the representations.
We control for this factor by grouping words into
three polysemy bands as in McCarthy et al. (2016),

Chiffre 5: Comparison of BERT average Self Sim for
mono and poly words in different polysemy bands in
the poly-bal and poly-same sentence pools.

which correspond to a specific number of senses
(k): faible: 2 ≤ k ≤ 3, mid: 4 ≤ k ≤ 6, haut:
k > 6. For English, the three bands are popu-
lated with a different number of words: faible: 551,
mid: 663, haut: 551. In the other languages,
we form bands containing 300 words each.17 In
Chiffre 4, we compare mono words with words
in each polysemy band in terms of their average
Self Sim. Values for mono words are taken from
Section 3. For poly words, we use representations
from the poly-rand sentence pool, which better
approximates natural occurrence in a corpus. Pour
comparison, we report in Figure 5 results obtained
in English using sentences from the poly-same
and poly-bal pools.18

In English, the pattern is clear in all plots:
Self Sim is higher for mono than for poly
words in any band, confirming that BERT is

17We only used 418 of these poly words in Section 3 dans

order to have balanced mono and poly pools.

16In mBERT for Greek, the difference is significant in ten

18We omit the plots for poly-bal and poly-same for

layers.

the other models due to space constraints.

831

able to distinguish mono from poly words at
different polysemy levels. The range of Self Sim
values for a band is inversely proportional to its
k: Words in low get higher values than words
in high. The results denote that the meaning
of highly polysemous words is more variable
(lower Self Sim) than the meaning of words with
fewer senses. As expected, scores are higher and
inter-band similarities are closer in poly-same
(cf. Chiffre 5(b)) compared with poly-rand
and poly-bal, where distinctions are clearer.
The observed differences confirm that BERT can
predict the polysemy level of words, even from
instances describing the same sense.

We observe similar patterns with ELMo (cf.
Chiffre 3(b)) and context2vec representations
in poly-rand,19 but smaller absolute inter-
band differences. In poly-same, both models
fail to correctly order the bands. Dans l'ensemble, notre
that BERT encodes higher
results highlight
quality knowledge about polysemy. We test the
significance of the inter-band differences detected
in poly-rand using the same approach as in
Section 3.4.1. These are significant in all but a
few20 layers of the models.

The bands are also correctly ranked in the
other three languages but with smaller inter-band
differences than in English, especially in Greek
where clear distinctions are only made in a few
middle layers. This variation across languages can
be explained to some extent by the quality of
the automatic EuroSense annotations, which has
a direct impact on the quality of the sentence
pools. Results of a manual evaluation conducted
by Delli Bovi et al. (2017) showed that WSD
precision is ten points higher in English (81.5)
and Spanish (82.5) than in French (71.8). Le
Greek portion, cependant, has not been evaluated.

Plots in the second row of Figure 4 show
results obtained using mBERT. Similarly to
the previous experiment (Section 3.4), mBERT
overall makes less clear distinctions than the
monolingual models. The low and mid bands
often get similar Self Sim values, which are
close to mono in French and Greek. Toujours, inter-
band differences are significant in most layers of

Chiffre 6: The left plots show the similarity between
random words in models for each language. Plots on the
right side show the difference between the similarity of
random words and Self Sim in poly-rand.

mBERT and the monolingual French, Spanish,
and Greek models.21

4.2 Anisotropy Analysis

In order to better understand the reasons behind
the smaller inter-band differences observed with
mBERT, we conduct an additional analysis of
the models’ anisotropy. We create 2,183 random
word pairs from the English mono, faible, mid
and high bands, et 1,318 pairs in each of
the other languages.22 We calculate the cosine
similarity between two random instances of the
words in each pair and take the average over all
pairs (RandSim). The plots in the left column
of Figure 6 show the results. We observe a clear
difference in the scores obtained by monolingual
models (solid lines) and mBERT (dashed lines).
Clairement, mBERT assigns higher similarities to

19Average Self Sim values for context2vec in the

poly-rand setting: faible: 0.37, mid: 0.36, haut: 0.36.

20low→ mid in ELMo’s third layer, and mid→high in

21With the exception of mono→ low in mBERT for
Greek, and low→mid in Flaubert and in mBERT for French.
221,318 is the total number of words across bands in

context2vec and in BERT’s first layer.

French, Spanish, and Greek.

832

je

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

p

:
/
/

d
je
r
e
c
t
.

m

je
t
.

e
d
toi

/
t

un
c
je
/

je

un
r
t
je
c
e
–
p
d

F
/

d
o

je
/

.

1
0
1
1
6
2

/
t

je

un
c
_
un
_
0
0
4
0
0
1
9
5
5
2
0
4

/

/
t

je

un
c
_
un
_
0
0
4
0
0
p
d

.

F

b
oui
g
toi
e
s
t

t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

random words, an indication that its semantic
space is more anisotropic than the one built by
monolingual models. High anisotropy means that
representations occupy a narrow cone in the vector
espace, which results in lower quality similarity
estimates and in the model’s limited potential to
establish clear semantic distinctions.

We also compare RandSim to the average
Self Sim obtained for poly words in the poly-
rand sentence pool (cf. Section 3.1). In a quality
semantic space, we would expect Self Sim
(between same word instances) to be much higher
than RandSim. The right column of Figure 6
shows the difference between these two scores.
diff in a layer l is calculated as in Equation (2):

diffl = AvgSelfSiml(poly-rand)−RandSiml
(2)
We observe that the difference is smaller in the
space built by mBERT, which is more anisotropic
than the space built by monolingual models.
This is particularly obvious in the upper layers
of the model. This result confirms the lower
quality of mBERT’s semantic space compared
to monolingual models.

Enfin, we believe that another factor behind
the worse mBERT performance is that
le
multilingual WP vocabulary is mostly English-
driven,
resulting in arbitrary partitionings of
words in the other languages. This word splitting
procedure must have an impact on the quality of
the lexical information in mBERT representations.

5 Analysis by Frequency and PoS

Given the strong correlation between word
frequency and number of senses (Zipf, 1945),
we explore the impact of frequency on BERT
representations. Our goal is to determine the extent
to which it influences the good mono/poly
detection results obtained in Sections 3.4 et 4.1.

5.1 Dataset Composition

We perform this analysis in English using
frequency information from Google Ngrams
(Brants and Franz, 2006). For French, Spanish,
and Greek, we use frequency counts gathered
from the OSCAR corpus (Su´arez et al., 2019). Nous
split the words into four ranges (F ) corresponding
to the quartiles of frequencies in each dataset.
Each range f in F contains the same number of
words. We provide detailed information about the

je

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

p

:
/
/

d
je
r
e
c
t
.

m

je
t
.

e
d
toi

/
t

un
c
je
/

je

un
r
t
je
c
e
–
p
d

F
/

d
o

je
/

.

1
0
1
1
6
2

/
t

je

un
c
_
un
_
0
0
4
0
0
1
9
5
5
2
0
4

/

/
t

je

un
c
_
un
_
0
0
4
0
0
p
d

.

F

b
oui
g
toi
e
s
t

t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Chiffre 7: Composition of the English word bands in
terms of frequency (un) and grammatical category (b).

composition of the English dataset in Figure 7.23
Chiffre 7(un) shows that mono words are much less
frequent than poly words. Chiffre 7(b) shows the
distribution of different PoS categories in each
band. Nouns are the prevalent category in all
bands and verbs are less present among mono
words (10.8%), as expected. Enfin, adverbs are
hardly represented in the high polysemy band
(1.2% of all words).

5.2 Self-Sim by Frequency Range and PoS

We examine the average BERT Self Sim per
frequency range in poly-rand. Due to space
constraints, we only report detailed results for the
English BERT model in Figure 8 (plot (un)). Le
clear ordering by range suggests that BERT can
successfully distinguish words by their frequency,
especially in the last layers. Plot (b) in Figure 8
shows the average Self Sim for words of each
PoS category. Verbs have the lowest Self Sim
which is not surprising given that they are highly
polysemous (as shown in Figure 7(b)). We observe
the same trend for monolingual models in the other
three languages.

23The composition of each band is the same as in Sections 3

et 4.

833

Chiffre 8: Average Self Sim for English words of
different frequencies and part of speech categories
with BERT representations.

5.3 Controlling for Frequency and PoS

We conduct an additional experiment where we
control for the composition of the poly bands
in terms of grammatical category and word
frequency. We call these two settings POS-bal and
FREQ-bal. We define npos, the smallest number of
words of a specific PoS that can be found in a band.
We form the POS-bal bands by subsampling from
each band the same number of words (npos) of that
PoS. Par exemple, all POS-bal bands have nn nouns
and nv verbs. We follow a similar procedure to
balance the bands by frequency in the FREQ-bal
setting. Dans ce cas, nf is the minimum number
of words of a specific frequency range f that can
be found in a band. We form the FREQ-bal dataset
by subsampling from each band the same number
of words (nf ) of a given range f in F .

Tableau 2 shows the distribution of words per
PoS and frequency range in the POS-bal and FREQ-
bal settings for each language. The table reads as
follows: The English POS-bal bands contain 198
nouns, 45 verbs, 64 adjectives, et 7 adverbs;
similarly for the other two languages. Greek is
not included in this POS-based analysis because
all sense-annotated Greek words in EuroSense are
nouns. In FREQ-bal, each English band contains 40
words that occur less than 7.1M times in Google
Ngrams, and so on and so forth.

We examine the average Self Sim values
obtained for words in each band in poly-
rand. Chiffre 9 shows the results for monolingual
models. We observe that the mono and poly
words in the POS-bal and FREQ-bal bands are ranked
similarly to Figure 4. This shows that BERT’s
polysemy predictions do not rely on frequency
or part of speech. The only exception is Greek
BERT, which cannot establish correct inter-band
distinctions when the influence of frequency is
neutralized in the FREQ-bal setting. A general
observation that applies to all models is that

POS-bal

Nouns Verbs Adjectives Adverbs
198
171
167

64
29
40

45
32
22

7
9
0

FREQ-bal

99
70m
43

7.1M 20M
40
23m
17
64m 233m
12
14m
13

39
40m
41

49M.
62
210m
67
793m
58
111m
70

682M.
39
41M.
38
59M.
48
1.9M.
42

dans
fr
es

dans

fr

es

el

Tableau 2: Content of the polysemy bands in the
POS-bal and FREQ-bal settings. All bands for a
language contain the same number of words of a
specific grammatical category or frequency range.
M stands for a million and m for a thousand
occurrences of a word in a corpus.

although inter-band distinctions become less clear,
the ordering of the bands is preserved. We observe
the same trend with ELMo and context2vec.

tests

Statistical

show that all

inter-band
distinctions established by English BERT are still
significant in most layers of the model.24 This is
not the case for ELMo and context2vec, which can
distinguish between mono and poly words but
fail to establish significant distinctions between
polysemy bands in the balanced settings. Pour
French and Spanish, the statistical analysis shows
that all distinctions in POS-bal are significant in at
least one layer of the models. The same applies
to the mono→poly distinction in FREQ-bal but
finer-grained distinctions disappear.25

6 Classification by Polysemy Level

Our finding that word instance similarity differs
across polysemy bands suggests that this feature
can be useful for classification. Dans cette section,
we probe the representations for polysemy using
a classification experiment where we test their
ability to guess whether a word is polysemous,
and which poly band it
falls in. We use

24Note that the sample size in this analysis is smaller

compared to that used in Sections 3.4 et 4.1.

25With a few exceptions: mono→low and mid→ high

are significant in all BETO layers.

834

je

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

p

:
/
/

d
je
r
e
c
t
.

m

je
t
.

e
d
toi

/
t

un
c
je
/

je

un
r
t
je
c
e
–
p
d

F
/

d
o

je
/

.

1
0
1
1
6
2

/
t

je

un
c
_
un
_
0
0
4
0
0
1
9
5
5
2
0
4

/

/
t

je

un
c
_
un
_
0
0
4
0
0
p
d

.

F

b
oui
g
toi
e
s
t

t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

je

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

p

:
/
/

d
je
r
e
c
t
.

m

je
t
.

e
d
toi

/
t

un
c
je
/

je

un
r
t
je
c
e
–
p
d

F
/

d
o

je
/

.

1
0
1
1
6
2

/
t

je

un
c
_
un
_
0
0
4
0
0
1
9
5
5
2
0
4

/

/
t

je

un
c
_
un
_
0
0
4
0
0
p
d

.

F

b
oui
g
toi
e
s
t

t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Chiffre 9: Average Self Sim inside the poly bands balanced for frequency (FREQ-bal) and part of speech (POS-bal).
Self Sim is calculated using representations generated by monolingual BERT models from sentences in each
language-specific pool. We do not balance the Greek dataset for PoS because it only contains nouns.

the poly-rand sentence pools and a standard
train/dev/test split (70%/15%/15%) of the data.
For the mono/poly distinction (c'est à dire., the data
used in Section 3), this results in 584/126/126
words per set in each language. To guarantee a
fair evaluation, we make sure there is no overlap
between the lemmas in the three sets. We use two
types of features: (je) the average Self Sim for
a word; et (ii) all pairwise cosine similarities
collected for its instances, which results in 45
features per word (pairCos). We train a binary
logistic regression classifier for each type of
representation and feature.

As explained in Section 4, the three poly
bands (faible, mid, and high) and mono contain
a different number of words. For classification
into polysemy bands, we balance each class by
randomly subsampling words from each band.
In total, we use 1,168 words for training, 252 pour
development, et 252 for testing (70%/15%/15%)
in English. In the other languages, we use a
split of 840/180/180 words. We train multi-class
logistic regression classifiers with the two types
of features, Self Sim and pairCos. We compare
the results of the classifiers to a baseline that
predicts always the same class, and to a frequency-
based classifier which only uses the words’ log
frequency in Google Ngrams, or in the OSCAR
corpus, as a feature.

Tableau 3 presents classification accuracy on the
test set. We report results obtained with the best
layer for each representation type and feature as

mono/poly

poly bands

Self Sim pairCos Self Sim pairCos

0.7610
0.778
0.692
0.61

0.587
0.669

Model
BERT
mBERT
ELMo
context2vec
Frequency
Flaubert
mBERT
Frequency
BETO
mBERT
Frequency
GreekBERT 0.704
0.607
mBERT
Frequency
Baseline

0.709
0.6911

N
E

R.
F

S
E

L
E

0.77

0.61

0.67

0.63
0.50

0.798
0.758
0.633
0.61

0.556
0.649

0.667
0.647

0.644
0.657

0.4910
0.4612
0.372
0.34

0.4610
0.4312
0.343
0.31

0.298
0.387

0.426
0.389

0.41

0.37

0.279
0.388

0.485
0.437

0.41

0.344
0.3211

0.386
0.349

0.35
0.25

Tableau 3: Accuracy of binary (mono/poly)
and multi-class (poly bands) classifiers using
Self Sim and pairCos features on the test sets.
Comparison to a baseline that predicts always the
same class and a classifier that only uses log fre-
quency as feature. Subscripts denote the layers
used.

determined on the development sets. In English,
best accuracy is obtained by BERT in both
le binaire (0.79) and multiclass settings (0.49),
followed by mBERT (0.77 et 0.46). Despite its
simplicity, the frequency-based classifier obtains
better results than context2vec and ELMo, et
performs on par with mBERT in the binary
setting. This shows that frequency information

835

is highly relevant for the mono-poly distinction.
All classifiers outperform the same class baseline.
These results are very encouraging, montrant que
BERT embeddings can be used to determine
whether a word has multiple meanings, et
provide a rough indication of its polysemy level.
Results in the other three languages are not as high
as those obtained in English, but most models
results than the frequency-based
give higher
classifier.26

instances, on the contrary, share a different number
of substitutes depending on their proximity. Le
need for manual annotations, cependant, constrains
the method’s applicability to specific datasets.

We propose to extend and scale up the
McCarthy et al. (2016) clusterability approach
using contextualized representations, pour
make it applicable to a larger vocabulary. These
experiments are carried out in English due to the
lack of evaluation data in other languages.

7 Word Sense Clusterability

7.2 Données

We have shown that representations from pre-
trained LMs encode rich information about
words’ degree of polysemy. They can successfully
distinguish mono from poly lemmas, and predict
the polysemy level of words. Our previous
experiments involved a set of controlled settings
representing different sense distributions and
polysemy levels. Dans cette section, we explore
whether these representations can also point to the
clusterability of poly words in an uncontrolled
setting.

7.1 Task Definition

Instances of some poly words are easier to group
into interpretable clusters than others. This is, pour
example, a simple task for the ambiguous noun
rock which can express two clearly separate senses
(STONE and MUSIC), but harder for book, lequel
might refer to the CONTENT or OBJECT senses of the
word (par exemple., I read a book vs. I bought a book). Dans
what follows, we test the ability of contextualized
representations to estimate how easy this task is
for a specific word, c'est, its partitionability into
senses.

Following McCarthy et al. (2016), we use
the clusterability metrics proposed by Ackerman
and Ben-David (2009) to measure the ease of
clustering word instances into senses. McCarthy
et autres. base their clustering on the similarity of
manual meaning-preserving annotations (lexical
substitutes and translations). Instances of different
senses, tel que: Put granola bars in a bowl
vs. That’s not a very high bar, present no
overlap in their in-context substitutes: {snack,
biscuit, block, slab} vs. {pole, marker, hurdle,
barrier, level, obstruction}. Semantically related

26Only exceptions are Greek mBERT in the multi-class

setting, and Flaubert in both settings.

We run our experiments on the usage similarity
(Usim) dataset (Erk et al., 2013) for comparison
with previous work. Usim contains ten instances
pour 56 target words of different PoS from the
SemEval Lexical Substitution dataset (McCarthy
and Navigli, 2007). Word instances are manually
annotated with pairwise similarity scores on a
scale from 1 (completely different) à 5 (same
meaning).

We represent target word instances in Usim
in two ways: using contextualized represen-
tations generated by BERT, context2vec, et
ELMo (BERT-REP, c2v-REP, ELMo-REP);27 en utilisant
substitute-based representations with automat-
ically generated substitutes. The substitute-based
approach allows for a direct comparison with the
method of McCarthy et al. (2016). They represent
each instance i of a word w in Usim as a vector(cid:3)je,
where each substitute s assigned to w over all its
instances (i ∈ I) becomes a dimension (ds). Pour
a given i, the value for each ds is the number of
annotators who proposed substitute s. ds contains
a zero entry if s was not proposed for i. Nous
refer to this type of representation as Gold-SUB.
We generate our substitute-based representations
with BERT using the simple ‘‘word similarity’’
approach in Zhou et al. (2019). For an instance i
of word w in context C, we rank a set of candidate
substitutes S = {s1, s2, . . . , sn} based on the
cosine similarity of the BERT representations for
i and for each substitute sj ∈ S in the same con-
text C. We use representations from the last layer
of the model. As candidate substitutes, we use
the unigram paraphrases of w in the Paraphrase

27We do not use the first layer of ELMo in this experiment.
It is character-based, so most representations of a lemma are
identical and we cannot obtain meaningful clusters.

836

je

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

p

:
/
/

d
je
r
e
c
t
.

m

je
t
.

e
d
toi

/
t

un
c
je
/

je

un
r
t
je
c
e
–
p
d

F
/

d
o

je
/

.

1
0
1
1
6
2

/
t

je

un
c
_
un
_
0
0
4
0
0
1
9
5
5
2
0
4

/

/
t

je

un
c
_
un
_
0
0
4
0
0
p
d

.

F

b
oui
g
toi
e
s
t

t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Database (PPDB) XXL package (Ganitkevitch
et coll., 2013; Pavlick et al., 2015).28

For each instance i of w, we obtain a ranking
R of all substitutes in S. We remove low-quality
substitutes (c'est à dire., noisy paraphrases or substitutes
referring to a different sense of w) using the
filtering approach proposed by Gar´ı Soler et al.
(2019). We check each pair of substitutes in
subsequent positions in R, starting from the top;
if a pair is unrelated in PPDB, all substitutes from
that position onwards are discarded. The idea is
that good quality substitutes should be both high-
ranked and semantically related. We build vectors
as in McCarthy et al. (2016), using the cosine
similarity assigned by BERT to each substitute as
a value. We call this representation BERT-SUB.

7.3 Sense Clustering

The clusterability metrics that we use are metrics
initially proposed for estimating the quality of the
optimal clustering that can be obtained from a
dataset; the better the quality of this clustering,
the higher the clusterability of the dataset it is
derived from (Ackerman and Ben-David, 2009).
In order to estimate the clusterability of a word
w, we thus need to first cluster its instances in
the data. We use the k-means algorithm which
requires the number of senses for a lemma.
This is, bien sûr, different for every lemma in
our dataset. We define the optimal number of
clusters k for a lemma in a data-driven manner
using the Silhouette coefficient (SIL) (Rousseeuw,
1987), without recourse to external resources.29
For a data point i, SIL compares the intra-cluster
the average distance from i to
distance (c'est à dire.,
every other data point in the same cluster) avec
the average distance of i to all points in its
nearest cluster. The SIL value for a clustering is
obtained by averaging SIL for all data points, et
it ranges from −1 to 1. We cluster each type of
representation for w using k-means with a range
of k values (2 ≤ k ≤ 10), and retain the k of the
clustering with the highest mean SIL. En plus,
since BERT representations’ cosine similarity
correlates well with usage similarity (Gar´ı Soler
et coll., 2019), we experiment with Agglomerative
Clustering with average linkage directly on the

28We use PPDB (http://www.paraphrase.org) à
reduce variability in our substitute sets, compared to the ones
that would be proposed by looking at the whole vocabulary.
29We do not use McCarthy et al.’s graph-based approach
because it is not compatible with all our representation types.

cosine distance matrix obtained with BERT
representations (BERT-AGG). For comparison,
we also use Agglomerative Clustering on the gold
usage similarity scores from Usim, transformed
into distances (Gold-AGG).

7.4 Clusterability Metrics

We use in our experiments the two best performing
metrics from McCarthy et al. (2016): Variance
Ratio (VR) (Zhang, 2001) and Separability (SEP)
(Ostrovsky et al., 2012). VR calculates the ratio
of the within- and between-cluster variance for
a given clustering solution. SEP measures the
difference in loss between two clusterings with
k − 1 and k clusters and its range is [0,1). We use
k-means’ sum of squared distances of data points
to their closest cluster center as the loss. Details
about these two metrics are given in Appendix A.30
We also experiment with SIL as a clusterability
metric, as it can assess cluster validity. For VR and
SIL, a higher value indicates higher clusterability.
The inverse applies to SEP.

We calculate Spearman’s ρ correlation between
the results of each clusterability metric and two
gold standard measures derived from Usim: Uiaa
and Umid. Uiaa is the inter-annotator agreement
for a lemma in terms of average pairwise
correlation between annotators’
Spearman’s
judgments. Higher Uiaa values indicate higher
clusterability, meaning that sense partitions are
clearer and easier to agree upon. Umid is the
proportion of mid-range judgments (entre 2
et 4) assigned by annotators to all instances of
a target word. It indicates how often usages do
not have identical (5) or completely different (1)
meaning. Donc, higher Umid values indicate
lower clusterability.

7.5 Results and Discussion

The clusterability results are given in Table 4.
Agglomerative Clustering on the gold Usim
similarity scores (Gold-AGG) gives best results
on the Uiaa evaluation in combination with
the SIL clusterability metric (ρ = 0.80). Ce
is unsurprising,
since Uiaa and Umid are
derived from the same Usim scores. Depuis
our automatically generated representations, le
strongest correlation with Uiaa (0.69) is obtained

30Note that the VR and SEP metrics are not compatible with
Gold-AGG which relies on Usim similarity scores, because
we need vectors for their calculation. For BERT-AGG, nous
calculate VR and SEP using BERT embeddings.

837

je

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

p

:
/
/

d
je
r
e
c
t
.

m

je
t
.

e
d
toi

/
t

un
c
je
/

je

un
r
t
je
c
e
–
p
d

F
/

d
o

je
/

.

1
0
1
1
6
2

/
t

je

un
c
_
un
_
0
0
4
0
0
1
9
5
5
2
0
4

/

/
t

je

un
c
_
un
_
0
0
4
0
0
p
d

.

F

b
oui
g
toi
e
s
t

t

o
n
0
7
S
e
p
e
m
b
e
r
2
0
2
3

Gold Metric

Uiaa

Umid

BERT-REP
SEP (cid:7) −0.48*10
VR (cid:8)
0.1712
SIL (cid:8)
0.61*11
SEP (cid:8)
0.43*9
VR (cid:7) −0.249
SIL (cid:7) −0.46*10

c2v-REP ELMo-REP BERT-SUB Gold-SUB BERT-AGG Gold-AGG
−0.12
0.14
0.06
−0.01
−0.08
0.05

−0.48*11
0.33*12
0.69*10
0.43*9
−0.32*5
−0.44*8

−0.20
0.34*
0.32*
0.16
−0.24
−0.38*

–
–
0.80*
–
–
−0.48*

−0.242
0.192
0.212
0.083
−0.153
−0.062

−0.03
0.09
0.10
0.05
−0.15
−0.11

Tableau 4: Spearman’s ρ correlation between automatic metrics and gold standard clusterability
estimates. Significant correlations (where the null hypothesis ρ = 0 is rejected with α < 0.05) are marked with *. The arrows indicate the expected direction of correlation for each metric. Subscripts indicate the layer that achieved best performance. The two strongest correlations obtained with each gold standard measure are in boldface. layer analysis of We present a per the correlations obtained with the best performing BERT representations (BERT-AGG) and the SIL the absolute metric in Figure 10. We report values of the correlation coefficient for a more straightforward comparison. For Uiaa, the higher layers of the model make the best predictions: Correlations increase monotonically up to layer 10, and then they show a slight decrease. Umid prediction shows a more irregular pattern: It peaks at layers 3 and 8, and decreases again in the last layers. l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u / t a c l / l a r t i c e - p d f / d o i / . 1 0 1 1 6 2 / t l a c _ a _ 0 0 4 0 0 1 9 5 5 2 0 4 / / t l a c _ a _ 0 0 4 0 0 p d . f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 Figure 10: Spearman’s ρ correlations between the gold standard Uiaa and Umid scores, and clusterability estimates obtained using Agglomerative Clustering on a cosine distance matrix of BERT representations. with BERT-AGG and the SIL clusterability metric. The SIL metric also works well with BERT-REP achieving the strongest correlation with Umid (−0.46). It constitutes, thus, a good alternative to the SEP and VR metrics used in previous studies when combined with BERT representations. Interestingly, the correlations obtained using raw BERT contextualized representations are much higher than the ones observed with McCarthy et al. (2016)’s representations that rely on manual substitutes (Gold-SUB). These were in the range of 0.20–0.34 for Uiaa and 0.16–0.38 for Umid (in absolute value). The results demonstrate that BERT representations offer good estimates of the partitionability of words into senses, improving over substitute annotations. As expected, the substitution-based approach performs better with clean manual substitutes (Gold-SUB) than with automatically generated ones (BERT-SUB). 8 Conclusion We have shown that contextualized BERT repre- sentations encode rich information about lexical polysemy. Our experimental results suggest that this high quality knowledge about words, which allows BERT to detect polysemy in different configurations and across all layers, is acquired during pre-training. Our findings hold for the English BERT as well as for BERT models in other languages, as shown by our experiments on French, Spanish, and Greek, and to a lesser extent for multilingual BERT. Moreover, English BERT representations can be used to obtain a good estimation of a word’s partitionability into senses. These results open up new avenues for research in multilingual semantic analysis, and we can con- sider various theoretical and application-related extensions for this work. The polysemy and sense-related knowledge revealed by the models can serve to develop novel methodologies for improved cross-lingual alignment of embedding spaces and cross-lingual transfer, pointing to more polysemous (or less clusterable) words for which transfer might be 838 harder. Predicting the polysemy level of words can also be useful for determining the context needed for acquiring representations that properly reflect the meaning of word instances in running text. From a more theoretical standpoint, we expect this work to be useful for studies on the organization of the semantic space in different languages and on lexical semantic change. Acknowledgments This work has been supported by the French Na- tional Research Agency under project ANR-16- CE33-0013. The work is also part of the FoTran project, funded by the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 771113). We thank the anonymous reviewers and the TACL Action Editor for their careful reading of our paper, their thorough reviews, and their helpful suggestions. References Margareta Ackerman and Shai Ben-David. 2009. Clusterability: A Theoretical Study. Journal of Machine Learning Research, 5:1–8. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2017. Fine- grained analysis of sentence embeddings using auxiliary prediction tasks. In Proceedings of ICLR. Toulon, France. Eneko Agirre and David Martinez. 2004. Unsupervised WSD based on automatically retrieved examples: The importance of bias. the 2004 Conference on In Proceedings of Empirical Methods in Natural Language Processing, pages 25–32, Barcelona, Spain. Association for Computational Linguistics. Laura Aina, Kristina Gulordava, and Gemma Boleda. 2019. Putting words in context: LSTM language models and lexical ambiguity. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3342–3348, Florence, Italy. Association for Computational Linguistics. Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009. The WaCky Wide Web: A collection of very large linguistically processed web-crawled corpora. Journal of Language Resources and Evaluation, 43(3):209–226. https://doi .org/10.1007/s10579-009-9081-4 Yoav Benjamini and Yosef Hochberg. 1995. Con- trolling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1):289–300. https:// doi.org/10.1111/j.2517-6161.1995 .tb02031.x Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python: Analyzing Text with the Natural Language Toolkit. O’Reilly Media, Inc., Beijing. Thorsten Brants and Alex Franz. 2006. Web 1T 5- gram Version 1. In LDC2006T13. Philadelphia, Pennsylvania. Linguistic Data Consortium. Jos´e Camacho-Collados and Mohammad Taher Pilehvar. 2018. From word to sense embed- dings: A survey on vector representations of meaning. Journal of Artificial Intelligence Re- search, 63:743–788. https://doi.org/10 .1613/jair.1.11259 José Cañete, Gabriel Chaperon, Rodrigo Fuentes, and Jorge Pérez. 2020. Spanish pre-trained BERT model and evaluation data. In Proceed- ings of the ICLR 2020 Workshop on Practi- cal ML for Developing Countries (PML4DC). Addis Ababa, Ethiopia. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? An analysis of BERT’s atten- tion. In Proceedings of the ACL 2019 Work- shop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Italy. Association for Computa- Florence, tional Linguistics. https://doi.org/10 .18653/v1/W19-4828 Alexis Conneau, German Kruszewski, Guillaume Lample, Lo¨ıc Barrault, and Marco Baroni. 2018. What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long 839 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u / t a c l / l a r t i c e - p d f / d o i / . 1 0 1 1 6 2 / t l a c _ a _ 0 0 4 0 0 1 9 5 5 2 0 4 / / t l a c _ a _ 0 0 4 0 0 p d . f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 pages Papers), 2126–2136, Melbourne, Australia. Association for Computational Lin- guistics. https://doi.org/10.18653 /v1/P18-1198 Claudio Delli Bovi, Jose Camacho-Collados, Alessandro Raganato, and Roberto Navigli. 2017. EuroSense: Automatic harvesting of multilingual sense annotations from paral- lel text. In Proceedings of the 55th Annual Meeting of the Association for Computa- tional Linguistics (Volume 2: Short Papers), pages 594–600, Vancouver, Canada. Associa- tion for Computational Linguistics. https:// doi.org/10.18653/v1/P17-2094 Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language the 2019 understanding. In Proceedings of Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. lexical Haim Dubossarsky, Simon Hengchen, Nina Tahmasebi, and Dominik Schlechtweg. 2019. Time-Out: Temporal referencing for robust In modeling of Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 457–470, Florence, Italy. Association for Computational Linguistics. https:// doi.org/10.18653/v1/P19-1044 semantic change. Katrin Erk, Diana McCarthy, and Nicholas Gaylord. 2009. Investigations on word senses and word usages. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 10–18, Suntec, Singapore. Association for Computational Linguistics. Katrin Erk, Diana McCarthy, and Nicholas Gaylord. 2013. Measuring word meaning in con- text. Computational Linguistics, 39(3):511–554. https://doi.org/10.1162/COLI a 00142 840 Kawin Ethayarajh. 2019. How contextual are contextualized word representations? Com- paring the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 55–65, Hong Kong, China. Associa- tion for Computational Linguistics. https:// doi.org/10.18653/v1/D19-1006 Christiane Fellbaum. 1998. WordNet: An Lexical Database. Language, Electronic Speech, and Communication. Cambridge, MA. MIT Press. https://doi.org/10 .7551/mitpress/7287.001.0001 Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The para phrase database. In Proceedings of the 2013 the North American Chap- Conference of ter of the Association for Computational Linguistics: Human Language Technologies, pages 758–764, Atlanta, Georgia. Association for Computational Linguistics. Aina Gar´ı Soler, Marianna Apidianaki, and Alexandre Allauzen. 2019. Word usage simi- larity estimation with sentence representations In Proceedings and automatic substitutes. the Eighth Joint Conference on Lexi- of (*SEM cal and Computational Semantics 2019), pages 9–21, Minneapolis, Minnesota. Association for Computational Linguistics. https://doi.org/10.18653/v1/S19 -1002 Mario Giulianelli, Marco Del Tredici, and Raquel Fern´andez. 2020. Analysing lexical semantic change with contextualized word representations. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 3960–3973, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020 .acl-main.365 John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Pro- ceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Nat- ural Language Processing (EMNLP-IJCNLP), l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u / t a c l / l a r t i c e - p d f / d o i / . 1 0 1 1 6 2 / t l a c _ a _ 0 0 4 0 0 1 9 5 5 2 0 4 / / t l a c _ a _ 0 0 4 0 0 p d . f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 pages 2733–2743, Hong Kong, China. Associa- tion for Computational Linguistics. https:// doi.org/10.18653/v1/D19-1275 John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis- tics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129–4138, Minneapolis, Minnesota. Association for Computational Linguistics. Eric Huang, Richard Socher, Christopher Manning, and Andrew Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 873–882, Jeju Island, Korea. Association for Computational Linguistics. Alexander Jakubowski, Milica Gasic, and Marcus Zibrowius. 2020. Topology of word embeddings: Singularities reflect polysemy. In Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics, pages 103–113, Barcelona, Spain (Online). Association for Computational Linguistics. Adam Kilgarriff. 2004. How dominant is the commonest sense of a word? In Lecture Notes in Computer Science (vol. 3206), Text, Speech and Dialogue, Sojka Petr, Kopeˇcek Ivan, Pala Karel (eds.), pages 103–112, Springer. Berlin, Heidelberg. https://doi.org/10.1007 /978-3-540-30120-2_14 John Koutsikakis, Ilias Chalkidis, Prodromos Malakasiotis, and Ion Androutsopoulos. 2020. GREEK-BERT: The Greeks visiting Sesame Street. In Proceedings of the 11th Hellenic Conference on Artificial Intelligence (SETN 2020), 110–117, Athens, Greece. https://doi.org/10.1145/3411408 .3411440 pages Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of BERT. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural 841 Processing (EMNLP-IJCNLP), Language 4365–4374, Hong Kong, China. pages Association for Computational Linguistics. https://doi.org/10.18653/v1/D19 -1445 Hang Le, Lo¨ıc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoˆıt Crabb´e, Laurent Besacier, and Didier Schwab. 2020. FlauBERT: Unsupervised language model Pre-training for In Proceedings of The 12th Lan- French. guage Resources and Evaluation Conference, pages 2479–2490, Marseille, France. European Language Resources Association. Claudia Leacock, Martin Chodorow, and George A. Miller. 1998. Using corpus statistics and WordNet relations for sense identification. Computational Linguistics, 24(1):147–165. Tal Linzen. 2018. What can linguistics and deep learning contribute to each other? arXiv pre- print:1809.04179v2. https://doi.org/10 .1162/tacl a 00115 Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Compu- tational Linguistics, 4:521–535. Daniel Loureiro and Jose Camacho-Collados. 2020. Don’t neglect the obvious: On the role of unambiguous words in word sense disambigua- tion. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 3514–3520, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020 .emnlp-main.283 Diana McCarthy, Marianna Apidianaki, and Katrin Erk. 2016. Word sense clustering and clusterability. Computational Linguistics, 42(2):245–275. https://doi.org/10.1162 /COLI a 00247 Diana McCarthy, Rob Koeling, Julie Weeds, and John Carroll. 2004. Finding predominant word senses in untagged text. In Proceedings of the 42nd Annual Meeting of the Associa- tion for Computational Linguistics (ACL-04), pages 279–286. Barcelona, Spain. https:// doi.org/10.3115/1218955.1218991 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u / t a c l / l a r t i c e - p d f / d o i / . 1 0 1 1 6 2 / t l a c _ a _ 0 0 4 0 0 1 9 5 5 2 0 4 / / t l a c _ a _ 0 0 4 0 0 p d . f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 In Proceedings of Diana McCarthy and Roberto Navigli. 2007. SemEval-2007 Task 10: English lexical the substitution task. Fourth International Workshop on Semantic Evaluations (SemEval-2007), pages 48–53, Prague, Czech Republic. Association for Computational Linguistics. https://doi .org/10.3115/1621474.1621483 Oren Melamud, Jacob Goldberger, and Ido Dagan. 2016. context2vec: Learning generic context embedding with bidirectional LSTM. In Proceedings of the 20th SIGNLL Confer- ence on Computational Natural Language Learning, pages 51–61, Berlin, Germany. Association for Computational Linguistics. https://doi.org/10.18653/v1/K16 -1006 Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint:1301.3781v3. George A. Miller, Claudia Leacock, Randee Tengi, and Ross T. Bunker. 1993. A semantic concordance. In Human Language Technology: Proceedings of a Workshop Held at Plains- boro. New Jersey. https://doi.org/10 .3115/1075671.1075742 Roberto Navigli and Simone Paolo Ponzetto. 2012. BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artificial Intel- ligence, 193:217–250. https://doi.org /10.1016/j.artint.2012.07.001 Arvind Neelakantan, Jeevan Shankar, Alexandre Passos, and Andrew McCallum. 2014. Efficient non-parametric estimation of multiple embed- dings per word in vector space. In Proceedings of the 2014 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pages 1059–1069, Doha, Qatar. Association for Computational Linguistics. https://doi .org/10.3115/v1/D14-1113 Rafail Ostrovsky, Yuval Rabani, Leonard J. Schulman, and Chaitanya Swamy. 2012. The effectiveness of Lloyd-type methods for the the ACM k-means problem. (JACM), 59(6):28. https://doi.org/10 .1145/2395116.2395117 Journal of Ellie Pavlick, Pushpendre Rastogi, Juri Gan- itkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2015. PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classi- fication. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 425–430, Beijing, China. Association for Computational Linguistics. https://doi.org/10.3115 /v1/P15-2070 the 2018 Conference of Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contex- In Proceed- tualized word representations. ings of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics. https://doi.org/10.18653/v1/N18 -1202 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u / t a c l / l a r t i c e - p d f / d o i / . 1 0 1 1 6 2 / t l a c _ a _ 0 0 4 0 0 1 9 5 5 2 0 4 / / t l a c _ a _ 0 0 4 0 0 p d . f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 for evaluating Mohammad Taher Pilehvar and Jose Camacho- Collados. 2019. WiC: The Word-in-Context dataset context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Short Papers), Volume pages 1267–1273, Minneapolis, Minnesota. Association for Computational Linguistics. (Long and 1 Tiago Pimentel, Rowan Hall Maudslay, Damian Blasi, and Ryan Cotterell. 2020. Speakers lexical semantic gaps with context. In fill Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 4004–4015, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020 .emnlp-main.328 Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9. 842 Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B. Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and measuring the geometry of BERT. In Advances in Neural Information Processing Systems, pages 8592–8600, Vancouver, Canada. Joseph Reisinger and Raymond J. Mooney. 2010. Multi-prototype vector-space models of word meaning. In Human Language Tech- nologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 109–117, Los Angeles, California. Association for Computational Linguistics. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Trans- actions of the Association for Computational Linguistics, 8:842–866. Alex Rosenfeld and Katrin Erk. 2018. Deep neural models of semantic shift. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 474–484, New Orleans, Louisiana. Association for Computational Linguistics. https://doi .org/10.1162/tacl_a_00349 Peter J. Rousseeuw. 1987. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20:53–65. Dominik Schlechtweg, Barbara McGillivray, Simon Hengchen, Haim Dubossarsky, and Nina Tahmasebi. 2020. SemEval-2020 Task 1: Unsu- pervised lexical semantic change detection. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1–23, Barcelona (online). International Committee for Compu- tational Linguistics. https://doi.org/10 .1016/0377-0427(87)90125-7 Pedro Javier Ortiz Su´arez, Benoˆıt Sagot, and Laurent Romary. 2019. Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In Proceedings of the 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7). Cardiff, UK. Leibniz-Institut Sprache. f¨ur Deutsche Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2020. oLMpics—On what language model pre-training captures. Trans- actions of the Association for Computational Linguistics, 8:743–758. https://doi.org /10.1162/tacl_a_00342 Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593–4601, Florence, Italy. Association for Computational Linguistics. https://doi.org/10.18653/v1/P19 -1452 Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. In Advances Attention is all you need. in Neural Information Processing Systems, pages 5998–6008. Long Beach, California, USA. Elena Voita, Rico Sennrich, and Ivan Titov. repre- 2019a. The bottom-up evolution of sentations in the transformer: A study with machine translation and language modeling objectives. In Proceedings of the 2019 Con- ference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pages 4396–4406, Hong Kong, China. Association for Com- putational Linguistics. https://doi.org /10.18653/v1/D19-1448 Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019b. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting the Association for Computational Lin- of guistics, pages 5797–5808, Florence, Italy. Association for Computational Linguistics. https://doi.org/10.18653/v1/P19 -1580 Ivan Vuli´c, Edoardo Maria Ponti, Robert Litschko, Goran Glavaˇs, and Anna Korhonen. 2020. Prob- ing pretrained language models for lexical 843 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u / t a c l / l a r t i c e - p d f / d o i / . 1 0 1 1 6 2 / t l a c _ a _ 0 0 4 0 0 1 9 5 5 2 0 4 / / t l a c _ a _ 0 0 4 0 0 p d . f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3 semantics. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 7222–7240, Online. Association for Computational Linguis- tics. https://doi.org/10.18653/v1 /2020.emnlp-main.586 Gregor Wiedemann, Steffen Remus, Avi Chawla, and Chris Biemann. 2019. Does BERT make any sense? Interpretable word sense disam- biguation with contextualized embeddings. In Proceedings of the 15th Conference on Natural Language Processing (KONVENS 2019): Long Papers, pages 161–170, Erlangen, Germany. German Society for Computational Linguistics & Language Technology. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empir- ical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020 .emnlp-demos.6 Bin Zhang. 2001. Dependence of clustering algorithm performance on clustered-ness of data. HP Labs Technical Report HPL-2001-91. Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei, and Ming Zhou. 2019. BERT-based lexical substitution. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3368–3373, Florence, Italy. Association for Computational Linguistics. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning In Proceedings of books and movies: Towards story-like visual explanations by watching movies and reading the 2015 IEEE books. International Conference on Computer Vision (ICCV’15), pages 19–27, Santiago, Chile. IEEE Computer Society. https://doi.org/10 .1109/ICCV.2015.11 George Kingsley Zipf. 1945. The meaning- frequency relationship of words. Journal of General Psychology, 33(2):251–256. https:// doi.org/10.1080/00221309.1945 .10544509 A Clusterability Metrics Variance Ratio. First, the variance of a cluster y is calculated: σ2(Y ) = 1 |y| (cid:2) i(cid:2)y (yi − ¯y)2 (3) where ¯y denotes the centroid of cluster y. Then the within-cluster variance W and the between-cluster variance B of a clustering solution C are calculated in the following way: W (C) = B(C) = k(cid:2) j=1 k(cid:2) j=1 pjσ2(xj) pj(¯xj − ¯x)2 (4) (5) |xj | where x is the set of all data points and pj = |x| . xj are the data points in cluster j. Finally, the VR of a clustering C is obtained as the ratio between B(C) and W (C): V R = B(C) W (C) (6) Separability (SEP). In an optimal clustering Ck of the dataset x with k clusters, SEP is defined as follows: SEP (x, k) = loss(Ck) loss(Ck−1) (7) 844 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . e d u / t a c l / l a r t i c e - p d f / d o i / . 1 0 1 1 6 2 / t l a c _ a _ 0 0 4 0 0 1 9 5 5 2 0 4 / / t l a c _ a _ 0 0 4 0 0 p d . f b y g u e s t t o n 0 7 S e p e m b e r 2 0 2 3

Télécharger le PDF