Efficient Contextual Representation Learning
With Continuous Outputs
Liunian Harold Li†, Patrick H. Chen∗, Cho-Jui Hsieh∗, Kai-Wei Chang∗
†Peking University
∗University of California, Los Angeles
liliunian@pku.edu.cn, patrickchen@g.ucla.edu
{chohsieh, kwchang}@cs.ucla.edu
Abstracto
Contextual representation models have achieved
great success in improving various downstream
natural language processing tasks. Sin embargo,
these language-model-based encoders are dif-
ficult to train due to their large parameter
size and high computational complexity. Por
carefully examining the training procedure,
we observe that the softmax layer, which pre-
dicts a distribution of the target word, often in-
duces significant overhead, especially when
the vocabulary size is large. Por lo tanto, we re-
visit the design of the output layer and consider
directly predicting the pre-trained embedding
of the target word for a given context. Cuando
applied to ELMo, the proposed approach achieves
a 4-fold speedup and eliminates 80% trainable
parameters while achieving competitive per-
formance on downstream tasks. Further anal-
ysis shows that the approach maintains the
speed advantage under various settings, incluso
when the sentence encoder is scaled up.
1 Introducción
En años recientes, text representation learning ap-
se acerca, such as ELMo (Peters et al., 2018a),
GPT (Radford et al., 2018), BERT (Devlin et al.,
2019), and GPT-2 (Radford et al., 2019), tener
been developed to represent generic contextual
information in natural languages by training an
encoder with a language model objective on
a large unlabelled corpus. During the training
proceso, the encoder is given part of the text
and asked to predict the missing pieces. Previo
studies show that the encoders trained in this way
can capture generic contextual information of the
input text and improve a variety of downstream
tasks significantly.
Sin embargo, training contextual representations
is known to be a resource-hungry process. Para
ejemplo, ELMo is reported to take about 2
weeks to train on a one-billion-token corpus
with a vocabulary of 800,000 words using three
GPUs.1 This slow training procedure hinders the
development cycle, prevents fine-grained param-
eter tuning, and makes training contextual repre-
sentations inaccessible to the broader community.
Recent work also raises concerns about the envi-
ronmental
implications of training such large
modelos (Strubell et al., 2019). Además, the suc-
cess of these models stems from a large amount of
data they used. It is challenging, if not impossible,
to train a contextual representation model on a
larger corpus with tens or hundreds of billions of
tokens.
En este trabajo, we explore how to accelerate
contextual representation learning. We identify the
softmax layer as the primary cause of inefficiency.
This component takes up a considerable portion
de todo
trainable parameters (80% for ELMo)
and consumes a huge amount of training time.
Sin embargo, it is often not needed in the final model
as the goal of contextual representation learning
is to build a generic encoder. Por lo tanto, es
rather a waste to allocate extensive computational
resources to the softmax layer.
Inspired by Kumar and Tsvetkov (2019), nosotros estafamos-
sider learning contextual representation models
with continuous outputs. In the training process,
the contextual encoder is learned by minimizing
the distance between its output and a pre-trained
target word embedding. The constant time com-
plexity and small memory footprint of the output
layer perfectly serve our desire to decouple learn-
ing contexts and words and devote most com-
putational resources to the contextual encoder. En
addition, we combine the approach with open-
vocabulary word embeddings such that the model
can be trained without the need to pre-define a
1https://github.com/allenai/bilm-tf/
issues/55.
611
Transacciones de la Asociación de Lingüística Computacional, volumen. 7, páginas. 611–624, 2019. https://doi.org/10.1162/tacl a 00289
Editor de acciones: Lucas Zettlemoyer. Lote de envío: 1/2019; Lote de revisión: 6/2019; Publicado 9/2019.
C(cid:13) 2019 Asociación de Lingüística Computacional. Distribuido bajo CC-BY 4.0 licencia.
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
mi
d
tu
/
t
a
C
yo
/
yo
a
r
t
i
C
mi
–
pag
d
F
/
d
oh
i
/
.
1
0
1
1
6
2
/
t
yo
a
C
_
a
_
0
0
2
8
9
1
9
2
3
6
5
6
/
/
t
yo
a
C
_
a
_
0
0
2
8
9
pag
d
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
closed word set as the vocabulary. We also provide
an alternative interpretation of learning contextual
encoders with continuous outputs that sheds light
on how the pre-trained embedding could affect the
performance of the model.
We conduct a comprehensive empirical study
to analyze the proposed approach and several
existing methods that are originally proposed to
reduce the complexity of the output layer in lan-
guage models, such as the adaptive softmax, y
the sub-word methods. We incorporate these ap-
proaches with ELMo and conduct a comprehen-
sive study to compare them in terms of training
speed and performance on five downstream tasks.
We demonstrate that the proposed approach ef-
fectively reduces the training time and trainable
parameters while maintaining competitive perfor-
mance compared with the baselines. Our approach
also exhibits consistent computational advanxtage
under different conditions (p.ej., with different vo-
cabulary sizes, with different sentence encoders,
and with different number of GPUs).
El código fuente está disponible en https://github.
com/uclanlp/ELMO-C.
2 Background and Related Work
Contextual representation We review contex-
tual representation models from two aspects:
how they are trained and how they are used in
downstream tasks.
CoVe (McCann et al., 2017) uses the source lan-
guage encoder from a machine translation model
as a contextual representation model. Peters et al.
(2018a) advocate for the use of larger unlabelled
corpora and proposes ELMo, a forward and a back-
ward LSTM-based (Hochreiter and Schmidhuber,
1997) modelo de lenguaje, whereas GPT (Radford
et al., 2018) and GPT-2 (Radford et al., 2019) build
a language model with the Transformer (Vaswani
et al., 2017). BERT (Devlin et al., 2019) introducción-
duces the masked language model and provides
deep bidirectional representation.
There are two existing strategies for applying
pre-trained contextual representations to down-
stream tasks: 1) feature-based and 2) fine-tuning.
In the feature-based approach, fixed features
are extracted from the contextual encoder (p.ej.,
ELMo, CoVe) and inserted as an input into a
task-specific model. In the fine-tuning approach,
the contextual encoder is designed as a part of
the network architecture for downstream tasks,
and its parameters are fine-tuned with the down-
stream task. BERT is designed for the fine-tuning
approach but it is also evaluated with the feature-
based approach. GPT-2 is a scaled-up version
of GPT and exhibits strong performance under
zero-shot settings.
Speeding up language models training Con-
siderable efforts have been devoted to accelerat-
ing the training process of language models. Uno
line of research focuses on developing faster
sequence encoder architectures such as CNN
(Kim y cols., 2016; Dauphin et al., 2017), QRNN
(Bradbury et al., 2016), SRU (Lei et al., 2018),
and the Transformer (Vaswani et al., 2017).
These architectures have been extensively used
for learning language representations (Radford
et al., 2018; Devlin et al., 2019; Tang et al.,
2018). Another line of work focuses on the large-
vocabulary issue, as a large and ever-growing vo-
cabulary results in an intractable softmax layer.
Our work falls into the second line and we review
existing solutions in detail.
Several studies for language modeling focus
on directly reducing the complexity of the soft-
max layer. Following Kumar and Tsvetkov (2019),
we group them into two categories: sampling-
based approximations and structural approxima-
ciones. Sampling-based approximations include the
sampled softmax (Bengio et al., 2003) and NCE
(Mnih and Teh, 2012). The sampled softmax ap-
proximates the normalization term of softmax by
sampling a subset of negative targets, and NCE
replaces the softmax with a binary classifier. On
the other hand, structural approximations such as
the hierarchical softmax (Morin and Bengio, 2005)
and the adaptive softmax (Grave et al., 2016), forma
a structural hierarchy to avoid expensive nor-
malization. The adaptive softmax, En particular,
groups words in the vocabulary into either a short-
list or clusters of rare words. For frequent words,
a softmax over the short-list would suffice, cual
reduces computation and memory usage signifi-
cantly. The adaptive softmax has been shown to
achieve results close to those of the full softmax
while maintaining high GPU efficiency (Merity
et al., 2018).
Regarding contextual representation models,
ELMo used the sampled softmax and GPT and
BERT resorted to a subword method. Specifi-
cally, they used WordPiece (Wu et al., 2016) o
BPE (Sennrich et al., 2016) to split the words into
612
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
mi
d
tu
/
t
a
C
yo
/
yo
a
r
t
i
C
mi
–
pag
d
F
/
d
oh
i
/
.
1
0
1
1
6
2
/
t
yo
a
C
_
a
_
0
0
2
8
9
1
9
2
3
6
5
6
/
/
t
yo
a
C
_
a
_
0
0
2
8
9
pag
d
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
subwords and the language models were trained
to take subwords as input and also predict sub-
palabras. This method is efficient and scalable, como
the subword vocabulary can be kept small. Uno
potential drawback of these subword-level lan-
guage models, sin embargo, is that they produce rep-
resentations for fragments of words. Por lo tanto,
it takes extra effort to generate word-level repre-
sentaciones (see the discussion in Section 4.2).
The high cost of the softmax layer has also
been noted in the sentence representation learning
literature. Following the success of Word2Vec
(Mikolov et al., 2013), methods such as SkipThought
(Kiros et al., 2015) have been developed to learn
distributed sentence representations by predicting
the context sentences of a given sentence, cual
involves sequentially decoding words of the target
oraciones. Jernite et al. (2017) and Logeswaran
and Lee (2018) notice the inefficiency of the
softmax layer during decoding and propose to use
discriminative instead of generative objectives,
eliminating the need for decoding. Sin embargo, estos
approaches are not directly applicable to contex-
tual representation learning.
3 Acercarse
A contextual representation model, at its core, es
a language model pre-trained on a large unlabeled
cuerpo. En el siguiente, we review the objective
of language models and the architectures of exist-
ing contextual representation models. Nosotros entonces
introduce the proposed model.
Language model objective Given a set of text
sequences as the training corpus, we can construct
a collection of word-context pairs (w, C), y el
goal of a language model is to predict the word
w based on the context c. In a forward language
modelo, the context c is defined as the previous
words in the sequence, whereas for a backward
modelo de lenguaje, the context of a word is defined
as the following words. For a masked language
modelo, some words in the input sentence are
enmascarado (p.ej., replaced by a [MASK] simbólico) y
the objective is to predict the masked words from
the remainder. Different contextual representa-
tion models optimize different objectives. Para
ejemplo, ELMo trains a forward and backward
language model and BERT trains a masked-
modelo de lenguaje.
613
Model architecture A typical neural language
model consists of three parts: 1) an input layer, 2)
a sequence encoder, y 3) a softmax layer. Given
a word-context pair (w, C), the input layer uses a
word embedding or a character-CNN model (kim
et al., 2016) to convert the input words in c into
word vectors. Then the sequence encoder embeds
the context into a context vector c ∈ Rm using a
multi-layer LSTM (Hochreiter and Schmidhuber,
1997), a Gated CNN (Dauphin et al., 2017), o un
Transformador (Vaswani et al., 2017). The softmax
layer then multiplies the context vector c with
an output word embedding2 W ∈ RV ×m and
uses a softmax function to produce a conditional
distribution p(w|C) over the vocabulary of size V .
In a language model, the learning objective
yo(w, C) para (w, C) is then expressed as:
yo(w, C) = − log p(w|C)
= − log softmax(cW T
= −c · w + log Xw′
)
exp.(c · w′
), (1)
where w ∈ Rm is a row from W corresponding
to the target word w and the second term sums
over the vocabulary. After the model is trained, el
contextual representations are generated from the
latent states of the sequence encoder. Por ejemplo,
ELMo combines the hidden states of the LSTMs
to generate contextualized word embedding for
each word in a sentence. We refer the reader to
Peters et al. (2018a) for details.
Note that the size of W and the computational
complexity of the second term in Eq. (1) escala
linearly to the vocabulary size, V . Por lo tanto,
when V is large, the softmax layer becomes the
speed bottleneck.
Our approach The scaling issue of softmax also
occurs in other language generation and sequence-
to-sequence models. In the literature, several ap-
proaches have been proposed to approximate the
softmax layer or bypass it with a subword method
(mira la sección 2). Recientemente, Kumar and Tsvetkov
(2019) propose to treat the context vector as con-
tinuous outputs and directly minimize the distance
2The dimension of the original output from the sequence
encoder may not match the dimension of the output word
incrustar. In that case, a projection layer is added after the
original sequence encoder to ensure that the two dimensions
match.
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
mi
d
tu
/
t
a
C
yo
/
yo
a
r
t
i
C
mi
–
pag
d
F
/
d
oh
i
/
.
1
0
1
1
6
2
/
t
yo
a
C
_
a
_
0
0
2
8
9
1
9
2
3
6
5
6
/
/
t
yo
a
C
_
a
_
0
0
2
8
9
pag
d
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
between the context vector and the pre-trained
word embedding associated with the target word,
yo(w, C) = d(C, w)
(2)
The distance function l could be the L2 distance
c·w
kc − wk2, the cosine distance
kckkwk or a prob-
abilistic distance metric.
We argue that the idea of learning with con-
tinuous outputs particularly suits contextual rep-
resentation learning. As the goal is to obtain a
strong contextual encoder, it makes sense to use a
pre-trained output word embedding and decouple
learning the contextual encoder and the output
incrustar. In the remainder of this section, nosotros
discuss the computational efficiency of the pro-
posed approach and its combination with the open-
vocabulary word embedding. We also provide an
alternative way to interpret training contextual en-
coders with continuous outputs.
3.1 Computational Efficiency
The continuous output layer has a reduced arith-
metic complexity and trainable parameter size.
We illustrate these improvements and how they
contribute to reducing the overall training time of a
contextual representation model in the following.
For comparison, we include the sampled softmax,
the adaptive softmax, and the subword method in
the discussion.
3.1.1 Learning with Continue Outputs
Arithmetic complexity The arithmetic com-
plejidad (es decir., FLOPs) of evaluating loss with con-
tinue outputs (es decir., ecuación. 2) takes O(metro), as we
only need to calculate the distance between two
m-dimensional vectors. The complexity of the
sampled softmax is proportional to the number of
negative samples per batch. When the vocabulary
is huge, a large number of negative samples are
needed (Jozefowicz et al., 2016). For the adaptive
softmax, the time complexity is determined by the
capacities of the short-list and the rare-word clus-
ters, which grows sub-linearly to the vocabulary
tamaño. The complexity of the subword method is
determined by the subword vocabulary size. En
contrast, the time spent on the continuous output
layer and loss evaluation remains constant with
respect to the vocabulary size and is negligible.
Trainable parameter size The output word
embedding usually takes up a huge part of the
parameters of a language model. Por ejemplo, el
softmax layer in ELMo trained on the One Billion
Word Benchmark (Chelba et al., 2013) takes up
más que 80% of the trainable parameters of
the entire model. Even if an approximation such
as the sampled softmax is used, the number of
trainable parameters is not reduced. Enfoques
like the adaptive softmax reduce the dimension of
softmax embedding for rare words, the trainable
parameter size of which is effectively reduced but
still remains sizable. For a model trained on the
same corpus (Grave et al., 2016), the adaptive
softmax still amounts to 240 million parameters
whereas the sequence encoder has only around
50 million parameters. On the contrary, we learn
a contextual encoder with Eq. (2) using a pre-
trained word embedding, reducing the trainable
parameters besides the encoder from tens or hun-
dreds of millions to zero.
3.1.2 Overall Training Time
We now discuss how the efficiency improvements
to the output layer contribute to the reduction
of the overall training time, in the context of
synchronous stochastic gradient descent training
on multiple GPUs. En general, the following three
factors determine the training time.
Arithmetic complexity The arithmetic com-
plexity of a model includes the complexity of the
forward and backward propagation on the in-
put layer, the sequence encoder, and the output
capa. It also includes the overhead of the opti-
mization algorithm such as gradient clipping and
model updates. The complexity of this optimiza-
tion overhead is often proportional to the number
of parameters that need updating. With the con-
tinuous output layer, not only the arithmetic com-
plexity but also the optimization overhead are
reduced.
GPU memory consumption The training time
is also affected by GPU memory consumption,
as less GPU memory consumption leads to larger
batch size. For the same amount of data and hard-
ware resource, larger batch size means better
parallelism and less training time. Our approach
exhibits small GPU memory footprints, due to
reductions of the arithmetic complexity (con
fewer intermediate results to keep) and trainable
parameter size (with fewer parameters to store).
Como resultado, training with continuous outputs is 2
a 4 times more memory-efficient than with the
softmax layer (mira la sección 5.2).
614
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
mi
d
tu
/
t
a
C
yo
/
yo
a
r
t
i
C
mi
–
pag
d
F
/
d
oh
i
/
.
1
0
1
1
6
2
/
t
yo
a
C
_
a
_
0
0
2
8
9
1
9
2
3
6
5
6
/
/
t
yo
a
C
_
a
_
0
0
2
8
9
pag
d
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Note that as the output word embedding is
fixed, we can keep that embedding in the main
memory and only load the required part to the
GPU memory. Despite the fact that this comes
with an overhead of moving part of the output
word embedding from CPU to GPU memory at
each iteration, the benefit of parallelism often
dominates over the communication overhead on
mainstream hardware, where the GPU memory
is often comparatively limited. We also note that
larger batch size may lead to difficulty in opti-
mization. Several methods have been developed
to ease the large-batch training issue (Goyal et al.,
2017; You et al., 2018). We show that these meth-
ods are sufficient for resolving the optimization
difficulty in our experiment (Sección 4).
Communication cost To train large neural net-
work models, using multiple GPUs almost becomes
a necessity. Además, one way to scale up
current systems is to increase the number of GPUs
usado. In such cases, the communication cost across
GPUs needs to be taken into consideration. El
cost occurs from synchronizing the parameters and
their gradients across GPUs, which is proportional
to the size of parameters that need to be updated.
For the sampled softmax, due to the use of the
sparse gradient, the communication cost is pro-
portional to the number of the sampled words. Para
the adaptive softmax and the subword language
modelo, the communication cost is proportional
to the trainable parameter size. The continuous
output layer, por otro lado, incurs little com-
munication cost across GPUs.
3.2 Open-Vocabulary Training
We utilize the open-vocabulary word embedding
as both the input and output layer embedding. Open-
vocabulary word embeddings, such as the FastText
embedding and the MIMICK model (Pinter et al.,
2017), utilize character or subword information to
provide word embeddings. They could represent
an unlimited number of words with a fixed number
of parameters. Como resultado, we can train contextual
encoders with an open vocabulary, which means
we do not need to pre-define a closed word set as
the vocabulary and the model can be trained on
any sequences of words.
Open-vocabulary input layer To be easily ap-
plied in various tasks, the contextual encoder usu-
ally has an open-vocabulary input layer. ELMo
uses a character-CNN but it is relatively slow.
615
Thus we use a pre-trained open-vocabulary word
embedding as the input layer of the contextual
encoder, reducing the time complexity of the input
layer to a negligible level. This also aligns with
the main spirit of our approach, which is to spend
computational resources on the most important
part, the sequence encoder.
Open-vocabulary output layer For the soft-
max layer, including efficient variants such as the
adaptive softmax, the output vocabulary has to
be pre-defined so that the normalization term can
be calculated. As the softmax layer’s arithmetic
complexity and parameter size grow when the vo-
cabulary size grows, the vocabulary is often trun-
cated to avoid expensive computation.
With the continuous output layer, we can con-
duct training on an arbitrary sequence of words, como
long as the output embedding for those words can
be derived. This can be achieved by using the
open-vocabulary embedding. This feature is espe-
cially attractive if we are training on domains or
languages with a long-tail word distribution such
as the biomedical domain, where truncating the
vocabulary may not be acceptable.
3.3 Interpretation of Learning Contextual
Encoders with Continuous Outputs
En el siguiente, we justify the intuition behind
learning with continue outputs and discuss how
the pre-trained word embedding affects the per-
formance of the model.
Language models are essentially modeling the
word-context conditional probability matrix, eso
es, A ∈ RN ×V where Ac,w = p(w|C), N is the
number of all possible contexts, and V is the
vocabulary size (Levy and Goldberg, 2014; Cual
et al., 2017). The continuous output layer can
be viewed as modeling A after using the word
embedding as a projection matrix.
Para ilustrar esto, consider the global objective
of the layer with the cosine distance:3
L = X(w,C)
#(w, C)yo(w, C)
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
mi
d
tu
/
t
a
C
yo
/
yo
a
r
t
i
C
mi
–
pag
d
F
/
d
oh
i
/
.
1
0
1
1
6
2
/
t
yo
a
C
_
a
_
0
0
2
8
9
1
9
2
3
6
5
6
/
/
t
yo
a
C
_
a
_
0
0
2
8
9
pag
d
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
#(w, C)c · w
= − X(w,C)
= − Xc
= − Xc
#(C)c·X
w
#(C)c·X
w
pag(w|C)w,
C.A,ww,
3Por simplicidad, we take the cosine distance as a running
example but the conclusions hold for other distance functions.
Modelo
Input
Sequence Encoder Output
ELMO
ELMO-C (OURS)
ELMO-A
ELMO-Sub
CNN
FASTTEXTCC
FASTTEXTCC
Subword
LSTM
LSTM w/ LN
LSTM w/ LN
LSTM w/ LN
ELMO-CONEB
ELMO-CRND
ELMO-CCNN
ELMO-CCNN-CC
ELMO-CCC-CNN
FASTTEXTONEB LSTM w/ LN
FASTTEXTCC
LSTM w/ LN
Trained CNN LSTM w/ LN
Trained CNN LSTM w/ LN
LSTM w/ LN
FASTTEXTCC
Sampled Softmax
CONT w/ FASTTEXTCC
Adaptive Softmax
Softmax
CONT w/ FASTTEXTONEB
CONT w/ Random Embedding
CONT w/ Trained CNN
CONT w/ FASTTEXTCC
CONT w/ Trained CNN
Mesa 1: Specifications of variants of ELMo models compared in Sections 4 y 5. CONT
means the model has continuous outputs. LN means layer normalization.
dónde #(w, C) is the number of occurrences of the
pair (w, C) in the corpus. We assume all vectors
(c and w) are normalized.
To optimize the inner product between c and
Pw p(w|C)w, we essentially align the direction
of context vector c with the expected word vector
under context c, Pw p(w|C)w = Ew∼p(w|C)w. En
otras palabras, given a word embedding matrix
W ∈ RV ×d, our approach aligns c with the cor-
responding row (AW )C,: in AW . Por lo tanto, the ob-
jective can be viewed as conducting multivariate
regression to approximate (AW )C,: given the context.
Based on this view, the performance of the
contextual representation model depends on how
much information of the original matrix A is
preserved after projection with W . In the spirit
of PCA (Jolliffe, 2011), to keep the variance of
A, we would like to have (AW )C,: y (AW )c′,:
distant from each other if c and c′ are very different.
Por lo tanto, a pre-trained word embedding, cual
projects words with different meanings into
different positions in space, is a natural choice
for the projection matrix W and can help preserve
much of the variance of A.
4 Experimento
We accelerate ELMo with the proposed approach
and show that the resulting model ELMO-C is
computationally efficient and maintains competi-
tive performance, compared to the original ELMo
modelo (ELMO), an ELMo variant with the adap-
tive softmax (ELMO-A4), and another variant with
the subword method (ELMO-Sub).
4We include ELMO-A instead of a model with sampled
softmax because the adaptive softmax has been shown to
have superior performance (Grave et al., 2016).
4.1 Setup
Modelos
En el siguiente, we introduce the mod-
els in detail. Mesa 1 provides a summary. El
original ELMo consists of a character-CNN as
the input layer, a forward and backward LSTM
with projection as the sequence encoder, y un
sampled softmax as the output layer. Adagrad
(Duchi et al., 2011) is used as the optimizer. Nosotros
conduct experiments using the reimplementation
of ELMO in AllenNLP (Gardner et al., 2018) y
build the others upon it.
The key difference between ELMO-C and ELMO
is that ELMO-C produces continuous outputs and
we train it with the cosine distance loss. A FastText
embedding trained on Common Crawl (Mikolov
et al., 2017) (FASTTEXTCC) is used as the output
incrustar. Based on preliminary experiments,
we also make three minor changes: 1) we use
FASTTEXTCC as the input layer as it is more efficient
than the character-CNN model; 2) we add a layer
norm (Ba et al., 2016) after the projection layer of
the LSTM to improve the convergence speed; 3)
we use Adam with the learning rate schedule from
Chen et al. (2018) to help training with a large
batch size.
Our main goal is to study how different output
layers affect the training speed and performance,
which cannot be achieved by just comparing
ELMO-C and ELMO, due to the aforementioned
minor changes to ELMO-C. Por lo tanto, presentamos-
duce two additional baseline models (ELMO-A
and ELMO-Sub), which differ from ELMO-C in a
minimal way. Específicamente,
their sequence en-
coders and training recipes are kept the same as
ELMO-C. Thus ELMO-C, ELMO-A, and ELMO-Sub
can be directly compared.
616
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
mi
d
tu
/
t
a
C
yo
/
yo
a
r
t
i
C
mi
–
pag
d
F
/
d
oh
i
/
.
1
0
1
1
6
2
/
t
yo
a
C
_
a
_
0
0
2
8
9
1
9
2
3
6
5
6
/
/
t
yo
a
C
_
a
_
0
0
2
8
9
pag
d
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
ELMOORG BASE FASTTEXTCC
Time
−
Batch
−
Params −
SNLI
Coref
SST-5
NER
srl
88.7
NA
54.7
92.22
84.6
−
−
−
−
−
−
87.7
88.0
68.90
NA
51.4
51.30 ± 0.77
90.15 90.97 ± 0.43
80.2
81.4
ELMO
14 X 3
128
499METRO
ELMO-A
ELMO-Sub
ELMO-C
5.7 X 4
256
196METRO
3.9 X 4
320
92METRO
2.5 X 4
768
76METRO
88.9
72.9
88.5
72.9
52.96 ± 2.26 53.58 ± 0.77 53.02 ± 2.08 53.80 ± 0.73
92.51 ± 0.28 92.28 ± 0.20 92.17 ± 0.56 92.24 ± 0.10
83.4
87.1
72.4
88.8
72.9
82.7
82.4
82.4
Mesa 2: Computational efficiency of the main competing models and their performance on five NLP
benchmarks. Time is the overall training time in Days x Cards format. Batch is the maximal batch size
per card. Params is the number of trainable parameters in millions. Due to the small test sizes for NER
and SST-5, we report mean and standard deviation across three runs. Our approach (ELMO-C) exhibits
better computational efficiency and shows comparable performance compared with ELMO, ELMO-A,
and ELMO-Sub.
ELMO-A uses the adaptive softmax as its output
capa. We carefully choose the hyper-parameters
of the adaptive softmax to obtain an efficient yet
strong baseline. It has only half of the parameters
of the one reported in Grave et al. (2016) pero
achieves a perplexity of 35.8, lower than ELMO’s
39.7.
ELMO-Sub takes subwords as input and also
predicts subwords. De este modo, unlike other models, es
vocabulary consists of around 30,000 subwords
created using BPE (Sennrich et al., 2016). Para
this reason, a lookup-table-style embedding rather
than FASTTEXTCC is used as its input layer and a
vanilla softmax is used as its output layer. Its input
and output word embedding are tied and trained
from scratch.
For reference, we also list the results of ELMo
and the baseline reported in Peters et al. (2018a)
as ELMOORG and BASE. Sin embargo, these models are
evaluated using different configurations. Finalmente,
we also include FASTTEXTCC a (non-contextual)
word embedding model, as another baseline.
All contextual representation models are trained
on the One Billion Word Benchmark for 10
epochs and the experiments are conducted on a
workstation with 8 GeForce GTX 1080Ti GPUs,
40 Intel Xeon E5 CPUs, and 128G main memory.
Downstream benchmarks We follow Peters
et al. (2018a) and use the feature-based approach
to evaluate contextual representations on down-
stream benchmarks. ELMo was evaluated on six
benchmarks and we conduct evaluations on five
de ellos. Equipo (Rajpurkar et al., 2016) is not
available for implementation reasons.5 In the
following, we briefly review the benchmarks and
task-specific models. For details please refer to
Peters et al. (2018a).
• SNLI (Bowman et al., 2015): The textual
entailment task seeks to determine whether
a ‘‘hypothesis’’ can be entailed from a
‘‘premise’’. The task-specific model is ESIM
(Chen et al., 2017).
• Coref: Coreference resolution is the task
of clustering mentions in text that refer to
the same underlying entities. The data set
is from CoNLL 2012 shared task (Pradhan
et al., 2012) and the model is from Lee et al.
(2018). Note that we use an improved version
of the Coref system (Lee et al., 2017) used in
Peters et al. (2018a).
• SST-5 (Socher et al., 2013): The task in-
volves selecting one of five labels to describe
a sentence from a movie review. We use the
BCN model from McCann et al. (2017).
• NER: The CoNLL 2003 NER task (Sang
and De Meulder, 2003) consists of newswire
from the Reuters RCV1 corpus tagged with
four different entity types. The model is a
biLSTM-CRF from Peters et al. (2018a),
similar to Collobert et al. (2011).
• SRL: Semantic role labeling models the
predicate-argument structure of a sentence. Él
5The SQuAD experiment in Peters et al. (2018a) era
conducted with an implementation in TensorFlow. El
experiment setting is not currently available in AllenNLP
(https://github.com/allenai/allennlp/
pull/1626), nor can it be easily replicated in PyTorch.
617
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
mi
d
tu
/
t
a
C
yo
/
yo
a
r
t
i
C
mi
–
pag
d
F
/
d
oh
i
/
.
1
0
1
1
6
2
/
t
yo
a
C
_
a
_
0
0
2
8
9
1
9
2
3
6
5
6
/
/
t
yo
a
C
_
a
_
0
0
2
8
9
pag
d
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
seeks to answer ‘‘Who did what to whom’’.
The model is from He et al. (2017) y el
data set is from Pradhan et al. (2013).
For SNLI, SST-5, NER, and SRL, we use
the same downstream models as in Peters et al.
(2018a) re-implemented in AllenNLP.6 For Coref,
Peters et al. (2018a) uses the model from Lee et al.
(2017) and we use an improved model (Lee et al.,
2018) from the same authors. For all the tasks,
the hyper-parameters are tuned to maximize the
performance for the original ELMo and all models
are tested under the same configurations.
4.2 Main Results
We report the main results in Table 2. Our ap-
proach (ELMO-C) enjoys a substantial compu-
tational advantage while maintaining competitive
or even superior performance, compared to ELMO,
ELMO-A, and ELMO-Sub.
Model efficiency For model efficiency,
el
statistics of ELMO is reported by the original
authors and they use three GTX 1080 Tis. Nosotros
train ELMO-A, ELMO-Sub, and ELMO-C using
four GTX 1080 Tis. Roughly speaking, comparado
with ELMO, ELMO-C is 4.2x faster and 6x more
memory-efficient. To give a clear view of the
speedup the CONT layer brings, comparamos
ELMO-C with ELMO-A. ELMO-A differs from
ELMO-C only in the output layer. Still, ELMO-
C has a 2.28x speed advantage and is 3x more
memory-efficient. Compared with ELMO-Sub, nuestro
approach holds a 1.56x speed advantage and is
2x more memory-efficient. The results here only
show the overall efficiency of our approach under
the setting of ELMo and a detailed analysis of
the efficiency is desirable, which we provide in
Sección 5.2.
Performance on downstream tasks ELMO-C
works especially well on semantic-centric tasks,
such as SNLI, Coref, and SST-5. Sin embargo, para
tasks that required a certain level of syntactic
información, such as NER and SRL (He et al.,
2018), ELMO-C suffers from slight performance
degradation compared with ELMO, but it remains
competitive with ELMO-A and ELMO-Sub. Nosotros
suspect that the performance degradation is related
to the pre-trained embedding and conduct further
analysis in Section 5.1.
6For SRL, the score reported by AllenNLP is lower than
the score reported by CoNLL official script.
Además, we notice that the performance of
ELMO-Sub is unstable. It shows satisfying per-
formance on SST-5, NER, and SRL. Sin embargo,
it lags behind on Coref and even fails to outper-
form FASTTEXTcc on SNLI. ELMO-Sub provides
subword-level contextual representations, cual
we suspect could be the cause of unstable perfor-
mance. Específicamente, to get the representation for a
word in evaluation on word-level tasks, we follow
Devlin et al. (2019) to use the representation of its
first subword. This could be sub-optimal if precise
word-level representation is desired (p.ej., the suf-
fix of a word is an important feature). Estos resultados
are consistent with the observation in Kitaev and
Klein (2018). They find that special design has to
be made to apply BERT to constituency parsing
because of the subword segmentation. Sin embargo,
we notice that the scope of our experiment is lim-
ited. It is likely that when ELMO-Sub is scaled
up or used with the fine-tuning method, the afore-
mentioned issue is alleviated—we leave that to
future work.
5 Análisis
We conduct analysis regarding the effect of the
pre-trained word embedding on the performance
of the contextual encoder. We also investigate the
contributions of different factors to the overall
training time and study the speedup of our ap-
proach under various conditions.
5.1 Effect of the Pre-trained Embedding
We show the effect of the pre-trained embedding
by introducing several variants of ELMO-C (sum-
marized in Table 1) and list their performance in
Mesa 3.
Quality of the pre-trained embedding We
aim to understand how the quality of the pre-
trained output word embedding W affects the
performance of the contextual encoder. To study
este, we train a FastText word embedding on the
One Billion Word Benchmark, a much smaller
corpus than Common Crawl. We then train an
ELMO-C variant, ELMO-CONEB, by using this em-
bedding in the input and output layers. Com-
to ELMO-C, ELMO-CONEB holds up
paring it
surprisingly well and it is competitive on SNLI,
Coref, and SST-5 while being inferior on NER
and SRL.
This motivates us to take a step further and
use a completely random output word embedding.
618
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
mi
d
tu
/
t
a
C
yo
/
yo
a
r
t
i
C
mi
–
pag
d
F
/
d
oh
i
/
.
1
0
1
1
6
2
/
t
yo
a
C
_
a
_
0
0
2
8
9
1
9
2
3
6
5
6
/
/
t
yo
a
C
_
a
_
0
0
2
8
9
pag
d
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Tarea
ELMO-C
ELMO-CONEB ELMO-CRND
ELMO-CCNN ELMO-CCNN-CC
ELMO-CCC-CNN
SNLI
Coref
SST-5
NER
srl
88.8
72.9
53.80 ± 0.73
92.24 ± 0.10
82.4
88.4
72.4
88.4
73.0
52.70 ± 0.90 53.01 ± 1.67
92.03 ± 0.47 91.99 ± 0.35
82.2
82.9
88.0
72.8
88.2
72.9
53.38 ± 0.68 54.33 ± 1.26
92.24 ± 0.36 92.04 ± 0.33
82.8
83.4
88.4
72.6
54.16 ± 0.96
91.93 ± 0.53
83.3
Mesa 3: Performance of ablation models on five NLP benchmarks. ELMO-C is included for reference.
We replace the output embedding of ELMO-C
with a random embedding matrix, of which each
element is randomly drawn from a standard normal
distribución. We denote this model as ELMO-CRND.
We find that this model performs well (Mesa 3),
with only a mild performance drop compared to
ELMO-C. The performance of ELMO-CRND shows
the robustness of the proposed approach and
demonstrates that the deep LSTM is expressive
enough to fit a complex output space. Sin embargo,
we find that the pre-trained input word embedding
is still indispensable because using a randomly
initialized input embedding would lead to brittle
actuación (p.ej., 85.8 on SNLI).
Pre-trained CNN layer as word embedding
En la sección 4, we observed that models using Fast-
Text embedding (ELMO-C and ELMO-A) as input
performed worse than ELMo on SRL, una tarea
relying heavily on syntactic information. Nosotros
suspect that the FastText embedding is weaker
on capturing syntactic information than the
character-CNN trained in ELMo (Peters et al.,
2018b). To verify this, we train ELMO-C using
the trained CNN layer from ELMo as the input
capa (ELMO-CCNN-CC) or the output embedding
(ELMO-CCC-CNN). We observe that the two models
exhibit notably better performance on SRL (ver
Mesa 3). We also consider a ELMO-CCNN model,
which uses the CNN layer as both the input and
output embedding. On SRL, ELMO-CCNN per-
forms favorably compared to ELMO-C but slightly
worse than ELMO-CCNN-CC or ELMO-CCC-CNN.
We suspect that this is because ELMO-CCNN-CC
and ELMO-CCC-CNN benefit from different kinds
of embeddings in the input layer and the output
capa.
5.2 Computational Efficiency
Próximo, we study the computational efficiency of the
continuous output layer against several baselines
from two aspects. Primero, en la sección 3.1, we dis-
cussed three factors governing the overall training
time of the model: 1) arithmetic complexity, 2)
GPU memory consumption, y 3) communica-
tion cost. We aim to study how each factor affects
the overall training time of each model. Segundo,
in the above experiments, we focus on ELMo
with the LSTM as the sequence encoder. Nosotros
wonder whether the continuous output layer can
deliver attractive speedup for sequence encoders
of different types and sizes.
We investigate the continuous output
capa
(CONT) and three common baseline output layers:
1) the subword-level language model (SUBWORD),
2) the adaptive softmax layer (ADAPTIVE), y 3)
the sampled softmax layer (SAMPLED). Addition-
ally, we include a variant of the sampled softmax
denoted as FIXED where the output word embed-
ding is initialized by the FastText embedding and
fixed during the training. This output layer is
similar to a special case of CONT with a ranking
loss, where the model encourages its output to be
close to the target word embedding but far from a
negative sample.
In total, we study five different output layers.
For several output layers, the trade-off between
computational efficiency and model performance
is controlled by their hyper-parameters. Nosotros
choose hyper-parameters close to those reported
in the literature to strike a balance between speed
and performance.
5.2.1 Speedup Breakdown
We pair the five different output layers with the
same input layer (fixed word embedding) y
sequence encoder (ELMo’s LSTM with projec-
ción). We then test the training speed of these
models under three scenarios, which are designed
to reflect the individual effect of the arithmetic
complejidad, the GPU memory consumption, y
the communication cost:
• S1 (small batch): We use one GPU card and
set the batch size to be 1. The asynchronous
execution feature of the GPU is disabled. El
time needed to finish one batch is reported.
619
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
mi
d
tu
/
t
a
C
yo
/
yo
a
r
t
i
C
mi
–
pag
d
F
/
d
oh
i
/
.
1
0
1
1
6
2
/
t
yo
a
C
_
a
_
0
0
2
8
9
1
9
2
3
6
5
6
/
/
t
yo
a
C
_
a
_
0
0
2
8
9
pag
d
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Vocab
Params Batch
S1 (small batch) S2 (large batch) S3 (multiple GPUs)
CONT
FIXED
SUBWORD
ADAPTIVE
SAMPLED
∞
∞
∞
40k
800k
2000k
40k
800k
76METRO 640
76METRO 512
92METRO 320
97METRO 384
196METRO 256
213METRO 192
96METRO 512
483METRO 256
64
2000K 1102M
0.47s
1.17X
1.09X
1.08X
1.16X
1.25X
1.07X
1.15X
1.16X
115.28s
1.24X
1.53X
1.30X
1.47X
1.82X
1.18X
1.35X
2.35X
34.58s
1.24X
1.55X
1.34X
1.89X
2.49X
1.30X
1.91X
16.09X
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
mi
d
tu
/
t
a
C
yo
/
yo
a
r
t
i
C
mi
–
pag
d
F
/
d
oh
i
/
.
1
0
1
1
6
2
/
t
yo
a
C
_
a
_
0
0
2
8
9
1
9
2
3
6
5
6
/
/
t
yo
a
C
_
a
_
0
0
2
8
9
pag
d
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Mesa 4: Statistics on the computation efficiency of different models. For CONT, we report the actual
training time in seconds. For other models, we report the relative training time compared to CONT.
Params: Number of trainable parameters of the whole model in millions. Batch: Maximal batch size per
card.
• S2 (large batch): We use one GPU card and
the maximal batch size. The time needed to
finish training on one million words for each
model is reported.
• S3 (multiple GPUs): Usamos 4 GPU cards
and the maximal batch size. The time needed
to finish training on one million words for
each model is reported.
are quite slow and that undermines the advantage
of the continuous output layer. For ELMO-Sub,
the small yet non-negligible softmax layer adds
overhead to the arithmetic complexity. FIXED,
ADAPTIVE, and SAMPLED have similar arithmetic
complexity but ADAPTIVE has the highest com-
plexity when the vocabulary size is large (p.ej.,
2,000k).
En mesa 4, we report the training speed of
the models under each scenario.7 In addition, nosotros
report the parameter size and the maximal batch
size on one GPU card. For ADAPTIVE and SAMPLED,
the vocabulary size also affects the training speed
so we test them under three different vocabulary
sizes:8 40k, 800k, and 2,000K.
Arithmetic complexity The arithmetic com-
plexity of the models is reflected by the speed
under S1, where the GPU memory is always
abundant and the arithmetic complexity is the
dominating factor. CONT holds a mild advan-
llevar (1.07x-1.25x) over baseline models, cual
is expected because the LSTM layers in ELMo
7CONT under S3 is slightly slower than the ELMO-C model
reported in Section 4.2. This is because when training the
ELMO-C model reported in 4.2, we actually train a forward
ELMO-C on two cards and train a backward ELMO-C on
two other cards, which reduces the communication cost by
half. This optimization is only applicable to our approach in
the setting of ELMo and does not work for other baseline
methods. In this experiment, we disable this optimization for
generosity.
8The 2,000K vocabulary is created on the tokenized 250-
billion-word Common Crawl corpus (Panchenko et al., 2017),
which covers words that appear more than 397 veces.
GPU memory consumption The effect of
GPU memory consumption can be observed by
comparing the statistics under S1 and S2. El
difference between S2 and S1 is that the parallel
computing of the GPU is fully utilized. Para
CONT, its great GPU memory efficiency helps
it gain larger speedup under S2, especially against
common baselines such as SUBWORD, ADAPTIVE,
and SAMPLED. For ELMO-Sub, in addition to the
overhead from the softmax layer, breaking words
into subwords leads to longer sequences, cual
increases the training time by 1.1x. Thus it is
1.53x slower than CONT under S2. SAMPLED suffers
from its huge parameter size and exhibits poor
scalability with respect to the vocabulary size
(2.35x slower when the vocabulary size reaches
2,000k).
Communication cost The effect of the com-
munication cost across GPUs can be observed
by comparing the statistics under S2 and S3.
As the communication cost and GPU memory
consumption both are highly dependent on the
parameter size, the observations are similar.
620
LSTM LSTMX2 TRANS BASE ELMO TRANS LARGE
GPT
CONT
FIXED
SUBWORD
ADAPTIVE
SAMPLED
3.97s
1.93X
2.32X
4.58X
2.50X
10.42s
1.32X
1.49X
2.20X
1.60X
15.87s
1.52X
1.78X
2.62X
2.91X
34.58s
1.24X
1.55X
1.89X
1.91X
48.55s
1.37X
1.72X
3.28X
OOM
43.53s
1.14X
1.44X
2.33X
8.31X
Mesa 5: Time needed to finish training on one million words for each model using
4 GPU cards and the maximal batch size. For CONT, we report the actual training
time in seconds. For other models, we report the relative training time compared to
CONT. OOM means that the GPU memory is not sufficient. CONT shows substantial
speedup over common baselines under all scenarios.
5.2.2 The Continuous Output Layer with
Different Sequence Encoders
For this experiment, we pair the output layers
with different sequence encoders and investigate
their training time. We start from a single-layer
LSTM with a hidden size of 2048 (LSTM) y
a two-layer version (LSTMX2), both reported in
Grave et al. (2016). They are all smaller than the
sequence encoder used in ELMO. We then scale
up to the forward and backward Transformer
reported in Peters et al. (2018b) (TRANS BASE)
and the multi-layer LSTM with projection in
ELMO(ELMO). Finalmente, we test two larger Trans-
anterior, TRANS LARGE, a scaled-up version of TRANS
BASE, and a uni-directional Transformer (denotado
as GPT) with the same size as BERTBASE (Devlin
et al., 2019) and GPT (Radford et al., 2018),
respectivamente. For all models but GPT, the lengths
of the input sequences are fixed at 20. For GPT,
we use input sequences of length 512, following
its original setting. For ADAPTIVE and SAMPLED, nosotros
fix the vocabulary size at 800K.
We report the training time of each model
using four GPU cards and the maximal batch
tamaño (S3) en mesa 5. We find that the continuous
output layer remains attractive, even when the
sequence encoder is as large as GPT. In that case,
the speedup of CONT over SUBWORD, ADAPTIVE,
and SAMPLED is still substantial (1.44X – 8.31X). En
addition, we observe that for sequence encoders
of the same type, more complex they get, menos
speedup CONT enjoys, which is expected. Para
instancia, from LSTM to LSTMX2, the speedup of
CONT decreases noticeably. Sin embargo, the speedup
the continuous output brings also depends on
the architecture of the sequence encoder. Para
instancia, though TRANS BASE and TRANS LARGE are
more complex than LSTMX2, CONT enjoys larger
speedup with those transformers. Profiling the
training process of sequence decoders such as
LSTM and the Transformer on GPU devices is an
interesting research topic but out of the scope of
este estudio.
6 Conclusión
We introduced an efficient framework to learn
the softmax
contextual representation without
capa. The experiments with ELMo showed that
we significantly accelerate the training of the
current models while maintaining competitive
performance on various downstream tasks.
Expresiones de gratitud
We wish to thank the anonymous reviewers,
the editor, Mark Yatskar, Muhao Chen, Xianda
zhou, and members at UCLANLP lab for helpful
comments. We also thank Yulia Tsvetkov and
Sachin Kumar for help with implementing the
continuous output layer as well as Jieyu Zhao,
Kenton Lee, and Nelson Liu for providing re-
producible source code for experiments. este trabajo
was supported by National Science Foundation
grant IIS-1760523 and IIS-1901527.
Referencias
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E
Hinton. 2016. Layer normalization. arXiv preprint
arXiv:1607.06450.
Yoshua Bengio and Jean-S´ebastien Sen´ecal. 2003.
Quick training of probabilistic neural nets by
importance sampling. In AISTATS.
621
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
mi
d
tu
/
t
a
C
yo
/
yo
a
r
t
i
C
mi
–
pag
d
F
/
d
oh
i
/
.
1
0
1
1
6
2
/
t
yo
a
C
_
a
_
0
0
2
8
9
1
9
2
3
6
5
6
/
/
t
yo
a
C
_
a
_
0
0
2
8
9
pag
d
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Samuel R. Bowman, Gabor Angeli, Christopher
Potts, and Christopher D. Manning. 2015. A
large annotated corpus for learning natural
language inference. In EMNLP.
James Bradbury, Stephen Merity, Caiming Xiong,
and Richard Socher. 2016. Quasi-recurrent neu-
ral networks. arXiv preimpresión arXiv:1611.01576.
Ciprian Chelba, Tomas Mikolov, Mike Schuster,
Qi Ge, Thorsten Brants, Phillipp Koehn, y
Tony Robinson. 2013. One billion word bench-
mark for measuring progress in statistical lan-
guage modeling. arXiv preimpresión arXiv:1312.3005.
Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin
Johnson, Wolfgang Macherey, George Foster,
Leon Jones, Mike Schuster, Noam Shazeer,
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit,
Lukasz Kaiser, Zhifeng Chen, Yonghui Wu,
and Macduff Hughes. 2018. The best of both
worlds: Combining recent advances in neural
machine translation. In ACL.
Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei,
Hui Jiang, and Diana Inkpen. 2017. Enhanced
LSTM for natural language inference. In ACL.
Ronan Collobert, Jason Weston, L´eon Bottou,
Michael Karlen, Koray Kavukcuoglu, and Pavel
PAG. Kuksa. 2011. Natural language processing
(almost) from scratch. Journal of Machine Learn-
ing Research, 12:2493–2537.
Yann N. Dauphin, Angela Fan, Michael Auli, y
David Grangier. 2017. Language modeling with
gated convolutional networks. In ICML.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, y
Kristina Toutanova. 2019. BERT: Pre-entrenamiento
de transformadores bidireccionales profundos para el lenguaje
comprensión. In NAACL-NLT.
John Duchi, Elad Hazan, and Yoram Singer.
2011. Adaptive subgradient methods for online
learning and stochastic optimization. Diario de
Machine Learning Research, 12:2121–2159.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind
Tafjord, Pradeep Dasigi, Nelson F. Liu,
Matthew E. Peters, Michael Schmitz, y lucas
Zettlemoyer. 2018. AllenNLP: A deep semantic
language processing platform. arXiv
natural
preprint arXiv:1803.07640.
Priya Goyal, Piotr Doll´ar, Ross Girshick, Pieter
Noordhuis, Lukasz Wesolowski, Aapo Kyrola,
Andrew Tulloch, Yangqing Jia, and Kaiming
Él. 2017. Accurate,
large minibatch SGD:
training Imagenet in 1 hour. arXiv preprint
arXiv:1706.02677.
Edouard Grave, Armand Joulin, Moustapha Ciss´e,
David Grangier, and Herv´e J´egou. 2016. Effi-
cient softmax approximation for GPUs. arXiv
preprint arXiv:1609.04309.
Luheng He, Kenton Lee, mike lewis, y lucas
Zettlemoyer. 2017. Deep semantic role label-
En g: What works and what’s next. In ACL.
Shexia He, Zuchao Li, Hai Zhao, and Hongxiao
Bai. 2018. Syntax for semantic role labeling, a
ser, or not to be. In ACL.
Sepp Hochreiter y Jürgen Schmidhuber. 1997.
Memoria larga a corto plazo. Computación neuronal,
9:1735–1780.
Yacine Jernite, Samuel R Bowman, and David
Sontag. 2017. Discourse-based objectives for
fast unsupervised sentence representation learn-
En g. arXiv preimpresión arXiv:1705.00557.
Ian Jolliffe. 2011. Principal component analysis. En
International Encyclopedia of Statistical Science,
Springer Berlin Heidelberg.
Rafal Jozefowicz, Oriol Vinyals, Mike Schuster,
Noam Shazeer, and Yonghui Wu. 2016. Explor-
ing the limits of language modeling. arXiv pre-
print arXiv:1602.02410.
Yoon Kim, Yacine Jernite, David Sontag, y
Alejandro M.. Rush. 2016. Character-aware
neural language models. En AAAI.
Ryan Kiros, Yukun Zhu, Ruslan R. Salakhutdinov,
Richard Zemel, Raquel Urtasun, Antonio Torralba,
and Sanja Fidler. 2015. Skip-thought vectors.
In NIPS.
Nikita Kitaev and Dan Klein. 2018. Plurilingüe
constituency parsing with self-attention and
pre-training. arXiv preimpresión arXiv:1812.11760.
Sachin Kumar and Yulia Tsvetkov. 2019. Von
Mises-Fisher loss for training sequence to
sequence models with continuous outputs. En
ICLR.
622
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
mi
d
tu
/
t
a
C
yo
/
yo
a
r
t
i
C
mi
–
pag
d
F
/
d
oh
i
/
.
1
0
1
1
6
2
/
t
yo
a
C
_
a
_
0
0
2
8
9
1
9
2
3
6
5
6
/
/
t
yo
a
C
_
a
_
0
0
2
8
9
pag
d
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Kenton Lee, Luheng He, mike lewis, y lucas
Zettlemoyer. 2017. End-to-end neural coref-
erence resolution. In EMNLP.
Kenton Lee, Luheng He, and Luke Zettlemoyer.
2018. Higher-order coreference resolution with
coarse-to-fine inference. In NAACL-HLT.
and Luke Zettlemoyer. 2018a. Deep contex-
tualized word representations. In NAACL-HLT.
Matthew E. Peters, Mark Neumann, Luke
Zettlemoyer, and Wen-tau Yih. 2018b. Dissect-
ing contextual word embeddings: Arquitectura
and representation. In EMNLP.
Tao Lei, Yu Zhang, Sida I. Wang, Hui Dai, y
Yoav Artzi. 2018. Simple recurrent units for
highly parallelizable recurrence. In EMNLP.
Yuval Pinter, Robert Guthrie, and Jacob Eisenstein.
2017. Mimicking word embeddings using sub-
word RNNs. In EMNLP.
Omer Levy and Yoav Goldberg. 2014. Neural word
embedding as implicit matrix factorization. En
NIPS.
Lajanugen Logeswaran and Honglak Lee. 2018.
An efficient framework for learning sentence
representaciones. ICLR.
Bryan McCann,
James Bradbury, Caiming
xiong, and Richard Socher. 2017. Learned
in translation: Contextualized word vectors. En
NIPS.
Stephen Merity, Nitish Shirish Keskar, y
Richard Socher. 2018. An analysis of neural
language modeling at multiple scales. arXiv
preprint arXiv:1803.08240.
Tomas Mikolov, Kai Chen, Greg Corrado, y
Jeffrey Dean. 2013. Efficient estimation of word
representations in vector space. arXiv preprint
arXiv:1301.3781.
Tomas Mikolov, Edouard Grave, Piotr Bojanowski,
Christian Puhrsch, and Armand Joulin. 2017.
Advances in pre-training distributed word rep-
resentaciones. arXiv preimpresión arXiv:1712.09405.
A Mnih and YW Teh. 2012. A fast and simple
training neural probabilistic
algorithm for
language models. In ICML.
Frederic Morin and Yoshua Bengio. 2005. Hier-
archical probabilistic neural network language
modelo. In AISTATS.
Alexander Panchenko, Eugen Ruppert, Stefano
Faralli, Simone Paolo Ponzetto, and Chris
Biemann. 2017. Building a web-scale dependency-
parsed corpus from CommonCrawl. arXiv
preprint arXiv:1710.01779.
Matthew E. Peters, Mark Neumann, Mohit Iyyer,
Matt Gardner, Christopher Clark, Kenton Lee,
Sameer Pradhan, Alessandro Moschitti, Nianwen
Xue, Hwee Tou Ng, Anders Bj¨orkelund, Olga
Uryupina, Yuchen Zhang, and Zhi Zhong.
2013. Towards robust linguistic analysis using
OntoNotes. In CoNLL.
Sameer Pradhan, Alessandro Moschitti, Nianwen
Xue, Olga Uryupina, and Yuchen Zhang. 2012.
CoNLL-2012 shared task: Modeling multi-
lingual unrestricted coreference in ontonotes.
In Joint Conference on EMNLP and CoNLL-
Tarea compartida.
Alec Radford, Karthik Narasimhan, Tim Salimans,
and Ilya Sutskever. 2018. Improving language
understanding by generative pre-training. OpenAI
Blog.
Alec Radford, Jeffrey Wu, niño rewon, David
Luan, Dario Amodei, and Ilya Sutskever. 2019.
Language models are unsupervised multitask
learners. OpenAI Blog.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev,
y Percy Liang. 2016. Equipo: 100, 000+
questions for machine comprehension of text.
In EMNLP.
Erik F Sang and Fien De Meulder. 2003. Introduc-
tion to the CoNLL-2003 shared task: Idioma-
independent named entity recognition. arXiv
preprint cs/0306050.
Rico Sennrich, Barry Haddow, and Alexandra
Birch. 2016. Neural machine translation of rare
words with subword units. In ACL.
Richard Socher, Alex Perelygin, Jean Wu, Jason
Chuang, Cristóbal D.. Manning, Andrew Y.
Ng, and Christopher Potts. 2013. Recursive
deep models for semantic compositionality over
a sentiment treebank. In EMNLP.
623
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
mi
d
tu
/
t
a
C
yo
/
yo
a
r
t
i
C
mi
–
pag
d
F
/
d
oh
i
/
.
1
0
1
1
6
2
/
t
yo
a
C
_
a
_
0
0
2
8
9
1
9
2
3
6
5
6
/
/
t
yo
a
C
_
a
_
0
0
2
8
9
pag
d
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Emma Strubell, Ananya Ganesh, y andres
McCallum. 2019. Energy and policy consid-
erations for deep learning in NLP. arXiv
preprint arXiv:1906.02243.
Shuai Tang, Hailin Jin, Chen Fang, Zhaowen Wang,
and Virginia de Sa. 2018. Speeding up context-
based sentence representation learning with
non-autoregressive convolutional decoding. En
Workshop on Representation Learning for NLP.
Ashish Vaswani, Noam Shazeer, Niki Parmar,
Jakob Uszkoreit, Leon Jones, Aidan N.. Gómez,
Lukasz Kaiser, y Illia Polosukhin. 2017.
Attention is all you need. In NIPS.
Kaiser, Stephan Gows, Yoshikiyo Kato, Taku
Kudo, Hideto Kazawa, Keith Stevens, Jorge
Kurian, Nishant Patil, Wei Wang, Cliff Young,
Jason Smith, Jason Riesa, Alex Rudnick, Oriol
Viñales, Gregory S. Corrado, Macduff Hughes,
and James A. Dean. 2016. Google’s neural
machine translation system: Bridging the gap
between human and machine translation. arXiv
preprint arXiv:1609.08144.
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov,
and William W. cohen. 2017. Breaking the soft-
max bottleneck: A high-rank RNN language
modelo. arXiv preimpresión arXiv:1711.03953.
Yonghui Wu, Mike Schuster, Zhifeng Chen,
Quoc V. Le, Mohammad Norouzi, Wolfgang
Macherey, Maxim Krikun, Yuan Cao, Qin
gao, Klaus Macherey, Jeff Klingner, Apurva
Shah, Melvin Johnson, Xiaobing Liu, lucas
Yang You, Zhao Zhang, Cho-Jui Hsieh, James
Demmel, and Kurt Keutzer. 2018. ImageNet
training in minutes. In Proceedings of the 47th
International Conference on Parallel Process-
En g, ICPP 2018.
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
mi
d
tu
/
t
a
C
yo
/
yo
a
r
t
i
C
mi
–
pag
d
F
/
d
oh
i
/
.
1
0
1
1
6
2
/
t
yo
a
C
_
a
_
0
0
2
8
9
1
9
2
3
6
5
6
/
/
t
yo
a
C
_
a
_
0
0
2
8
9
pag
d
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
8
S
mi
pag
mi
metro
b
mi
r
2
0
2
3