02/02/2021

On the Softmax Bottleneck of Recurrent Language Models

Dwarak Govind Parthiban, Yongyi Mao, Diana Inkpen

Keywords:

Abstract Paper Similar Papers

Abstract: Recent research has pointed to a limitation of word-level neural language models with softmax outputs. This limitation, known as the softmax bottleneck refers to the inability of these models to produce high-rank log probability (log P) matrices. Various solutions have been proposed to break this bottleneck, including Mixture of Softmaxes, SigSoftmax, and Linear Monotonic Softmax with Piecewise Linear Increasing Functions. They were reported to offer better performance in terms of perplexity on test data. A natural perception from these results is a strong positive correlation between the rank of the log P matrix and the model's performance. In this work, we show via an extensive empirical study that such a correlation is fairly weak and that the high-rank of the log P matrix is neither necessary nor sufficient for better test perplexity. Although our results are empirical, they are established in part via the construction of a rich family of models, which we call Generalized SigSoftmax. They are able to create diverse ranks for the log P matrices. We also present an investigation as to why the proposed solutions achieve better performance.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38949026
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at AAAI 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 16:00 
 0
 0
 0
 0
 5:34 
 0
 0
 0
 0
 15:02 
 0
 0
 0
 0
 3:23 
 0
 0
 0
 0
 11:05 
 0
 0
 0
 0
 14:30 
 0
 0
 0
 0
 12:50 
 0
 0
 0
 0
 10:58 
 0
 0
 0
 0
 7:48 
 0
 0
 0
 0
 13:22 
 0
 0
 0
 1
 14:12 
 0
 0
 0
 0
 9:37 
 0
 0
 0
 1
 14:29 
 0
 0
 0
 0
 10:35 
 0
 0
 0
 0
 3:07 
 0
 0
 0
 1
 3:14 
 0
 0
 1
 1
 14:35 
 0
 0
 0
 0
 5:01 
 0
 0
 0
 0
 14:00 
 0
 0
 0
 0
 4:46 
 0
 0
 0
 0
 14:48 
 0
 0
 0
 0
 14:15 
 0
 0
 0
 0
 5:13 
 0
 0
 0
 0
 3:19 
 0
 0
 0
 0
 5:25 
 0
 0
 0
 0
 13:45 
 0
 0
 0
 0
 13:02 
 0
 0
 0
 0
 17:13 
 0
 0
 0
 0
 11:15 
 0
 0
 0
 0
 10:57 
 0
 0
 0
 0
 2:48 
 0
 0
 0
 1
 5:22 
 0
 0
 0
 0
 14:05 
 0
 0
 0
 0
 3:29 
 0
 0
 0
 0
 15:24 
 0
 0
 0
 0
 12:05 
 0
 0
 0
 0
 1:00 
 0
 0
 0
 0
 3:20 
 0
 0
 0
 0
 2:58 
 0
 0
 0
 0
 12:40