08/12/2020

Linguistic Profiling of a Neural Language Model

Alessio Miaschi, Dominique Brunato, Felice Dell’Orletta, Giulia Venturi

Keywords:

Abstract Paper Similar Papers

Abstract: In this paper we investigate the linguistic knowledge learned by a Neural Language Model (NLM) before and after a fine-tuning process and how this knowledge affects its predictions during several classification problems. We use a wide set of probing tasks, each of which corresponds to a distinct sentence-level feature extracted from different levels of linguistic annotation. We show that BERT is able to encode a wide range of linguistic characteristics, but it tends to lose this information when trained on specific downstream tasks. We also find that BERT’s capacity to encode different kind of linguistic properties has a positive influence on its predictions: the more it stores readable linguistic information of a sentence, the higher will be its capacity of predicting the expected label assigned to that sentence.

The video of this talk cannot be embedded. You can watch it here:
https://underline.io/lecture/6267-linguistic-profiling-of-a-neural-language-model
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at COLING 2020 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 7:02 
 0
 0
 0
 0
 11:18 
 0
 0
 0
 0
 11:06 
 0
 0
 0
 0
 12:39 
 0
 0
 0
 0
 12:23 
 0
 0
 0
 0
 11:43 
 0
 0
 0
 0
 14:33 
 0
 0
 0
 0
 4:42 
 0
 0
 0
 0
 4:52 
 0
 0
 0
 0
 13:59 
 0
 0
 0
 0
 13:59 
 0
 0
 0
 0
 6:56 
 0
 0
 0
 0
 11:57 
 0
 0
 0
 0
 2:48 
 0
 0
 0
 0
 4:47 
 0
 0
 0
 0
 10:15 
 0
 0
 0
 0
 11:05 
 0
 0
 0
 0
 4:45 
 0
 0
 0
 0
 11:50 
 0
 0
 0
 0
 11:00 
 0
 0
 0
 0
 16:44 
 0
 0
 0
 0
 12:35 
 0
 0
 0
 0
 12:45 
 0
 0
 0
 0
 11:44 
 0
 0
 0
 0
 11:06 
 0
 0
 0
 0
 10:53 
 0
 0
 0
 0
 3:34 
 0
 0
 0
 0
 10:30 
 0
 0
 0
 0
 12:00 
 0
 0
 0
 0
 4:49 
 0
 0
 0
 0
 12:17 
 0
 0
 0
 0
 6:38 
 0
 0
 0
 0
 13:17 
 0
 0
 0
 0
 14:52 
 0
 0
 0
 0
 9:52 
 0
 0
 0
 0
 44:12 
 0
 0
 0
 0
 14:10 
 0
 0
 0
 0
 5:21 
 0
 0
 0
 0
 15:16 
 0
 0
 0
 0
 12:53