08/12/2020

On the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT

Abhilasha Ravichander, Eduard Hovy, Kaheer Suleman, Adam Trischler, Jackie Chi Kit Cheung

Keywords:

Abstract Paper Similar Papers

Abstract: Contextualized word representations have become a driving force in NLP, motivating widespread interest in understanding their capabilities and the mechanisms by which they operate. Particularly intriguing is their ability to identify and encode conceptual abstractions. Past work has probed BERT representations for this competence, finding that BERT can correctly retrieve noun hypernyms in cloze tasks. In this work, we ask the question: <i>do probing studies shed light on systematic knowledge in BERT representations?</i> As a case study, we examine hypernymy knowledge encoded in BERT representations. In particular, we demonstrate through a simple consistency probe that the ability to correctly retrieve hypernyms in cloze tasks, as used in prior work, does not correspond to systematic knowledge in BERT. Our main conclusion is cautionary: even if BERT demonstrates high probing accuracy for a particular competence, it does not necessarily follow that BERT ‘understands’ a concept, and it cannot be expected to systematically generalize across applicable contexts.

The video of this talk cannot be embedded. You can watch it here:
https://underline.io/lecture/6663-on-the-systematicity-of-probing-contextualized-word-representations-the-case-of-hypernymy-in-bert
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at COLING Workshops 2020 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 7:02 
 0
 0
 0
 0
 13:59 
 0
 0
 0
 0
 9:52 
 0
 0
 0
 0
 10:15 
 0
 0
 0
 0
 13:17 
 0
 0
 0
 0
 12:39 
 0
 0
 0
 0
 9:11 
 0
 0
 0
 0
 11:57 
 0
 0
 0
 0
 4:42 
 0
 0
 0
 0
 9:38 
 0
 0
 0
 0
 18:55 
 0
 0
 0
 0
 11:06 
 0
 0
 0
 0
 11:50 
 0
 0
 0
 0
 12:23 
 0
 0
 0
 0
 6:14 
 0
 0
 0
 0
 6:38 
 0
 0
 0
 0
 2:48 
 0
 0
 0
 0
 6:59 
 0
 0
 0
 0
 7:46 
 0
 0
 0
 0
 6:56 
 0
 0
 0
 0
 11:09 
 0
 0
 0
 0
 14:06 
 0
 0
 0
 0
 4:47 
 0
 0
 0
 0
 13:01 
 0
 0
 0
 0
 10:51 
 0
 0
 0
 0
 15:05 
 0
 0
 0
 0
 14:33 
 0
 0
 0
 0
 14:47 
 0
 0
 0
 0
 11:44 
 0
 0
 0
 0
 10:01 
 0
 0
 0
 0
 4:55 
 0
 0
 0
 0
 11:43 
 0
 0
 0
 0
 19:08 
 0
 0
 0
 0
 12:45 
 0
 0
 0
 0
 12:22 
 0
 0
 0
 0
 14:09 
 0
 0
 0
 0
 13:34 
 0
 0
 0
 0
 15:15 
 0
 0
 0
 0
 10:45 
 0
 0
 0
 0
 11:36