02/02/2021

Natural Language Inference in Context - Investigating Contextual Reasoning over Long Texts

Hanmeng Liu, Leyang Cui, Jian Liu, Yue Zhang

Keywords:

Abstract Paper Similar Papers

Abstract: Natural language inference (NLI) is a fundamental NLP task, investigating the entailment relationship between two texts. Popular NLI datasets present the task at sentence-level. While adequate for testing semantic representations, they fall short for testing contextual reasoning over long texts, which is a natural part of the human inference process. We introduce ConTRoL, a new dataset for ConTextual Reasoning over Long texts. Consisting of 8,325 expert-designed "context-hypothesis" pairs with gold labels, ConTRoL is a passage-level NLI dataset with a focus on complex contextual reasoning types such as logical reasoning. It is derived from competitive selection and recruitment test (verbal reasoning test) for police recruitment, with expert level quality. Compared with previous NLI benchmarks, the materials in ConTRoL are much more challenging, involving a range of reasoning types. Empirical results show that state-of-the-art language models perform by far worse than educated humans. Our dataset can also serve as a testing-set for downstream tasks like checking the factual correctness of summaries.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38948031
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at AAAI 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 18:56 
 0
 0
 0
 0
 19:57 
 0
 0
 0
 0
 16:40 
 0
 0
 0
 0
 7:31 
 0
 0
 0
 0
 14:09 
 0
 0
 0
 0
 9:48 
 0
 0
 0
 0
 4:58 
 0
 0
 0
 0
 11:19 
 0
 0
 0
 1
 12:13 
 0
 0
 0
 0
 11:31 
 0
 0
 0
 0
 12:16 
 0
 0
 0
 0
 4:38 
 0
 0
 0
 0
 10:30 
 0
 0
 0
 0
 12:39 
 0
 0
 0
 0
 14:56 
 0
 0
 0
 0
 11:13 
 0
 0
 0
 0
 4:57 
 0
 0
 0
 0
 4:11 
 0
 0
 0
 0
 13:17 
 0
 0
 0
 0
 7:48 
 0
 0
 0
 0
 8:54 
 0
 0
 0
 0
 19:17 
 0
 0
 0
 0
 7:28 
 0
 0
 0
 0
 16:13 
 0
 0
 0
 0
 11:06 
 0
 0
 0
 0
 13:17 
 0
 0
 0
 0
 12:23 
 0
 0
 0
 0
 11:36 
 0
 0
 0
 0
 12:01 
 0
 0
 0
 0
 11:42 
 0
 0
 0
 0
 11:43 
 0
 0
 0
 0
 11:26 
 0
 0
 0
 1
 10:41 
 0
 0
 0
 0
 11:17 
 0
 0
 0
 0
 9:33 
 0
 0
 0
 0
 11:59 
 0
 0
 0
 0
 14:42 
 0
 0
 0
 0
 15:01 
 0
 0
 0
 0
 6:58 
 0
 0
 0
 0
 17:52