02/02/2021

Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training

Peng Shi, Patrick Ng, Zhiguo Wang, Henghui Zhu, Alexander Hanbo Li, Jun Wang, Cicero Nogueira dos Santos, Bing Xiang

Keywords:

Abstract Paper Similar Papers

Abstract: Most recently, there has been significant interest in learning contextual representations for various NLP tasks, by leveraging large scale text corpora to train powerful language models with self-supervised learning objectives, such as Masked Language Model (MLM). Based on a pilot study, we observe three issues of existing general-purpose language models when they are applied in the text-to-SQL semantic parsers: fail to detect the column mentions in the utterances, to infer the column mentions from the cell values, and to compose target SQL queries when they are complex. To mitigate these issues, we present a model pretraining framework, Generation-Augmented Pre-training (GAP), that jointly learns representations of natural language utterance and table schemas, by leveraging generation models to generate high-quality pre-train data. GAP Model is trained on 2 million utterance-schema pairs and 30K utterance-schema-SQL triples, whose utterances are generated by generation models. Based on experimental results, neural semantic parsers that leverage GAP Model as a representation encoder obtain new state-of-the-art results on both Spider and Criteria-to-SQL benchmarks.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38948654
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at AAAI 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 5:11 
 0
 0
 0
 0
 3:28 
 0
 0
 0
 0
 14:44 
 0
 0
 0
 0
 10:51 
 0
 0
 0
 0
 5:13 
 0
 0
 0
 0
 11:48 
 0
 0
 0
 0
 11:42 
 0
 0
 0
 0
 16:04 
 0
 0
 0
 0
 6:48 
 0
 0
 0
 0
 11:57 
 0
 0
 0
 0
 12:00 
 0
 0
 0
 0
 8:33 
 0
 0
 0
 0
 8:45 
 0
 0
 0
 0
 10:59 
 0
 0
 0
 0
 5:15 
 0
 0
 0
 0
 11:33 
 0
 0
 0
 0
 18:01 
 0
 0
 0
 0
 12:49 
 0
 0
 0
 0
 4:45 
 0
 0
 0
 0
 11:48 
 0
 0
 0
 0
 3:22 
 0
 0
 0
 0
 15:13 
 0
 0
 0
 0
 11:42 
 0
 0
 0
 0
 3:14 
 0
 0
 0
 0
 11:38 
 0
 0
 0
 0
 11:03 
 0
 0
 0
 0
 14:40 
 0
 0
 0
 0
 11:47 
 0
 0
 0
 0
 12:12 
 0
 0
 0
 0
 11:04 
 0
 0
 0
 0
 10:37 
 0
 0
 0
 0
 14:42 
 0
 0
 0
 0
 10:25 
 0
 0
 0
 0
 11:34 
 0
 0
 0
 1
 10:41 
 0
 0
 0
 0
 14:16 
 0
 0
 0
 0
 13:55 
 0
 0
 0
 0
 15:39 
 0
 0
 0
 0
 11:51 
 0
 0
 0
 0
 9:42