02/02/2021

Encoding Syntactic Knowledge in Transformer Encoder for Intent Detection and Slot Filling

Jixuan Wang, Kai Wei, Martin Radfar, Weiwei Zhang, Clement Chung

Keywords:

Abstract Paper Similar Papers

Abstract: We propose a novel Transformer encoder-based architecture with syntactical knowledge encoded for intent detection and slot filling. Specifically, we encode syntactic knowledge into the Transformer encoder by jointly training it to predict syntactic parse ancestors and part-of-speech of each token via multi-task learning. Our model is based on self-attention and feed-forward layers and does not require external syntactic information to be available at inference time. Experiments show that on two benchmark datasets, our models with only two Transformer encoder layers achieve state-of-the-art results. Compared to the previously best performed model without pre-training, our models achieve absolute F1 score and accuracy improvement of 1.59 % and 0.85 % for slot filling and intent detection on the SNIPS dataset, respectively. Our models also achieve absolute F1 score and accuracy improvement of 0.1 % and 0.34 % for slot filling and intent detection on the ATIS dataset, respectively, over the previously best performed model. Furthermore, the visualization of the self-attention weights illustrates the benefits of incorporating syntactic information during training.

The video of this talk cannot be embedded. You can watch it here:
https://slideslive.com/38949019
(Link will open in new window)
 0
 0
 0
 0
  Share    
This is an embedded video. Talk and the respective paper are published at AAAI 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment
no comments yet
code of conduct: tbd Characters remaining: 140

Similar Papers

 0
 0
 0
 0
 3:17 
 0
 0
 0
 0
 12:42 
 0
 0
 0
 0
 2:59 
 0
 0
 0
 0
 16:08 
 0
 0
 0
 0
 11:55 
 0
 0
 0
 0
 5:13 
 0
 0
 0
 0
 11:47 
 0
 0
 0
 0
 16:00 
 0
 0
 0
 0
 11:21 
 0
 0
 0
 0
 3:23 
 0
 0
 0
 0
 1:00 
 0
 0
 0
 0
 10:16 
 0
 0
 0
 0
 15:16 
 0
 0
 0
 0
 18:43 
 0
 0
 0
 0
 7:54 
 0
 0
 0
 0
 10:32 
 0
 0
 0
 0
 11:54 
 0
 0
 0
 0
 12:07 
 0
 0
 0
 0
 15:25 
 0
 0
 0
 0
 10:01 
 0
 0
 0
 0
 11:09 
 0
 0
 0
 0
 6:59 
 0
 0
 0
 0
 11:44 
 0
 0
 0
 0
 6:06 
 0
 0
 0
 0
 10:50 
 0
 0
 0
 0
 14:39 
 0
 0
 0
 0
 9:40 
 0
 0
 0
 0
 11:49 
 0
 0
 0
 1
 10:52 
 0
 0
 0
 0
 5:00 
 0
 0
 0
 0
 11:27 
 0
 0
 0
 0
 14:05 
 0
 0
 0
 0
 12:15 
 0
 0
 0
 0
 14:52 
 0
 0
 0
 0
 19:06 
 0
 0
 0
 0
 9:23 
 0
 0
 0
 0
 13:35 
 0
 0
 0
 0
 13:16 
 0
 0
 0
 0
 9:31 
 0
 0
 0
 0
 10:32