Published on Fri Nov 01 2019

Forget Me Not: Reducing Catastrophic Forgetting for Domain Adaptation in Reading Comprehension

Y. Xu, X. Zhong, A. J. J. Yepes, J. H. Lau

The creation of large-scale open domain reading comprehension data sets has enabled the development of end-to-end neural comprehension models. To use these models for domains with limited training data, one of the most effective approach is to first pretrain them on large out-of-domain source data.

0
0
0
Abstract

The creation of large-scale open domain reading comprehension data sets in recent years has enabled the development of end-to-end neural comprehension models with promising results. To use these models for domains with limited training data, one of the most effective approach is to first pretrain them on large out-of-domain source data and then fine-tune them with the limited target data. The caveat of this is that after fine-tuning the comprehension models tend to perform poorly in the source domain, a phenomenon known as catastrophic forgetting. In this paper, we explore methods that overcome catastrophic forgetting during fine-tuning without assuming access to data from the source domain. We introduce new auxiliary penalty terms and observe the best performance when a combination of auxiliary penalty terms is used to regularise the fine-tuning process for adapting comprehension models. To test our methods, we develop and release 6 narrow domain data sets that could potentially be used as reading comprehension benchmarks.

Fri May 31 2019
Artificial Intelligence
MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension
A large number of reading comprehension (RC) datasets has been created recently. Little analysis has been done on whether they generalize to one another. We show that training on multiple RC datasets leads to robust generalization and transfer.
0
0
0
Sat Sep 28 2019
NLP
Integrated Triaging for Fast Reading Comprehension
Integrated Triaging is a framework that prunes almost all context in early layers of a network. This pruning increases the efficiency of MRC models and prevents the later layers from overfitting to prevalent short paragraphs in the training set.
0
0
0
Thu Mar 28 2019
NLP
Sogou Machine Reading Comprehension Toolkit
The Sogou Machine Reading Comprehension (SMRC) toolkit can be used to provide the fast and efficient development of modern machine learning models. The toolkit provides dataset readers, a flexible preprocessing pipeline, necessary neural network components, and built-in models.
0
0
0
Mon Nov 25 2019
NLP
Unsupervised Domain Adaptation of Language Models for Reading Comprehension
Reading comprehension (RC) is a task to learn the capability for question answering with textual sources. State-of-the-art models on RC still do not have general linguistic intelligence. We provide two domain adaptation models to solve the UDARC problem.
0
0
0
Sun Apr 21 2019
NLP
Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension
Machine reading comprehension tasks require a machine reader to answer questions relevant to the given document. We present the first free-form multiple-Choice Chinese machine reading Comprehension dataset (C^3)
0
0
0
Wed Oct 31 2018
NLP
Improving Machine Reading Comprehension with General Reading Strategies
Reading strategies have been shown to improve comprehension levels, especially for readers lacking adequate prior knowledge. Inspired by reading strategies identified in cognitive neuroscience, we propose three general strategies aimed to improve non-extractive machine reading comprehension (MRC) By fine-tuning a pre-trained language model (Radford et al., 2018) with our proposals, we obtain a 5.8% absolute increase in accuracy.
0
0
0