Published on Sat Apr 15 2017

RACE: Large-scale ReAding Comprehension Dataset From Examinations

Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, Eduard Hovy

RACE consists of near 28,000 passages and near 100,000 questions generated by human experts. The proportion of questions that requires reasoning is much larger in RACE than in other benchmark datasets for reading comprehension. There is a significant gap between the performance of the state-of-the-

1
8
9
Abstract

We present RACE, a new dataset for benchmark evaluation of methods in the reading comprehension task. Collected from the English exams for middle and high school Chinese students in the age range between 12 to 18, RACE consists of near 28,000 passages and near 100,000 questions generated by human experts (English instructors), and covers a variety of topics which are carefully designed for evaluating the students' ability in understanding and reasoning. In particular, the proportion of questions that requires reasoning is much larger in RACE than that in other benchmark datasets for reading comprehension, and there is a significant gap between the performance of the state-of-the-art models (43%) and the ceiling human performance (95%). We hope this new dataset can serve as a valuable resource for research and evaluation in machine comprehension. The dataset is freely available at http://www.cs.cmu.edu/~glai1/data/race/ and the code is available at https://github.com/qizhex/RACE_AR_baselines.

Mon Aug 05 2019
NLP
Beyond English-Only Reading Comprehension: Experiments in Zero-Shot Multilingual Transfer for Bulgarian
Reading comprehension models achieved near-human performance on large-scale datasets such as SQuAD, CoQA, MS Macro, RACE, etc. This is largely due to the release of pre-trained contextualized representations such as BERT and ELMo, which can be fine-
0
0
0
Mon Sep 25 2017
NLP
Dataset for the First Evaluation on Chinese Machine Reading Comprehension
Machine Reading Comprehension (MRC) has become enormously popular recently. However, existing reading comprehension datasets are mostly in English. To add diversity in reading comprehension, we propose a new Chinese reading comprehension dataset.
0
0
0
Sun Apr 21 2019
NLP
Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension
Machine reading comprehension tasks require a machine reader to answer questions relevant to the given document. We present the first free-form multiple-Choice Chinese machine reading Comprehension dataset (C^3)
0
0
0
Thu Nov 21 2019
NLP
Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets
Existing analysis work in machine reading comprehension (MRC) is largely concerned with evaluating the capabilities of systems. We propose a semi-automated, ablation-based methodology for this challenge. We evaluate to what degree the questions do not require the required skill.
0
0
0
Fri Dec 20 2019
NLP
SberQuAD -- Russian Reading Comprehension Dataset: Description and Analysis
0
0
0
Tue Nov 14 2017
NLP
DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applications
This paper introduces DuReader, a new large-scale, open-domain Chinese reading comprehension (MRC) dataset. DuReader has three advantages over previous MRC datasets: (1) data sources, (2) question types, and (3) scale.
0
0
0