BeerQA

What is BeerQA?

BeerQA is an open-domain question answering dataset that features questions requiring information from one or more Wikipedia documents to answer, which presents a more realistic challenge for open-domain question answering. BeerQA is constructed based on the Stanford Question Answering Dataset (SQuAD) and the HotpotQA dataset, by a team of NLP researchers at JD AI Research, Samsung Research, and Stanford University.

For more details about BeerQA, please refer to our EMNLP 2021 paper:

(Qi, Lee, Sido, & Manning 2021)

Getting started

BeerQA is distributed under a CC BY-SA 4.0 License. The training and development sets can be downloaded below.

Training set (153MB)

Dev set (16MB)

Test set questions (2.1MB)

A more comprehensive summary about data download, preprocessing, baseline model training, and evaluation is included in our GitHub repository, and linked below.

Getting started guide

Once you have built your model, you can use the evaluation script we provide below to evaluate model performance by running python eval_beerqa.py <path_to_gold> <path_to_prediction>

Evaluation script (4.8KB)

Sample dev prediction (878KB)

To submit your models and evaluate them on the official test sets, please read our submission guide hosted on Codalab.

Submission Guide

We also release the processed Wikipedia used in the process of creating BeerQA (also under a CC BY-SA 4.0 License), serving both as the corpus for the fullwiki setting in our evaluation, and hopefully as a standalone resource for future researches involving processed text on Wikipedia. Below please find the link to the documentation for this corpus.

Processed Wikipedia README

Reference

If you use BeerQA in your research, please cite our paper with the following BibTeX entry

@inproceedings{qi2021answering,
  title={Answering Open-Domain Questions of Varying Reasoning Steps from Text},
  author = {Qi, Peng and Lee, Haejun and Sido, Oghenetegiri "TG" and Manning, Christopher D.},
  booktitle = {Empirical Methods for Natural Language Processing ({EMNLP})},
  year = {2021}
}