Open-Domain Long-Form Question–Answering Using Transformer-Based Pipeline | SN Computer Science Skip to main content
Springer Nature Link
Log in

Open-Domain Long-Form Question–Answering Using Transformer-Based Pipeline

  • Original Research
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

For a long time, question–answering has been a crucial part of natural language processing (NLP). This task refers to fetching accurate and complete answers for a question using certain support documents or knowledge sources. In recent years, much work has been done in this field, especially after the introduction of transformer models. However, analysis reveals that the majority of research done in this domain mainly focuses on answering questions curated to have short answers, and fewer works focus on long-form question–answering (LFQA). LFQA systems generate explanatory answers for questions and pose more challenges than the short-form version. This paper investigates the long-form question–answering task by proposing a system in the form of a pipeline consisting of various transformer-based models, enabling the system to give explanatory answers to open-domain long-form questions. The pipeline mainly consists of a retriever module and a generator module. The retriever module retrieves the relevant support documents containing evidence to answer a question from a comprehensive knowledge source. On the other hand, the generator module generates the final answer using the relevant documents retrieved by the retriever module. The Explain Like I’m Five (ELI5) dataset is used to train and evaluate the system, and the final results are documented using proper metrics. The system is implemented in the Python programming language using the PyTorch framework. According to the evaluation, the proposed LFQA pipeline outperforms the existing research works when evaluated on the Knowledge-Intensive Language Tasks (KILT) benchmark and is thus effective in question–answering tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Data Availability

The Dataset and Knowledge Source used and analyzed in the current study, i.e., ELI5 [5] and KILT Wikipedia [6] respectively, are publicly available and can be used with proper citations.

References

  1. Chen W, Chang M-W, Schlinger E, Wang W, Cohen WW. Open question answering over tables and text. 2020. arXiv preprint arXiv:2010.10439.

  2. Krishna K, Roy A, Iyyer M, Hurdles to progress in long-form question answering. 2021. arXiv preprint arXiv:2103.06332.

  3. Nakano R, Hilton J, Balaji S, Wu J, Ouyang L, Kim C, Hesse C, Jain S, Kosaraju V, Saunders W et al. Webgpt: browser-assisted question-answering with human feedback. 2021. arXiv preprint arXiv:2112.09332.

  4. Bui M-Q, Tran V, Nguyen H-T, Le Nguyen M, How state-of-the-art models can deal with long-form question answering. In: Proceedings of the 34th Pacific Asia conference on language, information and computation, 2020; pp. 375–382.

  5. Fan A, Jernite Y, Perez E, Grangier D, Weston J, Auli M, Eli5: long form question answering. 2019. arXiv preprint arXiv:1907.09190.

  6. Petroni F, Piktus A, Fan A, Lewis P, Yazdani M, De Cao N, Thorne J, Jernite Y, Karpukhin V, Maillard J et al. Kilt: a benchmark for knowledge intensive language tasks. 2020. arXiv preprint arXiv:2009.02252.

  7. Zhang T, Kishore V, Wu F, Weinberger KQ, Artzi Y. Bertscore: evaluating text generation with bert. 2019. arXiv preprint arXiv:1904.09675.

  8. Bahri S, Sumpeno S, Nugroho SMS. An information retrieval approach to finding similar questions in question–answering of Indonesian government e-procurement services using tf* idf and lsi model. In: 2018 10th international conference on information technology and electrical engineering (ICITEE). New York: IEEE; 2018. p. 626–631 .

  9. Huang X, Zhang Y, Wei B, Yao L, A question–answering system over traditional Chinese medicine. In: 2015 IEEE international conference on bioinformatics and biomedicine (BIBM). New York: IEEE; 2015. p. 1737–1739.

  10. Mohapatra SK, Upadhyay A. Using TF-IDF on Kisan call centre dataset for obtaining query answers. In: 2018 international conference on communication, computing and internet of things (IC3IoT). New York: IEEE; 2018. p. 479–482.

  11. Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput. 1997;9(8):1735–80.

    Article  Google Scholar 

  12. Karimi E, Majidi B, Manzuri MT. Relevant question answering in community based networks using deep lSTM neural networks. In: 2019 7th Iranian joint congress on fuzzy and intelligent systems (CFIS). New York: IEEE; 2019. p. 1–5.

  13. Chen L, Zeng G, Zhang Q, Chen X, Wu D, Question answering over knowledgebase with attention-based lstm networks and knowledge embeddings. In: 2017 IEEE 16th International Conference on Cognitive Informatics & Cognitive Computing (ICCI* CC). New York: IEEE; 2017. p. 243–246.

  14. Luo D, Su J, Yu S, A bert-based approach with relation-aware attention for knowledge base question answering. In: 2020 International Joint Conference on Neural Networks (IJCNN). New York: IEEE; 2020. p. 1–8.

  15. Devlin J, Chang M-W, Lee K, Toutanova K. Bert: pre-training of deep bidirectional transformers for language understanding. 2018. arXiv preprint arXiv:1810.04805.

  16. Rajpurkar P, Zhang J, Lopyrev K, Liang P. Squad: 100,000+ questions for machine comprehension of text. 2016. arXiv preprint arXiv:1606.05250.

  17. Trischler A, Wang T, Yuan X, Harris J, Sordoni A, Bachman P, Suleman K, Newsqa: a machine comprehension dataset. 2016. arXiv preprint arXiv:1611.09830.

  18. Kočiskỳ T, Schwarz J, Blunsom P, Dyer C, Hermann KM, Melis G, Grefenstette E. The narrative QA reading comprehension challenge. Trans Assoc Comput Linguist. 2018;6:317–28.

    Article  Google Scholar 

  19. Reddy S, Chen D, Manning CD. COQA: a conversational question answering challenge. Trans Assoc Comput Linguist. 2019;7:249–66.

    Article  Google Scholar 

  20. Chen D, Fisch A, Weston J, Bordes A. Reading Wikipedia to answer open-domain questions. 2017. arXiv preprint arXiv:1704.00051.

  21. Tang T, Li J, Zhao WX, Wen J-R. MVP: multi-task supervised pre-training for natural language generation. 2022. arXiv preprint arXiv:2206.12131.

  22. Bhojanapalli S, Chakrabarti A, Veit A, Lukasik M, Jain H, Liu F, Chang Y-W, Kumar S. Leveraging redundancy in attention with reuse transformers. 2021. arXiv preprint arXiv:2110.06821.

  23. Riabi A, Scialom T, Keraron R, Sagot B, Seddah D, Staiano J. Synthetic data augmentation for zero-shot cross-lingual question answering. 2020. arXiv preprint arXiv:2010.12643.

  24. Huang Z, Liang D, Xu P, Xiang B. Improve transformer models with better relative position embeddings. 2020. arXiv preprint arXiv:2009.13658.

  25. Nitish S, Darsini R, Shashank G, Tejas V, Arya A. Bidirectional encoder representation from transformers (bert) variants for procedural long-form answer extraction. In: 2022 12th international conference on cloud computing, data science & engineering (confluence). New York: IEEE; 2022. p. 71–76.

  26. Butler R, Duggirala VD, Banaei-Kashani F. ILFQA: a platform for efficient and accurate long-form question answering. In: Proceedings of the fifteenth ACM international conference on web search and data mining; 2022. p. 1565–1568.

  27. Guu K, Lee K, Tung Z, Pasupat P, Chang M, Realm: retrieval-augmented language model pre-training. 2020. arXiv preprint arXiv:2002.08909.

  28. Karpukhin V, Oğuz B, Min S, Lewis P, Wu L, Edunov S, Chen D, Yih W-T. Dense passage retrieval for open-domain question answering. 2020. arXiv preprint arXiv:2004.04906.

  29. Lee K, Chang M-W, Toutanova K. Latent retrieval for weakly supervised open domain question answering. 2019. arXiv preprint arXiv:1906.00300.

  30. Lewis P, Perez E, Piktus A, Petroni F, Karpukhin V, Goyal N, Küttler H, Lewis M, Yih W-T, Rocktäschel T. Retrieval-augmented generation for knowledge-intensive NLP tasks. Adv Neural Inf Process Syst. 2020;33:9459–74.

    Google Scholar 

  31. Su D, Li X, Zhang J, Shang L, Jiang X, Liu Q, Fung P. Read before generate! Faithful long form question answering with machine reading. 2022. arXiv preprint arXiv:2203.00343.

  32. Johnson J, Douze M, Jégou H. Billion-scale similarity search with GPUS. IEEE Trans Big Data. 2019;7(3):535–47.

    Article  Google Scholar 

  33. Sachan DS, Lewis M, Joshi M, Aghajanyan A, Yih W-T, Pineau J, Zettlemoyer L. Improving passage retrieval with zero-shot question generation. 2022. arXiv preprint arXiv:2204.07496.

  34. Wolf T, Debut L, Sanh V, Chaumond J, Delangue C, Moi A, Cistac P, Rault T, Louf R, Funtowicz M. Transformers: state-of-the-art natural language processing. In: Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations; 2020. p. 38–45.

  35. Song K, Tan X, Qin T, Lu J, Liu T-Y. Mpnet: masked and permuted pre-training for language understanding. Adv Neural Inf Process Syst. 2020;33:16857–67.

    Google Scholar 

  36. Sanh V, Debut L, Chaumond J, Wolf T, Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. 2019. arXiv preprint arXiv:1910.01108.

  37. Sanh V, Webson A, Raffel C, Bach SH, Sutawika L, Alyafeai Z, Chaffin A, Stiegler A, Scao TL, Raja A et al. Multitask prompted training enables zero-shot task generalization. 2021. arXiv preprint arXiv:2110.08207.

  38. Lewis M, Liu Y, Goyal N, Ghazvininejad M, Mohamed A, Levy O, Stoyanov V, Zettlemoyer L, Bart: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. 2019. arXiv preprint arXiv:1910.13461.

  39. Raffel C, Shazeer N, Roberts A, Lee K, Narang S, Matena M, Zhou Y, Li W, Liu PJ. Exploring the limits of transfer learning with a unified text-to-text transformer. J Mach Learn Res. 2020;21(140):1–67.

    MathSciNet  MATH  Google Scholar 

  40. Fedus W, Zoph B, Shazeer N. Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. 2021.

Download references

Author information

Authors and Affiliations

  1. Department of Information Technology, National Institute of Technology Karnataka, Surathkal, Mangalore, Karnataka, 575025, India

    Aprameya Dash, Mohit Awachar, Anshul Patel & Bhawana Rudra

Authors
  1. Aprameya Dash
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Mohit Awachar
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Anshul Patel
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Bhawana Rudra
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Bhawana Rudra.

Ethics declarations

Conflict of Interest

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article is part of the topical collection “Research Trends in Computational Intelligence” guest edited by Anshul Verma, Pradeepika Verma, Vivek Kumar Singh, and S. Karthikeyan.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dash, A., Awachar, M., Patel, A. et al. Open-Domain Long-Form Question–Answering Using Transformer-Based Pipeline. SN COMPUT. SCI. 4, 595 (2023). https://doi.org/10.1007/s42979-023-02039-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-023-02039-x

Keywords