Abstract
Text Generation aims to utilize contextual details to generate linguistically appropriate language. Research has demonstrated that integrating various linguistic features can significantly enhance the quality of text generation tasks. In light of this, this paper proposes an innovative approach Diversity Text Generation (DiversityGen)-and makes advancements in three aspects. Firstly, data augmentation techniques are employed to transform the original data, thereby enhancing the latent features of the text. Secondly, in the conversion of the model’s distributed vector output into text, a combination of Top-K and Beam Search decoding methods (Top-k-bs-m) is utilized. This extends the search space through random sampling during Beam Search decoding, thereby improving decoding performance and generating diversified text. Lastly, the concept of Over Generation (OGen) is introduced, wherein the results are filtered using three probability maximization-based methods to optimize output diversity. Experimental results demonstrate the effectiveness of this approach in question generation and text summarization tasks, surpassing current benchmarks.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Wenhao, Yu., et al.: A survey of knowledge-enhanced text generation. ACM Comput. Surv. 54(11s), 1–38 (2022)
Shao, Z., Huang, M., Wen, J., Xu, W., Zhu, X.: Long and diverse text generation with planning-based hierarchical variational model. arXiv preprint arXiv:1908.06605 (2019)
Heilman, M.: Automatic factual question generation from text. Ph.D. thesis, Carnegie Mellon University (2011)
El-Kassas, W.S., Salama, C.R., Rafea, A.A., Mohamed, H.K.: Automatic text summarization: a comprehensive survey. Expert Syst. Appl. 165, 113679 (2021)
Qin, Y., Wang, Z., Xi, D.: Tree cycleGAN with maximum diversity loss for image augmentation and its application into gear pitting detection. Appl. Soft Comput. 114, 108130 (2022)
Lu, X., et al.: Neurologic a* esque decoding: Constrained text generation with lookahead heuristics. arXiv preprint arXiv:2112.08726 (2021)
Chen, M., et al.: Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)
Chen, J., Tam, D., Raffel, C., Bansal, M., Yang, D.: An empirical survey of data augmentation for limited data learning in NLP. Trans. Assoc. Comput. Linguist. 11, 191–211 (2023)
Caccia, M., Caccia, L., Fedus, W., Larochelle, H., Pineau, J., Charlin, L.: Language GANs falling short. arXiv preprint arXiv:1811.02549 (2018)
Lachaux, M.-A., Joulin, A., Lample, G.: Target conditioning for one-to-many generation. arXiv preprint arXiv:2009.09758 (2020)
Shu, L., et al.: Controllable text generation with focused variation. arXiv preprint arXiv:2009.12046 (2020)
Cao, Y., Wan, X.: Divgan: Towards diverse paraphrase generation via diversified generative adversarial network. In: Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 2411–2421 (2020)
Zhang, Z., Zhu, K.: Diverse and specific clarification question generation with keywords. In: Proceedings of the Web Conference 2021, pp. 3501–3511 (2021)
Schlichtkrull, M.S., Cheng, W.: Evaluating for diversity in question generation over text. arXiv preprint arXiv:2008.07291 (2020)
Lee, D.B., Lee, S., Jeong, W.T., Kim, D., Hwang, S.J.: Generating diverse and consistent QA pairs from contexts with information-maximizing hierarchical conditional VAEs. arXiv preprint arXiv:2005.13837 (2020)
Jovanovic, M., Campbell, M.: Generative artificial intelligence: trends and prospects. Computer 55(10), 107–112 (2022)
Freitag, M., Al-Onaizan, Y.: Beam search strategies for neural machine translation. arXiv preprint arXiv:1702.01806 (2017)
Gao, Y., Zhang, Z., Hong, R., Zhang, H., Fan, J., Yan, S.: Towards feature distribution alignment and diversity enhancement for data-free quantization. In: 2022 IEEE International Conference on Data Mining (ICDM), pp. 141–150. IEEE (2022)
Harkous, H., Groves, I., Saffari, A.: Have your text and use it too! end-to-end neural data-to-text generation with semantic fidelity. arXiv preprint arXiv:2004.06577 (2020)
Radford, A., Jeffrey, W., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI blog 1(8), 9 (2019)
Holtzman, A., Buys, J., Du, L., Forbes, M., Choi, Y.: The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751 (2019)
Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21(1), 5485–5551 (2020)
Lewis, M., et al.: BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461 (2019)
Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 (2016)
Zheng, L., Guha, N., Anderson, B.R., Henderson, P., Ho, D.E.: When does pretraining help? Assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings. In: Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law, pp. 159–168 (2021)
Manor, L., Li, J.J.: Plain English summarization of contracts. In: roceedings of the Natural Legal Language Processing Workshop 2019, Minneapolis, Minnesota, June 2019, pp. 1–11. Association for Computational Linguistics (2019)
Qi, W., et al.: Prophetnet: predicting future n-gram for sequence-to-sequence pre-training. arXiv preprint arXiv:2001.04063 (2020)
Wu, Z., Jia, X., Qu, F., W, Y.: . Enhancing pre-trained models with text structure knowledge for question generation. arXiv preprint arXiv:2209.04179 (2022)
Back, S., Kedia, A., Chinthakindi, S.C., Lee, H., Choo, J.: Learning to generate questions by recovering answer-containing sentences (2020)
Chan, Y.-H., Fan, Y.-C.: A recurrent BERT-based model for question generation. In: Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pp. 154–162 (2019)
Acknowledgments
This research is supported by Sichuan Science and Technology Program (no. 2024NSFSC0520) and Humanities and Social Science Fund of Ministry of Education (no. 23YJA740013).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Xu, L., Chen, X., Wang, B., Jin, P. (2024). Exploring Language Diversity to Improve Neural Text Generation. In: Cao, C., Chen, H., Zhao, L., Arshad, J., Asyhari, T., Wang, Y. (eds) Knowledge Science, Engineering and Management. KSEM 2024. Lecture Notes in Computer Science(), vol 14888. Springer, Singapore. https://doi.org/10.1007/978-981-97-5489-2_22
Download citation
DOI: https://doi.org/10.1007/978-981-97-5489-2_22
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-97-5488-5
Online ISBN: 978-981-97-5489-2
eBook Packages: Computer ScienceComputer Science (R0)