{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2023,10,17]],"date-time":"2023-10-17T16:05:50Z","timestamp":1697558750394},"reference-count":31,"publisher":"MDPI AG","issue":"4","license":[{"start":{"date-parts":[[2023,4,11]],"date-time":"2023-04-11T00:00:00Z","timestamp":1681171200000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Entropy"],"abstract":"Natural-language processing is well positioned to help stakeholders study the dynamics of ambiguous Climate Change-related (CC) information. Recently, deep neural networks have achieved good results on a variety of NLP tasks depending on high-quality training data and complex and exquisite frameworks. This raises two dilemmas: (1) the networks are highly reliant on powerful hardware devices and processing is time-consuming, which is not only inconducive to execution on edge devices but also leads to resource consumption. (2) Obtaining large-scale effective annotated data is difficult and laborious, especially when it comes to a special domain such as CC. In this paper, we propose a CC-domain-adapted BERT distillation and reinforcement ensemble (DARE) model for tackling the problems above. Specifically, we propose a novel data-augmentation strategy which is a Generator-Reinforced Selector collaboration network for countering the dilemma of CC-related data scarcity. Extensive experimental results demonstrate that our proposed method outperforms baselines with a maximum of 26.83% on SoTA and 50.65\u00d7 inference time speed-up. Furthermore, as a remedy for the lack of CC-related analysis in the NLP community, we also provide some interpretable conclusions for this global concern.<\/jats:p>","DOI":"10.3390\/e25040643","type":"journal-article","created":{"date-parts":[[2023,4,12]],"date-time":"2023-04-12T06:57:08Z","timestamp":1681282628000},"page":"643","source":"Crossref","is-referenced-by-count":1,"title":["DARE: Distill and Reinforce Ensemble Neural Networks for Climate-Domain Processing"],"prefix":"10.3390","volume":"25","author":[{"ORCID":"http:\/\/orcid.org\/0009-0008-6012-2284","authenticated-orcid":false,"given":"Kun","family":"Xiang","sequence":"first","affiliation":[{"name":"Department of Science and Engineering, Hosei University, Tokyo 184-8584, Japan"}]},{"given":"Akihiro","family":"Fujii","sequence":"additional","affiliation":[{"name":"Department of Science and Engineering, Hosei University, Tokyo 184-8584, Japan"}]}],"member":"1968","published-online":{"date-parts":[[2023,4,11]]},"reference":[{"key":"ref_1","unstructured":"Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv."},{"key":"ref_2","doi-asserted-by":"crossref","unstructured":"Sun, S., Cheng, Y., Gan, Z., and Liu, J. (2019). Patient knowledge distillation for bert model compression. arXiv.","DOI":"10.18653\/v1\/D19-1441"},{"key":"ref_3","doi-asserted-by":"crossref","unstructured":"Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., and Liu, Q. (2019). Tinybert: Distilling bert for natural language understanding. arXiv.","DOI":"10.18653\/v1\/2020.findings-emnlp.372"},{"key":"ref_4","unstructured":"Tang, R., Lu, Y., Liu, L., Mou, L., Vechtomova, O., and Lin, J. (2019). Distilling task-specific knowledge from bert into simple neural networks. arXiv."},{"key":"ref_5","unstructured":"Sanh, V., Debut, L., Chaumond, J., and Wolf, T. (2019). Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter. arXiv."},{"key":"ref_6","first-page":"7350","article-title":"Knowledge distillation from internal representations","volume":"34","author":"Aguilar","year":"2020","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"ref_7","first-page":"7422","article-title":"Learning to augment for data-scarce domain bert knowledge distillation","volume":"35","author":"Feng","year":"2021","journal-title":"Proc. AAAI Conf. Artif. Intell."},{"key":"ref_8","doi-asserted-by":"crossref","unstructured":"Ma, X., Xu, P., Wang, Z., Nallapati, R., and Xiang, B. (2019, January 3). Domain adaptation with bert-based domain classification and data selection. Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), Hong Kong, China.","DOI":"10.18653\/v1\/D19-6109"},{"key":"ref_9","doi-asserted-by":"crossref","unstructured":"Du, C., Sun, H., Wang, J., Qi, Q., and Liao, J. (2020, January 5\u201310). Adversarial and domain-aware bert for cross-domain Sentiment analysis. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online.","DOI":"10.18653\/v1\/2020.acl-main.370"},{"key":"ref_10","unstructured":"Rietzler, A., Stabinger, S., Opitz, P., and Engl, S. (2019). Adapt or get left behind: Domain adaptation through bert language model finetuning for aspect-target sentiment classification. arXiv."},{"key":"ref_11","doi-asserted-by":"crossref","unstructured":"Zhuge, M., Gao, D., Fan, D.-P., Jin, L., Chen, B., Zhou, H., Qiu, M., and Shao, L. (2021, January 20\u201325). Kaleido-bert: Vision-language pre-training on fashion domain. Proceedings of the IEEE\/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA.","DOI":"10.1109\/CVPR46437.2021.01246"},{"key":"ref_12","doi-asserted-by":"crossref","unstructured":"Whang, T., Lee, D., Lee, C., Yang, K., Oh, D., and Lim, H. (2019). An effective domain adaptive post-training method for bert in response selection. arXiv.","DOI":"10.21437\/Interspeech.2020-2153"},{"key":"ref_13","doi-asserted-by":"crossref","unstructured":"Sung, C., Dhamecha, T., Saha, S., Ma, T., Reddy, V., and Arora, R. (2019, January 3\u20137). Pre-training bert on domain resources for short answer grading. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China.","DOI":"10.18653\/v1\/D19-1628"},{"key":"ref_14","doi-asserted-by":"crossref","first-page":"584","DOI":"10.1093\/jamia\/ocaa001","article-title":"Does bert need domain adaptation for clinical negation detection?","volume":"27","author":"Lin","year":"2020","journal-title":"J. Am. Med. Inform. Assoc."},{"key":"ref_15","doi-asserted-by":"crossref","unstructured":"Cody, E.M., Reagan, A.J., Mitchell, L., Dodds, P.S., and Danforth, C.M. (2015). Climate change sentiment on twitter: An unsolicited public opinion poll. PLoS ONE, 10.","DOI":"10.1371\/journal.pone.0136092"},{"key":"ref_16","doi-asserted-by":"crossref","unstructured":"Diakopoulos, N., Zhang, A.X., Elgesem, D., and Salway, A. (2014, January 1\u20134). Identifying and analyzing moral evaluation frames in climate change blog discourse. Proceedings of the Eighth International AAAI Conference on Weblogs and Social Media, Ann Arbor, MI, USA.","DOI":"10.1609\/icwsm.v8i1.14570"},{"key":"ref_17","first-page":"99","article-title":"Understanding social media\u2019s take on climate change through large-scale analysis of targeted opinions and emotions","volume":"38","author":"Pathak","year":"2017","journal-title":"AI Mag."},{"key":"ref_18","doi-asserted-by":"crossref","unstructured":"Salway, A., Elgesem, D., Hofland, K., Reigem, \u00d8., and Steskal, L. (2016, January 7\u201312). Topically-focused blog corpora for multiple languages. Proceedings of the 10th Web as Corpus Workshop, Berlin, Germany.","DOI":"10.18653\/v1\/W16-2603"},{"key":"ref_19","doi-asserted-by":"crossref","unstructured":"Mohammad, S., Kiritchenko, S., Sobhani, P., Zhu, X., and Cherry, C. (2016, January 16\u201317). Semeval-2016 task 6: Detecting stance in tweets. Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), San Diego, CA, USA.","DOI":"10.18653\/v1\/S16-1003"},{"key":"ref_20","doi-asserted-by":"crossref","unstructured":"Jiang, Y., Song, X., Harrison, J., Quegan, S., and Maynard, D. (2017, January 7). Comparing attitudes to climate change in the media using sentiment analysis based on latent dirichlet allocation. Proceedings of the 2017 EMNLP Workshop: Natural Language Processing Meets Journalism, Copenhagen, Denmark.","DOI":"10.18653\/v1\/W17-4205"},{"key":"ref_21","doi-asserted-by":"crossref","unstructured":"Luo, Y., Card, D., and Jurafsky, D. (2020). Detecting stance in media on global warming. arXiv.","DOI":"10.18653\/v1\/2020.findings-emnlp.296"},{"key":"ref_22","unstructured":"Koenecke, A., and Feliu-Faba, J. (2019). Learning twitter user sentiments on climate change with limited labeled data. arXiv."},{"key":"ref_23","unstructured":"K\u00f6lbel, J.F., Leippold, M., Rillaerts, J., and Wang, Q. (2020). Ask Bert: How Regulatory Disclosure of Transition and Physical Climate Risks Affects the Cds Term Structure, University of Zurich. Swiss Finance Institute Research Paper no. 21-19."},{"key":"ref_24","unstructured":"Luccioni, A., Baylor, E., and Duchene, N. (2020). Analyzing sustainability reports using natural language processing. arXiv."},{"key":"ref_25","doi-asserted-by":"crossref","unstructured":"Webersinke, N., Kraus, M., Bingler, J.A., and Leippold, M. (2021). Climatebert: A pretrained language model for climate-related text. arXiv.","DOI":"10.2139\/ssrn.4229146"},{"key":"ref_26","doi-asserted-by":"crossref","first-page":"1735","DOI":"10.1162\/neco.1997.9.8.1735","article-title":"Long short-term memory","volume":"9","author":"Hochreiter","year":"1997","journal-title":"Neural Comput."},{"key":"ref_27","doi-asserted-by":"crossref","first-page":"1","DOI":"10.1145\/3544558","article-title":"A survey on data augmentation for text classification","volume":"55","author":"Bayer","year":"2021","journal-title":"ACM Comput. Surv."},{"key":"ref_28","doi-asserted-by":"crossref","first-page":"71","DOI":"10.1016\/j.aiopen.2022.03.001","article-title":"Data augmentation approaches in natural language processing: A survey","volume":"3","author":"Li","year":"2022","journal-title":"AI Open"},{"key":"ref_29","doi-asserted-by":"crossref","first-page":"e514","DOI":"10.1016\/S2542-5196(21)00179-0","article-title":"Systematic mapping of global research on climate and health: A machine learning review","volume":"5","author":"Sietsma","year":"2021","journal-title":"Lancet Planet. Health"},{"key":"ref_30","doi-asserted-by":"crossref","first-page":"120","DOI":"10.1049\/cje.2018.11.004","article-title":"A text sentiment classification modeling method based on coordinated cnn-lstm-attention model","volume":"28","author":"Zhang","year":"2019","journal-title":"Chin. J. Electron."},{"key":"ref_31","unstructured":"Diggelmann, T., Boyd-Graber, J., Bulian, J., Ciaramita, M., and Leippold, M. (2020). Climate-fever: A dataset for verification of real-world climate claims. arXiv."}],"container-title":["Entropy"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/www.mdpi.com\/1099-4300\/25\/4\/643\/pdf","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2023,4,12]],"date-time":"2023-04-12T08:02:36Z","timestamp":1681286556000},"score":1,"resource":{"primary":{"URL":"https:\/\/www.mdpi.com\/1099-4300\/25\/4\/643"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2023,4,11]]},"references-count":31,"journal-issue":{"issue":"4","published-online":{"date-parts":[[2023,4]]}},"alternative-id":["e25040643"],"URL":"https:\/\/doi.org\/10.3390\/e25040643","relation":{},"ISSN":["1099-4300"],"issn-type":[{"value":"1099-4300","type":"electronic"}],"subject":[],"published":{"date-parts":[[2023,4,11]]}}}