Multi-source domain adaptation for dependency parsing via domain-aware feature generation | International Journal of Machine Learning and Cybernetics Skip to main content
Springer Nature Link
Log in

Multi-source domain adaptation for dependency parsing via domain-aware feature generation

  • Original Article
  • Published:
International Journal of Machine Learning and Cybernetics Aims and scope Submit manuscript

Abstract

With deep representation learning advances, supervised dependency parsing has achieved a notable enhancement. However, when the training data is drawn from various predefined out-domains, the parsing performance drops sharply due to the domain distribution shift. The key to addressing this problem is to model the associations and differences between multiple source and target domains. In this work, we propose an innovative domain-aware adversarial and parameter generation network for multi-source cross-domain dependency parsing where a domain-aware parameter generation network is used for identifying domain-specific features and an adversarial network is used for learning domain-invariant ones. Experiments on the benchmark datasets reveal that our model outperforms strong BERT-enhanced baselines by 2 points in the average labeled attachment score (LAS). Detailed analysis of various domain representation strategies shows that our proposed distributed domain embedding can accurately capture domain relevance, which motivates the domain-aware parameter generation network to emphasize useful domain-specific representations and disregard unnecessary or even harmful ones. Additionally, extensive comparison experiments show deeper insights on the contributions of the two components.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
¥17,985 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Japan)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Algorithm 1
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

Here are links to the datasets used in this article: http://hlt.suda.edu.cn/index.php/Nlpcc-2019-shared-task.

Notes

  1. http://hlt.suda.edu.cn/index.php/Nlpcc-2019-shared-task.

  2. https://github.com/google-research/bert.

References

  1. McDonald R.T, Nivre J, Quirmbach-Brundage Y, Goldberg Y, Das D, Ganchev K, Hall K.B, Petrov S, Zhang H, Täckström O, Bedini C, Castelló N.B, Lee J (2013) Universal dependency annotation for multilingual parsing. In: Proceedings of ACL, pp 92–97

  2. Park Y, Kang S (2019) Natural language generation using dependency tree decoding for spoken dialog systems. IEEE Access 7:7250–7258

    Article  Google Scholar 

  3. Xia Q, Li Z, Zhang M, Zhang M, Fu G, Wang R, Si L (2019) Syntax-aware neural semantic role labeling. In: Proceedings of AAAI, pp 7305–7313

  4. Peng R, Hao T, Fang Y (2021) Syntax-aware neural machine translation directed by syntactic dependency degree. Neural Comput Appl 33(23):16609–16625

    Article  Google Scholar 

  5. Chen D, Manning CD (2014) A fast and accurate dependency parser using neural networks. In: Proceedings of EMNLP, pp 740–750

  6. Kiperwasser E, Goldberg Y (2016) Simple and accurate dependency parsing using bidirectional LSTM feature representations. TACL 4:313–327

    Article  Google Scholar 

  7. Dozat T, Manning C (2017) Deep biaffine attention for neural dependency parsing. arXiv preprint. https://arxiv.org/abs/1611.01734

  8. Zhang X, Le Roux J, Charnois T (2021) Strength in numbers: Averaging and clustering effects in mixture of experts for graph-based dependency parsing. In: Proceedings of the 17th international conference on parsing technologies and the IWPT 2021 shared task on parsing into enhanced universal dependencies (IWPT 2021), pp 106–118

  9. Pupier A, Coavoux M, Lecouteux B, Goulian J (2022) End-to-end dependency parsing of spoken French. In: Proceedings of Interspeech 2022, pp 1816–1820

  10. Altıntaş M, Tantuğ AC (2023) Improving the performance of graph based dependency parsing by guiding bi-affine layer with augmented global and local features. Intell Syst Appl 18:200190

    Google Scholar 

  11. Al-Ghamdi S, Al-Khalifa H, Al-Salman A (2023) Fine-tuning bert-based pre-trained models for Arabic dependency parsing. Appl Sci 13(7):4225

    Article  Google Scholar 

  12. Koo T, Carreras X, Collins M (2008) Simple semi-supervised dependency parsing. In: Proceedings of ACL, pp 595–603

  13. Yu M, Zhao T, Bai Y (2013) Learning domain differences automatically for dependency parsing adaptation. In: Proceedings of IJCAI, pp 1876–1882

  14. Sato M, Manabe H, Noji H, Matsumoto Y (2017) Adversarial training for cross-domain universal dependency parsing. In: Proceedings of the CoNLL 2017 Shared Task, pp 71–79

  15. Clark K, Luong M, Manning CD, Le QV (2018) Semi-supervised sequence modeling with cross-view training. In: Proceedings of EMNLP, pp 1914–1925

  16. Li Y, Li Z, Zhang M (2020) Semi-supervised domain adaptation for dependency parsing via improved contextualized word representations. In: Proceedings of COLING, pp 3806–3817

  17. Peng X, Li Z, Zhang M, Wang R, Zhang Y, Si L (2019) Overview of the NLPCC 2019 shared task: Cross-domain dependency parsing. In: Proceedings of NLPCC, pp 760–771

  18. Daumé III H (2007) Frustratingly easy domain adaptation. In: Proceedings of ACL, pp 256–263

  19. Kim Y, Stratos K, Sarikaya R (2016) Frustratingly easy neural domain adaptation. In: Proceedings of COLING, pp 387–396

  20. Wang R, Utiyama M, Liu L, Chen K, Sumita E (2017) Instance weighting for neural machine translation domain adaptation. In: Proceedings of EMNLP, pp 1482–1488

  21. Kim Y, Stratos K, Kim D (2017) Domain attention with an ensemble of experts. In: Proceedings of ACL, pp 643–653

  22. Ganin Y, Lempitsky VS (2015) Unsupervised domain adaptation by backpropagation. In: Proceedings of ICML, pp 1180–1189

  23. Rozantsev A, Salzmann M, Fua P (2019) Beyond sharing weights for deep domain adaptation. IEEE Trans Pattern Anal Mach Intell 41(4):801–814

    Article  Google Scholar 

  24. Sun T, Shao Y, Li X, Liu P, Yan H, Qiu X, Huang X (2020) Learning sparse sharing architectures for multiple tasks. In: Proceedings of AAAI, pp 8936–8943

  25. Chakraborty T, La Gatta V, Moscato V, Sperlì G (2023) Information retrieval algorithms and neural ranking models to detect previously fact-checked information. Neurocomputing 557:126680

    Article  Google Scholar 

  26. Charniak E (1997) Statistical parsing with a context-free grammar and word statistics. In: Proceedings of AAAI, pp 598–603

  27. Steedman M, Sarkar A, Osborne M, Hwa R, Clark S, Hockenmaier J, Ruhlen P, Baker S, Crim J (2003) Bootstrapping statistical parsers from small datasets. In: Proceedings of EACL, pp 331–338

  28. Reichart R, Rappoport A (2007) Self-training for enhancement and domain adaptation of statistical parsers trained on small datasets. In: Proceedings of ACL, pp 616–623

  29. Yu J, Elkaref M, Bohnet B (2015) Domain adaptation for dependency parsing via self-training. In: Proceedings of IWPT, pp 1–10

  30. Sarkar A (2001) Applying co-training methods to statistical parsing. In: Proceedings of NAACL

  31. Li Z, Zhou J, Zhao H, Wang R (2019) Cross-domain transfer learning for dependency parsing. In: Proceedings of NLPCC, pp 835–844

  32. Sun K, Li Z, Zhao H (2023) Cross-lingual universal dependency parsing only from one monolingual treebank. IEEE Trans Pattern Anal Mach Intell 45(11):13393–13407

    Article  Google Scholar 

  33. Li Z, Peng X, Zhang M, Wang R, Si L (2019) Semi-supervised domain adaptation for dependency parsing. In: Proceedings of ACL, pp 2386–2395

  34. Zeng J, Su J, Wen H, Liu Y, Xie J, Yin Y, Zhao J (2018) Multi-domain neural machine translation with word-level domain context discrimination. In: Proceedings of EMNLP, pp 447–457

  35. Cai Y, Wan X (2019) Multi-domain sentiment classification based on domain-aware embedding and attention. In: Proceedings of IJCAI, pp 4904–4910

  36. Guo J, Shah D.J, Barzilay R (2018) Multi-source domain adaptation with mixture of experts. In: Proceedings of EMNLP, pp 4694–4703

  37. Li R, Wang X, Yu H (2020) Metamt, a meta learning method leveraging multiple domain data for low resource machine translation. In: Proceedings of AAAI, pp 8245–8252

  38. Wright D, Augenstein I (2020) Transformer based multi-source domain adaptation. In: Proceedings of EMNLP, pp 7963–7974

  39. Jia C, Liang X, Zhang Y (2019) Cross-domain NER using cross-domain language modeling. In: Proceedings of ACL, pp 2464–2474

  40. McClosky D, Charniak E, Johnson M (2010) Automatic domain adaptation for parsing. In: Proceedings of NAACL, pp 28–36

  41. Wagner J, Barry J, Foster J (2020) Treebank embedding vectors for out-of-domain dependency parsing. In: Proceedings of ACL, pp 8812–8818

  42. Lample G, Ballesteros M, Subramanian S, Kawakami K, Dyer C (2016) Neural architectures for named entity recognition. In: NAACL, pp 260–270

  43. McDonald RT, Pereira F, Ribarov K, Hajic J (2005) Non-projective dependency parsing using spanning tree algorithms. In: Proceedings of HLT-EMNLP, pp 523–530

  44. Li Y, Zhang M, Li Z, Zhang M, Wang Z, Huai B, Yuan NJ (2021) APGN: adversarial and parameter generation networks for multi-source cross-domain dependency parsing. In: Findings of the association for computational linguistics: EMNLP 2021, pp 1724–1733

  45. Ganin Y, Ustinova E, Ajakan H, Germain P, Larochelle H, Laviolette F, Marchand M, Lempitsky VS (2017) Domain-adversarial training of neural networks. In: Domain adaptation in computer vision applications. Advances in computer vision and pattern recognition, pp 189–209

  46. Hajic J, Ciaramita M, Johansson R, Kawahara D, Martí M.A, Màrquez L, Meyers A, Nivre J, Padó S, Stepánek J, Stranák P, Surdeanu M, Xue N, Zhang, Y (2009) The conll-2009 shared task: Syntactic and semantic dependencies in multiple languages. In: Hajic, J. (ed.) Proceedings of the thirteenth conference on computational natural language learning: shared task, CoNLL 2009, pp 1–18

  47. Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013) Distributed representations of words and phrases and their compositionality. In: Proceedings of NIPS, pp 3111–3119

  48. Li Y, Li Z, Zhang M, Wang R, Li S, Si, L (2019) Self-attentive biaffine dependency parsing. In: Proceedings of IJCAI, pp 5067–5073

Download references

Acknowledgements

National Natural Science Foundation of China (U21B2027, 61972186, 62266028, 62266027), Yunnan Provincial Major Science and Technology Special Plan Projects (202103AA080015, 202202AD080003, 202203AA080004), Yunnan Fundamental Research Projects (202301AS070047), Kunming University of Science and Technology’s “Double First-rate” Construction Joint Project (202201BE070001-021), Yunnan High and New Technology Industry Project (201606).

Funding

This article is funded by National Natural Science Foundation of China, U21B2027.

Author information

Authors and Affiliations

  1. Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming, 650500, Yunnan, China

    Ying Li, Zhenguo Zhang, Yantuan Xian, Zhengtao Yu, Shengxiang Gao, Cunli Mao & Yuxin Huang

  2. Yunnan Provincial Key Laboratory of Artificial Intelligence, Kunming, 650500, Yunnan, China

    Ying Li, Zhenguo Zhang, Yantuan Xian, Zhengtao Yu, Shengxiang Gao, Cunli Mao & Yuxin Huang

Authors
  1. Ying Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhenguo Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yantuan Xian
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhengtao Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shengxiang Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Cunli Mao
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yuxin Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Y.L: Writing-Original Draft, Methodology. Z.Z: Data curation, Investigation. Y.X, Z.Y, S.G, C.Mand Y.H: Supervision, Validation.

Corresponding author

Correspondence to Yantuan Xian.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Y., Zhang, Z., Xian, Y. et al. Multi-source domain adaptation for dependency parsing via domain-aware feature generation. Int. J. Mach. Learn. & Cyber. 15, 6093–6106 (2024). https://doi.org/10.1007/s13042-024-02306-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13042-024-02306-0

Keywords

Search

Navigation