Abstract
Attention-based models have attracted crazy enthusiasm both in natural language processing and graph processing. We propose a novel model called Graph Encoder Representations from Transformers (GERT). Inspired by the similar distribution between vertices in graphs and words in natural language, GERT utilizes the equivalent of sentences-vertices obtained from truncated random walks to learn the local information of vertices. Then, GERT combines the strengths of local information learned from random walks and long-distance dependence obtained from transformer encoder models to represent latent features. Compared to other transformer models, the advantages of GERT include extracting local and global information, being suitable for homogeneous and heterogeneous networks, and possessing stronger strengths in extracting latent features. On top of GERT, we integrate convolution to extract information from the local neighbors and obtain another novel model Graph Convolution Encoder Representations from Transformers (GCERT). We demonstrate the effectiveness of proposed models on six networks DBLP, BlogCatalog, CiteSeerX, CoRE, Flickr, and PubMed. Evaluation results show that our models improve \(F_1\) scores of current state-of-the-art methods up to \(10\%\).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Chen, J., Zhang, Q., Huang, X.: Incorporate group information to enhance network embedding. In: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pp. 1901–1904. ACM (2016)
Chen, Yu., Wu, L.: Graph neural networks: graph structure learning. In: Graph Neural Networks: Foundations, Frontiers, and Applications, pp. 297–321. Springer, Singapore (2022). https://doi.org/10.1007/978-981-16-6054-2_14
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Dong, Y., Chawla, N.V., Swami, A.: metapath2vec: scalable representation learning for heterogeneous networks. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 135–144. ACM (2017)
Fan, S., et al.: Metapath-guided heterogeneous graph neural network for intent recommendation (2019)
Fu, T.Y., Lee, W.C., Lei, Z.: HIN2Vec: explore meta-paths in heterogeneous information networks for representation learning. In: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pp. 1797–1806. ACM (2017)
Grover, A., Leskovec, J.: node2vec: scalable feature learning for networks. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 855–864. ACM (2016)
Hamilton, W., Ying, Z., Leskovec, J.: Inductive representation learning on large graphs. In: Advances in Neural Information Processing Systems, pp. 1024–1034 (2017)
Hu, B., Fang, Y., Shi, C.: Adversarial learning on heterogeneous information networks (2019)
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)
Lee, J.B., Rossi, R., Kong, X.: Graph classification using structural attention. In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1666–1674. ACM (2018)
Lee, J.B., Rossi, R.A., Kim, S., Ahmed, N.K., Koh, E.: Attention models in graphs: A survey. arXiv preprint arXiv:1807.07984 (2018)
Li, J., Zhu, J., Zhang, B.: Discriminative deep random walk for network classification. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1004–1013 (2016)
Meng, Z., Liang, S., Bao, H., Zhang, X.: Co-embedding attributed networks. In: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, pp. 393–401. ACM (2019)
Ou, M., Cui, P., Pei, J., Zhang, Z., Zhu, W.: Asymmetric transitivity preserving graph embedding. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1105–1114. ACM (2016)
Pan, S., Wu, J., Zhu, X., Zhang, C., Wang, Y.: Tri-party deep network representation. Network 11(9), 12 (2016)
Perozzi, B., Al-Rfou, R., Skiena, S.: DeepWalk: online learning of social representations. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 701–710. ACM (2014)
Sohn, K., Lee, H., Yan, X.: Learning structured output representation using deep conditional generative models. In: Advances in Neural Information Processing Systems, pp. 3483–3491 (2015)
Tang, L., Liu, H.: Relational learning via latent social dimensions. In: Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 817–826. ACM (2009)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 5998–6008 (2017)
Wang, S., Tang, J., Aggarwal, C., Chang, Y., Liu, H.: Signed network embedding in social media. In: Proceedings of the 2017 SIAM International Conference on Data Mining, pp. 327–335. SIAM (2017)
Wang, X., et al.: Heterogeneous graph attention network. In: The World Wide Web Conference, pp. 2022–2032. ACM (2019)
Wu, L., et al.: Graph neural networks for natural language processing: A survey. arXiv preprint arXiv:2106.06090 (2021)
Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., Yu, P.S.: A comprehensive survey on graph neural networks. arXiv preprint arXiv:1901.00596 (2019)
Zhang, J., Jiang, Z., Li, T.: CHIN: classification with META-PATH in heterogeneous information networks. In: Florez, H., Diaz, C., Chavarriaga, J. (eds.) ICAI 2018. CCIS, vol. 942, pp. 63–74. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01535-0_5
Zhang, Z., et al.: ANRL: attributed network representation learning via deep neural networks. In: Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence (IJCAI). vol. 18, pp. 3155–3161 (2018)
Acknowledgements
This work was supported by the National Natural Science Foundation of China, 92267107.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, J., Jiang, Z., Li, C., Wang, Z. (2023). Semi-supervised Classification Based on Graph Convolution Encoder Representations from BERT. In: Yang, X., et al. Advanced Data Mining and Applications. ADMA 2023. Lecture Notes in Computer Science(), vol 14178. Springer, Cham. https://doi.org/10.1007/978-3-031-46671-7_14
Download citation
DOI: https://doi.org/10.1007/978-3-031-46671-7_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-46670-0
Online ISBN: 978-3-031-46671-7
eBook Packages: Computer ScienceComputer Science (R0)