FT-GAN: Fine-Grained Tune Modeling for Chinese Opera Synthesis

Authors

  • Meizhen Zheng Department of Artificial Intelligence, School of Informatics, Xiamen University, Xiamen, China Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China
  • Peng Bai Department of Artificial Intelligence, School of Informatics, Xiamen University, Xiamen, China Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China
  • Xiaodong Shi Department of Artificial Intelligence, School of Informatics, Xiamen University, Xiamen, China Key Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China
  • Xun Zhou Department of Artificial Intelligence, School of Informatics, Xiamen University, Xiamen, China
  • Yiting Yan Department of Artificial Intelligence, School of Informatics, Xiamen University, Xiamen, China

DOI:

https://doi.org/10.1609/aaai.v38i17.29943

Keywords:

NLP: Speech, NLP: Generation

Abstract

Although singing voice synthesis (SVS) has made significant progress recently, with its unique styles and various genres, Chinese opera synthesis requires greater attention but is rarely studied for lack of training data and high expressiveness. In this work, we build a high-quality Gezi Opera (a type of Chinese opera popular in Fujian and Taiwan) audio-text alignment dataset and formulate specific data annotation methods applicable to Chinese operas. We propose FT-GAN, an acoustic model for fine-grained tune modeling in Chinese opera synthesis based on the empirical analysis of the differences between Chinese operas and pop songs. To further improve the quality of the synthesized opera, we propose a speech pre-training strategy for additional knowledge injection. The experimental results show that FT-GAN outperforms the strong baselines in SVS on the Gezi Opera synthesis task. Extensive experiments further verify that FT-GAN performs well on synthesis tasks of other operas such as Peking Opera. Audio samples, the dataset, and the codes are available at https://zhengmidon.github.io/FTGAN.github.io/.
AAAI-24 / IAAI-24 / EAAI-24 Proceedings Cover

Downloads

Published

2024-03-24

How to Cite

Zheng, M., Bai, P., Shi, X., Zhou, X., & Yan, Y. (2024). FT-GAN: Fine-Grained Tune Modeling for Chinese Opera Synthesis. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17), 19697-19705. https://doi.org/10.1609/aaai.v38i17.29943

Issue

Vol. 38 No. 17: AAAI-24 Technical Tracks 17

Section

AAAI Technical Track on Natural Language Processing II