Rapping-Singing Voice Synthesis based on Phoneme-level Prosody Control

Konstantinos Markopoulos, Nikolaos Ellinas, Alexandra Vioni, Myrsini Christidou, Panos Kakoulidis, Georgios Vamvoukakis, Georgia Maniati, June Sig Sung, Hyoungmin Park, Pirros Tsiakoulis and Aimilios Chalamandaris

Abstract: In this paper, a Text-to-Rapping/Singing system is introduced, which can be adapted to any speaker's voice. It utilizes a Tacotron-based multispeaker acoustic model trained on read-only speech data and which provides prosody control at the phoneme level. Dataset augmentation and additional prosody manipulation based on traditional DSP algorithms are also investigated. The neural TTS model is fine-tuned to an unseen speaker's limited recordings, allowing rapping/singing synthesis with the target's speaker voice. The detailed pipeline of the system is described, which includes the extraction of the target pitch and duration values from an a capella song and their conversion into target speaker's valid range of notes before synthesis. An additional stage of prosodic manipulation of the output via WSOLA is also investigated for better matching the target duration values. The synthesized utterances can be mixed with an instrumental accompaniment track to produce a complete song. The proposed system is evaluated via subjective listening tests as well as in comparison to an available alternate system which also aims to produce synthetic singing voice from read-only training data. Results show that the proposed approach can produce high quality rapping/singing voice with increased naturalness.

Speaker ground truth samples

US_TR_M1
US_TR_F1
US_TR_F2
KO_TR_F1
KO_TR_F2
US_AD_M1
KO_AD_M1
Mellotron

Rapping/Singing speech synthesis

Reference
US_TR_M1
US_TR_F1
US_TR_F2
KO_TR_F1
KO_TR_F2
US_AD_M1
KO_AD_M1
Mellotron
Reference
US_TR_M1
US_TR_F1
US_TR_F2
KO_TR_F1
KO_TR_F2
US_AD_M1
KO_AD_M1
Mellotron
Reference
US_TR_M1
US_TR_F1
US_TR_F2
KO_TR_F1
KO_TR_F2
US_AD_M1
KO_AD_M1
Mellotron
Reference
US_TR_M1
US_TR_F1
US_TR_F2
KO_TR_F1
KO_TR_F2
US_AD_M1
KO_AD_M1
Mellotron
Reference
US_TR_M1
US_TR_F1
US_TR_F2
KO_TR_F1
KO_TR_F2
US_AD_M1
KO_AD_M1
Mellotron
Reference
US_TR_M1
US_TR_F1
US_TR_F2
KO_TR_F1
KO_TR_F2
US_AD_M1
KO_AD_M1
Mellotron
Reference
US_TR_M1
US_TR_F1
US_TR_F2
KO_TR_F1
KO_TR_F2
US_AD_M1
KO_AD_M1
Mellotron

Comparison of 15 vs 30 duration labels

Singing

Reference
US_TR_M1
US_TR_F1
US_TR_F2
KO_TR_F1
KO_TR_F2
US_AD_M1
KO_AD_M1
US_TR_M1
US_TR_F1
US_TR_F2
KO_TR_F1
KO_TR_F2
US_AD_M1
KO_AD_M1

Rapping

Reference
US_TR_M1
US_TR_F1
US_TR_F2
KO_TR_F1
KO_TR_F2
US_AD_M1
KO_AD_M1
US_TR_M1
US_TR_F1
US_TR_F2
KO_TR_F1
KO_TR_F2
US_AD_M1
KO_AD_M1

Assessment of Post-processing (WSOLA)

Singing

Reference
US_TR_M1
US_TR_F1
US_TR_F2
KO_TR_F1
KO_TR_F2
US_AD_M1
KO_AD_M1
US_TR_M1
US_TR_F1
US_TR_F2
KO_TR_F1
KO_TR_F2
US_AD_M1
KO_AD_M1

Rapping

Reference
US_TR_M1
US_TR_F1
US_TR_F2
KO_TR_F1
KO_TR_F2
US_AD_M1
KO_AD_M1
US_TR_M1
US_TR_F1
US_TR_F2
KO_TR_F1
KO_TR_F2
US_AD_M1
KO_AD_M1

Multispeaker musical mix

Multitrack