The DeepSinger AI developed by the team was trained on data from different music sources, using algorithms capturing the singer's vocal timbre. The sounds they make when people sing have far more complex rhythms and patterns compared to simple speech. Another challenge to be solved by the team was that although there is a reasonable amount of speaking/speaking training data available, training data sets for singing is very uncommon. Combine these problems with the fact that songs need to have an analysis of both sound and lyrics, and the challenge of a singing performance generation is extremely complex.

By creating a data pipeline that mined and transformed audio data, the DeepSinger system created by the researchers overcame all these challenges. The singing clips were extracted from different music websites, and then the singing was isolated from the rest of the audio and split into sentences. The next step was to determine the length of each phoneme within the lyrics, resulting in a series of samples each representing a unique lyric phoneme. Data cleaning is done to deal with any distorted training samples after the lyrics are sorted according to the confidence score and the accompanying audio samples.

For a number of languages, it appears that the very same methods succeed. DeepSinger was trained on more than 92 hours of audio in Chinese, Canton, and English, consisting of clips from 89 different singers. The study results showed that the DeepSinger model was able to consistently produce high-quality "singing" samples based on metrics such as pitch accuracy and how natural the vocals sounded.

Some of the audio samples generated can be listened to at https://speechresearch.github.io/deepsinger/