Machine Learning Art

Speech Cloning

SOTA speech synthesis models

I’m going to show you how easy it is to create state-of-the-art speech synthesis models with Toucan toolkit. You’ll see how everything is pure Python and PyTorch based to keep it as simple and beginner-friendly as possible.

IMS Toucan is a toolkit for teaching, training and using state-of-the-art Speech Synthesis models, developed at the Institute for Natural Language Processing (IMS), University of Stuttgart, Germany. The basic PyTorch Modules of FastSpeech 2 are taken from ESPnet, the PyTorch Modules of HiFiGAN are taken from the ParallelWaveGAN repository which are also authored by the brilliant Tomoki Hayashi.

Project Page (scroll down)

New Features

🔵 Vocoders can be used to perform super-resolution and spectrogram inversion simultaneously.

🔵 Articulatory representations of phonemes as the input for all models. This allows to easily use multilingual data to benefit less resource-rich languages.

🔵 A checkpoint trained with a variant of model agnostic meta learning from which you should be able to fine-tune a model with very little data in almost any language (except for tonal languages, as mentioned in the last point).

🔵 A small self-contained Aligner that is trained with CTC and an auxiliary spectrogram reconstruction objective, inspired by this implementation.

🔵 By conditioning the TTS on an ensemble of speaker embeddings as well an an embedding lookup table for language embeddings, multi-lingual and multi-speaker models are possible.

🔵 Vocoders can also be used to do some slight speech-enhancement by corrupting a small percentage of their input spectrograms.

🟠 Exactly cloning the speaking style of a reference utterance is also possible and it works in conjunction with everything else! So any utterance in any language spoken by any speaker can be replicated and controlled to allow for maximum customizability.

Installation

conda create --prefix ./toucan_conda_venv --no-default-packages python=3.8

pip install --no-cache-dir -r requirements.txt

pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html

Creating a new Pipeline

To create a new pipeline to train a HiFiGAN vocoder, you only need a set of audio files. To create a new pipeline for a FastSpeech 2, you need audio files, corresponding text labels, and an already trained Aligner model to estimate the duration information that FastSpeech 2 needs as input.

Training a Model

Once you have a pipeline built, training is super easy.

python run_training_pipeline.py <shorthand of the pipeline>

Using a trained Model for Inference

You can load your trained models using an inference interace. Simply instanciate it with the proper directory handle identifying the model you want to use, the rest should work out in the background. You might want to set a language embedding or a speaker embedding. The methods for that should be self-explanatory.

Conclusion:

The goal of synthesizing natural and high-quality speech from text involves two aspects: The first is the direct modeling and generation of waveforms at a sampling rate of 48 kHz, with a good compromise between the acoustic model and the HiFiNet vocoder, which provides higher perceptual quality than previous lower sampling rate systems; Second, the variation information in speech is modeled by a systematic design that includes both explicit and implicit modeling, which improves prosody and naturalness. Overall, the naturalness (MOS) of System F is significantly higher than all other systems and is not significantly different from natural speech; the speaker similarity (SMOS) of System F is better than all other systems, showing the superiority of the Toucan system.

@inproceedings{lux2021toucan,
  title={{The IMS Toucan system for the Blizzard Challenge 2021}},
  author={Florian Lux and Julia Koch and Antje Schweitzer and Ngoc Thang Vu},
  year={2021},
  booktitle={Proc. Blizzard Challenge Workshop},
  volume={2021},
  publisher={{Speech Synthesis SIG}}
}

@article{lux2022laml,
  title={{Language-Agnostic Meta-Learning for Low-Resource Text-to-Speech with Articulatory Features}},
  author={Florian Lux and Ngoc Thang Vu},
  year={2022},
  journal={arXiv preprint arXiv:2203.03191},
}

project page:

https://github.com/DigitalPhonetics/IMS-Toucan

Demo:

https://huggingface.co/spaces/Flux9665/IMS-Toucan

🟠 SpeechCloning Demo:

https://huggingface.co/spaces/Flux9665/SpeechCloning

I invite you to explore the concept of “AI creativity” by reading and learning from the many articles found on 🔵 MLearning.ai 🟠

I am an Art Curator, founder at EvArtology. I advise companies and institutions in the creative industries on using AI tools in their daily work. Human collaboration with ML models can be very creative and bring huge benefits. The new era begins now.

Data Scientists must think like an artist when finding a solution when creating a piece of code. Artists enjoy working on interesting problems, even if there is no obvious answer.

All our writers (members) receive the opportunity to be promoted on our social media, which increases the popularity of articles published on MLearning.ai

1. Linkedin (6.5K+ ML-professionals)
2. Twitter (4.7K+ followers)
3. Instagram (2.2K + followers )
4. Sketchfab * — individual vRooML!
5. Facebook
6. Youtube
7. Apple Podcasts
8. Substack

🔵 Submission Suggestions