AB
AB
AiBoss

All Categories

project

Fish Speech - An open-source, high-efficiency text-to-speech (TTS) tool

Fish Speech is an open-source text-to-speech (TTS) tool developed by Fish Audio, supporting Chinese, English, and Japanese. Trained on approximately 150,000 hours of multilingual data, it achieves near-human-level speech synthesis results...

What is Fish Speech?

Fish Speech is an open-source text-to-speech (TTS) tool developed by Fish Audio, supporting Chinese, English, and Japanese. Trained on approximately 150,000 hours of multilingual data, it achieves near-human-level speech synthesis. Currently, it's updated to version 1.2. Key features include low memory requirements (only 4GB), fast inference speed, high customizability, and flexibility, allowing users to quickly clone speech without complex training. Fish Speech also supports various speech generation models, such as VITS2 and Bert-VITS2, making it suitable for scenarios such as intelligent assistants, automated customer service, and language learning.

Features of Fish Speech

  • Highly efficient text-to-speech conversionFish Speech utilizes advanced algorithms to quickly convert input text into natural-sounding, fluent speech. Through optimized acoustic and language models, it ensures the naturalness and accuracy of the speech, delivering high-quality output across various scenarios.
  • Multilingual supportSupporting Chinese, English, and Japanese, Fish Speech's multilingual capabilities can overcome language barriers, providing services to users in different countries and regions and facilitating global application scenarios.
  • Voice cloning capabilityUsers can upload a voice clip of themselves or someone else as a reference. Fish Speech uses deep learning technology to learn and imitate the characteristics of that voice, achieving a personalized voice clone. This feature has broad application potential in personalized voice assistants, audiobook production, and other fields.
  • Low video memory requirementsIt only requires 4GB of video memory to run, which greatly lowers the hardware threshold and allows more users to use Fish Speech on their own computers without having to invest in expensive hardware.
  • Fast reasoning speedFish Speech optimizes the inference process, reduces waiting time, and improves the efficiency of speech synthesis. Users can obtain the required voice output in a short time, enhancing the overall user experience.
  • Multiple speech generation modelsFish Speech supports a variety of models, including VITS2, Bert-VITS2, GPT VITS, MQTTS, GPT Fast, and GPT-SoVITS. Users can choose the appropriate model according to their needs to obtain the best speech synthesis effect.
  • Easy to useFish Speech is designed with user experience in mind, simplifying the installation and configuration process. Users don't need in-depth technical knowledge to quickly start using it with a single click, greatly lowering the barrier to entry.
  • Fine-tuning capabilityLORA fine-tuning technology allows users to make detailed adjustments to the model to adapt to specific speech styles or expressions, providing users with more creative space.
  • Performance optimizationBy employing advanced techniques such as gradient checkpointing, causal sampling, and flash-attn, Fish Speech has achieved significant performance improvements during model training and inference, ensuring efficiency and stability when processing large-scale data.

Fish Speech official website

How to install and use Fish Speech

Operating requirements

  • GPU memory: 4GB (for inference), 16GB (for fine-tuning)
  • System: Linux, Windows

Windows configuration

Windows professionals can consider using WSL2 or Docker to run the codebase.

For non-professional Windows users, the following are basic operating methods that do not require a Linux environment (including model compilation functionality, i.e.) torch.compile):

  1. Unzip the project archive.
  2. Click install_env.bat to install the environment.
    • You can edit install_env.bat. USE_MIRROR The option to decide whether to use a mirror site for downloading is selected.
    • USE_MIRROR=false Download the latest stable version from the original website. torch environment.USE_MIRROR=true To download the latest from the mirror site torch Environment. Default is... true.
    • You can edit install_env.bat. INSTALL_TYPE This option determines whether to enable downloading in a compileable environment.
    • INSTALL_TYPE=preview Download the development version of the compilation environment.INSTALL_TYPE=stable Download the stable version without a compilation environment.
  3. If step 2 INSTALL_TYPE=preview, then this step will be executed (it can be skipped; this step is to activate the compilation model environment).
    1. Download the LLVM compiler using the following link.
    2. Download and install the Microsoft Visual C++ Redistributable Package to resolve potential .dll missing issues.
    3. Download and install Visual Studio Community Edition to obtain the MSVC++ compiler tools and resolve LLVM header file dependency issues.
        • Visual Studio Download
        • After installing the Visual Studio Installer, download Visual Studio Community 2022.
        • Click as shown in the image below修改Button, find使用C++的桌面开发Select the item and check the download box.
  4. Double-click start.bat to enter the Fish-Speech training and inference configuration WebUI page.
    • (Optional) Want to go directly to the inference page? Edit the project root directory. API_FLAGS.txtThe first three lines should be modified to the following format: 
      --infer
# --api
# --listen ...
...
    • (Optional) Want to start the API server? Edit the project root directory. API_FLAGS.txtThe first three lines should be modified to the following format: 
      --infer
# --api
# --listen ...
...
  5. (Optional) Double-click run_cmd.bat Enter the conda/python command-line environment of this project

Linux configuration

# 创建一个 python 3.10 虚拟环境, 你也可以用 virtualenv
conda create -n fish-speech python=3.10
conda activate fish-speech

# 安装 pytorch
pip3 install torch torchvision torchaudio

# 安装 fish-speech
pip3 install -e .

# (Ubuntu / Debian 用户) 安装 sox
apt install libsox-dev
Back