VibeVoice

Overview

VibeVoice is a text-to-speech framework for generating expressive, long-form, multi-speaker conversational audio, including podcasts. It uses continuous acoustic and semantic speech tokenizers operating at 7.5 Hz, a next-token diffusion approach with an LLM for textual context and dialogue flow, and a diffusion head for acoustic detail. The project states it can synthesize speech up to 90 minutes long with up to 4 distinct speakers.

Key Features

• Continuous acoustic and semantic speech tokenizers operate at an ultra-low 7.5 Hz frame rate to preserve audio fidelity while improving computational efficiency for long sequences.
• A next-token diffusion framework uses an LLM to understand textual context and dialogue flow, with a diffusion head generating high-fidelity acoustic details.
• Supports synthesis of speech up to 90 minutes long with up to 4 distinct speakers.
• Demonstrates context-aware expression such as spontaneous emotion and spontaneous singing.
• Includes examples for podcasts with background music, cross-lingual Mandarin-English speech, and long conversational speech.

Use Cases

• Generating podcast-style multi-speaker conversations from text.
• Producing long-form conversational audio with multiple distinct speakers.
• Creating expressive speech with emotional or singing-style delivery.
• Exploring cross-lingual Mandarin and English speech generation.

Before You Adopt

• License: MIT. Review its terms before using, modifying, or distributing the project.
• The project is positioned as a research framework for speech synthesis collaboration.
• The repository was disabled after release due to out-of-scope use, so availability may be constrained.