Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation
Abstract
Ovi is a unified audio-video generation model using twin-DiT modules with blockwise cross-modal fusion, enabling natural synchronization and high-quality multimodal outputs.
Audio-video generation has often relied on complex multi-stage architectures or sequential synthesis of sound and visuals. We introduce Ovi, a unified paradigm for audio-video generation that models the two modalities as a single generative process. By using blockwise cross-modal fusion of twin-DiT modules, Ovi achieves natural synchronization and removes the need for separate pipelines or post hoc alignment. To facilitate fine-grained multimodal fusion modeling, we initialize an audio tower with an architecture identical to that of a strong pretrained video model. Trained from scratch on hundreds of thousands of hours of raw audio, the audio tower learns to generate realistic sound effects, as well as speech that conveys rich speaker identity and emotion. Fusion is obtained by jointly training the identical video and audio towers via blockwise exchange of timing (via scaled-RoPE embeddings) and semantics (through bidirectional cross-attention) on a vast video corpus. Our model enables cinematic storytelling with natural speech and accurate, context-matched sound effects, producing movie-grade video clips. All the demos, code and model weights are published at https://aaxwaz.github.io/Ovi
Community
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- UniVerse-1: Unified Audio-Video Generation via Stitching of Experts (2025)
- X-Streamer: Unified Human World Modeling with Audiovisual Interaction (2025)
- HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generation (2025)
- MIDAS: Multimodal Interactive Digital-humAn Synthesis via Real-time Autoregressive Video Generation (2025)
- UniFlow-Audio: Unified Flow Matching for Audio Generation from Omni-Modalities (2025)
- WAVE: Learning Unified&Versatile Audio-Visual Embeddings with Multimodal LLM (2025)
- RAP: Real-time Audio-driven Portrait Animation with Video Diffusion Transformer (2025)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Move your head and start walking
8/12/16 Nivel a Laser Profissional Ajustável Nível 360° Laser Nível a Level Medição Da Linha De Ferramentas
Goku standing in fighting stance. Anime realistic.
We love the paper! Great job! Working on something special based on your work. My team has a couple questions if you get the opportunity to answer them. What adjustments or optimizations were made to go from the 5-second model to the 10-second model? (We would be interested in reading a paper on that 😁). We're currently working on getting inference down to the milisecond timescale (BIG fun! Think Hyperbolic spaces instead of standard euclidean RoPE). Huge fans! Thanks for the awesome model and paper!
Also we noticed in the fusion.py code that RoPE isnt being applied to the text cross attention. We found that at line 107, the text cross-attention happens, x = flash_attention(q, k, v, k_lens=context_lens). However the only time RoPE is applied is during the cross-modal attention, as it's being applied to q and k_target at line 118. Which means that only the cross-modal attention (between the audio and the video) get the benefits of the rope embeddings.
Is there a specific reason to not have the text cross-attention use RoPE?
That's kind of exciting to have some stuff to tinker with! Thanks for all t he great work!
Models citing this paper 2
Datasets citing this paper 0
No dataset linking this paper