Sketch2Sound - An AI audio generation technology developed by Adobe in collaboration with Northwestern University

What is Sketch2Sound?

Sketch2Sound is an AI audio generation technology developed by Adobe Research and Northwestern University. It generates high-quality sound effects based on sound imitation and text cues. Sketch2Sound extracts three control signals—loudness, brightness, and pitch—from sound imitation, encodes these signals, and uses them in a conditional text-to-sound generation system. Lightweight and requiring only a few fine-tuning steps and a single-layer linear fit, Sketch2Sound can be implemented on various text-to-audio models. It provides sound designers with tools that combine the semantic flexibility of text cues with the accuracy of sound imitation, enhancing the expressiveness and controllability of sound creation.

Sketch2Sound's main functions

• Combining voice imitation with text prompts: Sketch2Sound can understand sound imitation (such as verbal imitation) and text prompts, generating high-quality sound effects that match both.
• Extract control signals: Extract three key control signals from any input sound imitation: loudness, spectral centroid, and pitch probabilities.
• Generate arbitrary sounds: Using extracted control signals and text prompts, synthesize arbitrary sounds, including mimicking specific sounds or creating new sound effects.
• Lightweight implementation: It can be implemented on any text-to-audio latent diffusion transformer (DiT) with only 40,000 steps of fine-tuning and a separate linear layer for each control signal.

The technical principles of Sketch2Sound

• Control signal extraction: Using audio signal processing techniques, loudness, brightness, and pitch probabilities are extracted from the input sound imitation.
• Potential diffusion model: Based on a pre-trained text-to-sound latent diffusion transformer (DiT), the model includes a variational autoencoder (VAE) and a transformer decoder, which compresses audio into a continuous sequence of vectors and then generates a new sequence of latent vectors to synthesize audio.
• Conditional generation: Adding a linear projection layer to the latent diffusion model allows the control signal to be directly added to the model's noise latent variables, thus conditionalizing the model.
• Fine-tuning and adaptation: The pre-trained text-to-audio model is fine-tuned to enable it to handle time-varying control signals, thus achieving self-supervised fine-tuning.
• Control during reasoning: During inference, users select median filters of different sizes to adjust the timing details of the control signal, thereby striking a trade-off between the accuracy of sound imitation and the quality of the generated audio.
• Semantic flexibility and expressiveness: By combining the semantic flexibility of text prompts with the expressiveness of sound imitation, it provides users with a natural and intuitive method for creating sound.

Sketch2Sound project address

• Project official website:hugofloresgarcia.art/sketch2sound
• arXiv technical paper:https://arxiv.org/pdf/2412.08550

Application scenarios of Sketch2Sound

• Film and video production: In film and video post-production, generate sound effects that are synchronized with the visuals, such as sound effects that simulate specific environments (e.g., forests, cities, battlefields).
• Game Development: Design realistic sound effects and ambient sounds for video games to enhance immersion and interactivity.
• Music Production: Music producers create new musical elements or simulate the sound of specific instruments.
• Sound design education: In the teaching of sound design, it serves as a tool to help students understand the composition of sound and the basic methods of manipulating sound.
• Interactive media and installation art: In interactive art projects, corresponding sound feedback is generated based on the audience's behavior or input.