SemanticAudio - an audio generation and editing framework developed by The Chinese University of Hong Kong and other companies. - AiBoss

What is SemanticAudio?

SemanticAudio is an audio generation and editing framework jointly developed by the Chinese University of Hong Kong, LIGHTSPEED, and Shanghai Jiao Tong University. The framework breaks down text-to-audio generation into two stages: "semantic planning" and "acoustic synthesis." It plans the identity, timing, and structure of sound events in a high-level semantic space before rendering them into high-quality audio. The framework supports text-guided audio editing without training and significantly outperforms mainstream methods such as TangoFlux on both AudioCaps and TTABench benchmarks, achieving a dual improvement in semantic alignment and generation quality.

Main functions of SemanticAudio

Text-to-audio generationInput a natural language description and generate high-quality audio such as ambient sounds, action sounds, and complex sound scenes.
Semantic planning generationThe global layout of sound events is planned in the high-level semantic space, and then acoustic details are synthesized.
No audio editing training requiredThrough the FlowEdit ODE mechanism, sound attributes can be modified directly in the semantic space, supporting operations such as replacement and adjustment.
Frame-level semantic embeddingExtract semantic representations that preserve temporal structure to accurately describe the sequence of events and local changes in complex audio.

SemanticAudio's technical principles

Two-phase Flow Matching ArchitectureSemanticAudio breaks down audio generation into two stages: semantic planning and acoustic synthesis. The Semantic Planner first generates a compact semantic representation from the text, depicting the global layout of sound events. The Acoustic Synthesizer then uses this semantic plan as a condition to generate high-quality acoustic latent variables and decode them into audio. This decoupling ensures clear division of labor within the model, avoiding the coupling problem of a single-stage model handling both semantic understanding and acoustic rendering simultaneously in the acoustic space.
Construction and compression of semantic spaceThe Perception Encoder extracts frame-level semantic embeddings to preserve temporal structure, which are then compressed to 128 dimensions using a lightweight MLP. This low-dimensional semantic space retains key sound identity and temporal information and can be efficiently learned by the Flow Matching model, becoming a crucial intermediate layer connecting textual semantics and acoustic details.
FlowEdit ODEThe semantic trajectory editing is guided by the velocity field difference between the source and target texts. The source audio is encoded as a semantic latent variable, the velocity field difference between the two texts is calculated to determine the editing direction, ODE stepping is performed to obtain the target semantic representation, and then it is restored by an acoustic synthesizer. The editing is performed in a high-level semantic space, and attribute-level modifications can be achieved without additional training, inversion, or pairing data.
The core value of semantic-acoustic decouplingExplicit decoupling allows the model to first plan the identities and timing of multiple events in the semantic space before synthesizing acoustic details, avoiding event omissions and sequence errors under complex prompts. The semantic space provides editors with clearer and more interpretable objects for manipulation, making modifications closer to "changing the sound content itself," and enabling more stable and flexible text-guided editing.

How to use SemanticAudio

Visit the Demo pageVisit https://semanticaudio1.github.io/ to experience the online demo.
Input text prompt: Describe the target sound scene, such as "a dog barks and then a car horn sounds".
Semantic planningSemantic Planner generates compact semantic representations and plans the layout of sound events.
Acoustic SynthesisAcoustic Synthesizer renders semantic plans into high-quality audio.
Audio editing (optional)Input source audio and target text, and use FlowEdit ODE to modify sound properties in semantic space.

SemanticAudio's core advantages

Significant improvement in semantic alignmentBy first planning the semantic layout and then synthesizing the acoustic details, AudioCaps LAION-CLAP achieves a score of 0.381, which is better than TangoFlux's 0.361, demonstrating more accurate understanding of complex cue words.
Editable without trainingThe FlowEdit ODE mechanism uses velocity field differences to guide trajectories in the semantic space, supports attribute-level modifications, improves CLAP by +0.094, outperforms similar methods, and requires no additional training.
Balancing production quality and controllabilityFD 19.1 and MOS 3.72 significantly improve text-audio semantic alignment while maintaining high-fidelity listening quality.
Strong handling of complex prompt wordsExplicitly plan the identities and temporal relationships of multiple events to effectively avoid problems such as missing events, incorrect order, and insufficient text alignment.
More in line with human creative processesIt adopts a paradigm of first planning the sound scene structure and then adding acoustic details to achieve a generation method of thinking things through before speaking.

Comparison of SemanticAudio's similar products

Dimension	SemanticAudio	TangoFlux
Architecture	Two-stage (semantic planning + acoustic synthesis)	Single-stage acoustic space generation
Semantic alignment	LAION-CLAP 0.381	LAION-CLAP 0.361
Editing ability	No training required, attribute-level modification	Requires FlowEdit compatibility; the effect is relatively weak.
Complex prompt words	Explicitly planning the timing is less prone to errors.	Events that are easily overlooked or whose order is confused
Generate quality	FD 19.1, MOS 3.72	FD 22.6
Core differences	Semantic-acoustic decoupling: Think before you speak	Modeling directly in acoustic space

Application scenarios of SemanticAudio

Post-production of film and televisionBased on the natural language descriptions in the script, SemanticAudio can generate complex sound effects scenes with multiple sound events and precise timing, improving post-production efficiency.
Game developmentFor different game scenarios, developers can dynamically generate matching ambient sounds and action sound effects through text commands, enabling rapid iteration of audio content.
Content creationCreators of short videos, podcasts, and other content can quickly obtain customized audio materials by simply inputting a description, lowering the barrier to professional audio production.
Virtual RealityThe system generates immersive ambient sounds in real time based on the user's natural language commands, enhancing the sense of presence and interactive experience in the virtual space.
Audio editing toolsUsers can directly modify the attributes of existing audio or replace sound events using natural language, achieving professional-level editing without the need for training.