Propose MIDI-DDSP, a hierarchical music generation model with explicit and interpretable representation for controlling musical performance and synthesis.

MIDI-DDSP can reconstruct high-fidelity audio, accurately predict performance attributes for a note sequence, independently manipulate the attributes of a given performance, and as a complete system, generate realistic audio from a novel note sequence.

Code | Blog | Colab Notebook | Huggingface Space | Command-line MIDI Synthesis

Proposed a sequence-to-sequence model with a CNN as encoder and a Transformer as decoder, with data augmentation, data regulation, pre-training and fine-tuning for accurate automatic audio captioning.

The proposed system ranked 2nd in all participants achieved by a SPIDEr score of 0.214.

The proposed system won Reproducible System Award.

paper

Propose a Peking Opera singing synthesis system with expressiveness and dynamics in Peking opera singing. The model is trained by inputting musical note and learned from the spectrogram.

demo paper

Collect and construct a symbolic music dataset of Chinese Guqin Dataset. Dataset is also used for proposing a statistical approach to distinguishing different music genre.

Together with Yuxuan Wu and Yi Deng, we created a song about the growth of the AI model from its own perspective using music generation models. The song was ranked 3rd in the contest. To learn more about the song, please visit here.

We propose MusicLDM, a text-to-music diffusion model that trains efficiently on limited dataset and avoid plagiarism. We leverage a beat tracking model and propose two different mixup strategies for data augmentation: beat-synchronous audio mixup and beat-synchronous latent mixup. Such mixup strategies encourage the model to interpolate between musical training samples and generate new music within the convex hull of the training data, making the generated music more diverse while still staying faithful to the corresponding style. To learn more: sample page, code,paper.

Together with Stability.ai, we created CLAP, a large-scale contrastive language-audio representation learning model. CLAP has been used in many projects and has been chosen widely be community for standard feature extraction model to calculate Frechet Audio Distance. To learn more: model github, data,paper.

We introduce MusicLDM, a text-to-music diffusion model designed to train efficiently on limited datasets and avoid musical plagiarism. By leveraging beat-synchronous audio mixup strategies, MusicLDM generates novel, stylistically faithful music. To learn more: ReaLchords webpage,ReaLchords paper, GenJam webpage.

FLAM aligns language and audio at the frame level, enables open-set sound event detection and unlocking fine-grained multimodal tasks such as text-conditioned audio generation, localization, and separation.

paper