Streaming Generation for Music Accompaniment

Yusong Wu1, Mason Wang2, Heidi Lei2, Stephen Brade2, Lancelot Blanchard2, Shih-Lun Wu2,
Aaron Courville1,3, Anna Huang2
1Mila, Université de Montréal 2MIT 3Canada CIFAR AI Chair

Original Examples (Full-Mixture Setting)

These are the examples from the first version of this page. They were generated by models trained with an earlier version of our data pipeline, which had two bugs:

  1. Input mixture. Instead of drawing the number of input stems uniformly from 1 to N−1, every example used all N−1 other stems of the track as the input mixture, i.e., always the complementary submix of the target stem.
  2. Silent tail. The pre-computed 20-second windows contained only 10 seconds of audio followed by silence, so for streaming models with tf > 0 the last tf seconds of input lookahead were silent, in both training and evaluation.

We fixed both, then retrained and re-evaluated all models. The paper reports the corrected (random-subset) setting in the main text and this earlier full-mixture setting in the appendix. The overall trends agree: quality follows the same S-shaped dependence on tf, models with tf ≤ 0 stay close to random pairing and largely ignore the input, and longer prompts help most when tf ≤ 0. Relative to the ground-truth and random-pairing levels, decoder models with tf ≥ 0 score higher in the full-mixture setting: with all other tracks present, the input carries more harmonic and rhythmic information about the target, which makes the task easier.

The listening study in the paper was conducted with these models, on examples with at most six input tracks. ← Back to the examples from the retrained models

Each example row shares the same vocal input used in the listening study. For every system we provide the rendered mix (input plus accompaniment) and the standalone generated accompaniment.

Example 1 - Bass Input
Ground Truth
Mix
Output
Offline StemGen
Mix
Output
Offline Prefix Decoder
Mix
Output
tf = -1 s
Mix
Output
tf = 0 s
Mix
Output
tf = 1 s
Mix
Output
Random Pairing
Mix
Output
Example 2 - Piano Input
Ground Truth
Mix
Output
Offline StemGen
Mix
Output
Offline Prefix Decoder
Mix
Output
tf = -1 s
Mix
Output
tf = 0 s
Mix
Output
tf = 1 s
Mix
Output
Random Pairing
Mix
Output
Example 3 - Piano Input
Ground Truth
Mix
Output
Offline StemGen
Mix
Output
Offline Prefix Decoder
Mix
Output
tf = -1 s
Mix
Output
tf = 0 s
Mix
Output
tf = 1 s
Mix
Output
Random Pairing
Mix
Output
Example 4 - Guitar Input
Ground Truth
Mix
Output
Offline StemGen
Mix
Output
Offline Prefix Decoder
Mix
Output
tf = -1 s
Mix
Output
tf = 0 s
Mix
Output
tf = 1 s
Mix
Output
Random Pairing
Mix
Output
Example 5 - Percussive Input
Ground Truth
Mix
Output
Offline StemGen
Mix
Output
Offline Prefix Decoder
Mix
Output
tf = -1 s
Mix
Output
tf = 0 s
Mix
Output
tf = 1 s
Mix
Output
Random Pairing
Mix
Output

Interactive Model Comparison With More Configurations

Select a sample and configure two different models to compare their outputs side by side.

Configuration A

Mix
Output

Configuration B

Mix
Output