Streaming Generation for Music Accompaniment

Yusong Wu1, Mason Wang2, Heidi Lei2, Stephen Brade2, Lancelot Blanchard2, Shih-Lun Wu2,
Aaron Courville1,3, Anna Huang2
1Mila, Université de Montréal 2MIT 3Canada CIFAR AI Chair
arXiv Code
Update: an earlier version of our data pipeline always used the complementary submix (all other stems) as the input instead of a random subset of them. The examples below are from models retrained with the fix; the original examples, with more details, are here.

Each example row shares the same input. For every system we provide the rendered mix (input plus accompaniment) and the standalone generated accompaniment.

Example 1 - Bass Input
Ground Truth
Mix
Output
Offline StemGen
Mix
Output
Offline Prefix Decoder
Mix
Output
tf = -1 s
Mix
Output
tf = 0 s
Mix
Output
tf = 1 s
Mix
Output
Random Pairing
Mix
Output
Example 2 - Piano Input
Ground Truth
Mix
Output
Offline StemGen
Mix
Output
Offline Prefix Decoder
Mix
Output
tf = -1 s
Mix
Output
tf = 0 s
Mix
Output
tf = 1 s
Mix
Output
Random Pairing
Mix
Output
Example 3 - Piano Input
Ground Truth
Mix
Output
Offline StemGen
Mix
Output
Offline Prefix Decoder
Mix
Output
tf = -1 s
Mix
Output
tf = 0 s
Mix
Output
tf = 1 s
Mix
Output
Random Pairing
Mix
Output
Example 4 - Guitar Input
Ground Truth
Mix
Output
Offline StemGen
Mix
Output
Offline Prefix Decoder
Mix
Output
tf = -1 s
Mix
Output
tf = 0 s
Mix
Output
tf = 1 s
Mix
Output
Random Pairing
Mix
Output
Example 5 - Percussive Input
Ground Truth
Mix
Output
Offline StemGen
Mix
Output
Offline Prefix Decoder
Mix
Output
tf = -1 s
Mix
Output
tf = 0 s
Mix
Output
tf = 1 s
Mix
Output
Random Pairing
Mix
Output

Interactive Model Comparison With More Configurations

Select a sample and configure two different models to compare their outputs side by side.

Configuration A

Mix
Output

Configuration B

Mix
Output