OpenAI's Jukebox for music generation

[ArthurZucker]

Model description

Jukebox is an autoregressive model for music generation published by OpenAI in 2020. It is based on a hierarchical VQ-VAE and Scalable Transformers (based on Sparse Transformers ) to create long music samples that can be conditioned on Genres, Artists, Timing and Lyrics.

3 sampling strategies will be made available :

• Ancestral sampling : tokens are generated in an autoregressive fashion and are then upsampled
• Windowed sampling : in order to generate long sequences, samples are repeatedly produced from overlapping windows using the previous codes as context.
• Primed sampling : a continuation of a previous audio is obtained using the VQ-VAE encoding of the audio as initial tokens for the ancestral sampling

The generated tokens are then passed through the VQ-VAE decoder to obtain the final audio.

The Lyric conditional informations are obtained using a Lyric Transformer model.

Open source status

• The model implementation is available
• The model weights are available

Provide useful links for the implementation

The code is available at https://github.com/openai/jukebox, and weights are available at https://openaipublic.azureedge.net/jukebox/models/.

Authors :

• prafullasd
• heewooj
• jongwook
• mcleavey