Audio-reactive StyleGAN: supplementary examples

Back to research overview

This is supplemental material demonstrating the techniques discussed in this paper for the NeurIPS 2020 Workshop on Machine Learning for Creativity and Design. Code can be found in this repository.

Prior Work

The results of current publicly available audio-reactive StyleGAN interpolations are shown below.

Created with wzrd.ai with maximal values for all "advanced settings."

The remaining videos are generated using the techniques described in the paper.

Chromagram-weighted Sequences

Chromagram-weighted sum of latent vectors. Each note pulls the sum vector towards its own latent when played. This gives each chord its own unique appearance.
Bass and chord chromagrams mapped to the lower and upper blocks of the latent hierarchy, respectively. The bass controls the structure of the video and the chords control the color and detail.

Onset Envelopes

Drum onsets mapped to noise and latents. The kick controls the noise amplitude and the snare the weight of a latent vector.
Combining the bass/chord chromagram sequence with the drum onsets mapped to noise.

Network Bending / Model Rewriting

Kick onsets control a zooming transformation and the snare onsets horizontal translation. The melody which comes in halfway is used to make a chromagram-weighted sequence which is mixed in with its onsets.
The glitch effect envelope used on the audio file is also used to control the strength of a random scaling effect on the weights of the network. Scaling per parameter in the first 6 convolutional layers are sampled from a normal distribution with mean -1.

Long-Term Features

Below is a video which combines separate interpolations designed for individual sections of the song.

The RMS of the bass (which is only present in the main section) is used to interpolate between 2 sets of latents: warm colors for the intro/outro and cool colors for the main section.

The 1920x1080 resolution is achieved with a little network bending trick. Applying a mirroring transformation on the learned constant layer (lowest layer in the generator) doubles the width. This propagates all the way up through the fully-convolutional network leaving an output with twice the width.

The following video for a 30 minute DJ mix was generated using a combination of all the techniques discussed. The focus is on using looping latent sequences as these require less fine-tuning and post-processing which would be prohibitive for such a long audio-reactive video. Timestamps of transitions were entered manually to switch between latent vectors which were selected for each song. A moving average of the RMS was used to mix between faster latent loops and higher standard deviation noise during the high energy sections and less intense latents/noise in the softer sections.

A mix of some of my favorite Drum & Bass songs for Can't Be Cancelled, a digital charity festival which raised money for Meals On Wheels.