The basis for these videos is a next frame prediction feedback loop with a correction model trying to restore compounding frame prediction error artifacts back to the original style (as explained in more detail in this post).

Here I was specifically investigating if I could get these feedback loops, which in principle endlessly append new frames, to loop back on themselves seamlessly.

I settled on a process as follows: first run a couple of rollouts of the normal next frame prediction feedback (optionally with correction step). Be sure to use the same input frame to start the process! Take one of the two rollouts and reverse it. Crossfade the last couple of seconds of the first rollout into the first couple of seconds the reversed rollout (simply reversing a single rollout could work just as well, a classic trick to get things looping).

However, this approach has obvious drawbacks: the loop is generally a bit jarring and the blend over point is easily recognizable. To try to alleviate this, I tried another pass over the loop using the correction model to restore the original style. This led to quite a bit of jitter.

The key to getting it butterysmooth was to instead apply a temporal gaussian filter (blurring neighboring frames in time) and then apply another corrector model trained to restore blurred frames back to their original quality. The gaussian filter helps to smooth over the jolts between rollouts as well as the crossover period in the middle.

｡𖦹°

The technique is pretty robust even to inputs which have quite jarring loop points. In honor of Halloween, a spooky lavalamp with a few more fancy rollouts spliced together.

2018-10-31

Back to gallery