It's Raining: Next Frame Prediction III
A next frame prediction feedback loop with an extra temporal correction model trying to restore compounding frame prediction error artifacts back to the original style. This frame prediction model is the final descendent of models each trained on feedback loops from previous frame prediction models, trying to distill the essential qualities of the next frame feedback process itself.
The basic ideas behind the process are explained in more detail in this post. The visual style is largely based on CycleGAN interpretations of the Lakspe dataset.
In this series of next frame prediction feedback loops, I experimented with training different models to map back and forth between different styles and motion characteristics. I also played around with mixing and matching multiple different image-to-image models together in a single feedback loop.
Lakspe Frame Prediction 135
2019-07-01
This model shows more sluggish upwards motion, rather than the downward motion of the first model. The dynamics of each checkpoint can change significantly, even for the same model over the course of a training run. Often these feedback loops will converge to a stable state (especially focused in the sides or corners of the frame as we see in both models), but sometimes they will diverge and produce more chaotic motion.
Lakspe Frame Prediction 310
2019-07-01
This is the same model as the previous feedback loop but trained for more than twice as many epochs. Here we can see the motion swap back and forth between two modes: slow bubbling of greens & oranges and fast whipping in pink & blue.
Lakspe Gliders
2019-07-01
Here we see a very different motion style from another Lakspe model. This one throws out wavefronts of stable bubbles down to the left. We can see the translation invariance of the convolutional CycleGAN generator at play here. Each bubble has a thick black border surrounding it with the same structure clearly present in each bubble. I can’t help but see the similarities here to Gliders in Conway’s Game of Life or Reaction Diffusion systems.
Invicta Blubberoll Resynthesis
2019-06-29
This video is from a resynthesis of the Blubberoll feedback loop. This is done by training a CycleGAN model to transform the Blubberoll style into the Lakspe style and applying it to each frame of the Blubberoll video. Now there are four models contributing to the development of each frame: the forward Blubberoll model, the translation to the Lakspe style (these frames are pictured), the reverse translation back to the Blubberoll style, and finally the Blubberoll temporal correction model.
Meandermation
2019-06-29
This meandermation uses mismatched pairs of prediction & correction models. The next frame prediction model is based on the same model as the first Lakspe Frame Prediction model, but the correction model is taken from Zeinu. This leads to a slow undulating motion which blends the qualities of the two models together. Finally, the result is resynthesized into the Lakspe style using the same process as the Blubberoll resynthesis.