Datamoshing on realtime

Wikipedia’s article on compression artifacts states:

The minimization of perceivable artifacts is a key goal in implementing a lossy compression algorithm. However, artifacts are occasionally intentionally produced for artistic purposes, a style known as glitch art[1] or datamoshing.

We’ve been looking for a way to create such artifacts on a videocamera feed on realtime as opposed to the usual post-processing of already existing video files.

Take a look at this video to see the machine in action:

Simply put, this artifacts usually come up when the uncompression algorithm fails to load certain kind of frames.  In a compressed video file there are two kinds of frames:

  1. I-frames which would be just any frame in any movie.
  2. P-frames which are pixel-motion data frames.  What these frames do is tell the uncompressor where to move the current-frame’s pixels in order to get the next frame image.

It’s not a very complex subject, but it’s better explained elsewhere.  I strongly suggest taking a look at this great tutorial on datamosh.

So, since we wanted to use stream video, not recorded footage, we had to concentrate on the concept and forget about the specific tools.  Here’s how we did it with Max/Jitter.  Download this patch if you wish:

real_time_datamosh

Datamosh.maxpat.capture

Using cv.jit.opticalflow we get the motion of the pixels.  It outputs two matrices, one for the motion on the X axis (left outlet) and one for the motion on Y axis (right outlet).  We then pack them into a 2-plane matrix which is passed to the right inlet of a jit.repos object.  jit.repos repositions one matrix’s pixels with the values of another one.

Feeding one frame from the camera to the left inlet of the jit.repos object, will serve the same purpose than an I-frame on a compressed movie file.  Then we move its pixels with the opticalflow information coming into the right inlet of the jit.repos.  We save the output of the jit.repos in a (named) matrix that will feed back the the jit.repos left input again.

That’s all there is to it.  I wonder why it took us so long to figure it out!