Last Wednesday, “A Neural Algorithm of Artistic Style” was posted to ArXiv, featuring some of the most compelling imagery generated by deep convolutional neural networks (DCNNs) since Google Research’s “DeepDream” post.On Sunday, Kai Sheng Tai posted the first public implementation. I immediately stopped working on my implementation and started playing with his. Unfortunately, his results don’t quite match the paper, and it’s unclear why. I’m just getting started with this topic, so as I learn I want to share my understanding of the algorithm here, along with some results I got from testing his code.In two parts, the paper describes an algorithm for rendering a photo in the style of a given painting:2. Instead of trying to match the activations exactly, try to match the correlation of the activations. They call this “style reconstruction”, and depending on the layer you reconstruct you get varying levels of abstraction. The correlation feature they use is called a Gram matrix: the dot product between the vectorized feature activation matrix and its transpose. If this sounds confusing, see the footnotes.Finally, instead of optimizing for just one of these things, they optimize for both simultaneously: the style of one image, and the content of another image.Here is an attempt to recreate the results from the paper using Kai’s implementation:Not quite the same, and possibly explained by a few differences between Kai’s implementation and the original paper:As a final comparison, consider the images Andrej Karpathy posted from his own implementation.The same large-scale, high-level features are missing here, just like in the style reconstruction of “Seated Nude” above.Beside’s Kai’s, I’ve seen one more implementation from a PhD student named Satoshi: a brief example in Python with Chainer. I haven’t spent as much time with it, as I had to adapt it to run on my CPU due to lack of memory. But I did notice:After running Tübingen in the style of The Starry Night with a 1:10e3 ratio and 100 iterations, it seems to converge on something matching the general structure but lacking the overall palette:I’d like to understand this algorithm well enough to generalize it to other media (mainly thinking about sound right now), so if you have an insights or other implementations please share them in the comments!I’ve started testing another implementation that popped up this morning from Justin Johnson. His follow the original paper very closely, except for using unequal weights when balancing different layers used for style reconstruction. All the following examples were run for 100 iterations with the default ratio of 1:10e0.Justin switched his implementation to use L-BFGS and equally weighted layers, and to my eyes this matches the results in the original paper. Here are his results for one of the harder content/style pairs:Other implementations that look great, but I haven’t tested enough:The definition of the Gram matrix confused me at first, so I wrote it out as code. Using a literal translation of equation 3 in the paper, you would write in Python, with numpy:It turns out that the original description is computed more efficiently than this literal translation. For example, Kai writes in Lua, with Torch:Satoshi computes it for all the layers simultaneously in Python with Chainer:Or again in Python, with numpy and Caffe layers:From a quick cheer to a standing ovation, clap to show how much you enjoyed this story.Artist working with code.