In December 2017, a user named “DeepFakes” posted realistic looking explicit videos of famous celebrities on Reddit. He generated these fake videos using deep learning, the latest in AI, to insert celebrities’ faces into adult movies.In the following weeks, the internet exploded with articles about the dangers of face swapping technology: harassing innocents, propagating fake news, and hurting the credibility of video evidence forever.In this post, I explore the capabilities of this tech, describe how it works, and discuss potential applications.DeepFakes offers the ability to swap one face for another in an image or a video. Face swapping has been done in films for years, but it required skilled video editors and CGI experts to spend many hours to achieve decent results.This is so remarkable that I’m going to repeat it: anyone with hundreds of sample images, of person A and person B can feed them into an algorithm, and produce high quality face swaps — video editing skills are not needed.This also means that it can be done at scale, and given that so many of us have our faces online, it’s trivially easy to insert almost anyone into fake videos. Scary, but hopefully it’s not all doom and gloom, after all, we as a society have already come to accept that photos can easily be faked.Before dreaming up how to use this tech, I wanted to get a handle on how it works and how well it performs.I picked two popular late night TV hosts, Jimmy Fallon and John Oliver, because I can find lots of videos of them with similar poses and lighting — and also enough variation (like lip sync battles) to keep it interesting.Luckily for me, there’s an active GitHub repo that contains the original DeepFakes code and many more improvements. It’s fairly straightforward to use, but the onus is still on the user to collect and prepare training data.To make experimentation easy, I wrote a script to work directly with YouTube videos. This makes collecting and preprocessing training data painless, and converting videos one-step. Click here to view my Github repo, and see how easily I generated the videos below (I also share my model weights).The following videos were generated by training a model on about 15k images of each person’s face (30k images total). I got faces for each celebrity from 6–8 YouTube videos of 3–5 minutes each, with 20 frames per second per video, and by filtering out frames that don’t have their faces present. All of this was done automatically — all I did was specify a list of YouTube video urls.The total training time was about 72 hours on a NVIDIA GTX 1080 TI GPU. Training is primarily constrained by GPU, but downloading videos, and chopping them into frames is I/O bound and can be parallelized.Note that while I had thousands of images of each person, decent face swaps can be achieved with as few as 300 images. I went this route because I pulled face images from videos, and it’s far easier to pick a handful of videos as training data, than to find hundreds of images.The images below are low resolution to keep the size of the animated GIF file small. There’s a YouTube video below with higher resolution and sound.While not perfect, the results above are quite convincing. The key thing to remember is: the algorithm learned how to do this by seeing lots of examples, I didn’t modify the videos in any way. Magical? Let’s look under the covers.At the core of the Deepfakes code is an autoencoder, a deep neural network that learns how to take an input, compress it down into a small representation or encoding, and then to regenerate the original input from this encoding.Putting a bottleneck in the middle forces the network to recreate these images instead of just returning what it sees. The encodings help it capture broader patterns, hypothetically, like how and where to draw Jimmy Fallon’s eyebrow.Deepfakes goes further by having one encoder to compress a face into an encoding, and two decoders, one to turn it back into person A (Fallon), and the other to person B (Oliver). It’s easier to understand with a diagram:In the above, we’re showing how these 3 components get trained:Once training is complete, we can perform a clever trick: pass in an image of Fallon into the encoder, and then instead of trying to reconstruct Fallon from the encoding, we now pass it to Decoder B to reconstruct Oliver.It’s remarkable to think that the algorithm can learn how to generate these images just by seeing thousands of examples, but that’s exactly what has happened here, and with fairly decent results.While the results are exciting, there are clear limitations to what we can achieve with this technology today:These are tenable problems to be sure: tools can be built to collect images from online channels en masse; algorithms can help flag when there is insufficient or mismatched training data; clever optimizations or model reuse can help reduce training time; and a well engineered system can be built to make the entire process automatic.But ultimately, the question is: why? Is there enough of a business model to make doing all this worth it?Given what’ve now learned about what’s possible, let’s talk about ways in which this could be useful:Hollywood has had this technology at its fingertips, but not at this low cost. If they can create great looking videos with this technique, it will change the demand for skilled editors over time.But it could also open up new opportunities: for instance, making movies with unknown actors, and then superimposing famous celebrities onto them. This could work for YouTube videos or even news channels filmed by regular folks.In more out-there scenarios, studios could change actors based on their target market (more Schwarzenager for the Austrians), or Netflix could allow viewers to pick actors before hitting play. More likely, this tech could generate revenue for the estates of long dead actors by bringing them back to life.Some of the comment threads on DeepFakes videos on YouTube are abuzz about what a great meme generator this technology could create. Jib Jab is a company that has been selling video greeting cards with simple face swapping for years (they are hilarious). But the big opportunity is to create the next big viral hit; after all photo filters attracted masses of people to Instagram and SnapChat, and face swapping apps have done well before.Given how fun the results can be, there’s likely room for a hit viral app if you can get the costs low enough to generate these models.Imagine if Target could have a celebrity showcase their clothes for a month, just by paying her agent a fee, grabbing some existing headshots, and clicking a button. This would create a new revenue stream for celebrities, social media influencers, or anyone who happens to be in the spotlight at the moment. And it would give businesses another tool to promote brands and drive conversion. It also raises interesting legal questions about ownership of likeness, and business model questions on how to partition and price rights to use them.Imagine a world where the ads you see as you surf the web include you, your friends, and your family. While this may come across as creepy today, does it seem so far fetched to think that this won’t be the norm in a few years?After all, we are visual creatures, and advertisers have been trying to elicit emotional responses from us for years, e.g. Coke may want to convey joy by putting your friends in a hip music video, or Allstate may tug at your fears by showing your family in an insurance ad. Or the approach may be more direct: Banana Republic could superimpose your face on a body type that matches yours, and convince you that it’s worth trying out their new leather jackets.Whoever the original Deepfakes user is, they opened a Pandora’s box of difficult questions about how fake video generation will affect society. I hope that in the same way we have come to accept that images can easily be faked, we will adapt to video uncertainty too, though not everyone shares this hope.What Deepfakes also did is shine a light on how interesting this technology is. Deep generative models like the autoencoder that Deepfakes uses, allow us to create synthetic but realistic looking data (including images or videos), only by showing an algorithm lots of examples. This means that once these algorithms are turned into products, regular folks will have access to powerful tools that will make them more creative, hopefully towards positive ends.There have already been some interesting applications of this technique, like style transfer apps that make your photos look like famous paintings, but given the high volume and exciting nature of the research that is being published in this space, there’s clearly a lot more to come.I’m interested in exploring how to build value from the latest in AI research; if you have an interest in taking this technology to market to solve a real problem, please drop me a note.A few fun tidbits for the curious:From a quick cheer to a standing ovation, clap to show how much you enjoyed this story.I’ve been a product manager, engineer, and founder for over a decade in Seattle and Silicon Valley. Currently exploring new ideas at the Allen Institute for AI.how hackers start their afternoons.