Learning Practice: How to use TensorFlow to achieve fast style migration? _tensorflow

Introduction of Style migration

Style Transfer is one of the most interesting applications of deep learning, as shown in this way, we can use this method to "migrate" the style of a picture to another picture:

However, the speed of the original style migration (click to view the paper) is very slow. On the GPU, it takes about 10 minutes to generate a picture, and it may take several hours if you use only the CPU without using the GPU to run the program. This time will also increase as the size of the picture increases rapidly.

The reason for this is that in the original style migration process, the process of generating pictures as a "training" process. Each generation of a picture is equivalent to training a model, which may be iterated hundreds of thousands of times. If you know a little bit about machine learning, you'll know that it takes too much time to train a model from scratch than to perform a model that's already been trained. And that's why the original style is slow to migrate.

Introduction to Rapid style migration

Is there a way to do that instead of creating a picture as a "training" process, and as an "execution" process? The answer is yes. See Quick Style migration (fast neural style transfer): Perceptual losses for real-time style transfer and super-resolution

The Fast style migration network structure consists of two parts. One is "Generate Network" (in the original Transformation network), one is "Loss network" (Loss Network). The build network receives a picture as input, and then the output is also a picture (that is, the result of the style migration). The following figure, the left is the build network, the right is the loss network:

Training stage: First select a style picture. The goal of the training is to make the build network effective in generating pictures. The goal is defined by the loss network.

Implementation phase: Given a picture, input it into the network, output the image style after the migration results.

We can find that in the "execution" phase of the model we can complete the creation of the style picture. So the speed of generating a picture is very small, on the GPU generally less than 1 seconds, running on the CPU will only take a few seconds.

TensorFlow implementation of fast style migration

Say no more, directly on the GitHub address of my code (click to see)

And the transformation effect is as follows.

Original Picture:

Style after migrating pictures:

The above images are generated approximately 2.9s under the GPU (Titan black) generated around the need for 0.8S,CPU (i7-6850k).

With regard to the fast style migration, there are already two implementations of TensorFlow on GitHub:

Junrushao1994/fast-neural-style.tf

Olavhn/fast-neural-style

But the first project provided only a few trained models, no code to provide training, no specific network structure provided. So it's not really useful.

The second model is implemented in a complete, model training, but the effect is not very good, in the author's own blog, gave an example, you can see the resulting picture has a lot of noise points:

My project is to make a lot of changes and adjustments on the basis of Olavhn/fast-neural-style.

Iv. Some implementation details

1, combined with TensorFlow Slim

In the original implementation, the author used the VGG19 model as a loss network. In the original paper, the VGG16 is used. To maintain consistency, I used the TensorFlow slim to repackage the loss network.

Slim is an extended library of TensorFlow that provides a number of functions related to image classification, and has already been well trained models (such as VGG, Inception series, and ResNet series).

The following figure is the model supported by Slim:

Using slim to replace the original network, in the loss function, we can not only use VGG16, but also easy to use VGG19, resnet and other network structures. Specific implementation please refer to the source code.

2, improve the transfer of convolution of two trick

When we need to use the network to generate images, we usually use the transpose convolution to sample the image directly.

This paper points out some problems of the transpose convolution, and thinks that due to unreasonable coincidence, the transfer convolution make the resulting picture always has "checkerboard-like noise point", it put forward the use of the picture to enlarge, and then do the convolution of the way to replace the convolution to do on the sample, you can improve the quality of the resulting picture, the following figure is two methods of comparison:

The implementation of the corresponding TensorFlow:

The above is the first trick.

The second trick is mentioned in the article Instance normalization:the Missing ingredient for Fast stylization, substituting Instance normalization for the usual batch N Ormalization, can improve the quality of style migration.

3. Pay attention to the use of optimizer and saver

This is a small detail about the implementation of TensorFlow.

In TensorFlow, optimizer and saver are the default to train and save all the variables in the model. But in this project, the whole network is divided into two parts: Generation Network and Loss network. Our goal is to train and build the network so we just need to train and save the variables in the build network. When constructing optimizer and saver, be aware that only variables in the build network are passed in.

Identify the variables that need to be trained to pass to optimizer:

V. Summary

Anyway, I did a very interesting project. The code is not very much, if only with a trained model to generate pictures, the use of CPU can also be in a few seconds to run the results, do not need to build a GPU environment. Suggested that interested students can play on their own.

About training, there is also a comparison of Teng (Dan) of the tuning experience, the next time to share again, the first to write here today. Thank you.