TensorFlow implements neural style image transfer

Just beginning to contact TensorFlow, practice a small project, also refer to other bloggers of the article, I hope you put forward valuable comments.

The code and images in the article have been uploaded to GitHub (Https://github.com/Quanfita/Neural-Style). What is image style migration.

Each of the following pictures is a different art style. Intuitively it's hard to find out what these different styles of pictures have to do with the exact language to describe the go and connect. How to change the style of an image into another style is a problem that is difficult to define. As a programmer, how to use code to achieve.

(Image from https://zhuanlan.zhihu.com/p/26746283)

After the advent of neural networks, several of Gatys's papers presented solutions that allowed the machine to understand the vague concept of style: [Texture Synthesis Using convolutional neural Networks (1505.07376)] (https://arxiv.org/abs/1505.07376.pdf), [A Neural algorithm of artistic Style (1508.06576)] (https://arxiv.org/abs/ 1508.06576.pdf), [preserving Color in neural artistic Style Transfer (1606.05897V1)] (https://arxiv.org/abs/ 1606.05897v1.pdf).

Neural style becomes a very interesting deep learning application: Enter a picture representing the content and a picture representing the style, and the deep Learning Network will output a new piece of work that blends this style and content. TensorFlow is the most popular deep learning framework for Google Open source. There is an open source TensorFlow implementation of the Neural style Code (address) on GitHub.

Let's just take a look at neural style. This paper describes how to solve this problem. (Below is the main content of the paper) content of the thesis

First, there are several concepts:

convolutional Neural Network (CNN)

An input image is represented in a series of filtered images at each layer of the convolutional neural network. As layers are processed one layer at a level, filtered images are continuously reduced by sampling downward (e.g. through a pooled layer). This makes the number of neurons in each layer of neural nets smaller. (That is, the deeper the layer, because the pool layer, a single feature map will become smaller, so the number of neurons in each layer will be less and fewer).

Content Remodeling

In the only knowledge of the output of the layer, by reshaping the input image, you can see the different stages of the CNN image information. 5 levels on the original vgg-network: Conv1_1,conv1_2,conv1_3,conv1_4,conv1_5 The input image is reshaped.

The image entered is a row of houses in the figure above, and the 5 levels are a,b,c,d,e. Image refactoring (ABC) at the lower layer is perfect; at the higher layer (DE), detailed pixel information is lost. In other words, this extracts the contents of the image, but discards the pixels.

Style Remodeling

On top of the original CNN characterization (feature map), a new feature space was established (feature space), which captures the style of the input image. The characterization of the style calculates the similarity between the different layers of CNN without features. We reconstruct the style of the input image through the characterization of the styles established on different subsets of the CNN hidden layer. This creates a style that is consistent with the input image and discards the global content.

The key to this paper is that the representation of content and style can be separated in CNN. It is possible to operate two representations independently to produce new, perceptible images. The paper generates a picture that blends content and stylistic representations from two different images.

A picture, which simultaneously conforms to the photo's content characterization, and the artistic painting style representation. The overall layout of the original photo is preserved, while the color and the local structure are provided by art paintings.

The style representation is a multi-scale representation, including the multilayer of neural networks. In the image shown in Figure 2, the characterization of the style contains the hierarchy of the entire neural network. Styles can also contain only a fraction of the lower levels. (See figure below, the first line is Volume base 1, line 5th is the output of the volume base 5). If the style representation in the higher level is met, the local image structure will be increased on a large scale, thus making the image smoother and more coherent visually.

In short, the author directly regards the local feature as the approximate picture content, thus obtains the picture content and the picture style (plainly is the texture) separate system, the rest is the picture's content and another picture's style to combine.

The content and style of the image cannot be completely decomposed. When the style and content come from a different two images, the synthesized new image does not exist at the same moment that perfectly conforms to the two constraints. However, the loss function minimized in image synthesis includes both content and style, and they are well separated. So we can put the emphasis on the content and style in a smooth way.

Method:

Suppose the response of a layer is Fl∈rnlxml F l∈r N lxm l f^{l}\in r^{n_{l}\times m_{l}}, where Nl N l N_{L} is the number of L l l layer filter, Ml M l m_{l} is fil The size of TER. Flij F i J l F_{ij}^{l} represents the output of the I-I I-I filter at position J of the L-l-l layer.

P⃗p→\vec{p} represents an image that provides content, x⃗x→\vec{x} represents the generated image, and PL and FL represent their response to the L-l layer respectively, so the content Loss of the L layer:

Lcontent (p⃗,x