Against the sample machine learning _note1_ machine learning

A brief introduction to Learning _note1 against Sample machine

Machine learning methods, such as SVM, neural network, etc., although in the problem such as image classification has been outperform the ability of human beings to deal with similar problems, but also has its inherent defects, that our training sets are fed natural input, so under normal conditions to deal with better. However, if we want to attack the ML model, we can generate a counter sample (adversarial examples) by a certain means, taking the image as an example, against the sample at each pixel point only a small perturbation (pertubations), so the human eye is indistinguishable , that is, before and after the generation we humans will classify it as the same category. However, the ML model will have a bad feature in the face of these counter samples, which will produce the wrong points. The basic idea of counter sample generation is: in the process of training model, we fixed the input to adjust the parameters, so that the final result can correspond to the corresponding input, and when we generate the confrontation sample, we will fix the model, by adjusting the input, It is possible to observe in which characteristic direction only a small perturbation is required to allow our model to give the classification result of the wrong points we want. The aim of the study against sample machines is to hope that our model is more robust against samples.

The above illustration shows how the counter sample works. Model decision boundary is the classification boundary of our trained models, this boundary can better separate two kinds of samples, but if we do a small perturbation to the two test point, we can make it cross the boundary to produce misclassification , so our task decision boundary should also be divided into the original category of these confrontation samples.

For this problem, similar to the general security problem, we generally consider two angles, namely attack and defense. The attack attempts to better generate the counter samples so that the classification results conform to attacker's own expectations, and defense hopes to protect against attacks by improving the robustness of the model so as not to be sensitive to these adversarial examples. The common attack method for generating a countermeasure sample is fast gradient sign method (FGSM) and jacobian-based saliency map approach (JSMA). As shown in the following figure, the disturbances in the generated counter samples are insensitive to human vision, but for the ML model, a picture of a panda, originally 57.7percent, was sentenced to a gibbon for a 99.3 chance of being modified.

For defense, common methods are: adversarial training: This method is very ordinary thinking, that is, in the course of training the network, each image generated a number of counter samples, and then give them the same label as the original to feed the network training, This makes the network relatively robust against the sample. Open source Cleverhans is a library that fights training against samples generated by fgsm or Jsma. Defensive distillation: This method is used to smooth the direction of the perturbation of the sample against the decision Surface,distillation (narrowly translated into distillation. ) is a method used by Hinton to make small models imitate large models, and the basic idea is that when we train the classification model, the One-hot vectors, called hard label, are trained on a model with hard label. Not only do we retain that dimension of the largest probability after softmax, but rather the entire probability vector as a label (a little bit like the idea of a personal sensation and label smoothing), which is called Soft label, Each input sample is not only a small amount of information (because the classification result is too certain, that is, the picture is determined to be the category, the other class is completely unrelated to the class), but rather a vector with a certain probability for each category one-hot. As a result of the training network, you will get some additional information, such as a picture may be more difficult to distinguish between the two classes, so that they have a higher probability, so that the label actually with the large model training information, so you can improve the effect of small models. (flag: Above for personal understanding, after reading Hinton's reference paper) the so-called defensive distillation first trained to use a hard label on a network, then get soft tags and train another network (distillation network), Using a distilled network to classify will be more robust to adversarial.

A failed defense case is the gradient masking, which is the direct output class rather than the probability, which makes it impossible to gradient a small perturbation image, but by training a gradient network, this is the basis for perturbation, You can also attack a network that has been defense through this method.

The above illustration shows that even if the defense causes gradient to be masked, we can train the substitution model to generate a confrontation sample.

Compared with attack, the defense of machine learning is more difficult. There is a lack of a better theoratical model to show that some kind of adversarial sample can be excluded.

For the design of a stable and reliable system, there is a need for testing and verification, the so-called testing, which evaluates the system under a number of different conditions and observes its performance under these conditions; and verification means, Give a convincing reason that the system does not misbehave under broad range of circumstances. Just testing is not enough because testing only gives a lower bound of the system's failure rate, but for the purpose of security protection, we need to know the upper bound of the failure rate. But the verification of machine learning cannot have a guarantee against the sample, so it is not perfect.

Reference
cleverhans-blog:http://www.cleverhans.io/