Using Pytorch to implement visualising Image classification Models and saliency Maps

Source from Cs231n-assignment3-networkvisualization saliency map

Saliency map is a feature map that tells us the effect of the pixel points in the image on the image classification results.

When calculating it, the first thing to do is to calculate the gradient of the normalized fraction in the correct classification corresponding to the image pixels (this is a scalar). If the shape of the image is (3, H, W), the shape of the gradient is also (3, H, W); For each pixel in the image, this gradient tells us that the amplitude of the fractional change is correctly categorized when the pixel point has changed slightly.

When calculating saliency map, you need to calculate the absolute value of the gradient and then take the maximum of three color channels, so the final saliency map shape is (H, W) A grayscale image of a channel.

The following figure is an example:

The image above is the picture, the following image is a feature map, you can see the following image of the bright part of the neural network is interested in the section. Theoretical Basis

Take note:

Program Explanation

The following is a calculation of the feature map function, and contextual information is obtained through annotations.

def compute_saliency_maps (X, Y, model): "" Calculates the saliency map of the correct class using the model image (image) X and the tag (label) Y. Input:-X: Input image; Tensor of shape (N, 3, H, W)-y: The mark corresponding to x;

    Longtensor of shape (N,)-model: A pre-trained neural network model is used to calculate X.
    return value:-Saliency:a Tensor of shape (N, H, W) giving the saliency maps for the input images. "" "# Make sure the model was in ' test ' mode Model.eval () # Wrap the input tensors in Variables X_var = Va Riable (X, requires_grad=true) Y_var = Variable (y) saliency = None ############################################
    ################################## # # First forward operation, the input image pass through already trained model, and then reverse operation, # to get the corresponding image, the correct classification of the gradient of fractions # ############################################################################## # forward Operation scores = Model (X_va R) # Gets the correct class of fractions, scores for [5] of tensor scores = Scores.gather (1, Y_var.view ( -1, 1)). Squeeze () #反向计算, from the output fraction to the input image A series of gradient calculations Scores.backward (torch. Floattensor ([1.0, 1.0,1.0,1.0,1.0]) # parameter for the corresponding length of the gradient initialization # Scores.backward () must have parameters, because at this time the scores is non-scalar, 5 elements of the vector # to get the correct score corresponding to the input image pixel point gradient salien cy = X_var.grad.data saliency = Saliency.abs () # takes absolute value saliency, i = Torch.max (saliency,dim=1) # take absolute maximum from 3 color channels Number of channels saliency = Saliency.squeeze () # remove 1 D # Print (saliency) return saliency

Then define a display image function for image display.

def show_saliency_maps (X, y):
    # Convert X and y from numpy arrays to Torch tensors
    x_tensor = Torch.cat ([preproces S (Image.fromarray (x)) for x in X], dim=0)
    y_tensor = torch. Longtensor (y)

    # Compute saliency maps for images in X
    saliency = Compute_saliency_maps (X_tensor, y_tensor, model)

    # Convert The saliency map from Torch Tensor to numpy array and show Images
    # and saliency maps together.
    saliency = Saliency.numpy ()
    N = x.shape[0] for

    i in range (N):
        plt.subplot (2, N, i + 1)
        plt.imshow (X[i])
        plt.axis (' off ')
        Plt.title (Class_names[y[i])
        Plt.subplot (2, N, n + i + 1)
        plt.imshow (saliency[i ], Cmap=plt.cm.hot)
        plt.axis (' off ')
        PLT.GCF (). Set_size_inches (5)
    plt.show ()

Show_ Saliency_maps (X, y)

Output

Another method of gradient calculation, which passes the gradient calculated by the loss function

    out = Model (X_var)  
    Loss_func = Torch.nn.CrossEntropyLoss ()
    loss = Loss_func (out, Y_var) Loss.backward 
    () 
  grads = X_var.grad
    grads = grads.abs ()
    mx, index_mx = Torch.max (grads, 1)
#     print (MX, index_mx)
    saliency = mx.data
#     print (saliency)

The output of this method is:

Resources:
1, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. "Deep Inside convolutional networks:visualising Image classification Models and saliency Maps", Iclr Workshop 2014.
2, http://cs231n.stanford.edu/syllabus.html