Skip a jump Python software ideas and source analysis

jump a hop. Python Assistant software and image recognition source analysis

This article will comb the GitHub on the wechat_jump_game of the realization of the idea, and analysis of its image processing part of the source code

First, cut the crap, see the effect.
Core Ideas

gets the distance between the pieces to the center point of the next box
calculate the time to touch the screen
Click on the screen Important Method

Calculate the distance between the pieces to the center point of the next block use the ADB Shell screencap-p command to get the current screen picture of the phone and then find out the coordinates of the pieces and the coordinates of the next square with the information on the image and then calculate the distance by the distance formula between two points.

Calculate the time of the touch screen T=a * S
where S is the pixel distance from the upper step, T is the press time (MS), A is a coefficient this coefficient will change with the screen resolution, the coefficient is 1.35 under the 1920*1080 screen, the coefficient is 1.475 under the 2560*1440 screen.

Click on screen adb shell input swipe x y x y time (ms)
This command can be clicked on the phone screen x,y location time (ms) image processing part of the source code analysis

The image processing part of the code is in the Find_piece_and_board (IM) method

The coordinates of the pieces and the coordinates of the center of the next block are calculated by inputting the image IM

In the Find_piece_and_board approach, the following two nested for loops are included:

    For I in range (int (H/3), int (H * 2/3):
        Last_pixel = im_pixel[0, I] for J in
        Range (1, W):
            pixel = im_p Ixel[j, I]
            # is not a solid color line, then record the value of scan_start_y, ready to jump out of the loop
            if pixel[0]!= last_pixel[0] or pixel[1!= last_pixel[1] or pixel[ 2]!= last_pixel[2]:
                scan_start_y = i-50
                break
        if scan_start_y:
            break

The function of this code is to look down from the position of screen 2/3 to find a line that is not a solid color. And will find the position of ordinate-50 as, looking for the starting coordinates of the pieces and squares. This simplifies the search effort later, because there is nothing above the horizontal axis.

Next is the code to find the coordinates of the pieces.

# Find pieces coordinates
    # piece_x_sum The number of Piece_x_c points in the total axis of the Piece_y_max ordinate maximum
    # starting from scan_start_y, the pieces should be on the top half of the screen, not more than 2/3
  for i in range (scan_start_y, int (H * 2/3)):
        for J in range (Scan_x_border, W-scan_x_border):  # Horizontal axis also reduces part of the scan Overhead
            pixel = im_pixel[j, i]
            # According to the color of the lowest line of the pawn, find the average of those points in the last line, this color should be OK, temporarily do not propose
            if (< Pixel[0] <) and (5 3 < Pixel[1] < (< pixel[2] <):
                piece_x_sum = J
                Piece_x_c + = 1
                Piece_y_max = max (i , Piece_y_max)

    if not all ((Piece_x_sum, Piece_x_c): return
        0, 0, 0, 0

    # Average horizontal axis
    piece_x = Int (piece_x_ Sum/piece_x_c)
    # Ordinate maximum-the height of the base half
    piece_y = piece_y_max-piece_base_height_1_2  # up half of the height of the pawn chassis

The important basis to find pieces is that the pieces of the color is more single and the color of the box has a big gap. If the RGB pixel value of a pixel is in the range B (95, 110), then the pixel is considered to belong to the pawn. According to the above information can calculate the average horizontal axis of the pieces, as well as the maximum ordinate value.

So it is not difficult to calculate the coordinates of the pieces (the average axis of the pieces, the maximum ordinate of the pieces-the height of the base half) the height of the base is different from the resolution of the phone. Need to be configured well in advance.

Finally, the code to find the coordinates of the next square center point

# Looking for the highest chessboard # The chessboard will not be on the same side with the pieces # limit the horizontal axis of the board scan to avoid the note bug if piece_x < w/2: Board_x_start = piece_x b  Oard_x_end = W Else:board_x_start = 0 board_x_end = piece_x for i in range (int (H/3), int (H * 2
        /3): Last_pixel = im_pixel[0, I] if board_x or board_y:break board_x_sum = 0

            Board_x_c = 0 for J in range (int (board_x_start), int (board_x_end)): pixel = im_pixel[j, I]
                # The next chessboard is close to the pawn # It's better to fix your head than a small grid. Bug if ABS (j-piece_x) < piece_body_width: Continue # When the dome is repaired a line caused by a small bug, this color judgment should OK, temporarily do not put forward to if ABS (Pixel[0]-last_pixel[0]) + ABS (pixel [1]-last_pixel[1]) + ABS (Pixel[2]-last_pixel[2]) > 10:board_x_sum + = J Board_x_c + = 1 if board_x_sum: # The average horizontal axis of the highest chessboard board_x = Board_x_sum/board_x_c Last_pixel = Im_pix El[board_x, I]

The code begins by limiting the width of the search to the screen where the pieces are located, and if the pieces are on the left side of the screen, search for the squares on the right side and vice versa Because squares and pieces are not on the same side of the screen.

Then there is the top vertex of the search box Top-down.
Vertex coordinates on square (average horizontal, current line ordinate)

And then down the ordinate +247 position to start looking up the same color as the top vertex point, for the next vertex.
Of course, this method has a little limitation for the solid color of the plane effect is good but for the surface of the non solid color. Errors may be judged.

If the middle of the previous jump hit, then the next target Center will appear r245 g245 b245 point, using this property to make up for the last piece of code may exist error
If the previous jump for some reason did not jump to the middle, and the next jump just can not correctly identify the pattern, it is possible that the game failed, because the pattern area is usually relatively large, failure probability is lower can improve the scheme

The first is that the current scheme requires multiple profiles for multiple resolutions to record the coefficients at different resolutions and the height of the pieces ' chassis half. Randomly tested 6 mobile phones, two of which don't work because they don't have accessories.

The first is the coefficient a, the observation equation T=a * S,a is a training amount, using the machine learning framework such as TensorFlow, to fit this unary equation.

Observe the effect of half the height of the piece chassis on the code. It is not difficult to find out the vertical axis of the center of the pawn chassis. And the position of the pawn Chassis Center is exactly the most wide part of the piece. So you can find the most width of the pieces of the vertical axis of the chess Pieces Chassis center ordinate. This gets rid of the dependency on the configuration file, allowing the code to function on any phone.

The second is for the Square Center coordinate position judgment method error rate is high, although have center white dot can make up but in a lot of jump process still will appear error. 3 Mobile phones Keep running all afternoon, the highest score is only 2009 points.

The reason for the high error rate is to use the pure color method to judge, but in the actual game the color rich box is also many. If you want to change, you can't rely on the color method to judge, but you should calculate the position of the square by the shape of the geometric image. It is not difficult to find that the Chinese game is only a prism and a circle of two shapes.

First, the contour of the image is extracted by canny or other contour lookup operator, and then the central coordinates of the circle and the prism are extracted by Hough transform.

These are some of my personal insights. Welcome to the comments area together to explore learning