Python image recognition find coordinates (Appium by identifying the image click Coordinates)

* * * If you just want to know the image similarity recognition, see the first step directly* * * If you want to know appium according to image recognition Click Coordinates, need to see tertiary step Background |when you do a UI test, you find that the iOS custom UI control is not recognized by Appium. So consider finding coordinates by identifying the image and then solving the problem by clicking on the coordinates.  python-dependent packages | OpenCV, NumPy, AIRCV  First step: Find the coordinate point of the picture on the original picture

ImportAIRCV as ACdefMatchimg (imgsrc,imgobj,confidencevalue=0.5):#imgsrc= Original image, imgobj= the image to be looked upIMSRC =Ac.imread (imgsrc) imobj=Ac.imread (imgobj) Match_result= Ac.find_template (imsrc,imobj,confidence)#{' confidence ': 0.5435812473297119, ' Rectangle ': ((394, 384), (394, 416), (, 384), (416)), ' Result ': (422.0, 4 00.0)}    ifMatch_result is  notnone:match_result['Shape']= (Imsrc.shape[1],imsrc.shape[0])#0 is high, 1 is wide    returnMatch_result

 Description: Returns the coordinate result of the matching picture by AIRCV's Find_template () method1. Enter the parameter:Find_template (original image imsrc, image to be searched imobj, lowest similarity confidence) 2. Return results:

{'confidence'rectangle'result' : (422.0, 400.0)

confidence: matching similarity rateRectangle: Match the coordinates of a picture on the original image with quadrilateral shapesResult: Match the center point of the image on the original image, which is the point of the click we're looking for Note: If the result matches to a confidence less than the confidence of the incoming parameter pass, then none is returned and the dictionary is not returnedReference Document:HTTPS://GITHUB.COM/NETEASEGAME/AIRCV   Step Two: convert the coordinates of the picture to the actual coordinate point on the phone screenbecause after the resolution on the PC, and on the phone resolution is not the same, and we through the first step to find the coordinates point is the PC's coordinate point, is generally much larger than the mobile phone, so need to convert the coordinates 

Photo_position=self.driver.get_screenshot_as_file (Imgfile)#screen-cut phonex= Self.driver.get_window_size () ['width']y= Self.driver.get_window_size () ['Height']size_width,size_height= x, y#get the wide and high size of your phone dConfidencevalue= 0.8#Define similarity degreePosition = matchimg (imsrc,imobj,confidence)#in the first step, the Find_template () method is actually ifPosition! =none:x, y= position['result'] shape_x, shape_y= Tuple (Map (int,position['Shape'])) position_x,position_y=int (photo_position_x+ (photo_width/shape_x*x)), int (photo_position_y+ (photo_height/shape_y*y)) Self.driver.tap ([(Position_x, position_y)])

 description of the idea:1. Through the Appium method driver.get_screenshot_as_file (filename)2. Obtain a wide and high dictionary through the get_window_size of Appium, and then get the width and height3. Match the matching result coordinates and the size of the original image on the PC by matching the phone screen obtained on the PC.4. Get the actual coordinate point on the phone via the aspect ratio of the screen on the PC and the phone, and the actual coordinate point on the PC5. Finally, through the Appium method, the coordinates on the phone are clicked Drive.tap ([x, y]) Note: In order to match the accuracy of the results, preferably on the PC 1:1 under the original, do not enlarge, otherwise the similarity will be much worseThe third step: optimize, capture the image on the phone part of the area, to match the similarity, improve the matching accuracybecause some pictures are too small, if they are matched on a large map, they are often not matched. Well, if you know the approximate location of the picture, you can match that area . There are two areas of approach:1. Depending on the element to which the appium is positioned

Driver.find_element (*element). Screenshot (Imgfile)

 2. Based on the upper-left coordinate of the rectangle (percent x, y) and width height (percent)

Image.open (imgfile). Crop ((pc_location_x,pc_location_y,pc_location_x+pc_width,pc_location_y+pc_height)). Save ( Imgfile)

 capture the entire phone screen first, then calculate it based on the percentage and the width of the PC, and get the coordinates by PIL's crop () method .then get the coordinates on the phone based on the ratio of PC and mobile 

Python image recognition find coordinates (Appium by identifying the image click Coordinates)