Appium through image recognition technology OpenCV solve the problem of analog input password of cipher keyboard

Appium Automation test Framework, the specific information inconvenience I repeat, online can find a lot of information

In my project in the process of doing automated testing, encounter the password keyboard, need to simulate the input of the problem, the use of appium provided by the Sent_keys method regardless of use, because the password keyboard must be clicked on the keyboard to enter the password can be passed.

The principle of the password keyboard is to click a button each time, for an encryption, click the end, by login to be able to log in through. The Send_keys input method is to directly use the system's keyboard to send key values to the input box, no click operation.

Password keyboard A difficult point is that it does not have ID, and Appium provides the positioning method is mostly through the id,name,class and other elements of the page positioning, looked at all the positioning method, found that through the coordinate value can also be positioned, and password keyboard to locate success, must be recognized through the image, Identify the location coordinate information of the keyboard

The following function is to identify the coordinates of the cipher keyboard by means of an image recognition method.

The password keyboard encountered is shown in the following figure

There is also the contents of the input box, I have to intercept.

Solution idea:

1, image recognition, and then click the coordinates

2, development cooperation, to the specific app package, lift the security keyboard.

The second method is not available for a non-own app.

Image recognition found OpenCV inside the match template method, you can find the specified picture in the picture,

steps: 1, the first need to operate the location of the screenshot and save it, remember to intercept the original size, as small as possible, such as:

2, and then on the interception screen to find the corresponding position of these numbers

3, according to the position on the screen click

Implementation code:

As an example of the need to enter a password, I wrote the following code, you need to pass the screenshot image path, but also need to enter the password. I have saved pictures of each number in the local area and N0.jpg,n1.jpg,... Save, and put it together with the script.

I use Python script, of course you also need to install OPENCV, download path: http://opencv.org/releases.html Download after the build below the python inside the Cv2.pyd file directory under the Lib\ Site-packages inside

The code is as follows

#coding: Utf-8 import OS import time import CV2 import numpy as NP import Matplotlib.pyplot as Plt def get_pay_keyboard_n Umber_location (IM, pwd): Numbers = set (List (pwd)) templates = {} positions = {} Nimgpath = "" #数字图片不在同目录
    Use for i in numbers:templates[i] = Os.path.join (Nimgpath, "n{}.jpg". Format (i)) start = Time.time () Img_rgb = Cv2.imread (IM) for Tenum, Tepath in Templates.items (): # print (tepath) template = Cv2.imre AD (Tepath) h, w = template.shape[:-1] res = cv2.matchtemplate (Img_rgb, template, Cv2. tm_ccoeff_normed) threshold =. 95 # match parameter, 1 to exact match loc = Np.where (res >= threshold) If Len (Loc  ) > 0:positions[tenum] = Zip (*loc[::-1]) [0] Else:print ("Can not found number: [{}] in Image: [{}]. ". Format (Tepath, IM)) end = Time.time () print (End-start) return [positions[n] for n in pwd] if __name__ = "_ _main__ ": ls = Get_pay_keYboard_number_location (' keyboard.jpg ', ' 1234567890 ') print (LS) 

The code above implements the coordinates of getting each number and returns with the list.

Once you get the coordinates, you can go directly through the appium tap, such as

Touchaction (Driver). long_press (x = 180,y = 1400). Perform ()

Click on the operation.

Code is not necessarily rigorous, please correct me