Talking about how to use code to realize mobile phone tapping recognition

Zheng
Github:https://github.com/crazyzty
1. Preface Now there are many Android phones in the market support tapping screen unlock is a very useful function, but a dozen support percussion screen, and then can only be used to unlock or lock the screen, and our application layer of developers do not go in, completely can not play up. Developers, developers, since we as a developer why not make some big news, so this time I will teach you how to use code to achieve the phone's percussion recognition, it sounds interesting, some eager to. In fact, there are already applications for this feature on iOS: Knock, an app that taps to unlock Mac computers, costs $4.99, about 33 yuan. Sometimes I really want to do iOS development, can happily for their own application pricing, happy to earn extra money. Anyway, since iOS can be implemented, then we can't be outdated Android, and now I will lead you to use the code to realize the tap recognition of the phone.
This post takes Java as the sample language and Android as the sample platform.
2. Function Realization2.1. Realization of IdeasWhen it comes to percussion recognition, what do you think you can do with it, sensor? Yes, that's right. As the only way to identify the gesture gesture, we naturally need to use sensors to recognize the strokes, but there are a wide variety of Android sensors, which one should we choose?
In the era of Android2.3, the Android system has already defined 11 sensors, to the current Android6.0 era, the number of system-defined sensors has reached 26, so many sensors we actually use which, in fact, we only need to consider the 2.3 times provide the 11 sensors can be, because one side The sensor part, such as the heartbeat sensor, which requires hardware support, causes many mobile phones to be unable to support such sensors, on the other hand, 11 sensors in the 2.3 era are already quite powerful and can support the recognition of most gesture gestures, so now let me enumerate the 11 sensors:
Sensor_type_accelerometer acceleration
Sensor_type_magnetic_field Magnetic
Sensor_type_orientation direction
Sensor_type_gyroscope Gyroscope
Sensor_type_light Light Sensing
Sensor_type_pressure pressure
Sensor_type_temperature temperature
Sensor_type_proximity Close
Sensor_type_gravity Gravity
Sensor_type_linear_acceleration linear acceleration
Sensor_type_rotation_vector rotation vector
For a detailed description of these 11 sensors, you can go to http://www.oschina.net/question/163910_28354 to view, in fact, I have always suspected that the G3 of the LG is related to the light sensor or proximity sensor, because I hover with my fingers in LG G3 the head is not always able to tap unlock, after moving back to normal, and tapping the lock screen should only be related to the touch screen, because no matter how I block the sensor, the function of tapping the lock screen is completely unaffected.
After the 11 sensors to understand, we need to choose which sensor to achieve the function, we have to simulate the phone tapping situation, the phone flat on the table, the finger hit the phone when the finger to the phone a force, while the desktop to give a reaction to the cell phone, Considering that the desktop does not deform, the mobile phone is subjected to a force balance acceleration of 0, but then the phone's accelerometer data will change, the answer will be, the mobile phone accelerometer data will have a short but obvious change, for what, the mobile phone load balance acceleration of 0 is because it is a whole, But the internal components will also be affected by the complexity of the force of the left and right, not by force at the same time to achieve the balance of force, in fact, a change of thinking. With a cell phone similar to the shape of the inner smooth container, the container inside a few glass balls, a few strokes, the container will not move, but the glass ball is not moving it. Although the internal components of mobile phones are much more stable than glass balls, they also have to follow the Basic Law and accept the power honestly.
The above scenes are flat on the desktop scene, the actual life of the scene is often more complex and diverse, but no matter in which scene, no doubt on the phone's tapping operation should lead to the acceleration sensor out of the obvious changes in the data, then we now understand the need to choose what sensor as a tool for our tapping recognition, But there are two accelerometer-related sensors, accelerometer and linear accelerometer, which one should we choose? The accelerometer provides the acceleration of the mobile phone under the influence of gravity, and the linear acceleration sensor provides data that excludes the gravitational effects of the mobile phone acceleration, which can visually reflect the force of the mobile phone after gravity removal. , which is suitable for tapping recognition, then whether we should choose a linear accelerometer, on the contrary, we want to choose the accelerometer, Android provides linear accelerometer based on software, different platforms for linear acceleration sensor processing may not be the same, in fact, in the beating of Samsung S4,LG G3 in the back of a model, there is no big change in the data of the linear accelerometer, for the sake of insurance, we still choose hardware-based accelerometer more appropriate. By the way, when I saw the pressure sensor, I thought I was monitoring the pressure sensor on the cell phone, which is definitely suitable for identifying the tap, and then seeing the description to see the pressure.
As mentioned above, the tapping operation on the mobile phone will lead to the acceleration sensor out of the data of the obvious changes, so the implementation of this function, to determine whether there is a tapping operation method is to detect the mobile phone linear acceleration compared to the normal condition has a significant change. In order to eliminate the effects of gravity during the implementation of the function, it is necessary to process the data of the accelerometer into linear acceleration, since the conversion to linear acceleration is a process that needs to be calibrated, so a certain amount of data needs to be put in for calibration to obtain a more accurate linear acceleration, Taking into account that real life can lead to false recognition of the scene, such as shaking the mobile phone will bring a longer time and obvious linear acceleration changes, so the concept of steady state, the mobile phone is relatively stable, there is no significant linear acceleration of the situation for a long time as a steady state, In the steady state of the recognition of the percussion, in addition to this recognition, considering the use of the mobile phone frame is too low, so only to consider identifying the phone screen or the back of the tap, so that in the process of recognition can ignore the x, Y axis data, only consider the linear acceleration of the z axis.
2.2. Function IntroductionThe realization of the function is to identify the phone screen or the back of the tapping operation, the function of the process: Register the sensor, collect data, input a specified number of data calibration to obtain a more accurate linear acceleration, after calibration to determine whether the current steady state, if the non-stationary, then wait for the next data, if the steady state, The method is called to determine whether there is a strike operation, while the strike recognition will also deal with the resulting linear acceleration and the number of hits, steady state display to the interface,
2.3. Function Realization2.3.1. Getting sensor data The method of registering a sensor is a native method of the system, not too much to explain, but it is important to note that the parameters that identify the sampling interval of the sensor data when registering an accelerometer are best used with sensor_delay_game, because the acceleration data caused by the knock is very short, If using SENSOR_DELAY_UI or sensor_delay_normal often does not capture the acceleration change caused by the percussion, of course, if the use of sensor_delay_fastest naturally does not have this problem, but the performance will be more expensive.
After registering the sensor can wait for processing data in the callback method, below I give the implementation code, comprehensive code to explain the implementation process.

<span style= "FONT-SIZE:14PX;"         > public void onsensorchanged (Sensorevent sensorevent) {if (sensorevent.sensor = = null) {return; } if (sensorEvent.sensor.getType () = = Accelerometersensortype) {Float Accelerationz = Sensoreve            NT.VALUES[2];                if (Accelerationz > 0) {recognitionknockratio = 20;                Recognitionuniqueratio = 10;            Smoothsectionmaxratio = 5f;                } else {recognitionknockratio = 7.5f;                Recognitionuniqueratio = 6;            Smoothsectionmaxratio = 2.5f;            } Gravityz = Alpha * Gravityz + (1-alpha) * ACCELERATIONZ;            Linearaccelerationz = Accelerationz-gravityz;                if (calibratelinearacceleration) {calibratelinearaccelerationindex++;                if (Calibratelinearaccelerationindex <= calibratelinearaccelerationsectionnumber) {return;    }            Calibratelinearacceleration = false; } if (Sensordatashowindex >= sensordatashownumber) {sensordatashowindex = Sensordatashownumb                Er-sensordatashowdurationnumber;                Iterator<?> it = linearaccelerationzshowlist.listiterator (0);                    for (int i = 0; i < Sensordatashowdurationnumber; i++) {it.next ();                It.remove ();            } mainactivity.updatesensordata (Linearaccelerationzshowlist);            } linearaccelerationzshowlist.add (Linearaccelerationz);            sensordatashowindex++;                if (!stable) {linearaccelerationzlist.add (Linearaccelerationz);                    if (Linearaccelerationzlist.size () >= stablesectionnumber) {stablerecognition ();                Linearaccelerationzlist.clear ();            } return; } knockrecognition (LinearacceleratiOnZ); }}</span>

Sensor data callback method for the acceleration sensor data obtained are processed separately , first of all, according to the z-axis acceleration of the positive and negative, for Recognitionknockratio,recognitionuniqueratio,smoothsectionmaxratio three variables assigned different values, as to why this processing, is because the actual tapping operation of the Android phone found that the accelerometer is sensitive to the positive impact operation feedback, the back of the operation feedback is relatively slow, the feedback to the data plane, the impact of the impact of the acceleration sensor data changes compared to the impact of a lot of striking back, Therefore, in order to hit the screen and the back to assign different values, but in fact, standing in the perspective of the phone, using the current data is completely unable to analyze the operation caused by the sharp change in the speed of the strike from the front or the tap on the back, so use the z-axis acceleration of the positive or negative to simple judgment, After all, the majority of the z-axis acceleration is positive, that is, the back of the phone to the ground, users are more likely to tap the phone screen, and the negative is the mobile phone screen biased to the ground, users are more likely to tap the back of the As for the cause of the percussion screen and hit the back of the accelerometer sensor feedback sensitivity of the difference is two, one is the accelerometer compared to the back of the screen closer, and also the Android phone shell problem, this is particularly evident on LG G3, LG G3 is a certain radian plastic shell , the sensor data changed on the back of the tap is much lower than the tapping screen, while the metal shell of the Samsung S6, the sensor data changes on the back of the tap is close to the tapping screen. In fact, the above three coefficients are empirical coefficients, and for different types of mobile phones as far as possible to provide different values, the reason can be see just said LG G3 and Samsung S6, once again feeling the diversity of Android phones, Android phones too many kinds, The different hardware design causes the coefficient to be applied on a mobile phone to be completely unusable on another phone, and if only those models, like the iphone, are handled a lot. The
then filters the acceleration to obtain linear acceleration, and the method of obtaining linear acceleration refers to the suggested method in the Android sensorevent source code:

<span style= "FONT-SIZE:14PX;" > * <p> * It should be apparent this in order to measure the real acceleration of * the device, the Cont    Ribution of the force of gravity must is eliminated. * This can is achieved by applying a <i>high-pass</i> filter.    Conversely, A * <i>low-pass</i> filter can used to isolate the force of gravity.     * </p> * * <pre class= "Prettyprint" > * * public void onsensorchanged (Sensorevent event) *    {*//alpha is calculated as T/(T + DT) *//with T, the Low-pass filter ' s time-constant    *//And DT, the event delivery rate * * Final float alpha = 0.8;    * * Gravity[0] = Alpha * Gravity[0] + (1-alpha) * event.values[0];    * GRAVITY[1] = Alpha * Gravity[1] + (1-alpha) * event.values[1];    * GRAVITY[2] = Alpha * gravity[2] + (1-alpha) * event.values[2]; * * Linear_acceleration[0] = event.values[0]-gravity[0];    * Linear_acceleration[1] = event.values[1]-gravity[1];    * Linear_acceleration[2] = event.values[2]-gravity[2]; *} *</span>

Acceleration is treated by high-pass filtering and low-pass filtering to eliminate the gravitational effects to obtain linear acceleration, but in the process it is necessary to pass a certain amount of data for calibration to obtain a more precise linear acceleration, Here we set the Calibratelinearaccelerationsectionnumber as the data length used to calibrate the data, Use Calibratelinearaccelerationindex and calibratelinearacceleration to control when the calibration ends.
After calibration, use the Linearaccelerationzshowlist storage to display the sensor linear acceleration on the application interface, and then, if it is in a non-stationary state, the steady state is determined, and if it is stationary, the strike is recognized.
2.3.2. Steady state identification As mentioned above, if the user is shaking the mobile phone and other operations, it will produce significant acceleration changes, it is likely to lead to false recognition, so the concept of a steady state, that is, the mobile phone acceleration is not a significant change in the state of a long time, Extended to the real situation is the user does not have a clear mobile phone status, strictly speaking, the general user in the mobile phone significantly mobile, such as shaking the phone while the possibility of tapping operation is very low, so the concept of steady state can be formally applied to the implementation of the function.
Already understand the concept of steady state, then we should define what the condition belongs to the steady state, what is non-steady state, below I give the implementation code, comprehensive code to explain.

<span style= "FONT-SIZE:14PX;"        > private void Stablerecognition () {int exceptionnumber = 0;        float Accelerationzvalue;        float minaccelerationzvalue = Integer.max_value;        float maxaccelerationzvalue = Integer.min_value;            for (int i = stableSectionNumber-1; I >= 0; i--) {accelerationzvalue = Linearaccelerationzlist.get (i);            if (Math.Abs (accelerationzvalue) > Maxstableoffset) {exceptionnumber++; } else {if (Accelerationzvalue > Maxaccelerationzvalue) {maxaccelerationzvalue = AC                Celerationzvalue; } else {if (Accelerationzvalue < Minaccelerationzvalue) {Minaccelerationzva                    Lue = Accelerationzvalue;        }}}} stable = Exceptionnumber <= maxexceptionnumber;           if (Stable) {if (linearaccelerationzstablesection = = 0) {     Linearaccelerationzstablesection = (maxaccelerationzvalue-minaccelerationzvalue)/2; } if (Linearaccelerationzstablesection > Maxstableoffset) {Linearaccelerationzstablese            ction = Maxstableoffset;        }} mainactivity.updatestable (Stable);        LogFunction.log ("Stable", "" "+ Stable);        LogFunction.log ("Exceptionnumber", "" "+ Exceptionnumber);    LogFunction.log ("Linearaccelerationzstablesection", "" "+ linearaccelerationzstablesection); }</span>

In this function realization process, the way to judge the steady state is to sample 50 points, and then calculate the absolute value of each point, if greater than the maximum deviation maxstableoffset is considered an anomaly, the anomaly is greater than the maximum number of outliers maxexceptionnumber is considered non-stationary, The inverse is considered steady state. After judging the steady state, if it is in steady state, half of the difference between the maximum acceleration of the z-axis and the minimum acceleration is considered as the fluctuation interval linearaccelerationzstablesection after rejecting the anomaly data. Maxstableoffset and Maxexceptionnumber are the same empirical coefficients, which are derived from the linear acceleration analysis of the different scenarios that Android phones actually provide. Now there is a problem, that is, if the original state is in the stable, and then the user suddenly operate on the phone, the mobile phone state to the non-steady-state how to deal with it, do not worry, this problem will be in the process of tapping recognition processing.
2.3.3. Tap Recognition now to the core of the entire feature implementation: Tap recognition, as mentioned above, can cause a noticeable change in the accelerometer data, but how do we use code to detect strokes and how to troubleshoot user errors caused by other actions on the phone? In fact, these problems will be dealt with here, and now I give the implementation code, comprehensive code to explain.

<span style= "FONT-SIZE:14PX;" > private void Knockrecognition (float Linearaccelerationz) {Float Linearaccelerationzabsolute = Math.Abs (lin        EARACCELERATIONZ);        float Linearaccelerationzabsoluteradio = linearaccelerationzabsolute/linearaccelerationzstablesection; if (Linearaccelerationzabsoluteradio > Recognitionuniqueratio) {uniquelinearaccelerationzlist.add (Lin            EARACCELERATIONZ);        Currentforecastnumber = Forecastnumber;                    } else {if (uniquelinearaccelerationzlist.size () > 0) {if (Currentforecastnumber > 0) {                currentforecastnumber--;                } else {Handleuniquelinearaccelerationz (); }}} if (Linearaccelerationzabsoluteradio < smoothsectionmaxratio) {float OFFSETW            eight = 0.001f; Linearaccelerationzstablesection = Weightedmean (Offsetweight, LinearaccelErationzabsolute, linearaccelerationzstablesection); }}</span>

Knockrecognition is the method used to deal with linear acceleration to confirm whether there is a percussion operation, first, the linear acceleration of the incoming parameter is processed to obtain the absolute value of linear acceleration, Then, if the ratio of the absolute value of the linear acceleration to the fluctuation interval is greater than recognitionuniqueratio, it is assumed that the cell phone is being subjected to force and that the linear acceleration is added to the unique linear acceleration list to determine whether it is a strike operation or another user operation. Conversely, if it is less than or equal to Recognitionuniqueratio, it is believed that the mobile phone is in a relatively stable state, at this time if the unique linear acceleration list length is greater than 0, if Currentforecastnumber is greater than 0, The Currentforecastnumber minus 1, if currentforecastnumber is less than or equal to 0, starts processing a unique linear acceleration list, and formally begins to recognize whether to strike in the process of dealing with a unique linear acceleration list. and whether the current state transitions to non-stationary. At the same time, if the ratio of the absolute value of the linear acceleration to the fluctuation interval is less than smoothsectionmaxratio, the linear acceleration absolute value is used to smooth the fluctuation interval.

here, You must have doubts about Currentforecastnumber, what does this variable mean, and why this variable is possible, because this one-stroke can lead to a unique linear acceleration of two close but discontinuous. Without currentforecastnumber this variable would result in a realistic strike that could be identified as two strokes.
and if the ratio of the absolute value of the linear acceleration to the fluctuation interval is less than smoothoffsetmaxratio, the linear acceleration absolute value is used to smooth the fluctuation range, because on the one hand the state of the mobile phone may change at any time, and the fluctuation range should change with the change of mobile phone On the other hand, there may be problems in the calculation of the fluctuation interval in steady state recognition, and it does not reflect the acceleration fluctuation of the current mobile phone correctly, so it is necessary to learn from the latest data to smooth the fluctuation range. And why the ratio is less than smoothsectionmaxratio is because the ratio is more than the smoothsectionmaxratio of the basic non-normal linear acceleration, not suitable for the smooth fluctuation range, If the real-world linear acceleration and the fluctuation interval ratio are basically more than smoothsectionmaxratio, it shows that the mobile phone is mostly in the unsteady state, waiting for a new steady-state recognition to reset the fluctuation range, as stated above, Recognitionuniqueratio,smoothoffsetmaxratio belong to the empirical coefficient, can be set independently.

<span style= "FONT-SIZE:14PX;" > private void Handleuniquelinearaccelerationz () {LogFunction.log ("linearaccelerationzstablesection", "" + Li        Nearaccelerationzstablesection);        int recognitionknocknumber = 1;        int uniquelinearaccelerationzlistlength = Uniquelinearaccelerationzlist.size ();        float Accelerationzoffsetabsolute;        float Maxaccelerationzoffsetabsolute = 0; for (int i = 0; i < uniquelinearaccelerationzlistlength; i++) {Accelerationzoffsetabsolute = Math.Abs (uniqu            Elinearaccelerationzlist.get (i)); if (Maxaccelerationzoffsetabsolute < Accelerationzoffsetabsolute) {Maxaccelerationzoffsetabsolute = acc            Elerationzoffsetabsolute; } LogFunction.log ("Uniquelinearaccelerationzlist index" + I, "" + Uniquelinearaccelerationzl        Ist.get (i));        } uniquelinearaccelerationzlist.clear ();            LogFunction.log ("Uniquelinearaccelerationzlistlength",    "" + uniquelinearaccelerationzlistlength);            if (Uniquelinearaccelerationzlistlength > unstablelistlength) {stable = false;            Mainactivity.updatestable (Stable);        Return } LogFunction.log ("Maxaccelerationzoffsetabsolute/linearaccelerationzstablesection", "" "+ (Maxacce        lerationzoffsetabsolute/linearaccelerationzstablesection));            if (Maxaccelerationzoffsetabsolute > Linearaccelerationzstablesection * recognitionknockratio) {            LogFunction.log ("Recognitionknockratio", "" "+ recognitionknockratio);            LogFunction.log ("Recognitionuniqueratio", "" "+ recognitionuniqueratio);        Knockrecognitionsuccess (Recognitionknocknumber); }}</span>

Finally to the last Handleuniquelinearaccelerationz method, as the name implies, is used to deal with a unique linear acceleration list, in this method, the strike recognition and steady state whether the transformation of the decision, If the unique linear acceleration list length exceeds the non-steady-state unique linear acceleration list length, it is assumed that the current state of the mobile phone state now transitions to an unsteady and end method, and if the maximum offset in the acceleration offset data list is found to be more than a certain multiple of the range, it is recognized as percussion.
3. SummaryAt this point, the process of tapping recognition is finished. In fact, I provide the method of stroke recognition is still a false recognition of the situation, iOS knock I have used, with the ability to match the price, the recognition rate is quite good, do not know whether they are through machine learning or other methods attributed to a set of their identification coefficients, Of course, I am here to provide the percussion recognition is only a method of percussion recognition, I can not say that it is mature, because it has not been a real user test, we can completely follow their own ideas to replace the algorithm and even replace the sensor to achieve their own percussion recognition, and I am in fact equivalent to provide a realization of ideas.
This is the third blog, the first blog belongs to the test water has chosen to do a relatively biased door but not a good deal of a small module: write ID3 tags for MP3 files, the second blog selected a very rigorous practical module: audio synthesis, the first two modules have a common denominator is that various specifications have been very clear, Although the code implementation may be different but the implementation of the same ideas, and the third blog of the percussion test is undoubtedly a lot looser, so I also wrote the first time to implement the idea of this section, because I am not sure whether my implementation of the idea is completely correct, as the actual application of the sensor there are countless possibilities, We can completely follow our own ideas to try, the wrong big deal in a different direction.
In addition, if there is any problem in the blog or code, welcome all friends to come forward.
This is the end of this blog post, all the code in this article has been hosted in Https://github.com/CrazyZty/KnockDetect, you can freely download.

Talking about how to use code to realize mobile phone tapping recognition