Sweep blessing behind, pay treasure ar red Envelope technology innovation and story __ system architecture

During the Spring Festival, Alipay's "sweep blessing" activities are unusually hot. AR is a kind of new interactive way, compared with traditional marketing method, it can make users deeply participate in interaction and bring new experience to users. and pay Bao red envelopes, pinned to the future of the user's expectations, so its team will consider AR and red envelopes combined to explore a new play. In the AR field walk in front of Alipay, its AR red envelope technology selection, technical framework and the technical stories behind it. It's written in front.

December 12, 2004, I joined the Yahoo China, also joined the Internet Army. So far, I still vaguely remember the feeling of awe of the Internet at that time. 2009, in Yahoo four years of foreign life Flash, I was fortunate to join the Alipay team, and participate in paying the treasure wireless payment. Since then, I have to accept the wireless wave of baptism every day, the trajectory of life has also undergone great changes.

Today is my No. 4096 day to Ali, talk about "red envelopes", let me recall, 2015 to participate in the first time to pay the Treasure Spring Festival Password project, the team won the Ant Golden Service CEO Award, 2016 responsible for the identification of monkeys, began to have an AR embryonic form, followed by the children's time "set card" play, It is a five-year-old predecessor; 2016 began to build Alipay AR system, fully supported the 2017-year AR real red envelopes, AR sweep Blessing and a series of activities ...

Mention AR red envelopes, this is a bold innovation of our team, now to share with you the technology and the story behind Alipay ar red envelopes. Alipay ar red Envelope background ar development and old play

AR is the abbreviation of Augmented reality technology, its history can be traced back to 1968----------------------Professor Sutherland, Harvard University Electrical Engineering, invented the "Sword of Damocles" head-wearing display

In 1998, Sportvision, a leading company in the field of sports-graphic packaging and movement data tracking, used AR technology on television on the rugby game to overlay the scoring lines onto the field on the screen. Since then, AR technology has begun to be used in weather forecasts, where the computer images are superimposed onto realistic images and maps, and since then AR has really begun its explosive development.

In the 1999, Augmented Reality development tool Artoolkit came into being, and many developers joined the industry to push the field forward. At present, companies such as Microsoft, Google and Apple have invested heavily in the Ar field, aiming to drive the next generation of Windows interactive systems to change. Why do you pay Bao ar red envelopes

AR is a Mercedes-Benz train, from the individual to the team, from the team to the company, are hoping to catch the train, but how to catch up with what technology is a new challenge.

AR is a kind of new interactive way, compared with traditional marketing method, it can make users deeply participate in interaction and bring new experience to users. and pay Bao red envelopes, pinned to the future of the user's expectations, so the team will consider AR and red envelopes combined to explore a new play. AR New Play

In the 2015, we began to pay attention to the field of AR, which was developed in 2016 for a New Year red envelope program is to identify monkeys with mobile phones. This program let everyone have the feeling of AR, the project team also want to do some AR innovative play. However, because of the lack of relevant technical reserves at that time, they gave up.

Although the scheme gave up, but to explore the new way ar play this thing, deeply branded in our hearts. In the spring of 2016, our innovation group began to reserve relevant technical knowledge: identification tracking, OpenGL, 3D rendering engine, modeling capability, AR architecture, and so on. These related technologies have also been practiced in the social scenes such as small video beauty, face map and so on, which provide the guarantee for the subsequent AR related technology rapid prototyping.

How to promote the new play of AR. At that time, our team set two goals, one is to find a small scene of the AR concept, the second is in the Spring Festival red envelopes using AR technology, the latter is our annual ultimate goal.

2016 "Mid-Autumn Festival sweep the Moon", is Alipay in the field of AR first taste, experience and response is not bad. Take this first step, a variety of business needs are coming, AR new play is more and more-double 12 line under the Business Circle "sweep double 12 set four treasure" activities, 2017 Spring Festival ar Real Red envelopes, sweep Coca-Cola Fuwa collar Red, sweep Fuji Fu ... The challenge of paying Bao ar red envelopes

New play will bring new challenges, do not say anything else, just how a way to present AR red envelopes, is a very big challenge.

Most of the AR play is for the preset 2D pictures, off-line training good identification model, and then realize the recognition interaction. Because it is 2D pictures and verified beforehand, so it can better guarantee the effect of recognition.

Alipay ar virtual red envelope is a bold attempt in the industry. Users are free to choose the real 3D scene, take a picture as a clue map, others according to the clues to the location of the map to shoot the picture to compare. And this way presents AR red envelopes, natural also to the image matching algorithm performance proposed a new challenge, mainly related to the following aspects.

Algorithm robustness: Due to the real 3D scene photography, matching algorithm needs to tolerate a certain occlusion and angle, distance, light, picture quality and other changes;

Computational complexity: Due to real-time matching on the terminal, the complexity of the algorithm can not be too high, otherwise it will lead to the user to look for red envelopes longer, excessive consumption of mobile phones and other problems;

Mobile phone traffic: Mobile end applications need to consider the user's traffic problem, the need for data size and recognition ratio between the balance;

Mobile compatibility: To have billion-class users of Alipay, models, especially andriod models are widely distributed. For our three-dimensional model rendering has brought great challenges, we need to develop a comprehensive compatibility scheme to ensure the normal use of AR function. Technology selection of AR real red envelopes

The initial scheme of AR red envelopes is similar to many existing products, relying only on lbs and mobile gyroscope to interact: users send a red envelope at a certain location, others turn on the camera when they reach the location, and they can see the red envelopes from others when they are shot in a certain direction. This scheme is more mature because it relies only on lbs and handset gyroscopes, but there are two main problems:

Positioning accuracy: The initial positioning of the product is to let nearby people to receive red envelopes (such as dozens of meters near the person or the same floor of the people), which is very high requirements for close range positioning. There are a number of potential solutions, including WiFi positioning, indoor positioning, etc., but are not very common or immature, there is no good solution;

Subjective experience: Due to the GPS and gyro data itself there is a large error, location and user action can not be accurate, user experience is not very good, there is no real hidden ↔ to find the experience.

Therefore, in AR red envelopes, we hope to redesign the interactive mode and propose a picture matching scheme. In sending red envelopes, the user takes a picture of the scene; others want to get a red envelope, they must reach the location and shoot the same scene, only the location and picture matching to receive, greatly increased the accuracy and user interest. Finally, the program was adopted, the emergence of the AR real red envelopes. The final technical framework of AR real red envelopes ar real Red envelopes

The whole AR virtual red envelope consists of two scenes: one is to hide the red envelopes, the user to a scene to take a picture, the red envelopes hidden in this scene; the second is to find red envelopes, users in the map page to select a red envelope, go to the corresponding location through the clue map, open the camera to shoot for the picture, matching the success of a red

Hidden Red Envelopes

Not all the scenes are suitable for hiding red envelopes, such as white walls everywhere, will create a clue map is not unique situation. Therefore, when we hide the red envelopes, we will verify that the current scene satisfies the preset requirements. After the success of the possession of the picture will be uploaded to the server processing: first, the picture into a signature for image matching, and the second is to generate a clue map for red envelopes. Looking for a red envelope

According to the user's current position, the service end will be issued near the red envelope information. The user can choose to pick up a red envelope through a list of red envelopes, at which point the client will pull the signature and clue map of the red envelope to the service end. The clue map is displayed locally, prompting the user for a red envelope. The signature is used for client-side local real-time alignment. When a user is looking for a red envelope, the image captured by the camera is converted to a signature, compared with the red envelope signature issued by the service end, if the local match is successful, the picture information is passed to the service side for two times to prevent cheating. After the service-side check is passed, the red envelopes can be collected. The blessings of Esau's blessing

In the activities of the FU-fu card, in order to solve the problem of high concurrency, using the client + server-side parallel processing architecture system, can support two kinds of identification methods:

All pictures are transmitted to the server to do processing, so that the recognition accuracy is higher, but the number of servers can be handled is limited; the second is to go first client detection, the client can not recognize the upload service. The client detection ability is slightly weak, but the computation ability is dispersed to each terminal, can alleviate the pressure of the service side greatly.

Activity opened, due to the active participation of users, we opened the client identification early in the activity, so that the system successfully supported 140,000 times per second of the peak word recognition, to ensure the smooth conduct of the entire activity. But the client's recognition accuracy is limited also causes is not the blessing word is is the blessing word question, this also is our follow-up optimization direction. How to prevent falsely

Normal users to obtain red envelopes successfully, need to follow clues to find red envelopes hidden location and complete real shooting, that is, location + image matching two conditions need to meet, can receive red envelopes.

AR Real Red envelopes at the beginning of the line, in order to let users have a better experience, clues to do a more concise and beautiful. The first version of the clue is to block the original image by adding a uniform horizontal stripe mask (similar to the shutter effect), while ensuring that the user can easily identify the picture.

However, because the scheme is relatively simple, some users through the PS clues to complement the total occlusion part, the follow-up even appeared the automated PS tool. We then upgraded, each of the clues to the red envelopes are dynamically selected a mask, the mask will also be random dynamic rotation/offset and so on. The new clue diagram scheme greatly increases the difficulty of automatic PS, and solves the problem of PS basically.

In addition to dealing with images of attacks, for lbs tampering and other cheating means, ant security team has done a lot of prevention and control technology, through the user's behavior data to analyze potential users of cheating, but also effectively intercept most of the cheating users. The future of AR

Ar is a novel form of interaction, although the field has been a long history of development, but the technical maturity and the popularity of users are not very high. Currently on the market AR application scene is still relatively narrow, there are still a lot of digging space, especially in combination with the characteristics of various industries to think, will certainly produce a lot of good creative play.

AR Red Envelopes officially on the line, those days map pages are full of red envelopes, we are also very excited, that everyone is interested in this new thing and can accept. In addition to personal red envelopes play, many businesses are more interested in this new type of interaction, but also actively contact us. Businesses want users to film their own brand logo to deepen the user's awareness of the brand, that this is a novel marketing approach, there will be more growth in the future.

Follow-up we will be in the direction of the AR Open platform to provide an open AR platform for more businesses and users to participate in. At the same time, the combination of offline payment scene, around the user experience, to create more innovative products. Written in the last

The birth of the first version of Windows Mobile wallet made me move towards mobility, and then I led the team through Symbian from peak to recession, and then through the tangle of WP projects that have gone through Android from development to present, Experienced iOS from strange grammar to later ...

Technology is evolving, and we are constantly innovating. 2013 opened the payment of PO Sonic Payment, 2014 bank card identification and identification for the wallet business ... At the same time, team-related algorithm technology, engineering ability and innovative thinking in the process also get a lot of precipitation, for three consecutive years to participate in the Red Envelope Project laid the foundation for this year AR Red envelope new play laid the foundation. The greater the capacity, the greater the responsibility, since the commitment to the responsibility, I believe that our efforts will not be a mission.