Ext: 58 The road of the architecture evolution of high performance mobile push push platform in the same city

Ext.: http://geek.csdn.net/news/detail/58738

Wen/Sun Xuan

This article describes in detail the three stages of the evolution of the 58 high performance mobile push push platform architecture, and introduces what is mobile push push, why it is needed, how to compare the principle and scheme, and how to design the first stage (single platform) architecture of mobile push push, and solve the typical performance problem of mobile push push. As well as high availability, high performance, high stability how to guarantee.

What is mobile push push

Mobile push Push is one of the most basic requirements of mobile Internet, which is used to meet the message arrival app client in mobile connected environment. Take a turn (58 fair's real personal idle trading platform) For example, when the buyer placed a single, we through the mobile push push message to tell the seller, when the seller has shipped, we through the mobile push message to tell the buyer, so that both buyers and sellers in time to master the real-time orders of second-hand commodity trading dynamics.

Why do I need to move push push?

Mobile Internet environment, often appear weak network environment, especially 2G, 3G and other network environment, the network is not stable, the app client and the corresponding server side of the long connection has been disconnected, the message can not touch the app client. And our business needs to send messages (such as the transfer of app trading messages, etc.), operation (Transfer app operations, etc.), alert (non-consumption reminders to the red envelopes), etc. to the app client, triggering the user to see these messages, Reach the target by clicking on these push messages.

Push principle and scheme comparison

There are three main ways to implement mobile push push.

1. Mobile app polling method (pull)
   The app client periodically initiates a push message query request to achieve the purpose of the message push. The advantages and disadvantages of the pull scheme are obvious, the overall architecture is simple but the real-time is poor, we can improve the frequency of the query, increase the real-time, but this will cause power, traffic consumption is too high.
2. Mobile app based SMS push mode (SMS push)
   Send a push message via SMS and place the SMS Blocker module on the client to intercept the SMS and post it to the app processing. This scheme has good real-time and high arrival rate, but it has high cost.
3. Mobile app long connection mode (Push)
   Mobile push push is based on TCP long connection implementation, the message real-time good, this is the current mainstream implementation, need to maintain the app client and server side of the long connection heartbeat, will bring additional power, traffic consumption, in the architecture design, need to make some compromises to avoid the traffic and power consumption. In addition, the push push technology architecture is highly complex, maintaining a large number of long connection requests for mobile app clients, and establishing an encrypted channel to communicate with the app client, which integrates into a small amount of limited long connections inside, compressing and decompressing the communication data to save traffic.

At present, mobile push push technology is basically combined with these 3 scenarios, but for different mobile terminal platforms, there are different implementations, here in detail on the iOS and Android platform specific implementation.

iOS platform

For the iOS platform, because of its particularity, mobile push push is relatively simple, iOS app is not allowed service background resident, so you have no other choice, there is no way to develop their own push service to complete the push delivery, only through the Apple APNs way to complete. The iOS mobile push push process is shown in 1.

Figure 1 iOS mobile push push process

Android Platform

On the Android platform, there are more options available because there are no restrictions on service residency: it can be done through Google's official C2DM, open source solutions (such as XMPP), third-party, or fully self-developed mobile push push solutions.
Google C2DM is shown in the main process 2.

Figure 2 C2DM Mobile push push process

Google C2DM and the Apple APNs process are roughly similar, but the biggest problem is that mobile push push servers are easily blocked abroad, and push push delays are large. In addition, because the Android Community Division is more serious, many manufacturers directly removed the C2DM module, so in the domestic this program is extremely unreliable, has become a theoretical solution.

Mobile Push push Open source solution

For the open source mobile push push protocol, common has XMPP, etc., in fact, Google's C2DM base is based on the XMPP protocol implementation, we found through offline testing, open source mobile push push solution has two main problems: first, there is no ACK mechanism, the message arrives no guarantee, unreliable Second, when the number of mobile push message requests increases concurrently, the system begins to become unstable and even the module goes down. Therefore, the direct use of mobile push push open source solution is not very reliable, I personally recommend that: before the large-scale use of open-source mobile push push solution, must be in the open source technology solution overall grasp, otherwise, if there is a problem, can not be timely positioning and repair, the consequences will be disastrous.

With third-party mobile push push scenarios

In addition, there are a number of third-party push products available on the mobile push push market, but there are several issues that need to be faced:

• Arrival rate
  Although the third-party mobile push push product has a promotional reach rate of more than 90%, but actually used, found far beyond. Of course, the problem of low arrival rate, in addition to the third-party mobile push push platform itself technical reasons, but also with the business push the user selection has a great relationship, if the user is more active, the arrival rate will be higher, if the user is not active, or the user has uninstalled the corresponding app client, will inevitably cause the arrival rate further reduced.

• Real-time & degree of control
  The push channel of a third-party mobile push push product is shared and will target multiple push customers, and if one customer pushes a particular push, other real-time messages may be impacted, which are not controlled by the business push and will be more passive.

Fully self-developed mobile push push solution

We have considered the implementation of a fully autonomous mobile push push platform, if from scratch to do, need to solve a few difficulties: first, the mobile push push service to the mobile app client large-length connection maintenance management. Second, the app client resident service stability, how to make the push service resident? We can use the parent-child process to monitor each other in a way to do, once the other process is found, will be re-established, continue to cycle monitoring. Third, when the phone memory is low, the system will kill the push service, and even some operating systems are more powerful, it will be like iOS system does not allow the third party push service resident. Finally, the increase of the mobile push push arrival rate, in addition to technical means, there are some means of PR, such as mobile app Client Push service by adding a whitelist on the appropriate operating system to make it permanent permanent. In short, it is not a simple matter how to make mobile push push reach higher in a complex mobile-connected scenario.

58 the same city mobile push push solution

We consider the above-mentioned open source, third-party, fully independent research and development programs, 58 of the same city did not choose to completely independent development from scratch, but the adoption of a third-party mobile push push platform and self-developed high-performance provider Scheme (3), to meet the daily tens of millions of levels of throughput, and by dynamic combination and expansion of the way, combined with offline mobile push push data analysis, different mobile phones using different push strategies, targeted optimization. On the Android platform, we combine a variety of third-party mobile push platforms to effectively increase the reach rate.

Figure 3 58 The same city mobile push push platform technology architecture

First stage (single platform): How the architecture is designed

Background & Requirements

2011 we developed 58 help, this is a 58 users and merchants to communicate between the Instant messaging software, users can add friends to each other, send and receive messages and so on. 58 Help messages push a long connection based on the app client and server, and once the long connection is disconnected, the messages from the IM server will not be pushed to the app client and the user will not be able to see the messages. On the iOS platform, after the 58 help app switch to the background, the app and im long connection disconnect, the message can not reach, this time we need to use the iOS APNs mechanism, IM message needs to send to Apns,apns and then forward the corresponding message to the 58 Help app. Android switch to the background, the app and im long connection is maintained, IM messages can be pushed normally, so at this stage we need to solve the problem is on the iOS platform, when the 58 Help app cut Backstage, im in long connection after the message reached the demand.

Design goals

Based on the above background and requirements, when we design the mobile push push first stage (single platform) architecture, we must first meet on the iOS platform, when the IM long connection disconnects, the IM message can touch the app client. Second, our mobile push Push protocol design is also very scalable, and in the foreseeable future, the push push platform will gradually access more apps, so we designed the target iosprovider to be a generic iOS push service. Different apps use the same iosprovider to push the mobile push message by using a different mobile push-push certificate, and for different apps, we dynamically extend the access using the configuration file mode, Iosprovider establishes and maintains multiple TSL connections with APNs based on the app certificate configured. The format of the configuration file is as follows:

第一个域#第二个域#第三个域#第四个域

Where the first domain is the push service type, in case of expansion, 1 is APNs, the second domain is an internally defined AppID number, the corresponding service app, the third domain is the Apple certificate file name for the app, and the fourth domain is the number of connections established with APNs;
Each app access is configured as a single line, for example:

#88 #zhuanzhuan.p12 #641 #66 #58tongcheng.p12 #32

In addition, Iosprovider needs to manage the APNs connection pool for each access app, dynamically add and remove TSL connections, have a dynamic reconnection mechanism, and have a separate feedback receive thread, which is used to asynchronously receive APNS return invalid tokens, feedback to the mobile push push business party, Optimized for the next mobile push push message. Iosprodiver selects the corresponding APNs connection based on type, AppID, and sends the APNS package to the APNS server by pushing the thread assembly, as shown in 4.

Figure 4 Iosprovider Frame composition

Second stage (multi-platform): How the architecture is designed for optimization

With the advent of the mobile internet era, 58 have developed a number of apps, each app has mobile push message push demand (messages, operations, overdue reminders, etc.), and each app has multiple terminals: Android, iOS and so on. In this context of demand, our mobile push push platform needs to continue to evolve, how to evolve it?

The iOS mobile push push channel can be a good fit for business push, but it doesn't have the ability to push Android mobile push, so we need to develop an Android mobile push push channel. How to do? Combining the options currently available, we have selected solutions based on third-party push platforms and self-developed high-performance androidprovider.

First of all, focus on the process for Android mobile push push: First, the app client to the third party mobile push push platform registration, to obtain the corresponding app unique logo (Token). Second, the app sends token information to Androidprovider and stores it centrally for subsequent token-based mobile push push. Thirdly, Androidprovider connects to a third-party mobile push push platform via HTTPS or TSL and sends messages that need to be pushed to a third-party mobile push push platform. Finally, when the third-party mobile push push platform receives the ANDROIDPROVER push message, it pushes the message to the app in time to complete the push process, as shown in 5.

Figure 5 Android mobile push push process

The overall structure of the Androidprovider subsystem is divided into four levels, the first layer for the business party mobile push push access, for many mobile push push business party access. The second layer is the network interaction layer, which is used to receive the message data of the mobile push push business party and to send the processing data of the request processing layer to the business push call. The third layer is the request processing layer, which handles the data that the network interaction layer puts into the request queue, assembles the data required by the third party mobile push push interface, invokes the downstream interface through HTTP or HTTPS, waits for the return of the request result, and puts the result returned by the request into the response queue. The fourth tier is a third-party mobile push push platform, provided by a third party and opened to the consumer interface for invoking its functions, as shown in 6.

Figure 6 Androidproiver System architecture diagram

As more and more mobile app access, mobile push push demand diversification, while mobile push push business logic complex (multi-terminal, batch send, business rules diverse), public policy each business party development (late night anti-disturb function, sending frequency and transmission rate limit, etc.), resulting in inefficient development. To solve these problems, we abstracted the common logic, and made a uniform encapsulation, transparent to the business callers, these common logic include: Common strategy and general control, 7 shows.

Figure 7 Android mobile push push evolution Business architecture

In the second stage of mobile push push (multi-platform), we have the ability of Android, iOS channel service to meet the demand of push message. However, we do not provide a unified delivery interface, the business party needs to send the respective package (Android, IOS) different push channels, in addition, the performance of the push channel has yet to be improved, the push channel stability has yet to be improved, in addition to the push channel contains the relative common business logic, the push channel is not "pure".

Phase III: How architectures and protocols are designed and optimized

Mobile push Push the second stage also has a series of problems, so in the third phase needs to be addressed, and as more apps are accessed, we need to provide a unified, high-performance mobile push push platform for the company. Based on the third-party mobile push push platform, we have developed high-performance provider that meet the daily push of tens of millions of levels, the push platform has high stability, easy access, and provides a high push arrival rate.

Mobile Push Push Platform Phase III how do we architect and design? First we meet the downstream access to a variety of connection management (HTTP, HTTPS, TCP, SSL, TSL), with a variety of connectivity dynamic scalability, so as to meet the provider layer on the mobile push push connection requirements. Second, the platform to have high concurrency features, through a fully asynchronous design and multi-threaded support, to achieve high concurrency and support 100,000 QPS throughput. Again we need to deal with errors downstream of the access, once the connection has been disconnected and other errors, to be able to automatically use the new connection, and sent to the app client has been issued to send a push message to re-send, to ensure that the message is not lost. We need to encapsulate the channel and provide a unified and friendly access interface to shield the difference between the underlying iOS and Android access. Finally, in the context of Android mobile push, we have access to more third-party push platforms to achieve higher reach rates.

Based on these considerations, the 58 mobile push push platform in the same city uses a low-coupling layered architecture Design (3), divided into three-layer push Entry, push Transfer, Provider (Iosprovider and Androidprovider). The push entry is the entry of the business party call, we adopt the way of asynchronous message queue, provide the speed of high business party sending, and have the function of message buffering, so that the peak mass mobile push message pushes to the whole platform less impact, also play the role of protection push system. Push transfer will receive messages from the push entry layer for parsing, legality checking of push messages, and if the format is not valid, it is discarded directly, and the received push message format is converted into the internal message format. The sub-platform forwards to Iosprovider or Androidprovider, provider receives the message of the push transfer, it is converted according to the message format (APNs protocol, Android protocol) that is needed downstream, and the message is issued. In the process of issuance, the message will be re-issued to ensure that the message sent to the third-party push platform.

How is the interior of the provider module designed? Take Iosprovider as an example, it is divided into three levels: access logic, business logic, APNS export. The access logic mainly deals with the network interaction and the request distribution, and the business logic mainly deals with thread splitting extension, concurrency processing and error handling, and the sending logic of APNs egress processing to APNs, 8.

Figure 8 Iosprovider Module Structure diagram

For the mobile push push platform, the pursuit of the rate is our most core indicator, not one. So in Android, we have integrated a number of third-party push platform, through the model control, the different models using different channels, and further improve the push arrival rate. Androidprovider layer for the control of the message push policy, first push a channel, according to this push channel ACK situation, whether to continue to push other channels. Push multiple push channels, the push message will be repeated to the app client situation, the app client needs to be based on the push message ID deduplication, received repeated push messages ignored processing.

Typical performance problem analysis and resolution and how high availability, high performance, and high stability guarantee

In the continuous evolution of mobile push push, we encountered the problem of low androidprovider concurrency, careful analysis, because we use the HTTPS library, because the library HTTPS connection implementation is not thread-safe, the request for each HTTPS is locked serialization processing, To ensure the security of the thread. After discovering the problem, we temporarily resolved it by adding multiple process deployments online, giving us enough time to analyze the root cause of the problem. After in-depth analysis, found that the reason is that we do not have enough knowledge of the HTTPS library, resulting in a large lock grain, through the HTTPS library to provide a smaller granularity of the lock, we not only solve the problem of thread insecurity, but also increased the androidprovider concurrency, 9.

Figure 9 HTTPS Library fine-grained lock implementation method

In conclusion, the 58 unified high performance mobile push push platform ensures high availability of the push platform through stateless design and redundant deployment, providing high performance of the push platform through pure asynchronous, dynamic multithreading support, through quality assurance, multiple monitoring mechanisms (process monitoring, semantic monitoring, error log monitoring, Data fluctuation monitoring, etc.), there are problems in time to find processing to ensure the high stability of the push platform.

Finally, I would like to thank the project team of students, special thanks to Yiu Jing classmate, with your continuous efforts and pay, only today this article, but also thank your wife, you are behind the silent support, only today this article.

Sun Xuan: 58 Market Group System architect, technical Director, Technical Committee structure group director, is also 58 with the city of instant messaging, the head of the technology, responsible for the 58 core system architecture and optimization work. Distributed System storage experts, former Baidu senior engineer, participate in the Community Search Department of a number of basic system design and implementation.

This article is "programmer" original article, without permission not reproduced, subscribe to the 2016 "programmer" click http://dingyue.programmer.com.cn

Ext: 58 The road of the architecture evolution of high performance mobile push push platform in the same city