1 billion times + message, 4 billion times + Total Request 丨 58 Help For example to see 58 typical technical framework evolution

1 billion times + message, 4 billion times + Total Request 丨 58 Help For example to see 58 typical technical framework Evolution

Source: "Programmer" October 2015 period

Author: Sun Xuan

The author introduces: 58 Market Group System architect, technical Director, Technical Committee of the Organization of the Director, is also 58 in the city of Instant messaging, C2C technical director, responsible for 58 core system architecture and optimization work. The Distributed system storage expert, former Baidu senior engineer, participates in the Community Search department several basic system design and realizes.

58 with the city as China's largest life service platform, covering the real estate, recruitment, second-hand, second-hand cars, yellow pages and other business, in each business category can be seen to facilitate the exchange of user Communication 58 help. This article selected 58 help as the 58 representative of the city's typical technical framework, described in detail the 58 help since the line, with the user volume, data and product direction of development, 58 help in the technical framework of the continuous evolution. 58 Help by Instant Messaging (IM) and non-Im business processing part of the composition, at present, the entire help system to handle 1 billion times a day + send messages, add friends and other traditional IM requests, as well as 3 billion + non-Im business request operations, the total request to reach 4 billion times +. The help has also broken through 1 million online, and so many requests and online users have brought us a lot of challenges.

This article details the 58 Help technology architecture evolution of the four stages: Phase one: the traditional IM architecture how we design. How to meet tens of millions of online performance requirements. Phase two: from the traditional IM to the merchant management platform, how the 58-help architecture evolves. Phase three: From the Merchant management platform to the mobile marketing tool, how the 58-help architecture evolves. Stage four: How to build a mobile push system that satisfies 58 help

Stage One: Traditional IM

The first 58 help is only a traditional IM, mainly to meet 58 users and 58 merchants to communicate and transfer information, core functions include adding friends, user relations, send and receive messages. In the client product configuration is divided into PC desktop clients, Web page clients, of which the PC desktop client mainly serves 58 merchants, 58 users use web page client and 58 merchant communication. For the traditional IM use business scenario for phase One, we designed the following technical framework [Figure 1]:

Chart 1 58 Help technology architecture A

The entire technical architecture of the traditional IM server is divided into five tiers:

Access Layer (ENTRY)

As the interface subsystem of client and server, this layer directly faces the massive long connection request of PC desktop client, Web Web page client, mobile apps client, and is responsible for establishing the encrypted channel of communication with client, integrating a few limited long connections inside, compressing and decompressing the communication data, and forwards the corresponding request to the logical layer (LOGIC). In addition, the access layer to implement a preliminary attack and defense strategy, used to resist the illogical level of attack, monitoring some data (connection frequency, contract frequency, contract rate, etc.), ip/uid and other indicators to ban, im in some emergency situations, must be able to quickly implement ban, so need real-time authorization ip/ UID, such as black and white list, so that the white List dynamic entry into force, blacklist automatic closure.

Logical Layer (LOGIC)

The logical subsystem is mainly responsible for the entire 58-help business logic processing, including user-related (user login login, user information set up query), friends related (add friends, get friends, delete friends, modify friends information, etc.), message-related (send and receive friend messages, send and receive strangers messages, message confirmation, common message processing, Off-line message, etc.) and other complex logic processing.

Routing Layer (ROUTER)

The routing subsystem mainly deals with the data related to the user's logon session at a time (for example: User online status [online, leave, stealth], login IP, etc.), this part of the data involved changes very quickly, there is no need to cure, directly stored in memory. It also records the routing information of the user login access layer and supports the function of communicating to the specified UIDs.

Data layer (Database ACCESS)

The unified data access subsystem masks the underlying storage engine, and the upper layer does not care whether the actual storage is an RDBMS or NoSQL or other KV storage engine, providing a friendly, unified access interface to the upper layer and encapsulating atomic logic. And through the curd interface of the abstract encapsulation, adding new data operations only need to increase the configuration, without changing the data layer code, easier to complete the storage of critical data.

Data storage layer (STORAGE)

The data storage layer is a 58-help data-curing storage system that uses persistent storage such as RDBMS (MySQL), NoSQL (MongoDB) and so on, and accelerates queries through distributed caching (Memcached, Redis, etc.).

Phase one of the traditional IM architecture is layered low coupling, in each layer we adopt a stateless design, each layer of subsystems can be dynamically expanded to meet the demand growth requirements. In each layer we deploy through the subsystem redundancy, eliminating the single point and ensuring the high availability of the system. The call between each layer is designed to be dynamic load balancing, and when the underlying subsystem fails, the upper subsystem call automatically switches to the available service node and continues to complete the user request operation. Single-line support 50w+ online users, the single line reached the 3W+QPS to meet the Tens users at the same time online needs.

Phase two: from traditional IM to business management platform

With the continuous development and change of products, 58 help is no longer limited to the traditional IM, but a gradual evolution to the business management platform, continuous access to real estate, recruitment and other classified information services, providing business management functions, post, post refresh, sticky posts and other operations. More features, such as recruitment resume recommendation, real estate publishing needs to complete the operation and display on the client. In response to these changes, the traditional IM architecture has been adjusted, the main changes are reflected in the client. Technical framework [Figure 2] is as follows:

Chart 2 58 Help Technology Architecture II

Third-party business (recruitment, real estate, second-hand cars, etc.) access, and IM business type, from a technical point of view, in order to meet the corresponding product features, the long connection is not dependent on a more elegant implementation, so you do not have to continue to use the long connection, We used a client to invoke a Third-party service over HTTP, the development efficiency is very high, the third party business can quickly access, but because the client and the third party business coupling tightly, brings many compatibility difficulty, once the third party business strategy changes, the interface changes, the client will update with the adaptation change, Client upgrades cost very much, and can not be real-time updates, such as the need to upgrade, users can choose not to upgrade, these will bring a lot of trouble.

To address the problem of "simple brute" invocation by the client via HTTP, the infrastructure of the traditional technology architecture continues to evolve in the phase, forming the following architecture [Figure 3]:

Chart 3 58 Help technology architecture Three

The evolution of the above technology architecture, we found that the client calls the third party business by the HTTP direct invocation of the way to pass through the 58-Help server relay calls. The client no longer has direct "contact" with third party business, all third party requests will pass 58 to help service side relay, 58 help service side continues to call the third party business interface, even if the downstream service strategy changes, the business change, we only need 58 help the service side to make the corresponding adjustment, to 58 help client completely transparent, This avoids the costly problem of client upgrades. However, the problems of this technical architecture are also obvious: the first to first time user requests to pass through 5 subsystems (Client->entry->logic->extlogic->transit->destservice), 5 times Network Exchange, the interaction path is long, the response delay is high. Second, there are problems on the line, positioning problems need to see at least 5 subsystem log, increased rapid analysis and problem solving problems, high cost. Third, the client through 58 to help the server to invoke the third party business, still used is the way of TCP long connection, with the advent of mobility, mobile network is relatively not stable, just imagine using 58 to help refresh second-hand posts, into the elevator, the network instability, refreshing function can no longer use, prompting you users have dropped the line, Please login again after use, such a user manifests must be very bad.

Phase three: From Business management platform to mobile marketing tool

With the business development and product changes, we from the business management platform evolved into a business with mobile marketing tools, positioning for business mobile platform, to meet recruitment, real estate, used cars, yellow pages and other classified business businesses. The 58 Help technology framework [Figure 4] continues to evolve as follows:

Chart 4 58 Help technical Framework Four

From the product level, we have access to more classified business, this part of the business in the technical framework we have adopted a more lightweight WebService services, the client uses HTTP to invoke the 58 help WebService services, WebService services and then call downstream services, Through this framework, the first: we solve the Third-party service changes in the impact of client upgrades; second: Reduce the dependence on TCP long connection, stability and user experience better; third, users only need 2 times network interaction, the response delay is improved; four: The user requests the call path to shorten, we locate and analyze the online problem , faster and more efficient.

Through the above technical framework can be seen, 58 help the final structure is divided into two parts: IM-related (users, friends, messages, etc.) we adopted a TCP long connection in the technical framework to ensure the timely arrival of the message; third-party business part we adopt the HTTP short connection way, At the technical level to reduce dependence on long connections, while satisfying product features, the user experience is better.

Stage four: How to build a mobile push system that satisfies 58 help

With the 58 help to move in depth, we need to build their own mobile push system to solve the mobile environment network instability, app application can not reach the problem, so that message, Operation, alert and other types of messages to the user.

The main three ways to implement mobile push:

1. Client polling (pull)

The client periodically initiates a query request to achieve the purpose of the push. Pull's advantages and disadvantages are obvious, simple architecture but poor real-time, want to improve real-time, only to speed up the query frequency, but this will cause electricity, flow consumption is too high.

2. SMS Push

Send the message through the SMS, and in the client-side SMS interception module, will receive the SMS interception, and analysis and then forwarded to the application processing. This scheme has good real-time performance and high arrival rate, but its cost is very high.

3. Service-End long connection (push)

This is the mainstream implementation of the current, real-time and good, and low consumption of electricity, technical structure of a high degree of complexity.

The current mobile push technology is combined with these 3 aspects, but for different mobile terminal platform, there are different implementations, here a brief introduction to IOS and Android on the specific implementation of the program.

IOS Platform

For iOS is relatively simple, you have no other choice, iOS application is not allowed backstage resident, so you have no way to develop their own push service to complete the push sent, only through the Apple APNs Channel to complete the push, push process [Figure 5] as follows:

Chart 5 iOS Move push process

Android Platform