Order synchronization has skills, double eleven peaks are no longer afraid

As can be seen from the figure, the entire double eleven singles day order processing process mainly involves three systems: platform (Tmall, Taobao), ERP/OMS (used to process orders), WMS (package, delivery within the warehouse) . It includes eight states: the pull order can be completed by the order API provided by the platform, the ERP/OMS performs the order transfer, the review and the order, the WMS provides the picking, packing, and delivery, and finally the status is written back. After the order is written back, the order status will be displayed in the logistics details of the Taobao order.

For the entire link of the double eleven singles day orders, the following mainly recommend several products provided by the platform to make the order synchronization and processing more efficient: order synchronization through data push (order synchronization service), face sheet printing through the rookie electronic face sheet The warehouse docks through the Qimen data bus, and the order status is written back through the batch API.

Order synchronization and processing key interfaces

Order Synchronization API:

Taobao.trades.sold.get (get orders that have been sold within three months)

Taobao.trades.sold.increment.get (incremental acquisition of changed orders)

Taobao.trade.fullinfo.get (Get single order details)

Single-sided single API:

Taobao.wlb.waybill.i.get (Get the electronic face number)

Status writeback API:

Taobao.tmc.message.produce (order status reflow)

Taobao.logistics.offline.send (logistics delivery write back)

Traditional order synchronization method

The traditional order synchronization method is mainly divided into three processes.

The first process is the initialization process. After the new user comes in, it needs to be authorized. Then the background needs to start a new thread asynchronously to synchronize the order back. Generally, the order that has been sold within three months is synchronized, and the order needs to be called. The details interface gets the single order details.

The second is the incremental synchronization process. Orders are constantly being generated and changed, and incremental orders need to be synchronized in the background, so some asynchronous threads should be scheduled in the background to synchronize the incremental orders of all authorized users in the application, and then the order details can be obtained through the order details interface. And store the last synchronization time point for the next use.

The third is the real-time synchronization process, which is mainly applied to real-time processing scenarios such as automatic delivery. Taobao open platform provides the ability to actively notify the message service. After receiving the real-time message of the order change, through the order details interface, you can get the order details to update the local data.

Through these three processes, we can combine many solutions. The first one is the “full quantity + incremental” scheme, which can realize the initialization of the order and the incremental synchronization, which is suitable for most business scenarios; the second is the “full quantity + real time” scheme, and the problem is the partial order. The change may not send a message, it is suitable for real-time order processing scenarios where the data accuracy requirements are not very high; the third is the “full quantity + real-time + pair-to-single” solution, which can ensure the real-time reliability of the entire order synchronization, suitable for the data. Accuracy and real-time requirements are extremely high.

Efficient use of the order synchronization API

Taobao.trades.sold.get (obtain orders that have been sold within three months): Use the use_has_next=true method to page, each page can reduce the db count once, improve the efficiency of API paging query; for sellers with large orders , need to cut the time period, such as by day, double eleven singles day peak period or even cut to get by minute; need to call taobao.trade.fullinfo.get for details, batch interface settings fields=tid can go directly to the DB Index for greater efficiency.

Taobao.trades.sold.increment.get (incremental acquisition of changed orders): cannot be turned from the page to the page, otherwise the order that has been turned over will change, which will cause the order of the subsequent pages to move forward, and the forward order will be missed. The correct way is to turn pages from the back; first request to use use_has_next=false to get the total number of changes, subsequent requests through the use_has_next=true page, each page can be reduced by one db count; need to call taobao.trade.fullinfo.get for details In the case of the batch interface setting fields=tid, you can directly take the DB index and get higher efficiency.

Common reasons for missing orders

Bulk order query type input parameter setting is incomplete

Incremental sync order does not turn pages from back to front

Incremental synchronization order time slices do not overlap

Incremental synchronization orders do not go back a certain amount of time (for example, 5~30 minutes)

The program is simple and rude, and the time slice is not split, causing API calls to frequently time out.

Some attribute changes of the order will not send a proactive notification message.

In rare cases, some orders have changed, but the modification time has not changed.

Deeper reasons for missing orders

The picture shows the distribution of the computer room of Taobao Tmall Trading, which is divided into the central computer room and the unit computer room. The main engine room of the buyer's library is placed in the central computer room, and the main library of the buyer's library is also placed in the unit computer room. After the consumer places an order, the order will directly fall into the buyer's library. The central computer room and the unit computer room will synchronize the two-way data through the DRC. At the same time, based on the buyer's library, the seller's library is used to query the sold baby. On the basis of the seller's library, the stocking library of the seller's library is synchronized, and this stocking library is the library that actually provides the bulk API for trading. The transaction batch API directly queries the inventory of the seller's library, and the transaction details API is the associated buyer's library. The buyer library is a real-time library, and the seller library is synchronized by the buyer library. In the case of high pressure (double eleven singles day) between the seller's main library and the standby database, a synchronization delay may occur, resulting in the omission of batch inquiry orders. The slippage in this case is very difficult to solve. The more rude practice is to synchronize the order of yesterday with the full amount every morning, but the cost is very high.

In response to the above-mentioned order omissions, Taobao launched the order synchronization service.

Data Push - Order Synchronization Service

The background generated by the order synchronization service is that the cost of the missing order is high, the real-time polling of the API is not high, and the data of the big seller is difficult to synchronize. The positioning of the service is a data synchronization service for the public network, so that the internal database of Alibaba can be synchronized through the order. Service data push can be pushed to the ISV database in real time and reliably. At present, consistency has been achieved (data is completely consistent, no leakage), high real-time (second-level data synchronization), generalization (thousands of thousands of faces, self-service access and provisioning), scalability (scale-out, support billion Level data synchronization).

Data push architecture

As shown in the above figure, the entire data push architecture is divided into two parts: the server that is deployed on the data push of the intranet, the API service and the message service of the Taobao open platform; the data push client and ISV deployed in the cluster. Application server and RDS for ISV.

Key design points for data push include:

Reliable message + single-replenishment: Guarantee the order to be pushed to the RDS in real time through reliable message service; through the incremental API, big data, ODPS to make a single replenishment, to ensure that the order status is exactly the same as Taobao;

Packet isolation + dual room disaster recovery: The data push client adopts the method of packet isolation, assigning at least two machines to each group, and assigning them to different computer rooms. Even if one of the computer rooms is down, it will not affect. Normal push of data and isolate the effects between applications;

Intermediate table + large field: The design of the intermediate table and large field on the database, adapt to the personalized field push requirements of various ISVs;

Bulk Insert + Update Priority: The order RDS uses batch insert to improve the writing efficiency. After the order is changed, the update priority is adopted to avoid querying the DB once for each update, and reducing the number of queries to the DB;

Compatible API request results: The data stored in the RDS is a JSON return format compatible with the API request, and can be directly interpreted into a transaction object using the SDK provided by the TOP;

Exclusive access to RDS+: RDS is exclusive to each application. Access frequency and performance are controlled by ISV itself and are not affected by others like API. The entire RDS and data push client are put together, data push can be accessed nearby, and synchronization is more efficient.

Data push intermediate table design

The intermediate table is divided into key fields, system fields, and large fields. The key field is the more important field in the order, such as transaction ID, transaction status, transaction type, and so on. The system field is used for the entire data push to add, delete, change, check, and use the order. The large field is the entire JSON string returned by the Order Details API, and you can customize the fields you need to return.

Data Push Best Practices

First look at the official instructions for use (search for "order synchronization service use")

Set the fields and historical data to be pushed as needed

Set the system library retention time as needed (the minimum can be set to 2 days)

It is recommended to use "jdp_modified field + reverse sort + reverse page turn" to get incremental order

Do not count the table of the sys_info library, the query should try to index

Use multiple RDSs to ensure that users are evenly distributed

RDS recommended specifications, "Every 10,000 users + 2 days reserved" select RDS with IOPS of 600~1200

Official warehouse docking standard - Qimen

For the interworking between ERP and WMS, ISV used to be connected through R&D. It requires communication standards between the two parties, and the development of docking plan, joint debugging, etc. The cost of docking is high, the cycle is long, and the docking is repeated. In response to these problems, Taobao Open Platform launched Qimen, providing standardized docking for merchant ERP and WMS. As can be seen from the figure, the entire structure of the previous ERP and WMS docking is a mesh structure, and an ERP needs to be connected with all WMSs, and vice versa, which is an N-to-N situation. Qimen introduced the concept of data bus, standardized the scene of data interaction between ERP and WMS, and formed a data bus. The advantage of this is that ERP only needs to interface with Qimen data bus to realize the docking of any WMS. The reverse is also the case of an N to 1.

Qimen's advantage

ERP and WMS interoperability standards. Currently, 36 standard API interfaces have been defined to cover common scenarios.

Wide coverage, mainstream ERP and WMS are connected

Efficient docking, white paper, API online documentation, Qimen SDK, bezel self-test

Stable, proven, highly reliable, high-performance gateway that supports merchant-class flow control

Security, API requests are not tamperable, non-replayable, two-way authentication

Retrospective, all API requests record persistent storage, system disputes are well documented

Qimen Best Practices

Use baffle test to reduce joint cost

Reduce coding costs with Qimen SDK

Communication between different systems is prone to timeout problems, and core operations are best to be idempotent.

Fields not provided by the standard, can be personalized through extendProps

Increase the efficiency of the batch interface when the network latency is low

Efficient order writeback method - batch API

In addition to the API calls in the Jushi Tower, some merchants will call APIs on the mobile or office network, such as calling the API in the KL plugin. The efficiency is generally poor, because in the weak network environment, each API request The three-way handshake of TCP establishes a higher connection duration, and the total duration of each API call is much higher than the call in the cluster. In the past, the open platform also provided two batch API call methods to solve this problem: one is TQL, but the TQL calling mode is not easy to understand, the serial call is high, and the security risk is high; the other one is ATS, but the ATS provides high cost and complicated use. , real-time performance is poor. Bulk APIs can achieve high performance, low latency, generalization, and ease of use.

Bulk API architecture

As shown in the figure above, the top layer is the SDK, which is used to merge the requests of various APIs and route them to the corresponding services for calling. The second layer is lived in different places, and the SDK can be called to the corresponding computer room according to the principle of proximity. The third layer is the TOP gateway platform. After a request comes in, it will perform verification of the API verification, security, flow control and other rules. After the verification is passed, the API will be split, and a batch API request will be split into N. API request, and then submit the request to the corresponding service-providing machine in the background, wait for the background service to process, and then wake up the worker thread through the NIO network event, the worker thread will convert the result returned by the back-end data, including security. Filtering, log management, etc., until all requests are processed, the API will be collected, sorted, merged, exported, and finally the request will be returned to the client's SDK. The entire server uses the all-asynchronous processing in the process of processing the batch API. No matter how many API requests are initiated at one time, it will not affect the stability and efficiency of the server. This is the most fundamental difference between the batch API and TQL and ATS. local.

Batch API protocol

The protocol of a single API implements an API call by signing, setting of header parameters, and assembling of business parameters. The difference between the batch API protocol and the single API protocol is relatively small. The only difference is that the parameters in the body introduce a common parameter, which can be placed in the public field to avoid repeated transmission and reduce network overhead. The parameters of the different APIs are placed in the list of sub-parameters and separated by the separator of -s-.

Several features of the batch API: Payload carries each API in the form of form, which is split by \r\nS-\r\n by default (also supports custom delimiters); the first line #PUBLIC# starts, extractable Public parameters and API names; parameter priority: sub-API parameters ===override ===> #PUBLIC# parameters

Several features of the batch API: Payload carries each API in the form of form, which is split by \r\nS-\r\n by default (also supports custom delimiters); the first line #PUBLIC# starts, extractable Public parameters and API name; Parameter priority: Sub-API parameters ===Override ===> #PUBLIC#Parameter ===Override===> URL parameters; Signature mode, based on rest signature: byte2hex (hmac(key1value1key2value2.. .payloadsecret)).

Batch API compared to normal API

The calling method of a single API is familiar, such as calling the order details API, first creating a Taobao client, then creating a new request to call the order details, setting the order field to be queried, and executing a request for each request. The batch API introduces the object of TaobaoBatchRequest. It does not add the same field every time the request is made. It only needs to create a new request, append it to the batch request, and finally only initiate an API call. Comparison of bulk API calls with serial API calls has found that bulk API calls and single API calls are close in time, far more efficient than serial API calls.