Alibaba self-developed real time computing platform supports double eleven singles day

Alibaba self-developed real time computing platform supports double eleven Alibaba announced on November 7 that its big data team self-developed real-time data computing platform Galaxy, currently can calculate more than 5 million data per second, expected double eleven real time calculation. The number will exceed 10 million, and the number of daily processing messages will exceed 1 trillion. For each transaction data, the system will repeatedly detect more than 70 times in real time to ensure data quality.

One minute broke through 100 million, and more than 10 million people poured into Tmall. This is the first minute of the 2013 Double Eleven Shopping Carnival. These data are broadcast in real time on the big screen of data in Taobao City, Hangzhou. Each number that jumps on the big screen comes from the close cooperation between more than 60 systems inside Alibaba: when you are killing the double eleven hot items at the fastest speed, these systems have completed countless rounds of data collection, Transfer, process, calculate, and feed back to the page. This is exactly the technology that Alibaba has never disclosed - how to achieve real time computing while guaranteeing data quality?

Galaxy is Alibaba's self-developed universal incremental computing platform that provides real-time data computing capabilities from minute level to second level or even millisecond level. Galaxy solves many of the challenges of computing versatility, development costs, data quality, and provides scalable, scalable cluster services.

At present, Galaxy can calculate 5 million data per second, the number of records processed per day exceeds 250 billion, and the daily processing data volume is nearly 2PB. Imagine: When you are still trying to figure out how much 1024×1024 is equal, Galaxy has already got the data, the calculation is over, and the result is 5 million times in this second. This year's double eleven, the amount of data generated by users browsing, transactions, mobile APP, etc. will increase on a large scale. Galaxy's computing volume is expected to exceed 10 million per second on that day, and the number of daily processing messages will exceed 1 trillion.

Alibaba's data quality team said: "Galaxy is not only fast, but also guaranteed to be wrong." In addition to Galaxy, Alibaba has developed a system that can detect online data in real time, and can complete the data in less than 1 second. In the process of generating the verification, each transaction can be recommended more than 70 times in real time, so as to ensure that the data of the double eleven is not wrong.

For example, if a US user places an order at the Double Eleven event and has just paid the payment, the "paid" status data will not be transmitted back because the international network suddenly flashes. At this time, the possibility of presenting to the buyer is the status of "transaction failure". However, through the data real-time detection system, the alarm can be processed before the problem is discovered by the consumer. Perhaps, no consumer has come back, this problem has been corrected, and I have never felt that "transaction failure" has ever appeared.

In addition, Galaxy has also designed data "leak-proof" measures, even if the server suddenly crashes, it can also ensure that data is not lost, and resume work after rapid recovery. Imagine: If you make an appointment with a friend to watch a movie at night, you suddenly have a high fever and fainting. Under normal circumstances, you have to go to the hospital to get cured, and you have to rest for a few days. Galaxy not only can repair itself, but also can return the time back to that night, and you and friends continue to go to the movies.

At present, Galaxy has gradually supported most of the real-time business and applications of the Alibaba Group, including Taobao, Tmall, Alibaba Cloud, rookie, poly cost-effective, wireless, search, advertising, data cube and other services to provide real time computing services.