Large data with large science

Recently went to Wuhan to participate in the national testing conference, including hardware testing, http://www.aliyun.com/zixun/aggregation/10185.html "> Software testing, encountered many old friends and new friends, we talked a lot." I and you have exchanged on the CACM see "Big Data meets Big science", also quite feeling.

At Stanford's National Accelerator Laboratory, the observatory will install a 3.2 billion-pixel (3.2GP) camera by 2020, with extremely high-resolution sky images every 15 seconds every night after 10. The system needs to store 1 billion bytes (100PB) of data, equivalent to 20 million DVDs. Of course, the camera gets more raw data than that. The camera's vision has 40 billion ~500 billion astronomical targets. It is almost impossible to store these pixels for long periods of time, and only real-time processing and extraction of critical data. Large-scale scientific instruments produce large amounts of data from Large Hadron Collider to advanced beam processors and molecular imaging tools that the current parallel supercomputer cannot handle.

The reality that can be seen now is that 1. Moore's law has failed because the transistor size has reached the physical limit. 2. Supercomputers can no longer heap on the CPU. Tens of thousands, even hundreds of thousands of of CPUs, the GPU heap of supercomputers, the power consumption is staggering, and parallel computing is actually difficult to achieve. Most of the time, the CPU idle, and memory busy. 3. Neumann computer architecture must be changed. The storage-calculation method is no longer applicable to the new situation. For many applications, the actual bottleneck is not the processing time, but the constant access to the memory.

An obvious fact is that, although China's Tianhe supercomputer several times ranked first in the world, but the United States recently did not participate in the rankings of the competition, ranked a few also don't care.

How to solve the problem of big data? The scientists mainly take three approaches: one is to try to reduce the dataset from the beginning of observation, one is to learn from the private enterprise based on cloud computing experience, the other is to explore new technologies, such as quantum computing.

Quantum computation may be very effective for cracking passwords, factoring, and quantum physical simulations, but it is hard to say whether combinatorial optimization, aviation scheduling, and adiabatic algorithms are effective. So, big Science produces big data, big data technology relies on big science. Physics, optics, biology and computational science come together to study the collection, distribution, storage and processing of these data. You can't rely on computers alone. I wrote: "Large data technology relies on computers, large data analysis depends on the experts in various fields, now it seems that large data technology also rely on large science experts."

In such a crucial moment of innovation, the Chinese should make a difference. Do not think every day to send SCI, cast CNS, title, grumble, think about these big questions! However, I told my friends at the meeting that fault-tolerant computing is an eternal theme in quantum computing, and that people are paying close attention to fault-tolerant computing. The practical value of high-end fault, let alone, we all know.

I would like to add a few words: the micro-nano electronics industry is still thriving, the market is still very large; supercomputers, especially their applications, are going to be a little forward-looking from a scientific standpoint.