Large Data Infrastructure Forum: How to build a network infrastructure that meets the age of large data?

"Csdn Live Report" December 2014 12-14th, sponsored by the China Computer Society (CCF), CCF large data expert committee contractor, the Chinese Academy of Sciences and CSDN jointly co-organized to promote large data research, application and industrial development as the main theme of the 2014 China Data Technology Conference (big Data Marvell Conference 2014,BDTC 2014) and the second session of the CCF Grand Symposium was opened at Crowne Plaza Hotel, New Yunnan, Beijing.

2014 China Large Data Technology conference the second day morning of the Big Data Infrastructure Forum, Mellannox Asia-Pacific market Development Director Liu Tong, Wuhan Green Network Information Service Co., Ltd. Deputy general manager Rai, Wave Cloud product department and Channel Promotion department general manager Wang Feng, Baidu Infrastructure Department senior architect Chaihua and Brocade China Region technical director Yuzhanyun Five experts focus on the network level of related issues.

Mellannox Asia-Pacific market development Director Liu Tong

Mellannox Asia Pacific Market Development Director Liu Tong brought the theme "High-speed network InfiniBand accelerated large data Applications" speech. He said the infrastructure and the application of the upper level are very closely linked, the combination of hardware and software is more meaningful. In his view, InfiniBand is a completely autonomous and very intelligent two-layer network, from the day of birth it is the SDN network.

"Up to 56gb/s bandwidth (100G EDR InfiniBand also available) and a delay of less than 700 nanoseconds provide a guarantee for high-speed data transfer. InfiniBand built-in support RDMA (remote memory direct access) network transmission technology, providing memory 0 copies, kernel Bypass, protocol uninstall advantages, thus greatly downloading network traffic to CPU resources, these advantages are traditional TCP/IP communication protocol does not have. "he said.

Figure: Wuhan Green Network Information Service Co., Ltd. Deputy general manager Rai

Wuhan Green Network Information Service Co., Ltd. deputy general manager Rai published the theme of "Large data in network services applications and Network Evolution" speech. He elaborated on the two contents of the large data exploration in network operation and the evolution of intelligent network infrastructure in the era of large data. He describes the three tiers of a web-based, large data architecture: The first one is large data collection and preprocessing; the second is the DMP data management platform brings together the data resources of each province, and carries on the security to the privacy processing; The third is that partners can leverage the resources of the operator's large data platform, To carry out in-depth analysis of data mining.

He believes that IDC is the basic strategic resource of Internet industry, as the information Center, directly support the development of the whole internet industry. In the big Data age, the network evolution has undergone three changes: 1. The challenge of non-directional burst traffic; 2. Cloud Data center interconnection; 3. Cloud Open Data Center internal network support cloud services. Induction is the change in the network architecture driven by business change: 1. The traffic flow is mainly horizontal from longitudinal direction to 2. Large isomerization, more nodes number across data center; virtualization: 3. Communication between virtual machines of the same physical server, virtual machine identity when communicating with virtual machines across physical servers, virtual machine migration.

Figure: Wave Cloud Product Department and general manager of Channel Promotion department Wang Feng

Wave Cloud Product department and Channel Propulsion department general manager Wang Feng published the theme of "Open Fusion Cloud data Center" speech. Past data centers are for one application, the efficiency of our data centers is not very high, and then we propose calculations and storage of software definitions, and he says he hopes that everything in the future in the field of information can be defined by software, and that all resources can be abstracted, pooled, and automated.

The traditional data Center architecture, computing, storage and network resources are independent of each other, different applications require different devices and networks, the use of software-defined data centers can simplify it application infrastructure, so that it is no longer the entire cloud computing, large data, the heaviest part. Finally, he summed up two points: 1. SDDC will completely change the data center's device configuration, service form, and even the way it applications are carried out; 2. We have a few more stages before the data center for full integration, pooling, and software definition arrives.

Figure: Baidu Infrastructure Division senior architect Chaihua

Baidu Infrastructure Division senior architect Chaihua introduced Baidu distributed computing platform. He first introduced the development of Baidu MapReduce: 2004 MapReduce paper published in 2007 on the line based on the Haddop Baidu mr,2011 Baidu Mr Single cluster size reached 5000, 2013, Baidu Mr Single cluster reached 13000 units, 2014, Baidu DCE release, Dag engine online.

In the speech, Chaihua emphatically introduced Baidu distributed computing "Seven Musketeers": 1. Large-scale off-line computing platform DCE (MAPREDUCE/DAG model), 2. Large scale machine learning platform MPI (BSP model), 3. New generation of large-scale machine learning Framework elf (Parameter server architecture), 4. Millisecond-level real-time computing system dstream,5. Small-Batch streaming computing task manger,6. spark,7 of Open source distributed computing framework based on memory. A rig for distributed trace systems.

Yuzhanyun, technical director of Brocade China Region

Yuzhanyun, technical director of Brocade China, has brought the technology sharing "SDN technology for large data visualization", which focuses on the application of SDN technology in large scale, ultra-high performance data acquisition. The architecture of Vyatta platform includes three core layers: NFV connection Service, SDN architecture service and feature orchestration. Each layer of this open architecture is modular, allowing customers to choose the products and/or solutions that are appropriate for their specific data center requirements. This platform is open and the features of the module achieve interoperability, providing customers with better choices. Inter-tier communication uses standards-based transparent protocols and methods, including rest APIs and Netconf/yang patterns, to ensure configuration interoperability. At last, the paper analyzes how to apply the Sdn/openflow technology to the data center of the tbps bandwidth capacity and the automatic collection of backbone network at any time.

More highlights, please pay attention to the live topic 2014 China Large Data Technology Congress (BDTC), Sina Weibo @csdn cloud computing, subscribe to CSDN large data micro-signal.