Data Processing

A large data processing algorithm for energy efficiency in heterogeneous clusters Ding, Qin Xiaolin, Liuliang, Wang Tao Spring cluster energy consumption has exceeded its own hardware acquisition costs, and large data processing needs large-scale cluster time-consuming, so how to carry out high energy efficient data processing is a problem for both the owner and the user. is also a huge challenge to energy and the environment. Existing research is generally done by shutting down some nodes to reduce energy consumption, or by designing new data storage strategies to implement energy efficient data processing. Through analysis, it is found that even the least-used nodes exist ...

Analysis of large scale graph data processing technology in cloud computing environment Liyuan This article begins with the introduction of cloud computing, and studies the storage mode of graph data based on cloud computing, the segmentation of graph data, the calculation model of graph data and the query processing of graph data. It is hoped that the research in this paper can help to improve the data processing technology of large-scale map. Analysis of large scale graph data processing technology in cloud computing environment

With the explosion of information data, the analysis and processing of large data, especially unstructured data, is becoming more and more important. For mass data processing, there are obvious performance bottlenecks in traditional software architecture, analysis algorithm and deployment model, and new solutions are needed urgently. Collection Cloud "Haina" cabinet is an integrated soft, hardware system integrated large data processing cabinet, through the distributed analysis algorithm, the optimization of the calculation and storage equipment configuration, perfect software and hardware environment collocation, to provide a comprehensive machine processing mechanism for large data services for the user to provide the most optimized solution! Gao Ke ...

Recently, LinkedIn Open source has a technology--samza, it is a distributed flow processing framework, dedicated to real-time data processing, very much like the Twitter stream processing system storm. The difference is that Samza is based on Hadoop and uses LinkedIn's own Kafka distributed messaging system. Storm and Samza are very similar, as LinkedIn's Chris Riccomini blog: "[Samza] can help you build applications, process message queues ——...

Discussion on cloud computing and meteorological data processing Lau in the information age large data has become the main feature, in the increasingly developed network technology today, cloud computing is increasingly becoming the current and future network science and technology development of an important topic, become the future of computer technology development of the core direction. Cloud computing has more robust and robust performance both in terms of security and computing power.  In the meteorological department, cloud computing can provide huge data and make difficult and accurate calculations, which provides a new way of development for meteorological research. Discussion on cloud computing and meteorological data processing

Dynamic service for large-scale data processing private cloud system Wang Zhu, Merlin, Lilei, Chang, Hu Guang Summary: In order to adapt to a private cloud environment with large amount of data, computationally intensive, complex process of computing task demand, for reference to the public cloud computing related theory and technology, combining the characteristics of private cloud environment, This paper presents an implementation scheme of a dynamic service private cloud system which adapts to large-scale data processing. This scheme uses the job file to describe the computing task, constructs the processing work flow dynamically with the job logic structure, and introduces the MapReduce parallel framework through the data stream driver service request.

The data processing Guo of downhole personnel based on cloud storage Li Jing trillion in view of the actual situation of the major coal mines in China, this paper studies the personnel positioning system in the underground, and uses the cloud storage instead of the original storage mode in order to improve the insecurity and incomplete of the massive data produced by the positioning. The use of cloud storage body to customize this feature, in the enterprise to build a private cloud, with Hadoop as the technical framework, the use of HBase Rowkey to determine the primary key of the search, HDFs Namenode and Datanode to complete the interaction between the data, fast, efficient ...

Design of traffic cloud data processing platform based on Hadoop the rapid development of urban traffic in the city has brought more and more convenience to people at the same time, also appeared the city traffic pressure is too large, not efficient use of traffic resources. The goal of urban Intelligent transportation planning is to analyze the traffic situation and provide quality traffic guidance for travelers in the face of super large-scale travel volume. In this paper, the concept of traffic cloud is introduced, the traffic data and its calculation are concentrated in the "cloud", using the MapReduce distributed computing of Hadoop ...

hitachi Large data processing open source middleware temp_12040815388338.pdf

Ministry of Industry recently announced the 2014 1-October software business economic performance. January-October, China's software and information technology services to achieve software business income of 2.9853 trillion yuan, an increase of 20.3%. Among them, data processing and storage services to achieve revenue of 531.5 billion yuan, an increase of 25.3%, the growth rate to maintain the industry first, higher than the entire industry average level of 5.1%, compared to the January-September rebound 0.7%. Ministry of Industry data shows that in January-October, China's software and information technology services to achieve software business income of 2.9853 trillion yuan, an increase of 2.

Study on the efficiency optimization of Excel data processing based on cloud computing Liu Xianmei; Shangchong in order to improve the data processing efficiency of Excel software, it is necessary to establish a computing environment which can meet the high accuracy, efficiency and easy to use. In this paper, the optimal interpolation model of cylindrical spiral is established, the borehole curvature is calculated by five sets of oblique data in Excel, and then the Symphony de parallel software is built in the local private cloud environment by increasing the number of computing nodes and calculating the accuracy according to the characteristic of the computing nodes. The results show that with the increase of the number of nodes,...

Application analysis of meteorological cloud storage and data processing based on Hadoop Shi Sheng Zhou Tianbo Sunday Jie This paper mainly introduces the structure of hado0p architecture, and describes the MapReduce implementation of Hadoop architecture in detail with examples.   On the basis of this, we develop an example of meteorological numerical statistics on the basis of Hadoop architecture, and analyze the operational efficiency of the single node mode, pseudo distribution mode and complete distribution mode based on this example. Close ...

A database engine for large-scale data processing Wang Yi, Liu Great Wall, Ma Jianqing when data volumes rise from GB to terabytes or even petabytes, high-performance parallel databases can be computationally costly to ensure scalability and fault tolerance. To solve this problem, a parallel database engine flexdb for large-scale data processing is designed. The parallel computing framework of map Reduce is used as the communication layer to dispatch and coordinate the computation and communication of the nodes in the cluster. The experimental results show that the system of FLEXDB ...

Mass data processing technology of intermittent energy based on mapreduce model Mei-hua-mai enhanced Wu Guanglei in view of the limitation of traditional intermittent energy data processing technology, a batch energy data processing technology based on MapReduce model is proposed, which makes use of Low-cost commercial computer to form a cluster. The parallel processing of massive data ensures the reliability, low cost, high efficiency and scalability of the mass processing, and discusses the platform implementation of the technology. At last, the efficiency of mass data processing under different data platform is compared and validated based on Mapre ...

GAVL is an uncompressed audio and video http://www.aliyun.com/zixun/aggregation/14345.html "> Data Processing Library. It provides common conversion features such as video scaling, color space conversion, audio resampling, and more. GAVL is compatible with all major multimedia decoder APIs, supporting audio and video formats, including low-end traditional formats as well as professionally edited high-definition video formats. Gavl 1.4.0 This version adds a generic ...

Discussion on the evolution of large data processing technology architecture Ninguevo Jing New application to the real-time requirements of large data processing technology architecture, which challenges the traditional large data processing technology architecture. Must transform the architecture to meet the real-time requirements of large data-related businesses. This paper introduces the bottleneck of the Hadoop off-line processing architecture and the advantages of storm real-time processing architecture, at the same time, with the experience of changing the large data processing technology architecture in the actual project, this paper expounds the key technologies in the process of implementing the architecture change, and the experimental results prove that the use of the changed technical framework can

The core concept of the cascading API is piping and streaming. A pipeline is a series of processing steps (parsing, looping, filtering, and so on) that define the data processing to be performed, and the flow is the union of pipelines with data sources and data receivers (Data-sink). Cascading is a new data processing API for Hadoop clusters that uses expressive APIs to build complex processing workflows, and ...

Preg_replace string is parsed and replaced. Grammar: Mixed preg_replace (mixed pattern, mixed replacement, mixed subject); Return value: Mixed type data function type: Data processing &http://www.aliyun.com/zixun/aggregation/37954.html ">nbsp; content description ...

MapReduce principle and its application in natural language processing Shiri Wang as white as river in view of the current mass data processing in the processing speed, storage space, fault tolerance, access time and other aspects of the problems, the Google MapReduce programming model of the principle, the implementation process, and so on, This paper summarizes the main applications of MapReduce programming model in natural language processing and information retrieval from four aspects of MapReduce and index construction, statistical machine translation, clustering algorithm and text categorization, with a view to MapReduce ...

MPP database technology, supporting industry large data applications Dr. Vounie, cto/Tianjin Nanda General Data Technology Co., Ltd.--MPP database technology--gbase 8a MPP Cluster features--gbase 8a MPP Cluster Telecom, financial industry case Large data processing-mpp non-Hadoop hybrid architecture Trend MPP database technology, supporting the industry's large application