Data Processing

Research on the strategy of replica creation in cloud storage environment Hainan University leaf Xianglong with the development of Internet technology, especially in recent years, the data show the development of explosion, the traditional methods of data processing can not meet the needs of people. The emergence of cloud storage brings the gospel to people, which can meet the needs of user data processing well. However, the emergence of cloud storage also brings the problems of system fault tolerance, access efficiency and data reliability. To solve these problems, the paper introduces the replica technology for cloud storage. The introduction of replica technology, but also to the system to bring a copy of consistent maintenance, negative ...

Research on cloud computing model based on Hadoop and meteorological application Nanjing University of Information Engineering Zhangjian The main work of this paper is as follows: firstly, the characteristics of meteorological data are analyzed, and the problems of storing meteorological data directly in Hadoop are pointed out. According to the characteristics of meteorological data, a file merging algorithm based on Trie tree deformation is designed. On this basis, the experimental verification, data efficient processing, security and other aspects of effective promotion and protection. Secondly, the design and implementation of a large number of meteorological heterogeneous data storage and computing framework based on Hadoop, for meteorological data ...

Now the children are treasure, a little bit of sickness have to go to the best hospital. Sun Hong, director of Beijing Children's Hospital Information Center affiliated to the Capital Medical University, said that 300 people were not on the phone every day. Can cloud computing help Beijing's children's hospital solve this problem? Sun Hong has been working with their hospital's information center colleagues to study the problem. At the fourth session of China's cloud computing Conference, he briefed the participants on their experience "the integration of medical management process reengineering and cloud computing technology relying on information methods", and he hoped that medical resources would be better allocated to patients. General Traditional ...

Absrtact: The author is Dr. Li Tingwei, president of Greater China area of Bo Tong Company, authorize hardware to invent first. With the introduction of Google Glasses, wearable technology has become a hot topic for the public to talk about. 　　And among the most interesting is the wearable equipment in the author for the company's Greater China Region President Dr. Li Tingwei, authorized hardware again invented the first. With the introduction of Google Glasses, wearable technology has become a hot topic for the public to talk about. One of the most compelling concerns in this area is the discussion of the use of wearable equipment in health care —...

The Data Platform Business Unit first used not the MaxCompute (original ODPS) currently in use, but Hadoop. The original Hadoop cluster was named Cloud Ladder 1. At that time, Alibaba was also developing its own computing platform, which was the original ODPS, and named it. Cloud Ladder 2.

Apache Pig, a high-level query language for large-scale data processing, works with Hadoop to achieve a multiplier effect when processing large amounts of data, up to N times less than it is to write large-scale data processing programs in languages ​​such as Java and C ++ The same effect of the code is also small N times. Apache Pig provides a higher level of abstraction for processing large datasets, implementing a set of shell scripts for the mapreduce algorithm (framework) that handle SQL-like data-processing scripting languages ​​in Pig ...

Madagascar is a software package for multidimensional data analysis and repeatable computational experiments. Its mission is to provide a convenient and http://www.aliyun.com/zixun/aggregation/17547.html "> Powerful environmental and geophysical and related fields of digital image and data processing work for researchers a convenient technology transfer tool. Use Madagascar's project management system in technology development to process historical data and become a "recipe for calculation" plus ...

Study on Load Balancing optimization in MapReduce Hong Min Lau Zhao Liu Yuanyuan Hong data Analysis and processing is an important task in large-scale distributed data processing applications. Because of its simplicity and flexibility, the MapReduce programming model is becoming the core model of large-scale distributed data processing systems such as Hadoop systems. Because the data being processed may not be evenly divided, the MapReduce programming model may have data skew problems when it handles connection operations. Data skew problem severely reduces mapreduce execution ...

Smart city is a topic of great concern to the Government and many research institutes in recent years, and the Director of GIS Institute of China Institute of Surveying and Mapping, Dr. How to sublimate the digital city to the wisdom City Wisdom City is the concept that the intelligent city is a more intelligent way to change the way people communicate through a new generation of technology, improve the real-time processing of information, induction speed and response speed, increase business flexibility, continuity, promote the harmonious development of society. The wisdom of the city is generally understood to be easy to publicize ...

Big data appears in all areas of daily life and scientific research, and the continued growth of data has forced people to reconsider the storage and management of data.

For large-scale data processing, MapReduce has the following three basic design ideas. 1. To deal with big data parallel processing: a large divide and conquer If the data can be divided into the same calculation of the data block, and there is no data dependence between these data blocks, then improve the processing speed of the best ...

As the "new Oil of the future", large data is becoming another hot spot in the field of information technology after cloud computing and Internet of things. However, the existing information security means can not meet the requirements of information security in the age of large data. While large data brings challenges to information security, it also provides new opportunities for the development of information security. The author believes that large data has become a significant target of cyber attacks, increased the risk of privacy disclosure, threats to existing storage and security measures, as the carrier of high-level sustainable attacks. On the one hand, large data technology has become a means to exploit the hacker's attack, on the other hand, it provides new support for information security. ...

Now Apache Hadoop has become the driving force behind the development of the big data industry. Techniques such as hive and pig are often mentioned, but they all have functions and why they need strange names (such as Oozie,zookeeper, Flume). Hadoop has brought in cheap processing of large data (large data volumes are usually 10-100GB or more, with a variety of data types, including structured, unstructured, etc.) capabilities. But what's the difference? Today's enterprise data warehouses and relational databases are good at dealing with ...

Absrtact: The author is Dr. Li Tingwei, president of Greater China area of Bo Tong Company, authorize hardware to invent first. With the introduction of Google Glasses, wearable technology has become a hot topic for the public to talk about. 　　And among the most interesting is the wearable equipment in the author for the company's Greater China Region President Dr. Li Tingwei, authorized hardware again invented the first. With the introduction of Google Glasses, wearable technology has become a hot topic for the public to talk about. One of the most compelling concerns in this area is the discussion of the use of wearable equipment in health care-wearable equipment ...

The following small series summarizes 10 best data mining tools for everyone, which can help you analyze big data from various angles and make correct business decisions through data.

1. What is MapReduce MapReduce is a computational model, framework and platform for big data parallel processing. It implies the following three meanings: 1) MapReduce is a cluster-based high-performance parallel computing platform (Cluster Infrastructure). It allows for the deployment of a distributed and parallel computing cluster of tens, hundreds to thousands of nodes with commercially available commercial servers. 2) MapReduce is a parallel computing and running software framework (Software ...

Xml_parser_free releases the memory consumed by the resolution. Syntax: boolean xml_parser_free (int parser); Return value: Boolean function type: Data processing &http://www.aliyun.com/zixun/aggregation/37954.html ">nbsp; content description This function is used to release the memory used by the current XML parsing. Parameter parser ...

The development of the society, the prosperity of the market, let the transportation become more and more busy. In the face of tens of thousands of flights, ships and vehicles every day, the face of tens of thousands of passengers, goods, how to make people travel more convenient? How to make the transportation of goods more efficient? These problems are being paid attention to both inside and outside the industry. The emergence of large data technology undoubtedly provides us with an updated and more effective way to solve the above problems. Large data is hailed as the cloud computing, the Internet after the IT industry another important technological changes, using its transport industry accumulated massive data ...

Efficient deployment of high-level remote sensing image resources and services under cloud environment the specific research contents of Zhejiang University 曾志偉 include the following aspects: 1 on the basis of analyzing grid computing and cloud computing, this paper proposes a method to solve large data High division remote sensing efficient processing based on grid computing and cloud computing fusion strategy. Then, according to the WebService specification, the integrated description mechanism of resources and services is studied, especially for the descriptive method of stateful resources, which facilitates the organization and management of resources and services, and realizes ...

This time, we share the 13 most commonly used open source tools in the Hadoop ecosystem, including resource scheduling, stream computing, and various business-oriented scenarios. First, we look at resource management.