Hadoop: Facing the challenge of big data

Http://www.aliyun.com/zixun/aggregation/14417.html ">apache Hadoop to address the challenges of large data by simplifying the implementation of data-intensive, highly parallel distributed applications. Hadoop is being used by many companies, universities and other organizations around the world, which allows the analysis task to be divided into work fragments and distributed to thousands of computers, providing rapid analysis time and the distribution of massive data storage. Hadoop provides an economical way to store massive amounts of data. It provides an extensible and reliable mechanism for handling large amounts of data with a commercial hardware cluster. It also provides novel and advanced analytical techniques that allow for complex analytical processing of data in different structures.

Hadoop differs from previous distributed scenarios in the following ways:

Data is distributed in advance.

To ensure reliability and availability, data is backed up throughout the computer cluster.

Data processing attempts to be carried out in the location of the datastore, thus avoiding the generation of bandwidth bottlenecks.

In addition, Hadoop provides a simple programming way to abstract the complexities that exist in a previously distributed implementation. As a result, Hadoop provides a powerful mechanism for data analysis, including the following:

Mass-storage--hadoop allows applications to use thousands of computers and petabytes of data. Over the past 10 years, computer experts have realised that cheap "commercial" systems can be used together for High-performance computing applications that were previously handled only by supercomputers. By configuring hundreds of "small" computers as clusters, you can get more computing power in general than a single supercomputer at a relatively low price. Hadoop can leverage a cluster of more than thousands of machines to deliver huge storage and processing power at affordable prices.

Distributed processing for fast data access--hadoop clustering provides efficient storage of massive data while providing fast data access. Prior to Hadoop, parallel computing was difficult to distribute between machines in a cluster to perform tasks. This is because the cluster execution model relies on shared data stores that require very high I/O performance. Hadoop moves program execution to data. Moving the application to data alleviates many of the high performance challenges. In addition, Hadoop applications are often designed to process data sequentially. This avoids random data access (disk seek operations) and further reduces the I/O load.

Reliability, failover, and scalability-in the past, when a machine cluster was used, the implementation of parallel applications needed to be painstaking in dealing with reliability issues. Although the reliability of a single machine is quite high, the probability of failure increases with the increase of cluster size. In a large cluster (thousands of machines), it is not uncommon to fail every day. Given the way Hadoop is designed and implemented, a machine fails (or a group of machines fail) will not result in inconsistent results. Hadoop detects failure and retry execution (using different nodes). In addition, the scalability built into Hadoop allows you to seamlessly add additional (repaired) servers to the cluster and use them for data storage and program execution.

For most Hadoop users, the most important feature of Hadoop is the clear separation of the business logic program from the framework support code. For users who want to focus on business logic, Hadoop hides the complexity of the framework, providing a simple and easy-to-use platform for complex, distributed computing to solve difficult problems.