Hadoop platform lab report

1. Preface

Through the Massive Data Processing Experiment Based on the mapreduce cloud computing platform, we learned about the basic architecture of hadoop and how to compile mapreduceProgramIn this experiment, I mainly use two programs: wordcount (Word Frequency Statistics) and invertedindex (reverse index ). Before proceeding to the two programs, I will introduce my understanding of hadoop.

2. Introduction and features of hadoop 2.1. hadoop Distributed File System (HDFS)

Hadoop Distributed File System (HDFS) is designed as a distributed file system suitable for running on a common hardware (commodity hardware. It has a lot in common with the existing distributed file system. But at the same time, it is quite different from other distributed file systems. HDFS is a highly fault tolerant system and is suitable for deployment on cheap machines. HDFS provides high-throughput data access and is suitable for applications on large-scale datasets. HDFS relaxed some POSIX constraints to achieve stream reading of file system data.

2.2. Simple consistency Model

An HDFS application requires a file access model that reads data multiple times at a time. A file does not need to be changed after it is created, written, or closed. This assumption simplifies data consistency and makes high-throughput data access possible. MAP/reduce applications or web crawler applications are very suitable for this model. It is also planned to expand this model in the future to support additional write operations on files.

2.3. "mobile computing is more cost-effective than mobile data"

The closer an application request is to be processed, the more efficient it is. This can reduce the impact of network congestion and improve the system data throughput. Moving computing to the vicinity of data is obviously better than moving data to the application. HDFS provides an interface for applications to move themselves to the vicinity of data.

2.4. Data Replication

HDFS is designed to reliably store large files across machines in a large cluster. It stores each file into a series of data blocks. Except for the last one, all data blocks are of the same size. For fault tolerance, all data blocks of a file will have copies. The data block size and copy coefficient of each file are configurable. The application can specify the number of copies of a file. The copy coefficient can be specified or changed later when the file is created. All files in HDFS are written at one time, and there must be only one writer at any time.

Namenode manages data block replication. It periodically receives heartbeat signals and block status reports (blockreport) from each datanode in the cluster ). The received heartbeat signal means that the datanode node is working properly. The block Status Report contains a list of all data blocks on the datanode.

2.5. Copy Selection

To reduce the overall bandwidth consumption and read latency, HDFS tries its best to allow the reader to read the copy closest to it. If there is a copy on the same rack of the read program, the copy will be read. If an HDFS cluster spans multiple data centers, the client will first read copies of the local data center.

2.6. Persistence of File System metadata

The namespace of HDFS is stored on namenode. For any operation that modifies the metadata of the file system, namenode uses a transaction log called editlog to record it. For example, if you create a file in HDFS, The namenode will insert a record to indicate it. Similarly, the copy coefficient of the modified file will also insert a record to the editlog. Namenode stores this editlog in the file system of the local operating system. The namespace of the entire file system, including data block-to-file ing and file attributes, is stored in a file called fsimage, this file is also stored in the local file system where namenode is located.

Namenode stores the entire file system namespace and blockmap images in the memory. This key metadata structure is very compact, so a namenode with 4 GB memory is sufficient to support a large number of files and directories. When namenode is started, it reads the editlog and fsimage from the hard disk, applies all the transactions in the editlog to the fsimage in the memory, and saves the new fsimage from the memory to the local disk, then delete the old editlog, because the transaction of the old editlog has been applied to fsimage. This process is called a checkpoint ). In the current implementation, checkpoints only occur when namenode is started. In the near future, periodic checkpoints will be supported.

Datanode stores HDFS data as files in a local file system and does not know the information about HDFS files. It stores each HDFS data block in a separate file in the local file system. Datanode does not create all files in the same directory. In fact, it uses a testing method to determine the optimal number of files in each directory and create subdirectories when appropriate. Creating all local files in the same directory is not the best choice, because the local file system may not be able to efficiently support a large number of files in a single directory. When a datanode is started, it will scan the local file system, generate a list Of all HDFS data blocks corresponding to these local files, and then send it as a report to namenode. This report is the block status report.

2.7. Cluster balancing

HDFS architecture supports data balancing policies. If the free space on a datanode node falls below the specified critical point, the system automatically moves data from this datanode to another idle datanode according to the balance policy. When requests to a file suddenly increase, a plan may also be started to create a new copy of the file and rebalance other data in the cluster. These balancing policies are not yet implemented.

2.8. Data Integrity

Data blocks obtained from a datanode may be damaged, which may be caused by storage device errors, network errors, or software bugs of datanode. The HDFS client software implements the checksum check of HDFS file content. When the client creates a new HDFS file, it calculates the checksum of each data block of the file and saves the checksum as a separate hidden file in the same HDFS namespace. After the client obtains the file content, it checks whether the data obtained from datanode matches the checksum in the corresponding checksum file. If not, the client can obtain a copy of the data block from another datanode.

3. hadoop Process Overview

After hadoop is started, the corresponding hadoop process will be started. You can enter JPs in the terminal to view the current process. The following describes the specific meanings and functions of these processes.

3.1. namenode and datanode

HDFS adopts the Master/Slave architecture. An HDFS cluster consists of a namenode and a certain number of datanodes. Namenode is a central server responsible for managing the file system namespace and client access to files. A datanode in a cluster is generally a node responsible for managing the storage on its node. HDFS exposes the namespace of the file system, allowing you to store data in the form of files. Internally, a file is actually divided into one or more data blocks, which are stored in a group of datanode. Namenode executes the namespace operations of the file system, such as opening, closing, renaming a file or directory. It is also responsible for determining the ing between data blocks and specific datanode nodes. Datanode is responsible for processing read/write requests from the file system client. Create, delete, and copy data blocks under the unified scheduling of namenode.

Namenode and datanode are designed to run on common commercial machines. These machines generally run the GNU/Linux operating system (OS ). HDFS is developed in Java. Therefore, namenode or datanode can be deployed on any machine that supports Java. HDFS can be deployed on multiple types of machines because of its highly portable Java language. A typical Deployment scenario is to run only one namenode instance on one machine, while other machines in the cluster run one datanode instance respectively. This architecture does not reject running multiple datanode on one machine, but this is rare.

The single namenode structure in the cluster greatly simplifies the system architecture. Namenode is the arbitration and manager of all HDFS metadata, so that user data will never flow through namenode.

3.2. Secondary namenode

Namenode appends changes to the file system to an edits on the local file system ). When a namenode is started, it first reads the HDFS status from an image file (fsimage) and then applies edits operations in the log file. Then it writes the new HDFS status to (fsimage) and starts normal operations using an empty edits file. Because namenode only merges fsimage and edits in the startup phase, log files may become very large over time, especially for large clusters. Another side effect of the log file being too large is that the next namenode startup takes a long time.

Secondary namenode periodically merges fsimage and edits logs to limit the size of edits log files. Because the memory requirement and namenode are on an order of magnitude, secondary namenode and namenode usually run on different machines. Secondary namenode is started on the node specified in CONF/masters through the bin/start-dfs.sh.

The Checkpoint Process of secondary namenode is started by two configuration parameters:

L fs. Checkpoint. Period: specifies the maximum interval between two consecutive checkpoints. The default value is 1 hour.

L fs. Checkpoint. Size defines the maximum value of the edits log file. Once this value is exceeded, the checkpoint is forcibly executed (even if the maximum interval of the checkpoint is not reached ). The default value is 64 MB.

The directory where secondary namenode stores the latest checkpoint is the same as that of namenode. Therefore, namenode can read the checkpoint image on secondary namenode as needed. If all historical images and edits files except the latest checkpoint are lost on the namenode, the latest checkpoint can be introduced to the namenode. You can perform the following operations:

L create an empty folder at the location specified by DFS. Name. dir;

L assign the position of the checkpoint directory to the configuration parameter Fs. Checkpoint. dir;

L start namenode and add-importcheckpoint.

Namenode reads the checkpoint from the fs. Checkpoint. dir directory and stores it in the DFS. Name. dir directory. If the DFS. Name. dir directory contains a valid image file, namenode fails to be started. Namenode checks the consistency of the image file under the fs. Checkpoint. dir directory, but does not change it.

3.3. jobtracker

Create an inputformat instance, call its getsplits () method, split the input directory file into filesplist as the Mapper task input, and generate a Mapper task to join the queue.

3.4. tasktracker

Calculate the next map/reduce from jobtracker. The Mapper task first creates a recordreader from inputformat, cyclically reads the content of filesplits to generate the key and value, and passes it to the Mapper function. After processing, the intermediate result is written as sequencefile. Reducer task obtains the required intermediate content (33%) from the jetty of tasktracker running mapper using the HTTP protocol, sort/Merge (66%), and executes the reducer function, write the result directory according to outputformat. Tasktracker reports the running status to jobtracker every 10 seconds. every 10 seconds after a Tasker is completed, the task is requested from jobtracker.

4. hadoop MAP/reduce tutorial 4.1. Overview

Hadoop MAP/reduce is a simple software framework based on which applications can run on a large cluster consisting of thousands of commercial machines, it also processes T-level datasets in parallel in a reliable and fault-tolerant manner. A map/reduce job usually divides the input dataset into several independent data blocks, and the map task processes them in full parallel. The framework sorts the map output first and then inputs the result to the reduce task. Generally, input and output of jobs are stored in the file system. The entire framework is responsible for task scheduling and monitoring, and re-execution of failed tasks.

Generally, the MAP/reduce framework and Distributed File System run on the same group of nodes. That is to say, computing nodes and storage nodes are usually used together. This configuration allows the framework to efficiently Schedule Tasks on nodes that have stored data, which can make efficient use of the network bandwidth of the entire cluster.

The MAP/reduce framework is composed of a separate master jobtracker and a slave tasktracker for each cluster node. The master is responsible for scheduling all the tasks that constitute a job. These tasks are distributed on different slave, and the master monitors their execution and re-executes the failed tasks. Slave is only responsible for executing tasks assigned by the master.

The application should at least specify the input/output location (PATH), and provide map and reduce functions by implementing appropriate interfaces or abstract classes. With the parameters of other jobsJob configuration (Job Configuration). ThenJob ClientSubmit jobs (jar packages/executable programs) and configuration information to jobtracker. The latter is responsible for distributing the software and configuration information to slave, scheduling tasks, and monitoring their execution, both the status and diagnostic information are provided to the job-client.

Although the hadoop framework is implemented using javatm, MAP/reduce applications do not have to be written in Java. Hadoop streaming is a utility used to run jobs. It allows users to create and run any executable programs (such as shell tools) as Mapper and reducer. Hadoop pipes is a swig-Compatible C ++ API (not based on jnitm technology). It can also be used to implement MAP/reduce applications.

5. Input and Output

The MAP/reduce framework runs in In other words, the Framework regards the input of a job as a group. Key-value pairs also generate a group As the output of the job, these two key-value pairs may have different types. The framework needs to serialize the key and value classes. Therefore, these classes need to implement the writable interface. In addition, to facilitate the framework to perform sorting operations, the key class must implement the writablecomparable interface.

the input and output types of a map/reduce job are as follows: (input) -> Map -> -> combine -> -> reduce -> (Output)