Hadoop Simple API Web Application Development

Hadoop Simple API Web Application Development

Recently, I wrote a Web program to call the Hadoop api. I developed the Hadoop-provided management method twice, enhancing the operability. Now we will briefly introduce the functions and methods. Hadoop version 1.xx

1. File Viewing

First page

This function mainly uses the Hadoop FileSystem class to obtain HDFS file objects and uses this method to operate HDFS. The FileStatus class is used for determining the file type.

2. Job monitoring

Job monitoring first obtains a JobClient object, calls its getAllJobs method, obtains a list of all jobs in the cluster, and obtains the job running status based on JobStatus.

In job monitoring, each job is implemented by the system by creating a new thread. On the job creation page, a new thread is created to run the job, this does not prevent the main thread from creating other jobs or managing clusters.

The updates of job processes on the page are automatically refreshed through the page, which is a waste of system resources. It may be better to use Ajax and will be improved later.

New Features of Hadoop2.5.2

Install and configure Hadoop2.2.0 on CentOS

Build a Hadoop environment on Ubuntu 13.04

Cluster configuration for Ubuntu 12.10 + Hadoop 1.2.1

Build a Hadoop environment on Ubuntu (standalone mode + pseudo Distribution Mode)

Configuration of Hadoop environment in Ubuntu

Detailed tutorial on creating a Hadoop environment for standalone Edition

Build a Hadoop environment (using virtual machines to build two Ubuntu systems in a Winodws environment)