Apachehadoop2.4.064bit Installation Guide under windows8.1 (no virtual machine required)

Work needs: hadoop is about to begin, and big data is about big data and cloud. It is just a conceptual hype that emerged after the ERP and SOAP trends with the advent of the smart Earth. However, Apache is a magic organization, and Java won't be so popular without it. To put it bluntly: first download the tar.gz package of apachehadoop2.4.0and decompress it locally

Work needs: hadoop is about to begin, and big data is about big data and cloud. It is just a conceptual hype that emerged after the ERP and SOAP trends with the advent of the smart Earth. However, Apache is a magic organization, and Java won't be so popular without it. To put it bluntly: first download the tar.gz package of Apache hadoop 2.4.0and decompress it locally

Work needs: hadoop is about to begin, and big data is about big data and cloud. It is just a conceptual hype that emerged after the ERP and SOAP trends with the advent of the smart Earth. However, Apache is a magic organization, and Java won't be so popular without it. Back to the truth:

First, download the tar.gz package of Apache hadoop 2.4.0and decompress it to a local disk. Note that the path does not contain spaces. Otherwise, the path in windows 8.3 format is required in your configuration file!

The second is to ensure that the operating system is 64bit, And the. netframework4.0 or later version has been installed. You know, how can this problem be mixed in the world of Microsoft!

Third, ensure that the 64-bit version of JDK1.7 is installed. The author uses JDK1.7.

Please download hadoop-commin-2.2.zip from github. Apache hadoop plugin and libwinutils. lib are officially downloaded. This github is a 64-bit version of The Link Library package compiled by developers. Download and decompress the package directly, and overwrite the bin directory under the Official hadoop directory.

If you want to compile and install hadoop in windows, please refer to the official original document: Hadoop2.X Windows Installation Guide

Next, configure (most of them are excerpted from official documents ):

The decompressed directory of my Hadoop is D: \ Hadoop_2_4_0.

First, modify the hadoop-env.cmd script file for % h_home % \ etc \ hadoop and add the environment variable definition at the end of the file. Find the original JAVA_HOME setting of the file and change it to your 64-bit JDK path. Pay attention to this 1.1! For example, "set JAVA_HOME = D: \ OLD_JDK \ jdk1.7.0 _ 51"

set HADOOP_PREFIX=c:\deployset HADOOP_CONF_DIR=%HADOOP_PREFIX%\etc\hadoopset YARN_CONF_DIR=%HADOOP_CONF_DIR%set PATH=%PATH%;%HADOOP_PREFIX%\bin

Then find or create a core-site.xml file in this path, with the following changes:

   
      
   
    fs.default.name
       
   
    hdfs://0.0.0.0:19000
     
  
 

Next is the hdfs-site.xml file, the same modifications are as follows. By default, the configuration file uses the \ tmp directory as the storage location of the hdfs file. For example, if I decompress hadoop under D: \, it creates D: \ tmp to store the HDFS file system.

   
      
   
    dfs.replication
       
   
    1
     
  
 

The basic configuration is OK.

Next we will configure an YARN example:

Modify or create a mapred-site.xml file with a template file of the same name under the configuration file path, you can copy it directly, and then modify the content. Replace % USERNAME % in the configuration file with your windows user name.

    
       
   
    mapreduce.job.user.name
        
   
    %USERNAME%
      
     
       
   
    mapreduce.framework.name
        
   
    yarn
      
    
      
   
    yarn.apps.stagingDir
       
   
    /user/%USERNAME%/staging
     
    
      
   
    mapreduce.jobtracker.address
       
   
    local
     
  
  

Finally, create a yarn-site.xml file with the following changes:

   
      
   
    yarn.server.resourcemanager.address
       
   
    0.0.0.0:8020
     
    
      
   
    yarn.server.resourcemanager.application.expiry.interval
       
   
    60000
     
    
      
   
    yarn.server.nodemanager.address
       
   
    0.0.0.0:45454
     
    
      
   
    yarn.nodemanager.aux-services
       
   
    mapreduce_shuffle
     
    
      
   
    yarn.nodemanager.aux-services.mapreduce.shuffle.class
       
   
    org.apache.hadoop.mapred.ShuffleHandler
     
    
      
   
    yarn.server.nodemanager.remote-app-log-dir
       
   
    /app-logs
     
    
      
   
    yarn.nodemanager.log-dirs
       
   
    /dep/logs/userlogs
     
    
      
   
    yarn.server.mapreduce-appmanager.attempt-listener.bindAddress
       
   
    0.0.0.0
     
    
      
   
    yarn.server.mapreduce-appmanager.client-service.bindAddress
       
   
    0.0.0.0
     
    
      
   
    yarn.log-aggregation-enable
       
   
    true
     
    
      
   
    yarn.log-aggregation.retain-seconds
       
   
    -1
     
    
      
   
    yarn.application.classpath
       
   
    %HADOOP_CONF_DIR%,%HADOOP_COMMON_HOME%/share/hadoop/common/*,%HADOOP_COMMON_HOME%/share/hadoop/common/lib/*,%HADOOP_HDFS_HOME%/share/hadoop/hdfs/*,%HADOOP_HDFS_HOME%/share/hadoop/hdfs/lib/*,%HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/*,%HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/lib/*,%HADOOP_YARN_HOME%/share/hadoop/yarn/*,%HADOOP_YARN_HOME%/share/hadoop/yarn/lib/*
     
  
 

OK. All configurations have been modified. Click the command prompt (Administrator) to run the command prompt and switch to the hadoop installation directory. Perform the following operations:

1. Switch to the etc/hadoop directory, run the hadoop-env.cmd script, set the current command window to execute environment variables.

2. format the HDFS File System (switch to the bin directory and execute the command ):

%HADOOP_PREFIX%\bin\hdfs namenode -format

3. Example of running HDFS:

%HADOOP_PREFIX%\sbin\start-dfs.cmd

If no error is reported, the deployment is OK !!!