Windows configuration local Hadoop runtime environment

Many people like to develop Hadoop programs locally with Windows, a tutorial that configures Hadoop under Windows.

First go to the official website to download Hadoop, here need to download a tool winutils, this tool is compiled for Hadoop, after downloading the Hadoop file, and then put the Winutils.exe into the Hadoop file bin directory

Then modify the following files under Hadoop/etc/hadoop:

Core-site.xml:

<?XML version= "1.0" encoding= "UTF-8"?><?xml-stylesheet type= "text/xsl" href= "configuration.xsl "?><!--Licensed under the Apache License, Version 2.0 (the "License");  You are not a use of this file except in compliance with the License. Obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 unless required by applicable law O  R agreed to writing, software distributed under the License are distributed on a "as is" BASIS, without warranties OR  CONDITIONS of any KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. -<!--Put Site-specific property overrides in this file. -<Configuration>  < Property>    <name>Fs.defaultfs</name>    <value>Hdfs://localhost:9000/</value>  </ Property>  < Property>    <name>Io.native.lib.available</name>    <value>False</value>  </ Property>  < Property>    <name>Hadoop.native.lib</name>    <value>False</value>  </ Property>  < Property>    <name>Io.compression.codecs</name>    <value>Org.apache.hadoop.io.compress.GzipCodec, Org.apache.hadoop.io.compress.DefaultCodec, com.hadoop.co Mpression.lzo.LzoCodec, Com.hadoop.compression.lzo.LzopCodec, Org.apache.hadoop.io.compress.BZip2Codec , Org.apache.hadoop.io.compress.SnappyCodec</value></ Property>< Property>    <name>Io.compression.codec.lzo.class</name>    <value>Com.hadoop.compression.lzo.LzoCodec</value></ Property></Configuration>

Hdfs-site.xml:

<?XML version= "1.0" encoding= "UTF-8"?><?xml-stylesheet type= "text/xsl" href= "configuration.xsl "?><!--Licensed under the Apache License, Version 2.0 (the "License");  You are not a use of this file except in compliance with the License. Obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 unless required by applicable law O  R agreed to writing, software distributed under the License are distributed on a "as is" BASIS, without warranties OR  CONDITIONS of any KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. -<!--Put Site-specific property overrides in this file. -<Configuration>         < Property>                 <name>Dfs.replication</name>                  <value>1</value>         </ Property>       < Property>                <name>Dfs.namenode.name.dir</name>                 <value>File:///D:/Hadoop/namenode</value>         </ Property>       < Property>               <name>Dfs.datanode.data.dir</name>                 <value>File:///D:/Hadoop/datanode</value>         </ Property></Configuration>  

Mapred-site.xml:

<?XML version= "1.0"?><?xml-stylesheet type= "text/xsl" href= "configuration.xsl "?><!--Licensed under the Apache License, Version 2.0 (the "License");  You are not a use of this file except in compliance with the License. Obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 unless required by applicable law O  R agreed to writing, software distributed under the License are distributed on a "as is" BASIS, without warranties OR  CONDITIONS of any KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. -<!--Put Site-specific property overrides in this file. -<Configuration>    < Property>        <name>Mapreduce.framework.name</name>        <value>Yarn</value>    </ Property>    < Property>        <name>Mapred.compress.map.output</name>        <value>True</value>    </ Property>    < Property>        <name>Mapred.map.output.compression.codec</name>        <value>Com.hadoop.compression.lzo.LzoCodec</value>    </ Property>     < Property>         <name>Mapred.child.env</name>         <value>Ld_library_path=</value>        <name>Mapreduce.framework.name</name>        <value>Yarn</value>    </ Property>    < Property>        <name>Mapred.compress.map.output</name>        <value>True</value>    </ Property>    < Property>        <name>Mapred.map.output.compression.codec</name>        <value>Com.hadoop.compression.lzo.LzoCodec</value>    </ Property>     < Property>         <name>Mapred.child.env</name>         <value>Ld_library_path=d:\hadoop-2.7.3-win64\lib</value>     </ Property></Configuration>

Then CMD executes in the bin directory of Hadoop:

HDFs Namenode-format

Then execute it under the Sbin directory:

Start-all.cmd

Then the browser opens http://localhost:8088:

Execute Hadoop command: Hadoop fs-ls/

Empty, create a new folder: Hadoop fs-mkdir/data

Then view: Hadoop fs-ls/

This allows the local pseudo-distributed environment of Hadoop to be configured.

Windows configuration local Hadoop runtime environment