"Big Data Series" Windows environment to build Hadoop development environment read data from Hadoop URLs

Objective

Build a Java project in a Windows environment after the Hadoop cluster has been built test operations files in HDFs

Version One

Package Com.slp.hadoop274.hdfs;import Java.io.ioexception;import Java.io.inputstream;import java.net.URL;import Java.net.urlconnection;import org.apache.hadoop.fs.fsurlstreamhandlerfactory;import org.junit.Test;/** *  * @ Author Sangliping * Complete HDFs operation */public class Testhdfs {    /**     * Read HDFs file     * @throws ioexception      */@ testpublic void ReadFile () throws Ioexception{url url = new URL ("hdfs://192.168.181.201:8020/user/sanglp/hadoop/ Copyfromlocal "); URLConnection con = url.openconnection (); InputStream is = Con.getinputstream (); byte[] buf = new byte[is.available ()]; Is.read (BUF); Is.close (); String str = new String (buf, "UTF-8"); System.out.println (str);}}

The above error occurs when running the test because the URL does not recognize the HDFS protocol .

Version two,

Package Com.slp.hadoop274.hdfs;import Java.io.ioexception;import Java.io.inputstream;import java.net.URL;import Java.net.urlconnection;import org.apache.hadoop.fs.fsurlstreamhandlerfactory;import org.junit.Test;/** *  * @ Author Sangliping * Complete HDFs operation */public class Testhdfs {static{//Register HDFS protocol Otherwise the URL does not recognize the protocol url.seturlstreamhandlerfactory (new Fsurlstreamhandlerfactory ());}    /**     * Read HDFs file     * @throws IOException *      /@Testpublic void ReadFile () throws Ioexception{url url = new URL ("HDFs ://192.168.181.201:8020/user/sanglp/hadoop/copyfromlocal "); URLConnection con = url.openconnection (); InputStream is = Con.getinputstream (); byte[] buf = new byte[is.available ()]; Is.read (BUF); Is.close (); String str = new String (buf, "UTF-8"); System.out.println (str);}}

This time you can correctly print out the contents of the file copyfromlocal HDFs file.

Attached: You can put the log4j.properties file in etc under the Hadoop decompression file into the project file src file to make the console printing more friendly.

Big data series Windows environments build Hadoop development environment read data from Hadoop URLs