Using Spring-hadoop Summary

Springhadoop is called HDFs through the spring framework, and the biggest difference from calling HDFs directly is that spring generates the configuration and filesystem objects needed to operate HDFs through dependency injection. All other APIs that call HDFs do not change

1. Create the Resources folder in the project's main directory and add it to the project's resource file.

??

2. Create Beans.xml and Application.properties files under the Resources folder, respectively, to inject objects and manage Profiles 3. Add dependencies for Hadoop and Spring-hadoop

    <!--You can define variables here, you can manage the version number and<properties>    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>    <!--Unified management of Hadoop versions --    2.6.0-cdh5.7.0</properties>    <repositories>    <!--add warehouses to download Hadoop--    <repository>        <id>Cloudera</id>        <url>Https://repository.cloudera.com/artifactory/cloudera-repos</url>    </repository></repositories><dependencies>    <!--adding Hadoop dependency --    <dependency>        <groupId>Org.apache.hadoop</groupId>        <artifactId>Hadoop-client</artifactId>        <version>${hadoop.version}</version>        <!--This package is already available in the production environment, so there is no need to hit the project--        <scope>Provided</scope>    </dependency>    <!--add Spring-hadoop dependency    <dependency>        <groupId>Org.springframework.data</groupId>        <artifactId>Spring-data-hadoop</artifactId>        <version>2.5.0.RELEASE</version>    </dependency></dependencies>

4. Configuring the Configuration and FileSystem objects in Beans.xml and application.properties files

Beans.xml file

 <!-- The property profile Application.properties is referenced so that the attribute--> in the configuration file can be used in the XML file;  <context: Property-placeholder   location=   "application.properties"  span class= "kw" >/>  <!--configuration of the configurtion--> of HDFs;  <HDP: Configuration   id=   "hadoopconfiguration"   <!--configuration Namenode address-->  fs.defaultfs=${spring.hadoop.fs-uri}</HDP:CONFIGURATION>    id=  Span class= "st" > "FileSystem"   configuration-ref=    user=  " root " /> 

Application.properties file

spring.hadoop.fs-uri=hdfs://hadoop01:8020

5. Finally, the Java Unit test code

Package Spring;import Org.apache.hadoop.fs.FSDataInputStream;import Org.apache.hadoop.fs.FileSystem;import Org.apache.hadoop.fs.Path;import org.apache.hadoop.io.IOUtils;import Org.junit.After;import Org.junit.Before;import org.junit.Test;import Org.springframework.context.ApplicationContext;import Org.springframework.context.support.ClassPathXmlApplicationContext;import java.io.IOException;/** * @authorHT* @create 2018-01-30 23:03* @desc Springhadoop test class **/ Public classSpringhadoopapp {//spring Context    PrivateApplicationContext Mcontext;//hdfs File System objects    PrivateFileSystem Mfilesystem;/*** Test Create folder     */    @Test     Public void mkdir()throwsIOException {Mfilesystem.mkdirs(New Path("/test")); }/*** Test reads files from the server     */    @Test     Public void text()throwsIOException {Fsdatainputstream is = Mfilesystem.Open(New Path("/test/install.log.syslog")); Ioutils.copybytes(Is,system. out,1024x768); Is.Close(); }@Before     Public void setUp() {//Get spring context, spring's dependency injection, is to inject the object into the beans, similar to the moudle in the Dagger2, specifically responsible for generating the objectMcontext =New Classpathxmlapplicationcontext("Beans.xml");//Get FileSystem object by Beans.xml fileMfilesystem = (FileSystem) mcontext.Getbean("FileSystem"); }@After     Public void TearDown()throwsIOException {Mcontext =NULL; Mfilesystem.Close(); }}

Using Spring-hadoop Summary