Java operation HDFs to database or cache

When you use Hadoop tools to analyze your data, you need to do it in storage or in the cache, or it doesn't make sense.

Here is the code that uses JAVAAPI to manipulate HDFs into the cache:

<span style= "Font-family:microsoft yahei;font-size:14px;" >public class Interesttoredisjob {FileSystem HDFs = null;public interesttoredisjob () {init ();} private void init () {Configuration conf = new Configuration (); Conf.set ("Fs.default.name", "HDFs address"); try {HDFs = Filesystem.get (conf);} catch (IOException e) {e.printstacktrace ();}} public void Readfiletoreids (String path) throws Ioexception{path paths = new Path (path); filestatus[] files = hdfs.liststatus (paths);</span>

<span style= "Font-family:microsoft yahei;font-size:14px;" ><span style= "White-space:pre" ></span>//here is their own implementation of a path of concern, also can not be applied, in the "1" directly judge the part-r-and other indicators </span>

<span style= "Font-family:microsoft yahei;font-size:14px;" >pathfilter filter = new Resultnamefilter ("part-r-"); Text line = new text (); Redisclient Redis = new Redisclient (); for (Filestatus file:files) {if (File.isdir () | |!filter.accept (File.getpath ())) {/ /"1" continue;} Else{fsdatainputstream input = Null;try{input = Hdfs.open (File.getpath ()); Linereader reader = new Linereader (input) while (Reader.readline (line) > 0) {System.out.println (line); string[] arr = line.tostring (). Split ("\ T");</span>

<span style= "Font-family:microsoft yahei;font-size:14px;" ><span style= "White-space:pre" ></span>//do deposit Redis processing redis.savehsetvalue (arr[0], "interest", arr[1]);}} catch (Exception e) {e.printstacktrace ();} Finally{if (input! = null) {Input.close ();}}}}} public static void Main (string[] args) {interesttoredisjob job = new Interesttoredisjob (); try {job. Readfiletoreids ("Your Path");} catch (IOException e) {e.printstacktrace ();}}} </span>

the above code needs to be modified after use, do not paste directly.

Java operation HDFs to database or cache