Hadoop read/write elasticsearch

1. References:

Http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/configuration.html

Http://www.elasticsearch.org/guide/en/elasticsearch/hadoop/current/mapreduce.html#_emphasis_old_emphasis_literal_org_apache_hadoop_mapred_literal_api

 

2. mapreduce Configuration

 

 

// The following elasticsearch configurations are mainly provided to the elasticsearch format class for reading.

Configuration conf =NewConfiguration ();

Conf. Set (configurationoptions. es_nodes, "127.0.0.1 ");

Conf. Set (configurationoptions. es_port, "9200 ");

Conf. Set (configurationoptions. es_index_auto_create, "yes ");

// Set the index/type of the read and write Resources

Conf. Set (configurationoptions. es_resource, "helloes/demo"); // read target index/Type

 

 

 

// If you only want to retrieve part of the data, you can configure es_query

// Conf. Set (configurationoptions. es_query ,"? Q = me *");

 

// Configure elasticsearch as the format developed by hadoop.

Job job = job.Getinstance(Conf, elasticsearchindexmapper.Class. Getsimplename ());

Job. setjarbyclass (elasticsearchindexbuilder.Class);

Job. setspeculativeexecution (False); // Disable speculative execution

Job. setinputformatclass (esinputformat.Class);

 

// If the data is output to HDFS, specify the format of map output value. Select text format

Job. setoutputformatclass (textoutputformat.Class);

Job. setmapoutputvalueclass (text.Class);

Job. setmapoutputkeyclass (nullwritable.Class);

 

 

// If you select to input

Job. setoutputformatclass (esoutputformat.Class); // Output

Job. setmapoutputvalueclass (extends mapwritable.Class); // Output value class

Job. setmapoutputkeyclass (text.Class); // Output key value class

 

 

Job. setmapperclass (elasticsearchindexmapper.Class);

Fileinputformat.Addinputpath(Job,NewPATH ("HDFS: // localhost: 9000/es_input "));

Fileoutputformat.Setoutputpath(Job,NewPATH ("HDFS: // localhost: 9000/es_output "));

Job. setnumreducetasks (0 );

Job. waitforcompletion (True);

 

3. Corresponding mapper class elasticsearchindexmapper

Public ClassElasticsearchindexmapperExtendsMapper {

@ Override

Protected Void Map(Object key, object value, context)

ThrowsIoexception, interruptedexception {

// Assume that I want to export data to HDFS.

 

Linkedmapwritable Doc = (linkedmapwritable) value;

Text docval =NewText ();

Docval. Set (Doc. tostring ());

Context. Write (nullwritable. Get (), docval );

}

}

4. Summary

The most important thing for hadoop-es reading and writing is the parameter configuration of esinputformat and esoutputformat ).

In addition, other data source operations (such as MySQL) are similar. Find the corresponding inputformat and outputformat to configure the environment parameters.

Hadoop read/write elasticsearch