GitHub Address: HTTPS://GITHUB.COM/QINDONGLIANG/HIVE-SOLR
Welcome everyone fork and use
For an introduction to this project, please refer to the article in front of the penny Fairy:
http://qindongliang.iteye.com/blog/2283862
Latest update:
(1) Added support for Solrcloud cluster
(2) Fixed a bug in the handling of NULL columns and null values in hive when deserializing
(3) optimized for ignoring null and null values when building indexes
Some tests:
Data volume: Approximately 12 million, 8 fields, one of which is large text, 2 is a word breaker, and the data volume before index is about 20G
Total index time: about 15 minutes
Post-index Volume: approximately 6G per shard, approximately 18G
Hive: Limit the maximum number of concurrent maps to 30, for fear of affecting hbase services, note after using hive to build the index, you need to manually commit the memory index flush to disk
Batch processing: 100,000 data per map, batch processing submitted once, not commit, this value is set according to circumstances, too big Easy solrcloud easy to lose data, too small will affect the speed
Solrcloud Cluster version 5.1 uses 3 machines, each with one shard, no replicas, jetty memory gives 10G
Cpu:24 kernel, note that large text participle fields consume CPU
The JVM parameter tuning for SOLR:
(1) Increase the proportion of survivorratio area, reduce the memory space of survivor area
(2) Adjust the proportion of small newratio area, increase the memory space of Cenozoic
(3) Increase the permanent generation of maxpermsize memory to 256M
(4) Adjust the maxtenuringthreshold=0, so that large objects accelerate into the old age, to avoid copying back and forth in the Survivor and Eden areas, using YGC more times
Other parameters are also the default configuration
SOLR Server configuration:
(1) Turn off automatic commit
(2) Set RAMBUFFERSIZEMB to 1000, approx. 1G
(3) Set Maxbuffereddocs, etc.-1, disable Maxbufferdocs
(4) Set Mergefactor to 100
What's the problem can sweep the code to pay attention to the public number: I am the Siege division (Woshigcs), in the backstage message consultation.
Technical debt can not be owed, health debt can not be owed, seeking road, we walk.
Open source Big Data Index project HIVE-SOLR
