How do I use hive to integrate SOLR?

(a) HIVE+SOLR profile

As the offline data warehouse of the Hadoop ecosystem, hive can easily use SQL to analyze the huge amount of historical data offline, and according to the analysis results, to do some other things, such as report statistics query.
SOLR, as a high-performance search server, provides fast, powerful, full-text retrieval capabilities.

(b) Why is hive integration SOLR required?

Sometimes, we need to store the results of the analysis of hive into the full-text search service in SOLR, for example, we have a business, the search log on our e-commerce website using hive analysis after the storage to SOLR inside to do report query, because it involves search keywords, This field is required to be able to participle query and non-segmentation query, through the Word segmentation query can see the related product changes in a certain period of time a trend chart. Sometimes, we need to load the data in SOLR into hive and use SQL to do some of the join analysis functions, which are complementary to each other to better suit our business needs. Of course there are some open-source projects on the web that have hive integration SOLR, but because of the older version, it is not possible to run in the new version, and after the transformation, it can be run in the latest version.

(c) How can I enable hive integration with SOLR?

The so-called integration is actually some of the components that rewrite the Mr Programming interface of Hadoop. We all know that Mr's programming interfaces are very flexible and highly abstracted, and Mr is not just able to load data sources from HDFs, but can also load data from any non-HDFS system, if we need to customize:
InputFormat
OutputFormat
Recordreader
Recordwriter
Inputsplit
Component, though a bit of a hassle, but loading data from anywhere can really do it, including Mysql,sqlserver,oracle,mongodb, Solr,es,redis and so on.

This is the Mr Programming interface for custom Hadoop, in addition to some of the components above, additional definitions of Serde components and assembly Storagehandler are required, and serde in hive refers to serializer and Deserializer, which is what we call serialization and deserialization, hive needs to use Serde and fileinput to read and write a row of rows of data in a hive table.
Read the process:
HDFS files/every Source--Inputfileformat------Deserializer Row Object
The process of writing:
Row Object---serializer---Outputfileformat, HDFS files/every source

(iv) What can I do after hive integration Solr?

(1) Read SOLR data to Hive's supported SQL syntax, able to perform various aggregations, statistics, analyses, joins, etc.
(2) Generate SOLR Index, one sentence of SQL, can be an Mr Way to index large-scale data

(v) How to install the deployment and use?
Source here, not pasted, has uploaded GitHub, there is a need for friends can use Git clonehttps://github.com/qindongliang/hive-solr after modifying a little pom file after the execution
MVN Clean Package
command to build the jar package and copy the jar package to the Hive's Lib directory

Examples are as follows:
(1) Hive reads SOLR data

Build table:

SQL code

2. --The Presence table is deleted
4. Drop table if exists SOLR;
8. --Create an external table
10. Create external table SOLR (
12. --Define fields, where the fields need to be consistent with SOLR's fields
14. Rowkey String,
16. Sname string
20. )
22. --Define the storehandler of the store
24. Stored by "Com.easy.hive.store.SolrStorageHandler"
26. --Configure SOLR properties
28. Tblproperties (' solr.url ' = ' http://192.168.1.28:8983/solr/a ',
30. ' solr.query ' = ' *:* ',
32. ' solr.cursor.batch.size ' =' 10000 ',
34. ' solr.primary_key ' =' Rowkey '
36. );
38. Execute the bin/hive command for hive command-line terminal:
42. --Query all data
44. SELECT * FROM SOLR limit 5;
46. --Query the specified field
48. Select Rowkey from SOLR;
50. --to aggregate statistical SOLR data in the form of Mr
52. Select Sname,count (*) as C from solr Group by sname ORDER by C Desc /c13>

(2) Example of building an index to SOLR using hive

First build the Data source table:

SQL code

2. --delete if it exists
4. Drop table if exists index_source;
8. --Building a data table
10. CREATE TABLE index_source (ID string, yname string,sname string) ROW FORMAT delimited fields TERMINATED by ', '  STORED as textfile;
14. --Import local data into the data source
16. Load Data local inpath '/ROOT/SERVER/HIVE/TEST_SOLR ' into table index_source;
18. Next, build an association table for SOLR:
22. --delete a table that already exists
24. Drop table if exists INDEX_SOLR;
28. --Create an associated SOLR table
30. Create external table INDEX_SOLR (
32. ID string,
34. Yname String,
36. Sname string
38. )
40. --Defining the storage engine
42. Stored by "Com.easy.hive.store.SolrStorageHandler"
44. --Set SOLR service properties
46. Tblproperties (' solr.url ' = ' http://192.168.1.28:8983/solr/b ',
48. ' solr.query ' = ' *:* ',
50. ' solr.cursor.batch.size ' =' 10000 ',
52. ' solr.primary_key ' =' id '
54. );

Finally, execute the following SQL command to build the SOLR index to the data in the data source:

SQL code

1. --register HIVE-SOLR jar package, otherwise the MR Mode will not start properly   
2. add jar /root/server/hive/lib/hive-solr.jar;  
3. --execute insert command   
4. insert  Overwrite table index_solr select * from  index_source ;   
5. span class= "comment" >--after successful execution, can be viewed in the SOLR terminal interface, or in hive inside the following SOLR query   
6. select * from index_solr limit 10 ;   

(vi) Can they also integrate with other frameworks?

Of course, as an open source stand-alone framework, we can do a variety of combinations, hive can be integrated with Elasticsearch, can also be integrated with MongoDB, SOLR can be integrated with spark, can also be integrated with pig, but all need us to customize the relevant components to the line, The idea is roughly the same as that of the project.

(vii) Basic environment for this test

Apache Hadoop2.7.1
Apache Hive1.2.1
Apache Solr5.1.0

(eight) Thanks and reference information:

Https://github.com/mongodb/mongo-hadoop/tree/master/hive/src/main/java/com/mongodb/hadoop/hive
Https://github.com/lucidworks/hive-solr
Https://github.com/chimpler/hive-solr
Https://cwiki.apache.org/confluence/display/Hive/DeveloperGuide#DeveloperGuide-HowtoWriteYourOwnSerDe

How do I use hive to integrate SOLR?