How to build indexes using spark massively parallel

Building an index with spark is simple, because spark provides a more advanced, abstract Rdd distributed elastic dataset that builds large-scale indexes compared to previous mapreduce using Hadoop, and Spark has a number of advantages such as more flexible API operations, higher performance, and more concise syntax.

First look at the overall topology diagram:





Then, take a look at the spark program written in Scala:

Java code

2. Package Com.easy.build.index
6. Import Java.util
10. Import Org.apache.solr.client.solrj.beans.Field
12. Import Org.apache.solr.client.solrj.impl.HttpSolrClient
14. Import Org.apache.spark.rdd.RDD
16. Import Org.apache.spark. {sparkconf, Sparkcontext}
20. Import Scala.annotation.meta.field
22. /**
24. * Created by Qindongliang on 2016/1/21.
26. */
30. Register the model, the time type can be a string, as long as the background index is configured as long, the annotation map form as follows
32. Case class Record (
34. @ (Field@field) ("Rowkey") rowkey:string,
36. @ (Field@field) ("title") title:string,
38. @ (Field@field) ("content") content:string,
40. @ (Field@field) ("Isdel") isdel:string,
42. @ (Field@field) ("T1") t1:string,
44. @ (Field@field) ("T2") t2:string,
46. @ (Field@field) ("T3") t3:string,
48. @ (Field@field) ("Dtime") dtime:string
54. )
58. /***
60. * Spark Build Index ==>SOLR
62. */
64. Object Sparkindex {
68. //SOLR Client
70. Val client=New Httpsolrclient ("Http://192.168.1.188:8984/solr/monitor");
72. number of articles submitted by//batch
74. Val batchcount=10000;
78. def main2 (Args:array[string]) {
82. Val d1=New Record ("Row1","title", "content","1","n","", "+","3");
84. Val d2=New Record ("Row2","title", "content","1","n","", "+","45");
86. Val d3=New Record ("row3","title", "content","1","n", "$","$",null);
88. Client.addbean (D1);
90. Client.addbean (D2)
92. Client.addbean (D3)
94. Client.commit ();
96. println ("submitted successfully! ")  
102. }
108. /*** 
110. * Iterate partition data (an iterator collection) and process
112. * @param lines processing data for each partition
114. */
116. Def indexpartition (Lines:scala. Iterator[string]): Unit ={
118. //Initialize the collection, you can initialize some content, such as database connection, before the partition iteration begins
120. Val datas = new util. Arraylist[record] ()
122. //iterate over each piece of data and meet the criteria to submit the data
124. Lines.foreach (Line=>indexlinetomodel (Line,datas))
126. //After the operation partition is finished, you can close some resources, or do some operations, the last time you commit the data
128. COMMITSOLR (Datas,true);
130. }
134. /*** 
136. * Submit index data to SOLR
138. *
140. * @param datas Index data
142. * @param isend is the last commit
144. */
146. Def COMMITSOLR (Datas:util. Arraylist[record],isend:boolean): Unit ={
148. committed only when the last commit and the collection length equals the number of batches
150. if ((Datas.size () >0&&isend) | | Datas.size () ==batchcount) {
152. Client.addbeans (datas);
154. Client.commit (); //Submit data
156. Datas.clear (); //Clear collection for easy reuse
158. }
160. }
166. /*** 
168. * Get partitioned data specific per row, and map
170. * To model for subsequent indexing processing
172. *
174. * @param line specific data
176. * @param datas Add a collection of data for bulk-submit indexes
178. */
180. Def Indexlinetomodel (Line:string,datas:util. Arraylist[record]): Unit ={
182. //Array data Cleaning conversion
184. Val fields=line.split ("\1",-1). map (field =>etl_field (field))
186. //Map the cleaned array into a tuple type
188. Val tuple=buildtuble (Fields)
190. //Convert tuple to bean type
192. Val recoder=record.tupled (tuple)
194. //Add entity classes to the collection for easy batch submission
196. Datas.add (Recoder);
198. //Submit index to SOLR
200. COMMITSOLR (Datas,false);
202. }
208. /*** 
210. * Map arrays into tuple collections for easy binding with beans
212. * @param array Field Collection
214. * @return Tuple Collection
216. */
218. def buildtuble (array:array[string]):(string, String, String, String, String, String, String, ={
220. Array Match {
222. Case Array (S1, S2, S3, S4, S5, S6, S7, s8) = (S1, S2, S3, S4, S5, S6, S7,S8)
224. }
226. }
232. /*** 
234. * Processing of field
236. * Null value is replaced with NULL, so the index does not index this field
238. * The normal value is returned as it is.
240. *
242. * @param field used to take specific rules of data
244. * @return The data that was mapped out
246. */
248. def Etl_field (field:string): string={
250. Field Match {
252. case "" = null
254. Case _ = Field
256. }
258. }
262. /*** 
264. * Purge a class of indexed data based on conditions
266. * @param query criteria to delete
268. */
270. def deletesolrbyquery (query:string): Unit ={
272. Client.deletebyquery (query);
274. Client.commit ()
276. println ("Delete succeeded!")
278. }
284. def main (args:array[string]) {
286. //Delete some data based on conditions
288. Deletesolrbyquery ("t1:03")
290. //Remote Commit, you need to submit the packaged Jar
292. Val jarpath = "Target\\spark-build-index-1.0-snapshot.jar";
294. //Remote Commit, disguised as the relevant Hadoop user, otherwise, may not have access to the HDFS system
296. System.setproperty ("User.Name", "webmaster");
298. //Initialize sparkconf
300. Val conf = new sparkconf (). Setmaster ("spark://192.168.1.187:7077"). Setappname ("Build Index");
302. //upload the runtime-dependent jar package
304. Val seq = seq (jarpath): + "D:\\tmp\\lib\\noggit-0.6.jar": + "D:\\tmp\\lib\\httpclient-4.3.1.jar": + "d:\\  Tmp\\lib\\httpcore-4.3.jar ": + " D:\\tmp\\lib\\solr-solrj-5.1.0.jar ": + " D:\\tmp\\lib\\httpmime-4.3.1.jar "
306. Conf.setjars (seq)
308. //Initialize Sparkcontext context
310. Val sc = new Sparkcontext (conf);
312. all data in this directory will be indexed and the format must be agreed
314. Val Rdd = Sc.textfile ("hdfs://192.168.1.187:9000/user/monitor/gs/");
316. //Build an index with an RDD
318. Indexrdd (RDD);
320. //Close index resource
322. Client.close ();
324. //Close Sparkcontext context
326. Sc.stop ();
332. }
338. /*** 
340. * Process RDD data, build index
342. * @param RDD
344. */
346. def Indexrdd (rdd:rdd[string]): Unit ={
348. //traversing partitions, building indexes
350. Rdd.foreachpartition (Line=>indexpartition (line));
352. }
360. }

OK, so far, our build index program is finished, this example is used in the remote commit mode, in fact it can also support spark on yarn (cluster or client) mode, but it is important to note that you do not need to explicitly specify the value of Setmaster, and by the time the task is submitted , through the--master to specify the mode of operation, in addition, dependent on the relevant jar package, also need to be submitted to the cluster through the--jars parameter, otherwise, the runtime will report an exception, and finally see the example of SOLR is a stand-alone mode, so using spark to build an index does not reach the maximum value , the most powerful thing is that many search clusters, as I draw the architecture diagram, each machine is a shard, this is the Solrcloud mode, or in the Elasticsearch cluster Shard, so that can really achieve efficient batch index construction

How to build indexes using spark massively parallel