Spark To operate hbase

Spark is a computing framework. In the spark environment, it not only supports operations on standalone files and HDFS files, but also supports spark operations on hbase.

Data sources in enterprises are retrieved from hbase, which involves reading hbase data. This article uses spark shell for hbase operations to help you practice and operate hbase as soon as possible.

I. Environment:

Haoop2.2.0

Hbase version 0.96.2-hadoop2, r1581096

Spark1.0.0

This article assumes that the environment has been set up, and the spark haoop cluster can be seen when the spark environment is set up.

Note that hadoop2.2.0 is compatible with hbase. hbase 0.96.2 is used here.

Ii. Principles

Spark is actually operating hbase in the same way as Java client operating hbase:

Scala and Java are both JVM-based languages. You only need to load hbase classes into classpath to call operations. Other frameworks are similar.

Similarities: Connect to the hmaster as a client, and then use hbase APIs to operate hbase.

Differences: The only difference is that spark can process hbase data as RDD and use spark for parallel computing.

3. Practice 1. First check the dependent jar package. Before that, if the hbase jar package is not in spark-shell classpath, add it. Setting Method: Add spark_classpath =/home/Victor/software/hbase/lib/* to the Spark-evn.sh and start bin/spark-shell again. After the startup is complete and the worker is successfully registered, import jar package.
2. In hbase2.1 hbase, there is a score table in hbase, which contains two CF, course and grade. The data is as follows:

hbase(main):001:0> scan 'scores'ROW                                    COLUMN+CELL                                                                                                      Jim                                   column=course:art, timestamp=1404142440676, value=67                                                             Jim                                   column=course:math, timestamp=1404142434405, value=77                                                            Jim                                   column=grade:, timestamp=1404142422653, value=3                                                                  Tom                                   column=course:art, timestamp=1404142407018, value=88                                                             Tom                                   column=course:math, timestamp=1404142398986, value=97                                                            Tom                                   column=grade:, timestamp=1404142383206, value=5                                                                  shengli                               column=course:art, timestamp=1404142468266, value=17                                                             shengli                               column=course:math, timestamp=1404142461952, value=27                                                            shengli                               column=grade:, timestamp=1404142452157, value=8                                                                 3 row(s) in 0.3230 seconds

2.1 initialize connection Parameters

Scala> Import Org. apache. spark. _ import Org. apache. spark. _ Scala> Import Org. apache. spark. RDD. newhadooprddimport Org. apache. spark. RDD. newhadooprddscala> Import Org. apache. hadoop. conf. configuration; import Org. apache. hadoop. conf. configurationscala> Import Org. apache. hadoop. hbase. hbaseconfiguration; import Org. apache. hadoop. hbase. hbaseconfigurationscala> Import Org. apache. hadoop. hbase. mapreduce. tableinputformatimport Org. apache. hadoop. hbase. mapreduce. tableinputformatscala> Val configuration = hbaseconfiguration. create (); // initialize configuration: Org. apache. hadoop. conf. configuration = configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hbase-default.xml> configuration. set ("hbase. zookeeper. property. clientport "," 2181 "); // set the zookeeper client port Scala> configuration. set ("hbase. zookeeper. quorum "," localhost "); // set zookeeper quorumscala> configuration. set ("hbase. master "," localhost: 60000 "); // sets hbase masterscala> configuration. addresource ("/home/Victor/software/hbase/CONF/hbase-site.xml") // load the hbase configuration Scala> Import Org. apache. hadoop. hbase. client. hbaseadminimport Org. apache. hadoop. hbase. client. hbaseadmin

Scala> Val hadmin = new hbaseadmin (configuration); // instantiate hbase management 00:39:24, 649 info [main] zookeeper. zookeeper (zookeeper. java: <init> (438)-initiating client connection, connectstring = localhost: 2181 sessiontimeout = 90000 watcher = hconnection-0xc7eea5, quorum = localhost: 2181, baseznode =/hbase2014-07-01 00:39:24, 707 info [main] zookeeper. recoverablezookeeper (recoverablezookeeper. java: <init> (120)-process identifier = hconnection-0xc7eea5 connecting to zookeeper ensemble = localhost: 21812014-07-01 00:39:24, 753 info [main-sendthread (localhost: 2181)] zookeeper. clientcnxn (clientcnxn. java: logstartconnect (966)-opening socket connection to server localhost/127.0.0.1: 2181. will not attempt to authenticate using SASL (unknown error) 00:39:24, 755 info [main-sendthread (localhost: 2181)] zookeeper. clientcnxn (clientcnxn. java: primeconnection (849)-socket connection established to localhost/127.0.0.1: 2181, initiating session2014-07-01 00:39:24, 938 info [main-sendthread (localhost: 2181)] zookeeper. clientcnxn (clientcnxn. java: onconnected (1207)-session establishment complete on server localhost/127.0.0.1: 2181, sessionid = 0x146ed61c4ef0015, negotiated timeout = 40000 hadmin: Org. apache. hadoop. hbase. client. hbaseadmin = [email protected]

Next, use the haoop API to create an RDD.

scala> val hrdd = sc.newAPIHadoopRDD(configuration, classOf[TableInputFormat],      | classOf[org.apache.hadoop.hbase.io.ImmutableBytesWritable],     | classOf[org.apache.hadoop.hbase.client.Result])2014-07-01 00:51:06,683 WARN  [main] util.SizeEstimator (Logging.scala:logWarning(70)) - Failed to check whether UseCompressedOops is set; assuming yes2014-07-01 00:51:06,936 INFO  [main] storage.MemoryStore (Logging.scala:logInfo(58)) - ensureFreeSpace(85877) called with curMem=0, maxMem=3089104892014-07-01 00:51:06,946 INFO  [main] storage.MemoryStore (Logging.scala:logInfo(58)) - Block broadcast_0 stored as values to memory (estimated size 83.9 KB, free 294.5 MB)hrdd: org.apache.spark.rdd.RDD[(org.apache.hadoop.hbase.io.ImmutableBytesWritable, org.apache.hadoop.hbase.client.Result)] = NewHadoopRDD[0] at newAPIHadoopRDD at <console>:22

Read records: here we take 1 piece of data, and we can see that the format is based on our set hadooprdd. Key is a constant immutablebyteswritable, and value is the result of hbase.

scala> hrdd take 12014-07-01 00:51:50,371 INFO  [main] spark.SparkContext (Logging.scala:logInfo(58)) - Starting job: take at <console>:252014-07-01 00:51:50,423 INFO  [spark-akka.actor.default-dispatcher-16] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Got job 0 (take at <console>:25) with 1 output partitions (allowLocal=true)2014-07-01 00:51:50,425 INFO  [spark-akka.actor.default-dispatcher-16] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Final stage: Stage 0(take at <console>:25)2014-07-01 00:51:50,426 INFO  [spark-akka.actor.default-dispatcher-16] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Parents of final stage: List()2014-07-01 00:51:50,477 INFO  [spark-akka.actor.default-dispatcher-16] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Missing parents: List()2014-07-01 00:51:50,478 INFO  [spark-akka.actor.default-dispatcher-16] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Computing the requested partition locally2014-07-01 00:51:50,509 INFO  [Local computation of job 0] rdd.NewHadoopRDD (Logging.scala:logInfo(58)) - Input split: localhost:,2014-07-01 00:51:50,894 INFO  [main] spark.SparkContext (Logging.scala:logInfo(58)) - Job finished: take at <console>:25, took 0.522612687 sres5: Array[(org.apache.hadoop.hbase.io.ImmutableBytesWritable, org.apache.hadoop.hbase.client.Result)] = Array((4a 69 6d,keyvalues={Jim/course:art/1404142440676/Put/vlen=2/mvcc=0, Jim/course:math/1404142434405/Put/vlen=2/mvcc=0, Jim/grade:/1404142422653/Put/vlen=1/mvcc=0}))

Locate the result object

scala> val res = hrdd.take(1)2014-07-01 01:09:13,486 INFO  [main] spark.SparkContext (Logging.scala:logInfo(58)) - Starting job: take at <console>:242014-07-01 01:09:13,487 INFO  [spark-akka.actor.default-dispatcher-15] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Got job 4 (take at <console>:24) with 1 output partitions (allowLocal=true)2014-07-01 01:09:13,487 INFO  [spark-akka.actor.default-dispatcher-15] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Final stage: Stage 4(take at <console>:24)2014-07-01 01:09:13,487 INFO  [spark-akka.actor.default-dispatcher-15] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Parents of final stage: List()2014-07-01 01:09:13,488 INFO  [spark-akka.actor.default-dispatcher-15] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Missing parents: List()2014-07-01 01:09:13,488 INFO  [spark-akka.actor.default-dispatcher-15] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Computing the requested partition locally2014-07-01 01:09:13,488 INFO  [Local computation of job 4] rdd.NewHadoopRDD (Logging.scala:logInfo(58)) - Input split: localhost:,2014-07-01 01:09:13,504 INFO  [main] spark.SparkContext (Logging.scala:logInfo(58)) - Job finished: take at <console>:24, took 0.018069267 sres: Array[(org.apache.hadoop.hbase.io.ImmutableBytesWritable, org.apache.hadoop.hbase.client.Result)] = Array((4a 69 6d,keyvalues={Jim/course:art/1404142440676/Put/vlen=2/mvcc=0, Jim/course:math/1404142434405/Put/vlen=2/mvcc=0, Jim/grade:/1404142422653/Put/vlen=1/mvcc=0}))scala> res(0)res33: (org.apache.hadoop.hbase.io.ImmutableBytesWritable, org.apache.hadoop.hbase.client.Result) = (4a 69 6d,keyvalues={Jim/course:art/1404142440676/Put/vlen=2/mvcc=0, Jim/course:math/1404142434405/Put/vlen=2/mvcc=0, Jim/grade:/1404142422653/Put/vlen=1/mvcc=0})scala> res(0)._2res34: org.apache.hadoop.hbase.client.Result = keyvalues={Jim/course:art/1404142440676/Put/vlen=2/mvcc=0, Jim/course:math/1404142434405/Put/vlen=2/mvcc=0, Jim/grade:/1404142422653/Put/vlen=1/mvcc=0}scala> val rs = res(0)._2rs: org.apache.hadoop.hbase.client.Result = keyvalues={Jim/course:art/1404142440676/Put/vlen=2/mvcc=0, Jim/course:math/1404142434405/Put/vlen=2/mvcc=0, Jim/grade:/1404142422653/Put/vlen=1/mvcc=0}scala> rs.asInstanceOf             cellScanner              containsColumn           containsEmptyColumn      containsNonEmptyColumn   copyFrom                 getColumn                getColumnCells           getColumnLatest          getColumnLatestCell      getExists                getFamilyMap             getMap                   getNoVersionMap          getRow                   getValue                 getValueAsByteBuffer     isEmpty                  isInstanceOf             list                     listCells                loadValue                raw                      rawCells                 setExists                size                     toString                 value                    

Traverse this record and retrieve the value of each cell:

scala> val kv_array = rs.rawwarning: there were 1 deprecation warning(s); re-run with -deprecation for detailskv_array: Array[org.apache.hadoop.hbase.KeyValue] = Array(Jim/course:art/1404142440676/Put/vlen=2/mvcc=0, Jim/course:math/1404142434405/Put/vlen=2/mvcc=0, Jim/grade:/1404142422653/Put/vlen=1/mvcc=0)

Traverse records

scala> for(keyvalue <- kv) println("rowkey:"+ new String(keyvalue.getRow)+ " cf:"+new String(keyvalue.getFamily()) + " column:" + new String(keyvalue.getQualifier) + " " + "value:"+new String(keyvalue.getValue()))warning: there were 4 deprecation warning(s); re-run with -deprecation for detailsrowkey:Jim cf:course column:art value:67rowkey:Jim cf:course column:math value:77rowkey:Jim cf:grade column: value:3

Number of query records

scala> hrdd.count2014-07-01 01:26:03,133 INFO  [main] spark.SparkContext (Logging.scala:logInfo(58)) - Starting job: count at <console>:252014-07-01 01:26:03,134 INFO  [spark-akka.actor.default-dispatcher-16] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Got job 5 (count at <console>:25) with 1 output partitions (allowLocal=false)2014-07-01 01:26:03,134 INFO  [spark-akka.actor.default-dispatcher-16] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Final stage: Stage 5(count at <console>:25)2014-07-01 01:26:03,134 INFO  [spark-akka.actor.default-dispatcher-16] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Parents of final stage: List()2014-07-01 01:26:03,135 INFO  [spark-akka.actor.default-dispatcher-16] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Missing parents: List()2014-07-01 01:26:03,166 INFO  [spark-akka.actor.default-dispatcher-16] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Submitting Stage 5 (NewHadoopRDD[0] at newAPIHadoopRDD at <console>:22), which has no missing parents2014-07-01 01:26:03,397 INFO  [spark-akka.actor.default-dispatcher-16] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Submitting 1 missing tasks from Stage 5 (NewHadoopRDD[0] at newAPIHadoopRDD at <console>:22)2014-07-01 01:26:03,401 INFO  [spark-akka.actor.default-dispatcher-16] scheduler.TaskSchedulerImpl (Logging.scala:logInfo(58)) - Adding task set 5.0 with 1 tasks2014-07-01 01:26:03,427 INFO  [spark-akka.actor.default-dispatcher-16] scheduler.FairSchedulableBuilder (Logging.scala:logInfo(58)) - Added task set TaskSet_5 tasks to pool default2014-07-01 01:26:03,439 INFO  [spark-akka.actor.default-dispatcher-5] scheduler.TaskSetManager (Logging.scala:logInfo(58)) - Starting task 5.0:0 as TID 0 on executor 0: 192.168.2.105 (PROCESS_LOCAL)2014-07-01 01:26:03,469 INFO  [spark-akka.actor.default-dispatcher-5] scheduler.TaskSetManager (Logging.scala:logInfo(58)) - Serialized task 5.0:0 as 1305 bytes in 7 ms2014-07-01 01:26:11,015 INFO  [Result resolver thread-0] scheduler.TaskSetManager (Logging.scala:logInfo(58)) - Finished TID 0 in 7568 ms on 192.168.2.105 (progress: 1/1)2014-07-01 01:26:11,017 INFO  [Result resolver thread-0] scheduler.TaskSchedulerImpl (Logging.scala:logInfo(58)) - Removed TaskSet 5.0, whose tasks have all completed, from pool default2014-07-01 01:26:11,036 INFO  [spark-akka.actor.default-dispatcher-4] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Completed ResultTask(5, 0)2014-07-01 01:26:11,057 INFO  [spark-akka.actor.default-dispatcher-4] scheduler.DAGScheduler (Logging.scala:logInfo(58)) - Stage 5 (count at <console>:25) finished in 7.605 s2014-07-01 01:26:11,067 INFO  [main] spark.SparkContext (Logging.scala:logInfo(58)) - Job finished: count at <console>:25, took 7.933270634 sres71: Long = 3

Iv. Summary
Spark's hbase operations are basically the same as the Java client's hbas operations. All of them are the client's connection to the hmaster and the Java API is used to operate hbase. However, spark provides a combination of RDD and Scala syntax to improve programming efficiency.
-- EOF --
Original article, reproduced please indicate from: http://blog.csdn.net/oopsoom/article/details/36071323