Apache Spark Technology Practice 3 -- Installation and Use of spark Cassandra Connector

You are welcome to reprint it. Please indicate the source, huichiro.

Prerequisites

Assume that the following software has been installed.

• JDK
• SBT
• Git
• Scala

Install cassandra

Take archlinux as an example. Run the following command to install CASSANDRA:

yaourt -S cassandra

Start cassandra

cassandra -f

Create keyspace and table and run/Usr/bin/cqlshGo to the cql console and run the following statement to create the keyspace and table

CREATE KEYSPACE test WITH replication = {‘class‘: ‘SimpleStrategy‘, ‘replication_factor‘: 1 };CREATE TABLE test.kv(key text PRIMARY KEY, value int);

Add record and continue to use cql Console

INSERT INTO test.kv(key, value) VALUES (‘key1‘, 1);INSERT INTO test.kv(key, value) VALUES (‘key2‘, 2);

The verification record has been inserted successfully. Run the following cql command:

select * from test.kv;

Download and compile spark-Cassandra-connector

Download the latest spark-Cassandra-connector source code

git clone https://github.com/datastax/spark-cassandra-connector.git

Compile

sbt package

Please be patient.

Run spark-shell

First, make sure that Cassandra is installed and running properly. If you have any questions, go back to the start section to install Cassandra.

How to add the corresponding library to support spark-Cassandra-connector without a clear document. After one afternoon, I finally made a simple configuration.

bin/spark-shell --driver-class-path /root/working/spark-cassandra-connector/spark-cassandra-connector/target/scala-2.10/spark-cassandra-connector_2.10-1.1.0-SNAPSHOT.jar:/root/.ivy2/cache/org.apache.cassandra/cassandra-thrift/jars/cassandra-thrift-2.0.9.jar:/root/.ivy2/cache/org.apache.thrift/libthrift/jars/libthrift-0.9.1.jar:/root/.ivy2/cache/org.apache.cassandra/cassandra-clientutil/jars/cassandra-clientutil-2.0.9.jar:/root/.ivy2/cache/com.datastax.cassandra/cassandra-driver-core/jars/cassandra-driver-core-2.0.4.jar:/root/.ivy2/cache/io.netty/netty/bundles/netty-3.9.0.Final.jar:/root/.ivy2/cache/com.codahale.metrics/metrics-core/bundles/metrics-core-3.0.2.jar:/root/.ivy2/cache/org.slf4j/slf4j-api/jars/slf4j-api-1.7.7.jar:/root/.ivy2/cache/org.apache.commons/commons-lang3/jars/commons-lang3-3.3.2.jar:/root/.ivy2/cache/org.joda/joda-convert/jars/joda-convert-1.2.jar:/root/.ivy2/cache/joda-time/joda-time/jars/joda-time-2.3.jar:/root/.ivy2/cache/org.apache.cassandra/cassandra-all/jars/cassandra-all-2.0.9.jar:/root/.ivy2/cache/org.slf4j/slf4j-log4j12/jars/slf4j-log4j12-1.7.2.jar

The preceding command assumes that the source code of Spark-Cassandra-connector is downloaded in the $ home/working directory. Modify the command as needed.

How can I guess that the dependencies of these packages need to be specified? To put it bluntly, it is also very simple, that is, to execute the following command, and then check the running parameters in the Java Process.

# Run spark-Cassandra-connector Test Set SBT testsbt it: Test

When the preceding command is still running, use PS to view the Java running parameters. In this way, the required package dependency is known.

ps -ef|grep -i java

Test procedure

Because spark-shell creates SC by default, you need to stop the default SC first, and then use the new configuration to create SC that can be connected to Cassandra. The sample code is as follows:

sc.stopimport com.datastax.spark.connector._import org.apache.spark._val conf = new SparkConf()conf.set("spark.cassandra.connection.host", "127.0.0.1")val sc = new SparkContext("local[2]", "Cassandra Connector Test", conf)val table = sc.cassandraTable("test", "kv")table.count

If everything is normal, the following results are displayed:

res3: Long = 2

Summary

As we enter the practical stage, there will be more and more challenges. It is important to maintain sufficient confidence and patience.

This article is organized together with Kafka cluster in practice 1. It forms a complete processing chain from the front-end to the back-end storage.

Apache Spark Technology Practice 3 -- Installation and Use of spark Cassandra Connector