Development on Storm cluster, task calculation output to MySQL database, integrated JDBC (10)

Storm integrates JDBC and saves the results to MySQL.

First, the table is built in MySQL, and the field of the table is consistent with the schema of the output tuple:

Create Table result (    varchar(),    int);

Write a connection provider to get the MySQL database connection:

Need to introduce jar :/usr/local/apps/apache-storm-1.0.3/external/storm-jdbc 's Storm-jdbc-1.0.3.jar

 PackageMystorm.wordcount;Importjava.sql.Connection;ImportJava.sql.DriverManager;Importjava.sql.SQLException;ImportOrg.apache.storm.jdbc.common.ConnectionProvider;/*** Storm integrated JDBC Connection provider *@authorAdministrator **///provide a corresponding data connection for the Jdbcbolt component Public classMyconnectionproviderImplementsConnectionProvider {Private StaticString Driver = "Com.mysql.jdbc.driver"; Private StaticString url = "Jdbc:mysql://192.168.2.1:3306/test"; Private StaticString user = "root"; Private StaticString password = "123456"; Static{                Try{class.forname (driver); } Catch(ClassNotFoundException e) {//TODO auto-generated Catch block            NewExceptionininitializererror (e);//e.printstacktrace ();}} @Override Public voidCleanup () {//TODO auto-generated Method Stub} @Override PublicConnection getconnection () {//TODO auto-generated Method Stub        Try {            returndrivermanager.getconnection (Url,user,password); } Catch(SQLException e) {//TODO auto-generated Catch blockE.printstacktrace (); }        return NULL; } @Override Public voidprepare () {//TODO auto-generated Method Stub    }}

View Code

Then add a storm-supplied bolt component to the topology task we wrote earlier to write data to MySQL

    //Create a new Jdbcbolt component that inserts the data sent from the previous bolt component into the MySQL database//Storm integrated jar package path for various frameworks:/usr/local/apps/apache-storm-1.0.3/external//integrated JDBC Path:/USR/LOCAL/APPS/APACHE-STORM-1.0.3/EXTERNAL/STORM-JDBC    /*** * * * JAR Package introduced: * 1.external/sql/storm-sql-core/*.jar * 2.EXTERNAL/STORM-JDBC * 3.mysql Drive * 4.commons-lang3-3.1.jar * *@return     */    Private StaticIrichbolt Createjdbcbolt () {//Create a ConnectionProviderMyconnectionprovider Conectionprovider =NewMyconnectionprovider (); //Create a mapper, fill in the table nameJdbcmapper mapper =NewSimplejdbcmapper ("Result", Conectionprovider); //creating a bolt component from Mapper                return NewJdbcinsertbolt (conectionprovider,mapper). Withtablename ("Result"). Withquerytimeoutsecs (30); }

Then add this component to the third bolt component on the last side of the topology task:

/*** * topology entry for Word count, main program * *@authorAdministrator **/ Public classWordcounttopology { Public Static voidMain (string[] args)throwsalreadyaliveexception, Invalidtopologyexception, authorizationexception {topologybuilder builder=NewTopologybuilder (); //set the spout component of a taskBuilder.setspout ("Wordcount_spout",Newwordcountspout ()); //set the first bolt component of a taskBuilder.setbolt ("Wordcount_bolt",NewWordcountsplitbolt ())//(random assignment policy receives spout tasks). shufflegrouping ("Wordcount_spout"); //set up a second bolt componentBuilder.setbolt ("Wordcount_countbolt",NewWordcountboltcount ())//(accept data from the first bolt component, grouped by field). fieldsgrouping ("Wordcount_bolt",NewFields ("word")); //set up a third bolt component (storm-supplied), and save the record to the MySQL database. Builder.setbolt ("Wordcount_jdbcbolt", Createjdbcbolt ()). Shufflegrouping ("Wordcount_countbolt"); //Create a topology taskStormtopology WC =builder.createtopology (); //Configuration parameter InformationConfig conf =NewConfig (); //1. Local mode submissionLocalcluster Localcluster =NewLocalcluster (); Localcluster.submittopology ("Mywordcount", conf, WC); /**         * * Local mode operation results * * Spout data collected is: Beijing are the capital of China output Result: {beijing=1} output result: {Is=1, beijing=1} output result: {the=1, Is=1, beijing=1} output: {the=1, Capit Al=1, Is=1, beijing=1} output result: {the=1, capital=1, Of=1, Is=1, beijing=1} output: {the=1, capital=1, Chin A=1, Of=1, Is=1, beijing=1} spout The data collected is: I love Beijing output results: {the=1, capital=1, China=1, Of=1, I=1,  Results of Is=1, beijing=1} output: {the=1, love=1, capital=1, China=1, Of=1, I=1, Is=1, beijing=1} output: {the=1, Love=1, Capital=1, China=1, Of=1, I=1, Is=1, beijing=2} The data collected is: I love the result of the output: {spout, love =1, Capital=1, China=1, Of=1, i=2, Is=1, beijing=2} results of output: {the=1, love=2, capital=1, China=1, Of=1, i=2, Is=1,          Results of beijing=2} output: {the=1, love=2, capital=1, china=2, Of=1, i=2, Is=1, beijing=2} *           **/                //2 cluster mode submission//stormsubmitter.submittopology (Args[0], conf, WC);            }

View Code

Hit into jar package, put into the storm cluster run, will treasure the following error:

 Exception in thread "main" java.lang.noclassdeffounderror:org/apache/commons/lang/ validate at Org.apache.storm.jdbc.mapper.SimpleJdbcMapper.  <init> (Simplejdbcmapper.java:39 95 ) at Mystorm.wordcount.WordCountTopology.main (Wordcounttopology.java:  42) caused by:java.lang.ClassNotFoundException:org.apache.commons.lang.Validate at Java.net.URLClassLoader.findClass (URLClassLoader.java:  381 424 331 357 3 more 

There are also some jars that are not integrated and need to be integrated into the storm jar, placed in the Storm cluster home Lib, and replicated to other machines in the cluster:

     * 1.external/sql/storm-sql-core/*. Jar     * 2.EXTERNAL/STORM-JDBC     * 3.mysql Drive     * 4. Commons-lang3-3.1.jar

Development on Storm cluster, task calculation output to MySQL database, integrated JDBC (10)