How to Setup replicated LevelDB persistence in Apache ActiveMQ 5.9--reprint

Original address: https://simplesassim.wordpress.com/2013/11/03/how-to-setup-replicated-leveldb-persistence-in-apache-activemq-5-9/

1) Install Apache ZooKeeper.

2) Change the persistence adapter in the <Apache ActiveMQ Install Dir>/etc/activemq.xml file:

123	<persistenceAdapter>  <replicatedLevelDB directory="activemq-data" replicas="..." bind="tcp://0.0.0.0:61619" zkAddress="host:2181" zkPath="/activemq/leveldb-stores" /></persistenceAdapter>

3) Start Apache ZooKeeper.

4) Start Apache ActiveMQ.

You can find more details here.

Another article about-to-set up:

Setup ActiveMQ, Zookeeper, and replicated LevelDB running in JDK 8 and CentOS

Original address: http://www.elasticcloudapps.com/page0/files/c1f6bea32e025aa68542a95f9d664ea9-12.html

This guide describes the step-by-step guide to setup ActiveMQ to use replicated LevelDB persistence with Zookeeper. CentOS environment is used for servers. Zookeeper is used to replicate the LevelDB to support Master/slave Activemqs. Three VMware instances is used with 2 Core processes, 2G RAM and 20G disk space. For simplicity, stop the Iptable Service (firewall) in CentOS. If Iptable is required then you need to open set of ports. List of ports numbers is included in Pre-setup.

Overview
ActiveMQ cluster environment includes following
1Three VM instances with CentOS OS and JDK 8
2Three ActiveMQ instances.
3Three Zookeeper instances.




Pre-install

Following ports is required to open in Iptables host firewall.

• Zookeeper

• • 2181–the port that clients'll use to connect to the ZK ensemble
• • 2888–port used by ZK for quorum election
• • 3888–port used by ZK for leader election

• ActiveMQ

• • 61616–default Openwire Port
• • 8161–jetty Port for Web console
• • 61619–LEVELDB peer-to-peer replication port for ActiveMQ slaves.

• To check Iptables status.

• • Service Iptables Status

• To stop iptables service

• • Service Iptables Save
• • Service Iptables Stop
• • Chkconfig iptables off

• To start again

• • Service Iptables Start
• • Chkconfig iptables on

• Setup proper hostname, edit following files

• • Update HOSTNAME Value In/etc/sysconfig/network:e.g:hostname=msgq1.dev.int
• • ADD host name with IP address of the machine in/etc/hosts:e.g:192.168.163.160 msgq1.dev.int MSGQA1
• • Restart the instance and repeat same process each instances.

Installation

Java JDK

• Download JDK 8 from http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
• Extract jdk-8*.tar.gz to a folder of your choice (I used ~/dev/server). This would create folder ~/dev/server/jdk1.8.0_20.
• Set up Java_home directory.
• To setup for all users edit/etc/profile and add following line export java_home=home_dir/dev/server/jdk1.8.0_20.

Zookeeper

• Download Zookeeper from site http://zookeeper.apache.org/
• Extract the file into your folder of your choice. I have used ~/dev/server/
• Create Soft link zookeeper for extracted directory.
• For reliable ZooKeeper service, the ZK should is deployed in a cluster mode knows as ensemble. As long as a majority of the ensemble is up, the service would be available.
• Goto conf directory and create zookeeper configuration directory.
• CD <zookeeper_install_dir>/conf
• Copy Zoo_sample.cfg to Zoo.cfg
• Make sure the file has following lines

• • Ticktime=2000initlimit=5
    synclimit=2
    Datadir=~/dev/server/data/zk
    clientport=2181

• Add following lines to zoo.cfg at the end.

• • server.1=zk1_ipaddress:2888:3888
    Server.2=zk2_ipaddress : 2888:3888
    server.3=zk3_ipaddress:2888:3888
• • Zk1, ZK2 & Zk3 are IP addresses for the ZK servers. 
• • port 2181 is used T o Communicate with client
• • Port 2888 are used by peer ZK servers to COM Municate among themselves (Quorum port)  
• • Port 3888 is used for L Eader election (Leader election port).

• The last three lines of the Server.id=host:port:port format specifies that there is three nodes in the ensemble. In the ensemble, each ZooKeeper node must has a unique ID between 1 and 255. This ID was defined by creating a file named myID in the DataDir directory of each node.  For example, the node with the ID 1 (server.1=zk1:2888:3888) would have a myID file At/home/sthuraisamy/dev/server/data/zk With the text 1 inside it.
• Create myID file in Data directory for ZK1 (Server.1) and for other ZK servers as 2 & 3.

• • Echo 1 > myID

ActiveMQ
Download ACTIVEMQ distribution from http://apache.mirror.nexicom.net/activemq/5.10.0/apache-activemq-5.10.0-bin.tar.gz

• Extract the file into your folder of your choice. I have used ~/dev/server/
• Create Soft link activemq for extracted directory.
• Do following

• • CD <active_mq_dir>/bin
• • chmod 755 ACTIVEMQ
• • <ACTIVE_MQ_DIR>/BIN/ACTIVEMQ start

• To confirm the ACTIVEMQ was listening on port 61616 or check the log file and confirm port listening messages are populated .

• • Netstat-an|grep 61616

• In activemq config file, following bean classes define the settings

• • Propertyplaceholderconfigurer
• • Credentials
• • Broker section

• • • Constantpendingmessagelimitstrategy:limit the number of messages to is keep in memory for slow consumers.

• • Other settings to handle slower consumers, refer http://activemq.apache.org/slow-consumer-handling.html
• • Persistence adapter to define the keep of storage to messages.
• • For better performance

• • • Use Nio:refer Http://activemq.apache.org/configuring-transports.html#ConfiguringTransports-TheNIOTransport

• 
  
  

• • Replicated LevelDB Store using Zookeeper http://activemq.apache.org/replicated-leveldb-store.html
• • The settings need to being done in ActiveMQ after zookeeper is setup. Add following lines into Conf/activemq.xml

• • • Hostname should is assigned with separate IP address for each instance.

My approach is to get the software setup on a single VM instance in VM Ware Fusion, and create both more clones to having th  REE servers.  I have named the instances as MESSAGEQ1, MESSAGEQ2, and MESSAGEQ3. After starting instances confirm the myID file and IP address in the ZOO.CFG is setup properly with new instance ' s IP add Ress.

After configured everything

• Start the Zookeeper instances in all three nodes: <zookeeper_dir>/bin/zk_server.sh start
• Start the ActiveMQ instances in all three nodes: <ACTIVEMQ_DIR>/BIN/ACTIVEMQ start
• In my setup, I start the first node I didn ' t find any issue. After I has started the second node, I found exception in the log file.

No Ioexceptionhandler registered, ignoring IO exception | Org.apache.activemq.broker.BrokerService | LevelDB Ioexcepti
On handler.
Java.io.IOException:com.google.common.base.Objects.firstNonNull (Ljava/lang/object; Ljava/lang/objectljava/lang/object;
At Org.apache.activemq.util.IOExceptionSupport.create (ioexceptionsupport.java:39) [Activemq-client-5.10.0.jar : 5.10.0]
At Org.apache.activemq.leveldb.LevelDBClient.might_fail (leveldbclient.scala:552) [ activemq-leveldb-store-5.10.0.jar:5.10.0]
At Org.apache.activemq.leveldb.LevelDBClient.replay_init (leveldbclient.scala:657) [ activemq-leveldb-store-5.10.0.jar:5.10.0]
At Org.apache.activemq.leveldb.LevelDBClient.start (leveldbclient.scala:558) [Activemq-leveldb-store-5.10.0.jar : 5.10.0]
At Org.apache.activemq.leveldb.DBManager.start (dbmanager.scala:648) [activemq-leveldb-store-5.10.0.jar:5.10.0]
At Org.apache.activemq.leveldb.LevelDBStore.doStart (leveldbstore.scala:235) [Activemq-leveldb-store-5.10.0.jar : 5.10.0]
At Org.apache.activemq.leveldb.replicated.MasterLevelDBStore.doStart (masterleveldbstore.scala:110) [ activemq-leveldb-store-5.10.0.jar:5.10.0]
At Org.apache.activemq.util.ServiceSupport.start (servicesupport.java:55) [activemq-client-5.10.0.jar:5.10.0]
At org.apache.activemq.leveldb.replicated.electingleveldbstore$ $anonfun $START_MASTER$1.APPLY$MCV$SP ( electingleveldbstore.scala:226) [Activemq-lev
eldb-store-5.10.0.jar:5.10.0]
At org.fusesource.hawtdispatch.package$ $anon $4.run (hawtdispatch.scala:330) [Hawtdispatch-scala-2.11-1.21.jar : 1.21]
At Java.util.concurrent.ThreadPoolExecutor.runWorker (threadpoolexecutor.java:1142) [: 1.8.0_20]
At Java.util.concurrent.threadpoolexecutor$worker.run (threadpoolexecutor.java:617) [: 1.8.0_20]

After going through tickets in ActiveMQ found following ticket have been reported Https://issues.apache.org/jira/browse/AM Q-5225. Workaround described in the ticket would solve the issue. The work around for this issue,

1. Remove Pax-url-aether-1.5.2.jar from Lib directory
2. Comment out the log in query section

To confirm ActiveMQ listening for request
? In master, check with Netstat-an | grep 61616 and confirm the port is in listen mode.
? In slaves, you can run Netstat-an | grep 6161 and output should show you slave binding port 61619

Post-Install
? For Zookeeper, set the Java heap size. This was very important to avoid swapping, which would seriously degrade ZooKeeper performance. To determine the correct value, use load tests, and make sure you is well below the usage limit that would cause you to s Wap. Be conservative-use a maximum heap size of 3GB for a 4GB machine.
? Increase the open file number to support 51200. E.g:limit-n 51200.
? Review Linux Network setting PARAMETERS:HTTP://WWW.NATEWARE.COM/LINUX-NETWORK-TUNING-FOR-2013.HTML#.VA8PN2TCMXO
? Review ActiveMQ Transports Configuration settings:http://activemq.apache.org/configuring-transports.html
? Review ActiveMQ Persistence Configuration settings:http://activemq.apache.org/persistence.html
? Review Zookeeper Configuration Settings:http://zookeeper.apache.org/doc/trunk/zookeeperadmin.html#sc_configuration

Reference
? ticktime:the length of a single tick, which are the basic time unit used by ZooKeeper, as measured in milliseconds. It is used to regulate heartbeats, and timeouts. For example, the minimum session timeout would be ticks.
? Initlimit:amount of time, in ticks, to allow followers to connect and sync to a leader. Increased this value as needed, if the amount of data managed by ZooKeeper is large.
? Synclimit:amount of time, in ticks, to-allow followers-to-sync with ZooKeeper. If followers fall too far behind a leader, they'll be dropped.
? clientport:the port to listen for client connections; That's, the port, clients attempt to connect to.
? datadir:the location where ZooKeeper would store the In-memory database snapshots and, unless specified otherwise, the TR Ansaction Log of updates to the database.


ActiveMQ configuration file from msgq1
Xmlns:xsi= "Http://www.w3.org/2001/XMLSchema-instance"
xsi:schemalocation= "Http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/ Spring-beans.xsd
Http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd ">


File:${activemq.conf}/credentials.properties


Scope= "Singleton" init-method= "Start" destroy-method= "Stop" >
datadirectory= "${activemq.data}" >



<replicatedleveldb
Zkaddress= "192.168.163.160:2181,192.168.163.161:2181,192.168.163.162:2181"
directory= "~/dev/server/activemq/data/leveldb"
Hostname= "192.168.163.160"/>

class= "Org.apache.activemq.hooks.SpringContextHook"/>


ZK Configuration File
ticktime=2000
Initlimit=5
synclimit=2
Datadir=/home/sthuraisamy/dev/server/data/zk
clientport=2181

server.1=192.168.163.160:2888:3888
server.2=192.168.163.161:2888:3888
server.3=192.168.163.162:2888:3888

In ActiveMQ 5.10 Web Console You can view and delete the pending messages in a Queue

How to Setup replicated LevelDB persistence in Apache ActiveMQ 5.9--reprint