Jstorm when using Kafka as spout, there will be multi-threaded error problem in high concurrency situation
These two classes need to be appropriately modified to avoid the above problems:
Storm.kafka.PartitionManager
Storm.kafka.ExponentialBackoffMsgRetryManager
Modification of 1.storm.kafka.partitionmanager
//The variablePrivate Sortedmap<long, long> _pending =NewTreeMap ();//SwitchPrivate Sortedmap<long, long> _pending = Collections.synchronizedsortedmap (New Treemap<long, long>());/**----------------------------------------------------------------------------------------------------**///The methodPublicLongLastcompletedoffset () {return this._pending.isempty ()? This._emittedtooffset.longvalue ():((Long) this._ Pending.firstkey ()). Longvalue ();} // change to: public long Lastcompletedoffset () { Synchronized (_pending) {if (_pending.isempty ()) {return _emittedtooffset;} else {return _pending.firstkey (); } }}
Modification of 2.storm.kafka.exponentialbackoffmsgretrymanager
//will bePrivateQueue<exponentialbackoffmsgretrymanager.messageretryrecord> waiting =NewPriorityqueue (11,Newexponentialbackoffmsgretrymanager.retrytimecomparator ()); PrivateMap<long, exponentialbackoffmsgretrymanager.messageretryrecord> records =NewConcurrenthashmap ();//instead:PrivateQueue<messageretryrecord> waiting =NewPriorityblockingqueue<messageretryrecord> (11,Newretrytimecomparator ()); PrivateMap<long,messageretryrecord> records =NewConcurrenthashmap<long,messageretryrecord> ();
Jstorm multithreading issues when using Kafka as a spout