Storm transport optionaldataexception Solution

The storm version used is 0.9.2. After running for a period of time (time is not fixed, the fastest time is dozens of minutes), a worker will report the following exception:

java.lang.RuntimeException: java.lang.RuntimeException: java.io.OptionalDataException    at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]    at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]    at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]    at backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]    at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]    at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]Caused by: java.lang.RuntimeException: java.io.OptionalDataException    at backtype.storm.serialization.SerializableSerializer.read(SerializableSerializer.java:58) ~[storm-core-0.9.2-incubating.jar:0.9.2-          incubating]    at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:732) ~[kryo-2.21.jar:na]    at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109) ~[kryo-2.21.jar:na]    at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18) ~[kryo-2.21.jar:na]    at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:629) ~[kryo-2.21.jar:na]    at backtype.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:38) ~[storm-core-0.9.2-incubating.jar:0.9. 2-incubating]    at backtype.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:53) ~[storm-core-0.9.2-incubating.jar:0.9.2-     incubating]    at backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:396) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]    at backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]    at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]    ... 6 common frames omittedCaused by: java.io.OptionalDataException: null    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1370) ~[na:1.7.0_67]    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) ~[na:1.7.0_67]    at java.util.LinkedList.readObject(LinkedList.java:1136) ~[na:1.7.0_67]    at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source) ~[na:na]    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_67]    at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_67]    at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017) ~[na:1.7.0_67]    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893) ~[na:1.7.0_67]    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) ~[na:1.7.0_67]    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) ~[na:1.7.0_67]    at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) ~[na:1.7.0_67]    at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) ~[na:1.7.0_67]    at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) ~[na:1.7.0_67]    at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) ~[na:1.7.0_67]    at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) ~[na:1.7.0_67]    at backtype.storm.serialization.SerializableSerializer.read(SerializableSerializer.java:56) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]    ... 15 common frames omitted

According to the exception analysis, the executor thread encountered a problem in deserializing the information in the disruptor queue. When the data types written by serialization are inconsistent with those read, optionaldataexception may be reported. However, during code running, the transmitted object does not change. Google does not get much. Finally, it checks the code to find that the transmitted object class does not define serialversionuid, after completing the task, submit the task again. No exception is reported. Serialversionuid is used for version verification when the entity class code is changed. During the storm topology running, the entity class itself does not change, and the error is a bit strange.

During troubleshooting, we found that An NPE exception was reported at the same location of "disruptorqueue. Java: 128". The official troubleshooting explained this.

    This is caused by having multiple threads issue methods on the OutputCollector. All emits, acks, and fails must happen on the same thread. One subtle way this can happen is if you make a IBasicBolt that emits on a separate thread. IBasicBolt’s automatically ack after execute is called, so this would cause multiple threads to use the OutputCollector leading to this exception. When using a basic bolt, all emits must happen in the same thread that runs execute.

This means that when calling emit, ack, and fail using outputcollector, it must be performed in the same thread. However, some users may encounter NPE problems even if they call the agent in the same thread, after talking with this student, I learned that the read thread timeout was caused by the suspension of the JVM memory Recycle of worker. After the JVM memory parameter is optimized, the problem is solved.

During troubleshooting, I also got two good articles about Storm's transport mechanism and related tuning. I will share them with you.

Http://www.michael-noll.com/blog/2013/06/21/understanding-storm-internal-message-buffers/

Https://gist.github.com/mrflip/5958028

This article is from the "Lotso blog" blog, please be sure to keep this source http://lotso.blog.51cto.com/3681673/1562425

Storm transport optionaldataexception Solution