Kafka high-level consumer multithreaded Access exception

In the use of Kafka high-level consumer, the use of multi-threaded consumption data times wrong, simple analysis of the reason for download , Consumeriterator will not be blocked when the message and set the internal state to failed, which throws an exception when other threads access it.

Java code 650) this.width=650; "class=" star "src=" Http://woodding2008.iteye.com/images/icon_star.png "alt=" Favorites Code "style= "border:0px;"/>

2. Def hasnext (): Boolean = {

4. if         (state = = FAILED) //In failed state, additional thread access will be directly abnormal

6. Throw New IllegalStateException ("Iteratoris in failed state")

8. State Match {

10. Case Done = false

12. Case Ready = true

14. Case _ = Maybecomputenext ()

16. }

18. }

24. Def maybecomputenext (): Boolean = {

26. state = FAILED //Reset Status

28. Nextitem = Some (Makenext ())

30. if (state = = done) {

32. false

34. } Else {

36. State = Ready

38. true

40. }

42. }

44. Download

48. protected def makenext (): Messageandmetadata[k, V] = {

50. var currentdatachunk:fetcheddatachunk = null

52. //If we don ' t has an iterator, get one

54. var localcurrent = Current.get ()

56. if (localcurrent = = Null | |!localcurrent.hasnext) {

58. if (Consumertimeoutms < 0)

60. Currentdatachunk = Channel.take //channel is blockingqueue here will block

64. Else {

66. Currentdatachunk = Channel.poll (Consumertimeoutms, Timeunit.milliseconds)

68. if (Currentdatachunk = = null) {

70. //Reset State to make the iterator re-iterable

72. ResetState ()

74. Throw New Consumertimeoutexception

76. }

78. }

80. //Omit part of the code

82. }

Kafka high-level consumer multithreaded Access exception