Druid.io Series 6: Issues summary

We also encountered a lot of problems in the production environment using Druid, by reading the official website documents, source code and Community questions to solve or partially solve a lot of problems, now will encounter problems, solutions and tuning experience summarized as follows:

Issue one: Hadoop batch ingestion failed with a log error of "No buckets?..."

Solution: This problem has been bothering us for about a week, and for most of the people who have just contacted Druid, they will encounter the time zone problem.

In fact, the problem is simple, mainly because the cluster working time zone is inconsistent with the import data time zone. Because Druid is a time series database, it is very sensitive to time. The Druid base uses absolute milliseconds to store the time, and if you do not specify a time zone, the default output is 0 time zone, which is yyyy-mm-ddthh:mm:ss in ISO8601. Sssz. We use the East eight zone in our production environment, which is Asia/hong Kong time zone, so we need to adjust all UTC time of the cluster to utc+08:00, and the timestamp column format of the imported data must be: YYYY-MM-DDTHH:MM:SS. sss+08:00

Issue two: Druid and Hadoop have a version conflict on Jackson, log error message:

Solution: The current version of Druid is compiled with the Hadoop-2.3.0 version, either to replace the Hadoop version as 2.3.0 (which is what our production environment is doing), or to follow the approach in this article:
Https://github.com/druid-io/druid/blob/master/docs/content/operations/other-hadoop.md

Problem three: Segment proposal size between 300-700MB, we are currently one hours aggregation once, the original data per hour about 50G, the generated segments data around 10G, segments too big impact query performance, how to deal with it?

Solution:
There are two ways: reducing segmentgranularity
Aggregated by hour granularity, consider reducing to minute levels, such as 20 minutes. Adding Partitionsspec to the Tunningconfig, the official explanation is as follows:

Issue four: The index task cannot be released for a long time after handoff failure, and waiting for handoff, error log information is as follows:
2016-05-26 22:05:25,805 ERROR i.d.s.r.p.coordinatorbasedsegmenthandoffnotifier [Coordinator_handoff_scheduled_0] Exception while checking handoff for Datasource[hm_flowinfo_analysis] segment[segmentdescriptor{interval= 2016-05-26t21:00:00.000+0800/2016-05-26t22:00:00.000+0800, version= ' 2016-05-26t21:00:00.000+0800 ', Partitionnumber=0}], would try again after [60000]secs

Solution:
This problem is very common in Druid, and there are a lot of similar questions in the community, the problem is mainly focused on historical node has no memory to load segment. The methods for troubleshooting this type of problem are summarized below:

The

First refers to the developer's answer to the question  
I guess due to the network storage the segment are not being pushed to deep storage at all.  ;
Do you see a segment metadata entry in DB for above segment?  
If No and then check for any exception in the task Logs or Overlord logs related to segment publishing. 
If The metadata entry are present in the DB, make sure you h Ave enough free space available on the historical nodes to load the segments and there is no exceptions in Coordinator/hi storical while loading the segment.

And a simple explanation of the handoff phase workflow  
The coordinator is the process that detects the new segment built by the indexing task an D signals the historical nodes to load the segment. The indexing task would only complete once it gets notification a historical have picked up the segment so it knows it Can stop serving it. The coordinator logs should help determine whether or isn't the coordinator noticed the new segment, if it tried to signal a Historical to load it-failed, if there were rules preventing it from loading, etc. Historical logs would show you if a historical received the load order but failed for some reason (e.g. out of memory).

Therefore, to synthesize the above two aspects, the root cause of this problem is summarized as follows
Indexing task if it has not been released for a long time because it did not receive the return information after the historical nodes loaded successfully. The Coordinatorbasedsegmenthandoffnotifier class is primarily responsible for registering the segments waiting for handoff and checking the status of segments to be handoff.
Register waiting for handoff segments mainly use internal Concurrentmap to save the relevant information of segments;
Check the status of the segments to be handoff mainly by:
(1) First, through the Apache curator to zookeeper on the query coordinator instance;
(2) Coordinatorclient internally encapsulates a httpclient that sends a httpget request to a surviving coordinator instance to obtain the load information list for all segments in the current cluster;
(3) compared with the internal cache Concurrentmap and list, the successful handoff delete the information, the failure will print out the above log, while Coordinator will go to the meta repository every minute to synchronize the published segments, So handoff failure will try again every minute.

The final solution is summarized below

Check that the META repository has registered this segment information, and if not, check whether the task logs (middle Manager) or Overlord log for this segment release has an exception thrown;

If the segment is already registered in the Meta repository, check if coordinator has detected that this segments has been generated, and if coordinator attempts to notify historical nodes to load but fails, check for rules Prevent the loading of this segments, etc., and check the historical nodes in coordinator to notify load of this segment whether an exception occurred during the loading of the segment.

Issue five: In the 0.9.1.1 Version of the newly introduced Kafka indexing Service, if the Kafka retention time is set too short, while the Druid consumption speed is less than the retention rate, then the offset expiration occurs, That is, the data that has not yet been consumed has been Kafka deleted, the Peon log will always appear offsetoutofrangeexception, and all subsequent tasks fail.

Solution:
This problem occurs more extreme, because we use the scene data is large, while the cluster bandwidth resources are insufficient, all Kafka retention time is set to 2 hours, However, Kafka provides auto.offset.reset this policy has to deal with the issue of offset expiration, but configured in the spec file {"Auto.offset.reset": "Latest"}, Indicates that if offset expires automatically rewind to the latest offset, the configuration entry in the Overlord log is in effect by tracking the Overlord log and Peon log discovery, but the configuration entry in the Peon log is set to "none", "none" Indicates that offset expires without any processing, only throws an exception, that is, offsetoutofrangeexception. The peon does not take effect because it was written in the code, mainly to satisfy the exactly-once semantics, but the developer did not take into account how to solve the problem of the death cycle while writing dead.

For this issue, the developer gives the following answers:
Because of your 2 hour data retention, I guess you ' re hitting a case where the Druid Kafka indexing tasks is trying to re Ad offsets that has already been deleted. This causes problems with the Exactly-once transaction handling scheme, which requires-all offsets is read in order, Without skipping any. The Github issue https://github.com/druid-io/druid/issues/3195 is about making this better–basically you would has an O Ption to reset the Kafka indexing to latest (this would involve resetting the ingestion metadata Druid stores for the data SOURCE).
In the meantime, maybe it's possible to make this happen less often by either extending your Kafka retention, or by settin G your Druid taskduration lower than the default of 1 hour.

Therefore, there is no complete solution for this problem, but the following scenarios can be partially or temporarily resolved:

Try to increase Kafka retention time, we set 2 hours indeed too extreme

Reduce taskduration;

In the case where neither of the above solves the problem completely, the Druid_datasource table in the Meta repository can be emptied, which records the partition and offset information for all consumed Kafka topic. Restart the Middlemanager node at the same time.  
However, in the 0.9.2-milestone version, this feature should be further improved, as the following is introduced from GitHub:  
The Kafka Indexing Service can get into a stuck State where it's trying to read the Kafka offset following the last one recorded in the DataSource metadata table but can ' t read it because Kafka ' s retention period have elapsed and that message is no longer available. We probably don ' t want to automatically jump to a valid Kafka offset since we would has missed events and would no longer has ingested each message exactly once, so currently we start throwing exceptions and the user needs to acknowledge what Happened by removing the last committed offset from the DataSource table. 
It would is nice to include an API tha T would help them by either removing the DataSource table entry or setting it to a valid Kafka offset, but would still requ IRE a manual action/acknowledgment by the user.