Kafka Quota Control

Reprint Please specify address http://www.cnblogs.com/dongxiao-yang/p/5217754.html

Starting in 0.9, the Kafka cluster have the ability to enforce quotas on produce and fetch requests. Quotas is basically byte-rate thresholds defined per Client-id. A Client-id logically identifies an application making a request. Hence a single Client-id can span multiple producer and consumer instances and the quota would apply for all of them as a s Ingle entity i.e. if client-id= "test-client" have a produce quota of 10mb/sec, this is gkfx across all instances with tha T same ID.

Starting with version 0.9, the Kafka cluster has added the ability to quota (quotas) control for production and consumption requests. Quota is basically the concept of a single-client-id data byte-rate threshold. Logically, a client-id represents an application that generates a request. A client-id can theoretically have multiple producer or consumer instances, and Qouta treats these instances as an entire individual. If a program with a client-id of "test-client" is quota set to 10mb/s, an instance with the same ID will share this quota.

It is possible for producers and consumers to produce/consume very high volumes of data and thus monopolize broker RESOURC ES, cause network saturation and generally DOS other clients and the brokers themselves. Having quotas protects against these issues and are all tbe more important in large multi-tenant clusters where a small set Of badly behaved clients can degrade user experience for the well behaved ones. In fact, when running Kafka as a service this even makes it possible to enforce API limits according to an agreed upon con Tract.

Production/consumer has the potential to generate very high data throughput and thus preempt the broker server's resources, causing the network to saturate and usually DOS off other client connections and the broker server itself. The addition of the quota mechanism can prevent the occurrence of the above situation, but also can avoid the cluster in multi-user scenarios due to the abnormal traffic of the individual clients to affect the use of the rest of the normal client (that is, a rat excrement broke the meaning of a pot of soup). In fact, since Kafka is already running as a background service, it is feasible to enforce restrictive conventions through API interfaces.

By default, each unique Client-id receives a fixed quota in bytes/sec as configured by the cluster (Quota.producer.default , Quota.consumer.default). This quota was defined on a per-broker basis. Each client can publish/fetch a maximum of X bytes/sec per broker before it gets throttled. We decided that defining these quotas per broker are much better than having a fixed cluster wide bandwidth per client Beca Use this would require a mechanism to share client quota usage among all the brokers. This can is harder to get right than the quota implementation itself!

By default, each individual Client-id corresponds to a fixed quta speed for the cluster configuration (the default configuration is Quota.producer.default, Quota.consumer.default). Quota is a concept that is defined to the granularity of each broker. Each client can generate a write/read traffic request of up to X bytes/sec with a single broker before reaching the speed limit. The decision to define quota to each broker granularity is more appropriate than setting the bandwidth concept of a fixed complete cluster granularity, thus eliminating a mechanism to coordinate quota between cluster brokers. This coordination mechanism may be more troublesome than the implementation of the quota mechanism itself!

How does a broker react when it detects a quota violation? In our solution, the broker does isn't return an error rather it attempts to slow down a client exceeding its quota. It computes the amount of delay needed to bring a guilty client under it's quota and delays the response for that time. This approach keeps the quota violation transparent to clients (outside of client side metrics). This also keeps them from have to implement any special Backoff and retry behavior which can get tricky. In fact, bad client behavior (retry without backoff) can exacerbate the very problem quotas is trying to solve.

What will the broker do if it finds out that quota is out of the picture? Our current approach is that the broker does not return an error message but tries to slow down the client. The broker calculates the delay time that is required to limit the client's speed to quota, and then delay so much time before response to respond. The basic implementation of this mechanism is to quota the speed limit function to the client (no configuration on the client side), but also avoids the complicated and cumbersome backoff and retry logic that the client needs to implement. In fact, abnormal client behavior (retries without fallback mechanism) may worsen the problem that quota wants to solve.

Client byte rate was measured over multiple small windows (for e.g. windows of 1 second each) in order to detect and Cor Rect quota violations quickly. Typically, has large measurement windows (for e.g.) seconds to leads large of bursts traffic Owed by long delays which are not great in terms of user experience.

The client's byte rate is a sample of multiple small windows (e.g. 30 samples per second) to accurately and quickly detect quota overrun conditions. In general, long sampling windows (such as sampling 10 times in 30 seconds) can cause traffic congestion due to large delays, which can affect the user experience.

Quota overrides (Quota reset)

It is possible to override the default quota for client-ids that need a higher (or even lower) quota. The mechanism is similar to the Per-topic log config overrides. Client-id overrides is written to ZooKeeper under/config/clients. These overrides is read by all brokers and is effective immediately. This lets us change quotas without have to do a rolling restart of the entire cluster. See here for details.

You can modify the default quota for a client, similar to the config reset for each topic. The client's settings are located under the Zookeeper Service/config/clients path. This change is read by all brokers and takes effect immediately.

This allows us to change the quota without having to roll over the entire cluster. Refer to Here for the following links.

Kafka Quota Control