Logstash 1.5.3 with Kafka 0.9's quota


#1

In Kafka 0.9 support for quota is added as mentioned here. I am having some doubts about the behavior of quota in accordance with logstash.

So in Kafka 0.9 quota can be configured for each client-id. If one of the client violates the quota then the kafka broker does not return an error rather it attempts to slow down a client exceeding its quota. It computes the amount of delay needed to bring a guilty client under it's quota and delays the response for that time.

Now the doubt I am having is let's say a single instance of logstash is pushing logs from two different services (logs are pushed via logstash-forwarder) to kafka with clientIds A & B respectively. Client A is sending data under its quota while Client B is exceeding its quota. Since client B violates the quota as configured in Kafka then Kafka will delay the response to slow down B to bring its usage under quota. This will in turn exert a back-pressure on the logstash-forwarder & slow its log forwarding to logstash.

Due to this slow down

  1. Will the logstash-forwarder, who is forwarding logs for Client A (who is not violating the quota ) be effected in anyway or
  2. The slow down of logstash will effect both Client A & Client B or
  3. This will slow down all logstash outputs not just restricted to Kafka Outputs

#2

@Joe_Lawson - Any thoughts?


(system) #3