In Kafka 0.9 support for quota is added as mentioned here. I am having some doubts about the behavior of quota in accordance with logstash.
So in Kafka 0.9 quota can be configured for each client-id. If one of the client violates the quota then the kafka broker does not return an error rather it attempts to slow down a client exceeding its quota. It computes the amount of delay needed to bring a guilty client under it's quota and delays the response for that time.
Now the doubt I am having is let's say a single instance of logstash is pushing logs from two different services (logs are pushed via logstash-forwarder) to kafka with clientIds
Client A is sending data under its quota while
Client B is exceeding its quota. Since client B violates the quota as configured in Kafka then Kafka will delay the response to slow down B to bring its usage under quota. This will in turn exert a back-pressure on the logstash-forwarder & slow its log forwarding to logstash.
Due to this slow down
- Will the logstash-forwarder, who is forwarding logs for
Client A(who is not violating the quota ) be effected in anyway or
- The slow down of logstash will effect both
- This will slow down all logstash outputs not just restricted to Kafka Outputs