Really big retrieval lag for Logstash Kafka inputs producing data irregularly

[mwysokin (Michal Wysokinski)]

Hi!

I'm using logstash 2.4 with kafka input 5.1.6. In my config I created a field called input_lag in order to monitor how much time it takes logstash to process logs:

ruby {
      code => "event['lag_seconds'] = (Time.now.to_f - event['@timestamp'].to_f)"
}

I listen to several kafka topics from a single logstash instance and for the topics that produce logs regularly everything is OK and the lag is small (several seconds). However, for the topics that produce small amount of logs irregularly I get really big lags. Sometimes it's tens of thousands of seconds. My configuration for Kafka input is following:

input {
    kafka {
        bootstrap_servers => "kafka-broker1:6667,kafka-broker2:6667,kafka-broker3:6667"
        add_field => { "environment" => "my_env" }
        topics_pattern => "a|lot|of|topics|like|60|of|them"
        decorate_events => true
        group_id => "mygroup1"
        codec => json
        consumer_threads => 10
        auto_offset_reset => "latest"
        auto_commit_interval_ms => "5000"
   }
} 

The logstash instance seems healthy, as logs from other topics are being retrieved regularly. I've checked and if I connect to Kafka using its console consumer the delayed logs are there. I've also thought that it might be a problem with too many topics being served by a single logstash instance and extracted those small topics to separate logstash instances but the lag is exactly the same, so it's not the issue.

Any ideas? I suspect that logstash might be using some exponential delay for log retrieval, but have no idea how to confirm and fix that.