Ingesting latest logs from kafka with logstash

Elastic Stack Logstash
1

Hi,

I searched previous opened topics related that question but none of them have any reply. I’m ingesting logs from apache kafka with logstash, everything is fine but I noticed that whenever I started logstash instance it doesn't starts to ingest latest logs. The logs’ timestamps belong 13-14 hours ago.

I was thinking that after I set auto_offset_reset => “latest” parameter should have solved my issue but I guess It didn’t work.

This is input part of my logstash config:

input {
  kafka {
    bootstrap_servers => ["bootsrap1:39092,bootsrap2:39092,bootsrap3:39092,bootsrap4:39092,bootsrap5:39092"]
    topics => "th-cef"
    auto_offset_reset => "latest"
    client_id => "test10"
    type => "logs9"
    group_id => "sectechlogstash"
    consumer_threads => 19
  }
}

Any tips or help would be appreciated
Thanks

2

It's funny, I didn't change anything at all. After I created this topic, I just started again logstash instances and noticed that logstash ingest latest logs from kafka.

Thanks anyway to Elastic Community!

3

auto_offset_reset determines what it does if there is no offset for the consumer group, or if the offset is beyond the start of the queue in kafka. It is possible that the 14 hour old data was very close to expiring, and when you stopped and restarted logstash the offset for sectechlogstash moved beyond the expiry window.

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.