How can logstash support daily rolling kafka topics?

Elastic Stack Logstash
1

This is a continuous conversation of https://github.com/elastic/logstash/issues/10005
First I answer two questions repied by yaauie.

  1. What we think is to use topics like ["xxx_20180919", "xxx_20180920", "xxx_20180921"] and keep updating every day which can avoid the windows problem at 0:00 am.
  2. We have used kafka config log.retention.hours but it only clean the data of topics instead of removing the topics themselves, which still require us to delete the topic manually.

Mentioned before, if we use logstash topics_pattern like xxx_.*, the logstash will stuck if we delete a old topic and requires restarting. Does any one has an idea for our cases? we just want the logstash to consumer the kafka topics related to yesterday, today and tomorrow and updating every day.

Thank you very much!

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.