Is there a way to force Logstash to open files in chronological order so that they are written to the Kafka topic in chronological order?
I am setting up a Logstash instance that will read files written to a server (2 new files per minute) and output them to Kafka. My environment captures the contents of Kafka every 30 minutes and stores them to HDFS. I have this setup and it works well when everything is up and processing files in realtime. However if Logstash or Kafka are down for any length of time, when Logstash starts sending data to Kafka again numerous files are written to the HDFS. I believe this is being caused by Logstash opening multiple files and sending the contents to Kafka not in chronological order.