we have a running ELK stack, actually:
16 Mailappliances -> syslogserver -> to file -> filebeat -> logstash -> redis -> some magic script -> ES
Syslogserver + Filebeat is one machine.
Logstash, Redis and the magic scritp is another machine.
ES is also another machine(ClusteR).
We use this chain to process logs of our mail appliances, the logformat is very complicated and multilined.
The "magic script" relies on getting the logs in chronological order for each mailappliance.
As far as good, as long as no backlog happens between filebeat and logstash.
If logstash is to slow to process all incoming logs(or is down for some reason) filebeat creates a backlog and as soon as logstash is back up filebeat starts to send the backlog.
But it seems like the backlog is not something like "first in - first out".
It seems like Filebeat just got a bunch of files and starts to send the content of every file.
Let me make this a bit more clear:
Each mail appliance has its own logfile for the current day - lets say "2017-03-23_mail1.log", filebeat is sending the contents of the file to logstash.
Now suddenly logstash cant handle the load and filebeat starts to slow down.
now filebeat didnt finish sending the contents of the file and the syslogserver starts to create a new file for the next day, so there is a new file "2017-03-24_mail1.log" and filebeat also starts to send the content of this file.
Now, further down the chain our "magic script" can see different timestamps coming: one log got the timestamp "2017-03-24 00:00:10" and the next log got the timestamp "2017-03-23 22:00:22".
This is a big problem and we need to make sure that this script gets its logs in chronological order.
So, long story short:
Is there a way to tell filebeat to start sending the contents of the new file after finishing sending the contents of the old file?
for example, if the old file didnt change for 1 Minute -> start sending the new file
Or are we missing something? is there another solution?