I am running filebeat on a server where my script is offloading messages from a queue as a individual files for filebeat to consume.
The setup works fine, but every now and then i see the no. of files doesn't match the no. of records on kibana. Where are these missing files going? Is filebeat dropping them?
Here's some more detail.
I have MQ queues which have messages. I wrote a script to offload these messages, each message as a single file. I am running filebeat to pick these files up. In my testing i am seeing, if there are 300 messages(hence 300 files), in kibana it only shows as 299 records sometimes and 294 sometimes. It is not getting all the 300 files. ANother thing i noticed is that if i make my script more messages, say a 1000(hence 1000 files) , the files appear to be broken or only show partial messages in kibana.
Is there a setting i am missing to ensure all files are picked up? Is the issue in filebeat or elasticsearch? I did a bit of troubleshooting where it was showing 299 records instead of 300, and went through my filebeat.log to check the names of all the files that were picked up. It missed one file.
What am i doing wrong? The files are not big. About 1100 bytes each.