Hello all,
My filebeat forwarder stops sending logs to my logstash instance once every half hour or so. I looked at the DBG logs and it essentially says that the file hasn't changed and there are no logs to send. However, I know that there are new logs in the system and the only way I can send the new logs is to restart filebeat.
I looked at other topics that are similar to my problem and I think this might be caused by my rapid file rotation on the server. The logs Filebeat reads from the server can rotate about ever 30s. I heard this might cause a race condition and I'm wondering if there is any way to resolve that without changing the file rotation turnover.
My logstash configuration is the default one except for the path files and filebeat logging.
Thank you for all the help. My stack is consuming around 200k logs each half hour and I'm struggling to keep it up and running especially since it's my first time building the stack.