I was wondering about filebeat to logstash output resiliency. For example, we are using a single instance of logstash with a Elasticsearch cluster, what would happen if Logstash is done for a day?
Hey @ehsank777,
Filebeat keeps an internal queue of events before sending them to the output. This queue is stored in memory and can store by default till 4096 events. This queue can support in some cases some hiccups, but it shouldn't be trusted for resiliency.
Additionally, you can also enable a disk spool queue, that keeps a queue of events on disk. The number of events stored locally till the output is reachable again depends then on the available disk size. The disk size used for this queue can be limited.
You can learn more about these queues in this documentation page: https://www.elastic.co/guide/en/beats/filebeat/7.5/configuring-internal-queue.html
Is the behavior better documented anywhere? I tested the queue behavior a couple days ago, sent 500000 messages through filebeat stdin with the target inaccessible. A couple minutes later launched logstash (the intended target) and after some CPU load ALL 500000 messages were delivered to elasticsearch. This was the default configuration so I didn't mess with queue.mem or spool. I expected the cutoff point to be 4096 with the default config so that was surprising. Then I changed queue.mem -events to see how the process is affected but it didn't make any difference. So right now I don't know where those messages were queued in (memory? disk? how much can it grow?) and for how long (or for how many messages) I can depend on them being deliverable if the target goes down.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.