I am using Filebeat, Logstash, and Elasticsearch to collect logs. I have received a query that some logs are not being captured by Elasticsearch. Is it possible to lose logs in this ELK server configuration? For example, if the server with Filebeat or Logstash is under heavy load. Or is there a …

Thanks for the answer. Yes, I agree with you. It is the same issue from the purpose of preventing loss of logs. However, the former asks if the logs can be lost if the server goes down. Here I am asking what are the cases of losing logs. For example, are there cases that time out? If so, we wou…

[image] its-ogawa: Is there a specification such that if the logs cannot be delivered for a certain period of time, they will be discarded? Yes, there is, but it is not exactly a time out, it is related to the event queue. Both Filebeat and Logstash have a queue where the events are buffered…

Thanks for the answer. Excellent answer. For Logstash I have already set the queue.type: persisted. Please let me know if there is any indicator about the size of the queue. The ELK servers we are using has 32GB memory and 4TB disks in a cluster configuration. I see that there is a setting for …

[image] its-ogawa: Please let me know if there is any indicator about the size of the queue. It is all in the documentation, check it here , you can configure the size and the max number of events, per default it accept unlimited events but is limited to 1 GB of size, if you need to increase i…

Is there a time limit for Filebeat or Logstash to attempt to retransmit?

Elastic Stack Elasticsearch

warkolm (Mark Walkom) September 6, 2022, 2:10am 2

Isn't this the same as Does Elasticsearch lose logs in the event of an unexpected outage??

Topic		Replies	Views
Are the logs lost if elastic search down Elasticsearch	4	714	August 22, 2021
For how long filebeat attempts sending logs to logstash while it is unreachable Beats filebeat	4	487	March 16, 2020
Logstash - What happen if logstash can't contact elasticsearch node Logstash	2	569	April 8, 2020
Logging events lost when Elastic Search connection goes down Logstash	3	944	July 6, 2017
Data loss prevention? Beats filebeat	8	1660	February 18, 2021

Is there a time limit for Filebeat or Logstash to attempt to retransmit?

Related topics