Exploring logs queuing application with Filebeat

[Nishant_Hurnekar (Nishant)]

Hi,

Is there any functionality available in Filebeat where filebeat application can send logs to some queuing application.

We are exploring such option in Filebeat because our access logs are keep rotating on every 15 minutes.

We aware as (Filebeat keeps track of the last lines sent and will continue reading the files as soon as the output becomes available again)

How do we push logs to logstash server withing losing any single logs in case filebeat or logstash server crashed or unavailable.

Our system as below
Filebeat --> Logstash --> Elasticsearch (3 server in cluster) --> KIbana

Kindly give us clarity in this matter.

Regards
Nishant

[Christian_Dahlqvist]

Why are you rotating your access logs so aggressively?

Logstash now support persistent queues, which allows it to buffer data on disk if it is unable to send it on to downstream systems. If you configured multiple Logstash instance so that you got some redundancy, you should be able to handle outages in Logstash and Elasticsearch without losing data.

If Filebeat however were to crash and not be restarted before logs are rotated, preventing logs from being read, you could however still lose data. I am not sure if there is any other way than making the rotation less aggressive to get around that though.

[Nishant_Hurnekar (Nishant)]

Thanks for your quick response.

We are rotating logs very aggressively for quick troubleshooting purpose.

As suggested, we will implement persistent queues option into logstash to avoid losing data.

[Christian_Dahlqvist]

Why does aggressive log rotation help troubleshoot quickly?

[Nishant_Hurnekar (Nishant)]

Single day access log file size is more than 30GB. There are many issue raised on web severs like data mining , crawling etc. If we tried to extract single file for extracting particular timeslot logs then it will take more time and system resources.

[ruflin]

By default filebeat has an at least once guarantee and will keep the files open even after rotation. Having 15min of log rotation should also not be an issue here. Did you see any issues so far?

[Nishant_Hurnekar (Nishant)]

As per our current settings filebeat functionality seems fine. We are looking for solution in filebeat in case filebeat sudden got crashed and recovery took more than 30 mins. In this scenario, rotated files won't be capture by Filebeat.

[ruflin]

In case filebeat crashes (an issue which I'm not aware of yet) I would assume it takes the system seconds to restart it. It sounds like this issue is less about filebeat but more how you handle recovery?