Beats & Logstash - What happens if logstash goes offline?

Elastic Stack Beats
1

I've setup an elasticsearch cluster with 3 Elasticsearch nodes, 1 Kibana, 1 Logstash. Everything is working as expected with one exception.

I currently have my Beats (metric, file, etc.) sending log data to my Logstash node. My question is, if I have to take the Logstash nodes down for maintenance, upgrade, unexpected crash, etc., what happens to the log data that the beats are collecting?

I tried to simulate this by bringing logstash offline for 30 minutes, then bringing it back online. I ended up with a 30 minute gap in metricbeat data but actually got all the filebeat data. Is this expected behaviour? Is there a way to modify this and have metricbeat behave like filebeat in this respect?

2

You have several alternatives:

  • Use the file spool queue on Metricbeat (beta!)
  • You can use a queueing system as output before reaching Logstash (Kafka, Redis...) using the supported outputs & Logstash inputs
  • Configure Metricbeat to write to file and use a Filebeat to send the files to Logstash
3

Thanks so much for the quick response. This is exactly what I was looking for.

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.