Logs generated while logstash stopped weren't collected after restart

Elastic Stack Logstash
1

Hi, Im using ELK and filebeat to collect and analise logs on a server. The problem i had was that I stopped logstash for a few hours and started it again but it seems that the logs during that period were not collected by logstash after restarting it. So, is filebeat still pushing logs to logstash even if logstash is not working and filebeat wont push that logs again once logstash starts running?

Thank you.

2

Filebeat should pause while Logstash is down and send the logs when Logstash comes up again. Did the log files rotate during the Logstash outage? Is there anything interesting in the Filebeat logfile?

3

Did the log files rotate during the Logstash outage?

Log files rotated before the outage so all logs that were supposed to be collected are in the same log file.

Is there anything interesting in the Filebeat logfile?

Not really. It seems that filebeat was still publishing logs while logstash was not running.

The cause of the Logstash's outage is that I changed the user's password on Kibana GUI but forgot to apply that change to logstash.conf so the authentication error occurred in the pipeline. Does this information help?

4

Judging by https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#_retry_policy authentications problems aren't retried indefinitely so I'd expect such configuration problems to result in data loss.

5

I see. So, the pipeline between filebeat and logstash was okay and logstash received logs, but failed to connect to elasticsearch and didnt retry to send logs because of the retry policy, then logs were lost. Is that correct?

6

Yes, I think that's what happened here.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.