Filebeat on no internet

Elastic Stack Beats
1

We have our servers located geographically at remote places (where the internet is not as reliable) the Filebeat shipper is responsible to ship the logs to our ELK via KAFKA -> LOGSTASH -> ELASTIC

The thing I'm unable to find from filebeat.conf is what is the expected behaviour of the Filebeat when the internet is not available does it retry the same, or is there a max attempt last?

Also,
Given the internet is not available (for days) does Filebeat store the stuff(which it could not send now) somewhere other than the actual logs file Bcoz I'm sceptical that logrotate would be rotating the file which would result in Filebeat missing on sending those log events to ELK

2

Filebeat is meant to ship data out of the host as quickly as possible, so storing data locally for a few days is unusual.

I don't know the size of your log files, but if you don't have internet access for a few days, when Filebeat regains access there will be a burst of events being sent to Kafka, so keep that in mind.

There are retries and backoff mechanism in place for all outputs, take a look at them to see how you can better tune them for your needs: Configure the Kafka output | Filebeat Reference [8.4] | Elastic

Filebeat uses an internal queue that can be in memory or disk, for your case the best would be a disk queue with a size large enough to keep all this data: Configure the internal queue | Filebeat Reference [8.4] | Elastic

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.