How to Configure Failover Shipping to remote server when Logstash connection fails

Elastic Stack Beats
1

Hi Team,

I have a requirement where my filebeat will connect to Logstash and ships the file content. But I really dont have a failover mechanism now if my logstash goes down. If my logstash goes down, Log4j will rotate the files and I will end up losing data. So as a failover if logstash is unreachable for a specific amount of time, I need to move the files to a remote host where i have adequate storage. After a while when logstash is up, filebeat should continue to send to logstash.

Is there a way we can do this in filebeat ?

2

Filebeat by default is configured to keep the files open if they are not successfully shipped and it will start shipping again if LS becomes available. So the log rotation should not have any affect here.

You mentioned that in your case the storage is an issue? There is no way that Filebeat could move files to a different host for you. Also the problem is when you move the files back, Filebeat will start from scratch again as the files have a different inode id.

I think you better look into having multiple LS instances as a failover instead of trying to copy forth and back log files.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.