I have a requirement where my filebeat will connect to Logstash and ships the file content. But I really dont have a failover mechanism now if my logstash goes down. If my logstash goes down, Log4j will rotate the files and I will end up losing data. So as a failover if logstash is unreachable for a specific amount of time, I need to move the files to a remote host where i have adequate storage. After a while when logstash is up, filebeat should continue to send to logstash.
Filebeat by default is configured to keep the files open if they are not successfully shipped and it will start shipping again if LS becomes available. So the log rotation should not have any affect here.
You mentioned that in your case the storage is an issue? There is no way that Filebeat could move files to a different host for you. Also the problem is when you move the files back, Filebeat will start from scratch again as the files have a different inode id.
I think you better look into having multiple LS instances as a failover instead of trying to copy forth and back log files.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.