and close-inactive is default(5min).we have 100 transaction per second .I stop filebeat manually for specific time(for crash test) and when start it manually (with 2 million docs stored in second directory path )and some logs been duplicated. what is the solution ,is it possible solution to increase close-inactive time ?
It might be related to the log rotation strategies, more details on it you can find in article, do you use log rotation strategies that copy and truncate the input log file?
Log duplication may still happen independent of the log rotate strategy you use or any other setting you change.
One thing that helps avoiding log duplication in most of the cases is to use a custom _id value instead of letting Elasticsearch set the _id value.
But depending on how you are indexing your data (if you are using time based indices, if you are using data streams, if you are using rollover etc) you still may get some duplicate in some cases.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.