When I restart the logstash service, the old documents are coming out.
I tried to stopping the filebeat service where the logs are coming from and I deleted the old documents. But when I restart the logstash service the logs I deleted came back.
Has anyone encountered this?
Are you using FB->LS? If do, logs read on FB side. There is the param registry: filebeat.registry.path: ${path.data}/registry in filebeat.yml which keeps tracks about files which has been read.
If files were deleted from a disk, no chance to get back in LS.
What does filebeat.registry.path do?
Thank you for your time @Rios
It keeps track which files have been processed. If you testing often, and a file is processed, FB will not process again the same file.
Details about the fields inside the registry are here.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.