Hi, I'm trying to do the same as in this post: [image] Force logstash to reindex csv files Logstash Hello, Sometimes I need to force the re-indexing of files csv from the beginning. I try to delete the sincedb file and restart logstash, but re-indexing never restart fro…

So if I understood correctly you would like to read a CSV file using Filebeat. Then forward the events to Logstash and parse it. You can use the log input of Filebeat to read the logs. Set the output to Logstash. Then Logstash can read from its beats input and parse the CSV it has received. See mo…

Hi kvch, thank you for the reply. My pipeline is already up and running nicely: filebeat monitors a folder for all csv files, ships data to logstash where data is parsed and filtered, then sent to ES. When an entry in a csv file has its value modified, I want the pipeline to reread it and reindex …

I am afraid that it is a use case we are not supporting. Filebeat is not able to detect whether a line in a file has been modified or not. It only reads each line once (unless the entries from the registry are deleted, in this case the whole file is reread and resent).

There has to be a way. Do you intend to do something about this situation in a (near) future release? We need to use filebeat, filter data in logstash using the csv filter, and have a method to rewrite/reindex entries upon any entry changes in the csv. Please help, any additional info is highly ap…

We are not planning to support this use case. The architecture of Filebeat reader pipeline does not support processing files in this manner. So even if we decide to support it (which I doubt), it is going to be a massive undertaking, as the core of Filebeat is was not designed in this way.

ok, I'll try to do that. I'm sure someone else at some point faced the same scenario and had the same question in mind. This actually became a new requirement in our pipeline, which is why we went with filebeat and not directly with logstash. Thank you for the reply. I'll see if there's a workaro…

Discuss the Elastic Stack

Force logstash+FILEBEAT to reindex csv file

Elastic Stack Beats

kvch (Noémi Ványi) February 7, 2019, 10:37am 8

You could try opening an enhancement request on Github so it is recorded somewhere. But it might get rejected there.

Topic		Replies	Views
Force logstash to reindex csv files Logstash	4	2346	July 6, 2017
Force filebeat to reparse CSV Beats filebeat	4	657	July 14, 2020
Force logstash to reindex a CSV Logstash	4	618	July 26, 2020
Logstash 5.4 recreating the indices after deletion Logstash	5	731	April 3, 2018
Filebeats(5.40) keeps on recreating the indices after deletion Beats filebeat	8	1124	April 3, 2018

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries
Trademarks
Terms
Privacy
Brand
Code of Conduct

Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.

Force logstash+FILEBEAT to reindex csv file

Related topics