I receive new CSV file with catalogue data everyday. I need to remove all old indexed documents/data from existing Elasticsearch index before starting processing records from new CSV file. How can I do it? Thanks.
I don't see a way of doing it from inside Logstash. Look into using e.g. curl outside of Logtash.
Or, better, always index the data into an index containing today's date, then reconfigure an index alias to point to today's index. Old indexes can be deleted with Curator. By doing this you have 100% data availability, i.e. at no point in time is the data unavailable because reindexing is taking place.
You might be able to reconfigure the alias with an index template so that a newly created daily index will automatically be pointed to by the index.