Delete and Re-Create Index with ES output plugin

Elastic Stack Logstash
1

Hi All

I have a situation where my source dataset refreshes everyday, so there is need to first drop/delete the index manually using curl command and run the LS to create the index again.

Is there any option where we can drop the index and create the index using ES output plugin

Thanks

2

Just make sure you provide a unique id for every document

3

I have a unique Ids in all the documents, but it is of no use as the whole document refresh or new document comes up everyday and the old ones are uselss. So i have to delete and recreate the index. I need to automate this through LS

This is typical case in ETL tool and we call this case as "truncate and load"

Important part here is index name should be same, as this will impact other programs downstream

4

The best solution is to use a unique id based on attributes that doesn't changes, and output will update the whole document
Otherwise you need to use curator or ILM
May be if you provide more details son your use case we can better help

5

Above statement is important , this is the use case

6

Logstash does not support that. It is designed to primarily handle never ending streams of data, not batches with defined start and end actions.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.