Logstash to delete documents in a particular index each time it is run

Elastic Stack Logstash
1

Hello,

Is it possible in Logstash 6.2.2 to delete all the documents in an index when a pipeline is executed? I need this as SQL queries sent via JDBC interface returns data in the following format: Top 5 largest tables and their names (Table 1, Size 1; Table2, Size 2 etc.) . Obviously this changes as transactions are made to the database. In particular, I noticed that when I delete a table (it is no longer in top 5 largest tables after this obviously) its name and size is still displayed in Kibana and Elasticsearch as I overwrite with respect to table name.

I think of deleting all the documents associated with that index, but do not know how to do that through Logstash. Could you provide help in this matter?

2

I do not know that Logstash can be made to do that. You'd be perhaps better suited to using the Rollover API to just create a new index each time, but keep the same alias, than to do a huge number of atomic delete operations.

1 Like
3

Do you mean the this API?

4

Yes. You can make use of Elasticsearch Curator to automate this.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.