Import data vs. Manage index / index pattern


I don't know if I'm in the good section of the forum and I'm sorry about that but I need help to setup my import pattern.

We actually import data daily at 6 AM using Logstash (7.3) into Elasticsearch (7.4) in the index "ls-X" with the settings "action => update" and "doc_as_upsert => true".

So, each day, I'm updating the data in "ls-X" but I never have the same amount of documents.

For example, the first day, I import 100 documents. The next day, I import 125 documents (100 documents updated + 25 documents added). Finally, I import 75 documents.

So I have 75 documents updated and 50 documents that I would like to delete automatically.

How can I achieve this? Is there a way to delete data using Logstash?

I tried to create an index named "ls-X-%{+YYYY.MM.dd}". This way, I'm able to insert all the documents in a new index each day so I don't need to delete the documents.

But I don't know how to use the last day only in my index pattern in Kibana (7.4)? Is there a way to name the index pattern to be dynamic (like the index)?


This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.