Multiple Indexes Created Instead of One

Elastic Stack Logstash
1

Hi,

I'm using Logstash to create a new index and the index name is set like this:
- index => "name-%{+YYYY-MM-dd.HH-mm}"

The execution of Logstash takes a bit more than one minute to process so once it is finished I end up with 2 Indexes:
- name-2017-12-05.13-00 - With part of the documents
- name-2017-12-05.13-01 - With the rest of the documents

I would like to have everything on the same index and keep the minutes information on the name. Is there a way to achieve this?

2

There's no obvious way of doing this. Why do you need minute resolution for the indexes?

3

Our case consists in Indexing part of a database (that contains public and private data).
Private data should never be indexed (the privacy of the data is mutable, so something that was once private can now be public and vice-versa).

So the Index is fully built (Less than 2000 documents total) every 30 min and then swapped with the old one using alias.
And in this sense my naming of the index contains the minutes to separate the indexes built in the same hour.

I see that there is not a straightforward answer for this, therefore we are likely to decrease the "refresh rate" of the index to 1 hour, avoiding this problem.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.