How to create rolling index with date as the index name?

[Sumeet_Singh]

Hi, my elastic search index will ingest thousands of documents per second. The service which puts documents in the index doesn't creates a new index, instead it just gets current data in nodejs and indexes docs in "log-YYYY.MM.DD". So, we know index is created automatically if not present.

Now, my question is can this practice of creating index and putting docs at the same time cause performance issues or failures given that index wil be ingesting thousands of docs per second ?

If the answer to above question is yes, how can I create a rolling index whitch date as the index name? Say today is 5 May, 2021, so i want automatic creation of index for 6 May, 2021 in the format log-2021.05.06.

[warkolm (Mark Walkom)]

Welcome to our community!

The reason for using these approaches is to reduce performance hits, due to excessively large, or small, shard sizes.

We are moving away from indices with timestamps in them like that, take a look at ILM instead.

[Sumeet_Singh]

Hi @warkolm, I can't change the index name format, "log-YYYY.MM.DD" is what I want because some of my services are dependent on this name.

The reason for using these approaches is to reduce performance hits, due to excessively large, or small, shard sizes.

Which approach you're talking about? Can you please eleborate more on this?

[Christian_Dahlqvist]

That naming convention only allows a single index per day which means rollover can not create a new index when triggered. You therefore either need to index based on time stamp only and skip rollover or add a sequence number that increments (not per day).

[Sumeet_Singh]

"log-YYYY.MM.DD-01" is this what are you implying here? adding a sequence as suffix but it won't increment the date right? In simple words I want automatic creation of index with date in its name.
I want to skip the rollover but before that I want to ensure whether there can be performance issues or failures when index is being initialized at the time of ingestion.

[Wave (Andrew Sullivan)]

Sumeet,

Others have addressed the issue of rollover, but I just wanted to answer your first question. The answer is no, you should not see performance issues or failures ingesting thousands of docs per second assuming a properly sized and maintained cluster.

-Andrew