How to rollover to same index

Elastic Stack Elasticsearch
1

Hello Team,

I am trying to configure an ILM for a time series logs. Since they are logs, i am trying to impose a hot phase of 2d, and then essentially move them into the warm phase, where (i can proceed to mark them as read only, as they won't be written to, but will be queried, and shrink the indices.)

Below is the policy i had configured

PUT _ilm/policy/time_logs_data_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "2d"
          }
        }
      },
     "warm": {
        "actions": {
          "readonly" : { },
          "shrink" : {
            "number_of_shards": 20
          }
        }
      },
      "delete": {
        "min_age": "50d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
PUT _template/index_template
{
  "index_patterns": ["time_logs_*"],
  "settings": {
    "index.lifecycle.name": "time_logs_data_policy"
  }
}

My indices would be created as such - time_logs_2020_07_04, time_logs_2020_07_05 etc.

I see this exception with the configured ILM policy.

"""
java.lang.IllegalArgumentException: setting [index.lifecycle.rollover_alias] for index [time_logs_2020_06_24] is empty or not defined
	at org.elasticsearch.xpack.core.indexlifecycle.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:50)
	at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:133)
	at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:270)
	at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleService.triggered(IndexLifecycleService.java:213)
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:168)
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:196)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Now, i technically do not want to rollover to a new index, but essentially mark the index read only, and shrink it after 2d in hot phase. I see examples for actually rolling over to a new index.

eg. once time_logs_2020_07_02 is 2 days old, ready to be rolled over (on average, each index has 40 shards), i want to shrink it to 20 shards, and still maintain the same index name.

Could anyone help in figuring out, how do i maintain the same index, but still have ILM rollover policies declared ?

2

This is not how ILM was designed, so it won't work like that sorry. It needs a write alias attached to it that you then write to, otherwise you will need to create a policy for each of these individual indices, which defeats the purpose.

Why so many?

3

Thank you @warkolm for helping me out.
I understand. How would i set a write alias attached to it, so that, it can be still used to read effectively from it ?

eg. i have index for a day, time_logs_2020_07_01, after 2 days, i can rename this into something else, time_logs_2020_07_01-1 with the help of ILM i however, still need to mark the newly created index as read-only, and proceed to shrink it, since it won't be written to anymore, but will be queried from. Then i need to add the old index ( time_logs_2020_07_01) as alias to newly created read-only index. I then need to delete the old one.

I searched online, but i wasn't able to figure out this ILM policy out.

We are working with a index per day, and that's why there are so many shards. (it includes 20 primary + 20 replicas) If this is not ideal, what should i be looking into? What are the effects of this ?

4

Have you looked at ILM: Manage the index lifecycle | Elasticsearch Guide [8.11] | Elastic, it should explain what's needed.

How big are your indices?

5

I did, and that's how i wrote the initial policy, which then did not work as intended.
I am looking to shrink the already created indices, after 2 days. I think i should revisit, maybe i missed something. Thank you :slight_smile: @warkolm

Also, each index is around ~500-600gb.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.