Getting error when indices getting rollover with ILM policy

Elastic Observability
1

Hi there,

I am getting ILM errors on all of my indexes. For example:

stack_trace" : "java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [jaeger-span] does not point to index [jaeger-span-2024-08-02]\n\tat org.elasticsearch.xpack.core.ilm.RolloverStep.performAction(RolloverStep.java:73)\n\tat org.elasticsearch.xpack.ilm.IndexLifecycleRunner.maybeRunAsyncAction(IndexLifecycleRunner.java:294)\n\tat org.elasticsearch.xpack.ilm.IndexLifecycleRunner$2.clusterStateProcessed(IndexLifecycleRunner.java:250)\n\tat org.elasticsearch.cluster.service.MasterService$SafeClusterStateTaskListener.clusterStateProcessed(MasterService.java:534)\n\tat org.elasticsearch.cluster.service.MasterService$TaskOutputs.lambda$processedDifferentClusterState$1(MasterService.java:421)\n\tat java.base/java.util.ArrayList.forEach(ArrayList.java:1511)\n\tat org.elasticsearch.cluster.service.MasterService$TaskOutputs.processedDifferentClusterState(MasterService.java:421)\n\tat org.elasticsearch.cluster.service.MasterService.onPublicationSuccess(MasterService.java:281)\n\tat org.elasticsearch.cluster.service.MasterService.publish(MasterService.java:273)\n\tat org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:250)\n\tat org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:73)\n\tat org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:151)\n\tat org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:150)\n\tat org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:188)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:684)\n\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:252)\n\tat org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:215)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.base/java.lang.Thread.run(Thread.java:832)\n

In my case both the index and the alias exist. The ILM profile should roll over each index at max 20GB or 14 days but its failing with above error. Any suggestions on this is helpful.

2

Hi @Rajeev_Aminbhavi Welcome to the comunity.

Mostly likely the Alias / Index was not set up properly.

What version of the stack are you using?

How did you initially setup up the template?

Are you using indices or data streams?

In kibana Dev Tools run the following show the command and results

GET _alias/jaeger-span

3

Hi @stephenb ,

My ES version is 7.10.2, integrating with Jaeger with version 1.57. Its a single node cluster.

Here are steps followed for template

curl -XPUT 'http://localhost:9200/_ilm/policy/jaeger_policy' -H 'Content-Type: application/json' -d '
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",  
        "actions": {
          "rollover": {
            "max_size": "20GB",  
            "max_age": "7d"
                
          }
        }
      },
      "delete": {
        "min_age": "15d",      
        "actions": {
          "delete": {}         
        }
      }
    }
  }
}'


 curl -XPUT 'http://localhost:9200/_template/jaeger_template' -H 'Content-Type: application/json' -d '
{
  "index_patterns": ["jaeger-span-*"],  
  "settings": {
    "number_of_shards": 1,               
    "number_of_replicas": 0,             
    "index.lifecycle.name": "jaeger_policy",  
    "index.lifecycle.rollover_alias": "jaeger-span" 
  }
}'

curl -XPOST 'http://localhost:9200/_aliases' -H 'Content-Type: application/json' -d '
{
  "actions": [
    {
      "add": {
        "index": "jaeger-span-*",
        "alias": "jaeger-span",
        "is_write_index": true
      }
    }
  ]
}'

Using indices.

Thanks,
Rajeev

4

Hi @Rajeev_Aminbhavi

First, 7.10..2 is ancient. You are not going to get much help... you should really upgrade with a matter of urgency

Second you alias is not correct

as your error says...

index.lifecycle.rollover_alias [jaeger-span] does not point to index [jaeger-span-2024-08-02]

would need to be

        "index": "jaeger-span-2024-08-02",
        "alias": "jaeger-span",
        "is_write_index": true

This known as bootstraping and needs to be done in the beginning ...
Then when jaeger-span a new alias should be created

    "index": "jaeger-span-2024-08-03",
    "alias": "jaeger-span",
    "is_write_index": true
5

GET _alias/jaeger-span

curl -XGET localhost:9200/_alias/jaeger-span?pretty
{
  "jaeger-span-2024-09-10" : {
    "aliases" : {
      "jaeger-span" : {
        "is_write_index" : true
      }
    }
  },
  "jaeger-span-2024-09-09" : {
    "aliases" : {
      "jaeger-span" : {
        "is_write_index" : false
      }
    }
  },
  "jaeger-span-2024-09-000010" : {
    "aliases" : {
      "jaeger-span" : {
        "is_write_index" : false
      }
    }
  }
}
6

@stephenb ,

Can i bootstrap again if so can you please point to correct steps.

Thanks,
Rajeev

7

This is the current write index.... you can only have one...

I am not really sure what state you are in. You look like you are trying to mix daily indices and ILM, which are similar concepts but not the same. I am not sure what is writing your data.

Your ILM policy says rollover on 7 days or 20 GB... but looks like whatever is writing your data is writing daily indices...

If you were to upgrade and use data streams this is all handled for you...

8

This is confusing...

1 Like
9

Is it possible to apply ILM for daily indices, yes as you mentioned data is getting written to daily indices. How can this be corrected, any guidance or suggestions.

10

Yes, you can use ILM with daily indices where the date is in the name but that means that you do not use rollover, which is currently part of your policy. You would then write to the indices directly and not have a write alias. In this scenarion ILM actions and transitions are based on the index creation timestamp rather than the rollover timestamp.