ILM error killing all indexes

Elastic Stack Logstash
1

The es instance I'm working with crashed, and attempting to restart the kernel leads to the run command immediately being killed (runES.sh: line 1: 92654 Killed ./bin/Elasticsearch).

The log shows this error:

[2022-03-02T10:40:54,802][INFO ][o.e.x.i.IndexLifecycleRunner] [smartmedia] policy [logstash-policy] for index [logstash-testin
dex] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry att
empt [24812]
[2022-03-02T10:50:54,779][ERROR][o.e.x.i.IndexLifecycleRunner] [smartmedia] policy [logstash-policy] for index [logstash-testin
dex] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step
java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [logstash] does not point to index [logstash-testindex]
        at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:104) [x-pa
ck-core-7.8.1.jar:7.8.1]
        at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:173) [x-pack-ilm-7.8.1.ja
r:7.8.1]
        at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:329) [x-pack-ilm-7.8.1.
jar:7.8.1]
        at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:267) [x-pack-ilm-7.8.1.jar:7.
8.1]
        at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:183) [x-pack-core-7.8.1.
jar:7.8.1]
        at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:211) [x-pack-core-7.8
.1.jar:7.8.1]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]

If I understand correctly, the problem is this part: [logstash] does not point to index [logstash-testindex], however I do not know how to change an alias for an index if I can't even start the kernel. Is there a config file I need to change instead?

2

Can you share more of your Elasticsearch log please?

3

It's this repeated:

[2022-03-02T10:40:54,802][INFO ][o.e.x.i.IndexLifecycleRunner] [smartmedia] policy [logstash-policy] for index [logstash-testindex] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [24812]
[2022-03-02T10:50:54,779][ERROR][o.e.x.i.IndexLifecycleRunner] [smartmedia] policy [logstash-policy] for index [logstash-testindex] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step
java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [logstash] does not point to index [logstash-testindex]
	at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:104) [x-pack-core-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:173) [x-pack-ilm-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:329) [x-pack-ilm-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:267) [x-pack-ilm-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:183) [x-pack-core-7.8.1.jar:7.8.1]
	at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:211) [x-pack-core-7.8.1.jar:7.8.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
[2022-03-02T11:00:54,822][INFO ][o.e.x.i.IndexLifecycleRunner] [smartmedia] policy [logstash-policy] for index [logstash-testindex] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [24813]

That's the entire Elasticsearch.log file. Is there a different log file that would be helpful?

4

That should not cause Elasticsearch to stop/crash.

That's the issue.
So what does your policy look like?

5

I don't know as I'm not the person who set up or used logstash. How would I check the policy of an index when Elasticsearch isn't running? the properties file for logstash doesn't mention any index alias (see below)

status = error

appender.console.type = Console
appender.console.name = console
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n

######## Server JSON ############################
appender.rolling.type = RollingFile
appender.rolling.name = rolling
appender.rolling.fileName = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}_server.json
appender.rolling.layout.type = ESJsonLayout
appender.rolling.layout.type_name = server

appender.rolling.filePattern = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}-%d{yyyy-MM-dd}-%i.json.gz
appender.rolling.policies.type = Policies
appender.rolling.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling.policies.time.interval = 1
appender.rolling.policies.time.modulate = true
appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
appender.rolling.policies.size.size = 128MB
appender.rolling.strategy.type = DefaultRolloverStrategy
appender.rolling.strategy.fileIndex = nomax
appender.rolling.strategy.action.type = Delete
appender.rolling.strategy.action.basepath = ${sys:es.logs.base_path}
appender.rolling.strategy.action.condition.type = IfFileName
appender.rolling.strategy.action.condition.glob = ${sys:es.logs.cluster_name}-*
appender.rolling.strategy.action.condition.nested_condition.type = IfAccumulatedFileSize
appender.rolling.strategy.action.condition.nested_condition.exceeds = 2GB
################################################
######## Server -  old style pattern ###########
appender.rolling_old.type = RollingFile
appender.rolling_old.name = rolling_old
appender.rolling_old.fileName = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}.log
appender.rolling_old.layout.type = PatternLayout
appender.rolling_old.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] [%node_name]%marker %m%n

appender.rolling_old.filePattern = ${sys:es.logs.base_path}${sys:file.separator}${sys:es.logs.cluster_name}-%d{yyyy-MM-dd}-%i.log.gz
appender.rolling_old.policies.type = Policies
appender.rolling_old.policies.time.type = TimeBasedTriggeringPolicy
appender.rolling_old.policies.time.interval = 1
appender.rolling_old.policies.time.modulate = true
appender.rolling_old.policies.size.type = SizeBasedTriggeringPolicy
appender.rolling_old.policies.size.size = 128MB
appender.rolling_old.strategy.type = DefaultRolloverStrategy
appender.rolling_old.strategy.fileIndex = nomax
appender.rolling_old.strategy.action.type = Delete
appender.rolling_old.strategy.action.basepath = ${sys:es.logs.base_path}
appender.rolling_old.strategy.action.condition.type = IfFileName
6

I'd like to add that trying to run Elasticsearch currently leads to the process immediately being killed:

esStack/elasticsearch-7.8.1$ bash runES.sh
[0.002s][warning][logging] Output options for existing outputs are ignored.
runES.sh: line 1: 102120 Killed                  ./bin/elasticsearch

with nothing being added to the logs

7

These two issues are not related.

What is this script? It's not something that we provide.

8

It contains the following:

./bin/elasticsearch

Running directly with ./bin/Elasticsearch causes the same warning with the process being immediately killed

9

What user are you running it at?
What happens when you run ./bin/elasticsearch directly?

10

I don't understand the first question, however as I said above when I run ./bin/Elasticsearch I get the same error, with the process being immediately killed.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.