tami
August 23, 2019, 7:41pm
1
We are running ElasticSearch 6.7.2 in a 3.2.2 (master/client/data) node setup on kubernetes.
I have an ILM policy running to rollover a new index every 7 days, move to a warm stage, and then delete after 1 year. Here is the policy I am using:
{
"counter-policy" : {
"version" : 615,
"modified_date" : "2019-08-23T16:30:08.997Z",
"policy" : {
"phases" : {
"hot" : {
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_size" : "50gb",
"max_age" : "7d"
}
}
},
"delete" : {
"min_age" : "365d",
"actions" : {
"delete" : { }
}
},
"warm" : {
"min_age" : "30d",
"actions" : {
"forcemerge" : {
"max_num_segments" : 1
},
"shrink" : {
"number_of_shards" : 1
}
}
}
}
}
}
}
The rollover worked correctly the first couple of times, but now we are hitting a NullPtrException. I am calling _ilm/explain?pretty on my current write index, and am seeing this error (see next comment because character limits):
I am able to unstick this by using the retry step API, but it keeps happening again and again.
tami
August 23, 2019, 7:41pm
2
Here is the error, see the stack trace:
{
"indices" : {
"counter-000004" : {
"index" : "counter-000004",
"managed" : true,
"policy" : "counter-policy",
"lifecycle_date_millis" : 1565248631650,
"phase" : "hot",
"phase_time_millis" : 1565248632167,
"action" : "rollover",
"action_time_millis" : 1565249231840,
"step" : "ERROR",
"step_time_millis" : 1565548622691,
"failed_step" : "check-rollover-ready",
"step_info" : {
"type" : "null_pointer_exception",
"reason" : null,
"stack_trace" : "java.lang.NullPointerException\n\tat org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.evaluateConditions(TransportRolloverAction.java:253)\n\tat org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction$1.onResponse(TransportRolloverAction.java:128)\n\tat org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction$1.onResponse(TransportRolloverAction.java:125)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:85)\n\tat org.elasticsearch.action.support.TransportAction$1.onResponse(TransportAction.java:81)\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.onCompletion(TransportBroadcastByNodeAction.java:393)\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.start(TransportBroadcastByNodeAction.java:304)\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:236)\n\tat org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:78)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:120)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)\n\tat org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:87)\n\tat org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:76)\n\tat org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)\n\tat org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1269)\n\tat org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:71)\n\tat org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.masterOperation(TransportRolloverAction.java:124)\n\tat org.elasticsearch.action.admin.indices.rollover.TransportRolloverAction.masterOperation(TransportRolloverAction.java:67)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:124)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:211)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:208)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.start(TransportMasterNodeAction.java:164)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:141)\n\tat org.elasticsearch.action.support.master.TransportMasterNodeAction.doExecute(TransportMasterNodeAction.java:59)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:167)\n\tat org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:120)\n\tat org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:165)\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:139)\n\tat org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:81)\n\tat org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:87)\n\tat org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:76)\n\tat org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)\n\tat org.elasticsearch.xpack.core.ClientHelper.executeAsyncWithOrigin(ClientHelper.java:88)\n\tat org.elasticsearch.xpack.core.ClientHelper.executeWithHeadersAsync(ClientHelper.java:152)\n\tat org.elasticsearch.xpack.indexlifecycle.LifecyclePolicySecurityClient.doExecute(LifecyclePolicySecurityClient.java:55)\n\tat org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)\n\tat org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1269)\n\tat org.elasticsearch.client.support.AbstractClient$IndicesAdmin.rolloverIndex(AbstractClient.java:1777)\n\tat org.elasticsearch.xpack.core.indexlifecycle.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:115)\n\tat org.elasticsearch.xpack.indexlifecycle.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:133)\n\tat org.elasticsearch.xpack.indexlifecycle.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:270)\n\tat org.elasticsearch.xpack.indexlifecycle.IndexLifecycleService.triggered(IndexLifecycleService.java:213)\n\tat org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:168)\n\tat org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:196)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base/java.lang.Thread.run(Thread.java:835)\n"
},
"phase_execution" : {
"policy" : "counter-policy",
"phase_definition" : {
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_size" : "50gb",
"max_age" : "7d"
}
}
},
"version" : 483,
"modified_date_in_millis" : 1565248086813
}
}
}
}
dakrone
August 23, 2019, 8:14pm
3
Hi Aaron,
Thanks for reporting this, it was definitely a bug. It was previously fixed this for the 7.2.1 and later releases of elasticsearch, see: https://github.com/elastic/elasticsearch/pull/43353
tami
August 23, 2019, 8:17pm
4
I see. So you didn't backport this fix to any of the 6.* versions for ElasticSearch that also support ILM?
dakrone
August 23, 2019, 8:44pm
5
tami
August 23, 2019, 8:55pm
6
OK great thanks. We are currently using 6.7, will it get backported to there too? Or will we need to upgrade to 6.8
dakrone
August 23, 2019, 9:15pm
7
There aren't any more 6.7 releases planned (except in the case of security vulnerability fixes), so you'll need to upgrade to the latest 6.8 release once the next version is out.
tami
August 23, 2019, 9:28pm
8
OK thanks, we can upgrade to 6.8
tami
September 4, 2019, 11:08pm
9
Hi,
Is there any ETA on when a new 6.8.* version will be released containing this fix?
Thanks.
dakrone
September 5, 2019, 8:08pm
10
system
October 3, 2019, 8:08pm
11
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.