Hey all,
I have a bunch of indices stuck when they try to downsample. They're all elasticsearch monitor related indices coming from the Elastic Agent Elasticsearch integration.
When I manually tell ILM to try again, I get logs like the following.
{"@timestamp":"2024-05-28T15:19:21.777Z", "log.level": "INFO", "message":"Downsampling task [downsample-downsample-5m-.ds-metrics-elasticsearch.stack_monitoring.index-customspace-2024.01.16-000136-0-5m on shard [downsample-1m-.ds-metrics-elasticsearch.stack_monitoring.index-customspace-2024.01.16-000136][0] started", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[test-es-node-01][downsample_indexing][T#1]","log.logger":"org.elasticsearch.xpack.downsample.DownsampleShardIndexer","trace.id":"d5842b640588ebf650394594b322064c","elasticsearch.cluster.uuid":"clusteruuid","elasticsearch.node.id":"nodeid","elasticsearch.node.name":"test-es-node-01","elasticsearch.cluster.name":"testclustername"}
{"@timestamp":"2024-05-28T15:19:22.699Z", "log.level":"ERROR", "message":"Shard [[downsample-1m-.ds-metrics-elasticsearch.stack_monitoring.index-customspace-2024.01.16-000136][0]] failed to populate downsample index. Failures: [{null=org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:236] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #236],org.elasticsearch.xcontent.XContentParseException: [-1:236] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #236],org.elasticsearch.xcontent.XContentParseException: [-1:236] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #236],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237],org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237]}]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[test-es-node-01][write][T#4]","log.logger":"org.elasticsearch.xpack.downsample.DownsampleShardIndexer","trace.id":"d5842b640588ebf650394594b322064c","elasticsearch.cluster.uuid":"clusteruuid","elasticsearch.node.id":"nodeid","elasticsearch.node.name":"test-es-node-01","elasticsearch.cluster.name":"testclustername"}
{"@timestamp":"2024-05-28T15:19:22.707Z", "log.level":"ERROR", "message":"Shard [[downsample-1m-.ds-metrics-elasticsearch.stack_monitoring.index-customspace-2024.01.16-000136][0]] failed to populate downsample index. Failures: [{null=org.elasticsearch.xcontent.XContentParseException: [-1:237] Duplicate field 'min'\n at [Source: (byte[])[7299 bytes]; byte offset: #237]}]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[test-es-node-01][write][T#2]","log.logger":"org.elasticsearch.xpack.downsample.DownsampleShardIndexer","trace.id":"d5842b640588ebf650394594b322064c","elasticsearch.cluster.uuid":"clusteruuid","elasticsearch.node.id":"nodeid","elasticsearch.node.name":"test-es-node-01","elasticsearch.cluster.name":"testclustername"}
{"@timestamp":"2024-05-28T15:19:22.708Z", "log.level":"ERROR", "message":"Downsampling task [downsample-downsample-5m-.ds-metrics-elasticsearch.stack_monitoring.index-customspace-2024.01.16-000136-0-5m non-retriable failure [Rejecting request because bulk add has been cancelled by the caller]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[test-es-node-01][downsample_indexing][T#1]","log.logger":"org.elasticsearch.xpack.downsample.DownsampleShardPersistentTaskExecutor","trace.id":"d5842b640588ebf650394594b322064c","elasticsearch.cluster.uuid":"clusteruuid","elasticsearch.node.id":"nodeid","elasticsearch.node.name":"test-es-node-01","elasticsearch.cluster.name":"testclustername"}
{"@timestamp":"2024-05-28T15:19:23.599Z", "log.level":"ERROR", "message":"error while waiting for downsampling persistent task", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[test-es-node-01][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.downsample.TransportDownsampleAction","trace.id":"d5842b640588ebf650394594b322064c","elasticsearch.cluster.uuid":"clusteruuid","elasticsearch.node.id":"nodeid","elasticsearch.node.name":"test-es-node-01","elasticsearch.cluster.name":"testclustername","error.type":"org.elasticsearch.ElasticsearchException","error.message":"downsample task [downsample-downsample-5m-.ds-metrics-elasticsearch.stack_monitoring.index-customspace-2024.01.16-000136-0-5m] failed","error.stack_trace":"org.elasticsearch.ElasticsearchException: downsample task [downsample-downsample-5m-.ds-metrics-elasticsearch.stack_monitoring.index-customspace-2024.01.16-000136-0-5m] failed\n\tat org.elasticsearch.xpack.downsample.TransportDownsampleAction$2.onResponse(TransportDownsampleAction.java:425)\n\tat org.elasticsearch.xpack.downsample.TransportDownsampleAction$2.onResponse(TransportDownsampleAction.java:417)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.persistent.PersistentTasksService$1.onNewClusterState(PersistentTasksService.java:166)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onNewClusterState(ClusterStateObserver.java:380)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.clusterChanged(ClusterStateObserver.java:231)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateListener(ClusterApplierService.java:560)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:547)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:505)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:429)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:154)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:917)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:217)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:183)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n"}
{"@timestamp":"2024-05-28T15:19:23.600Z", "log.level":"ERROR", "message":"policy [metrics] for index [downsample-1m-.ds-metrics-elasticsearch.stack_monitoring.index-customspace-2024.01.16-000136] failed on step [{\"phase\":\"cold\",\"action\":\"downsample\",\"name\":\"rollup\"}]. Moving to ERROR step", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[test-es-node-01][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.xpack.ilm.IndexLifecycleRunner","trace.id":"d5842b640588ebf650394594b322064c","elasticsearch.cluster.uuid":"clusteruuid","elasticsearch.node.id":"nodeid","elasticsearch.node.name":"test-es-node-01","elasticsearch.cluster.name":"testclustername","error.type":"org.elasticsearch.ElasticsearchException","error.message":"downsample task [downsample-downsample-5m-.ds-metrics-elasticsearch.stack_monitoring.index-customspace-2024.01.16-000136-0-5m] failed","error.stack_trace":"org.elasticsearch.ElasticsearchException: downsample task [downsample-downsample-5m-.ds-metrics-elasticsearch.stack_monitoring.index-customspace-2024.01.16-000136-0-5m] failed\n\tat org.elasticsearch.xpack.downsample.TransportDownsampleAction$2.onResponse(TransportDownsampleAction.java:425)\n\tat org.elasticsearch.xpack.downsample.TransportDownsampleAction$2.onResponse(TransportDownsampleAction.java:417)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.persistent.PersistentTasksService$1.onNewClusterState(PersistentTasksService.java:166)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onNewClusterState(ClusterStateObserver.java:380)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.clusterChanged(ClusterStateObserver.java:231)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateListener(ClusterApplierService.java:560)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:547)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:505)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:429)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:154)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:917)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:217)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:183)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n"}
{"@timestamp":"2024-05-28T15:19:23.601Z", "log.level": "WARN", "message":"task [downsample-downsample-5m-.ds-metrics-elasticsearch.stack_monitoring.index-customspace-2024.01.16-000136-0-5m] failed with an exception", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[test-es-node-01][masterService#updateTask][T#6066]","log.logger":"org.elasticsearch.persistent.AllocatedPersistentTask","trace.id":"d5842b640588ebf650394594b322064c","elasticsearch.cluster.uuid":"clusteruuid","elasticsearch.node.id":"nodeid","elasticsearch.node.name":"test-es-node-01","elasticsearch.cluster.name":"testclustername","error.type":"org.elasticsearch.common.util.concurrent.EsRejectedExecutionException","error.message":"Rejecting request because bulk add has been cancelled by the caller","error.stack_trace":"org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: Rejecting request because bulk add has been cancelled by the caller\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.action.bulk.BulkProcessor2.addWithBackpressure(BulkProcessor2.java:336)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.action.bulk.BulkProcessor2.addWithBackpressure(BulkProcessor2.java:303)\n\tat org.elasticsearch.xpack.downsample.DownsampleShardIndexer$TimeSeriesBucketCollector.indexBucket(DownsampleShardIndexer.java:448)\n\tat org.elasticsearch.xpack.downsample.DownsampleShardIndexer$TimeSeriesBucketCollector$1.collect(DownsampleShardIndexer.java:414)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.search.aggregations.LeafBucketCollector.collect(LeafBucketCollector.java:86)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.search.aggregations.support.TimeSeriesIndexSearcher$LeafWalker.collectCurrent(TimeSeriesIndexSearcher.java:261)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.search.aggregations.support.TimeSeriesIndexSearcher.search(TimeSeriesIndexSearcher.java:169)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.search.aggregations.support.TimeSeriesIndexSearcher.search(TimeSeriesIndexSearcher.java:97)\n\tat org.elasticsearch.xpack.downsample.DownsampleShardIndexer.execute(DownsampleShardIndexer.java:164)\n\tat org.elasticsearch.xpack.downsample.DownsampleShardPersistentTaskExecutor$1.doRun(DownsampleShardPersistentTaskExecutor.java:220)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n"}
{"@timestamp":"2024-05-28T15:19:23.601Z", "log.level": "WARN", "message":"persistent task downsample-downsample-5m-.ds-metrics-elasticsearch.stack_monitoring.index-customspace-2024.01.16-000136-0-5m failed", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[test-es-node-01][generic][T#39]","log.logger":"org.elasticsearch.persistent.PersistentTasksClusterService","trace.id":"d5842b640588ebf650394594b322064c","elasticsearch.cluster.uuid":"clusteruuid","elasticsearch.node.id":"nodeid","elasticsearch.node.name":"test-es-node-01","elasticsearch.cluster.name":"testclustername","error.type":"org.elasticsearch.common.util.concurrent.EsRejectedExecutionException","error.message":"Rejecting request because bulk add has been cancelled by the caller","error.stack_trace":"org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: Rejecting request because bulk add has been cancelled by the caller\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.action.bulk.BulkProcessor2.addWithBackpressure(BulkProcessor2.java:336)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.action.bulk.BulkProcessor2.addWithBackpressure(BulkProcessor2.java:303)\n\tat org.elasticsearch.xpack.downsample.DownsampleShardIndexer$TimeSeriesBucketCollector.indexBucket(DownsampleShardIndexer.java:448)\n\tat org.elasticsearch.xpack.downsample.DownsampleShardIndexer$TimeSeriesBucketCollector$1.collect(DownsampleShardIndexer.java:414)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.search.aggregations.LeafBucketCollector.collect(LeafBucketCollector.java:86)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.search.aggregations.support.TimeSeriesIndexSearcher$LeafWalker.collectCurrent(TimeSeriesIndexSearcher.java:261)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.search.aggregations.support.TimeSeriesIndexSearcher.search(TimeSeriesIndexSearcher.java:169)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.search.aggregations.support.TimeSeriesIndexSearcher.search(TimeSeriesIndexSearcher.java:97)\n\tat org.elasticsearch.xpack.downsample.DownsampleShardIndexer.execute(DownsampleShardIndexer.java:164)\n\tat org.elasticsearch.xpack.downsample.DownsampleShardPersistentTaskExecutor$1.doRun(DownsampleShardPersistentTaskExecutor.java:220)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984)\n\tat org.elasticsearch.server@8.13.2/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n"}
{"@timestamp":"2024-05-28T15:19:29.167Z", "log.level": "INFO", "message":"[.ds-ilm-history-7-2024.05.28-000007/_tJHL0KQRb20mxy70tjkBA] update_mapping [_doc]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[test-es-node-01][masterService#updateTask][T#6066]","log.logger":"org.elasticsearch.cluster.metadata.MetadataMappingService","trace.id":"d5842b640588ebf650394594b322064c","elasticsearch.cluster.uuid":"clusteruuid","elasticsearch.node.id":"nodeid","elasticsearch.node.name":"test-es-node-01","elasticsearch.cluster.name":"testclustername"}
I'm seeing this on both my test/dev cluster and my prod cluster.
Anyone have any ideas? I checked the index mapping to see if there was something with multiple "min" values or keys, I didn't see anything.
I'm currently running my test stack on 8.12.2, prod on 8.13.2 (cannot recall why prod is ahead of test...) ES/Kibana/Fleet are running on a custom Docker images based on the official images. Filebeat and most Agents are installed via Apt on the server hosts. Docker hosting is Docker Swarm on Ubuntu 22.04.
Thanks in advance!