Cluster red, unassigned shards, no response on writes

decibel83 · December 28, 2022, 4:38pm

Hi everyone,
I'm dealing with a problem on Elasticsearch 7.17.5 (I've also tried to upgrade it to version 8 but I'm having the same problem with that version too) which is hanged, red status and unresponsive on any write request (for example POST and DELETE calls).

I tried to enlarge the Java heap size, restarting the server many times, with no success.

This is the _cluster/health output:

{
    "cluster_name": "my_elastic_cluster",
    "status": "red",
    "timed_out": false,
    "number_of_nodes": 1,
    "number_of_data_nodes": 1,
    "active_primary_shards": 234,
    "active_shards": 234,
    "relocating_shards": 0,
    "initializing_shards": 1,
    "unassigned_shards": 125,
    "delayed_unassigned_shards": 0,
    "number_of_pending_tasks": 252,
    "number_of_in_flight_fetch": 0,
    "task_max_waiting_in_queue_millis": 26331,
    "active_shards_percent_as_number": 65
}

Here you can find some information and logs:

/_cat/indices?v output: health status index uuid pri rep - Pastebin.com
/_cat/shards?v output: index shard prirep state - Pastebin.com
/var/log/elasticsearch/my_elastic_cluster.log file: [2022-12-28T16:14:53,129][INFO ][o.e.n.Node ] [elastic.my.domain.c - Pastebin.com

Could you help me to find out what's going on, and how to resume this server without loosing data, please?

Thank you very much!
Bye

leandrojmp · December 28, 2022, 5:50pm

How many nodes do you have?

Which version are you using? The log you shared says it is 7.17.7.

[2022-12-28T16:14:53,129][INFO ][o.e.n.Node ] [elastic.my.domain.com] version[7.17.7]

Also, this is everything you have in the log? I didn't find anything that could give a hint of your issue.

What is the status of the following request:

GET _cluster/allocation/explain

decibel83 · December 28, 2022, 6:46pm

This is a single node cluster dedicated to beats, with the following configuration:

bootstrap.memory_lock: false
cluster.initial_master_nodes:
- elastic.my.domain.com
cluster.name: my_elastic_cluster
discovery.seed_hosts:
- elastic.my.domain.com
http.port: '9200'
network.bind_host:
- 123.123.123.123
- 127.0.0.1
- localhost
network.host: 123.123.123.123
node.data: true
node.master: true
node.name: elastic.my.domain.com
transport.port: '9301'

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

action.auto_create_index: true

Yes, the version is 7.17.7, anyway I've already tried to upgrade to version 8 on this machine but I had the same problem, so I rolled back to 7.17.7.

That's all, I don't have any more lines in the log.

This is the output:

{
    "note": "No shard was specified in the explain API request, so this response explains a randomly chosen unassigned shard. There may be other unassigned shards in this cluster which cannot be assigned for different reasons. It may not be possible to assign this shard until one of the other shards is assigned correctly. To explain the allocation of other shards (whether assigned or unassigned) you must specify the target shard in the request to this API.",
    "index": "auditbeat-7.17.5-2022.11.28-000027",
    "shard": 0,
    "primary": true,
    "current_state": "unassigned",
    "unassigned_info": {
        "reason": "CLUSTER_RECOVERED",
        "at": "2022-12-28T16:15:07.145Z",
        "last_allocation_status": "throttled"
    },
    "can_allocate": "yes",
    "allocate_explanation": "can allocate the shard",
    "target_node": {
        "id": "94z2vzjeTPGgIWheedCseA",
        "name": "elastic.my.domain.com",
        "transport_address": "123.123.123.123:9301",
        "attributes": {
            "ml.machine_memory": "16821936128",
            "xpack.installed": "true",
            "transform.node": "true",
            "ml.max_open_jobs": "512",
            "ml.max_jvm_size": "8589934592"
        }
    },
    "allocation_id": "V_KwpEg6RkihWwMGTHKiRA",
    "node_allocation_decisions": [
        {
            "node_id": "94z2vzjeTPGgIWheedCseA",
            "node_name": "elastic.my.domain.com",
            "transport_address": "123.123.123.123:9301",
            "node_attributes": {
                "ml.machine_memory": "16821936128",
                "xpack.installed": "true",
                "transform.node": "true",
                "ml.max_open_jobs": "512",
                "ml.max_jvm_size": "8589934592"
            },
            "node_decision": "yes",
            "store": {
            "in_sync": true,
            "allocation_id": "V_KwpEg6RkihWwMGTHKiRA"
            }
        }
    ]
}

leandrojmp · December 28, 2022, 6:57pm

What is the result of:

GET _cat/health?v

Also, choose any UNASSIGNED index and run the following:

GET /_cluster/allocation/explain
{
  "index": "index-name",
  "shard": 0,
  "primary": true
}

It looks that your cluster is still bringing up your indices and you just need to wait.

decibel83 · December 28, 2022, 7:18pm

epoch      timestamp cluster          status node.total node.data shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1672254759 19:12:39  my_elastic_cluster red             1         1    234 234    0    1      125           305               2.9h                 65.0%

Also, choose any UNASSIGNED index and run the following:

GET /_cluster/allocation/explain
{
  "index": "index-name",
  "shard": 0,
  "primary": true
}

TEST root@elk ~# curl -X GET "localhost:9200/_cluster/allocation/explain?pretty" -H 'Content-Type: application/json' -d'
                                    {
                                      "index": "auditbeat-7.17.5-2022.11.28-000027",
                                      "shard": 0,
                                      "primary": true
                                    }
                                    '
{
  "index" : "auditbeat-7.17.5-2022.11.28-000027",
  "shard" : 0,
  "primary" : true,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "CLUSTER_RECOVERED",
    "at" : "2022-12-28T16:15:07.145Z",
    "last_allocation_status" : "throttled"
  },
  "can_allocate" : "yes",
  "allocate_explanation" : "can allocate the shard",
  "target_node" : {
    "id" : "94z2vzjeTPGgIWheedCseA",
    "name" : "elastic.my.domain.com",
    "transport_address" : "123.123.123.123:9301",
    "attributes" : {
      "ml.machine_memory" : "16821936128",
      "xpack.installed" : "true",
      "transform.node" : "true",
      "ml.max_open_jobs" : "512",
      "ml.max_jvm_size" : "8589934592"
    }
  },
  "allocation_id" : "V_KwpEg6RkihWwMGTHKiRA",
  "node_allocation_decisions" : [
    {
      "node_id" : "94z2vzjeTPGgIWheedCseA",
      "node_name" : "elastic.my.domain.com",
      "transport_address" : "123.123.123.123:9301",
      "node_attributes" : {
        "ml.machine_memory" : "16821936128",
        "xpack.installed" : "true",
        "transform.node" : "true",
        "ml.max_open_jobs" : "512",
        "ml.max_jvm_size" : "8589934592"
      },
      "node_decision" : "yes",
      "store" : {
        "in_sync" : true,
        "allocation_id" : "V_KwpEg6RkihWwMGTHKiRA"
      }
    }
  ]
}

Please note that this server is into this situation from some days...

DavidTurner · December 28, 2022, 8:30pm

What does GET _nodes/hot_threads?threads=9999 say?

decibel83 · January 5, 2023, 3:14pm

Hi David, sorry for my late answer!

I tried that call but it remains hanged without any answer even after many minutes:

TEST me@elk ~> curl -X GET "localhost:9200/_nodes/hot_threads?threads=9999"
...

I see no new error logs when I try to make this call.

This is the very same behaviour when I tried to delete some data or a index from this Elasticsearch instance...

Ayush_Mathur · January 5, 2023, 3:32pm

@decibel83 what does GET _cat/allocation?v returns ?

decibel83 · January 5, 2023, 3:47pm

No answer here too:

TEST me@elk ~> curl -X GET "localhost:9200/_cat/allocation?v"

Ayush_Mathur · January 5, 2023, 4:01pm

@decibel83 can you please run: curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED

DavidTurner · January 5, 2023, 5:07pm

That is interesting. Could you capture a thread dump of the elected master with jstack?

DavidTurner · January 5, 2023, 5:21pm

On reflection I'd probably want to see thread dumps from all nodes. I note that you have only one node, so of course that's the elected master and the one you should dump, but I'm just saying this for the sake of future readers who might encounter something similar.

decibel83 · January 25, 2023, 1:28pm

me@elk:/home/me# curl -XGET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 30942  100 30942    0     0   570k      0 --:--:-- --:--:-- --:--:--  570k
auditbeat-7.17.5-2022.11.28-000027                            0 p UNASSIGNED   CLUSTER_RECOVERED
filebeat-7.14.1-2022.04.08-000021                             0 p UNASSIGNED   CLUSTER_RECOVERED
filebeat-7.14.1-2022.01.18-000013                             0 p UNASSIGNED   CLUSTER_RECOVERED
apm-7.14.1-onboarding-2022.07.20                              0 p UNASSIGNED   CLUSTER_RECOVERED
apm-7.6.1-error-000009                                        0 r UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.14.1-2022.03.09-000018                            0 p UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.6.1-2022.01.19-000013                             0 p UNASSIGNED   CLUSTER_RECOVERED
filebeat-7.6.1-2022.02.08-000015                              0 p UNASSIGNED   CLUSTER_RECOVERED
.ds-ilm-history-5-2022.11.25-000025                           0 p UNASSIGNED   CLUSTER_RECOVERED
heartbeat-7.14.1-2022.02.17-000016                            0 p UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.14.1-2022.11.28-000056                            0 p UNASSIGNED   CLUSTER_RECOVERED
.kibana-event-log-7.13.1-000014                               0 p UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.14.1-2022.02.07-000015                            0 p UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.17.0-2022.03.29-000005                            0 p UNASSIGNED   CLUSTER_RECOVERED
.ds-.logs-deprecation.elasticsearch-default-2022.10.26-000009 0 p UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.6.1-2022.03.10-000018                             0 p UNASSIGNED   CLUSTER_RECOVERED
heartbeat-7.14.1-2022.02.07-000015                            0 p UNASSIGNED   CLUSTER_RECOVERED
.kibana-event-log-7.12.1-000017                               0 p UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.6.1-2022.01.29-000014                             0 p UNASSIGNED   CLUSTER_RECOVERED
filebeat-7.14.1-2022.11.21-000058                             0 p UNASSIGNED   CLUSTER_RECOVERED
.ds-ilm-history-5-2022.08.27-000019                           0 p UNASSIGNED   CLUSTER_RECOVERED
heartbeat-7.14.1-2022.11.28-000059                            0 p UNASSIGNED   CLUSTER_RECOVERED
.ds-.logs-deprecation.elasticsearch-default-2022.11.09-000010 0 p UNASSIGNED   CLUSTER_RECOVERED
filebeat-7.6.1-2022.03.10-000018                              0 p UNASSIGNED   CLUSTER_RECOVERED
.ds-ilm-history-5-2022.10.26-000023                           0 p UNASSIGNED   CLUSTER_RECOVERED
apm-7.6.1-onboarding-2022.10.19                               0 p UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.17.0-2022.11.28-000041                            0 p UNASSIGNED   CLUSTER_RECOVERED
.ds-.logs-deprecation.elasticsearch-default-2022.08.03-000003 0 p UNASSIGNED   CLUSTER_RECOVERED
filebeat-7.6.1-2022.11.18-000057                              0 p UNASSIGNED   CLUSTER_RECOVERED
apm-7.6.1-onboarding-2022.11.29                               0 p UNASSIGNED   CLUSTER_RECOVERED
apm-7.6.1-onboarding-2022.11.29                               0 r UNASSIGNED   CLUSTER_RECOVERED
apm-7.14.1-onboarding-2022.03.16                              0 p UNASSIGNED   CLUSTER_RECOVERED
metricbeat-7.14.1-2022.11.28-001899                           0 p UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.6.1-2022.11.28-000056                             0 p UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.14.1-2022.03.29-000020                            0 p UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.6.1-2022.03.30-000020                             0 p UNASSIGNED   CLUSTER_RECOVERED
filebeat-7.6.1-2022.03.30-000020                              0 p UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.14.1-2022.02.27-000017                            0 p UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.17.0-2022.04.08-000006                            0 p UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.17.0-2022.03.19-000004                            0 p UNASSIGNED   CLUSTER_RECOVERED
heartbeat-7.14.1-2022.01.08-000012                            0 p UNASSIGNED   CLUSTER_RECOVERED
filebeat-7.17.5-2022.11.28-000030                             0 p UNASSIGNED   CLUSTER_RECOVERED
heartbeat-7.14.1-2022.01.18-000013                            0 p UNASSIGNED   CLUSTER_RECOVERED
.kibana-event-log-7.14.1-000014                               0 p UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.6.1-2022.03.20-000019                             0 p UNASSIGNED   CLUSTER_RECOVERED
.kibana-event-log-7.17.5-000004                               0 p UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.17.6-2022.11.29-000012                            0 p UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.17.0-2022.02.16-000001                            0 p UNASSIGNED   CLUSTER_RECOVERED
metricbeat-7.17.5-2022.11.29-000710                           0 p UNASSIGNED   CLUSTER_RECOVERED
apm-7.14.1-onboarding-2022.12.15                              0 p UNASSIGNED   CLUSTER_RECOVERED
filebeat-7.14.1-2022.02.27-000017                             0 p UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.6.1-2022.02.18-000016                             0 p UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.14.1-2022.01.08-000012                            0 p UNASSIGNED   CLUSTER_RECOVERED
filebeat-7.6.1-2022.03.20-000019                              0 p UNASSIGNED   CLUSTER_RECOVERED
heartbeat-7.14.1-2022.03.19-000019                            0 p UNASSIGNED   CLUSTER_RECOVERED
filebeat-7.14.1-2022.11.16-000057                             0 p UNASSIGNED   CLUSTER_RECOVERED
.ds-ilm-history-5-2022.09.26-000021                           0 p UNASSIGNED   CLUSTER_RECOVERED
apm-7.14.1-onboarding-2022.03.09                              0 p UNASSIGNED   CLUSTER_RECOVERED
.kibana-event-log-7.12.1-000019                               0 p UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.6.1-2022.01.09-000012                             0 p UNASSIGNED   CLUSTER_RECOVERED
heartbeat-7.14.1-2022.02.27-000017                            0 p UNASSIGNED   CLUSTER_RECOVERED
apm-7.17.5-onboarding-2022.07.12                              0 p UNASSIGNED   CLUSTER_RECOVERED
filebeat-7.17.5-2022.11.23-000029                             0 p UNASSIGNED   CLUSTER_RECOVERED
filebeat-7.14.1-2022.11.26-000059                             0 p UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.6.1-2022.02.28-000017                             0 p UNASSIGNED   CLUSTER_RECOVERED
heartbeat-7.14.1-2022.03.29-000020                            0 p UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.14.1-2022.01.28-000014                            0 p UNASSIGNED   CLUSTER_RECOVERED
filebeat-7.14.1-2022.02.07-000015                             0 p UNASSIGNED   CLUSTER_RECOVERED
.ds-.logs-deprecation.elasticsearch-default-2022.07.20-000002 0 p UNASSIGNED   CLUSTER_RECOVERED
.kibana-event-log-7.17.5-000003                               0 p UNASSIGNED   CLUSTER_RECOVERED
filebeat-7.14.1-2022.03.19-000019                             0 p UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.14.1-2022.03.19-000019                            0 p UNASSIGNED   CLUSTER_RECOVERED
apm-7.6.1-profile-000009                                      0 r UNASSIGNED   CLUSTER_RECOVERED
.kibana-event-log-7.17.5-000002                               0 p UNASSIGNED   CLUSTER_RECOVERED
metricbeat-7.6.1-2022.11.29-000209                            0 p UNASSIGNED   CLUSTER_RECOVERED
filebeat-7.6.1-2022.01.09-000012                              0 p UNASSIGNED   CLUSTER_RECOVERED
.kibana-event-log-7.14.1-000015                               0 p UNASSIGNED   CLUSTER_RECOVERED
.ds-.logs-deprecation.elasticsearch-default-2022.07.06-000001 0 p UNASSIGNED   CLUSTER_RECOVERED
metricbeat-7.17.0-2022.11.29-000140                           0 p UNASSIGNED   CLUSTER_RECOVERED
filebeat-7.6.1-2022.02.18-000016                              0 p UNASSIGNED   CLUSTER_RECOVERED
filebeat-7.14.1-2022.01.28-000014                             0 p UNASSIGNED   CLUSTER_RECOVERED
filebeat-7.6.1-2022.01.19-000013                              0 p UNASSIGNED   CLUSTER_RECOVERED
.kibana-event-log-7.12.1-000020                               0 p UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.14.1-2022.04.08-000021                            0 p UNASSIGNED   CLUSTER_RECOVERED
filebeat-7.14.1-2022.03.29-000020                             0 p UNASSIGNED   CLUSTER_RECOVERED
filebeat-7.6.1-2022.01.29-000014                              0 p UNASSIGNED   CLUSTER_RECOVERED
filebeat-7.14.1-2022.01.08-000012                             0 p UNASSIGNED   CLUSTER_RECOVERED
heartbeat-7.17.5-2022.11.28-000030                            0 p UNASSIGNED   CLUSTER_RECOVERED
apm-7.14.1-onboarding-2022.12.02                              0 p UNASSIGNED   CLUSTER_RECOVERED
filebeat-7.6.1-2022.02.28-000017                              0 p UNASSIGNED   CLUSTER_RECOVERED
.kibana-event-log-7.13.1-000016                               0 p UNASSIGNED   CLUSTER_RECOVERED
.ds-.logs-deprecation.elasticsearch-default-2022.09.28-000007 0 p UNASSIGNED   CLUSTER_RECOVERED
filebeat-7.6.1-2022.11.28-000059                              0 p UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.17.0-2022.02.26-000002                            0 p UNASSIGNED   CLUSTER_RECOVERED
.kibana-event-log-7.13.1-000015                               0 p UNASSIGNED   CLUSTER_RECOVERED
heartbeat-7.14.1-2022.01.28-000014                            0 p UNASSIGNED   CLUSTER_RECOVERED
.kibana-event-log-7.14.1-000013                               0 p UNASSIGNED   CLUSTER_RECOVERED
metricbeat-7.17.6-2022.11.28-000028                           0 p UNASSIGNED   CLUSTER_RECOVERED
.ds-.logs-deprecation.elasticsearch-default-2022.10.12-000008 0 p UNASSIGNED   CLUSTER_RECOVERED
apm-7.6.1-onboarding-2022.07.12                               0 p UNASSIGNED   CLUSTER_RECOVERED
.ds-.logs-deprecation.elasticsearch-default-2022.08.17-000004 0 p UNASSIGNED   CLUSTER_RECOVERED
filebeat-7.6.1-2022.11.23-000058                              0 p UNASSIGNED   CLUSTER_RECOVERED
filebeat-7.17.5-2022.11.18-000028                             0 p UNASSIGNED   CLUSTER_RECOVERED
.kibana-event-log-7.17.5-000005                               0 p UNASSIGNED   CLUSTER_RECOVERED
apm-7.6.1-metric-000009                                       0 r UNASSIGNED   CLUSTER_RECOVERED
.ds-.logs-deprecation.elasticsearch-default-2022.08.31-000005 0 p UNASSIGNED   CLUSTER_RECOVERED
heartbeat-7.14.1-2022.03.09-000018                            0 p UNASSIGNED   CLUSTER_RECOVERED
apm-7.14.1-onboarding-2022.12.01                              0 p UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.17.0-2022.03.09-000003                            0 p UNASSIGNED   CLUSTER_RECOVERED
.kibana-event-log-7.13.1-000013                               0 p UNASSIGNED   CLUSTER_RECOVERED
.kibana-event-log-7.12.1-000018                               0 p UNASSIGNED   CLUSTER_RECOVERED
filebeat-7.14.1-2022.02.17-000016                             0 p UNASSIGNED   CLUSTER_RECOVERED
apm-7.6.1-transaction-000009                                  0 r UNASSIGNED   CLUSTER_RECOVERED
filebeat-7.14.1-2022.03.09-000018                             0 p UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.6.1-2022.02.08-000015                             0 p UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.14.1-2022.02.17-000016                            0 p UNASSIGNED   CLUSTER_RECOVERED
apm-7.6.1-span-000009                                         0 r UNASSIGNED   CLUSTER_RECOVERED
apm-7.6.1-onboarding-2022.03.22                               0 p UNASSIGNED   CLUSTER_RECOVERED
apm-7.6.1-onboarding-2022.10.18                               0 p UNASSIGNED   CLUSTER_RECOVERED
auditbeat-7.14.1-2022.01.18-000013                            0 p UNASSIGNED   CLUSTER_RECOVERED
.ds-.logs-deprecation.elasticsearch-default-2022.11.23-000011 0 r UNASSIGNED   CLUSTER_RECOVERED
.ds-.logs-deprecation.elasticsearch-default-2022.11.23-000011 0 p UNASSIGNED   CLUSTER_RECOVERED
apm-7.14.1-onboarding-2022.11.29                              0 p UNASSIGNED   CLUSTER_RECOVERED
.ds-.logs-deprecation.elasticsearch-default-2022.09.14-000006 0 p UNASSIGNED   CLUSTER_RECOVERED
.kibana-event-log-7.14.1-000012                               0 p UNASSIGNED   CLUSTER_RECOVERED

decibel83 · January 25, 2023, 1:36pm

root@elk:~# ps ax|grep elastic
 1774 ?        Ssl   73:33 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j2.formatMsgNoLookups=true -Djava.locale.providers=SPI,COMPAT --add-opens=java.base/java.io=ALL-UNNAMED -Djava.security.manager=allow -Xms8g -Xmx8g -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Djava.io.tmpdir=/tmp -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m -XX:MaxDirectMemorySize=4294967296 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=deb -Des.bundled_jdk=true -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
 1944 ?        Sl     0:00 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Here you can find the output of the command /usr/share/elasticsearch/jdk/bin/jstack 1944 : https://pastebin.com/H8cqLLtz.

Thank you!

DavidTurner · January 25, 2023, 1:59pm

I get this error:

Error, this is a private paste or is pending moderation. If this paste belongs to you, please login to Pastebin to view it.

Try https://gist.github.com instead?

decibel83 · January 25, 2023, 2:33pm

Here it is: jjstack output · GitHub

DavidTurner · January 25, 2023, 2:41pm

Thanks. This indicates you're hitting this bug:

github.com/elastic/elasticsearch

Generic threads deadlock related to ILMHistoryStore indexing

opened 04:30PM - 03 Feb 21 UTC

closed 03:58PM - 09 Jan 23 UTC

DaveCTurner

>bug :Data Management/ILM+SLM Team:Data Management

**Elasticsearch version** (`bin/elasticsearch --version`): 7.10.1 **Plugins i…nstalled**: Cloud **JVM version** (`java -version`): 15.0.1+9 **OS version** (`uname -a` if on a Unix-like system): Cloud **Description of the problem including expected versus actual behavior**: All generic threads are waiting with the following stack trace: ``` at java.util.concurrent.locks.LockSupport.park(Ljava/lang/Object;)V (LockSupport.java:211) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Ljava/util/concurrent/locks/AbstractQueuedSynchronizer$Node;IZZZJ)I (AbstractQueuedSynchronizer.java:714) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(I)V (AbstractQueuedSynchronizer.java:937) at java.util.concurrent.locks.ReentrantLock$Sync.lock()V (ReentrantLock.java:153) at java.util.concurrent.locks.ReentrantLock.lock()V (ReentrantLock.java:322) at org.elasticsearch.action.bulk.BulkProcessor.internalAdd(Lorg/elasticsearch/action/DocWriteRequest;)V (BulkProcessor.java:379) at org.elasticsearch.action.bulk.BulkProcessor.add(Lorg/elasticsearch/action/DocWriteRequest;)Lorg/elasticsearch/action/bulk/BulkProcessor; (BulkProcessor.java:361) at org.elasticsearch.action.bulk.BulkProcessor.add(Lorg/elasticsearch/action/index/IndexRequest;)Lorg/elasticsearch/action/bulk/BulkProcessor; (BulkProcessor.java:347) at org.elasticsearch.xpack.ilm.history.ILMHistoryStore.lambda$putAsync$0(Lorg/elasticsearch/action/index/IndexRequest;Lorg/elasticsearch/xpack/ilm/history/ILMHistoryItem;)V (ILMHistoryStore.java:150) at org.elasticsearch.xpack.ilm.history.ILMHistoryStore$$Lambda$8091+0x0000000801eda370.run()V (Unknown Source) at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run()V (ThreadContext.java:678) at java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run()V (ThreadPoolExecutor.java:630) at java.lang.Thread.run()V (Thread.java:832) ``` Meanwhile one of the scheduler threads is holding the lock on which they're waiting, and is blocked here: ``` jdk.internal.misc.Unsafe.park(ZJ)V (Native Method) java.util.concurrent.locks.LockSupport.park(Ljava/lang/Object;)V (LockSupport.java:211) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Ljava/util/concurrent/locks/AbstractQueuedSynchronizer$Node;IZZZJ)I (AbstractQueuedSynchronizer.java:714) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(I)V (AbstractQueuedSynchronizer.java:1046) java.util.concurrent.Semaphore.acquire()V (Semaphore.java:318) org.elasticsearch.action.bulk.BulkRequestHandler.execute(Lorg/elasticsearch/action/bulk/BulkRequest;J)V (BulkRequestHandler.java:59) org.elasticsearch.action.bulk.BulkProcessor.execute(Lorg/elasticsearch/action/bulk/BulkRequest;J)V (BulkProcessor.java:454) org.elasticsearch.action.bulk.BulkProcessor.execute()V (BulkProcessor.java:463) org.elasticsearch.action.bulk.BulkProcessor.access$400(Lorg/elasticsearch/action/bulk/BulkProcessor;)V (BulkProcessor.java:54) org.elasticsearch.action.bulk.BulkProcessor$Flush.run()V (BulkProcessor.java:503) org.elasticsearch.threadpool.Scheduler$ReschedulingRunnable.doRun()V (Scheduler.java:213) org.elasticsearch.common.util.concurrent.AbstractRunnable.run()V (AbstractRunnable.java:37) java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object; (Executors.java:515) java.util.concurrent.FutureTask.run()V (FutureTask.java:264) java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run()V (ScheduledThreadPoolExecutor.java:304) java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (ThreadPoolExecutor.java:1130) java.util.concurrent.ThreadPoolExecutor$Worker.run()V (ThreadPoolExecutor.java:630) java.lang.Thread.run()V (Thread.java:832) ``` The semaphore on which it is waiting is held, apparently, by another ongoing flush. I haven't chased this any further but I could believe that the ongoing flush needs a generic thread to make progress. **Steps to reproduce**: Unknown. **Provide logs (if relevant)**: Not available, but I can share a heap dump privately. **Workaround**: The deadlocked node must be restarted, it will not recover on its own. If the issue persists then the problematic component can be disabled by setting `indices.lifecycle.history_index_enabled: false` in the `elasticsearch.yml` file on each master-eligible node and then restarting them all.

A fix is planned for 8.7 but until then I suggest you use the workaround described in the issue.

system · February 22, 2023, 2:42pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.