# Elastic search data nodes kept crashing continuously
**URL:** https://discuss.elastic.co/t/elastic-search-data-nodes-kept-crashing-continuously/337993
**Category:** Elasticsearch
**Created:** [July 10, 2023, 6:39am UTC](https://discuss.elastic.co/t/elastic-search-data-nodes-kept-crashing-continuously/337993 "2023-07-10T06:39:33Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@pratiksha](https://discuss.elastic.co/u/pratiksha)
#### Post date: [July 10, 2023, 6:39am UTC](https://discuss.elastic.co/t/elastic-search-data-nodes-kept-crashing-continuously/337993/1 "2023-07-10T06:39:33Z")
We are using Elasticsearch v7.10.2 deployed in kubernetes environment. Elasticsearch cluster was running fine earlier. As part of kubernetes cluster we updated a certificate in our k8s cluster which has not impacted elasticsearch in any way. After this all the elasticsearch pods got restarted and we see that ingest and master nodes have come up fine but we have 5 nodes deployed as data which all are continuously restarting with Error code 137. Tried increasing RAM and heap to higher values but it did not help.
Below log is seen before the restarts:
{"type":"log","host":"elasticsearch-data-0.","level":"INFO","time": "2023-06-02T08:56:08.887Z","logger":"o.e.e.NodeEnvironment","timezone":"UTC","marker":"[elasticsearch-data-0] ","log":{"message":"using [1] data paths, mounts [[/data (10.148.95.33:/ttsvmnas004\_edennet\_data/elasticsearch-data-0-pvc-d9a0cf2c-e0fb-402f-b3c6-62fe7db60b00)]], net usable\_space [1.3tb], net total\_space [1.8tb], types [nfs]"}}
This is cluster health output:
{
"cluster\_name" : "test",
"status" : "red",
"timed\_out" : false,
"number\_of\_nodes" : 7,
"number\_of\_data\_nodes" : 1,
"active\_primary\_shards" : 7,
"active\_shards" : 7,
"relocating\_shards" : 0,
"initializing\_shards" : 0,
"unassigned\_shards" : 956,
"delayed\_unassigned\_shards" : 0,
"number\_of\_pending\_tasks" : 0,
"number\_of\_in\_flight\_fetch" : 0,
"task\_max\_waiting\_in\_queue\_millis" : 0,
"active\_shards\_percent\_as\_number" : 0.726895119418484
}
We tried updating memory to a higher value but it did not help. However, after 2 days the cluster without any change came back to proper state and data nodes stopped restarting and all shards got assigned and it is been working fine since then.
Cluster health output after cluster was fine
{ "cluster\_name" : "test", "status" : "green", "timed\_out" : false, "number\_of\_nodes" : 11, "number\_of\_data\_nodes" : 5, "active\_primary\_shards" : 480, "active\_shards" : 960, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 0, "delayed\_unassigned\_shards" : 0, "number\_of\_pending\_tasks" : 0, "number\_of\_in\_flight\_fetch" : 0, "task\_max\_waiting\_in\_queue\_millis" : 0, "active\_shards\_percent\_as\_number" : 100.0 }
Please help in understanding as what could have lead to this behavior.
Thanks in advance.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 10, 2023, 6:45am UTC](https://discuss.elastic.co/t/elastic-search-data-nodes-kept-crashing-continuously/337993/2 "2023-07-10T06:45:39Z")
> [@pratiksha](#):
>
> net usable\_space [1.3tb], net total\_space [1.8tb], types [nfs]"}}
Are you using NFS storage for the Elasticsearch data nodes? If so, this is generally not recommended and requires that the storage is mounted so it behaves like local storage [as outlined in the docs](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/modules-node.html#data-path).
The log entry you linked to does not show why the nodes crashed. Please provide the full Elasticsearch logs from the failed startup attempt.
Also note that you are using an old version that has been EOL a long time. I would recommend upgrading to at least version 7.17.
---
### Author:  [@pratiksha](https://discuss.elastic.co/u/pratiksha)
#### Post date: [July 10, 2023, 8:54am UTC](https://discuss.elastic.co/t/elastic-search-data-nodes-kept-crashing-continuously/337993/3 "2023-07-10T08:54:30Z")
NFS storage is used but it is not shared type.
Below are the complete logs before it restarts, there is no error log seen before restarts:
> {"type":"log","host":"elasticsearch-data-0.tmo-shcd","level":"WARN","time": "2023-06-02T08:56:07.268Z","logger":"o.e.c.l.LogConfigurator","timezone":"UTC","marker":"[elasticsearch-data-0] ","log":{"message":"Some logging configurations have %marker but don't have %node\_name. We will automatically add %node\_name to the pattern to ease the migration for users who customize log4j2.properties but will stop this behavior in 7.0. You should manually replace `%node_name` with `[%node_name]%marker ` in these locations:
> /etc/elasticsearch/esconfig/log4j2.properties"}}
> {"type":"log","host":"elasticsearch-data-0.tmo-shcd","level":"INFO","time": "2023-06-02T08:56:07.572Z","logger":"o.e.n.Node","timezone":"UTC","marker":"[elasticsearch-data-0] ","log":{"message":"version[7.10.2], pid[18], build[oss/rpm/be1945029bf6730639d5bef8b39d8de9aa5efca8/2022-03-04T11:04:07.696525Z], OS[Linux/4.18.0-305.34.2.el8\_4.x86\_64/amd64], JVM[Red Hat, Inc./OpenJDK 64-Bit Server VM/11.0.14.1/11.0.14.1+1-LTS]"}}
> {"type":"log","host":"elasticsearch-data-0.tmo-shcd","level":"INFO","time": "2023-06-02T08:56:07.572Z","logger":"o.e.n.Node","timezone":"UTC","marker":"[elasticsearch-data-0] ","log":{"message":"JVM home [/usr/lib/jvm/java-11-openjdk-11.0.14.1.1-1.el7\_9.x86\_64], using bundled JDK [false]"}}
> {"type":"log","host":"elasticsearch-data-0.tmo-shcd","level":"INFO","time": "2023-06-02T08:56:07.572Z","logger":"o.e.n.Node","timezone":"UTC","marker":"[elasticsearch-data-0] ","log":{"message":"JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-2703224334137262050, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/tmp/elasticsearch/heapdump.hprof, -XX:ErrorFile=/tmp/elasticsearch/hs\_err.log, -Xlog:gc\*=warning:file=/tmp/elasticsearch/gc.log:utctime,pid,tags,level:filecount=2,filesize=2m, -Des.cgroups.hierarchy.override=/, -Xms10g, -Xmx10g, -XX:MaxDirectMemorySize=5368709120, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch/esconfig, -Des.distribution.flavor=oss, -Des.distribution.type=rpm, -Des.bundled\_jdk=true]"}}
> {"type":"log","host":"elasticsearch-data-0.tmo-shcd","level":"INFO","time": "2023-06-02T08:56:08.489Z","logger":"o.e.p.p.PrometheusExporterPlugin","timezone":"UTC","marker":"[elasticsearch-data-0] ","log":{"message":"starting Prometheus exporter plugin"}}
> {"type":"log","host":"elasticsearch-data-0.tmo-shcd","level":"INFO","time": "2023-06-02T08:56:08.820Z","logger":"o.e.p.PluginsService","timezone":"UTC","marker":"[elasticsearch-data-0] ","log":{"message":"loaded module [aggs-matrix-stats]"}}
> {"type":"log","host":"elasticsearch-data-0.tmo-shcd","level":"INFO","time": "2023-06-02T08:56:08.820Z","logger":"o.e.p.PluginsService","timezone":"UTC","marker":"[elasticsearch-data-0] ","log":{"message":"loaded module [analysis-common]"}}
> {"type":"log","host":"elasticsearch-data-0.tmo-shcd","level":"INFO","time": "2023-06-02T08:56:08.820Z","logger":"o.e.p.PluginsService","timezone":"UTC","marker":"[elasticsearch-data-0] ","log":{"message":"loaded module [geo]"}}
> {"type":"log","host":"elasticsearch-data-0.tmo-shcd","level":"INFO","time": "2023-06-02T08:56:08.820Z","logger":"o.e.p.PluginsService","timezone":"UTC","marker":"[elasticsearch-data-0] ","log":{"message":"loaded module [ingest-common]"}}
> {"type":"log","host":"elasticsearch-data-0.tmo-shcd","level":"INFO","time": "2023-06-02T08:56:08.821Z","logger":"o.e.p.PluginsService","timezone":"UTC","marker":"[elasticsearch-data-0] ","log":{"message":"loaded module [ingest-geoip]"}}
> {"type":"log","host":"elasticsearch-data-0.tmo-shcd","level":"INFO","time": "2023-06-02T08:56:08.821Z","logger":"o.e.p.PluginsService","timezone":"UTC","marker":"[elasticsearch-data-0] ","log":{"message":"loaded module [ingest-user-agent]"}}
> {"type":"log","host":"elasticsearch-data-0.tmo-shcd","level":"INFO","time": "2023-06-02T08:56:08.821Z","logger":"o.e.p.PluginsService","timezone":"UTC","marker":"[elasticsearch-data-0] ","log":{"message":"loaded module [kibana]"}}
> {"type":"log","host":"elasticsearch-data-0.tmo-shcd","level":"INFO","time": "2023-06-02T08:56:08.821Z","logger":"o.e.p.PluginsService","timezone":"UTC","marker":"[elasticsearch-data-0] ","log":{"message":"loaded module [lang-expression]"}}
> {"type":"log","host":"elasticsearch-data-0.tmo-shcd","level":"INFO","time": "2023-06-02T08:56:08.821Z","logger":"o.e.p.PluginsService","timezone":"UTC","marker":"[elasticsearch-data-0] ","log":{"message":"loaded module [lang-mustache]"}}
> {"type":"log","host":"elasticsearch-data-0.tmo-shcd","level":"INFO","time": "2023-06-02T08:56:08.821Z","logger":"o.e.p.PluginsService","timezone":"UTC","marker":"[elasticsearch-data-0] ","log":{"message":"loaded module [lang-painless]"}}
> {"type":"log","host":"elasticsearch-data-0.tmo-shcd","level":"INFO","time": "2023-06-02T08:56:08.821Z","logger":"o.e.p.PluginsService","timezone":"UTC","marker":"[elasticsearch-data-0] ","log":{"message":"loaded module [mapper-extras]"}}
> {"type":"log","host":"elasticsearch-data-0.tmo-shcd","level":"INFO","time": "2023-06-02T08:56:08.821Z","logger":"o.e.p.PluginsService","timezone":"UTC","marker":"[elasticsearch-data-0] ","log":{"message":"loaded module [parent-join]"}}
> {"type":"log","host":"elasticsearch-data-0.tmo-shcd","level":"INFO","time": "2023-06-02T08:56:08.821Z","logger":"o.e.p.PluginsService","timezone":"UTC","marker":"[elasticsearch-data-0] ","log":{"message":"loaded module [percolator]"}}
> {"type":"log","host":"elasticsearch-data-0.tmo-shcd","level":"INFO","time": "2023-06-02T08:56:08.821Z","logger":"o.e.p.PluginsService","timezone":"UTC","marker":"[elasticsearch-data-0] ","log":{"message":"loaded module [rank-eval]"}}
> {"type":"log","host":"elasticsearch-data-0.tmo-shcd","level":"INFO","time": "2023-06-02T08:56:08.821Z","logger":"o.e.p.PluginsService","timezone":"UTC","marker":"[elasticsearch-data-0] ","log":{"message":"loaded module [reindex]"}}
> {"type":"log","host":"elasticsearch-data-0.tmo-shcd","level":"INFO","time": "2023-06-02T08:56:08.821Z","logger":"o.e.p.PluginsService","timezone":"UTC","marker":"[elasticsearch-data-0] ","log":{"message":"loaded module [repository-url]"}}
> {"type":"log","host":"elasticsearch-data-0.tmo-shcd","level":"INFO","time": "2023-06-02T08:56:08.822Z","logger":"o.e.p.PluginsService","timezone":"UTC","marker":"[elasticsearch-data-0] ","log":{"message":"loaded module [systemd]"}}
> {"type":"log","host":"elasticsearch-data-0.tmo-shcd","level":"INFO","time": "2023-06-02T08:56:08.822Z","logger":"o.e.p.PluginsService","timezone":"UTC","marker":"[elasticsearch-data-0] ","log":{"message":"loaded module [transport-netty4]"}}
> {"type":"log","host":"elasticsearch-data-0.tmo-shcd","level":"INFO","time": "2023-06-02T08:56:08.822Z","logger":"o.e.p.PluginsService","timezone":"UTC","marker":"[elasticsearch-data-0] ","log":{"message":"loaded plugin [ingest-attachment]"}}
> {"type":"log","host":"elasticsearch-data-0.tmo-shcd","level":"INFO","time": "2023-06-02T08:56:08.822Z","logger":"o.e.p.PluginsService","timezone":"UTC","marker":"[elasticsearch-data-0] ","log":{"message":"loaded plugin [prometheus-exporter]"}}
> {"type":"log","host":"elasticsearch-data-0.tmo-shcd","level":"INFO","time": "2023-06-02T08:56:08.887Z","logger":"o.e.e.NodeEnvironment","timezone":"UTC","marker":"[elasticsearch-data-0] ","log":{"message":"using [1] data paths, mounts [[/data (10.148.95.33:/ttsvmnas004\_edennet\_data/tmo-shcd-datadir-elasticsearch-data-0-pvc-d9a0cf2c-e0fb-402f-b3c6-62fe7db60b00)]], net usable\_space [1.3tb], net total\_space [1.8tb], types [nfs]"}}
> {"type":"log","host":"elasticsearch-data-0.tmo-shcd","level":"INFO","time": "2023-06-02T08:56:08.887Z","logger":"o.e.e.NodeEnvironment","timezone":"UTC","marker":"[elasticsearch-data-0] ","log":{"message":"heap size [10gb], compressed ordinary object pointers [true]"}}
---
### Author:  [@pratiksha](https://discuss.elastic.co/u/pratiksha)
#### Post date: [July 12, 2023, 9:37am UTC](https://discuss.elastic.co/t/elastic-search-data-nodes-kept-crashing-continuously/337993/4 "2023-07-12T09:37:31Z")
Please check this and help.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [July 12, 2023, 9:48am UTC](https://discuss.elastic.co/t/elastic-search-data-nodes-kept-crashing-continuously/337993/5 "2023-07-12T09:48:08Z")
I do not see any error in the logs so there is not much to go on.
---
### Author:  [@pratiksha](https://discuss.elastic.co/u/pratiksha)
#### Post date: [July 12, 2023, 9:50am UTC](https://discuss.elastic.co/t/elastic-search-data-nodes-kept-crashing-continuously/337993/6 "2023-07-12T09:50:30Z")
Yes, this is what is not clear that why data nodes are restrting after this log message as when there is no error itself. What could be the root cause in such scenario or how this can be debugged.
---
### Author:  [@pratiksha](https://discuss.elastic.co/u/pratiksha)
#### Post date: [July 21, 2023, 8:12am UTC](https://discuss.elastic.co/t/elastic-search-data-nodes-kept-crashing-continuously/337993/7 "2023-07-21T08:12:01Z")
Hi,
For current scenario there are no error logs observed in the application logs. However, Can heap dump details be helpful in identifying and debugging of such scenarios? Also, if there is any other way to help in debugging of such above mentioned scenarios.
> **[JVM heap dumps | Elastic Cloud on Kubernetes \[master\] | Elastic](https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-jvm-heap-dumps.html)**
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [July 21, 2023, 8:43am UTC](https://discuss.elastic.co/t/elastic-search-data-nodes-kept-crashing-continuously/337993/8 "2023-07-21T08:43:54Z")
> [@pratiksha](#):
>
> What could be the root cause in such scenario or how this can be debugged.
Pretty much the only way a node will shut down without logging anything is if it receives a `SIGKILL` signal. This could be sent by any sufficiently-privileged process running on the same machine, but often it's the kernel's OOM killer. You'll need to look for logs related to this (e.g. the OOM killer reports its activities in the kernel logs which you can view with `dmesg`).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 18, 2023, 8:44am UTC](https://discuss.elastic.co/t/elastic-search-data-nodes-kept-crashing-continuously/337993/9 "2023-08-18T08:44:45Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.