# Unexpected ES shut down - nothing there in logs to identify the problem **URL:** \ **Category:** Elasticsearch\ **Created:** [June 20, 2012, 10:53am UTC](https://discuss.elastic.co/t/unexpected-es-shut-down-nothing-there-in-logs-to-identify-the-problem/8166 "2012-06-20T10:53:37Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![jagdeep](https://avatars.discourse-cdn.com/v4/letter/j/ba8739/32.png) [@jagdeep](https://discuss.elastic.co/u/jagdeep)\ **Post date:** [June 20, 2012, 10:53am UTC](https://discuss.elastic.co/t/unexpected-es-shut-down-nothing-there-in-logs-to-identify-the-problem/8166/1 "2012-06-20T10:53:37Z")
I am facing a very unexpected shutdown of ES data node. It dumps the memory and automatically getting shut(even console is getting closed). I have already dealt with memory related issues but this one is pretty weird because nothing is there in the logs. Server configuration OS - Windows server 2008 RT standard CPU - 24 cores RAM - 32 GB ES configuration Data node 1 - ES\_MIN\_MEM = 1024m ES\_MAX\_MEM=20g MAster node 1 - ES\_MIN\_MEM = 1024m ES\_MAX\_MEM=5g Cluster state is { cluster\_name: elasticsearch\_\*\*\*\* status: green timed\_out: false number\_of\_nodes: 2 number\_of\_data\_nodes: 1 active\_primary\_shards: 330 active\_shards: 330 relocating\_shards: 0 initializing\_shards: 0 unassigned\_shards: 0 } Data node state { name: dmz14\_data\_node transport\_address: inet[/\*\*\*.\*\*\*.\ ***.** :9301] hostname: \*\*\*\*\*\*\*\*\*\* attributes: { tag: \*\*\*\*\*\*\*\*\*\*\*\* master: false } indices: { store: { size: 21.2gb size\_in\_bytes: 22845246770 } docs: { count: 2525295 deleted: 551707 } indexing: { index\_total: 1207877 index\_time: 21.1m index\_time\_in\_millis: 1266718 index\_current: 0 delete\_total: 0 delete\_time: 0s delete\_time\_in\_millis: 0 delete\_current: 0 } get: { total: 0 time: 0s time\_in\_millis: 0 exists\_total: 0 exists\_time: 0s exists\_time\_in\_millis: 0 missing\_total: 0 missing\_time: 0s missing\_time\_in\_millis: 0 current: 0 } search: { query\_total: 8075 query\_time: 31.7s query\_time\_in\_millis: 31702 query\_current: 0 fetch\_total: 1242 fetch\_time: 907ms fetch\_time\_in\_millis: 907 fetch\_current: 0 } cache: { field\_evictions: 0 field\_size: 73.9mb field\_size\_in\_bytes: 77579988 filter\_count: 217 filter\_evictions: 0 filter\_size: 981.7kb filter\_size\_in\_bytes: 1005320 } merges: { current: 0 current\_docs: 0 current\_size: 0b current\_size\_in\_bytes: 0 total: 2073 total\_time: 28m total\_time\_in\_millis: 1681573 total\_docs: 5233813 total\_size: 20.6gb total\_size\_in\_bytes: 22176964666 } refresh: { total: 18970 total\_time: 22.8m total\_time\_in\_millis: 1373282 } flush: { total: 4072 total\_time: 1.6m total\_time\_in\_millis: 98095 } } } This cluster works fine with search, indexing, facteing etc but after some irregular interval(between 24-48hrs) data node is getting shut. Surprisingly there is no activity on the cluster when this happens, i mean no search and indexing. Logs are also pretty much clear. Logs when system is crashed and after manual restart of the cluster are [2012-06-20 08:00:04,678][WARN][transport.netty] [dmz14\_data\_node] Message not fully read (response) for [188842] handler org.elasticsearch.action.support.master.TransportMasterNodeOperationAction $4@75f34e16, error [true], resetting [2012-06-20 08:00:12,681][WARN][transport.netty] [dmz14\_data\_node] Message not fully read (response) for [188852] handler org.elasticsearch.action.support.master.TransportMasterNodeOperationAction $4@799b0299, error [true], resetting [2012-06-20 08:00:21,105][WARN][transport.netty] [dmz14\_data\_node] Message not fully read (response) for [188862] handler org.elasticsearch.action.support.master.TransportMasterNodeOperationAction $4@f75cb00, error [true], resetting [2012-06-20 08:00:30,200][WARN][transport.netty] [dmz14\_data\_node] Message not fully read (response) for [188873] handler org.elasticsearch.action.support.master.TransportMasterNodeOperationAction $4@1ffcf6e7, error [true], resetting [2012-06-20 08:00:39,544][WARN][transport.netty] [dmz14\_data\_node] Message not fully read (response) for [188884] handler org.elasticsearch.action.support.master.TransportMasterNodeOperationAction $4@5892644d, error [true], resetting [2012-06-20 08:00:49,076][WARN][transport.netty] [dmz14\_data\_node] Message not fully read (response) for [188896] handler org.elasticsearch.action.support.master.TransportMasterNodeOperationAction $4@557e75f5, error [true], resetting [2012-06-20 08:00:59,341][WARN][transport.netty] [dmz14\_data\_node] Message not fully read (response) for [188908] handler org.elasticsearch.action.support.master.TransportMasterNodeOperationAction $4@21dadca4, error [true], resetting [2012-06-20 08:01:09,434][WARN][transport.netty] [dmz14\_data\_node] Message not fully read (response) for [188920] handler org.elasticsearch.action.support.master.TransportMasterNodeOperationAction $4@10dbd700, error [true], resetting [2012-06-20 08:01:17,249][WARN][transport.netty] [dmz14\_data\_node] Message not fully read (response) for [188930] handler org.elasticsearch.action.support.master.TransportMasterNodeOperationAction $4@35cbd34b, error [true], resetting [2012-06-20 08:01:23,942][WARN][transport.netty] [dmz14\_data\_node] Message not fully read (response) for [188938] handler org.elasticsearch.action.support.master.TransportMasterNodeOperationAction $4@a2a8fed, error [true], resetting [2012-06-20 08:01:30,307][WARN][transport.netty] [dmz14\_data\_node] Message not fully read (response) for [188947] handler org.elasticsearch.action.support.master.TransportMasterNodeOperationAction $4@62411fb4, error [true], resetting [2012-06-20 08:01:39,292][WARN][transport.netty] [dmz14\_data\_node] Message not fully read (response) for [188957] handler org.elasticsearch.action.support.master.TransportMasterNodeOperationAction $4@36ea5908, error [true], resetting [2012-06-20 10:08:20,126][INFO][node] [dmz14\_data\_node] {0.19.2}[16084]: initializing ... [2012-06-20 10:08:20,194][INFO][plugins] [dmz14\_data\_node] loaded [es-river-rss-oe, river-rss, lang- javascript], sites [head] [2012-06-20 10:08:25,750][INFO][node] [dmz14\_data\_node] {0.19.2}[16084]: initialized [2012-06-20 10:08:25,750][INFO][node] [dmz14\_data\_node] {0.19.2}[16084]: starting ... [2012-06-20 10:08:26,176][INFO][transport] [dmz14\_data\_node] bound\_address {inet[/0:0:0:0:0:0:0:0:9301]}, publish\_address {inet[/192.168.150.14:9301]} [2012-06-20 10:08:56,190][WARN][discovery] [dmz14\_data\_node] waited for 30s and no initial state was set by the discovery [2012-06-20 10:08:56,191][INFO][discovery] [dmz14\_data\_node] elasticsearch\_dmz/WXMk\_K-VSaulKWfPYcAHXQ [2012-06-20 10:08:56,403][INFO][http] [dmz14\_data\_node] bound\_address {inet[/0:0:0:0:0:0:0:0:8081]}, publish\_address {inet[/192.168.150.14:8081]} [2012-06-20 10:08:56,404][INFO][node] [dmz14\_data\_node] {0.19.2}[16084]: started I am working on ES for quite some time now and dealt with many issues sometime by tuning my setting and analyzing logs to identify the problem but this time I am clueless as there is no error in the logs. Please suggest how can I hunt this down. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 21, 2012, 7:33am UTC](https://discuss.elastic.co/t/unexpected-es-shut-down-nothing-there-in-logs-to-identify-the-problem/8166/2 "2012-06-21T07:33:56Z")
I assume you look at the logs and oyu don't see any "shutdown" / closing messages. This can happen for several reasons (I've see all of those): - an external process is calling "kill -9": Tricky to find, you need to find the process that does that... - the service wrapper (if you use it) kills the process. Check the service wrapper logs. - The JVM crashed. Check for jvm crash logs (hs\_err files). On Wed, Jun 20, 2012 at 12:53 PM, jagdeep [reach.jagdeep@gmail.com](mailto:reach.jagdeep@gmail.com) wrote: > I am facing a very unexpected shutdown of ES data node. It dumps the > memory and automatically getting shut(even console is getting closed). > I have already dealt with memory related issues but this one is pretty > weird because nothing is there in the logs. > > Server configuration > > OS - Windows server 2008 RT standard > CPU - 24 cores > RAM - 32 GB > > ES configuration > > Data node 1 - ES\_MIN\_MEM = 1024m > ES\_MAX\_MEM=20g > MAster node 1 - ES\_MIN\_MEM = 1024m > ES\_MAX\_MEM=5g > > Cluster state is { > cluster\_name: elasticsearch\_\*\*\*\* > status: green > timed\_out: false > number\_of\_nodes: 2 > number\_of\_data\_nodes: 1 > active\_primary\_shards: 330 > active\_shards: 330 > relocating\_shards: 0 > initializing\_shards: 0 > unassigned\_shards: 0 > } > > Data node state > > { > name: dmz14\_data\_node > transport\_address: inet[/_ **.** _.\ ***.** :9301] > hostname: \*\*\*\*\*\*\*\*\*\* > attributes: { > tag: \*\*\*\*\*\*\*\*\*\*\*\* > master: false > } > indices: { > store: { > size: 21.2gb > size\_in\_bytes: 22845246770 > } > docs: { > count: 2525295 > deleted: 551707 > } > indexing: { > index\_total: 1207877 > index\_time: 21.1m > index\_time\_in\_millis: 1266718 > index\_current: 0 > delete\_total: 0 > delete\_time: 0s > delete\_time\_in\_millis: 0 > delete\_current: 0 > } > get: { > total: 0 > time: 0s > time\_in\_millis: 0 > exists\_total: 0 > exists\_time: 0s > exists\_time\_in\_millis: 0 > missing\_total: 0 > missing\_time: 0s > missing\_time\_in\_millis: 0 > current: 0 > } > search: { > query\_total: 8075 > query\_time: 31.7s > query\_time\_in\_millis: 31702 > query\_current: 0 > fetch\_total: 1242 > fetch\_time: 907ms > fetch\_time\_in\_millis: 907 > fetch\_current: 0 > } > cache: { > field\_evictions: 0 > field\_size: 73.9mb > field\_size\_in\_bytes: 77579988 > filter\_count: 217 > filter\_evictions: 0 > filter\_size: 981.7kb > filter\_size\_in\_bytes: 1005320 > } > merges: { > current: 0 > current\_docs: 0 > current\_size: 0b > current\_size\_in\_bytes: 0 > total: 2073 > total\_time: 28m > total\_time\_in\_millis: 1681573 > total\_docs: 5233813 > total\_size: 20.6gb > total\_size\_in\_bytes: 22176964666 > } > refresh: { > total: 18970 > total\_time: 22.8m > total\_time\_in\_millis: 1373282 > } > flush: { > total: 4072 > total\_time: 1.6m > total\_time\_in\_millis: 98095 > } > } > } > > This cluster works fine with search, indexing, facteing etc but after > some irregular interval(between 24-48hrs) data node is getting shut. > Surprisingly there is no activity on the cluster when this happens, i > mean no search and indexing. Logs are also pretty much clear. > Logs when system is crashed and after manual restart of the cluster > are > > [2012-06-20 08:00:04,678][WARN][transport.netty] > [dmz14\_data\_node] Message not fully read (response) for [188842] > handler > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > $4@75f34e16, error [true], resetting > [2012-06-20 08:00:12,681][WARN][transport.netty] > [dmz14\_data\_node] Message not fully read (response) for [188852] > handler > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > $4@799b0299, error [true], resetting > [2012-06-20 08:00:21,105][WARN][transport.netty] > [dmz14\_data\_node] Message not fully read (response) for [188862] > handler > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > $4@f75cb00, error [true], resetting > [2012-06-20 08:00:30,200][WARN][transport.netty] > [dmz14\_data\_node] Message not fully read (response) for [188873] > handler > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > $4@1ffcf6e7, error [true], resetting > [2012-06-20 08:00:39,544][WARN][transport.netty] > [dmz14\_data\_node] Message not fully read (response) for [188884] > handler > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > $4@5892644d, error [true], resetting > [2012-06-20 08:00:49,076][WARN][transport.netty] > [dmz14\_data\_node] Message not fully read (response) for [188896] > handler > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > $4@557e75f5, error [true], resetting > [2012-06-20 08:00:59,341][WARN][transport.netty] > [dmz14\_data\_node] Message not fully read (response) for [188908] > handler > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > $4@21dadca4, error [true], resetting > [2012-06-20 08:01:09,434][WARN][transport.netty] > [dmz14\_data\_node] Message not fully read (response) for [188920] > handler > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > $4@10dbd700, error [true], resetting > [2012-06-20 08:01:17,249][WARN][transport.netty] > [dmz14\_data\_node] Message not fully read (response) for [188930] > handler > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > $4@35cbd34b, error [true], resetting > [2012-06-20 08:01:23,942][WARN][transport.netty] > [dmz14\_data\_node] Message not fully read (response) for [188938] > handler > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > $4@a2a8fed, error [true], resetting > [2012-06-20 08:01:30,307][WARN][transport.netty] > [dmz14\_data\_node] Message not fully read (response) for [188947] > handler > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > $4@62411fb4, error [true], resetting > [2012-06-20 08:01:39,292][WARN][transport.netty] > [dmz14\_data\_node] Message not fully read (response) for [188957] > handler > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > $4@36ea5908, error [true], resetting > [2012-06-20 10:08:20,126][INFO][node] > [dmz14\_data\_node] {0.19.2}[16084]: initializing ... > [2012-06-20 10:08:20,194][INFO][plugins] > [dmz14\_data\_node] loaded [es-river-rss-oe, river-rss, lang- > javascript], sites [head] > [2012-06-20 10:08:25,750][INFO][node] > [dmz14\_data\_node] {0.19.2}[16084]: initialized > [2012-06-20 10:08:25,750][INFO][node] > [dmz14\_data\_node] {0.19.2}[16084]: starting ... > [2012-06-20 10:08:26,176][INFO][transport] > [dmz14\_data\_node] bound\_address {inet[/0:0:0:0:0:0:0:0:9301]}, > publish\_address {inet[/192.168.150.14:9301]} > [2012-06-20 10:08:56,190][WARN][discovery] > [dmz14\_data\_node] waited for 30s and no initial state was set by the > discovery > [2012-06-20 10:08:56,191][INFO][discovery] > [dmz14\_data\_node] elasticsearch\_dmz/WXMk\_K-VSaulKWfPYcAHXQ > [2012-06-20 10:08:56,403][INFO][http] > [dmz14\_data\_node] bound\_address {inet[/0:0:0:0:0:0:0:0:8081]}, > publish\_address {inet[/192.168.150.14:8081]} > [2012-06-20 10:08:56,404][INFO][node] > [dmz14\_data\_node] {0.19.2}[16084]: started > > I am working on ES for quite some time now and dealt with many issues > sometime by tuning my setting and analyzing logs to identify the > problem but this time I am clueless as there is no error in the logs. > > Please suggest how can I hunt this down. ---
**Author:** ![jagdeep](https://avatars.discourse-cdn.com/v4/letter/j/ba8739/32.png) [@jagdeep](https://discuss.elastic.co/u/jagdeep)\ **Post date:** [June 21, 2012, 7:50am UTC](https://discuss.elastic.co/t/unexpected-es-shut-down-nothing-there-in-logs-to-identify-the-problem/8166/3 "2012-06-21T07:50:22Z")
Thanks Shay for the response. 1. I stopped using service wrapper to track this problem but its still persisting. 2. Will definitely check JVM crash files. 3. It dumps the memory before getting crashed, heap dump of size 21-26 GB. I can understand the heap dump at the time of faceting or indexing but at this time there is no activity what so ever. Any pointer towards this heap dump? On Jun 21, 12:33 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > I assume you look at the logs and oyu don't see any "shutdown" / closing > messages. This can happen for several reasons (I've see all of those): > > - an external process is calling "kill -9": Tricky to find, you need to > find the process that does that... > - the service wrapper (if you use it) kills the process. Check the service > wrapper logs. > - The JVM crashed. Check for jvm crash logs (hs\_err files). > > On Wed, Jun 20, 2012 at 12:53 PM, jagdeep [reach.jagd...@gmail.com](mailto:reach.jagd...@gmail.com) wrote: > > > I am facing a very unexpected shutdown of ES data node. It dumps the > > memory and automatically getting shut(even console is getting closed). > > I have already dealt with memory related issues but this one is pretty > > weird because nothing is there in the logs. > > > Server configuration > > > OS - Windows server 2008 RT standard > > CPU - 24 cores > > RAM - 32 GB > > > ES configuration > > > Data node 1 - ES\_MIN\_MEM = 1024m > > ES\_MAX\_MEM=20g > > MAster node 1 - ES\_MIN\_MEM = 1024m > > ES\_MAX\_MEM=5g > > > Cluster state is { > > cluster\_name: elasticsearch\_\*\*\*\* > > status: green > > timed\_out: false > > number\_of\_nodes: 2 > > number\_of\_data\_nodes: 1 > > active\_primary\_shards: 330 > > active\_shards: 330 > > relocating\_shards: 0 > > initializing\_shards: 0 > > unassigned\_shards: 0 > > } > > > Data node state > > > { > > name: dmz14\_data\_node > > transport\_address: inet[/_ **.** _.\ ***.** :9301] > > hostname: \*\*\*\*\*\*\*\*\*\* > > attributes: { > > tag: \*\*\*\*\*\*\*\*\*\*\*\* > > master: false > > } > > indices: { > > store: { > > size: 21.2gb > > size\_in\_bytes: 22845246770 > > } > > docs: { > > count: 2525295 > > deleted: 551707 > > } > > indexing: { > > index\_total: 1207877 > > index\_time: 21.1m > > index\_time\_in\_millis: 1266718 > > index\_current: 0 > > delete\_total: 0 > > delete\_time: 0s > > delete\_time\_in\_millis: 0 > > delete\_current: 0 > > } > > get: { > > total: 0 > > time: 0s > > time\_in\_millis: 0 > > exists\_total: 0 > > exists\_time: 0s > > exists\_time\_in\_millis: 0 > > missing\_total: 0 > > missing\_time: 0s > > missing\_time\_in\_millis: 0 > > current: 0 > > } > > search: { > > query\_total: 8075 > > query\_time: 31.7s > > query\_time\_in\_millis: 31702 > > query\_current: 0 > > fetch\_total: 1242 > > fetch\_time: 907ms > > fetch\_time\_in\_millis: 907 > > fetch\_current: 0 > > } > > cache: { > > field\_evictions: 0 > > field\_size: 73.9mb > > field\_size\_in\_bytes: 77579988 > > filter\_count: 217 > > filter\_evictions: 0 > > filter\_size: 981.7kb > > filter\_size\_in\_bytes: 1005320 > > } > > merges: { > > current: 0 > > current\_docs: 0 > > current\_size: 0b > > current\_size\_in\_bytes: 0 > > total: 2073 > > total\_time: 28m > > total\_time\_in\_millis: 1681573 > > total\_docs: 5233813 > > total\_size: 20.6gb > > total\_size\_in\_bytes: 22176964666 > > } > > refresh: { > > total: 18970 > > total\_time: 22.8m > > total\_time\_in\_millis: 1373282 > > } > > flush: { > > total: 4072 > > total\_time: 1.6m > > total\_time\_in\_millis: 98095 > > } > > } > > } > > > This cluster works fine with search, indexing, facteing etc but after > > some irregular interval(between 24-48hrs) data node is getting shut. > > Surprisingly there is no activity on the cluster when this happens, i > > mean no search and indexing. Logs are also pretty much clear. > > Logs when system is crashed and after manual restart of the cluster > > are > > > [2012-06-20 08:00:04,678][WARN][transport.netty] > > [dmz14\_data\_node] Message not fully read (response) for [188842] > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > $4@75f34e16, error [true], resetting > > [2012-06-20 08:00:12,681][WARN][transport.netty] > > [dmz14\_data\_node] Message not fully read (response) for [188852] > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > $4@799b0299, error [true], resetting > > [2012-06-20 08:00:21,105][WARN][transport.netty] > > [dmz14\_data\_node] Message not fully read (response) for [188862] > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > $4@f75cb00, error [true], resetting > > [2012-06-20 08:00:30,200][WARN][transport.netty] > > [dmz14\_data\_node] Message not fully read (response) for [188873] > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > $4@1ffcf6e7, error [true], resetting > > [2012-06-20 08:00:39,544][WARN][transport.netty] > > [dmz14\_data\_node] Message not fully read (response) for [188884] > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > $4@5892644d, error [true], resetting > > [2012-06-20 08:00:49,076][WARN][transport.netty] > > [dmz14\_data\_node] Message not fully read (response) for [188896] > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > $4@557e75f5, error [true], resetting > > [2012-06-20 08:00:59,341][WARN][transport.netty] > > [dmz14\_data\_node] Message not fully read (response) for [188908] > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > $4@21dadca4, error [true], resetting > > [2012-06-20 08:01:09,434][WARN][transport.netty] > > [dmz14\_data\_node] Message not fully read (response) for [188920] > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > $4@10dbd700, error [true], resetting > > [2012-06-20 08:01:17,249][WARN][transport.netty] > > [dmz14\_data\_node] Message not fully read (response) for [188930] > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > $4@35cbd34b, error [true], resetting > > [2012-06-20 08:01:23,942][WARN][transport.netty] > > [dmz14\_data\_node] Message not fully read (response) for [188938] > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > $4@a2a8fed, error [true], resetting > > [2012-06-20 08:01:30,307][WARN][transport.netty] > > [dmz14\_data\_node] Message not fully read (response) for [188947] > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > $4@62411fb4, error [true], resetting > > [2012-06-20 08:01:39,292][WARN][transport.netty] > > [dmz14\_data\_node] Message not fully read (response) for [188957] > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > $4@36ea5908, error [true], resetting > > [2012-06-20 10:08:20,126][INFO][node] > > [dmz14\_data\_node] {0.19.2}[16084]: initializing ... > > [2012-06-20 10:08:20,194][INFO][plugins] > > [dmz14\_data\_node] loaded [es-river-rss-oe, river-rss, lang- > > javascript], sites [head] > > [2012-06-20 10:08:25,750][INFO][node] > > [dmz14\_data\_node] {0.19.2}[16084]: initialized > > [2012-06-20 10:08:25,750][INFO][node] > > [dmz14\_data\_node] {0.19.2}[16084]: starting ... > > [2012-06-20 10:08:26,176][INFO][transport] > > [dmz14\_data\_node] bound\_address {inet[/0:0:0:0:0:0:0:0:9301]}, > > publish\_address {inet[/192.168.150.14:9301]} > > [2012-06-20 10:08:56,190][WARN][discovery] > > [dmz14\_data\_node] waited for 30s and no initial state was set by the > > discovery > > [2012-06-20 10:08:56,191][INFO][discovery] > > [dmz14\_data\_node] elasticsearch\_dmz/WXMk\_K-VSaulKWfPYcAHXQ > > [2012-06-20 10:08:56,403][INFO][http] > > [dmz14\_data\_node] bound\_address {inet[/0:0:0:0:0:0:0:0:8081]}, > > publish\_address {inet[/192.168.150.14:8081]} > > [2012-06-20 10:08:56,404][INFO][node] > > [dmz14\_data\_node] {0.19.2}[16084]: started > > > I am working on ES for quite some time now and dealt with many issues > > sometime by tuning my setting and analyzing logs to identify the > > problem but this time I am clueless as there is no error in the logs. > > > Please suggest how can I hunt this down. ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [June 22, 2012, 7:50pm UTC](https://discuss.elastic.co/t/unexpected-es-shut-down-nothing-there-in-logs-to-identify-the-problem/8166/4 "2012-06-22T19:50:57Z")
This smells like OOM (which doesn't always seem to get logged). ## Otis Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) Scalable Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) On Thursday, June 21, 2012 3:50:22 AM UTC-4, jagdeep wrote: > Thanks Shay for the response. > > 1. I stopped using service wrapper to track this problem but its still > persisting. > 2. Will definitely check JVM crash files. > 3. It dumps the memory before getting crashed, heap dump of size 21-26 > GB. I can understand the heap dump at the time of faceting or > indexing but at this time there is no activity what so ever. > > Any pointer towards this heap dump? > > On Jun 21, 12:33 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > I assume you look at the logs and oyu don't see any "shutdown" / closing > > messages. This can happen for several reasons (I've see all of those): > > > > - an external process is calling "kill -9": Tricky to find, you need to > > find the process that does that... > > - the service wrapper (if you use it) kills the process. Check the > > service > > wrapper logs. > > - The JVM crashed. Check for jvm crash logs (hs\_err files). > > > > On Wed, Jun 20, 2012 at 12:53 PM, jagdeep [reach.jagd...@gmail.com](mailto:reach.jagd...@gmail.com) > > wrote: > > > > > I am facing a very unexpected shutdown of ES data node. It dumps the > > > memory and automatically getting shut(even console is getting closed). > > > I have already dealt with memory related issues but this one is pretty > > > weird because nothing is there in the logs. > > > > > Server configuration > > > > > OS - Windows server 2008 RT standard > > > CPU - 24 cores > > > RAM - 32 GB > > > > > ES configuration > > > > > Data node 1 - ES\_MIN\_MEM = 1024m > > > ES\_MAX\_MEM=20g > > > MAster node 1 - ES\_MIN\_MEM = 1024m > > > ES\_MAX\_MEM=5g > > > > > Cluster state is { > > > cluster\_name: elasticsearch\_\*\*\*\* > > > status: green > > > timed\_out: false > > > number\_of\_nodes: 2 > > > number\_of\_data\_nodes: 1 > > > active\_primary\_shards: 330 > > > active\_shards: 330 > > > relocating\_shards: 0 > > > initializing\_shards: 0 > > > unassigned\_shards: 0 > > > } > > > > > Data node state > > > > > { > > > name: dmz14\_data\_node > > > transport\_address: inet[/_ **.** _.\ ***.** :9301] > > > hostname: \*\*\*\*\*\*\*\*\*\* > > > attributes: { > > > tag: \*\*\*\*\*\*\*\*\*\*\*\* > > > master: false > > > } > > > indices: { > > > store: { > > > size: 21.2gb > > > size\_in\_bytes: 22845246770 > > > } > > > docs: { > > > count: 2525295 > > > deleted: 551707 > > > } > > > indexing: { > > > index\_total: 1207877 > > > index\_time: 21.1m > > > index\_time\_in\_millis: 1266718 > > > index\_current: 0 > > > delete\_total: 0 > > > delete\_time: 0s > > > delete\_time\_in\_millis: 0 > > > delete\_current: 0 > > > } > > > get: { > > > total: 0 > > > time: 0s > > > time\_in\_millis: 0 > > > exists\_total: 0 > > > exists\_time: 0s > > > exists\_time\_in\_millis: 0 > > > missing\_total: 0 > > > missing\_time: 0s > > > missing\_time\_in\_millis: 0 > > > current: 0 > > > } > > > search: { > > > query\_total: 8075 > > > query\_time: 31.7s > > > query\_time\_in\_millis: 31702 > > > query\_current: 0 > > > fetch\_total: 1242 > > > fetch\_time: 907ms > > > fetch\_time\_in\_millis: 907 > > > fetch\_current: 0 > > > } > > > cache: { > > > field\_evictions: 0 > > > field\_size: 73.9mb > > > field\_size\_in\_bytes: 77579988 > > > filter\_count: 217 > > > filter\_evictions: 0 > > > filter\_size: 981.7kb > > > filter\_size\_in\_bytes: 1005320 > > > } > > > merges: { > > > current: 0 > > > current\_docs: 0 > > > current\_size: 0b > > > current\_size\_in\_bytes: 0 > > > total: 2073 > > > total\_time: 28m > > > total\_time\_in\_millis: 1681573 > > > total\_docs: 5233813 > > > total\_size: 20.6gb > > > total\_size\_in\_bytes: 22176964666 > > > } > > > refresh: { > > > total: 18970 > > > total\_time: 22.8m > > > total\_time\_in\_millis: 1373282 > > > } > > > flush: { > > > total: 4072 > > > total\_time: 1.6m > > > total\_time\_in\_millis: 98095 > > > } > > > } > > > } > > > > > This cluster works fine with search, indexing, facteing etc but after > > > some irregular interval(between 24-48hrs) data node is getting shut. > > > Surprisingly there is no activity on the cluster when this happens, i > > > mean no search and indexing. Logs are also pretty much clear. > > > Logs when system is crashed and after manual restart of the cluster > > > are > > > > > [2012-06-20 08:00:04,678][WARN][transport.netty] > > > [dmz14\_data\_node] Message not fully read (response) for [188842] > > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > $4@75f34e16, error [true], resetting > > > [2012-06-20 08:00:12,681][WARN][transport.netty] > > > [dmz14\_data\_node] Message not fully read (response) for [188852] > > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > $4@799b0299, error [true], resetting > > > [2012-06-20 08:00:21,105][WARN][transport.netty] > > > [dmz14\_data\_node] Message not fully read (response) for [188862] > > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > $4@f75cb00, error [true], resetting > > > [2012-06-20 08:00:30,200][WARN][transport.netty] > > > [dmz14\_data\_node] Message not fully read (response) for [188873] > > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > $4@1ffcf6e7, error [true], resetting > > > [2012-06-20 08:00:39,544][WARN][transport.netty] > > > [dmz14\_data\_node] Message not fully read (response) for [188884] > > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > $4@5892644d, error [true], resetting > > > [2012-06-20 08:00:49,076][WARN][transport.netty] > > > [dmz14\_data\_node] Message not fully read (response) for [188896] > > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > $4@557e75f5, error [true], resetting > > > [2012-06-20 08:00:59,341][WARN][transport.netty] > > > [dmz14\_data\_node] Message not fully read (response) for [188908] > > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > $4@21dadca4, error [true], resetting > > > [2012-06-20 08:01:09,434][WARN][transport.netty] > > > [dmz14\_data\_node] Message not fully read (response) for [188920] > > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > $4@10dbd700, error [true], resetting > > > [2012-06-20 08:01:17,249][WARN][transport.netty] > > > [dmz14\_data\_node] Message not fully read (response) for [188930] > > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > $4@35cbd34b, error [true], resetting > > > [2012-06-20 08:01:23,942][WARN][transport.netty] > > > [dmz14\_data\_node] Message not fully read (response) for [188938] > > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > $4@a2a8fed, error [true], resetting > > > [2012-06-20 08:01:30,307][WARN][transport.netty] > > > [dmz14\_data\_node] Message not fully read (response) for [188947] > > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > $4@62411fb4, error [true], resetting > > > [2012-06-20 08:01:39,292][WARN][transport.netty] > > > [dmz14\_data\_node] Message not fully read (response) for [188957] > > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > $4@36ea5908, error [true], resetting > > > [2012-06-20 10:08:20,126][INFO][node] > > > [dmz14\_data\_node] {0.19.2}[16084]: initializing ... > > > [2012-06-20 10:08:20,194][INFO][plugins] > > > [dmz14\_data\_node] loaded [es-river-rss-oe, river-rss, lang- > > > javascript], sites [head] > > > [2012-06-20 10:08:25,750][INFO][node] > > > [dmz14\_data\_node] {0.19.2}[16084]: initialized > > > [2012-06-20 10:08:25,750][INFO][node] > > > [dmz14\_data\_node] {0.19.2}[16084]: starting ... > > > [2012-06-20 10:08:26,176][INFO][transport] > > > [dmz14\_data\_node] bound\_address {inet[/0:0:0:0:0:0:0:0:9301]}, > > > publish\_address {inet[/192.168.150.14:9301]} > > > [2012-06-20 10:08:56,190][WARN][discovery] > > > [dmz14\_data\_node] waited for 30s and no initial state was set by the > > > discovery > > > [2012-06-20 10:08:56,191][INFO][discovery] > > > [dmz14\_data\_node] elasticsearch\_dmz/WXMk\_K-VSaulKWfPYcAHXQ > > > [2012-06-20 10:08:56,403][INFO][http] > > > [dmz14\_data\_node] bound\_address {inet[/0:0:0:0:0:0:0:0:8081]}, > > > publish\_address {inet[/192.168.150.14:8081]} > > > [2012-06-20 10:08:56,404][INFO][node] > > > [dmz14\_data\_node] {0.19.2}[16084]: started > > > > > I am working on ES for quite some time now and dealt with many issues > > > sometime by tuning my setting and analyzing logs to identify the > > > problem but this time I am clueless as there is no error in the logs. > > > > > Please suggest how can I hunt this down. ---
**Author:** ![jagdeep](https://avatars.discourse-cdn.com/v4/letter/j/ba8739/32.png) [@jagdeep](https://discuss.elastic.co/u/jagdeep)\ **Post date:** [June 24, 2012, 4:04pm UTC](https://discuss.elastic.co/t/unexpected-es-shut-down-nothing-there-in-logs-to-identify-the-problem/8166/5 "2012-06-24T16:04:04Z")
Hey Otis thanks.. I have dealt with OOM earlier, this does not seem oom because cluster is completely ideal when this is happening. I am keeping a track on JVM crash as suggested by Shay. On Jun 23, 12:50 am, Otis Gospodnetic [otis.gospodne...@gmail.com](mailto:otis.gospodne...@gmail.com) wrote: > This smells like OOM (which doesn't always seem to get logged). > > ## Otis > > Search Analytics -[Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) > Scalable Performance Monitoring -[Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) > > On Thursday, June 21, 2012 3:50:22 AM UTC-4, jagdeep wrote: > > > Thanks Shay for the response. > > > 1. I stopped using service wrapper to track this problem but its still > > persisting. > > 2. Will definitely check JVM crash files. > > 3. It dumps the memory before getting crashed, heap dump of size 21-26 > > GB. I can understand the heap dump at the time of faceting or > > indexing but at this time there is no activity what so ever. > > > Any pointer towards this heap dump? > > > On Jun 21, 12:33 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > > > I assume you look at the logs and oyu don't see any "shutdown" / closing > > > messages. This can happen for several reasons (I've see all of those): > > > > - an external process is calling "kill -9": Tricky to find, you need to > > > find the process that does that... > > > - the service wrapper (if you use it) kills the process. Check the > > > service > > > wrapper logs. > > > - The JVM crashed. Check for jvm crash logs (hs\_err files). > > > > On Wed, Jun 20, 2012 at 12:53 PM, jagdeep [reach.jagd...@gmail.com](mailto:reach.jagd...@gmail.com) > > > wrote: > > > > > > > I am facing a very unexpected shutdown of ES data node. It dumps the > > > > memory and automatically getting shut(even console is getting closed). > > > > I have already dealt with memory related issues but this one is pretty > > > > weird because nothing is there in the logs. > > > > > Server configuration > > > > > OS - Windows server 2008 RT standard > > > > CPU - 24 cores > > > > RAM - 32 GB > > > > > ES configuration > > > > > Data node 1 - ES\_MIN\_MEM = 1024m > > > > ES\_MAX\_MEM=20g > > > > MAster node 1 - ES\_MIN\_MEM = 1024m > > > > ES\_MAX\_MEM=5g > > > > > Cluster state is { > > > > cluster\_name: elasticsearch\_\*\*\*\* > > > > status: green > > > > timed\_out: false > > > > number\_of\_nodes: 2 > > > > number\_of\_data\_nodes: 1 > > > > active\_primary\_shards: 330 > > > > active\_shards: 330 > > > > relocating\_shards: 0 > > > > initializing\_shards: 0 > > > > unassigned\_shards: 0 > > > > } > > > > > Data node state > > > > > { > > > > name: dmz14\_data\_node > > > > transport\_address: inet[/_ **.** _.\ ***.** :9301] > > > > hostname: \*\*\*\*\*\*\*\*\*\* > > > > attributes: { > > > > tag: \*\*\*\*\*\*\*\*\*\*\*\* > > > > master: false > > > > } > > > > indices: { > > > > store: { > > > > size: 21.2gb > > > > size\_in\_bytes: 22845246770 > > > > } > > > > docs: { > > > > count: 2525295 > > > > deleted: 551707 > > > > } > > > > indexing: { > > > > index\_total: 1207877 > > > > index\_time: 21.1m > > > > index\_time\_in\_millis: 1266718 > > > > index\_current: 0 > > > > delete\_total: 0 > > > > delete\_time: 0s > > > > delete\_time\_in\_millis: 0 > > > > delete\_current: 0 > > > > } > > > > get: { > > > > total: 0 > > > > time: 0s > > > > time\_in\_millis: 0 > > > > exists\_total: 0 > > > > exists\_time: 0s > > > > exists\_time\_in\_millis: 0 > > > > missing\_total: 0 > > > > missing\_time: 0s > > > > missing\_time\_in\_millis: 0 > > > > current: 0 > > > > } > > > > search: { > > > > query\_total: 8075 > > > > query\_time: 31.7s > > > > query\_time\_in\_millis: 31702 > > > > query\_current: 0 > > > > fetch\_total: 1242 > > > > fetch\_time: 907ms > > > > fetch\_time\_in\_millis: 907 > > > > fetch\_current: 0 > > > > } > > > > cache: { > > > > field\_evictions: 0 > > > > field\_size: 73.9mb > > > > field\_size\_in\_bytes: 77579988 > > > > filter\_count: 217 > > > > filter\_evictions: 0 > > > > filter\_size: 981.7kb > > > > filter\_size\_in\_bytes: 1005320 > > > > } > > > > merges: { > > > > current: 0 > > > > current\_docs: 0 > > > > current\_size: 0b > > > > current\_size\_in\_bytes: 0 > > > > total: 2073 > > > > total\_time: 28m > > > > total\_time\_in\_millis: 1681573 > > > > total\_docs: 5233813 > > > > total\_size: 20.6gb > > > > total\_size\_in\_bytes: 22176964666 > > > > } > > > > refresh: { > > > > total: 18970 > > > > total\_time: 22.8m > > > > total\_time\_in\_millis: 1373282 > > > > } > > > > flush: { > > > > total: 4072 > > > > total\_time: 1.6m > > > > total\_time\_in\_millis: 98095 > > > > } > > > > } > > > > } > > > > > This cluster works fine with search, indexing, facteing etc but after > > > > some irregular interval(between 24-48hrs) data node is getting shut. > > > > Surprisingly there is no activity on the cluster when this happens, i > > > > mean no search and indexing. Logs are also pretty much clear. > > > > Logs when system is crashed and after manual restart of the cluster > > > > are > > > > > [2012-06-20 08:00:04,678][WARN][transport.netty] > > > > [dmz14\_data\_node] Message not fully read (response) for [188842] > > > > handler > > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > > > $4@75f34e16, error [true], resetting > > > > [2012-06-20 08:00:12,681][WARN][transport.netty] > > > > [dmz14\_data\_node] Message not fully read (response) for [188852] > > > > handler > > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > > > $4@799b0299, error [true], resetting > > > > [2012-06-20 08:00:21,105][WARN][transport.netty] > > > > [dmz14\_data\_node] Message not fully read (response) for [188862] > > > > handler > > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > > > $4@f75cb00, error [true], resetting > > > > [2012-06-20 08:00:30,200][WARN][transport.netty] > > > > [dmz14\_data\_node] Message not fully read (response) for [188873] > > > > handler > > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > > > $4@1ffcf6e7, error [true], resetting > > > > [2012-06-20 08:00:39,544][WARN][transport.netty] > > > > [dmz14\_data\_node] Message not fully read (response) for [188884] > > > > handler > > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > > > $4@5892644d, error [true], resetting > > > > [2012-06-20 08:00:49,076][WARN][transport.netty] > > > > [dmz14\_data\_node] Message not fully read (response) for [188896] > > > > handler > > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > > > $4@557e75f5, error [true], resetting > > > > [2012-06-20 08:00:59,341][WARN][transport.netty] > > > > [dmz14\_data\_node] Message not fully read (response) for [188908] > > > > handler > > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > > > $4@21dadca4, error [true], resetting > > > > [2012-06-20 08:01:09,434][WARN][transport.netty] > > > > [dmz14\_data\_node] Message not fully read (response) for [188920] > > > > handler > > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > > > $4@10dbd700, error [true], resetting > > > > [2012-06-20 08:01:17,249][WARN][transport.netty] > > > > [dmz14\_data\_node] Message not fully read (response) for [188930] > > > > handler > > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > > > $4@35cbd34b, error [true], resetting > > > > [2012-06-20 08:01:23,942][WARN][transport.netty] > > > > [dmz14\_data\_node] Message not fully read (response) for [188938] > > > > handler > > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > > > $4@a2a8fed, error [true], resetting > > > > [2012-06-20 08:01:30,307][WARN][transport.netty] > > > > [dmz14\_data\_node] Message not fully read (response) for [188947] > > > > handler > > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > > > $4@62411fb4, error [true], resetting > > > > [2012-06-20 08:01:39,292][WARN][transport.netty] > > > > [dmz14\_data\_node] Message not fully read (response) for [188957] > > > > handler > > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > > > $4@36ea5908, error [true], resetting > > > > [2012-06-20 10:08:20,126][INFO][node] > > > > [dmz14\_data\_node] {0.19.2}[16084]: initializing ... > > > > [2012-06-20 10:08:20,194][INFO][plugins] > > > > [dmz14\_data\_node] loaded [es-river-rss-oe, river-rss, lang- > > > > javascript], sites [head] > > > > [2012-06-20 10:08:25,750][INFO][node] > > > > [dmz14\_data\_node] {0.19.2}[16084]: initialized > > > > [2012-06-20 10:08:25,750][INFO][node] > > > > [dmz14\_data\_node] {0.19.2}[16084]: starting ... > > > > [2012-06-20 10:08:26,176][INFO][transport] > > > > [dmz14\_data\_node] bound\_address {inet[/0:0:0:0:0:0:0:0:9301]}, > > > > publish\_address {inet[/192.168.150.14:9301]} > > > > [2012-06-20 10:08:56,190][WARN][discovery] > > > > [dmz14\_data\_node] waited for 30s and no initial state was set by the > > > > discovery > > > > [2012-06-20 10:08:56,191][INFO][discovery] > > > > [dmz14\_data\_node] elasticsearch\_dmz/WXMk\_K-VSaulKWfPYcAHXQ > > > > [2012-06-20 10:08:56,403][INFO][http] > > > > [dmz14\_data\_node] bound\_address {inet[/0:0:0:0:0:0:0:0:8081]}, > > > > publish\_address {inet[/192.168.150.14:8081]} > > > > [2012-06-20 10:08:56,404][INFO][node] > > > > [dmz14\_data\_node] {0.19.2}[16084]: started > > > > > I am working on ES for quite some time now and dealt with many issues > > > > sometime by tuning my setting and analyzing logs to identify the > > > > problem but this time I am clueless as there is no error in the logs. > > > > > Please suggest how can I hunt this down. ---
**Author:** ![jagdeep](https://avatars.discourse-cdn.com/v4/letter/j/ba8739/32.png) [@jagdeep](https://discuss.elastic.co/u/jagdeep)\ **Post date:** [June 24, 2012, 4:17pm UTC](https://discuss.elastic.co/t/unexpected-es-shut-down-nothing-there-in-logs-to-identify-the-problem/8166/6 "2012-06-24T16:17:48Z")
Hi Shay, I have observed one more thing.. Earlier this crash was happening after 24-48 hours. 3 indexes had replica count of 1 whereas other indexes had replica count of 0. I configured only one data node for this cluster hence cluster state was yellow. I was ignoring this for a while. 3 days before I changed replica count to 0 and cluster state turned green. Its been 72 hours and system is still up and running. I am keeping a close eye on this cluster. As per my understanding this is normal and expected behavior of ES and dont sense yellow state or replica count behind this crash. Please share any pointer towards this observation. Thanks and regards Jagdeep On Jun 21, 12:33 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > I assume you look at the logs and oyu don't see any "shutdown" / closing > messages. This can happen for several reasons (I've see all of those): > > - an external process is calling "kill -9": Tricky to find, you need to > find the process that does that... > - the service wrapper (if you use it) kills the process. Check the service > wrapper logs. > - The JVM crashed. Check for jvm crash logs (hs\_err files). > > On Wed, Jun 20, 2012 at 12:53 PM, jagdeep [reach.jagd...@gmail.com](mailto:reach.jagd...@gmail.com) wrote: > > > I am facing a very unexpected shutdown of ES data node. It dumps the > > memory and automatically getting shut(even console is getting closed). > > I have already dealt with memory related issues but this one is pretty > > weird because nothing is there in the logs. > > > Server configuration > > > OS - Windows server 2008 RT standard > > CPU - 24 cores > > RAM - 32 GB > > > ES configuration > > > Data node 1 - ES\_MIN\_MEM = 1024m > > ES\_MAX\_MEM=20g > > MAster node 1 - ES\_MIN\_MEM = 1024m > > ES\_MAX\_MEM=5g > > > Cluster state is { > > cluster\_name: elasticsearch\_\*\*\*\* > > status: green > > timed\_out: false > > number\_of\_nodes: 2 > > number\_of\_data\_nodes: 1 > > active\_primary\_shards: 330 > > active\_shards: 330 > > relocating\_shards: 0 > > initializing\_shards: 0 > > unassigned\_shards: 0 > > } > > > Data node state > > > { > > name: dmz14\_data\_node > > transport\_address: inet[/_ **.** _.\ ***.** :9301] > > hostname: \*\*\*\*\*\*\*\*\*\* > > attributes: { > > tag: \*\*\*\*\*\*\*\*\*\*\*\* > > master: false > > } > > indices: { > > store: { > > size: 21.2gb > > size\_in\_bytes: 22845246770 > > } > > docs: { > > count: 2525295 > > deleted: 551707 > > } > > indexing: { > > index\_total: 1207877 > > index\_time: 21.1m > > index\_time\_in\_millis: 1266718 > > index\_current: 0 > > delete\_total: 0 > > delete\_time: 0s > > delete\_time\_in\_millis: 0 > > delete\_current: 0 > > } > > get: { > > total: 0 > > time: 0s > > time\_in\_millis: 0 > > exists\_total: 0 > > exists\_time: 0s > > exists\_time\_in\_millis: 0 > > missing\_total: 0 > > missing\_time: 0s > > missing\_time\_in\_millis: 0 > > current: 0 > > } > > search: { > > query\_total: 8075 > > query\_time: 31.7s > > query\_time\_in\_millis: 31702 > > query\_current: 0 > > fetch\_total: 1242 > > fetch\_time: 907ms > > fetch\_time\_in\_millis: 907 > > fetch\_current: 0 > > } > > cache: { > > field\_evictions: 0 > > field\_size: 73.9mb > > field\_size\_in\_bytes: 77579988 > > filter\_count: 217 > > filter\_evictions: 0 > > filter\_size: 981.7kb > > filter\_size\_in\_bytes: 1005320 > > } > > merges: { > > current: 0 > > current\_docs: 0 > > current\_size: 0b > > current\_size\_in\_bytes: 0 > > total: 2073 > > total\_time: 28m > > total\_time\_in\_millis: 1681573 > > total\_docs: 5233813 > > total\_size: 20.6gb > > total\_size\_in\_bytes: 22176964666 > > } > > refresh: { > > total: 18970 > > total\_time: 22.8m > > total\_time\_in\_millis: 1373282 > > } > > flush: { > > total: 4072 > > total\_time: 1.6m > > total\_time\_in\_millis: 98095 > > } > > } > > } > > > This cluster works fine with search, indexing, facteing etc but after > > some irregular interval(between 24-48hrs) data node is getting shut. > > Surprisingly there is no activity on the cluster when this happens, i > > mean no search and indexing. Logs are also pretty much clear. > > Logs when system is crashed and after manual restart of the cluster > > are > > > [2012-06-20 08:00:04,678][WARN][transport.netty] > > [dmz14\_data\_node] Message not fully read (response) for [188842] > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > $4@75f34e16, error [true], resetting > > [2012-06-20 08:00:12,681][WARN][transport.netty] > > [dmz14\_data\_node] Message not fully read (response) for [188852] > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > $4@799b0299, error [true], resetting > > [2012-06-20 08:00:21,105][WARN][transport.netty] > > [dmz14\_data\_node] Message not fully read (response) for [188862] > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > $4@f75cb00, error [true], resetting > > [2012-06-20 08:00:30,200][WARN][transport.netty] > > [dmz14\_data\_node] Message not fully read (response) for [188873] > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > $4@1ffcf6e7, error [true], resetting > > [2012-06-20 08:00:39,544][WARN][transport.netty] > > [dmz14\_data\_node] Message not fully read (response) for [188884] > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > $4@5892644d, error [true], resetting > > [2012-06-20 08:00:49,076][WARN][transport.netty] > > [dmz14\_data\_node] Message not fully read (response) for [188896] > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > $4@557e75f5, error [true], resetting > > [2012-06-20 08:00:59,341][WARN][transport.netty] > > [dmz14\_data\_node] Message not fully read (response) for [188908] > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > $4@21dadca4, error [true], resetting > > [2012-06-20 08:01:09,434][WARN][transport.netty] > > [dmz14\_data\_node] Message not fully read (response) for [188920] > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > $4@10dbd700, error [true], resetting > > [2012-06-20 08:01:17,249][WARN][transport.netty] > > [dmz14\_data\_node] Message not fully read (response) for [188930] > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > $4@35cbd34b, error [true], resetting > > [2012-06-20 08:01:23,942][WARN][transport.netty] > > [dmz14\_data\_node] Message not fully read (response) for [188938] > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > $4@a2a8fed, error [true], resetting > > [2012-06-20 08:01:30,307][WARN][transport.netty] > > [dmz14\_data\_node] Message not fully read (response) for [188947] > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > $4@62411fb4, error [true], resetting > > [2012-06-20 08:01:39,292][WARN][transport.netty] > > [dmz14\_data\_node] Message not fully read (response) for [188957] > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > $4@36ea5908, error [true], resetting > > [2012-06-20 10:08:20,126][INFO][node] > > [dmz14\_data\_node] {0.19.2}[16084]: initializing ... > > [2012-06-20 10:08:20,194][INFO][plugins] > > [dmz14\_data\_node] loaded [es-river-rss-oe, river-rss, lang- > > javascript], sites [head] > > [2012-06-20 10:08:25,750][INFO][node] > > [dmz14\_data\_node] {0.19.2}[16084]: initialized > > [2012-06-20 10:08:25,750][INFO][node] > > [dmz14\_data\_node] {0.19.2}[16084]: starting ... > > [2012-06-20 10:08:26,176][INFO][transport] > > [dmz14\_data\_node] bound\_address {inet[/0:0:0:0:0:0:0:0:9301]}, > > publish\_address {inet[/192.168.150.14:9301]} > > [2012-06-20 10:08:56,190][WARN][discovery] > > [dmz14\_data\_node] waited for 30s and no initial state was set by the > > discovery > > [2012-06-20 10:08:56,191][INFO][discovery] > > [dmz14\_data\_node] elasticsearch\_dmz/WXMk\_K-VSaulKWfPYcAHXQ > > [2012-06-20 10:08:56,403][INFO][http] > > [dmz14\_data\_node] bound\_address {inet[/0:0:0:0:0:0:0:0:8081]}, > > publish\_address {inet[/192.168.150.14:8081]} > > [2012-06-20 10:08:56,404][INFO][node] > > [dmz14\_data\_node] {0.19.2}[16084]: started > > > I am working on ES for quite some time now and dealt with many issues > > sometime by tuning my setting and analyzing logs to identify the > > problem but this time I am clueless as there is no error in the logs. > > > Please suggest how can I hunt this down. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [June 25, 2012, 12:06pm UTC](https://discuss.elastic.co/t/unexpected-es-shut-down-nothing-there-in-logs-to-identify-the-problem/8166/7 "2012-06-25T12:06:59Z")
Heya, no, yellow state would not lead to a crash of elasticsearch under no circumstances... On Sun, Jun 24, 2012 at 6:17 PM, jagdeep [reach.jagdeep@gmail.com](mailto:reach.jagdeep@gmail.com) wrote: > Hi Shay, > > I have observed one more thing.. Earlier this crash was happening > after 24-48 hours. 3 indexes had replica count of 1 whereas other > indexes had replica count of 0. I configured only one data node for > this cluster hence cluster state was yellow. I was ignoring this for a > while. > 3 days before I changed replica count to 0 and cluster state turned > green. Its been 72 hours and system is still up and running. I am > keeping a close eye on this cluster. > As per my understanding this is normal and expected behavior of ES and > dont sense yellow state or replica count behind this crash. Please > share any pointer towards this observation. > > Thanks and regards > Jagdeep > > On Jun 21, 12:33 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > I assume you look at the logs and oyu don't see any "shutdown" / closing > > messages. This can happen for several reasons (I've see all of those): > > > > - an external process is calling "kill -9": Tricky to find, you need to > > find the process that does that... > > - the service wrapper (if you use it) kills the process. Check the > > service > > wrapper logs. > > - The JVM crashed. Check for jvm crash logs (hs\_err files). > > > > On Wed, Jun 20, 2012 at 12:53 PM, jagdeep [reach.jagd...@gmail.com](mailto:reach.jagd...@gmail.com) > > wrote: > > > > > I am facing a very unexpected shutdown of ES data node. It dumps the > > > memory and automatically getting shut(even console is getting closed). > > > I have already dealt with memory related issues but this one is pretty > > > weird because nothing is there in the logs. > > > > > Server configuration > > > > > OS - Windows server 2008 RT standard > > > CPU - 24 cores > > > RAM - 32 GB > > > > > ES configuration > > > > > Data node 1 - ES\_MIN\_MEM = 1024m > > > ES\_MAX\_MEM=20g > > > MAster node 1 - ES\_MIN\_MEM = 1024m > > > ES\_MAX\_MEM=5g > > > > > Cluster state is { > > > cluster\_name: elasticsearch\_\*\*\*\* > > > status: green > > > timed\_out: false > > > number\_of\_nodes: 2 > > > number\_of\_data\_nodes: 1 > > > active\_primary\_shards: 330 > > > active\_shards: 330 > > > relocating\_shards: 0 > > > initializing\_shards: 0 > > > unassigned\_shards: 0 > > > } > > > > > Data node state > > > > > { > > > name: dmz14\_data\_node > > > transport\_address: inet[/_ **.** _.\ ***.** :9301] > > > hostname: \*\*\*\*\*\*\*\*\*\* > > > attributes: { > > > tag: \*\*\*\*\*\*\*\*\*\*\*\* > > > master: false > > > } > > > indices: { > > > store: { > > > size: 21.2gb > > > size\_in\_bytes: 22845246770 > > > } > > > docs: { > > > count: 2525295 > > > deleted: 551707 > > > } > > > indexing: { > > > index\_total: 1207877 > > > index\_time: 21.1m > > > index\_time\_in\_millis: 1266718 > > > index\_current: 0 > > > delete\_total: 0 > > > delete\_time: 0s > > > delete\_time\_in\_millis: 0 > > > delete\_current: 0 > > > } > > > get: { > > > total: 0 > > > time: 0s > > > time\_in\_millis: 0 > > > exists\_total: 0 > > > exists\_time: 0s > > > exists\_time\_in\_millis: 0 > > > missing\_total: 0 > > > missing\_time: 0s > > > missing\_time\_in\_millis: 0 > > > current: 0 > > > } > > > search: { > > > query\_total: 8075 > > > query\_time: 31.7s > > > query\_time\_in\_millis: 31702 > > > query\_current: 0 > > > fetch\_total: 1242 > > > fetch\_time: 907ms > > > fetch\_time\_in\_millis: 907 > > > fetch\_current: 0 > > > } > > > cache: { > > > field\_evictions: 0 > > > field\_size: 73.9mb > > > field\_size\_in\_bytes: 77579988 > > > filter\_count: 217 > > > filter\_evictions: 0 > > > filter\_size: 981.7kb > > > filter\_size\_in\_bytes: 1005320 > > > } > > > merges: { > > > current: 0 > > > current\_docs: 0 > > > current\_size: 0b > > > current\_size\_in\_bytes: 0 > > > total: 2073 > > > total\_time: 28m > > > total\_time\_in\_millis: 1681573 > > > total\_docs: 5233813 > > > total\_size: 20.6gb > > > total\_size\_in\_bytes: 22176964666 > > > } > > > refresh: { > > > total: 18970 > > > total\_time: 22.8m > > > total\_time\_in\_millis: 1373282 > > > } > > > flush: { > > > total: 4072 > > > total\_time: 1.6m > > > total\_time\_in\_millis: 98095 > > > } > > > } > > > } > > > > > This cluster works fine with search, indexing, facteing etc but after > > > some irregular interval(between 24-48hrs) data node is getting shut. > > > Surprisingly there is no activity on the cluster when this happens, i > > > mean no search and indexing. Logs are also pretty much clear. > > > Logs when system is crashed and after manual restart of the cluster > > > are > > > > > [2012-06-20 08:00:04,678][WARN][transport.netty] > > > [dmz14\_data\_node] Message not fully read (response) for [188842] > > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > $4@75f34e16, error [true], resetting > > > [2012-06-20 08:00:12,681][WARN][transport.netty] > > > [dmz14\_data\_node] Message not fully read (response) for [188852] > > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > $4@799b0299, error [true], resetting > > > [2012-06-20 08:00:21,105][WARN][transport.netty] > > > [dmz14\_data\_node] Message not fully read (response) for [188862] > > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > $4@f75cb00, error [true], resetting > > > [2012-06-20 08:00:30,200][WARN][transport.netty] > > > [dmz14\_data\_node] Message not fully read (response) for [188873] > > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > $4@1ffcf6e7, error [true], resetting > > > [2012-06-20 08:00:39,544][WARN][transport.netty] > > > [dmz14\_data\_node] Message not fully read (response) for [188884] > > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > $4@5892644d, error [true], resetting > > > [2012-06-20 08:00:49,076][WARN][transport.netty] > > > [dmz14\_data\_node] Message not fully read (response) for [188896] > > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > $4@557e75f5, error [true], resetting > > > [2012-06-20 08:00:59,341][WARN][transport.netty] > > > [dmz14\_data\_node] Message not fully read (response) for [188908] > > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > $4@21dadca4, error [true], resetting > > > [2012-06-20 08:01:09,434][WARN][transport.netty] > > > [dmz14\_data\_node] Message not fully read (response) for [188920] > > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > $4@10dbd700, error [true], resetting > > > [2012-06-20 08:01:17,249][WARN][transport.netty] > > > [dmz14\_data\_node] Message not fully read (response) for [188930] > > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > $4@35cbd34b, error [true], resetting > > > [2012-06-20 08:01:23,942][WARN][transport.netty] > > > [dmz14\_data\_node] Message not fully read (response) for [188938] > > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > $4@a2a8fed, error [true], resetting > > > [2012-06-20 08:01:30,307][WARN][transport.netty] > > > [dmz14\_data\_node] Message not fully read (response) for [188947] > > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > $4@62411fb4, error [true], resetting > > > [2012-06-20 08:01:39,292][WARN][transport.netty] > > > [dmz14\_data\_node] Message not fully read (response) for [188957] > > > handler > > org.elasticsearch.action.support.master.TransportMasterNodeOperationAction > > > > $4@36ea5908, error [true], resetting > > > [2012-06-20 10:08:20,126][INFO][node] > > > [dmz14\_data\_node] {0.19.2}[16084]: initializing ... > > > [2012-06-20 10:08:20,194][INFO][plugins] > > > [dmz14\_data\_node] loaded [es-river-rss-oe, river-rss, lang- > > > javascript], sites [head] > > > [2012-06-20 10:08:25,750][INFO][node] > > > [dmz14\_data\_node] {0.19.2}[16084]: initialized > > > [2012-06-20 10:08:25,750][INFO][node] > > > [dmz14\_data\_node] {0.19.2}[16084]: starting ... > > > [2012-06-20 10:08:26,176][INFO][transport] > > > [dmz14\_data\_node] bound\_address {inet[/0:0:0:0:0:0:0:0:9301]}, > > > publish\_address {inet[/192.168.150.14:9301]} > > > [2012-06-20 10:08:56,190][WARN][discovery] > > > [dmz14\_data\_node] waited for 30s and no initial state was set by the > > > discovery > > > [2012-06-20 10:08:56,191][INFO][discovery] > > > [dmz14\_data\_node] elasticsearch\_dmz/WXMk\_K-VSaulKWfPYcAHXQ > > > [2012-06-20 10:08:56,403][INFO][http] > > > [dmz14\_data\_node] bound\_address {inet[/0:0:0:0:0:0:0:0:8081]}, > > > publish\_address {inet[/192.168.150.14:8081]} > > > [2012-06-20 10:08:56,404][INFO][node] > > > [dmz14\_data\_node] {0.19.2}[16084]: started > > > > > I am working on ES for quite some time now and dealt with many issues > > > sometime by tuning my setting and analyzing logs to identify the > > > problem but this time I am clueless as there is no error in the logs. > > > > > Please suggest how can I hunt this down. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:22am UTC](https://discuss.elastic.co/t/unexpected-es-shut-down-nothing-there-in-logs-to-identify-the-problem/8166/8 "2017-07-06T03:22:43Z")