# No alive nodes found in your cluster all of sudden
**URL:** https://discuss.elastic.co/t/no-alive-nodes-found-in-your-cluster-all-of-sudden/106193
**Category:** Elasticsearch
**Created:** [November 2, 2017, 12:34pm UTC](https://discuss.elastic.co/t/no-alive-nodes-found-in-your-cluster-all-of-sudden/106193 "2017-11-02T12:34:08Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@ManishChughAtSmartsh](https://discuss.elastic.co/u/ManishChughAtSmartsh)
#### Post date: [November 2, 2017, 12:34pm UTC](https://discuss.elastic.co/t/no-alive-nodes-found-in-your-cluster-all-of-sudden/106193/1 "2017-11-02T12:34:08Z")
Hi,
I am running into a problem with ES service. Some cron jobs are running on my server and crawling data from other applications. Then the data is indexed in elastic.
The problem is Elastic service failed any time, and doesn't index data. Saying error
"no alive nodes found in your cluster".
I had to manually restart the Elasticsearch service, then it back up to work.
**Found Log at /var/log/messages:**
AHV-ID-8121 kernel: Out of memory: Kill process 36993 (java) score 143 or sacrifice child
AHV-ID-8121 kernel: Killed process 36993 (java) total-vm:6090372kB, anon-rss:2479712kB, file-rss:0kB, shmem-rss:0kB
AHV-ID-8121 systemd: elasticsearch.service: main process exited, code=killed, status=9/KILL
AHV-ID-8121 systemd: Unit elasticsearch.service entered failed state.
AHV-ID-8121 systemd: elasticsearch.service failed.
AHV-ID-8121 dockerd: time="2017-11-02T12:23:41.973997500Z" level=info msg="libcontainerd: new containerd process, pid: 38366"
AHV-ID-8121 systemd-logind: Removed session 143489.
(FYI: I am using very basic Elastic configuration)
What can be the cause? Any help Please!
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [November 2, 2017, 12:56pm UTC](https://discuss.elastic.co/t/no-alive-nodes-found-in-your-cluster-all-of-sudden/106193/2 "2017-11-02T12:56:48Z")
What is the full output from the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/cluster-stats.html)?
---
### Author:  [@ManishChughAtSmartsh](https://discuss.elastic.co/u/ManishChughAtSmartsh)
#### Post date: [November 2, 2017, 1:03pm UTC](https://discuss.elastic.co/t/no-alive-nodes-found-in-your-cluster-all-of-sudden/106193/3 "2017-11-02T13:03:56Z")
{
"\_nodes" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"cluster\_name" : "buythenews\_cluster",
"timestamp" : 1509627687303,
"status" : "yellow",
"indices" : {
"count" : 1,
"shards" : {
"total" : 5,
"primaries" : 5,
"replication" : 0.0,
"index" : {
"shards" : {
"min" : 5,
"max" : 5,
"avg" : 5.0
},
"primaries" : {
"min" : 5,
"max" : 5,
"avg" : 5.0
},
"replication" : {
"min" : 0.0,
"max" : 0.0,
"avg" : 0.0
}
}
},
"docs" : {
"count" : 110592,
"deleted" : 4
},
"store" : {
"size" : "168.6mb",
"size\_in\_bytes" : 176882811,
"throttle\_time" : "0s",
"throttle\_time\_in\_millis" : 0
},
"fielddata" : {
"memory\_size" : "0b",
"memory\_size\_in\_bytes" : 0,
"evictions" : 0
},
"query\_cache" : {
"memory\_size" : "0b",
"memory\_size\_in\_bytes" : 0,
"total\_count" : 0,
"hit\_count" : 0,
"miss\_count" : 0,
"cache\_size" : 0,
"cache\_count" : 0,
"evictions" : 0
},
"completion" : {
"size" : "0b",
"size\_in\_bytes" : 0
},
"segments" : {
"count" : 30,
"memory" : "978kb",
"memory\_in\_bytes" : 1001525,
"terms\_memory" : "879.4kb",
"terms\_memory\_in\_bytes" : 900523,
"stored\_fields\_memory" : "29.3kb",
"stored\_fields\_memory\_in\_bytes" : 30072,
"term\_vectors\_memory" : "0b",
"term\_vectors\_memory\_in\_bytes" : 0,
"norms\_memory" : "20.6kb",
"norms\_memory\_in\_bytes" : 21120,
"points\_memory" : "1.4kb",
"points\_memory\_in\_bytes" : 1450,
"doc\_values\_memory" : "47.2kb",
"doc\_values\_memory\_in\_bytes" : 48360,
"index\_writer\_memory" : "0b",
"index\_writer\_memory\_in\_bytes" : 0,
"version\_map\_memory" : "0b",
"version\_map\_memory\_in\_bytes" : 0,
"fixed\_bit\_set" : "0b",
"fixed\_bit\_set\_memory\_in\_bytes" : 0,
"max\_unsafe\_auto\_id\_timestamp" : -1,
"file\_sizes" : { }
}
},
"nodes" : {
"count" : {
"total" : 1,
"data" : 1,
"coordinating\_only" : 0,
"master" : 1,
"ingest" : 1
},
"versions" : [
"5.6.3"
],
"os" : {
"available\_processors" : 4,
"allocated\_processors" : 4,
"names" : [
{
"name" : "Linux",
"count" : 1
}
],
"mem" : {
"total" : "15.5gb",
"total\_in\_bytes" : 16650620928,
"free" : "1.6gb",
"free\_in\_bytes" : 1802637312,
"used" : "13.8gb",
"used\_in\_bytes" : 14847983616,
"free\_percent" : 11,
"used\_percent" : 89
}
},
"process" : {
"cpu" : {
"percent" : 24
},
"open\_file\_descriptors" : {
"min" : 183,
"max" : 183,
"avg" : 183
}
},
"jvm" : {
"max\_uptime" : "23.4s",
"max\_uptime\_in\_millis" : 23441,
"versions" : [
{
"version" : "1.8.0\_141",
"vm\_name" : "OpenJDK 64-Bit Server VM",
"vm\_version" : "25.141-b16",
"vm\_vendor" : "Oracle Corporation",
"count" : 1
}
],
"mem" : {
"heap\_used" : "278.1mb",
"heap\_used\_in\_bytes" : 291618768,
"heap\_max" : "1.9gb",
"heap\_max\_in\_bytes" : 2112618496
},
"threads" : 44
},
"fs" : {
"total" : "38.4gb",
"total\_in\_bytes" : 41293721600,
"free" : "34.2gb",
"free\_in\_bytes" : 36742766592,
"available" : "34.2gb",
"available\_in\_bytes" : 36742766592
},
"plugins" : [],
"network\_types" : {
"transport\_types" : {
"netty4" : 1
},
"http\_types" : {
"netty4" : 1
}
}
}
}
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [November 2, 2017, 1:06pm UTC](https://discuss.elastic.co/t/no-alive-nodes-found-in-your-cluster-all-of-sudden/106193/4 "2017-11-02T13:06:01Z")
That looks OK. Is there anything in the Elasticsearch logs?
---
### Author:  [@ManishChughAtSmartsh](https://discuss.elastic.co/u/ManishChughAtSmartsh)
#### Post date: [November 2, 2017, 2:00pm UTC](https://discuss.elastic.co/t/no-alive-nodes-found-in-your-cluster-all-of-sudden/106193/5 "2017-11-02T14:00:03Z")
Only following that i also added already:
In /var/log/messages:
AHV-ID-8121 kernel: Out of memory: Kill process 36993 (java) score 143 or sacrifice child
AHV-ID-8121 kernel: Killed process 36993 (java) total-vm:6090372kB, anon-rss:2479712kB, file-rss:0kB, shmem-rss:0kB
AHV-ID-8121 systemd: elasticsearch.service: main process exited, code=killed, status=9/KILL
AHV-ID-8121 systemd: Unit elasticsearch.service entered failed state.
AHV-ID-8121 systemd: elasticsearch.service failed.
AHV-ID-8121 dockerd: time="2017-11-02T12:23:41.973997500Z" level=info msg="libcontainerd: new containerd process, pid: 38366"
AHV-ID-8121 systemd-logind: Removed session 143489.
Can you please suggest me to where i can find some specific log for you? Thanks
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [November 2, 2017, 2:14pm UTC](https://discuss.elastic.co/t/no-alive-nodes-found-in-your-cluster-all-of-sudden/106193/6 "2017-11-02T14:14:51Z")
That will depend on how you installed it and which OS you are using.
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [November 2, 2017, 2:17pm UTC](https://discuss.elastic.co/t/no-alive-nodes-found-in-your-cluster-all-of-sudden/106193/7 "2017-11-02T14:17:04Z")
Looks like an OOM killer process or something like this, no?
> kernel: Out of memory: Kill process 36993 (java) score 143 or sacrifice child
---
### Author:  [@ManishChughAtSmartsh](https://discuss.elastic.co/u/ManishChughAtSmartsh)
#### Post date: [November 3, 2017, 9:47am UTC](https://discuss.elastic.co/t/no-alive-nodes-found-in-your-cluster-all-of-sudden/106193/8 "2017-11-03T09:47:09Z")
Hello,
I just found on my server(Pleask server statistics), my RAM is fully occupied so may OOM killer kills java processes. I will increase my server Memory and then can check if its still crashing or not. Till Thanks for all your quick response.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 1, 2017, 9:47am UTC](https://discuss.elastic.co/t/no-alive-nodes-found-in-your-cluster-all-of-sudden/106193/9 "2017-12-01T09:47:24Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.