# Elasticsearch dies every other day

**URL:** <https://discuss.elastic.co/t/elasticsearch-dies-every-other-day/18627>\
**Category:** Elasticsearch\
**Created:** [July 12, 2014, 6:22pm UTC](https://discuss.elastic.co/t/elasticsearch-dies-every-other-day/18627 "2014-07-12T18:22:33Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![Klavs\_Klavsen](https://avatars.discourse-cdn.com/v4/letter/k/90db22/32.png) [@Klavs\_Klavsen](https://discuss.elastic.co/u/Klavs_Klavsen)\
**Post date:** [July 12, 2014, 6:22pm UTC](https://discuss.elastic.co/t/elasticsearch-dies-every-other-day/18627/1 "2014-07-12T18:22:33Z")

</div>

my cluster - running 1.1.2 with oracle java 1.7\_55 - dies several days a  
week  
one the nodes gets "disconnected"..  
this time one of them logged: observer timed out. notifying listener.  
several times  
I then have to restart them all - this time I had several who thought they  
were masters  
they are physical machines on the same LAN  
the machines (4) index between 70 and 130k documents (lines from logstash)  
per minute. I input into different indexes (not just logstash-$date)

after having a queue built up - they will easily index 260 to 290k/min -  
until the queue is emptied  
so they seem to have no resource shortage, but they somehow "get tired and  
die"- very often ☹

any ideas how I should proceed in debugging this issue? It seems to fit  
that EVERY time a node dies - I have garbage collection log entries. This  
time I had these on one node:  
[2014-07-12 17:12:25,132][INFO][monitor.jvm]  
[p-elasticlog02] [gc][young][194961][25004] duration [767ms], collections  
[1]/[1s], total [767ms]/[34.3m], memory [27.7gb]-\>[26.7gb]/[31.7gb],  
all\_pools {[young] [1gb]-\>[5.3mb]/[1.1  
gb]}{[survivor] [149.7mb]-\>[124.1mb]/[149.7mb]}{[old]  
[26.5gb]-\>[26.6gb]/[30.4gb]}  
[2014-07-12 17:12:44,929][INFO][monitor.jvm]  
[p-elasticlog02] [gc][young][194980][25007] duration [804ms], collections  
[1]/[1.1s], total [804ms]/[34.3m], memory [27.8gb]-\>[26.9gb]/[31.7gb],  
all\_pools {[young] [1gb]-\>[11.2mb]/[  
1.1gb]}{[survivor] [149.7mb]-\>[146.1mb]/[149.7mb]}{[old]  
[26.6gb]-\>[26.7gb]/[30.4gb]}  
[2014-07-12 17:14:57,032][INFO][monitor.jvm]  
[p-elasticlog02] [gc][young][195109][25035] duration [837ms], collections  
[1]/[1s], total [837ms]/[34.4m], memory [28.9gb]-\>[28.1gb]/[31.7gb],  
all\_pools {[young] [1gb]-\>[141.1mb]/[1  
.1gb]}{[survivor] [149.7mb]-\>[145.7mb]/[149.7mb]}{[old]  
[27.7gb]-\>[27.8gb]/[30.4gb]}  
[2014-07-12 17:16:17,016][INFO][monitor.jvm]  
[p-elasticlog02] [gc][young][195187][25053] duration [756ms], collections  
[1]/[1.4s], total [756ms]/[34.5m], memory [29.5gb]-\>[28.7gb]/[31.7gb],  
all\_pools {[young] [926.8mb]-\>[27.6m  
b]/[1.1gb]}{[survivor] [149.7mb]-\>[138.6mb]/[149.7mb]}{[old]  
[28.4gb]-\>[28.5gb]/[30.4gb]}  
[2014-07-12 17:18:57,313][WARN][monitor.jvm]  
[p-elasticlog02] [gc][young][195303][25075] duration [1.1s], collections  
[1]/[40.6s], total [1.1s]/[34.6m], memory [30.4gb]-\>[29.2gb]/[31.7gb],  
all\_pools {[young] [1.1gb]-\>[12.3mb]/  
[1.1gb]}{[survivor] [149.7mb]-\>[0b]/[149.7mb]}{[old]  
[29.1gb]-\>[29.2gb]/[30.4gb]}  
[2014-07-12 17:18:57,314][WARN][monitor.jvm]  
[p-elasticlog02] [gc][old][195303][53] duration [39.1s], collections  
[2]/[40.6s], total [39.1s]/[1.5m], memory [30.4gb]-\>[29.2gb]/[31.7gb],  
all\_pools {[young] [1.1gb]-\>[12.3mb]/[1.1  
gb]}{[survivor] [149.7mb]-\>[0b]/[149.7mb]}{[old]  
[29.1gb]-\>[29.2gb]/[30.4gb]}

I collect a lot of counters from elasticsearch (Using elasticsearch  
collector in diamond (graphite collector written in python)) - so I have  
data on the ES nodes.

my config is this:  
index.warmer.enabled: false  
cluster.name: elasticsearch  
node.name: "p-elasticlog02"  
node.master: true  
node.data: true  
action.disable\_delete\_all\_indices: true  
indices.memory.index\_buffer\_size: 50%  
indices.fielddata.cache.size: 30%  
index.refresh\_interval: 5s  
index.index\_concurrency: 16  
threadpool.search.type: fixed  
threadpool.search.size: 400  
threadpool.search.queue\_size: 900  
threadpool.bulk.type: fixed  
threadpool.bulk.size: 500  
threadpool.bulk.queue\_size: 900  
threadpool.index.type: fixed  
threadpool.index.size: 300  
threadpool.index.queue\_size: -1  
path.data: /var/lib/elasticsearch/  
bootstrap.mlockall: true  
network.publish\_host: $hostip  
discovery.zen.ping.multicast.enabled: false  
discovery.zen.ping.unicast.hosts: ["p-elasticlog01.example.idk",  
"p-elasticlog02.example.idk", "p-elasticlog03.example.idk",  
"p-elasticlog04.example.idk", "p-elasticlog05.example.idk"]

I have 24 cores in each machine (doing pretty much nothing) - so I was  
considering trying to switch to g1gc f.ex. - as I've read it should be  
better in some respects.. any input?

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f485903b-b34c-4d58-bbf0-9a60a3866afe%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f485903b-b34c-4d58-bbf0-9a60a3866afe%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 12, 2014, 11:22pm UTC](https://discuss.elastic.co/t/elasticsearch-dies-every-other-day/18627/2 "2014-07-12T23:22:10Z")

</div>

What's your heap size how much data do you have in your indexes?  
Take a look at settings like discovery.zen.minimum\_master\_nodes to reduce  
the multi-master problem you see, although ideally you want an odd number  
of nodes for that (eg 3 or 5).

Regards,  
Mark Walkom

Infrastructure Engineer  
Campaign Monitor  
email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)  
web: [www.campaignmonitor.com](http://www.campaignmonitor.com)

On 13 July 2014 04:22, Klavs Klavsen [klavs@enableit.dk](mailto:klavs@enableit.dk) wrote:

> my cluster - running 1.1.2 with oracle java 1.7\_55 - dies several days a  
> week  
> one the nodes gets "disconnected"..  
> this time one of them logged: observer timed out. notifying listener.  
> several times  
> I then have to restart them all - this time I had several who thought they  
> were masters  
> they are physical machines on the same LAN  
> the machines (4) index between 70 and 130k documents (lines from logstash)  
> per minute. I input into different indexes (not just logstash-$date)
> 
> after having a queue built up - they will easily index 260 to 290k/min -  
> until the queue is emptied  
> so they seem to have no resource shortage, but they somehow "get tired and  
> die"- very often ☹
> 
> any ideas how I should proceed in debugging this issue? It seems to fit  
> that EVERY time a node dies - I have garbage collection log entries. This  
> time I had these on one node:  
> [2014-07-12 17:12:25,132][INFO][monitor.jvm]  
> [p-elasticlog02] [gc][young][194961][25004] duration [767ms], collections  
> [1]/[1s], total [767ms]/[34.3m], memory [27.7gb]-\>[26.7gb]/[31.7gb],  
> all\_pools {[young] [1gb]-\>[5.3mb]/[1.1  
> gb]}{[survivor] [149.7mb]-\>[124.1mb]/[149.7mb]}{[old]  
> [26.5gb]-\>[26.6gb]/[30.4gb]}  
> [2014-07-12 17:12:44,929][INFO][monitor.jvm]  
> [p-elasticlog02] [gc][young][194980][25007] duration [804ms], collections  
> [1]/[1.1s], total [804ms]/[34.3m], memory [27.8gb]-\>[26.9gb]/[31.7gb],  
> all\_pools {[young] [1gb]-\>[11.2mb]/[  
> 1.1gb]}{[survivor] [149.7mb]-\>[146.1mb]/[149.7mb]}{[old]  
> [26.6gb]-\>[26.7gb]/[30.4gb]}  
> [2014-07-12 17:14:57,032][INFO][monitor.jvm]  
> [p-elasticlog02] [gc][young][195109][25035] duration [837ms], collections  
> [1]/[1s], total [837ms]/[34.4m], memory [28.9gb]-\>[28.1gb]/[31.7gb],  
> all\_pools {[young] [1gb]-\>[141.1mb]/[1  
> .1gb]}{[survivor] [149.7mb]-\>[145.7mb]/[149.7mb]}{[old]  
> [27.7gb]-\>[27.8gb]/[30.4gb]}  
> [2014-07-12 17:16:17,016][INFO][monitor.jvm]  
> [p-elasticlog02] [gc][young][195187][25053] duration [756ms], collections  
> [1]/[1.4s], total [756ms]/[34.5m], memory [29.5gb]-\>[28.7gb]/[31.7gb],  
> all\_pools {[young] [926.8mb]-\>[27.6m  
> b]/[1.1gb]}{[survivor] [149.7mb]-\>[138.6mb]/[149.7mb]}{[old]  
> [28.4gb]-\>[28.5gb]/[30.4gb]}  
> [2014-07-12 17:18:57,313][WARN][monitor.jvm]  
> [p-elasticlog02] [gc][young][195303][25075] duration [1.1s], collections  
> [1]/[40.6s], total [1.1s]/[34.6m], memory [30.4gb]-\>[29.2gb]/[31.7gb],  
> all\_pools {[young] [1.1gb]-\>[12.3mb]/  
> [1.1gb]}{[survivor] [149.7mb]-\>[0b]/[149.7mb]}{[old]  
> [29.1gb]-\>[29.2gb]/[30.4gb]}  
> [2014-07-12 17:18:57,314][WARN][monitor.jvm]  
> [p-elasticlog02] [gc][old][195303][53] duration [39.1s], collections  
> [2]/[40.6s], total [39.1s]/[1.5m], memory [30.4gb]-\>[29.2gb]/[31.7gb],  
> all\_pools {[young] [1.1gb]-\>[12.3mb]/[1.1  
> gb]}{[survivor] [149.7mb]-\>[0b]/[149.7mb]}{[old]  
> [29.1gb]-\>[29.2gb]/[30.4gb]}
> 
> I collect a lot of counters from elasticsearch (Using elasticsearch  
> collector in diamond (graphite collector written in python)) - so I have  
> data on the ES nodes.
> 
> my config is this:  
> index.warmer.enabled: false  
> cluster.name: elasticsearch  
> node.name: "p-elasticlog02"  
> node.master: true  
> node.data: true  
> action.disable\_delete\_all\_indices: true  
> indices.memory.index\_buffer\_size: 50%  
> indices.fielddata.cache.size: 30%  
> index.refresh\_interval: 5s  
> index.index\_concurrency: 16  
> threadpool.search.type: fixed  
> threadpool.search.size: 400  
> threadpool.search.queue\_size: 900  
> threadpool.bulk.type: fixed  
> threadpool.bulk.size: 500  
> threadpool.bulk.queue\_size: 900  
> threadpool.index.type: fixed  
> threadpool.index.size: 300  
> threadpool.index.queue\_size: -1  
> path.data: /var/lib/elasticsearch/  
> bootstrap.mlockall: true  
> network.publish\_host: $hostip  
> discovery.zen.ping.multicast.enabled: false  
> discovery.zen.ping.unicast.hosts: ["p-elasticlog01.example.idk",  
> "p-elasticlog02.example.idk", "p-elasticlog03.example.idk",  
> "p-elasticlog04.example.idk", "p-elasticlog05.example.idk"]
> 
> I have 24 cores in each machine (doing pretty much nothing) - so I was  
> considering trying to switch to g1gc f.ex. - as I've read it should be  
> better in some respects.. any input?
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/f485903b-b34c-4d58-bbf0-9a60a3866afe%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f485903b-b34c-4d58-bbf0-9a60a3866afe%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/f485903b-b34c-4d58-bbf0-9a60a3866afe%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f485903b-b34c-4d58-bbf0-9a60a3866afe%40googlegroups.com?utm_medium=email&utm_source=footer)  
> .  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624aALkP-b4tf%2BzsSjgGni9GsRu4CLMfRzDQJYt806bdPWw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624aALkP-b4tf%2BzsSjgGni9GsRu4CLMfRzDQJYt806bdPWw%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Harry\_Waye\_2](https://avatars.discourse-cdn.com/v4/letter/h/b2d939/32.png) [@Harry\_Waye\_2](https://discuss.elastic.co/u/Harry_Waye_2)\
**Post date:** [July 13, 2014, 12:25pm UTC](https://discuss.elastic.co/t/elasticsearch-dies-every-other-day/18627/3 "2014-07-13T12:25:41Z")

</div>

Maybe your GC times are pausing the node for longer than the default 30s  
zen ping timeout. G1GC should produce shorter pauses although playing  
around with your heap size with CMS could also work. I guess your indexing  
hard such that there's pressure to push new objects into the old gen  
prematurely, which then results in long pauses when it tries to clean up.

I'd probably just go with G1GC to reduce pause times and set minimum nodes  
settings. It you wanted to have a look to see what GC is doing try turning  
on PrintGCDetails and related flags and use a log analyzer to see if that  
gives you any clues.

On Sunday, July 13, 2014 12:22:46 AM UTC+1, Mark Walkom wrote:

> What's your heap size how much data do you have in your indexes?  
> Take a look at settings like discovery.zen.minimum\_master\_nodes to reduce  
> the multi-master problem you see, although ideally you want an odd number  
> of nodes for that (eg 3 or 5).
> 
> Regards,  
> Mark Walkom
> 
> Infrastructure Engineer  
> Campaign Monitor  
> email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \<javascript:\>  
> web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
> 
> On 13 July 2014 04:22, Klavs Klavsen \<kl...@enableit.dk \<javascript:\>\>  
> wrote:
> 
> > my cluster - running 1.1.2 with oracle java 1.7\_55 - dies several days a  
> > week  
> > one the nodes gets "disconnected"..  
> > this time one of them logged: observer timed out. notifying listener.  
> > several times  
> > I then have to restart them all - this time I had several who thought  
> > they were masters  
> > they are physical machines on the same LAN  
> > the machines (4) index between 70 and 130k documents (lines from  
> > logstash) per minute. I input into different indexes (not just  
> > logstash-$date)
> > 
> > after having a queue built up - they will easily index 260 to 290k/min -  
> > until the queue is emptied  
> > so they seem to have no resource shortage, but they somehow "get tired  
> > and die"- very often ☹
> > 
> > any ideas how I should proceed in debugging this issue? It seems to fit  
> > that EVERY time a node dies - I have garbage collection log entries. This  
> > time I had these on one node:  
> > [2014-07-12 17:12:25,132][INFO][monitor.jvm]  
> > [p-elasticlog02] [gc][young][194961][25004] duration [767ms], collections  
> > [1]/[1s], total [767ms]/[34.3m], memory [27.7gb]-\>[26.7gb]/[31.7gb],  
> > all\_pools {[young] [1gb]-\>[5.3mb]/[1.1  
> > gb]}{[survivor] [149.7mb]-\>[124.1mb]/[149.7mb]}{[old]  
> > [26.5gb]-\>[26.6gb]/[30.4gb]}  
> > [2014-07-12 17:12:44,929][INFO][monitor.jvm]  
> > [p-elasticlog02] [gc][young][194980][25007] duration [804ms], collections  
> > [1]/[1.1s], total [804ms]/[34.3m], memory [27.8gb]-\>[26.9gb]/[31.7gb],  
> > all\_pools {[young] [1gb]-\>[11.2mb]/[  
> > 1.1gb]}{[survivor] [149.7mb]-\>[146.1mb]/[149.7mb]}{[old]  
> > [26.6gb]-\>[26.7gb]/[30.4gb]}  
> > [2014-07-12 17:14:57,032][INFO][monitor.jvm]  
> > [p-elasticlog02] [gc][young][195109][25035] duration [837ms], collections  
> > [1]/[1s], total [837ms]/[34.4m], memory [28.9gb]-\>[28.1gb]/[31.7gb],  
> > all\_pools {[young] [1gb]-\>[141.1mb]/[1  
> > .1gb]}{[survivor] [149.7mb]-\>[145.7mb]/[149.7mb]}{[old]  
> > [27.7gb]-\>[27.8gb]/[30.4gb]}  
> > [2014-07-12 17:16:17,016][INFO][monitor.jvm]  
> > [p-elasticlog02] [gc][young][195187][25053] duration [756ms], collections  
> > [1]/[1.4s], total [756ms]/[34.5m], memory [29.5gb]-\>[28.7gb]/[31.7gb],  
> > all\_pools {[young] [926.8mb]-\>[27.6m  
> > b]/[1.1gb]}{[survivor] [149.7mb]-\>[138.6mb]/[149.7mb]}{[old]  
> > [28.4gb]-\>[28.5gb]/[30.4gb]}  
> > [2014-07-12 17:18:57,313][WARN][monitor.jvm]  
> > [p-elasticlog02] [gc][young][195303][25075] duration [1.1s], collections  
> > [1]/[40.6s], total [1.1s]/[34.6m], memory [30.4gb]-\>[29.2gb]/[31.7gb],  
> > all\_pools {[young] [1.1gb]-\>[12.3mb]/  
> > [1.1gb]}{[survivor] [149.7mb]-\>[0b]/[149.7mb]}{[old]  
> > [29.1gb]-\>[29.2gb]/[30.4gb]}  
> > [2014-07-12 17:18:57,314][WARN][monitor.jvm]  
> > [p-elasticlog02] [gc][old][195303][53] duration [39.1s], collections  
> > [2]/[40.6s], total [39.1s]/[1.5m], memory [30.4gb]-\>[29.2gb]/[31.7gb],  
> > all\_pools {[young] [1.1gb]-\>[12.3mb]/[1.1  
> > gb]}{[survivor] [149.7mb]-\>[0b]/[149.7mb]}{[old]  
> > [29.1gb]-\>[29.2gb]/[30.4gb]}
> > 
> > I collect a lot of counters from elasticsearch (Using elasticsearch  
> > collector in diamond (graphite collector written in python)) - so I have  
> > data on the ES nodes.
> > 
> > my config is this:  
> > index.warmer.enabled: false  
> > cluster.name: elasticsearch  
> > node.name: "p-elasticlog02"  
> > node.master: true  
> > node.data: true  
> > action.disable\_delete\_all\_indices: true  
> > indices.memory.index\_buffer\_size: 50%  
> > indices.fielddata.cache.size: 30%  
> > index.refresh\_interval: 5s  
> > index.index\_concurrency: 16  
> > threadpool.search.type: fixed  
> > threadpool.search.size: 400  
> > threadpool.search.queue\_size: 900  
> > threadpool.bulk.type: fixed  
> > threadpool.bulk.size: 500  
> > threadpool.bulk.queue\_size: 900  
> > threadpool.index.type: fixed  
> > threadpool.index.size: 300  
> > threadpool.index.queue\_size: -1  
> > path.data: /var/lib/elasticsearch/  
> > bootstrap.mlockall: true  
> > network.publish\_host: $hostip  
> > discovery.zen.ping.multicast.enabled: false  
> > discovery.zen.ping.unicast.hosts: ["p-elasticlog01.example.idk",  
> > "p-elasticlog02.example.idk", "p-elasticlog03.example.idk",  
> > "p-elasticlog04.example.idk", "p-elasticlog05.example.idk"]
> > 
> > I have 24 cores in each machine (doing pretty much nothing) - so I was  
> > considering trying to switch to g1gc f.ex. - as I've read it should be  
> > better in some respects.. any input?
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/f485903b-b34c-4d58-bbf0-9a60a3866afe%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f485903b-b34c-4d58-bbf0-9a60a3866afe%40googlegroups.com)  
> > [https://groups.google.com/d/msgid/elasticsearch/f485903b-b34c-4d58-bbf0-9a60a3866afe%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f485903b-b34c-4d58-bbf0-9a60a3866afe%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > .  
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/322ce7e2-42bf-4115-a331-b8508a949d77%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/322ce7e2-42bf-4115-a331-b8508a949d77%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![smonasco\_2](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@smonasco\_2](https://discuss.elastic.co/u/smonasco_2)\
**Post date:** [July 13, 2014, 1:45pm UTC](https://discuss.elastic.co/t/elasticsearch-dies-every-other-day/18627/4 "2014-07-13T13:45:58Z")

</div>

You are certainly having a heap utilization issue. As the utilization gets close to 100% the GCs get aggressive.

Is your heap utilization staying close to the edge? Does it jump up at the end? What about field cache? What about hot threads?

200+ per second is higher than needed indexing. You may want to try scalling back index threads.

Otherwise if it's not a huge spike you may need more memory.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/cb2e2b9b-2821-4b07-a85f-9bb51873b14a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/cb2e2b9b-2821-4b07-a85f-9bb51873b14a%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [July 13, 2014, 4:01pm UTC](https://discuss.elastic.co/t/elasticsearch-dies-every-other-day/18627/5 "2014-07-13T16:01:57Z")

</div>

With this setting, you allow the flooding of your heap when using single  
doc indexing. Now you ask for the reason why your heap collapses. Not sure  
how Logstash indexer works but you should review your custom settings.

Jörg

On Saturday, July 12, 2014 8:22:33 PM UTC+2, Klavs Klavsen wrote:

> threadpool.index.queue\_size: -1

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3ee40446-452d-46fc-a9fe-9c2be7e53391%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3ee40446-452d-46fc-a9fe-9c2be7e53391%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Klavs\_Klavsen](https://avatars.discourse-cdn.com/v4/letter/k/90db22/32.png) [@Klavs\_Klavsen](https://discuss.elastic.co/u/Klavs_Klavsen)\
**Post date:** [July 14, 2014, 11:17am UTC](https://discuss.elastic.co/t/elasticsearch-dies-every-other-day/18627/6 "2014-07-14T11:17:29Z")

</div>

Changing it to:  
threadpool.index.queue\_size: 900

hope it helps 🙂

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f534e2a3-c3b9-4f7b-ada6-c48527192bbd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f534e2a3-c3b9-4f7b-ada6-c48527192bbd%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Klavs\_Klavsen](https://avatars.discourse-cdn.com/v4/letter/k/90db22/32.png) [@Klavs\_Klavsen](https://discuss.elastic.co/u/Klavs_Klavsen)\
**Post date:** [July 14, 2014, 12:31pm UTC](https://discuss.elastic.co/t/elasticsearch-dies-every-other-day/18627/7 "2014-07-14T12:31:19Z")

</div>

Den søndag den 13. juli 2014 01.22.46 UTC+2 skrev Mark Walkom:

> What's your heap size how much data do you have in your indexes?

data used = approx. 4,5TB  
heapsize = 32700M (less than 32GB 🙂

Heap memory used usually runs between 24 and 28GB - but when this happens,  
I see heap usage go to max heapsize.

5505713372 hits in 1322 shards.

Take a look at settings like discovery.zen.minimum\_master\_nodes to reduce

> the multi-master problem you see, although ideally you want an odd number  
> of nodes for that (eg 3 or 5).
> 
> I have 4 data nodes and one none-data node (does it count as a 5.?)

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1febddfc-9382-4f41-85d5-ff9fcd7df81b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1febddfc-9382-4f41-85d5-ff9fcd7df81b%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Klavs\_Klavsen](https://avatars.discourse-cdn.com/v4/letter/k/90db22/32.png) [@Klavs\_Klavsen](https://discuss.elastic.co/u/Klavs_Klavsen)\
**Post date:** [July 14, 2014, 12:47pm UTC](https://discuss.elastic.co/t/elasticsearch-dies-every-other-day/18627/8 "2014-07-14T12:47:41Z")

</div>

Here's the heap usage graph for the last week:  
[![](http://blog.klavsen.info/ES-heapusage.png) ](http://blog.klavsen.info/ES-heapusage.png)

I've marked where it crashes. I hope this can help give an idea of what  
goes wrong and what I could try?

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b0dc1d93-97dd-49aa-809e-db3d46fe0de1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b0dc1d93-97dd-49aa-809e-db3d46fe0de1%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![smonasco\_2](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@smonasco\_2](https://discuss.elastic.co/u/smonasco_2)\
**Post date:** [July 14, 2014, 1:18pm UTC](https://discuss.elastic.co/t/elasticsearch-dies-every-other-day/18627/9 "2014-07-14T13:18:30Z")

</div>

Is the non data node a client node? Here we are counting master eligible nodes. Whether you havever 4 or 5 I would go with min master eligible nodes 3. I'm sure what your replica setup is, but only 2 nodes is probably not a healthy cluster.

With this set to 3 you won't have a split brain.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b7b4fb77-478e-4709-9f6f-60bfad67feef%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b7b4fb77-478e-4709-9f6f-60bfad67feef%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![smonasco\_2](https://avatars.discourse-cdn.com/v4/letter/s/848f3c/32.png) [@smonasco\_2](https://discuss.elastic.co/u/smonasco_2)\
**Post date:** [July 14, 2014, 1:22pm UTC](https://discuss.elastic.co/t/elasticsearch-dies-every-other-day/18627/10 "2014-07-14T13:22:40Z")

</div>

Do you have thread counts and do any of them correlate to the crash times? I'm guessing that we'll find index threads leap up.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/fab19f43-8735-4208-873f-a4575a05b313%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fab19f43-8735-4208-873f-a4575a05b313%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 14, 2014, 11:00pm UTC](https://discuss.elastic.co/t/elasticsearch-dies-every-other-day/18627/11 "2014-07-14T23:00:47Z")

</div>

Looks like your cluster is at capacity.  
Try closing/deleting some old indexes, adding more nodes or more RAM.

Regards,  
Mark Walkom

Infrastructure Engineer  
Campaign Monitor  
email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)  
web: [www.campaignmonitor.com](http://www.campaignmonitor.com)

On 14 July 2014 23:22, smonasco [smonasco@gmail.com](mailto:smonasco@gmail.com) wrote:

> Do you have thread counts and do any of them correlate to the crash times?  
> I'm guessing that we'll find index threads leap up.
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/fab19f43-8735-4208-873f-a4575a05b313%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/fab19f43-8735-4208-873f-a4575a05b313%40googlegroups.com)  
> .  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624bY6\_%2B\_6cpiCoWRVToeKVK0%3DhQsGM7kQ8LSzVa%3D%2Ba7%3Dwg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624bY6_%2B_6cpiCoWRVToeKVK0%3DhQsGM7kQ8LSzVa%3D%2Ba7%3Dwg%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Klavs\_Klavsen](https://avatars.discourse-cdn.com/v4/letter/k/90db22/32.png) [@Klavs\_Klavsen](https://discuss.elastic.co/u/Klavs_Klavsen)\
**Post date:** [July 15, 2014, 12:32pm UTC](https://discuss.elastic.co/t/elasticsearch-dies-every-other-day/18627/12 "2014-07-15T12:32:53Z")

</div>

define "at capacity" ?  
I can easily give it more ram.. it has had 42GB -but that just made it jump  
even more 🙂

I changed index queue to 900 (instead of -1) and set two of the four nodes  
to using g1gc - which has altered the GC very much (it collects very  
frequently) - and those were the ones that died most often - and so far,  
they have not died.

heap memory usage is ~28GB at max. still letting it run for a few days.  
I simultaneously (I know that was stupid.. ) moved some content, that I  
know they search over longer periods (7 days and longer) into it's own  
index - so they do not search the entire logstash index.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/9f306df3-f9d7-44ec-8701-b63d5b3e51f7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9f306df3-f9d7-44ec-8701-b63d5b3e51f7%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 15, 2014, 10:31pm UTC](https://discuss.elastic.co/t/elasticsearch-dies-every-other-day/18627/13 "2014-07-15T22:31:29Z")

</div>

Every system has a limit of what it can process or maintain, you're  
probably very near your limit based on your setup.  
This is a fluid limit as it depends a lot on what you do, but generally if  
you aren't doing anything beyond what you'd consider "normal" and you are  
getting GC problems and crashing nodes, you're likely to be at that limit.

Regards,  
Mark Walkom

Infrastructure Engineer  
Campaign Monitor  
email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)  
web: [www.campaignmonitor.com](http://www.campaignmonitor.com)

On 15 July 2014 22:32, Klavs Klavsen [klavs@enableit.dk](mailto:klavs@enableit.dk) wrote:

> define "at capacity" ?  
> I can easily give it more ram.. it has had 42GB -but that just made it  
> jump even more 🙂
> 
> I changed index queue to 900 (instead of -1) and set two of the four nodes  
> to using g1gc - which has altered the GC very much (it collects very  
> frequently) - and those were the ones that died most often - and so far,  
> they have not died.
> 
> heap memory usage is ~28GB at max. still letting it run for a few days.  
> I simultaneously (I know that was stupid.. ) moved some content, that I  
> know they search over longer periods (7 days and longer) into it's own  
> index - so they do not search the entire logstash index.
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/9f306df3-f9d7-44ec-8701-b63d5b3e51f7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9f306df3-f9d7-44ec-8701-b63d5b3e51f7%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/9f306df3-f9d7-44ec-8701-b63d5b3e51f7%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9f306df3-f9d7-44ec-8701-b63d5b3e51f7%40googlegroups.com?utm_medium=email&utm_source=footer)  
> .  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624Z\_MS5aOMAiBUo\_yBh7vnEYTVcuL-UNbB1xWrM-Rn38Ow%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624Z_MS5aOMAiBUo_yBh7vnEYTVcuL-UNbB1xWrM-Rn38Ow%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Klavs\_Klavsen](https://avatars.discourse-cdn.com/v4/letter/k/90db22/32.png) [@Klavs\_Klavsen](https://discuss.elastic.co/u/Klavs_Klavsen)\
**Post date:** [July 16, 2014, 7:26am UTC](https://discuss.elastic.co/t/elasticsearch-dies-every-other-day/18627/14 "2014-07-16T07:26:46Z")

</div>

I updated the graph:  
[![](http://blog.klavsen.info/ES-graphs-update.png) ](http://blog.klavsen.info/ES-graphs-update.png)

I added overview of how many threads were running, and its appearent that  
what peaked when it crashed (left side of the two graphs - two spikes where  
it crashed) - correlated with a peak in search threads.  
Also the change to G1GC for two of the nodes - is very appearent in the  
heap\_mem\_usage graph 🙂

It's been stable for two days now.. nearing the record 🙂 I did also move a  
"culprit" who searched the main index for larger periods, to their own  
index.. and change threadpool.index.queue\_size: from -1 to 900.  
the index queue size does not seem to be hit at all, so I'm not sure that  
made a change.

Thank you for your input everyone.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/dd7fac5b-e97f-4abc-94e4-23e66efe5c36%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/dd7fac5b-e97f-4abc-94e4-23e66efe5c36%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Robert\_Gardam](https://avatars.discourse-cdn.com/v4/letter/r/919ad9/32.png) [@Robert\_Gardam](https://discuss.elastic.co/u/Robert_Gardam)\
**Post date:** [September 10, 2014, 2:53pm UTC](https://discuss.elastic.co/t/elasticsearch-dies-every-other-day/18627/15 "2014-09-10T14:53:25Z")

</div>

Hey! Your graphs are really nice. That looks like grafana. I was wondering  
how you're pipiing data there? I used the ES graphite plugin and found that  
it flooded my graphite with too much data.

Thanks

On Wednesday, July 16, 2014 9:26:46 AM UTC+2, Klavs Klavsen wrote:

> I updated the graph:  
> [http://blog.klavsen.info/ES-graphs-update.png](http://blog.klavsen.info/ES-graphs-update.png)
> 
> I added overview of how many threads were running, and its appearent that  
> what peaked when it crashed (left side of the two graphs - two spikes where  
> it crashed) - correlated with a peak in search threads.  
> Also the change to G1GC for two of the nodes - is very appearent in the  
> heap\_mem\_usage graph 🙂
> 
> It's been stable for two days now.. nearing the record 🙂 I did also move  
> a "culprit" who searched the main index for larger periods, to their own  
> index.. and change threadpool.index.queue\_size: from -1 to 900.  
> the index queue size does not seem to be hit at all, so I'm not sure that  
> made a change.
> 
> Thank you for your input everyone.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ff6462a4-e99a-4b82-b45a-303a9c87c3bd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ff6462a4-e99a-4b82-b45a-303a9c87c3bd%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:03am UTC](https://discuss.elastic.co/t/elasticsearch-dies-every-other-day/18627/16 "2017-07-06T01:03:17Z")

</div>