# Update to ES 1.4.2 gone terribly wrong - nodes won't start
**URL:** https://discuss.elastic.co/t/update-to-es-1-4-2-gone-terribly-wrong-nodes-wont-start/21402
**Category:** Elasticsearch
**Created:** [December 26, 2014, 8:52pm UTC](https://discuss.elastic.co/t/update-to-es-1-4-2-gone-terribly-wrong-nodes-wont-start/21402 "2014-12-26T20:52:48Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@Jean\_Noel\_Rivasseau](https://discuss.elastic.co/u/Jean_Noel_Rivasseau)
#### Post date: [December 26, 2014, 8:52pm UTC](https://discuss.elastic.co/t/update-to-es-1-4-2-gone-terribly-wrong-nodes-wont-start/21402/1 "2014-12-26T20:52:48Z")
Hello,
We had a 2 nodes, 10 shards cluster with about 150GB of data (not sure how
many documents, I would say about 150 millions since 1 doc is about 1K for
us). It worked fine under ES 1.3.4.
We tried to update it today to 1.4.2. Now the nodes won't start. They eat
up all RAM during the recovery phase (when booting, and starting engines on
all indices), then GC occurs when no more RAM is available for ES. ES then
eventually gets stuck with OOM errors.
I tried to increase the RAM given to ES to 30GB and it still eats all that
up and fails. Before with 1.3.4 we had 16GB allocated and no problems.
Why ES needs all that RAM? It's doing nothing, not servicing a single
request... it's only starting the recovery process...
We now have our production cluster down. It's really a huge problem for us,
please advise on any solutions or things we could try out.
Java is IcedTea 7.2.5.3
OS is Gentoo Linux
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7b5b4fb8-bbd6-4db2-ad10-8446ebd92140%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7b5b4fb8-bbd6-4db2-ad10-8446ebd92140%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Jean\_Noel\_Rivasseau](https://discuss.elastic.co/u/Jean_Noel_Rivasseau)
#### Post date: [December 27, 2014, 9:34am UTC](https://discuss.elastic.co/t/update-to-es-1-4-2-gone-terribly-wrong-nodes-wont-start/21402/2 "2014-12-27T09:34:44Z")
To be a bit more precise, we have 375GB of data. Still, I am under the
impression that 1.4.2 uses at least 3 times more RAM than 1.3.4. Everything
was fine under 1.3.4.
On Friday, December 26, 2014 11:52:48 PM UTC+3, Jean-Noël Rivasseau wrote:
> Hello,
>
> We had a 2 nodes, 10 shards cluster with about 150GB of data (not sure how
> many documents, I would say about 150 millions since 1 doc is about 1K for
> us). It worked fine under ES 1.3.4.
>
> We tried to update it today to 1.4.2. Now the nodes won't start. They eat
> up all RAM during the recovery phase (when booting, and starting engines on
> all indices), then GC occurs when no more RAM is available for ES. ES then
> eventually gets stuck with OOM errors.
>
> I tried to increase the RAM given to ES to 30GB and it still eats all that
> up and fails. Before with 1.3.4 we had 16GB allocated and no problems.
>
> Why ES needs all that RAM? It's doing nothing, not servicing a single
> request... it's only starting the recovery process...
>
> We now have our production cluster down. It's really a huge problem for
> us, please advise on any solutions or things we could try out.
>
> Java is IcedTea 7.2.5.3
> OS is Gentoo Linux
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8a769d4b-65b7-4809-93a9-86b36e64e459%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8a769d4b-65b7-4809-93a9-86b36e64e459%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [December 27, 2014, 12:02pm UTC](https://discuss.elastic.co/t/update-to-es-1-4-2-gone-terribly-wrong-nodes-wont-start/21402/3 "2014-12-27T12:02:58Z")
Check if you had flushed the translog of the indexes on 1.3.4 before
migration. Replaying the translog may add substantial amount of RAM usage.
Check if you had set index.codec.bloom.load=true, and if so, set it to
false.
Try Oracle JDK.
Try to add a third node.
Jörg
On Sat, Dec 27, 2014 at 10:34 AM, Jean-Noël Rivasseau [elvanor@gmail.com](mailto:elvanor@gmail.com)
wrote:
> To be a bit more precise, we have 375GB of data. Still, I am under the
> impression that 1.4.2 uses at least 3 times more RAM than 1.3.4. Everything
> was fine under 1.3.4.
>
> On Friday, December 26, 2014 11:52:48 PM UTC+3, Jean-Noël Rivasseau wrote:
>
> > Hello,
> >
> > We had a 2 nodes, 10 shards cluster with about 150GB of data (not sure
> > how many documents, I would say about 150 millions since 1 doc is about 1K
> > for us). It worked fine under ES 1.3.4.
> >
> > We tried to update it today to 1.4.2. Now the nodes won't start. They eat
> > up all RAM during the recovery phase (when booting, and starting engines on
> > all indices), then GC occurs when no more RAM is available for ES. ES then
> > eventually gets stuck with OOM errors.
> >
> > I tried to increase the RAM given to ES to 30GB and it still eats all
> > that up and fails. Before with 1.3.4 we had 16GB allocated and no problems.
> >
> > Why ES needs all that RAM? It's doing nothing, not servicing a single
> > request... it's only starting the recovery process...
> >
> > We now have our production cluster down. It's really a huge problem for
> > us, please advise on any solutions or things we could try out.
> >
> > Java is IcedTea 7.2.5.3
> > OS is Gentoo Linux
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/8a769d4b-65b7-4809-93a9-86b36e64e459%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8a769d4b-65b7-4809-93a9-86b36e64e459%40googlegroups.com)
> [https://groups.google.com/d/msgid/elasticsearch/8a769d4b-65b7-4809-93a9-86b36e64e459%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/8a769d4b-65b7-4809-93a9-86b36e64e459%40googlegroups.com?utm_medium=email&utm_source=footer)
> .
>
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFEdN84YqbCao%3DLkWuF0S11wnrstM1VtY9mRant%2BLS4wQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFEdN84YqbCao%3DLkWuF0S11wnrstM1VtY9mRant%2BLS4wQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@nik9000](https://discuss.elastic.co/u/nik9000)
#### Post date: [December 27, 2014, 1:33pm UTC](https://discuss.elastic.co/t/update-to-es-1-4-2-gone-terribly-wrong-nodes-wont-start/21402/4 "2014-12-27T13:33:41Z")
IcedTea isn't a JVM version. Give us `java -version`. It looks like that
version of IcedTea could be OpenJDK 7u71 which is generally fine (we use it
under plenty of loaf). It could also be jamvm or cacao or zero/shark. Those
probably won't work. Lots of folks suggest oraclejdk so you may as well try
like Joe says.
You can try using jmap to get a heap histogram when memory is filling up
and posting that somewhere.
You may be able to downgrade. The worst that can happen is some indexes can
open then you have to delete and rebuild them from source. This is only
likely to work because you can't get it to boot.
Also, running gentoo in production is either brave or crazy or both.
Nik
On Dec 26, 2014 3:52 PM, "Jean-Noël Rivasseau" [elvanor@gmail.com](mailto:elvanor@gmail.com) wrote:
> Hello,
>
> We had a 2 nodes, 10 shards cluster with about 150GB of data (not sure how
> many documents, I would say about 150 millions since 1 doc is about 1K for
> us). It worked fine under ES 1.3.4.
>
> We tried to update it today to 1.4.2. Now the nodes won't start. They eat
> up all RAM during the recovery phase (when booting, and starting engines on
> all indices), then GC occurs when no more RAM is available for ES. ES then
> eventually gets stuck with OOM errors.
>
> I tried to increase the RAM given to ES to 30GB and it still eats all that
> up and fails. Before with 1.3.4 we had 16GB allocated and no problems.
>
> Why ES needs all that RAM? It's doing nothing, not servicing a single
> request... it's only starting the recovery process...
>
> We now have our production cluster down. It's really a huge problem for
> us, please advise on any solutions or things we could try out.
>
> Java is IcedTea 7.2.5.3
> OS is Gentoo Linux
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/7b5b4fb8-bbd6-4db2-ad10-8446ebd92140%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7b5b4fb8-bbd6-4db2-ad10-8446ebd92140%40googlegroups.com)
> [https://groups.google.com/d/msgid/elasticsearch/7b5b4fb8-bbd6-4db2-ad10-8446ebd92140%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/7b5b4fb8-bbd6-4db2-ad10-8446ebd92140%40googlegroups.com?utm_medium=email&utm_source=footer)
> .
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3pYSC5qOxcmQ97PBw7fnBYqFT%2BQqUD4kv-r6dRS2Zubw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3pYSC5qOxcmQ97PBw7fnBYqFT%2BQqUD4kv-r6dRS2Zubw%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Jean\_Noel\_Rivasseau](https://discuss.elastic.co/u/Jean_Noel_Rivasseau)
#### Post date: [December 27, 2014, 1:37pm UTC](https://discuss.elastic.co/t/update-to-es-1-4-2-gone-terribly-wrong-nodes-wont-start/21402/5 "2014-12-27T13:37:31Z")
We were being affected by
>
>
> We have one particularly large index in our cluster - it contains 10s of million…s of documents and has quite a lot of nesteds too. Prior to 1.4.0 Beta 1 (including 1.2.x and 1.3.x) the index re-initialised on a node with 8GB allocated to ElasticSearch (16GB+ available in OS). Since 1.4.0 Beta 1 (and still on 1.4.0) we're getting an OOM exception (startup log and exception stack below). At this point, the node ceases recovery (expected, I guess) and becomes unresponsive. All data nodes suffer the same fate and the entire cluster becomes unresponsive.
>
> \`\`\`
> \[2014-11-07 17:12:39,895\]\[WARN \]\[common.jna \] unable to link C library. native methods (mlockall) will be disabled.
> \[2014-11-07 17:12:40,077\]\[INFO \]\[node \] \[dvlp\_FRONTEND2\] version\[1.4.0\], pid\[9052\], build\[bc94bd8/2014-11-05T14:26:12Z\]
> \[2014-11-07 17:12:40,077\]\[INFO \]\[node \] \[dvlp\_FRONTEND2\] initializing ...
> \[2014-11-07 17:12:40,129\]\[INFO \]\[plugins \] \[dvlp\_FRONTEND2\] loaded \[cloud-aws\], sites \[bigdesk, head, inquisitor, kopf\]
> \[2014-11-07 17:12:45,220\]\[INFO \]\[node \] \[dvlp\_FRONTEND2\] initialized
> \[2014-11-07 17:12:45,220\]\[INFO \]\[node \] \[dvlp\_FRONTEND2\] starting ...
> \[2014-11-07 17:12:45,438\]\[INFO \]\[transport \] \[dvlp\_FRONTEND2\] bound\_address {inet\[/0:0:0:0:0:0:0:0:50882\]}, publish\_address {inet\[FRONTEND2/192.168.10.73:50882\]}
> \[2014-11-07 17:12:45,452\]\[INFO \]\[discovery \] \[dvlp\_FRONTEND2\] dvlp/C2f-euXcRc-cEv3dnsBnXw
> \[2014-11-07 17:13:15,451\]\[WARN \]\[discovery \] \[dvlp\_FRONTEND2\] waited for 30s and no initial state was set by the discovery
> \[2014-11-07 17:13:15,468\]\[INFO \]\[http \] \[dvlp\_FRONTEND2\] bound\_address {inet\[/0:0:0:0:0:0:0:0:50881\]}, publish\_address {inet\[frontend2/192.168.10.73:50881\]}
> \[2014-11-07 17:13:15,468\]\[INFO \]\[node \] \[dvlp\_FRONTEND2\] started
> \[2014-11-07 17:13:48,552\]\[INFO \]\[discovery.zen \] \[dvlp\_FRONTEND2\] failed to send join request to master \[\[dvlp\_FRONTEND2\_coordinator\]\[jwhGk5NyTx-E1HInKTLDkg\]\[FRONTEND2\]\[inet\[/192.168.10.73:55591\]\]{datacentrename=site2, data=false, nodename=dvlp\_FRONTEND2\_coordinator, master=true}\], reason \[ElasticsearchTimeoutException\[Timeout waiting for task.\]\]
> \[2014-11-07 17:14:51,597\]\[INFO \]\[discovery.zen \] \[dvlp\_FRONTEND2\] failed to send join request to master \[\[dvlp\_FRONTEND2\_coordinator\]\[jwhGk5NyTx-E1HInKTLDkg\]\[FRONTEND2\]\[inet\[/192.168.10.73:55591\]\]{datacentrename=site2, data=false, nodename=dvlp\_FRONTEND2\_coordinator, master=true}\], reason \[ElasticsearchTimeoutException\[Timeout waiting for task.\]\]
> \[2014-11-07 17:15:54,633\]\[INFO \]\[discovery.zen \] \[dvlp\_FRONTEND2\] failed to send join request to master \[\[dvlp\_FRONTEND2\_coordinator\]\[jwhGk5NyTx-E1HInKTLDkg\]\[FRONTEND2\]\[inet\[/192.168.10.73:55591\]\]{datacentrename=site2, data=false, nodename=dvlp\_FRONTEND2\_coordinator, master=true}\], reason \[ElasticsearchTimeoutException\[Timeout waiting for task.\]\]
> \[2014-11-07 17:16:57,647\]\[INFO \]\[discovery.zen \] \[dvlp\_FRONTEND2\] failed to send join request to master \[\[dvlp\_FRONTEND2\_coordinator\]\[jwhGk5NyTx-E1HInKTLDkg\]\[FRONTEND2\]\[inet\[/192.168.10.73:55591\]\]{datacentrename=site2, data=false, nodename=dvlp\_FRONTEND2\_coordinator, master=true}\], reason \[ElasticsearchTimeoutException\[Timeout waiting for task.\]\]
> \[2014-11-07 17:18:00,664\]\[INFO \]\[discovery.zen \] \[dvlp\_FRONTEND2\] failed to send join request to master \[\[dvlp\_FRONTEND2\_coordinator\]\[jwhGk5NyTx-E1HInKTLDkg\]\[FRONTEND2\]\[inet\[/192.168.10.73:55591\]\]{datacentrename=site2, data=false, nodename=dvlp\_FRONTEND2\_coordinator, master=true}\], reason \[ElasticsearchTimeoutException\[Timeout waiting for task.\]\]
> \[2014-11-07 17:19:03,675\]\[INFO \]\[discovery.zen \] \[dvlp\_FRONTEND2\] failed to send join request to master \[\[dvlp\_FRONTEND2\_coordinator\]\[jwhGk5NyTx-E1HInKTLDkg\]\[FRONTEND2\]\[inet\[/192.168.10.73:55591\]\]{datacentrename=site2, data=false, nodename=dvlp\_FRONTEND2\_coordinator, master=true}\], reason \[ElasticsearchTimeoutException\[Timeout waiting for task.\]\]
> \[2014-11-07 17:20:06,684\]\[INFO \]\[discovery.zen \] \[dvlp\_FRONTEND2\] failed to send join request to master \[\[dvlp\_FRONTEND2\_coordinator\]\[jwhGk5NyTx-E1HInKTLDkg\]\[FRONTEND2\]\[inet\[/192.168.10.73:55591\]\]{datacentrename=site2, data=false, nodename=dvlp\_FRONTEND2\_coordinator, master=true}\], reason \[ElasticsearchTimeoutException\[Timeout waiting for task.\]\]
> \[2014-11-07 17:20:36,950\]\[INFO \]\[discovery.zen \] \[dvlp\_FRONTEND2\] failed to send join request to master \[\[dvlp\_FRONTEND2\_coordinator\]\[jwhGk5NyTx-E1HInKTLDkg\]\[FRONTEND2\]\[inet\[/192.168.10.73:55591\]\]{datacentrename=site2, data=false, nodename=dvlp\_FRONTEND2\_coordinator, master=true}\], reason \[NodeDisconnectedException\[\[dvlp\_FRONTEND2\_coordinator\]\[inet\[/192.168.10.73:55591\]\]\[internal:discovery/zen/join\] disconnected\]\]
> \[2014-11-07 17:20:41,171\]\[WARN \]\[transport.netty \] \[dvlp\_FRONTEND2\] Message not fully read (response) for \[85\] handler future(org.elasticsearch.transport.EmptyTransportResponseHandler@2060e2c8), error \[true\], resetting
> \[2014-11-07 17:20:41,171\]\[INFO \]\[discovery.zen \] \[dvlp\_FRONTEND2\] failed to send join request to master \[\[dvlp\_FRONTEND1\_coordinator\]\[4y8Hh5kAQPK2Ie3gzc58Ww\]\[FRONTEND1\]\[inet\[/192.168.10.70:55858\]\]{datacentrename=site1, data=false, nodename=dvlp\_FRONTEND1\_coordinator, master=true}\], reason \[RemoteTransportException\[Failed to deserialize exception response from stream\]; nested: TransportSerializationException\[Failed to deserialize exception response from stream\]; nested: StreamCorruptedException\[unexpected end of block data\]; \]
> \[2014-11-07 17:20:45,520\]\[INFO \]\[discovery.zen \] \[dvlp\_FRONTEND2\] failed to send join request to master \[\[dvlp\_FRONTEND2\_coordinator\]\[-O87CxU3RRSTHZkuC985Yw\]\[FRONTEND2\]\[inet\[/192.168.10.73:55591\]\]{datacentrename=site2, data=false, nodename=dvlp\_FRONTEND2\_coordinator, master=true}\], reason \[RemoteTransportException\[\[dvlp\_FRONTEND2\_coordinator\]\[inet\[/192.168.10.73:55591\]\]\[internal:discovery/zen/join\]\]; nested: ElasticsearchIllegalStateException\[Node \[\[dvlp\_FRONTEND2\_coordinator\]\[-O87CxU3RRSTHZkuC985Yw\]\[FRONTEND2\]\[inet\[FRONTEND2/192.168.10.73:55591\]\]{datacentrename=site2, data=false, nodename=dvlp\_FRONTEND2\_coordinator, master=true}\] not master for join request from \[\[dvlp\_FRONTEND2\]\[C2f-euXcRc-cEv3dnsBnXw\]\[FRONTEND2\]\[inet\[/192.168.10.73:50882\]\]{datacentrename=site2, nodename=dvlp\_FRONTEND2, master=false}\]\]; \], tried \[3\] times
> \[2014-11-07 17:20:48,831\]\[INFO \]\[cluster.service \] \[dvlp\_FRONTEND2\] detected\_master \[dvlp\_FRONTEND2\_coordinator\]\[-O87CxU3RRSTHZkuC985Yw\]\[FRONTEND2\]\[inet\[/192.168.10.73:55591\]\]{datacentrename=site2, data=false, nodename=dvlp\_FRONTEND2\_coordinator, master=true}, added {\[dvlp\_DEVBATH01.exabre.co.uk\_loadbalancer\]\[8i4izXAUQiWeS2arwV9LeA\]\[DEVBATH01\]\[inet\[/192.168.10.65:12184\]\]{datacentrename=site1, data=false, nodename=dvlp\_DEVBATH01.exabre.co.uk\_loadbalancer, master=true},\[dvlp\_FRONTEND2\_coordinator\]\[-O87CxU3RRSTHZkuC985Yw\]\[FRONTEND2\]\[inet\[/192.168.10.73:55591\]\]{datacentrename=site2, data=false, nodename=dvlp\_FRONTEND2\_coordinator, master=true},\[dvlp\_FRONTEND2\_loadbalancer\]\[joVXc\_fGTx-SC\_YwJ2YBmQ\]\[FRONTEND2\]\[inet\[/192.168.10.73:65341\]\]{datacentrename=site2, data=false, nodename=dvlp\_FRONTEND2\_loadbalancer, master=false},\[dvlp\_FRONTEND1\_loadbalancer\]\[snDHwo0YTR6VsAFV9nBcxw\]\[FRONTEND1\]\[inet\[/192.168.10.70:55054\]\]{datacentrename=site1, data=false, nodename=dvlp\_FRONTEND1\_loadbalancer, master=false},}, reason: zen-disco-receive(from master \[\[dvlp\_FRONTEND2\_coordinator\]\[-O87CxU3RRSTHZkuC985Yw\]\[FRONTEND2\]\[inet\[/192.168.10.73:55591\]\]{datacentrename=site2, data=false, nodename=dvlp\_FRONTEND2\_coordinator, master=true}\])
> \[2014-11-07 17:21:01,937\]\[INFO \]\[cluster.service \] \[dvlp\_FRONTEND2\] added {\[dvlp\_FRONTEND1\_coordinator\]\[4y8Hh5kAQPK2Ie3gzc58Ww\]\[FRONTEND1\]\[inet\[/192.168.10.70:55858\]\]{datacentrename=site1, data=false, nodename=dvlp\_FRONTEND1\_coordinator, master=true},}, reason: zen-disco-receive(from master \[\[dvlp\_FRONTEND2\_coordinator\]\[-O87CxU3RRSTHZkuC985Yw\]\[FRONTEND2\]\[inet\[/192.168.10.73:55591\]\]{datacentrename=site2, data=false, nodename=dvlp\_FRONTEND2\_coordinator, master=true}\])
> \[2014-11-07 17:25:25,598\]\[INFO \]\[monitor.jvm \] \[dvlp\_FRONTEND2\] \[gc\]\[old\]\[739\]\[27\] duration \[8s\], collections \[1\]/\[9s\], total \[8s\]/\[8.8s\], memory \[7.8gb\]-\>\[7.7gb\]/\[7.9gb\], all\_pools {\[young\] \[172.4mb\]-\>\[46.5mb\]/\[199.6mb\]}{\[survivor\] \[24.9mb\]-\>\[0b\]/\[24.9mb\]}{\[old\] \[7.6gb\]-\>\[7.7gb\]/\[7.7gb\]}
> \[2014-11-07 17:25:46,387\]\[INFO \]\[monitor.jvm \] \[dvlp\_FRONTEND2\] \[gc\]\[old\]\[746\]\[32\] duration \[5s\], collections \[1\]/\[6s\], total \[5s\]/\[23.6s\], memory \[7.9gb\]-\>\[7.9gb\]/\[7.9gb\], all\_pools {\[young\] \[195mb\]-\>\[199.6mb\]/\[199.6mb\]}{\[survivor\] \[0b\]-\>\[10.9mb\]/\[24.9mb\]}{\[old\] \[7.7gb\]-\>\[7.7gb\]/\[7.7gb\]}
> \[2014-11-07 17:28:16,136\]\[WARN \]\[index.warmer \] \[dvlp\_FRONTEND2\] \[dvlp\_13\_67\_item\_20140410\]\[7\] failed to load fixed bitset for \[org.elasticsearch.index.search.nested.NonNestedDocsFilter@fd00879d\]
> org.elasticsearch.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
> at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
> at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
> at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
> at org.elasticsearch.index.cache.fixedbitset.FixedBitSetFilterCache.getAndLoadIfNotPresent(FixedBitSetFilterCache.java:139)
> at org.elasticsearch.index.cache.fixedbitset.FixedBitSetFilterCache.access$100(FixedBitSetFilterCache.java:75)
> at org.elasticsearch.index.cache.fixedbitset.FixedBitSetFilterCache$FixedBitSetFilterWarmer$1.run(FixedBitSetFilterCache.java:287)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at org.apache.lucene.util.FixedBitSet.\(FixedBitSet.java:187)
> at org.apache.lucene.search.MultiTermQueryWrapperFilter.getDocIdSet(MultiTermQueryWrapperFilter.java:104)
> at org.elasticsearch.common.lucene.search.NotFilter.getDocIdSet(NotFilter.java:49)
> at org.elasticsearch.index.search.nested.NonNestedDocsFilter.getDocIdSet(NonNestedDocsFilter.java:46)
> at org.elasticsearch.index.cache.fixedbitset.FixedBitSetFilterCache$2.call(FixedBitSetFilterCache.java:142)
> at org.elasticsearch.index.cache.fixedbitset.FixedBitSetFilterCache$2.call(FixedBitSetFilterCache.java:139)
> at org.elasticsearch.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
> at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
> at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
> at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
> at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
> ... 8 more
> \[2014-11-07 17:28:29,215\]\[INFO \]\[monitor.jvm \] \[dvlp\_FRONTEND2\] \[gc\]\[old\]\[749\]\[40\] duration \[22.9s\], collections \[4\]/\[2.3m\], total \[22.9s\]/\[1m\], memory \[7.9gb\]-\>\[7.9gb\]/\[7.9gb\], all\_pools {\[young\] \[199.5mb\]-\>\[199.6mb\]/\[199.6mb\]}{\[survivor\] \[22.9mb\]-\>\[23.1mb\]/\[24.9mb\]}{\[old\] \[7.7gb\]-\>\[7.7gb\]/\[7.7gb\]}
> \[2014-11-07 17:28:23,797\]\[WARN \]\[index.warmer \] \[dvlp\_FRONTEND2\] \[dvlp\_13\_67\_item\_20140410\]\[7\] failed to load fixed bitset for \[org.elasticsearch.index.search.nested.NestedDocsFilter@fd00879d\]
> org.elasticsearch.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
> at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
> at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
> at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
> at org.elasticsearch.index.cache.fixedbitset.FixedBitSetFilterCache.getAndLoadIfNotPresent(FixedBitSetFilterCache.java:139)
> at org.elasticsearch.index.cache.fixedbitset.FixedBitSetFilterCache.access$100(FixedBitSetFilterCache.java:75)
> at org.elasticsearch.index.cache.fixedbitset.FixedBitSetFilterCache$FixedBitSetFilterWarmer$1.run(FixedBitSetFilterCache.java:287)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at org.apache.lucene.util.FixedBitSet.\(FixedBitSet.java:187)
> at org.apache.lucene.search.MultiTermQueryWrapperFilter.getDocIdSet(MultiTermQueryWrapperFilter.java:104)
> at org.elasticsearch.index.search.nested.NestedDocsFilter.getDocIdSet(NestedDocsFilter.java:50)
> at org.elasticsearch.index.cache.fixedbitset.FixedBitSetFilterCache$2.call(FixedBitSetFilterCache.java:142)
> at org.elasticsearch.index.cache.fixedbitset.FixedBitSetFilterCache$2.call(FixedBitSetFilterCache.java:139)
> at org.elasticsearch.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
> at org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
> at org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
> at org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
> at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
> ... 8 more
> \`\`\`
Setting index.load\_fixed\_bitset\_filters\_eagerly to false fixed everything
for now.
I could argue that not running Gentoo in production is crazy, but it really
depends on your personal preferences 🙂
On Saturday, December 27, 2014 4:34:02 PM UTC+3, Nikolas Everett wrote:
> IcedTea isn't a JVM version. Give us `java -version`. It looks like that
> version of IcedTea could be OpenJDK 7u71 which is generally fine (we use it
> under plenty of loaf). It could also be jamvm or cacao or zero/shark. Those
> probably won't work. Lots of folks suggest oraclejdk so you may as well try
> like Joe says.
>
> You can try using jmap to get a heap histogram when memory is filling up
> and posting that somewhere.
>
> You may be able to downgrade. The worst that can happen is some indexes
> can open then you have to delete and rebuild them from source. This is only
> likely to work because you can't get it to boot.
>
> Also, running gentoo in production is either brave or crazy or both.
>
> Nik
> On Dec 26, 2014 3:52 PM, "Jean-Noël Rivasseau" \<[elv...@gmail.com](mailto:elv...@gmail.com)
> \\> wrote:
>
> > Hello,
> >
> > We had a 2 nodes, 10 shards cluster with about 150GB of data (not sure
> > how many documents, I would say about 150 millions since 1 doc is about 1K
> > for us). It worked fine under ES 1.3.4.
> >
> > We tried to update it today to 1.4.2. Now the nodes won't start. They eat
> > up all RAM during the recovery phase (when booting, and starting engines on
> > all indices), then GC occurs when no more RAM is available for ES. ES then
> > eventually gets stuck with OOM errors.
> >
> > I tried to increase the RAM given to ES to 30GB and it still eats all
> > that up and fails. Before with 1.3.4 we had 16GB allocated and no problems.
> >
> > Why ES needs all that RAM? It's doing nothing, not servicing a single
> > request... it's only starting the recovery process...
> >
> > We now have our production cluster down. It's really a huge problem for
> > us, please advise on any solutions or things we could try out.
> >
> > Java is IcedTea 7.2.5.3
> > OS is Gentoo Linux
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/7b5b4fb8-bbd6-4db2-ad10-8446ebd92140%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7b5b4fb8-bbd6-4db2-ad10-8446ebd92140%40googlegroups.com)
> > [https://groups.google.com/d/msgid/elasticsearch/7b5b4fb8-bbd6-4db2-ad10-8446ebd92140%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/7b5b4fb8-bbd6-4db2-ad10-8446ebd92140%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c3e4a74c-cd54-455b-ade2-dee980d1b457%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c3e4a74c-cd54-455b-ade2-dee980d1b457%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [December 27, 2014, 2:12pm UTC](https://discuss.elastic.co/t/update-to-es-1-4-2-gone-terribly-wrong-nodes-wont-start/21402/6 "2014-12-27T14:12:25Z")
Just my 2¢ - after battling with gentoo, you will come back to RHEL/Centos.
[http://www.reddit.com/r/linux/comments/1xizt0/production\_server\_environment\_gentoo\_or\_debian/](http://www.reddit.com/r/linux/comments/1xizt0/production_server_environment_gentoo_or_debian/)
Jörg
On Sat, Dec 27, 2014 at 2:37 PM, Jean-Noël Rivasseau [elvanor@gmail.com](mailto:elvanor@gmail.com)
wrote:
> I could argue that not running Gentoo in production is crazy, but it
> really depends on your personal preferences 🙂
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGpVWMXXMBReyajYM%2BDZ1qs\_Pu0dA1SO-wXwNLvBr%2BEhQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGpVWMXXMBReyajYM%2BDZ1qs_Pu0dA1SO-wXwNLvBr%2BEhQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@nik9000](https://discuss.elastic.co/u/nik9000)
#### Post date: [December 27, 2014, 2:35pm UTC](https://discuss.elastic.co/t/update-to-es-1-4-2-gone-terribly-wrong-nodes-wont-start/21402/7 "2014-12-27T14:35:07Z")
Yikes! I've seen a few people hit by that but I keep forgetting about it.
I have some kind of cognitive dissonance for nested docs.
On Dec 27, 2014 8:37 AM, "Jean-Noël Rivasseau" [elvanor@gmail.com](mailto:elvanor@gmail.com) wrote:
> We were being affected by
>
> [Large index no longer initialises under 1.4.0 and 1.4.0 Beta 1 due to OutOfMemoryException · Issue #8394 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/8394)
>
> Setting index.load\_fixed\_bitset\_filters\_eagerly to false fixed
> everything for now.
>
> I could argue that not running Gentoo in production is crazy, but it
> really depends on your personal preferences 🙂
>
> On Saturday, December 27, 2014 4:34:02 PM UTC+3, Nikolas Everett wrote:
>
> > IcedTea isn't a JVM version. Give us `java -version`. It looks like
> > that version of IcedTea could be OpenJDK 7u71 which is generally fine (we
> > use it under plenty of loaf). It could also be jamvm or cacao or
> > zero/shark. Those probably won't work. Lots of folks suggest oraclejdk so
> > you may as well try like Joe says.
> >
> > You can try using jmap to get a heap histogram when memory is filling up
> > and posting that somewhere.
> >
> > You may be able to downgrade. The worst that can happen is some indexes
> > can open then you have to delete and rebuild them from source. This is only
> > likely to work because you can't get it to boot.
> >
> > Also, running gentoo in production is either brave or crazy or both.
> >
> > Nik
> > On Dec 26, 2014 3:52 PM, "Jean-Noël Rivasseau" [elv...@gmail.com](mailto:elv...@gmail.com) wrote:
> >
> > > Hello,
> > >
> > > We had a 2 nodes, 10 shards cluster with about 150GB of data (not sure
> > > how many documents, I would say about 150 millions since 1 doc is about 1K
> > > for us). It worked fine under ES 1.3.4.
> > >
> > > We tried to update it today to 1.4.2. Now the nodes won't start. They
> > > eat up all RAM during the recovery phase (when booting, and starting
> > > engines on all indices), then GC occurs when no more RAM is available for
> > > ES. ES then eventually gets stuck with OOM errors.
> > >
> > > I tried to increase the RAM given to ES to 30GB and it still eats all
> > > that up and fails. Before with 1.3.4 we had 16GB allocated and no problems.
> > >
> > > Why ES needs all that RAM? It's doing nothing, not servicing a single
> > > request... it's only starting the recovery process...
> > >
> > > We now have our production cluster down. It's really a huge problem for
> > > us, please advise on any solutions or things we could try out.
> > >
> > > Java is IcedTea 7.2.5.3
> > > OS is Gentoo Linux
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > msgid/elasticsearch/7b5b4fb8-bbd6-4db2-ad10-8446ebd92140%
> > > [40googlegroups.com](http://40googlegroups.com)
> > > [https://groups.google.com/d/msgid/elasticsearch/7b5b4fb8-bbd6-4db2-ad10-8446ebd92140%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/7b5b4fb8-bbd6-4db2-ad10-8446ebd92140%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > .
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/c3e4a74c-cd54-455b-ade2-dee980d1b457%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c3e4a74c-cd54-455b-ade2-dee980d1b457%40googlegroups.com)
> > [https://groups.google.com/d/msgid/elasticsearch/c3e4a74c-cd54-455b-ade2-dee980d1b457%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/c3e4a74c-cd54-455b-ade2-dee980d1b457%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0f45RkB89w8uaWtPQnd-KXpvj%2BjG66dZh422Jxm2ifng%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0f45RkB89w8uaWtPQnd-KXpvj%2BjG66dZh422Jxm2ifng%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 12:41am UTC](https://discuss.elastic.co/t/update-to-es-1-4-2-gone-terribly-wrong-nodes-wont-start/21402/8 "2017-07-06T00:41:48Z")