Hello,
Our Production Elasticsearch Cluster nodes are constantly getting a Java
out of memory.
Another strange phenomenon is that we have 2 nodes which their JVM is
extremely high between 90%-99% , after a cluster restart they have still
high JVM Memory.
The Production Cluster has :
3 Master Nodes, type: m3.large
3 Nodata Nodes, type: m3.xlarge
23 Data Nodes, type: c3.2xlarge.
More details:
Nodes: 29 Indices: 16 Shards: 740 Data: 12.11 TB CPU: 452%Memory: 145.13 GB
/ 240.74 GB Version: 1.4.2
Here is the logs from the nodes:
[2015-02-03 01:28:01,237][WARN ][index.warmer ]
[es-prod-hist09] [2014_12][20] failed to load fixed bitset for
[org.elasticsearch.index.search.nested.NonNestedDocsFilter@300b168e]
org.elasticsearch.common.util.concurrent.ExecutionError:
java.lang.OutOfMemoryError: Java heap space
at
org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
at
org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at
org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at
org.elasticsearch.index.cache.fixedbitset.FixedBitSetFilterCache.getAndLoadIfNotPresent(FixedBitSetFilterCache.java:137)
at
org.elasticsearch.index.cache.fixedbitset.FixedBitSetFilterCache.access$100(FixedBitSetFilterCache.java:73)
at
org.elasticsearch.index.cache.fixedbitset.FixedBitSetFilterCache$FixedBitSetFilterWarmer$1.run(FixedBitSetFilterCache.java:278)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.util.FixedBitSet.(FixedBitSet.java:187)
at
org.elasticsearch.index.cache.fixedbitset.FixedBitSetFilterCache$2.call(FixedBitSetFilterCache.java:145)
at
org.elasticsearch.index.cache.fixedbitset.FixedBitSetFilterCache$2.call(FixedBitSetFilterCache.java:137)
at
org.elasticsearch.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
at
org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at
org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at
org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at
org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
... 8 more
[2015-02-03 01:28:22,780][WARN ][index.warmer ]
[es-prod-hist09] [2014_12][20] failed to load fixed bitset for
[org.elasticsearch.index.search.nested.NonNestedDocsFilter@300b168e]
org.elasticsearch.common.util.concurrent.ExecutionError:
java.lang.OutOfMemoryError: Java heap space
at
org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
at
org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at
org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at
org.elasticsearch.index.cache.fixedbitset.FixedBitSetFilterCache.getAndLoadIfNotPresent(FixedBitSetFilterCache.java:137)
at
org.elasticsearch.index.cache.fixedbitset.FixedBitSetFilterCache.access$100(FixedBitSetFilterCache.java:73)
at
org.elasticsearch.index.cache.fixedbitset.FixedBitSetFilterCache$FixedBitSetFilterWarmer$1.run(FixedBitSetFilterCache.java:278)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.lucene.util.FixedBitSet.(FixedBitSet.java:187)
at
org.elasticsearch.index.cache.fixedbitset.FixedBitSetFilterCache$2.call(FixedBitSetFilterCache.java:145)
at
org.elasticsearch.index.cache.fixedbitset.FixedBitSetFilterCache$2.call(FixedBitSetFilterCache.java:137)
at
org.elasticsearch.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)
at
org.elasticsearch.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at
org.elasticsearch.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at
org.elasticsearch.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
at
org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
... 8 more
[2015-02-03 01:28:57,635][WARN ][index.warmer ]
[es-prod-hist09] [2014_12][20] failed to load fixed bitset for
[org.elasticsearch.index.search.nested.NonNestedDocsFilter@300b168e]
org.elasticsearch.common.util.concurrent.ExecutionError:
java.lang.OutOfMemoryError: Java heap space
at
org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
at
org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at
org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at
org.elasticsearch.index.cache.fixedbitset.FixedBitSetFilterCache.getAndLoadIfNotPresent(FixedBitSetFilterCache.java:137)
at
org.elasticsearch.index.cache.fixedbitset.FixedBitSetFilterCache.access$100(FixedBitSetFilterCache.java:73)
at
org.elasticsearch.index.cache.fixedbitset.FixedBitSetFilterCache$FixedBitSetFilterWarmer$1.run(FixedBitSetFilterCache.java:278)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.OutOfMemoryError: Java heap space
[2015-02-03 01:29:12,392][WARN ][index.warmer ]
[es-prod-hist09] [2014_12][20] failed to load fixed bitset for
[org.elasticsearch.index.search.nested.NonNestedDocsFilter@300b168e]
org.elasticsearch.common.util.concurrent.ExecutionError:
java.lang.OutOfMemoryError: Java heap space
at
org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
at
org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
Thanks,
For the Help.
Costya.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c8cb9599-4cda-4252-be78-94412f137b45%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.