Flush and clear cache unexpected behavior

Elastic Stack Elasticsearch
1

Hello,
for a long time now I've been trying to tune the heap size of our elasticsearch cluster.
I'm loading ~200g on a 2 node cluster , each with 8 cores + 60g Ram (heap size set to 30g).
ES version 1.6.0

While loading data the gc size gets smaller and smaller until it gets out of memory after ~100g is loaded.
I'm not running any queries therefore cache field & filter are empty.
Flush setting was not changed at all (default). Also cluster stats shows no cache is present.

As part of my attempts to figure this out I tried running both flush and clear cache separately, but it didn't do any good. Which make sense because there's nothing in the cache and flush is running by default setting.

But... when I ran flush and then clear cache, the heap size suddenly dropped from 28g to 2g!
I've tested it several times, also in version 1.4.3 and this behavior is consistent.
What I'm trying to understand is why this combination finally did the trick (as I said cache was empty),
and what settings should I use so that will not require manual execution. Because clearly memory is being
abused in some way...

Will appreciate any insights in this matter.

Thanks in advance
Oren

2

What sort of document structure is it?

3

Hi,
Please see below my template.

Also the exception I eventually get is:
java.lang.OutOfMemoryError: Java heap space
[2015-06-20 07:53:45,292][WARN ][index.warmer ] [es-userdata3.use.dynamicyield.com] [8765304_block14_1434780534546][1] failed to load fixed bitset for [org.elasticsearch.index.search.nested.NonNestedDocsFilter@edc74611]
org.elasticsearch.common.util.concurrent.ExecutionError: java.lang.OutOfMemoryError: Java heap space
at org.elasticsearch.common.cache.LocalCache$Segment.get(LocalCache.java:2201)
at org.elasticsearch.common.cache.LocalCache.get(LocalCache.java:3937)
at org.elasticsearch.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)
at org.elasticsearch.index.cache.fixedbitset.FixedBitSetFilterCache.getAndLoadIfNotPresent(FixedBitSetFilterCache.java:136)
at org.elasticsearch.index.cache.fixedbitset.FixedBitSetFilterCache.access$100(FixedBitSetFilterCache.java:73)
at org.elasticsearch.index.cache.fixedbitset.FixedBitSetFilterCache$FixedBitSetFilterWarmer$1.run(FixedBitSetFilterCache.java:277)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space

{
"template": "*",
"settings": {
	"index.cache.query.enable": true,
	"number_of_replicas": 0,
	"number_of_shards": 2
},
"mappings": {
	"user": {
		"dynamic": "strict",
		"_all": {
			"enabled": false
		},
		"properties": {
			"p1": {
				"type": "integer",
				"index": "not_analyzed",
				"norms": {
					"enabled": false
				},
				"doc_values": true
			},
			"p2": {
				"type": "integer",
				"index": "not_analyzed",
				"norms": {
					"enabled": false
				},
				"doc_values": true
			},
			"p3": {
				"type": "long",
				"index": "not_analyzed",
				"norms": {
					"enabled": false
				},
				"doc_values": true
			},
			"p4": {
				"type": "integer",
				"index": "not_analyzed",
				"norms": {
					"enabled": false
				},
				"doc_values": true
			},
			"p5": {
				"type": "long",
				"index": "not_analyzed",
				"norms": {
					"enabled": false
				},
				"doc_values": true
			},
			"p6": {
				"type": "string",
				"index": "not_analyzed",
				"norms": {
					"enabled": false
				},
				"doc_values": true
			},
			"p7": {
				"type": "string",
				"index": "not_analyzed",
				"norms": {
					"enabled": false
				},
				"doc_values": true
			},
			"p8": {
				"type": "string",
				"index": "not_analyzed",
				"norms": {
					"enabled": false
				},
				"doc_values": true
			},
			"p9": {
				"type": "string",
				"index": "not_analyzed",
				"norms": {
					"enabled": false
				},
				"doc_values": true
			},
			"p10": {
				"type": "string",
				"index": "not_analyzed",
				"norms": {
					"enabled": false
				},
				"doc_values": true
			},
			"p11": {
				"type": "nested",
				"properties": {
					"name": {
						"type": "string",
						"index": "not_analyzed",
						"norms": {
							"enabled": false
						},
						"doc_values": true
					},
					"count": {
						"type": "integer",
						"index": "not_analyzed",
						"norms": {
							"enabled": false
						},
						"doc_values": true
					}
				},
				"include_in_root": true
			},
			"p12": {
				"type": "nested",
				"properties": {
					"name": {
						"type": "string",
						"index": "not_analyzed",
						"norms": {
							"enabled": false
						},
						"doc_values": true
					},
					"count": {
						"type": "integer",
						"index": "not_analyzed",
						"norms": {
							"enabled": false
						},
						"doc_values": true
					}
				},
				"include_in_root": true
			},
			"p13": {
				"type": "nested",
				"properties": {
					"name": {
						"type": "string",
						"index": "not_analyzed",
						"norms": {
							"enabled": false
						},
						"doc_values": true
					},
					"count": {
						"type": "integer",
						"index": "not_analyzed",
						"norms": {
							"enabled": false
						},
						"doc_values": true
					}
				},
				"include_in_root": true
			},
			"p14": {
				"type": "nested",
				"properties": {
					"name": {
						"type": "string",
						"index": "not_analyzed",
						"norms": {
							"enabled": false
						},
						"doc_values": true
					},
					"count": {
						"type": "integer",
						"index": "not_analyzed",
						"norms": {
							"enabled": false
						},
						"doc_values": true
					}
				},
				"include_in_root": true
			},
			"p15": {
				"type": "nested",
				"properties": {
					"name": {
						"type": "string",
						"index": "not_analyzed",
						"norms": {
							"enabled": false
						},
						"doc_values": true
					},
					"count": {
						"type": "integer",
						"index": "not_analyzed",
						"norms": {
							"enabled": false
						},
						"doc_values": true
					}
				},
				"include_in_root": true
			},
			"p16": {
				"type": "nested",
				"properties": {
					"name": {
						"type": "string",
						"index": "not_analyzed",
						"norms": {
							"enabled": false
						},
						"doc_values": true
					},
					"count": {
						"type": "integer",
						"index": "not_analyzed",
						"norms": {
							"enabled": false
						},
						"doc_values": true
					}
				},
				"include_in_root": true
			},
			"p17": {
				"type": "nested",
				"properties": {
					"name": {
						"type": "string",
						"index": "not_analyzed",
						"norms": {
							"enabled": false
						},
						"doc_values": true
					},
					"property": {
						"type": "string",
						"index": "not_analyzed",
						"norms": {
							"enabled": false
						},
						"doc_values": true
					},
					"event": {
						"type": "string",
						"index": "not_analyzed",
						"norms": {
							"enabled": false
						},
						"doc_values": true
					},
					"count": {
						"type": "string",
						"index": "not_analyzed",
						"norms": {
							"enabled": false
						},
						"doc_values": true
					}
				},
				"include_in_root": true
			}
		}
	}
}

}

4

Just came across this thread:

Not trying another run with:
index.load_fixed_bitset_filters_eagerly: false

Thanks,

5

Just to update in case anyone else will face this issue.
Turns out that due to nested objects in my mapping, filter bit set gets build during indexing and wasting a lot of heap memory. Setting index.load_fixed_bitset_filters_eagerly to false fixed this problem.

Now I'm testing the performance impacts that this setting will have.
According to the thread I shared above, setting this to false will not prevent the heap usage when nested queries will be executed, but for now, my nested queries (and I use filters) don't take as much memory as this setting took by default.

Thanks,