How to control content of accounting requests circuit breaker

Hi,

I am running into issues with circuit breakers during indexing-heavy operations: it seems the accounting circuit breaker is filling up and not getting cleaned quickly enough.

[2018-09-04T12:19:05,954][WARN ][r.suppressed             ] path: /_bulk, params: {timeout=1m}
org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for 
[<http_request>] would be [4487528104/4.1gb], which is larger than the limit of [4485310054/4.1gb], 
usages [request=0/0b, fielddata=0/0b, in_flight_requests=7102543/6.7mb, 
accounting=4480425561/4.1gb]

From what I understand in the docs, the accounting requests circuit breaker is for resources from requests that have ended but have not been released on server-side.

I would like to know if there is a way to somehow force the release or control the content of the accounting requests circuit breaker. So far restarting the service has not helped. Increasing the amount of RAM has helped, but it is not a viable long-term solution for me.

Thanks.

What is the full output of the cluster stats API?

This is the current output:
{ "_nodes" : { "total" : 1, "successful" : 1, "failed" : 0 }, "cluster_name" : "elasticsearch", "timestamp" : 1536145181580, "status" : "yellow", "indices" : { "count" : 10, "shards" : { "total" : 128, "primaries" : 128, "replication" : 0.0, "index" : { "shards" : { "min" : 5, "max" : 20, "avg" : 12.8 }, "primaries" : { "min" : 5, "max" : 20, "avg" : 12.8 }, "replication" : { "min" : 0.0, "max" : 0.0, "avg" : 0.0 } } }, "docs" : { "count" : 15469398375, "deleted" : 18098506 }, "store" : { "size" : "1.8tb", "size_in_bytes" : 1979635604166 }, "fielddata" : { "memory_size" : "0b", "memory_size_in_bytes" : 0, "evictions" : 0 }, "query_cache" : { "memory_size" : "0b", "memory_size_in_bytes" : 0, "total_count" : 0, "hit_count" : 0, "miss_count" : 0, "cache_size" : 0, "cache_count" : 0, "evictions" : 0 }, "completion" : { "size" : "0b", "size_in_bytes" : 0 }, "segments" : { "count" : 3074, "memory" : "4.7gb", "memory_in_bytes" : 5110311136, "terms_memory" : "3.6gb", "terms_memory_in_bytes" : 3867356463, "stored_fields_memory" : "681mb", "stored_fields_memory_in_bytes" : 714101608, "term_vectors_memory" : "0b", "term_vectors_memory_in_bytes" : 0, "norms_memory" : "45.3kb", "norms_memory_in_bytes" : 46464, "points_memory" : "499.6mb", "points_memory_in_bytes" : 523955201, "doc_values_memory" : "4.6mb", "doc_values_memory_in_bytes" : 4851400, "index_writer_memory" : "0b", "index_writer_memory_in_bytes" : 0, "version_map_memory" : "0b", "version_map_memory_in_bytes" : 0, "fixed_bit_set" : "0b", "fixed_bit_set_memory_in_bytes" : 0, "max_unsafe_auto_id_timestamp" : -1, "file_sizes" : { } } }, "nodes" : { "count" : { "total" : 1, "data" : 1, "coordinating_only" : 0, "master" : 1, "ingest" : 1 }, "versions" : [ "6.4.0" ], "os" : { "available_processors" : 4, "allocated_processors" : 4, "names" : [ { "name" : "Linux", "count" : 1 } ], "mem" : { "total" : "15.5gb", "total_in_bytes" : 16657776640, "free" : "167mb", "free_in_bytes" : 175140864, "used" : "15.3gb", "used_in_bytes" : 16482635776, "free_percent" : 1, "used_percent" : 99 } }, "process" : { "cpu" : { "percent" : 87 }, "open_file_descriptors" : { "min" : 1579, "max" : 1579, "avg" : 1579 } }, "jvm" : { "max_uptime" : "22.1h", "max_uptime_in_millis" : 79873438, "versions" : [ { "version" : "1.8.0_171", "vm_name" : "OpenJDK 64-Bit Server VM", "vm_version" : "25.171-b10", "vm_vendor" : "Oracle Corporation", "count" : 1 } ], "mem" : { "heap_used" : "5.8gb", "heap_used_in_bytes" : 6275777672, "heap_max" : "7.9gb", "heap_max_in_bytes" : 8555069440 }, "threads" : 66 }, "fs" : { "total" : "3tb", "total_in_bytes" : 3298534883328, "free" : "822.8gb", "free_in_bytes" : 883570180096, "available" : "822.8gb", "available_in_bytes" : 883570180096 }, "plugins" : [ ], "network_types" : { "transport_types" : { "security4" : 1 }, "http_types" : { "security4" : 1 } } } }

It looks like you have a single node with 8GB of heap configured. If you look at the statistics, you can see that segment memory takes up a full 4.7GB of that:

		"segments": {
			"count": 3074,
			"memory": "4.7gb",
			"memory_in_bytes": 5110311136,
			"terms_memory": "3.6gb",
			"terms_memory_in_bytes": 3867356463,
			"stored_fields_memory": "681mb",
			"stored_fields_memory_in_bytes": 714101608,
			"term_vectors_memory": "0b",
			"term_vectors_memory_in_bytes": 0,
			"norms_memory": "45.3kb",
			"norms_memory_in_bytes": 46464,
			"points_memory": "499.6mb",
			"points_memory_in_bytes": 523955201,
			"doc_values_memory": "4.6mb",
			"doc_values_memory_in_bytes": 4851400,
			"index_writer_memory": "0b",
			"index_writer_memory_in_bytes": 0,
			"version_map_memory": "0b",
			"version_map_memory_in_bytes": 0,
			"fixed_bit_set": "0b",
			"fixed_bit_set_memory_in_bytes": 0,
			"max_unsafe_auto_id_timestamp": -1,
			"file_sizes": {}
		}
	}

Given the current workload, it seems the current heap is too small as you are hitting circuit breakers. You may be able to save memory by optimising mappings, but it is also possible that you may have reached the limit for what your node can handle given your data and that heap size and the headroom you need to handle indexing and querying.

I see. So the segments/terms are kept in memory all the time and cannot be somehow unloaded when not in use? If I spread the data across multiple indexes (on the same node) and close some indexes when they are not needed, would their segments still take up memory?

Closed indices do not take up memory, but are then also not searchable unless you first explicitly open them.

Alright. Thank you.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.