Circuit breaker always trips

Elastic Stack Elasticsearch
1

I have a cluster of 40 nodes. Today Elasticsearch started to return circuit breaker exceptions for all queries on two given nodes.
Even for this:

$ curl -s 'http://localhost:9200/'

{
   "error":{
      "root_cause":[
         {
            "type":"circuit_breaking_exception",
            "reason":"[parent] Data too large, data for [<http_request>] would be [13610582016/12.6gb], which is larger than the limit of [11885484441/11gb]",
            "bytes_wanted":13610582016,
            "bytes_limit":11885484441
         }
      ],
      "type":"circuit_breaking_exception",
      "reason":"[parent] Data too large, data for [<http_request>] would be [13610582016/12.6gb], which is larger than the limit of [11885484441/11gb]",
      "bytes_wanted":13610582016,
      "bytes_limit":11885484441
   },
   "status":503
}

How can I figure out what is the problem? Should the above log say how much memory that given query needs?
And if it shows that, what's going on? How can the above (and all other queries to those nodes) die with this error?
I see that the request breakers trip on those nodes. After cluster restart the same continues in some minutes.

If we reach the above situation, the node remains in the cluster but every operation to it fails.

Elasticsearch 5.6.4.

2

That looks weird.

Could you share your mappings?

3

I've sent it in a private message.
BTW, yesterday we started to use some new indexes with bigger docs (updated with scripts in bulk and returning the source in the bulk call), otherwise I don't know anything which has changed.
But I've already disabled that code and the problem still persists.
Bigger here means an enabled: false object and some (1-6) keys in the following structure:

4

I don’t understand why the circuit breaker complains with a match_all query.

@dakrone does this reminds you anything?

5

I've started to graph estimated breaker size:
image
According to the logs, the first entries are -so far- a repeated aggregation, which may be right, or not, I don't yet know:
[2017-11-24T21:52:55,425][WARN ][o.e.i.b.request ] [request] New used memory 10664646024 [9.9gb] for data of [<agg [messagesByFolders]>] would be larger than configured breaker: 10213706956 [9.5gb], breaking
[2017-11-24T21:52:56,049][WARN ][o.e.i.b.request ] [request] New used memory 11342156800 [10.5gb] for data of [<agg [messagesByFolders]>] would be larger than configured breaker: 10213706956 [9.5gb], breaking
[2017-11-24T21:52:56,666][WARN ][o.e.i.b.request ] [request] New used memory 11342156800 [10.5gb] for data of [<agg [messagesByFolders]>] would be larger than configured breaker: 10213706956 [9.5gb], breaking
[2017-11-24T21:52:57,824][WARN ][o.e.i.b.request ] [request] New used memory 11342156800 [10.5gb] for data of [<agg [messagesByFolders]>] would be larger than configured breaker: 10213706956 [9.5gb], breaking

But what I don't understand is why does it affect even a simple root query (http://localhost:9200/). Shouldn't breakers stop just the query which is over the allowed size? Why the estimated size grows linearly and denies all subsequent requests on the affected node(s) after some minutes?

Do I misunderstand the concept of breakers entirely?

6

I don’t know.

But for sure, I saw in you mappings that you disabled doc_values. Was there a reason for doing so?

7

Nothing more than saving space where we don't use them.

8

OK, so the questions seem to be:

  1. why does the request breakers' size constantly grow until it hits the limit and no query can get in? it's not a single query, but a bunch of queries (hence the steep curve on the graph, but it's not like a dirac delta).
    "breakers" : {
    "request" : {
    "limit_size_in_bytes" : 10187558092,
    "limit_size" : "9.4gb",
    "estimated_size_in_bytes" : 11343200256,
    "estimated_size" : "10.5gb",
    "overhead" : 1.0,
    "tripped" : 250
    },
  2. how could I see what queries account for these values?
  3. how can it be that the JVM heap size (reported by elastic) doesn't show this increased memory usage? (maybe a counter leak, it doesn't get decremented after a query finishes?)
  4. why is this localized to two nodes only (i will try to narrow it down to given indices, but it seems it happens only where the primary and replica shards of a given index are)

Also I think that the log/returned error (currently: [2017-11-24T21:52:55,425][WARN ][o.e.i.b.request ] [request] New used memory 10664646024 [9.9gb] for data of [<agg [messagesByFolders]>] would be larger than configured breaker: 10213706956 [9.5gb], breaking) should also include that individual query's estimated memory requirement, so if the estimated size grows by gigabytes every seconds, it may be traced better from the logs.

BTW, this is very similar to https://github.com/elastic/elasticsearch/issues/26943, except that I have a very rapid explosion in request breaker's size. (and doesn't have any plugins apart from icu)

9

OK, I've raised the breakers' limit to 2^63-1 bytes, it made the cluster stable.
image
So it seems this is a counter bug/leak, obviously no node can use 170 GiB of heap.

And I think I found the root cause.
Opened issue:

1 Like
10

Yep, that's definitely it. It's an issue where the stream is not closed, so the breaker isn't decrementing the accounting by the number of bytes. There's a PR open now to fix this.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.