Kibana DevTools autocomplete refresh cause circuit breaker\ OOM on coordinators nodes

[val16384]

Hey guys,

We have issue with constantly increasing of Old gen part in JVM heap on coordinator nodes till OOM.

As I can understand the issue - memory leaking\OOM precisely correlated with requests to ES
that generated by Kibana DevTool (when autocomplete enabled and Console used by admin)
In audit requests looks like
GET /*/_mapping

For some reasons such request is too heavy for our cluster

Is there any way to

1. Understand why this "standard Kibana" request so heavy for our cluster ?
   ( too many indecies? too many fields ? )

2. Understand why ES coordinator nodes never free memory filled with this requests result data ?
   (old gen heap consumption grow till node OOM)

3. Disable autocomplete\refresh for all users of DevTools (including new) ?
   (I tried to set kibana.autocompleteTerminateAfter=1 , but without any luck )

Any thoughts/advice would be great.

Thanks

[Christian_Dahlqvist]

Which version are you using?

[val16384]

We are using 7.9.1 ES oss flavour

[Christian_Dahlqvist]

What is the full output of the cluster stats API? How much heap does the coordinator nodes have? What load is the cluster under?

[val16384]

Output { "_nodes" : { "total" : 33, "successful" : 33, "failed" : 0 - Pastebin.com

Currently we have two coordinator nodes (used only for Kibana\ Grafana) with 32 Gb of memory and 16 Gb heap ( G1GC ).
We had increased coordinators size when problem occurs for first time.

Almost all load are bulk indexing (logs) with small amount of searching.

But issue with GET /*/_mappings reproduces without external load (data ingested via Kafka , and we have ability to stop flow for some time )

I think the root cause can be related with one of our index . It ingested with data that regulary update (dynamic mapping used) theirs mappings (looks like it uses some sort of UUID as field key ) Mapping for this index can be huge .

Currently I seeking for way to stabilize cluster\coordinator nodes till we will improve situation with mappings.

May by there is any way to disable filed autocomplete refresh for all Dev Tools users ?

[Christian_Dahlqvist]

The output only seems to be partial. Could you please show the full output?

[val16384]

Sure. My mistake

{ "_nodes" : { "total" : 33, "successful" : 33, "failed" : 0 -...

[Christian_Dahlqvist]

I see that you seem to have a full set of OpenDistro plugins installed. I have no experience with any of these plugins and do not know whether they may have issues around memory management or what impact they have on heap usage, so would recommend you post a question to the OpenDistro community.

In order to identify whether any of these plugins are contributing to the problems you could try uninstalling them and see if that affects the situation. If the situation persisted on a vanilla Elasticsearch cluster you would probably be able to get more help from this forum.