We have issue with constantly increasing of Old gen part in JVM heap on coordinator nodes till OOM.
As I can understand the issue - memory leaking\OOM precisely correlated with requests to ES
that generated by Kibana DevTool (when autocomplete enabled and Console used by admin)
In audit requests looks like GET /*/_mapping
For some reasons such request is too heavy for our cluster
Is there any way to
Understand why this "standard Kibana" request so heavy for our cluster ?
( too many indecies? too many fields ? )
Understand why ES coordinator nodes never free memory filled with this requests result data ?
(old gen heap consumption grow till node OOM)
Disable autocomplete\refresh for all users of DevTools (including new) ?
(I tried to set kibana.autocompleteTerminateAfter=1 , but without any luck )
Currently we have two coordinator nodes (used only for Kibana\ Grafana) with 32 Gb of memory and 16 Gb heap ( G1GC ).
We had increased coordinators size when problem occurs for first time.
Almost all load are bulk indexing (logs) with small amount of searching.
But issue with GET /*/_mappings reproduces without external load (data ingested via Kafka , and we have ability to stop flow for some time )
I think the root cause can be related with one of our index . It ingested with data that regulary update (dynamic mapping used) theirs mappings (looks like it uses some sort of UUID as field key ) Mapping for this index can be huge .
Currently I seeking for way to stabilize cluster\coordinator nodes till we will improve situation with mappings.
May by there is any way to disable filed autocomplete refresh for all Dev Tools users ?
I see that you seem to have a full set of OpenDistro plugins installed. I have no experience with any of these plugins and do not know whether they may have issues around memory management or what impact they have on heap usage, so would recommend you post a question to the OpenDistro community.
In order to identify whether any of these plugins are contributing to the problems you could try uninstalling them and see if that affects the situation. If the situation persisted on a vanilla Elasticsearch cluster you would probably be able to get more help from this forum.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.