Out of memory happens in elasticsearch node when Kibana connected

[Hugo_Ning (Hugo)]

Hi I'm using Kibana+ES. ES cluster contains 1 master and 2 clients.

And OOM happens in ES node when I connect kibana to one of them.

Heap dump shows that most of the memory is consumed by hundreds of byte(16.7M for each) :
POST /_xpack/monitoring/_bulk?system_id=kibana&system_api_version=6&interval=10000ms HTTP/1.1..content-type: application/x-ndjson..Host: ****..Content-Length: 1866..Connection: keep-alive....{"index":{"_type":"kibana_stats"}}.{"kibana":{"uuid":"****-****-****-****","name":"kibana","index":".kibana","host":"0","transport_address":"0:5601","version":"6.5.1","snapshot":false,"status":"green"},"usage":{"kql":{"optInCount":0,"optOutCount":0,"defaultQueryLanguage":"default-lucene"},"rollups":{"index_patterns":{"total":0},"saved_searches":{"total":0},"visualizations":{"total":0,"saved_searches":{"total":0}}},"index":".kibana","dashboard":{"total":0},"visualization":{"total":0},"search":{"total":0},"index_pattern":{"total":9},"graph_workspace":{"total":0},"timelion_sheet":{"total":0},"xpack":{"spaces":{"available":true,"enabled":true,"count":1},"reporting":{"available":true,"enabled":true,"browser_type":"chromium","_all":0,"csv":{"available":true,"total":0},"printable_pdf":{"available"

Call stack :

I'd really appreciate ideas how to solve this.

Thanks,

[Marius_Dragomir]

You should try disabling monitoring data collection in the settings first and then you could troubleshoot the issue easier.

[Hugo_Ning (Hugo)]

Hi @Marius_Dragomir, thank you for the reply,

I do disable monitoring feature now. But how should I troubleshoot the issue next since there will be no more OOM, do you have any suggestion ?

[shaunak (Shaunak Kashyap)]

It looks like you're running Kibana 6.5.1. Is your Elasticsearch of the same version too?

Exactly when and how often are you seeing the OOM error? Is it happening immediately after Kibana connects to Elasticsearch? Is it happening again, repeatedly — if so, at what frequency?

How much max heap (-Xmx) have you allocated to the ES node?

Finally, would it be possible for you to intercept the HTTP traffic between Kibana and the ES node and capture this large HTTP request body? If possible I'd like to see what's in it to inspect why it's so large.

Thanks,

Shaunak

[shaunak (Shaunak Kashyap)]

I can't say for sure yet but it might be that you are running into this issue: https://github.com/elastic/kibana/issues/32519.