Coordinate + kibana heap issues

[d4nnyx]

Hello,

after Kibana upgrade to v7.12 Im experiencing issues with underlying coordinate-only node. It regularly crashes because of low heap memory. Before upgrade whole cluster worked like a charm.

Cluster setup
3 master nodes - 8CPU, 16GB RAM per node
12 data nodes (ingests and logstash output points here) - 16CPU, 32GB RAM, 2TB disk per node
2 coordinate nodes + Kibana - 8CPU, 16GB RAM per node
Indexing avg rate 60K/s
Search avg rate 20/s (yeah, really only 20 per second)

Should coordinate node have more resources? Or is it meant to by more like a lightweight support node? We use coordinates node only to load balance Kibana requests.

"Crash situation", according to log output, begins with Kibana cannot obtain licence info, then plugins gets to red state and then Kibana stops respond because of Elastic which is after that out of memory.

Could be this problem connected to migration to new kibana system index in this version?

I'll be greatful for any advice.

[rcowart (Rob Cowart)]

The coordinating node is responsible for the gather phase of the query. So it needs to have enough resources to retrieve the query results from the data nodes and combine them into a single response. So the coordinating node does require a bit more resources, as it is more than a simple load balancer.

I assuming the resources above are physical resources for the entire node, not just the heap size for each node. What are your heap sizes?

[d4nnyx]

Yeah, I completely forgot to mention heap size
We stick to rule of thumb and give the half memory to heap.

Master 8GB heap
Data 16GB heap
Coord 8GB heap

[rcowart (Rob Cowart)]

That is reasonable heap for a typical coordinating node. I would suspect the issue to be elsewhere.

[d4nnyx]

This problem has showed up after upgrade to 7.12, so I would suspect some internal change in this version.