ECK performance optimisation on Kubernetes

Elastic Stack Elasticsearch
1

Can you pass on some recommendation on optimising elasticsearch cluster on Kubernetes.
Current Configuration :
3 different node groups hot-warm-cold
ES Nodegroup - 4 CPU 8Gib Mem /node
Kibana - 2 CPU 4 Gib Mem /node

Issue: Search is slow, data loading is slow and crashes even when we search for last 15 min.

Need recommendation on ES & Kibana cluster cpu mem limits etc.
Any recommendation on type of nodes ( instance types etc)

2

If you have more data in the cluster than can fit in the page cache, both indexing and query performance is often limited by storage performance. What type of storage are you using?

3

I’m using EBS volumes io1/gp3/sc1 types for respective nodes hot/warm/cold. I’m pretty much sure it’s not the volume issues as I’m using same types for even larger set of data.

4

How much data (in terms of shard count and GB) does each type of node hold? How much resources are allocated to each node type (is it 4CPU and 8GB RAM for all?)?

Do you have any monitoring data so you can track CPU usage, heap usage and IOPS/await for the different node types? (It would be good to check for resource constraints on all node types)

What type of searches are you running?

Are your searches targeting indices only on specific node types or indices across all node types?

What is the indexing throughput?

What exactly crashes?

Is there anything relevant in the Elasticsearch logs?

Which version of Elasticsearch are you using?

5

What type of searches are you running? "Text strings"

Are your searches targeting indices only on specific node types or indices across all node types? Only hot

What is the indexing throughput? How should i get that ?

What exactly crashes? Kibana UI

Is there anything relevant in the Elasticsearch logs? No, everything is normal there in logs.

Which version of Elasticsearch are you using? 8.12

6

What type of query clauses are you using? Is it normal text search or are you using query types that are known to be expensive and slow, e.g. wildcard queries?

Is there anything in the Kibana logs? What are you doing in Kibana when it crashes?

How are you targeting only the hot nodes? Do you have an alias you maintain that only links to indices on the hot nodes?

If you are querying an index pattern that matches indices on all node types all nodes will be involved in the query even if you are only searching for data from the last 15 minutes that will be located on the hot nodes.

Could you please answer this as well.