I'm using Kibana Logs UI to view my logs from my k8s pods. Although, when I do a simple query such as "kubernetes.pod.name: foo.bar" , I can check a high load on cluster and all Stack hangs.
Is that query doing a "match_all" documents from all matched indices? Is there way to deal with that?
(Other cached queries is ok).
Architecture: All my Stack is 7.4.2 (except Metricbeat 7.3.2) and running on k8s (Elastic Official Helm charts)
FIlebeat+Metricbeat (daemonsets) -> Elasticsearch
APM Agents -> APM Server -> Elasticsearch
Kibana -> Elasticsearch
- 03 hot nodes: 1TB SSD + 8CPU + 28GB RAM
- 03 warm nodes: 2TB SSD + 4CPU + 16GB RAM
- 03 shards - 0 Replica
- Active indices -> Hot nodes
- Read-only indices -> Warm nodes
- I'm using ILM (rollover indices to warm nodes with 60GB (20gb/shard) and delete them after 20 days)
Index rate is 25k/s