Search latency & index latency for elasticsearch

Elastic Stack Elasticsearch
1

We are continuously getting latency in both search & indexing. We are having a cluster of 15 nodes with 3 master eligible & rest 12 are data nodes each having 30 GB RAM and we have a traffic of 300 concurrent users. Any suggestions by which I can reduce the latency?

Here is the screenshot of kibana monitoring:

kibana_logs
kibana_logs.png1920×1080 188 KB

2

Hi,

if I see this correctly on the charts the highest reported latency for indexing is around 2.5 milliseconds and for search it spikes to roughly 12 milliseconds. Is a latency of 12 milliseconds problematic in practice or are you asking out of curiosity? Latency on this level can be caused by a lot of factors and you'd need to have a very heavily tuned system (including kernel and JVM tuning) to reduce those spikes. Some examples include:

  • Garbage collection pauses (in the range of tens of milliseconds but you should expect multi-second spikes occasionally as well)
  • Backpressure / blocking due to heavy disk I/O
  • CPU scheduling pressure / threads moved to different CPU cores
  • Kernel defragmenting memory
  • Network backpressure / dropped packets
  • ...

Daniel