How to speed up query performance?


(張皓翔) #1

Hi, I have 4 nodes in a cluster. I ingest netflow through ELK 40~50G data per day with 3 replica shards, and query about 3000 times per second. I found the query perfomance is not good. I am trying to find out the bottle neck.Now it almost doesn't have Garbage collection overhead.
(1)query syntax
(2)hardware limit
(3)shards allocation
(4)distribute the query node not specified one node to query.

I want to know which one is the main reason.



thank you in advance!


(Robert Cowart) #2
  1. Judging by the memory values, those might be really small nodes for that amount data.
  2. What kind of storage is being used. Spinning disk or network attached storage is bad. You will need local SSDs.
  3. How many shards per index. I would recommend 4 shards and only 1 replica.
  4. Do NOT use location awareness. This will cause Elasticsearch to "prefer local shards" and you will see that only 1 or 2 nodes have all of the query load, while the others sit idle. Eliminating location awareness allows the query load to be spread more evenly across the cluster. Unfortunately I have found no way to disable "prefer local shards" when location awareness is enabled. :frowning:

(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.