Performance of search requests in cluster from Kibana

[Evgeniy_Ivlev]

Hi!

I deployed new ES cluster with following configuration:
3 master nodes (4Gb RAM per node)
2 data nodes (24Gb RAM per node), Xmx 12G
1 node for Kibana (4 Gb RAM)

All nodes are virtual machines.
lscpu shows following hardware info:
CPU(s): 4
Model name: Intel(R) Xeon(R) CPU E5-2697A v4 @ 2.60GHz
CPU MHz: 2599.998
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 40960K

I'm going to use cluster for analytics system. I have following index configuratiuon:
Every day 12 new indexes are created from ES template.
The name of index is rt-geoID-YYYY-MM-DD
number of shards: 1
number of replicas: 1
Each index has size 1-8 Gb

Data source is connected to DataNode2, Kibana is connected to DataNode1.
There are no problems with data write and indexsing.
But there are large problems with speed of search. Look at image below.
When user performs search request from kibana by pattern rt-* for sevaral days, only DataNode2 has huge % utulization of disk.
I expect search request will be distributed between 2 data nodes.

Also I noticed If I search something by keyword field, first request is very long (Up to 90 sec), second one can be 2 times faster and more. Is there a way to cache something in order to perform first request fast?

[Christian_Dahlqvist]

Which version of Elasticsearch are you using? This may be related to the issue discussed in this thread, which resulted in this GitHub issue. It looks like this should have been fixed in 6.0.

[Evgeniy_Ivlev]

5.6.4 version of ES and Kibana

[Evgeniy_Ivlev]

I'd like to understand - if it possible to serve single request by 2 nodes? Or in the best case different request may be served by different nodes?

[Christian_Dahlqvist]

A single request can be served by shards on both nodes, leading to the query load being spread out. It seems, based on the GitHub issue I linked to, that Kibana adds preference, which under some circumstances can lead to happening properly.