Hi,
Each day elasticsearch creates index from two different devices. From the first device I get results really, really slow and even sometimes I got timeout error, from the second one it's still slow but I do not receive any error. They are both on the same machines, and are similar in docs count and size, 1) 107294970 180.8gb, 2) 143319153 202.9gb. The main difference is in fields number 433 vs 204. Could you suggest me what could be wrong, and how to fix this? Maybe I should use more shards per index? Currently there are 2 mains and 2 replicas.
Below there are stats from one of the indexes (the same day) per device after commiting search for 7 days.
Sorry for the external source but i was unable to paste it all in here.
1)
https://pastebin.com/jdAC0dRu
What kind of searches are you running? What is the indexing and query load? What is the specification of your cluster? What type of storage do you have?
Hi,
I think that query is not import, I even get timeout in simple kibana empty query (longer than 7 days).
Cluster specifications:
all virtual, 8 data node - 20cpu and 60GB ram each
Kibana requests query are through coordinator node. None of the cluster member use more than 75% of JVM memory.
Elasticsearch version: 6.7.2
What type of storage are you using?
SSD on 4 first nodes and HDD on the rest, the first ones are keeping 2 days of logs
There are other indexes on this cluster but for some reason, the trouble one is just this one.
What is the result if you query just the last 2 days or day 2 to 7? What does disk IO on the nodes backed by HDD look like while querying?
Ok, why do you think that there is an issue with the IOs?
Like I said, there are few similar indexes on this cluster and only one creates issues with searches longer that 7 days.
I am looking to eliminate common causes and disk I/O or iowait is very common. It could also be that the shard for this index is too large for the data you have and the queries you are running. Each query runs single-threaded against each shard so size will affect latency.
So the stats said nothing important in this case?
I could see that you have throttling going on which indicate slow storage.
It's really hard to accept, that there is problem only with this one index.. There are others, bigger in term of doc count and GB. The only real difference is the field number.
You have provided very little information to go on so it is very hard to tell. Is there anything else that sets this index apart from the others, e.g. different types of queries, document size, higher query volume etc?
It would also help to know if there is any difference in latency beteen the data store on the different types of nodes. If there is not at least we have ruled that out.
Are there any commands that I could execute for this indices so they could help improve this ticket?
Can you start by comparing query latency against 2 days worth of data on the different node types? Are you running queries or Kibana dashboards? If queries, could you try using the search profile API?
Like I said, I even get timeout with simple/empty query
1 index
SSD {
"took" : 292,
"timed_out" : false,
"_shards" : {
"total" : 2,
"successful" : 2,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 101519698,
HDD {
"took" : 191,
"timed_out" : false,
"_shards" : {
"total" : 2,
"successful" : 2,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 111305720,
2 index
SSD{
"took" : 410,
"timed_out" : false,
"_shards" : {
"total" : 2,
"successful" : 2,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 139194459,
HDD{
"took" : 417,
"timed_out" : false,
"_shards" : {
"total" : 2,
"successful" : 2,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 165511168,
What does that query look like? Based on that data it seems it executed quite quickly.
It's empty, in this api empty queries doesn't generate any errors, empty (only counting) query does
Can you show us the exact query? As the number of fields in the index seems to matter this could be explained if you were running a query string query that triggered searching against all_fields.
It's literally empty
GET abc-2019.06.*/_search
no errors
and
kibana -> discover -> index pattern (same time) timeout