Search two slow


(Willam Boss) #1

My index use 2T space, I have 10billion document, three nodes ,each node 4 cpus 13G mem , my match search is quite slow , it seem read io is use up , how to increase my search speed?


(David Pilato) #2

How many shards you have?
Could you start new nodes?

May I suggest you look at the following resources about sizing:

https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing

And https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right


(Willam Boss) #3

50 shards ,each shard use 46G space, about 3300 segments, I have no server for new node,do you have any suggestion?I think 10 billion document is the reason . there is 'hit' in result of restful query , if I want to only little records does elasticsearch search the whole index for count this 'hits'? and it only return 10 records each time , i think this is a waste of time , can elasticsearch remove 'hits' for speed up query ?
{ "took" : 61051, "timed_out" : false, "_shards" : { "total" : 50, "successful" : 50, "skipped" : 0, "failed" : 0 }, "hits" : { "total" : 2716157, "max_score" : 1.0,
like this hits total


(Christian Dahlqvist) #4

What is the use case? Are you continuously indexing into the index? Are you updating documents in the index? If so, what is the indexing/update rate?

What type of storage do you have?

What is the query that returns the response you provided?


(Willam Boss) #5

use for query match a filed with my own analyzer , I load all data once with bulk api, now I am not putting data into elastisearch , all my host are from azure cloud, storage should be ssd , read io almost 128M /second,query string like
"query":{"match":{"name":{"query":"ijmlajip","operator":"and"}}


(Christian Dahlqvist) #6

If you are not indexing or updating data, I would recommend you force merge the index down to a single segment. If you still seem limited by disk I/O, I would recommend looking into getting faster storage for the nodes, but I am not very familiar with what is available on Azure.


(Willam Boss) #7

you mean this ?
curl -X POST "es01:9200/myindex/_forcemerge?only_expunge_deletes=false&max_num_segments=100&flush=true"
now my index have 50 shards ,each shard has 20 segment , if I merge them to one segments I will have a big segment which take 46 GB space . is it too large ? are you sure this could improve search ?

set max_num_segments=1?


(Christian Dahlqvist) #8

It should. There are also additional recommendations provided here.


(Willam Boss) #9

how much speed will increase ? as you see my query take almost 60 seconds, If I merge all segments into one , can I compelete one simple query like
"query":{"match":{"name":{"query":"ijmlajip","operator":"and"}}
in 10 seconds?


(Christian Dahlqvist) #10

I do not know. As you have indicated storage likely being the limitation, I would recommend upgrading that. This is probably what will make the biggest difference. From what I have heard, Azure Premium Storage is recommended for I/O intensive workloads.


(Willam Boss) #11

each host has a limit of read io for 128m /second , if I have more hosts ,will search query be better? for index which take 2T space ,how many host should be ok ? I have 10 billion small documents ,do you really think 10 billion is not the bottleneck of my query ? if so how can i improve this ??

this is a profile of my query


(Christian Dahlqvist) #12

Elasticsearch typically performs a lot of small random reads during querying rather than large sequential ones, so fast storage that is able to handle this type of load is essential. Many throughput metrics for disks assume sequential reads, so might not be representative.

You need to test and benchmark. Have a look at the links David provided.

What type of Azure storage are you using at the moment?


(Willam Boss) #13

time for i in seq 1 1000; do

dd bs=4k if=/dev/sdd count=1 skip=$(( $RANDOM * 128 )) >/dev/null 2>&1;

done

real 0m1.931s
user 0m0.622s
sys 0m1.386s

the result show is ssd


(Christian Dahlqvist) #14

As I am not very familiar with Azure, that does not really tell me anything.


(Christian Dahlqvist) #15

I had a look at the Azure documentation about storage options. For Standard SSD disks (if that is what you are using) they state:

Standard SSD disks combine elements of Premium SSD disks and Standard HDD disks to form a cost-effective solution best suited for applications like web servers that do not need high IOPS on disks.

Elasticsearch definitely require high IOPS, which is why Premium SSD disks are recommended.


(Willam Boss) #16

merge segment quite slowly , can I reset some config to speed up this merge? just like "index.refresh_interval": "-1", "index.translog.durability": "async", "index.translog.sync_interval": "60s"
?


(Christian Dahlqvist) #17

No, I do not think you can speed this up as it is quite I/O intensive.


(Willam Boss) #18

thanks a lot ,your suggestions really help me ,my query response in ten seconds now


#19

Just passing by. Is it correct to say, to summarize, that the main improvement is to merge everything into one segment but it only works if you don't update the index?

In this case, if you have a realtime usecase and you make one index per day, it's actually a good idea to do a one-segment merge for all the past days indexes so the requests that spans across many indexes are faster?


(Christian Dahlqvist) #20

Yes, it can make searches faster, but also result in lower heap usage due to reduced need for global ordinals as outlined in this webinar.