How to increase efficiency of search queries in Elasticsearch

Elastic Stack Elasticsearch
1

I have made a Elasticsearch setup on a server with configuration - 64 GB RAM and 16 cores. I have created three nodes on this server. One node is both master and data and other two are data nodes. They all are using heap size of 10 GB each.

I have created an index in Elasticsearch which is having documents around 200K . Now when I am using a search query to retrieve all the data from that particular index it is taking around 20-30 seconds which is very huge.
Please suggest me what should I do to reduce this search time.

Thank you

2

Searching for matching documents is very different than extracting a full resultset.

Anyway what does your query look like. Which API are you using? Search? Scroll?

What is the use case?

3

Hi @dadoonet,

yes I'm extracting a full resultset. My query looks like this http://xxx.xx.x.xxx:9200/index-name/doc-type/_search?q=*&size=200000 . It is taking 20-30 seconds.

Actually this index contains my SQL data which i am using in my python code after every 10-15 seconds so i need to work fast

4

Why did you create 3 separate nodes on a host that size?

1 Like
5

Which version is that?
Unless you changed index settings, this query won't work in 6.x/7.x and may be 5.x.

6

I have searched for how to improve performance of Elasticsearch so I end up creating 3 nodes as of now. Is there any other way please suggest.

7

I'm using the latest version 7.1.1.
I have changed the settings for the index
PUT index-name/_settings
{
"max_result_window" : 200000
}
and it is working but it is taking too long. help me to reduce this time

8

You should use smaller size value like 1000 or 10000 and use the scroll API as documented.
Sorting by uid will also help the scroll API to run faster.

But doing that with search API is a bad practice.

9

Thanks for the suggestion. I have used scroll API this time but it is still taking same time to fetch the data for me.
concept of _uid is not there in 7.1 version see this https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-uid-field.html
so I have sorted it by _id but there is no change in the time taken.

10

If you also changed the number of shards to 1, then I guess that's the best you can have with your current hardware.

11

No of shards is already 1 on that particular index.
Do I need to change any other configuration for my Elasticsearch ?
What are the ideal configuration for setting up an Elasticsearch ? Maybe I have missed any.

12

Do you have SSD drives? Do you have only one nodes?

13

I am Using HDD right now with 3 nodes on it. One both master and data and two data nodes

14

So use SSD and that will be faster.
Then use only one node to make it even faster if really needed.

15

Thanks @dadoonet
SSD makes it faster but nodes creation has no effect. It is still taking 9 seconds but I want the result under 1 sec. Suggest something more.

16

What is "Node creation"? How is this related?

17

I just meant about no of Elasticsearch nodes .
No of nodes doesn't have any impact on search speed. Currently I am using one node and the speed is approx. same

18

How big is a document? What is its size?

19

When adding nodes is suggested in order to improve performance it is generally assumed these will run on different hardware thereby increasing the total amount of system resources available to the cluster. Having more than one node per host is generally only recommended when the host has more resources than s single node can efficiently utilise.

1 Like
20

My Index contains total 200k records of size 70mb