# How to increase efficiency of search queries in Elasticsearch

**URL:** <https://discuss.elastic.co/t/how-to-increase-efficiency-of-search-queries-in-elasticsearch/184951>\
**Category:** Elasticsearch\
**Created:** [June 10, 2019, 8:57am UTC](https://discuss.elastic.co/t/how-to-increase-efficiency-of-search-queries-in-elasticsearch/184951 "2019-06-10T08:57:09Z")\
**Posts on this page:** 19\
**Page:** 2

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 12, 2019, 4:20pm UTC](https://discuss.elastic.co/t/how-to-increase-efficiency-of-search-queries-in-elasticsearch/184951/21 "2019-06-12T16:20:55Z")

</div>

70mb in total?

What gives

```
GET /_cat/indices?v
```

---

<div class="post-metadata">

**Author:** ![avinash9999](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/avinash9999/32/91409_2.png) [@avinash9999](https://discuss.elastic.co/u/avinash9999)\
**Post date:** [June 13, 2019, 6:48am UTC](https://discuss.elastic.co/t/how-to-increase-efficiency-of-search-queries-in-elasticsearch/184951/22 "2019-06-13T06:48:31Z")

</div>

This is how it looks right now and I am fetching documents from second index

![Screenshot%20(2)](https://us1.discourse-cdn.com/elastic/original/3X/a/6/a692c96a559865655cb51758b5787c570d4e1f50.png)

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 13, 2019, 7:22am UTC](https://discuss.elastic.co/t/how-to-increase-efficiency-of-search-queries-in-elasticsearch/184951/23 "2019-06-13T07:22:13Z")

</div>

Please don't post images of text as they are hardly readable and not searchable.

Instead paste the text and format it with `</>` icon. Check the preview window.

Can you share the code you are using to extract all the data?

---

<div class="post-metadata">

**Author:** ![avinash9999](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/avinash9999/32/91409_2.png) [@avinash9999](https://discuss.elastic.co/u/avinash9999)\
**Post date:** [June 13, 2019, 9:22am UTC](https://discuss.elastic.co/t/how-to-increase-efficiency-of-search-queries-in-elasticsearch/184951/24 "2019-06-13T09:22:00Z")

</div>

I am using it in python

**first I made my connection**

`es = ES(host=host, port=port, timeout=100)`

**then I have search first 10000 documents**

```
scroll_data = es.search(
  index = index,
  scroll = '1m',
  size = 10000,
body={
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "language" : "3"
          }
        },
        {
          "match": {
            "statusid" : "5"
          }
        },
        {
          "range": {
            "cp": {
              "gte": "1"
            }
          }
        },
            { "wildcard": { "list" : "*17*" }}
      ]
    }
  }
})

```

\*\*then I have use scroll for the rest \*\*

```
sid = scroll_data['_scroll_id']
scroll_size = scroll_data['hits']['total']['value'] - size
final_scroll_data = []
final_scroll_data.extend(scroll_data['hits']['hits'])

while (scroll_size > 0):
    each_scroll = es.scroll(scroll_id = sid, scroll = '1m')
    sid = each_scroll['_scroll_id']
    scroll_size = scroll_size - len(each_scroll['hits']['hits'])
    final_scroll_data.extend(each_scroll['hits']['hits'])
```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 13, 2019, 9:37am UTC](https://discuss.elastic.co/t/how-to-increase-efficiency-of-search-queries-in-elasticsearch/184951/25 "2019-06-13T09:37:37Z")

</div>

You are using a wildcard query with wildcards at both ends, which is the most inefficient query you can run in Elasticsearch. What does CPU usage look like while the query is running? What is the specification of the host?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 13, 2019, 9:41am UTC](https://discuss.elastic.co/t/how-to-increase-efficiency-of-search-queries-in-elasticsearch/184951/26 "2019-06-13T09:41:29Z")

</div>

Yeah. This is weird:

```auto
{ "wildcard": { "list" : "*17*" }}

```

Look at the documentation: [Wildcard query | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-wildcard-query.html)

> Avoid beginning patterns with `*` or `?` . This can increase the iterations needed to find matching terms and slow search performance.

What gives:

```auto
GET your_index/_search
{
  "size": 1,
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "language" : "3"
          }
        },
        {
          "match": {
            "statusid" : "5"
          }
        },
        {
          "range": {
            "cp": {
              "gte": "1"
            }
          }
        },
            { "wildcard": { "list" : "*17*" }}
      ]
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![avinash9999](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/avinash9999/32/91409_2.png) [@avinash9999](https://discuss.elastic.co/u/avinash9999)\
**Post date:** [June 13, 2019, 9:49am UTC](https://discuss.elastic.co/t/how-to-increase-efficiency-of-search-queries-in-elasticsearch/184951/27 "2019-06-13T09:49:31Z")

</div>

@dadoonet, @Christian_Dahlqvist Thanks for the suggestion.

But with or without wildcard it is taking the time. It totally depends on data.

Right now I Have made two nodes on two different SSD machine and there is no performance improvement

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 13, 2019, 9:50am UTC](https://discuss.elastic.co/t/how-to-increase-efficiency-of-search-queries-in-elasticsearch/184951/28 "2019-06-13T09:50:26Z")

</div>

Can you share the output of the query I asked to run?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [June 13, 2019, 10:51am UTC](https://discuss.elastic.co/t/how-to-increase-efficiency-of-search-queries-in-elasticsearch/184951/29 "2019-06-13T10:51:12Z")

</div>

Given the small data size I would expect the data to be cached no matter what type of disk you have. What is CPU utilisation looking like? Do you have swap enabled? Are the other processes running on the host that could be interfering?

---

<div class="post-metadata">

**Author:** ![avinash9999](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/avinash9999/32/91409_2.png) [@avinash9999](https://discuss.elastic.co/u/avinash9999)\
**Post date:** [June 13, 2019, 11:58am UTC](https://discuss.elastic.co/t/how-to-increase-efficiency-of-search-queries-in-elasticsearch/184951/30 "2019-06-13T11:58:13Z")

</div>

> output of the query --

{  
"took" : 40,  
"timed\_out" : false,  
"\_shards" : {  
"total" : 1,  
"successful" : 1,  
"skipped" : 0,  
"failed" : 0  
},  
"hits" : {  
"total" : {  
"value" : 10000,  
"relation" : "gte"  
},  
"max\_score" : 4.0,  
"hits" : [  
{  
"\_index" : "sc-surveys2",  
"\_type" : "surveys",  
"\_id" : "140277170216",  
"\_score" : 4.0,  
"\_source" : {  
"@version" : "1",  
"qualificationanswerid" : 216,  
"description" : "Bio-Tech",  
"surveystatusid" : 5,  
"epc" : 0,  
"qualificationid" : 70,  
"qualificationanswerdesc" : "Bio-Tech",  
"id" : "140277170216",  
"supplierlist" : "17,603,554,28,27,623,581,307,101,126,30",  
"surveyid" : 1402771,  
"cpi" : 2.75,  
"languageid" : 3,  
"@timestamp" : "2019-06-13T09:35:59.449Z",  
"qualificationname" : "STANDARD\_INDUSTRY\_PERSONAL"  
}  
}  
]  
}  
}

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 13, 2019, 7:57pm UTC](https://discuss.elastic.co/t/how-to-increase-efficiency-of-search-queries-in-elasticsearch/184951/31 "2019-06-13T19:57:11Z")

</div>

What are now the first lines until hits if you change

```
 "size": 10000
```

---

<div class="post-metadata">

**Author:** ![avinash9999](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/avinash9999/32/91409_2.png) [@avinash9999](https://discuss.elastic.co/u/avinash9999)\
**Post date:** [June 14, 2019, 7:34am UTC](https://discuss.elastic.co/t/how-to-increase-efficiency-of-search-queries-in-elasticsearch/184951/32 "2019-06-14T07:34:21Z")

</div>

` when I run for-`

> "size": 10000

`it gives result like-`

> {  
> "took" : 344,  
> "timed\_out" : false,  
> "\_shards" : {  
> "total" : 1,  
> "successful" : 1,  
> "skipped" : 0,  
> "failed" : 0  
> },  
> "hits" : {  
> "total" : {  
> "value" : 10000,  
> "relation" : "gte"  
> },  
> "max\_score" : 4.0,  
> "hits" : [  
> {  
> "\_index" : "sc-surveys2",  
> "\_type" : "surveys",  
> "\_id" : "13943646059",

---

<div class="post-metadata">

**Author:** ![Pradyumna\_Achar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pradyumna_achar/32/99780_2.png) [@Pradyumna\_Achar](https://discuss.elastic.co/u/Pradyumna_Achar)\
**Post date:** [June 14, 2019, 4:25pm UTC](https://discuss.elastic.co/t/how-to-increase-efficiency-of-search-queries-in-elasticsearch/184951/33 "2019-06-14T16:25:09Z")

</div>

Just curious - the "took" represents the time in milliseconds right? So it is 344 milliseconds to retrieve 10,000 documents as opposed to the 9 seconds mentioned, or am I missing something?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 14, 2019, 5:49pm UTC](https://discuss.elastic.co/t/how-to-increase-efficiency-of-search-queries-in-elasticsearch/184951/34 "2019-06-14T17:49:02Z")

</div>

Exact. That's what I wanted to see.  
So the rest of time is most likely spent on the network I'd say.

---

<div class="post-metadata">

**Author:** ![avinash9999](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/avinash9999/32/91409_2.png) [@avinash9999](https://discuss.elastic.co/u/avinash9999)\
**Post date:** [June 16, 2019, 7:51pm UTC](https://discuss.elastic.co/t/how-to-increase-efficiency-of-search-queries-in-elasticsearch/184951/35 "2019-06-16T19:51:45Z")

</div>

okay thanks so is there anyway I can overcome this situation??

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 16, 2019, 8:13pm UTC](https://discuss.elastic.co/t/how-to-increase-efficiency-of-search-queries-in-elasticsearch/184951/36 "2019-06-16T20:13:43Z")

</div>

What if you run the extraction on the same machine elasticsearch is running?

---

<div class="post-metadata">

**Author:** ![avinash9999](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/avinash9999/32/91409_2.png) [@avinash9999](https://discuss.elastic.co/u/avinash9999)\
**Post date:** [June 17, 2019, 12:00pm UTC](https://discuss.elastic.co/t/how-to-increase-efficiency-of-search-queries-in-elasticsearch/184951/38 "2019-06-17T12:00:41Z")

</div>

when I run the extraction on same machine it is quick as compared when i run on other machine. But in my scenario I want to run the extraction from different machine.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [June 17, 2019, 12:45pm UTC](https://discuss.elastic.co/t/how-to-increase-efficiency-of-search-queries-in-elasticsearch/184951/39 "2019-06-17T12:45:21Z")

</div>

So we are now pretty sure that you have a network problem.  
I don't think that we can fix anything on elasticsearch side but at least you should investigate what kind of network connection you have between your client and elasticsearch server...

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 15, 2019, 12:45pm UTC](https://discuss.elastic.co/t/how-to-increase-efficiency-of-search-queries-in-elasticsearch/184951/40 "2019-07-15T12:45:25Z")

</div>

Closed on July 15, 2019, 12:45pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/how-to-increase-efficiency-of-search-queries-in-elasticsearch/184951.md?page=1)
