# How to calculate search time spent by Elasticsearch on its inverted index alone?

**URL:** <https://discuss.elastic.co/t/how-to-calculate-search-time-spent-by-elasticsearch-on-its-inverted-index-alone/232962>\
**Category:** Elasticsearch\
**Created:** [May 16, 2020, 11:15pm UTC](https://discuss.elastic.co/t/how-to-calculate-search-time-spent-by-elasticsearch-on-its-inverted-index-alone/232962 "2020-05-16T23:15:49Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![kmdabdulla](https://avatars.discourse-cdn.com/v4/letter/k/f9ae1b/32.png) [@kmdabdulla](https://discuss.elastic.co/u/kmdabdulla)\
**Post date:** [May 16, 2020, 11:15pm UTC](https://discuss.elastic.co/t/how-to-calculate-search-time-spent-by-elasticsearch-on-its-inverted-index-alone/232962/1 "2020-05-16T23:15:50Z")

</div>

Took field gives overall search time but is there any way to find the time elasticsearch spends only on its index alone?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [May 17, 2020, 9:26am UTC](https://discuss.elastic.co/t/how-to-calculate-search-time-spent-by-elasticsearch-on-its-inverted-index-alone/232962/2 "2020-05-17T09:26:52Z")

</div>

Have you looked at the [explain API](https://www.elastic.co/guide/en/elasticsearch/reference/7.7/search-explain.html)?

---

<div class="post-metadata">

**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\
**Post date:** [May 18, 2020, 5:16pm UTC](https://discuss.elastic.co/t/how-to-calculate-search-time-spent-by-elasticsearch-on-its-inverted-index-alone/232962/3 "2020-05-18T17:16:50Z")

</div>

In addition to Explain, the [Profile API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-profile.html) can be helpful too.

It's a little more complicated than just the time taken by the inverted index, since in reality there are many steps to interacting with the index (advancing iterators to documents, scoring, two-phase iteration for things like phrases, etc). But the query profiler should give you some more insight.

Do note that the profiler adds significant overhead to query execution, so times should only be looked at in a relative manner. And concurrent execution across several shards can make reading the results tricky. E.g. overall wall-clock `took` time might be 15s, but looking at the profile results you see 10 shards that each took 10s. This indicates that some shards were executing concurrently (otherwise the `took` time would be 100s), and some sequentially (otherwise the `took` time would be 10s)

---

<div class="post-metadata">

**Author:** ![kmdabdulla](https://avatars.discourse-cdn.com/v4/letter/k/f9ae1b/32.png) [@kmdabdulla](https://discuss.elastic.co/u/kmdabdulla)\
**Post date:** [May 28, 2020, 8:33pm UTC](https://discuss.elastic.co/t/how-to-calculate-search-time-spent-by-elasticsearch-on-its-inverted-index-alone/232962/4 "2020-05-28T20:33:57Z")

</div>

Sorry for getting back late. I will definitely try this. Thanks for your time.

---

<div class="post-metadata">

**Author:** ![kmdabdulla](https://avatars.discourse-cdn.com/v4/letter/k/f9ae1b/32.png) [@kmdabdulla](https://discuss.elastic.co/u/kmdabdulla)\
**Post date:** [May 28, 2020, 8:35pm UTC](https://discuss.elastic.co/t/how-to-calculate-search-time-spent-by-elasticsearch-on-its-inverted-index-alone/232962/5 "2020-05-28T20:35:05Z")

</div>

I am sorry for getting back late. Thanks for your comment. Will try it !

---

<div class="post-metadata">

**Author:** ![kmdabdulla](https://avatars.discourse-cdn.com/v4/letter/k/f9ae1b/32.png) [@kmdabdulla](https://discuss.elastic.co/u/kmdabdulla)\
**Post date:** [June 1, 2020, 8:29pm UTC](https://discuss.elastic.co/t/how-to-calculate-search-time-spent-by-elasticsearch-on-its-inverted-index-alone/232962/6 "2020-06-01T20:29:29Z")

</div>

I have indexed 2 million documents and I am trying to return all the matching document ids at once. and I use PHP client.

My mapping is as follows:

```
$params = [
    'index' => $index,
    'body' => [
        'settings' => [
            "number_of_shards" => 1,
            "number_of_replicas" => 0,
            "index.queries.cache.enabled" => false,
            "index.soft_deletes.enabled" => false,
            "index.refresh_interval" => -1,
            "index.requests.cache.enable" => false,
            "index.max_result_window"=> $result_window
        ],
        'mappings' => [
            '_source' => [
                "enabled" => false
             ],
             'properties' => [
                "text" => [
                        "type" => "text",
                        "index_options" => "docs"
                ]
        ]
     ]
    ]
];

```

My query string is as follows:

```
$json = '{
"from" : 0, "size" : '.$size.',
        "profile": true,
"query": {
    "bool": {
      "filter" : {
        "match" : {
            "text" : {
            "query" : "justin trump clinton harry",
            "operator" : "and"
            }
        }
    }
}
}
}';

```

The goal is to get all the matching documents at once. I need only document ids (check whether the given term exists in a document or not only) so I used index\_options as docs. I understand about scroll API but I want to use max\_result\_window. I am using only one shard, no replicas and I also avoided scoring of documents when I perform search operation.

My questions are as follows:

1. I want to retrieve only document ids and avoid document fetch phase, so I disabled the source field. To avoid other metadata, I tried the following as per this link [avoid fetch phase](https://github.com/elastic/elasticsearch/issues/17159). But I can still see document type and index name. Is there anything I need to do to get only document ids and avoid the fetch phase?

```auto
   "stored_fields": "_none_",
    "docvalue_fields": ["_id"]

```

1. Since I am retrieving all the matching documents scoring is irrelevant to me so I used filter clause but I was wondering why I am getting boostquery timing in profile API results below?. But you can also note that Booleanquery score timing is zero!

2. In order to know how much time Boolean query search took on Lucene index alone, should I just take the time reported by the Boolean query or do I need to add up all its children (term query) timings? Because when I add all those term query timings it is higher than the one reported by Boolean query. Any possible reason for this?

3. Do I need to include collector as well for my Boolean query timing, Because in [profile api](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-profile.html), it is said that "Lucene works by defining a "Collector" which is responsible for coordinating the traversal, scoring, and collection of matching documents. ". It also says that " It should be noted that Collector times are **independent** from the Query times. They are calculated, combined, and normalized independently! Due to the nature of Lucene’s execution, it is impossible to "merge" the times from the Collectors into the Query section, so they are displayed in separate portions". As for my understanding, it helps in traversing the postings list of Lucene index to execute Boolean query operation. Am I right in this regard?

4. Is there any similar API for investigating the indexing time in elasticsearch?. I was able to get indexing time in settings API but I am looking somethings similar to profile API.?

I apologize for having too much loaded questions. I will highly appreciate your help in this regard. I hope I have provided necessary information. Please let me know if you need anything.  
Please find the profile API output in the following reply!  
Thank You!

---

<div class="post-metadata">

**Author:** ![kmdabdulla](https://avatars.discourse-cdn.com/v4/letter/k/f9ae1b/32.png) [@kmdabdulla](https://discuss.elastic.co/u/kmdabdulla)\
**Post date:** [June 1, 2020, 8:33pm UTC](https://discuss.elastic.co/t/how-to-calculate-search-time-spent-by-elasticsearch-on-its-inverted-index-alone/232962/7 "2020-06-01T20:33:41Z")

</div>

```
Array                                                                                                                                  
(                                                                                                                                 
    [shards] => Array                                                                                                
        (                                                                                                               
            [0] => Array                                                                                                    
                (                                                                                                   
                    [id] => [tod2gbVKSRGinZVfdXTmxA][elasticindex-2][0]                                 
                    [searches] => Array                                                                          
                        (                                                                                     
                            [0] => Array                                                                                  
                                (                                                                                   
                                    [query] => Array                                                             
                                        (                                                                            
                                            [0] => Array                                                       
                                                (                                                                    
                                                    [type] => BoostQuery                                            
                                                    [description] => (ConstantScore(+text:justin +text:trump +text:clinton +text:harry))^0.0
                                                    [time_in_nanos] => 176108294                                                  
                                                    [breakdown] => Array                                           
                                                        (                                                               
                                                            [set_min_competitive_score_count] => 0                          
                                                            [match_count] => 0                                    
                                                            [shallow_advance_count] => 0                
                                                            [set_min_competitive_score] => 0                          
                                                            [next_doc] => 158666901                           
                                                            [match] => 0                                                  
                                                            [next_doc_count] => 439522                              
                                                            [score_count] => 439522                              
                                                            [compute_max_score_count] => 0                           
                                                            [compute_max_score] => 0                            
                                                            [advance] => 262234                                                   
                                                            [advance_count] => 1                                    
                                                            [score] => 14477781                                         
                                                            [build_scorer_count] => 2                                             
                                                            [create_weight] => 401058                              
                                                            [shallow_advance] => 0                                      
                                                            [create_weight_count] => 1                                      
                                                            [build_scorer] => 1421272                         
                                                        )                                                                 
                                                                                                                    
                                                    [children] => Array                                          
                                                        (                                                                 
                                                            [0] => Array
(                                                                                       
                                                                    [type] => BooleanQuery
                                                                    [description] => +text:justin +text:trump +text:clinton +text:harry
                                                                    [time_in_nanos] => 128547273
                                                                    [breakdown] => Array
                                                                        (
                                                                            [set_min_competitive_score_count] => 0
                                                                            [match_count] => 0
                                                                            [shallow_advance_count] => 0
                                                                            [set_min_competitive_score] => 0
                                                                            [next_doc] => 126071813
                                                                            [match] => 0
                                                                            [next_doc_count] => 439522
                                                                            [score_count] => 0
                                                                            [compute_max_score_count] => 0
                                                                            [compute_max_score] => 0
                                                                            [advance] => 260695
                                                                            [advance_count] => 1
                                                                            [score] => 0
                                                                            [build_scorer_count] => 2
                                                                            [create_weight] => 373620
                                                                            [shallow_advance] => 0
                                                                            [create_weight_count] => 1
                                                                            [build_scorer] => 1401619
                                                                        )

                                                                    [children] => Array
                                                                        (
                                                                            [0] => Array
                                                                                (
                                                                                    [type] => TermQuery
                                                                                    [description] => text:justin
                                                                                    [time_in_nanos] => 40691947

                                                                                )

                                                                            [1] => Array
                                                                                (
                                                                                    [type] => TermQuery
                                                                                    [description] => text:trump
                                                                                    [time_in_nanos] => 42972729
                                                                                )

                                                                            [2] => Array
                                                                                (
                                                                                    [type] => TermQuery
                                                                                    [description] => text:clinton
                                                                                    [time_in_nanos] => 29407195

                                                                                )

                                                                            [3] => Array
                                                                                (
                                                                                    [type] => TermQuery
                                                                                    [description] => text:harry
                                                                                    [time_in_nanos] => 33799904

                                                                                )

                                                                        )

                                                                )

                                                        )

                                                )

                                        )

                                    [rewrite_time] => 260704
                                    [collector] => Array
                                        (
                                            [0] => Array
                                                (
                                                    [name] => SimpleTopScoreDocCollector
                                                    [reason] => search_top_hits
                                                    [time_in_nanos] => 116380511
                                                )
                                        )

                                )

                        )

                )

        )

)
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 29, 2020, 8:33pm UTC](https://discuss.elastic.co/t/how-to-calculate-search-time-spent-by-elasticsearch-on-its-inverted-index-alone/232962/8 "2020-06-29T20:33:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.