# Selecting documents to search (performance issues)

**URL:** <https://discuss.elastic.co/t/selecting-documents-to-search-performance-issues/228928>\
**Category:** Elasticsearch\
**Created:** [April 20, 2020, 7:27pm UTC](https://discuss.elastic.co/t/selecting-documents-to-search-performance-issues/228928 "2020-04-20T19:27:45Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![djmarcus](https://avatars.discourse-cdn.com/v4/letter/d/fbc32d/32.png) [@djmarcus](https://discuss.elastic.co/u/djmarcus)\
**Post date:** [April 20, 2020, 7:27pm UTC](https://discuss.elastic.co/t/selecting-documents-to-search-performance-issues/228928/1 "2020-04-20T19:27:45Z")

</div>

I have an elasticsearch document repository with ~15M documents.

Each document has an unique 11-char string field (comes from a mongo DB) that is unique to the document. This field is indexed as `keyword`.

I'm using C#.

When I run a search, I want to be able to limit the search to a set of documents that I specify (via some list of the unique field ids).

My query text uses `bool` with `must` to supply a `filter` for the unique identifiers and additional clauses to actually search the documents. See example below.

To search a large number of documents, I generate multiple query strings and run them concurrently. Each query handles up to 64K unique ids (determined by the limit on `terms`).

In this case, I have 262,144 documents to search (list comes, at run time, from a separate mongo DB query). So my code generates 4 query strings (see example below).

I run them concurrently.

Unfortunately, this search takes over 22 seconds to complete.

When I run the same search but drop the `terms` node (so it searches all the documents), a single such query completes the search in 1.8 seconds.

An incredible difference.

So my question: **Is there an efficient way to specify which documents are to be searched (when each document has a unique self-identifying keyword field)?**

I want to be able to specify up to a few 100K of such unique ids.

Here's an example of my search specifying unique document identifiers:

```auto
{
    "_source" : "talentId",
    "from" : 0,
    "size" : 10000,
    "query" : {
        "bool" : {
            "must" : [
                {
                    "bool" : {
                        "must" : [ { "match_phrase" : { "freeText" : "java" } },
                                          { "match_phrase" : { "freeText" : "unix" } },
                                          { "match_phrase" : { "freeText" : "c#" } },
                                          { "match_phrase" : { "freeText" : "cnn" } } ]
                    }
                },
                {
                    "bool" : {
                        "filter" : {
                            "bool" : {
                                "should" : [
                                    {
                                        "terms" : {
                                            "talentId" : [ "goGSXMWE1Qg", "GvTDYS6F1Qg",
                                                           "-qa_N-aC1Qg", "iu299LCC1Qg",
                                                           "0p7SpteI1Qg", ... 4,995 more ... ]
                                        }
                                    }
                                ]
                            }
                        }
                    }
                }
            ]
        }
    }
} 

```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 20, 2020, 7:48pm UTC](https://discuss.elastic.co/t/selecting-documents-to-search-performance-issues/228928/2 "2020-04-20T19:48:02Z")

</div>

How are you finding these ids? Is there done grouping you could use instead to reduce the number of terms? As far as I know there is no magic way to dramatically improve the performance of this type of queries.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 18, 2020, 7:48pm UTC](https://discuss.elastic.co/t/selecting-documents-to-search-performance-issues/228928/3 "2020-05-18T19:48:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.