# Performance of "exists" query

**URL:** <https://discuss.elastic.co/t/performance-of-exists-query/59591>\
**Category:** Elasticsearch\
**Created:** [September 1, 2016, 6:44pm UTC](https://discuss.elastic.co/t/performance-of-exists-query/59591 "2016-09-01T18:44:43Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![cweiske](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cweiske/32/11732_2.png) [@cweiske](https://discuss.elastic.co/u/cweiske)\
**Post date:** [September 1, 2016, 6:44pm UTC](https://discuss.elastic.co/t/performance-of-exists-query/59591/1 "2016-09-01T18:44:43Z")

</div>

I've built a website search engine on top of Elasticsearch and have problems with crawling performance.

After extracting all URLs from a HTML page, I add them to ES. But in advance I check if a document with the given URL already [exists](https://www.elastic.co/guide/en/elasticsearch/guide/2.x/_most_important_queries.html#_exists_and_missing_queries):

```
GET /document/_search/exists
{"query":{"filtered":{"filter":{"term":{"url":"https:\/\/example.org\/"}}}}}

```

This takes 40ms, and since I do this for each of the URLs this adds up quite a bit.

My current mapping for the `url` field is:

```
    "url": {
        "type": "string",
        "index": "not_analyzed",
        "boost": 1.5
    },

```

* * *

Is there anything I can do to make this query faster?

---

<div class="post-metadata">

**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\
**Post date:** [September 1, 2016, 7:10pm UTC](https://discuss.elastic.co/t/performance-of-exists-query/59591/2 "2016-09-01T19:10:45Z")

</div>

If you added the documents routing on the URL then you could add the same routing on the query and only hit a single shard. That is fairly similar in concept to using the URL as the document's ID but doesn't have problems with very very long urls.

If you are doing this for lots of URLs at once then you could do a multi search requests and get all the results back at once. You'd want to set `size=0&terminate_after=1` which is what the `_search/exists` API does anyway. Actually, `_search/exists` is deprecated with a note that says to use `size=0&terminate_after=1` so maybe that is a good idea anyway.

It'd go with the multisearch idea first because it is something you can do without changing your index. If that isn't good enough then look at the routing.

---

<div class="post-metadata">

**Author:** ![cweiske](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cweiske/32/11732_2.png) [@cweiske](https://discuss.elastic.co/u/cweiske)\
**Post date:** [September 2, 2016, 6:41am UTC](https://discuss.elastic.co/t/performance-of-exists-query/59591/3 "2016-09-02T06:41:38Z")

</div>

> [@nik9000](#):
>
> If you added the documents routing on the URL then you could add the same routing on the query and only hit a single shard. That is fairly similar in concept to using the URL as the document's ID but doesn't have problems with very very long urls.

I tried the routing, but that does not make a difference. I'm also testing on a fairly tiny ES database with 200 documents.

I did not try multisearch yet, but have the feeling that maybe the HTTP request handling (albeit to 127.0.0.1) add most of the query time.  
Using keep-alive might solve the problems here.

---

<div class="post-metadata">

**Author:** ![cweiske](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cweiske/32/11732_2.png) [@cweiske](https://discuss.elastic.co/u/cweiske)\
**Post date:** [September 2, 2016, 8:58am UTC](https://discuss.elastic.co/t/performance-of-exists-query/59591/4 "2016-09-02T08:58:27Z")

</div>

I debugged this issue a bit more: Keep-Alive was used automatically, so this was not the problem.

I am using PHP 5.5.9 with the [HTTP\_Request2 library](http://pear.php.net/package/HTTP_Request2) and the socket adapter to send the `_search/exists` GET query with the body.

Switching to the Curl adapter broke, because curl apparently had an issue of sending GET request with a body. Switching to POST made it work with curl, too. Timing did change, too:

- curl with POST: 2.8s
- socket with POST: 18.8s

Then I optimized a bit and used the document's URL as identifier, which means I can now fetch the document with the URL: `GET /document/$url`:

- curl with HEAD on the URL: 1.8s
- socket with HEAD on the URL: 1.3s

Conclusion:

- There is a performance problem in HTTP\_Request2's socket adapter for requests with a body.
- The curl adapter is 10x faster for POST requests
- The socket adapter is 1.5x faster for HEAD requests.

This all had nothing to do with elastic search itself. Sorry.

---

<div class="post-metadata">

**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\
**Post date:** [September 2, 2016, 12:17pm UTC](https://discuss.elastic.co/t/performance-of-exists-query/59591/5 "2016-09-02T12:17:29Z")

</div>

Using the URL as the document id is a bit dangerous because URLs can get  
long but document IDs will have a limit in 5.0:

> <https://github.com/elastic/elasticsearch/pull/16036>

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:22pm UTC](https://discuss.elastic.co/t/performance-of-exists-query/59591/6 "2017-07-05T22:22:53Z")

</div>