# Need help to improve performance with ES

**URL:** <https://discuss.elastic.co/t/need-help-to-improve-performance-with-es/39973>\
**Category:** Elasticsearch\
**Created:** [January 24, 2016, 5:21pm UTC](https://discuss.elastic.co/t/need-help-to-improve-performance-with-es/39973 "2016-01-24T17:21:16Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Deep\_2](https://avatars.discourse-cdn.com/v4/letter/d/7feea3/32.png) [@Deep\_2](https://discuss.elastic.co/u/Deep_2)\
**Post date:** [January 24, 2016, 5:21pm UTC](https://discuss.elastic.co/t/need-help-to-improve-performance-with-es/39973/1 "2016-01-24T17:21:16Z")

</div>

Hi,

I have a single node elastic search deployment with 15K documents. The machine has 4 cores and 8 Gb of RAM. The node is handling 1300 request per second with 25% cpu utilization, 75% memory utilization. In the current deployment query response time is 100 ms.

We need the search query to run in \< 30 ms.

The search query is essentially geo location search that tries to fetch document that are within x miles of the input lat/lon with some additional filters and the documents are sorted on distance (nearest to furthest). Each document has multiple lat/lon. It seems geo\_distance uses only first lat/lon in the array of lat/long so the simple geo\_distance filter was not usable in the query.

Need help to optimize the query.

Regards,  
Deep

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [January 24, 2016, 6:32pm UTC](https://discuss.elastic.co/t/need-help-to-improve-performance-with-es/39973/2 "2016-01-24T18:32:16Z")

</div>

Have you tried denormalising and store a copy of each document for each location, possibly with the array of locations in a separate field if your application needs them? I suspect this would give better performance than using a groovy script, which you mention in [your other post](https://discuss.elastic.co/t/geo-distance-filter-with-multiple-lat-lon/39955).

---

<div class="post-metadata">

**Author:** ![Deep\_2](https://avatars.discourse-cdn.com/v4/letter/d/7feea3/32.png) [@Deep\_2](https://discuss.elastic.co/u/Deep_2)\
**Post date:** [January 24, 2016, 7:26pm UTC](https://discuss.elastic.co/t/need-help-to-improve-performance-with-es/39973/3 "2016-01-24T19:26:14Z")

</div>

Hi Christian,  
Thank you for your response.

As suggested let me denormalise and try.

But denormalizing would add more documents to the index, will that impact query performance. In some cases I would have 50 lat/lon in a document will this adversely impact query performance.

Regards,  
Deep

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [January 24, 2016, 7:31pm UTC](https://discuss.elastic.co/t/need-help-to-improve-performance-with-es/39973/4 "2016-01-24T19:31:16Z")

</div>

Even if all the 15k documents has 50 geopoints in them, 750000 documents in an index is still not much (unless the documents are huge). Given your amount of memory I would expect it all to be cached anyway.

---

<div class="post-metadata">

**Author:** ![Deep\_2](https://avatars.discourse-cdn.com/v4/letter/d/7feea3/32.png) [@Deep\_2](https://discuss.elastic.co/u/Deep_2)\
**Post date:** [January 24, 2016, 7:40pm UTC](https://discuss.elastic.co/t/need-help-to-improve-performance-with-es/39973/5 "2016-01-24T19:40:40Z")

</div>

Ok.  
On marvel i can see the data size as 100 Mb. Let me denormalise and share the results.

Thanks.

---

<div class="post-metadata">

**Author:** ![Deep\_2](https://avatars.discourse-cdn.com/v4/letter/d/7feea3/32.png) [@Deep\_2](https://discuss.elastic.co/u/Deep_2)\
**Post date:** [January 25, 2016, 12:11pm UTC](https://discuss.elastic.co/t/need-help-to-improve-performance-with-es/39973/6 "2016-01-25T12:11:53Z")

</div>

Hi Christian,

In our search query we need to use geo\_range query and to and from values are a part of the document. In the geo\_range query is it possible to access to and from values from within the document. We can access these values using a script.

Regards,  
Deep

---

<div class="post-metadata">

**Author:** ![Deep\_2](https://avatars.discourse-cdn.com/v4/letter/d/7feea3/32.png) [@Deep\_2](https://discuss.elastic.co/u/Deep_2)\
**Post date:** [February 5, 2016, 11:14am UTC](https://discuss.elastic.co/t/need-help-to-improve-performance-with-es/39973/7 "2016-02-05T11:14:55Z")

</div>

Hi Christian,

As suggested by you removing scripts and denormalising the data I can see reduction in response time. But the response time fluctuates and I can co-relate the increase in response time with increase in merge rate on marvel. When ever the merge rate goes up the response time increases.

I tried to control the increase in merge rate by increasing the index refresh interval duration. I increased it to 60s. But I still see spikes in merge rate.

Merging should happen only when the index is updated. Is this understanding correct?

How can I control Merge Rate?

Regards,  
Deep

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [February 5, 2016, 11:52am UTC](https://discuss.elastic.co/t/need-help-to-improve-performance-with-es/39973/8 "2016-02-05T11:52:32Z")

</div>

If you are not continuously indexing, e.g. if you perform periodic bulk uploads, you can force Elasticsearch to consolidate segments through the [force merge API](https://www.elastic.co/guide/en/elasticsearch/reference/2.2/indices-forcemerge.html). This can be resource intensive in terms of CPU and disk I/O, but once you have consolidated into one segment, there should be no more merging until you index/update/delete more data.

You also have the option to tune how aggressive you want merging to by [throttling merges](https://www.elastic.co/guide/en/elasticsearch/reference/2.2/breaking_20_setting_changes.html#_merge_and_merge_throttling_settings).

---

<div class="post-metadata">

**Author:** ![Deep\_2](https://avatars.discourse-cdn.com/v4/letter/d/7feea3/32.png) [@Deep\_2](https://discuss.elastic.co/u/Deep_2)\
**Post date:** [February 5, 2016, 12:14pm UTC](https://discuss.elastic.co/t/need-help-to-improve-performance-with-es/39973/9 "2016-02-05T12:14:10Z")

</div>

Hi Christian,

I am using elastic search 1.4.

The force merge API returns an error.

The API curl -XPOST '[http://localhost:9200/search/\_forcemerge](http://localhost:9200/search/_forcemerge)'  
{"error":"InvalidTypeNameException[mapping type name [_forcemerge] can't start with '_']","status":400}

Is this API supported in new versions of elastic search?

Regards,  
Deep

---

<div class="post-metadata">

**Author:** ![Deep\_2](https://avatars.discourse-cdn.com/v4/letter/d/7feea3/32.png) [@Deep\_2](https://discuss.elastic.co/u/Deep_2)\
**Post date:** [February 5, 2016, 12:22pm UTC](https://discuss.elastic.co/t/need-help-to-improve-performance-with-es/39973/10 "2016-02-05T12:22:30Z")

</div>

Hi Christian,

just figured out that in version 1.4 this api was optimize.

The api curl -XPOST "[http://localhost:9200/search/\_optimize?max\_num\_segments=1](http://localhost:9200/search/_optimize?max_num_segments=1)" worked.

Regards,  
Deep

---

<div class="post-metadata">

**Author:** ![Deep\_2](https://avatars.discourse-cdn.com/v4/letter/d/7feea3/32.png) [@Deep\_2](https://discuss.elastic.co/u/Deep_2)\
**Post date:** [February 5, 2016, 2:15pm UTC](https://discuss.elastic.co/t/need-help-to-improve-performance-with-es/39973/11 "2016-02-05T14:15:19Z")

</div>

Hi Christian,

After optimize it automatically creates multiple segments. There are no deletes/updated to the index.

What could be the reason for this?

Regards,  
Deep

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [February 5, 2016, 3:00pm UTC](https://discuss.elastic.co/t/need-help-to-improve-performance-with-es/39973/12 "2016-02-05T15:00:44Z")

</div>

If you look at index statistics, do the number of documents or the number of deleted documents change when segments are created?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:18pm UTC](https://discuss.elastic.co/t/need-help-to-improve-performance-with-es/39973/13 "2017-07-05T23:18:29Z")

</div>