# Unstable response-timing App Search

**URL:** <https://discuss.elastic.co/t/unstable-response-timing-app-search/309383>\
**Category:** Elastic Search\
**Tags:** elastic-app-search\
**Created:** [July 12, 2022, 7:32am UTC](https://discuss.elastic.co/t/unstable-response-timing-app-search/309383 "2022-07-12T07:32:16Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![SanderP](https://avatars.discourse-cdn.com/v4/letter/s/ccd318/32.png) [@SanderP](https://discuss.elastic.co/u/SanderP)\
**Post date:** [July 12, 2022, 7:32am UTC](https://discuss.elastic.co/t/unstable-response-timing-app-search/309383/1 "2022-07-12T07:32:16Z")

</div>

Dear all,

I'm looking into the performance of App Search and am noticing that this is not stable. The response-time of a identical request ranges from 200ms towards 1000+ ms. How could this be made more stable to around 200ms (or faster)? In general what influences the performance?

Engine: 600 objects  
Fields: 31  
Query: 99 objects returned  
No ingests running and no other queries running

The performance varies when testing in Postman. Within Kibana the same can be seen by viewing .ds-logs-enterprise\_search.api-default\* indexes and looking at duration field.

1. Shard cleanup - no improvements  
It is difficult to find blogs/pages that explain App Search performance and any guides of well maintaining the cluster. As underwater Elasticsearch is used I had a look at shard-counts. Initially I found advises to keep the amount of shards to 160 on a 4GB cluster. Only at a later moment I found this documentation which mentions 12000 indexes nowadays: [Size your shards | Elasticsearch Guide [8.4] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/size-your-shards.html#shard-count-recommendation)

Still I cleaned the system going from 848 to 752 shards. Most indexes remaining are out of the box App Search (hidden) indexes, so further cleaning was not possible.

1. Hardware profiling - no improvements  
Recently hardware profiles are added to the Cloud configuration portals. Seeing the documentation General Purpose would be the best fit, hence this was the latest configuration in the clone instance. As a test also CPU optimized was tested: [What are hardware profiles? | Elasticsearch Service Documentation | Elastic](https://www.elastic.co/guide/en/cloud/current/ec-getting-started-profiles.html)

2. Disabling unneeded fields in the result set - no improvements

3. Playing around with facets (minimizing facet usage) - no improvements

4. Increasing the deployment - no improvements  
Our monitoring on cpu and memory state that all is within the boundaries, but still as a test the whole deployment was increased from 4GB in two zones to 15GB in two zones (15GB (2 zone) - 525GB - 4vcpu storage’)

5. Appsearch instance from 2GB to 15GB (cpu 8vcpu) - no improvements

edit: I still need to go through these by the way, but any pointers would be welcome. These came from the Elastic team, as no support could be given in a ticket. Some of these are not really related to our App Search only deployment (no scripts, no aggregations, no expensive queries, ...)

```auto
http://https//discuss.elastic.co/t/tips-to-increase-query-performance/202304
https://www.elastic.co/guide/en/app-search/8.1/troubleshooting.html#troubleshooting-debugging-queries
https://www.elastic.co/guide/en/app-search/8.1/troubleshooting.html#troubleshooting-debug-header-debugging-queries
https://www.elastic.co/guide/en/app-search/8.1/troubleshooting.html#troubleshooting-profiling-queries
https://www.elastic.co/guide/en/kibana/8.1/xpack-profiler.html
https://www.elastic.co/guide/en/elasticsearch/reference/8.1/search-profile.html
https://www.elastic.co/blog/elastic-app-search-performant-rich-search-results-with-result-settings
https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-search-speed.html
https://www.elastic.co/blog/advanced-tuning-finding-and-fixing-slow-elasticsearch-queries

```

Kind regards,

---

<div class="post-metadata">

**Author:** ![SanderP](https://avatars.discourse-cdn.com/v4/letter/s/ccd318/32.png) [@SanderP](https://discuss.elastic.co/u/SanderP)\
**Post date:** [July 12, 2022, 9:20am UTC](https://discuss.elastic.co/t/unstable-response-timing-app-search/309383/2 "2022-07-12T09:20:51Z")

</div>

Using ([Troubleshooting | Elastic App Search Documentation [8.1] | Elastic](https://www.elastic.co/guide/en/app-search/8.1/troubleshooting.html#troubleshooting-5001-error)) I got the raw Elasticsearch query. This result is stable around 20 ms (from debug response and within dev\_tools). Profiling the same query on the involved engine returns a cumulative result of around 1.3ms with a aggregation of 0.08ms. I don't think ElasticSearch is the culprit therefor.

The reason could as well be Infra, so I will try see if the App Search query can be run from within the cluster. Another test would be to run the used ES query directly through Postman to validate the timing.

edit: Direct call to ElasticSearch with the query returned in the debug steps has a 'took' of 20 ms and a total time averaging on 50 ms. It is some times around 100ms but not higher. Conclusion so far is that Appsearch adds considerable time towards the cumulative response and is the reason for fluctuating timing (not the Infrastructure: Local network -\> Internet -\> App Search)

---

<div class="post-metadata">

**Author:** ![Carlos\_D](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carlos_d/32/126245_2.png) [@Carlos\_D](https://discuss.elastic.co/u/Carlos_D)\
**Post date:** [July 18, 2022, 8:09am UTC](https://discuss.elastic.co/t/unstable-response-timing-app-search/309383/3 "2022-07-18T08:09:21Z")

</div>

Hi @SanderP !

Could you please include the Enterprise Search version you're using? There is a known issue for performance in 8.2.0 - 8.2.2. Should that be your case, we recommend you to upgrade to 8.2.3 or higher.

If that is not the case, can you please set `log_level` to `debug` in `enterprise-search.yml` [configuration file](https://www.elastic.co/guide/en/enterprise-search/master/configuration.html#configuration-settings-logging)? That will send to the log the Elasticsearch queries performed by App Search so we can diagnose it further.

Thanks!

---

<div class="post-metadata">

**Author:** ![SanderP](https://avatars.discourse-cdn.com/v4/letter/s/ccd318/32.png) [@SanderP](https://discuss.elastic.co/u/SanderP)\
**Post date:** [July 18, 2022, 8:40am UTC](https://discuss.elastic.co/t/unstable-response-timing-app-search/309383/4 "2022-07-18T08:40:26Z")

</div>

Hello @Carlos_D ,

It is on 8.1.2  
With the debug statement of appsearch I already have the Elasticsearch query (which was performing well). Are there additional queries to be expected?

Based on App Search:  
{"page":{"current":1,"size":50},"query":"philips","facets":{"brand":[{"type":"value","size":50}]},"filters":{"all":[{"start\_of\_publication":{"to":"2022-07-05T13:27:48.572248905Z"}},{"end\_of\_publication":{"from":"2022-07-05T13:27:48.572248905Z"}},{"marketing\_status":["Final Published"]},{"power":{"from":0.0,"to":10.0}}]}}

```auto
{
                    "query": {
                        "bool": {
                            "must": {
                                "bool": {
                                    "must": [
                                        {
                                            "bool": {
                                                "should": [
                                                    {
                                                        "multi_match": {
                                                            "query": "philips",
                                                            "minimum_should_match": "1<-1 3<49%",
                                                            "type": "cross_fields",
                                                            "fields": [
                                                                "description^1.0",
                                                                "description.stem^0.95",
                                                                "description.prefix^0.1",
                                                                "description.joined^0.75",
                                                                "description.delimiter^0.4",
                                                                "locale^1.0",
                                                                "locale.stem^0.95",
                                                                "locale.prefix^0.1",
                                                                "locale.joined^0.75",
                                                                "locale.delimiter^0.4",
                                                                "value_ladder^1.0",
                                                                "value_ladder.stem^0.95",
                                                                "value_ladder.prefix^0.1",
                                                                "value_ladder.joined^0.75",
                                                                "value_ladder.delimiter^0.4",
                                                                "brand_image^1.0",
                                                                "brand_image.stem^0.95",
                                                                "brand_image.prefix^0.1",
                                                                "brand_image.joined^0.75",
                                                                "brand_image.delimiter^0.4",
                                                                "id^1.0",
                                                                "id.stem^0.95",
                                                                "id.prefix^0.1",
                                                                "id.joined^0.75",
                                                                "id.delimiter^0.4",
                                                                "categories^1.0",
                                                                "categories.stem^0.95",
                                                                "categories.prefix^0.1",
                                                                "categories.joined^0.75",
                                                                "categories.delimiter^0.4",
                                                                "sku^1.0",
                                                                "sku.stem^0.95",
                                                                "sku.prefix^0.1",
                                                                "sku.joined^0.75",
                                                                "sku.delimiter^0.4",
                                                                "brand^1.0",
                                                                "brand.stem^0.95",
                                                                "brand.prefix^0.1",
                                                                "brand.joined^0.75",
                                                                "brand.delimiter^0.4",
                                                                "image^1.0",
                                                                "image.stem^0.95",
                                                                "image.prefix^0.1",
                                                                "image.joined^0.75",
                                                                "image.delimiter^0.4",
                                                                "family_id^1.0",
                                                                "family_id.stem^0.95",
                                                                "family_id.prefix^0.1",
                                                                "family_id.joined^0.75",
                                                                "family_id.delimiter^0.4",
                                                                "sub_application_areas^1.0",
                                                                "sub_application_areas.stem^0.95",
                                                                "sub_application_areas.prefix^0.1",
                                                                "sub_application_areas.joined^0.75",
                                                                "sub_application_areas.delimiter^0.4",
                                                                "dimmable^1.0",
                                                                "dimmable.stem^0.95",
                                                                "dimmable.prefix^0.1",
                                                                "dimmable.joined^0.75",
                                                                "dimmable.delimiter^0.4",
                                                                "product_codes^1.0",
                                                                "product_codes.stem^0.95",
                                                                "product_codes.prefix^0.1",
                                                                "product_codes.joined^0.75",
                                                                "product_codes.delimiter^0.4",
                                                                "application_areas^1.0",
                                                                "application_areas.stem^0.95",
                                                                "application_areas.prefix^0.1",
                                                                "application_areas.joined^0.75",
                                                                "application_areas.delimiter^0.4",
                                                                "marketing_status^1.0",
                                                                "marketing_status.stem^0.95",
                                                                "marketing_status.prefix^0.1",
                                                                "marketing_status.joined^0.75",
                                                                "marketing_status.delimiter^0.4",
                                                                "catalogs^1.0",
                                                                "catalogs.stem^0.95",
                                                                "catalogs.prefix^0.1",
                                                                "catalogs.joined^0.75",
                                                                "catalogs.delimiter^0.4",
                                                                "name^1.0",
                                                                "name.stem^0.95",
                                                                "name.prefix^0.1",
                                                                "name.joined^0.75",
                                                                "name.delimiter^0.4"
                                                            ]
                                                        }
                                                    },
                                                    {
                                                        "multi_match": {
                                                            "query": "philips",
                                                            "minimum_should_match": "1<-1 3<49%",
                                                            "type": "best_fields",
                                                            "fuzziness": "AUTO",
                                                            "prefix_length": 2,
                                                            "fields": [
                                                                "description.stem^0.1",
                                                                "locale.stem^0.1",
                                                                "value_ladder.stem^0.1",
                                                                "brand_image.stem^0.1",
                                                                "id.stem^0.1",
                                                                "categories.stem^0.1",
                                                                "sku.stem^0.1",
                                                                "brand.stem^0.1",
                                                                "image.stem^0.1",
                                                                "family_id.stem^0.1",
                                                                "sub_application_areas.stem^0.1",
                                                                "dimmable.stem^0.1",
                                                                "product_codes.stem^0.1",
                                                                "application_areas.stem^0.1",
                                                                "marketing_status.stem^0.1",
                                                                "catalogs.stem^0.1",
                                                                "name.stem^0.1"
                                                            ]
                                                        }
                                                    }
                                                ]
                                            }
                                        }
                                    ]
                                }
                            },
                            "filter": {
                                "bool": {
                                    "must": [
                                        {
                                            "range": {
                                                "start_of_publication.date": {
                                                    "lt": "2022-07-05T13:27:48.572248905Z"
                                                }
                                            }
                                        },
                                        {
                                            "range": {
                                                "end_of_publication.date": {
                                                    "gte": "2022-07-05T13:27:48.572248905Z"
                                                }
                                            }
                                        },
                                        {
                                            "terms": {
                                                "marketing_status.enum": [
                                                    "Final Published"
                                                ]
                                            }
                                        },
                                        {
                                            "range": {
                                                "power.float": {
                                                    "gte": 0.0,
                                                    "lt": 10.0
                                                }
                                            }
                                        }
                                    ]
                                }
                            }
                        }
                    },
                    "sort": [
                        {
                            "_score": "desc"
                        },
                        {
                            "_doc": "desc"
                        }
                    ],
                    "aggs": {
                        "0__brand": {
                            "terms": {
                                "size": 50,
                                "field": "brand.enum"
                            },
                            "meta": {
                                "respond_with_rfc3339_date": null
                            }
                        }
                    },
                    "highlight": {
                        "fragment_size": 300,
                        "type": "plain",
                        "number_of_fragments": 1,
                        "order": "score",
                        "encoder": "html",
                        "require_field_match": false,
                        "fields": {}
                    },
                    "size": 50,
                    "from": 0,
                    "timeout": "30000ms",
                    "_source": [
                        "description",
                        "locale",
                        "value_ladder",
                        "diameter",
                        "brand_image",
                        "price",
                        "warranty",
                        "id",
                        "categories",
                        "power",
                        "sku",
                        "stock",
                        "brand",
                        "beam_angle",
                        "height",
                        "image",
                        "family_id",
                        "luminous_flux",
                        "length",
                        "sub_application_areas",
                        "color_temp",
                        "dimmable",
                        "product_codes",
                        "application_areas",
                        "catalogs",
                        "name",
                        "width"
                    ]
                }

```

---

<div class="post-metadata">

**Author:** ![SanderP](https://avatars.discourse-cdn.com/v4/letter/s/ccd318/32.png) [@SanderP](https://discuss.elastic.co/u/SanderP)\
**Post date:** [July 25, 2022, 8:57am UTC](https://discuss.elastic.co/t/unstable-response-timing-app-search/309383/5 "2022-07-25T08:57:58Z")

</div>

HI,

I'm on version 8.3 currently. The performance ranges from 350-500ms mostly with still sporadic spikes.

---

<div class="post-metadata">

**Author:** ![SanderP](https://avatars.discourse-cdn.com/v4/letter/s/ccd318/32.png) [@SanderP](https://discuss.elastic.co/u/SanderP)\
**Post date:** [August 15, 2022, 1:37pm UTC](https://discuss.elastic.co/t/unstable-response-timing-app-search/309383/6 "2022-08-15T13:37:33Z")

</div>

> [@Carlos\_D](#):
>
> If that is not the case, can you please set `log_level` to `debug` in `enterprise-search.yml` [configuration file](https://www.elastic.co/guide/en/enterprise-search/master/configuration.html#configuration-settings-logging)? That will send to the log the Elasticsearch queries performed by App Search so we can diagnose it further.

Hello @Carlos_D , would this still help next to the above debug flag from within the Appsearch call itself? Thanks for your reply?

---

<div class="post-metadata">

**Author:** ![Carlos\_D](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carlos_d/32/126245_2.png) [@Carlos\_D](https://discuss.elastic.co/u/Carlos_D)\
**Post date:** [August 16, 2022, 7:58am UTC](https://discuss.elastic.co/t/unstable-response-timing-app-search/309383/7 "2022-08-16T07:58:35Z")

</div>

Hi @SanderP! Sorry for the late response.

Please ensure you're on 8.3.3 version; there was a performance bug fixed for that release that was present on 8.3.0 - 8.3.2. As you already experienced this on 8.2, I don't believe you're impacted by this, but please make sure you're on 8.3.3 👍

> Hello @Carlos_D , would this still help next to the above debug flag from within the Appsearch call itself? Thanks for your reply?

I've checked your query, and you have already checked its execution time directly on Elasticsearch, so we can rule out the query side.

Enterprise Search does more communication with Elasticsearch under the hood before actually performing the actual query. That could be part of the extra time you are experiencing versus doing the raw Elasticsearch query.

One way we can check where time is actually spent is using [Application Performance Monitoring (APM)](https://www.elastic.co/apm). Enterprise Search [supports APM integration](https://www.elastic.co/guide/en/enterprise-search/master/monitoring.html#elastic-apm-integration), which you can use to understand performance issues.

I suggest checking APM for diving deeper into this issue.

---

<div class="post-metadata">

**Author:** ![SanderP](https://avatars.discourse-cdn.com/v4/letter/s/ccd318/32.png) [@SanderP](https://discuss.elastic.co/u/SanderP)\
**Post date:** [August 16, 2022, 11:22am UTC](https://discuss.elastic.co/t/unstable-response-timing-app-search/309383/8 "2022-08-16T11:22:28Z")

</div>

Hello @Carlos_D ,

I'm happy to say that 8.3.3 seemed to resolve the issue. The responses are currently in the 1xx ms range and usually below 150ms. I'll still explore APM to understand the benefits for future cases and will do more testing on 8.3.3. Thanks!

Small note, we also have a v7.16 deployment which I just validated. With 338 results the response is around 350ms but stable, so the fluctuating response times really seemed to be only in v8 till 8.3.3.

Note2: There was someone in 1:1 chat reporting the same issue so I'm going to update him to validate the same on their deployment.

---

<div class="post-metadata">

**Author:** ![Carlos\_D](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carlos_d/32/126245_2.png) [@Carlos\_D](https://discuss.elastic.co/u/Carlos_D)\
**Post date:** [August 16, 2022, 11:50am UTC](https://discuss.elastic.co/t/unstable-response-timing-app-search/309383/9 "2022-08-16T11:50:36Z")

</div>

Hey @SanderP :

Thanks for confirming upgrading to 8.3.3 worked! 👍

---

<div class="post-metadata">

**Author:** ![SanderP](https://avatars.discourse-cdn.com/v4/letter/s/ccd318/32.png) [@SanderP](https://discuss.elastic.co/u/SanderP)\
**Post date:** [August 17, 2022, 10:16am UTC](https://discuss.elastic.co/t/unstable-response-timing-app-search/309383/10 "2022-08-17T10:16:57Z")

</div>

Hello @Carlos_D ,

As a final check I tried to go back to 8.2.3 to see if it was still reproducible to exclude any other reason. Initially I ran into issues with the deployment of 8.2.3 with a snapshot from 8.3.3. During the initialization of the deployment and within snapshot restore options 8.3.3 is to be chosen, but apparently going back minor versions is not allowed: [Snapshot and restore | Elasticsearch Guide [8.3] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-restore.html#snapshot-index-compatibility)

As an alternative I used the below approach. Do you have insights if this approach can be used for copying over Appsearch indexes nowadays now that AS is being integrated more within the landscape as a real accessible Elasticsearch index?

1. Clean deployment 8.2.3
2. Created an engine with the same name from the other 8.3.3 deployment
3. Added whitelisting
4. Reindex-remote: [Reindex API | Elasticsearch Guide [8.3] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html#reindex-from-remote)
5. Added schema
6. The Search command was fluctuating more once more, hence proven

---

<div class="post-metadata">

**Author:** ![Carlos\_D](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carlos_d/32/126245_2.png) [@Carlos\_D](https://discuss.elastic.co/u/Carlos_D)\
**Post date:** [August 17, 2022, 12:30pm UTC](https://discuss.elastic.co/t/unstable-response-timing-app-search/309383/11 "2022-08-17T12:30:29Z")

</div>

Hi @SanderP :

Migrations will not work backwards. Always do a snapshot before upgrading so you can revert back to a previous version if needed.

The method you mention is not guaranteed to work; in particular, it does not copy over engine settings like curations, synonyms, relevance tuning... and other useful information like API Logs or analytics.

It is preferred to use App Search APIs to migrate documents and settings if needed, rather than doing specific index updates.

---

<div class="post-metadata">

**Author:** ![SanderP](https://avatars.discourse-cdn.com/v4/letter/s/ccd318/32.png) [@SanderP](https://discuss.elastic.co/u/SanderP)\
**Post date:** [August 17, 2022, 1:04pm UTC](https://discuss.elastic.co/t/unstable-response-timing-app-search/309383/12 "2022-08-17T13:04:28Z")

</div>

Great, thanks. This finalizes the issue for me!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 14, 2022, 1:04pm UTC](https://discuss.elastic.co/t/unstable-response-timing-app-search/309383/13 "2022-09-14T13:04:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.