Unstable response-timing App Search

Elastic Search
1

Dear all,

I'm looking into the performance of App Search and am noticing that this is not stable. The response-time of a identical request ranges from 200ms towards 1000+ ms. How could this be made more stable to around 200ms (or faster)? In general what influences the performance?

Engine: 600 objects
Fields: 31
Query: 99 objects returned
No ingests running and no other queries running

The performance varies when testing in Postman. Within Kibana the same can be seen by viewing .ds-logs-enterprise_search.api-default* indexes and looking at duration field.

  1. Shard cleanup - no improvements
    It is difficult to find blogs/pages that explain App Search performance and any guides of well maintaining the cluster. As underwater Elasticsearch is used I had a look at shard-counts. Initially I found advises to keep the amount of shards to 160 on a 4GB cluster. Only at a later moment I found this documentation which mentions 12000 indexes nowadays: Size your shards | Elasticsearch Guide [8.4] | Elastic

Still I cleaned the system going from 848 to 752 shards. Most indexes remaining are out of the box App Search (hidden) indexes, so further cleaning was not possible.

  1. Hardware profiling - no improvements
    Recently hardware profiles are added to the Cloud configuration portals. Seeing the documentation General Purpose would be the best fit, hence this was the latest configuration in the clone instance. As a test also CPU optimized was tested: What are hardware profiles? | Elasticsearch Service Documentation | Elastic

  2. Disabling unneeded fields in the result set - no improvements

  3. Playing around with facets (minimizing facet usage) - no improvements

  4. Increasing the deployment - no improvements
    Our monitoring on cpu and memory state that all is within the boundaries, but still as a test the whole deployment was increased from 4GB in two zones to 15GB in two zones (15GB (2 zone) - 525GB - 4vcpu storage’)

  5. Appsearch instance from 2GB to 15GB (cpu 8vcpu) - no improvements

edit: I still need to go through these by the way, but any pointers would be welcome. These came from the Elastic team, as no support could be given in a ticket. Some of these are not really related to our App Search only deployment (no scripts, no aggregations, no expensive queries, ...)

http://https//discuss.elastic.co/t/tips-to-increase-query-performance/202304
https://www.elastic.co/guide/en/app-search/8.1/troubleshooting.html#troubleshooting-debugging-queries
https://www.elastic.co/guide/en/app-search/8.1/troubleshooting.html#troubleshooting-debug-header-debugging-queries
https://www.elastic.co/guide/en/app-search/8.1/troubleshooting.html#troubleshooting-profiling-queries
https://www.elastic.co/guide/en/kibana/8.1/xpack-profiler.html
https://www.elastic.co/guide/en/elasticsearch/reference/8.1/search-profile.html
https://www.elastic.co/blog/elastic-app-search-performant-rich-search-results-with-result-settings
https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-search-speed.html
https://www.elastic.co/blog/advanced-tuning-finding-and-fixing-slow-elasticsearch-queries

Kind regards,

1 Like
2

Using (Troubleshooting | Elastic App Search Documentation [8.1] | Elastic) I got the raw Elasticsearch query. This result is stable around 20 ms (from debug response and within dev_tools). Profiling the same query on the involved engine returns a cumulative result of around 1.3ms with a aggregation of 0.08ms. I don't think ElasticSearch is the culprit therefor.

The reason could as well be Infra, so I will try see if the App Search query can be run from within the cluster. Another test would be to run the used ES query directly through Postman to validate the timing.

edit: Direct call to ElasticSearch with the query returned in the debug steps has a 'took' of 20 ms and a total time averaging on 50 ms. It is some times around 100ms but not higher. Conclusion so far is that Appsearch adds considerable time towards the cumulative response and is the reason for fluctuating timing (not the Infrastructure: Local network -> Internet -> App Search)

3

Hi @SanderP !

Could you please include the Enterprise Search version you're using? There is a known issue for performance in 8.2.0 - 8.2.2. Should that be your case, we recommend you to upgrade to 8.2.3 or higher.

If that is not the case, can you please set log_level to debug in enterprise-search.yml configuration file? That will send to the log the Elasticsearch queries performed by App Search so we can diagnose it further.

Thanks!

4

Hello @Carlos_D ,

It is on 8.1.2
With the debug statement of appsearch I already have the Elasticsearch query (which was performing well). Are there additional queries to be expected?

Based on App Search:
{"page":{"current":1,"size":50},"query":"philips","facets":{"brand":[{"type":"value","size":50}]},"filters":{"all":[{"start_of_publication":{"to":"2022-07-05T13:27:48.572248905Z"}},{"end_of_publication":{"from":"2022-07-05T13:27:48.572248905Z"}},{"marketing_status":["Final Published"]},{"power":{"from":0.0,"to":10.0}}]}}

{
                    "query": {
                        "bool": {
                            "must": {
                                "bool": {
                                    "must": [
                                        {
                                            "bool": {
                                                "should": [
                                                    {
                                                        "multi_match": {
                                                            "query": "philips",
                                                            "minimum_should_match": "1<-1 3<49%",
                                                            "type": "cross_fields",
                                                            "fields": [
                                                                "description^1.0",
                                                                "description.stem^0.95",
                                                                "description.prefix^0.1",
                                                                "description.joined^0.75",
                                                                "description.delimiter^0.4",
                                                                "locale^1.0",
                                                                "locale.stem^0.95",
                                                                "locale.prefix^0.1",
                                                                "locale.joined^0.75",
                                                                "locale.delimiter^0.4",
                                                                "value_ladder^1.0",
                                                                "value_ladder.stem^0.95",
                                                                "value_ladder.prefix^0.1",
                                                                "value_ladder.joined^0.75",
                                                                "value_ladder.delimiter^0.4",
                                                                "brand_image^1.0",
                                                                "brand_image.stem^0.95",
                                                                "brand_image.prefix^0.1",
                                                                "brand_image.joined^0.75",
                                                                "brand_image.delimiter^0.4",
                                                                "id^1.0",
                                                                "id.stem^0.95",
                                                                "id.prefix^0.1",
                                                                "id.joined^0.75",
                                                                "id.delimiter^0.4",
                                                                "categories^1.0",
                                                                "categories.stem^0.95",
                                                                "categories.prefix^0.1",
                                                                "categories.joined^0.75",
                                                                "categories.delimiter^0.4",
                                                                "sku^1.0",
                                                                "sku.stem^0.95",
                                                                "sku.prefix^0.1",
                                                                "sku.joined^0.75",
                                                                "sku.delimiter^0.4",
                                                                "brand^1.0",
                                                                "brand.stem^0.95",
                                                                "brand.prefix^0.1",
                                                                "brand.joined^0.75",
                                                                "brand.delimiter^0.4",
                                                                "image^1.0",
                                                                "image.stem^0.95",
                                                                "image.prefix^0.1",
                                                                "image.joined^0.75",
                                                                "image.delimiter^0.4",
                                                                "family_id^1.0",
                                                                "family_id.stem^0.95",
                                                                "family_id.prefix^0.1",
                                                                "family_id.joined^0.75",
                                                                "family_id.delimiter^0.4",
                                                                "sub_application_areas^1.0",
                                                                "sub_application_areas.stem^0.95",
                                                                "sub_application_areas.prefix^0.1",
                                                                "sub_application_areas.joined^0.75",
                                                                "sub_application_areas.delimiter^0.4",
                                                                "dimmable^1.0",
                                                                "dimmable.stem^0.95",
                                                                "dimmable.prefix^0.1",
                                                                "dimmable.joined^0.75",
                                                                "dimmable.delimiter^0.4",
                                                                "product_codes^1.0",
                                                                "product_codes.stem^0.95",
                                                                "product_codes.prefix^0.1",
                                                                "product_codes.joined^0.75",
                                                                "product_codes.delimiter^0.4",
                                                                "application_areas^1.0",
                                                                "application_areas.stem^0.95",
                                                                "application_areas.prefix^0.1",
                                                                "application_areas.joined^0.75",
                                                                "application_areas.delimiter^0.4",
                                                                "marketing_status^1.0",
                                                                "marketing_status.stem^0.95",
                                                                "marketing_status.prefix^0.1",
                                                                "marketing_status.joined^0.75",
                                                                "marketing_status.delimiter^0.4",
                                                                "catalogs^1.0",
                                                                "catalogs.stem^0.95",
                                                                "catalogs.prefix^0.1",
                                                                "catalogs.joined^0.75",
                                                                "catalogs.delimiter^0.4",
                                                                "name^1.0",
                                                                "name.stem^0.95",
                                                                "name.prefix^0.1",
                                                                "name.joined^0.75",
                                                                "name.delimiter^0.4"
                                                            ]
                                                        }
                                                    },
                                                    {
                                                        "multi_match": {
                                                            "query": "philips",
                                                            "minimum_should_match": "1<-1 3<49%",
                                                            "type": "best_fields",
                                                            "fuzziness": "AUTO",
                                                            "prefix_length": 2,
                                                            "fields": [
                                                                "description.stem^0.1",
                                                                "locale.stem^0.1",
                                                                "value_ladder.stem^0.1",
                                                                "brand_image.stem^0.1",
                                                                "id.stem^0.1",
                                                                "categories.stem^0.1",
                                                                "sku.stem^0.1",
                                                                "brand.stem^0.1",
                                                                "image.stem^0.1",
                                                                "family_id.stem^0.1",
                                                                "sub_application_areas.stem^0.1",
                                                                "dimmable.stem^0.1",
                                                                "product_codes.stem^0.1",
                                                                "application_areas.stem^0.1",
                                                                "marketing_status.stem^0.1",
                                                                "catalogs.stem^0.1",
                                                                "name.stem^0.1"
                                                            ]
                                                        }
                                                    }
                                                ]
                                            }
                                        }
                                    ]
                                }
                            },
                            "filter": {
                                "bool": {
                                    "must": [
                                        {
                                            "range": {
                                                "start_of_publication.date": {
                                                    "lt": "2022-07-05T13:27:48.572248905Z"
                                                }
                                            }
                                        },
                                        {
                                            "range": {
                                                "end_of_publication.date": {
                                                    "gte": "2022-07-05T13:27:48.572248905Z"
                                                }
                                            }
                                        },
                                        {
                                            "terms": {
                                                "marketing_status.enum": [
                                                    "Final Published"
                                                ]
                                            }
                                        },
                                        {
                                            "range": {
                                                "power.float": {
                                                    "gte": 0.0,
                                                    "lt": 10.0
                                                }
                                            }
                                        }
                                    ]
                                }
                            }
                        }
                    },
                    "sort": [
                        {
                            "_score": "desc"
                        },
                        {
                            "_doc": "desc"
                        }
                    ],
                    "aggs": {
                        "0__brand": {
                            "terms": {
                                "size": 50,
                                "field": "brand.enum"
                            },
                            "meta": {
                                "respond_with_rfc3339_date": null
                            }
                        }
                    },
                    "highlight": {
                        "fragment_size": 300,
                        "type": "plain",
                        "number_of_fragments": 1,
                        "order": "score",
                        "encoder": "html",
                        "require_field_match": false,
                        "fields": {}
                    },
                    "size": 50,
                    "from": 0,
                    "timeout": "30000ms",
                    "_source": [
                        "description",
                        "locale",
                        "value_ladder",
                        "diameter",
                        "brand_image",
                        "price",
                        "warranty",
                        "id",
                        "categories",
                        "power",
                        "sku",
                        "stock",
                        "brand",
                        "beam_angle",
                        "height",
                        "image",
                        "family_id",
                        "luminous_flux",
                        "length",
                        "sub_application_areas",
                        "color_temp",
                        "dimmable",
                        "product_codes",
                        "application_areas",
                        "catalogs",
                        "name",
                        "width"
                    ]
                }
5

HI,

I'm on version 8.3 currently. The performance ranges from 350-500ms mostly with still sporadic spikes.

6

Hello @Carlos_D , would this still help next to the above debug flag from within the Appsearch call itself? Thanks for your reply?

7

Hi @SanderP! Sorry for the late response.

Please ensure you're on 8.3.3 version; there was a performance bug fixed for that release that was present on 8.3.0 - 8.3.2. As you already experienced this on 8.2, I don't believe you're impacted by this, but please make sure you're on 8.3.3 :+1:

Hello @Carlos_D , would this still help next to the above debug flag from within the Appsearch call itself? Thanks for your reply?

I've checked your query, and you have already checked its execution time directly on Elasticsearch, so we can rule out the query side.

Enterprise Search does more communication with Elasticsearch under the hood before actually performing the actual query. That could be part of the extra time you are experiencing versus doing the raw Elasticsearch query.

One way we can check where time is actually spent is using Application Performance Monitoring (APM). Enterprise Search supports APM integration, which you can use to understand performance issues.

I suggest checking APM for diving deeper into this issue.

8

Hello @Carlos_D ,

I'm happy to say that 8.3.3 seemed to resolve the issue. The responses are currently in the 1xx ms range and usually below 150ms. I'll still explore APM to understand the benefits for future cases and will do more testing on 8.3.3. Thanks!

Small note, we also have a v7.16 deployment which I just validated. With 338 results the response is around 350ms but stable, so the fluctuating response times really seemed to be only in v8 till 8.3.3.

Note2: There was someone in 1:1 chat reporting the same issue so I'm going to update him to validate the same on their deployment.

1 Like
9

Hey @SanderP :

Thanks for confirming upgrading to 8.3.3 worked! :+1:

10

Hello @Carlos_D ,

As a final check I tried to go back to 8.2.3 to see if it was still reproducible to exclude any other reason. Initially I ran into issues with the deployment of 8.2.3 with a snapshot from 8.3.3. During the initialization of the deployment and within snapshot restore options 8.3.3 is to be chosen, but apparently going back minor versions is not allowed: Snapshot and restore | Elasticsearch Guide [8.3] | Elastic

As an alternative I used the below approach. Do you have insights if this approach can be used for copying over Appsearch indexes nowadays now that AS is being integrated more within the landscape as a real accessible Elasticsearch index?

  1. Clean deployment 8.2.3
  2. Created an engine with the same name from the other 8.3.3 deployment
  3. Added whitelisting
  4. Reindex-remote: Reindex API | Elasticsearch Guide [8.3] | Elastic
  5. Added schema
  6. The Search command was fluctuating more once more, hence proven
11

Hi @SanderP :

Migrations will not work backwards. Always do a snapshot before upgrading so you can revert back to a previous version if needed.

The method you mention is not guaranteed to work; in particular, it does not copy over engine settings like curations, synonyms, relevance tuning... and other useful information like API Logs or analytics.

It is preferred to use App Search APIs to migrate documents and settings if needed, rather than doing specific index updates.

12

Great, thanks. This finalizes the issue for me!

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.