Unstable response-timing App Search

Dear all,

I'm looking into the performance of App Search and am noticing that this is not stable. The response-time of a identical request ranges from 200ms towards 1000+ ms. How could this be made more stable to around 200ms (or faster)? In general what influences the performance?

Engine: 600 objects
Fields: 31
Query: 99 objects returned
No ingests running and no other queries running

The performance varies when testing in Postman. Within Kibana the same can be seen by viewing .ds-logs-enterprise_search.api-default* indexes and looking at duration field.

  1. Shard cleanup - no improvements
    It is difficult to find blogs/pages that explain App Search performance and any guides of well maintaining the cluster. As underwater ElasticSearch is used I had a look at shard-counts. Initially I found advises to keep the amount of shards to 160 on a 4GB cluster. Only at a later moment I found this documentation which mentions 12000 indexes nowadays: Size your shards | Elasticsearch Guide [8.3] | Elastic

Still I cleaned the system going from 848 to 752 shards. Most indexes remaining are out of the box App Search (hidden) indexes, so further cleaning was not possible.

  1. Hardware profiling - no improvements
    Recently hardware profiles are added to the Cloud configuration portals. Seeing the documentation General Purpose would be the best fit, hence this was the latest configuration in the clone instance. As a test also CPU optimized was tested: What are hardware profiles? | Elasticsearch Service Documentation | Elastic

  2. Disabling unneeded fields in the result set - no improvements

  3. Playing around with facets (minimizing facet usage) - no improvements

  4. Increasing the deployment - no improvements
    Our monitoring on cpu and memory state that all is within the boundaries, but still as a test the whole deployment was increased from 4GB in two zones to 15GB in two zones (15GB (2 zone) - 525GB - 4vcpu storage’)

  5. Appsearch instance from 2GB to 15GB (cpu 8vcpu) - no improvements

edit: I still need to go through these by the way, but any pointers would be welcome. These came from the Elastic team, as no support could be given in a ticket. Some of these are not really related to our App Search only deployment (no scripts, no aggregations, no expensive queries, ...)

http://https//discuss.elastic.co/t/tips-to-increase-query-performance/202304
https://www.elastic.co/guide/en/app-search/8.1/troubleshooting.html#troubleshooting-debugging-queries
https://www.elastic.co/guide/en/app-search/8.1/troubleshooting.html#troubleshooting-debug-header-debugging-queries
https://www.elastic.co/guide/en/app-search/8.1/troubleshooting.html#troubleshooting-profiling-queries
https://www.elastic.co/guide/en/kibana/8.1/xpack-profiler.html
https://www.elastic.co/guide/en/elasticsearch/reference/8.1/search-profile.html
https://www.elastic.co/blog/elastic-app-search-performant-rich-search-results-with-result-settings
https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-search-speed.html
https://www.elastic.co/blog/advanced-tuning-finding-and-fixing-slow-elasticsearch-queries

Kind regards,

1 Like

Using (Troubleshooting | Elastic App Search Documentation [8.1] | Elastic) I got the raw Elasticsearch query. This result is stable around 20 ms (from debug response and within dev_tools). Profiling the same query on the involved engine returns a cumulative result of around 1.3ms with a aggregation of 0.08ms. I don't think ElasticSearch is the culprit therefor.

The reason could as well be Infra, so I will try see if the App Search query can be run from within the cluster. Another test would be to run the used ES query directly through Postman to validate the timing.

edit: Direct call to ElasticSearch with the query returned in the debug steps has a 'took' of 20 ms and a total time averaging on 50 ms. It is some times around 100ms but not higher. Conclusion so far is that Appsearch adds considerable time towards the cumulative response and is the reason for fluctuating timing (not the Infrastructure: Local network -> Internet -> App Search)

Hi @SanderP !

Could you please include the Enterprise Search version you're using? There is a known issue for performance in 8.2.0 - 8.2.2. Should that be your case, we recommend you to upgrade to 8.2.3 or higher.

If that is not the case, can you please set log_level to debug in enterprise-search.yml configuration file? That will send to the log the Elasticsearch queries performed by App Search so we can diagnose it further.

Thanks!

Hello @Carlos_D ,

It is on 8.1.2
With the debug statement of appsearch I already have the Elasticsearch query (which was performing well). Are there additional queries to be expected?

Based on App Search:
{"page":{"current":1,"size":50},"query":"philips","facets":{"brand":[{"type":"value","size":50}]},"filters":{"all":[{"start_of_publication":{"to":"2022-07-05T13:27:48.572248905Z"}},{"end_of_publication":{"from":"2022-07-05T13:27:48.572248905Z"}},{"marketing_status":["Final Published"]},{"power":{"from":0.0,"to":10.0}}]}}

{
                    "query": {
                        "bool": {
                            "must": {
                                "bool": {
                                    "must": [
                                        {
                                            "bool": {
                                                "should": [
                                                    {
                                                        "multi_match": {
                                                            "query": "philips",
                                                            "minimum_should_match": "1<-1 3<49%",
                                                            "type": "cross_fields",
                                                            "fields": [
                                                                "description^1.0",
                                                                "description.stem^0.95",
                                                                "description.prefix^0.1",
                                                                "description.joined^0.75",
                                                                "description.delimiter^0.4",
                                                                "locale^1.0",
                                                                "locale.stem^0.95",
                                                                "locale.prefix^0.1",
                                                                "locale.joined^0.75",
                                                                "locale.delimiter^0.4",
                                                                "value_ladder^1.0",
                                                                "value_ladder.stem^0.95",
                                                                "value_ladder.prefix^0.1",
                                                                "value_ladder.joined^0.75",
                                                                "value_ladder.delimiter^0.4",
                                                                "brand_image^1.0",
                                                                "brand_image.stem^0.95",
                                                                "brand_image.prefix^0.1",
                                                                "brand_image.joined^0.75",
                                                                "brand_image.delimiter^0.4",
                                                                "id^1.0",
                                                                "id.stem^0.95",
                                                                "id.prefix^0.1",
                                                                "id.joined^0.75",
                                                                "id.delimiter^0.4",
                                                                "categories^1.0",
                                                                "categories.stem^0.95",
                                                                "categories.prefix^0.1",
                                                                "categories.joined^0.75",
                                                                "categories.delimiter^0.4",
                                                                "sku^1.0",
                                                                "sku.stem^0.95",
                                                                "sku.prefix^0.1",
                                                                "sku.joined^0.75",
                                                                "sku.delimiter^0.4",
                                                                "brand^1.0",
                                                                "brand.stem^0.95",
                                                                "brand.prefix^0.1",
                                                                "brand.joined^0.75",
                                                                "brand.delimiter^0.4",
                                                                "image^1.0",
                                                                "image.stem^0.95",
                                                                "image.prefix^0.1",
                                                                "image.joined^0.75",
                                                                "image.delimiter^0.4",
                                                                "family_id^1.0",
                                                                "family_id.stem^0.95",
                                                                "family_id.prefix^0.1",
                                                                "family_id.joined^0.75",
                                                                "family_id.delimiter^0.4",
                                                                "sub_application_areas^1.0",
                                                                "sub_application_areas.stem^0.95",
                                                                "sub_application_areas.prefix^0.1",
                                                                "sub_application_areas.joined^0.75",
                                                                "sub_application_areas.delimiter^0.4",
                                                                "dimmable^1.0",
                                                                "dimmable.stem^0.95",
                                                                "dimmable.prefix^0.1",
                                                                "dimmable.joined^0.75",
                                                                "dimmable.delimiter^0.4",
                                                                "product_codes^1.0",
                                                                "product_codes.stem^0.95",
                                                                "product_codes.prefix^0.1",
                                                                "product_codes.joined^0.75",
                                                                "product_codes.delimiter^0.4",
                                                                "application_areas^1.0",
                                                                "application_areas.stem^0.95",
                                                                "application_areas.prefix^0.1",
                                                                "application_areas.joined^0.75",
                                                                "application_areas.delimiter^0.4",
                                                                "marketing_status^1.0",
                                                                "marketing_status.stem^0.95",
                                                                "marketing_status.prefix^0.1",
                                                                "marketing_status.joined^0.75",
                                                                "marketing_status.delimiter^0.4",
                                                                "catalogs^1.0",
                                                                "catalogs.stem^0.95",
                                                                "catalogs.prefix^0.1",
                                                                "catalogs.joined^0.75",
                                                                "catalogs.delimiter^0.4",
                                                                "name^1.0",
                                                                "name.stem^0.95",
                                                                "name.prefix^0.1",
                                                                "name.joined^0.75",
                                                                "name.delimiter^0.4"
                                                            ]
                                                        }
                                                    },
                                                    {
                                                        "multi_match": {
                                                            "query": "philips",
                                                            "minimum_should_match": "1<-1 3<49%",
                                                            "type": "best_fields",
                                                            "fuzziness": "AUTO",
                                                            "prefix_length": 2,
                                                            "fields": [
                                                                "description.stem^0.1",
                                                                "locale.stem^0.1",
                                                                "value_ladder.stem^0.1",
                                                                "brand_image.stem^0.1",
                                                                "id.stem^0.1",
                                                                "categories.stem^0.1",
                                                                "sku.stem^0.1",
                                                                "brand.stem^0.1",
                                                                "image.stem^0.1",
                                                                "family_id.stem^0.1",
                                                                "sub_application_areas.stem^0.1",
                                                                "dimmable.stem^0.1",
                                                                "product_codes.stem^0.1",
                                                                "application_areas.stem^0.1",
                                                                "marketing_status.stem^0.1",
                                                                "catalogs.stem^0.1",
                                                                "name.stem^0.1"
                                                            ]
                                                        }
                                                    }
                                                ]
                                            }
                                        }
                                    ]
                                }
                            },
                            "filter": {
                                "bool": {
                                    "must": [
                                        {
                                            "range": {
                                                "start_of_publication.date": {
                                                    "lt": "2022-07-05T13:27:48.572248905Z"
                                                }
                                            }
                                        },
                                        {
                                            "range": {
                                                "end_of_publication.date": {
                                                    "gte": "2022-07-05T13:27:48.572248905Z"
                                                }
                                            }
                                        },
                                        {
                                            "terms": {
                                                "marketing_status.enum": [
                                                    "Final Published"
                                                ]
                                            }
                                        },
                                        {
                                            "range": {
                                                "power.float": {
                                                    "gte": 0.0,
                                                    "lt": 10.0
                                                }
                                            }
                                        }
                                    ]
                                }
                            }
                        }
                    },
                    "sort": [
                        {
                            "_score": "desc"
                        },
                        {
                            "_doc": "desc"
                        }
                    ],
                    "aggs": {
                        "0__brand": {
                            "terms": {
                                "size": 50,
                                "field": "brand.enum"
                            },
                            "meta": {
                                "respond_with_rfc3339_date": null
                            }
                        }
                    },
                    "highlight": {
                        "fragment_size": 300,
                        "type": "plain",
                        "number_of_fragments": 1,
                        "order": "score",
                        "encoder": "html",
                        "require_field_match": false,
                        "fields": {}
                    },
                    "size": 50,
                    "from": 0,
                    "timeout": "30000ms",
                    "_source": [
                        "description",
                        "locale",
                        "value_ladder",
                        "diameter",
                        "brand_image",
                        "price",
                        "warranty",
                        "id",
                        "categories",
                        "power",
                        "sku",
                        "stock",
                        "brand",
                        "beam_angle",
                        "height",
                        "image",
                        "family_id",
                        "luminous_flux",
                        "length",
                        "sub_application_areas",
                        "color_temp",
                        "dimmable",
                        "product_codes",
                        "application_areas",
                        "catalogs",
                        "name",
                        "width"
                    ]
                }

HI,

I'm on version 8.3 currently. The performance ranges from 350-500ms mostly with still sporadic spikes.

Hello @Carlos_D , would this still help next to the above debug flag from within the Appsearch call itself? Thanks for your reply?