I'm looking into the performance of App Search and am noticing that this is not stable. The response-time of a identical request ranges from 200ms towards 1000+ ms. How could this be made more stable to around 200ms (or faster)? In general what influences the performance?
Engine: 600 objects
Query: 99 objects returned
No ingests running and no other queries running
The performance varies when testing in Postman. Within Kibana the same can be seen by viewing .ds-logs-enterprise_search.api-default* indexes and looking at duration field.
- Shard cleanup - no improvements
It is difficult to find blogs/pages that explain App Search performance and any guides of well maintaining the cluster. As underwater ElasticSearch is used I had a look at shard-counts. Initially I found advises to keep the amount of shards to 160 on a 4GB cluster. Only at a later moment I found this documentation which mentions 12000 indexes nowadays: Size your shards | Elasticsearch Guide [8.3] | Elastic
Still I cleaned the system going from 848 to 752 shards. Most indexes remaining are out of the box App Search (hidden) indexes, so further cleaning was not possible.
Hardware profiling - no improvements
Recently hardware profiles are added to the Cloud configuration portals. Seeing the documentation General Purpose would be the best fit, hence this was the latest configuration in the clone instance. As a test also CPU optimized was tested: What are hardware profiles? | Elasticsearch Service Documentation | Elastic
Disabling unneeded fields in the result set - no improvements
Playing around with facets (minimizing facet usage) - no improvements
Increasing the deployment - no improvements
Our monitoring on cpu and memory state that all is within the boundaries, but still as a test the whole deployment was increased from 4GB in two zones to 15GB in two zones (15GB (2 zone) - 525GB - 4vcpu storage’)
Appsearch instance from 2GB to 15GB (cpu 8vcpu) - no improvements
edit: I still need to go through these by the way, but any pointers would be welcome. These came from the Elastic team, as no support could be given in a ticket. Some of these are not really related to our App Search only deployment (no scripts, no aggregations, no expensive queries, ...)
http://https//discuss.elastic.co/t/tips-to-increase-query-performance/202304 https://www.elastic.co/guide/en/app-search/8.1/troubleshooting.html#troubleshooting-debugging-queries https://www.elastic.co/guide/en/app-search/8.1/troubleshooting.html#troubleshooting-debug-header-debugging-queries https://www.elastic.co/guide/en/app-search/8.1/troubleshooting.html#troubleshooting-profiling-queries https://www.elastic.co/guide/en/kibana/8.1/xpack-profiler.html https://www.elastic.co/guide/en/elasticsearch/reference/8.1/search-profile.html https://www.elastic.co/blog/elastic-app-search-performant-rich-search-results-with-result-settings https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-search-speed.html https://www.elastic.co/blog/advanced-tuning-finding-and-fixing-slow-elasticsearch-queries