One of my customer is using App Search to search among an engine that contains less than 1000 documents, each having 70 fields (we had to raise the limit on the fields number).
The App Search instance is deployed on Elastic Cloud and has a pretty good configuration (with a CPU Optimized hardware, 2 x 8 GB Elasticsearch, 2 x 4 GB Enterprise Search). The version is 8.5.3.
However, issuing queries shows some bad response time and I decided to dig using APM.
Let's take this transaction sample :
We can see here that the Enterprise Search server take some milliseconds to get all of its meta data (which engine to use, corresponding to which index, etc.) then there is a big amount of time spent before actually issuing the queries to Elasticsearch, which takes a total amount of 29 ms.
To sum up, operations against Elasticsearch are performed in 55.1 ms but the transaction total time is 170 ms.
The multi_search query sampled here is made of 6 queries, each of them computing facets.
I made various tests about not computing facets, returning only one field, etc. but the overall response time does not change significantly.
Do we reach here some "glass ceiling" in terms of performance or is it something wrong on my side ? What are the performance tests showing at Elastic ?
Thanks
Thanks for your detailed explanation and the APM trace you shared.
The performance your customer is experiencing is expected from the App Search side. App Search performs multiple Elasticsearch queries and related processing for each search.
These requests are done in order to apply the Engine configuration (relevance tuning, result fields, synonyms, curations, etc) and issue the appropriate Elasticsearch query.
Our customers find these response times satisfactory, including e-commerce sites. Please check that this is a problem from the customer perspective.
In case your customer needs a faster response time, there is the following procedure that can be used:
Use the Search Explain API to retrieve the Elasticsearch query that App Search will use.
Issue the query directly to Elasticsearch, or using the Elasticsearch search API from App Search.
Using some caching on the frontend side for the retrieved Elasticsearch query, this will allow to be as close as possible to raw Elasticsearch performance.
Thanks a lot for these explanations.
If I am not wrong, the Search explain API will only explain which queries are really performed against Elasticsearch but IMO these are not the bottleneck here.
I agree that I can issue the query directly to ES but then I will have to know exactly which is the structure of the underlying index, which is not what my customer expects from App Search as a tool whose role is to act as a facade.
Caching might help indeed, but in this case there is a lot of document updates so ...
APM shows that most of the time is spent in the Enterprise Search Server itself (not issuing Elasticsearch queries I mean), are there some plan on your side to speed up things on the processing side ?
Thanks again
the Search explain API will only explain which queries are really performed against Elasticsearch but IMO these are not the bottleneck here.
The Search Explain API will retrieve the final Elasticsearch search that is done by Enterprise Search.
That includes the transformations done by the Enterprise Search server as well, not just the Elasticsearch requests.
I was thinking on retrieving the corresponding query for a fixed query string, and then replacing that query string with the actual query issued by the user. I was thinking on the following course of action:
The UI client issues a search that hits the backend
The backend uses the Search Explain API to retrieve the Elasticsearch query and caches it
The backend uses the cached Elasticsearch API to perform.
However, that won't work if there are synonyms or curations defined for the Engine, as those modify the issued query. It's more a workaround for a very specific use case than a general guidance.
I agree that I can issue the query directly to ES but then I will have to know exactly which is the structure of the underlying index, which is not what my customer expects from App Search as a tool whose role is to act as a facade.
Completely agree! App Search does some processing in order to help simplifying the query. It's a tradeoff between raw Elasticsearch efficiency, and schema and query flexibility.
If raw efficiency is a must, and it's a more desirable quality than flexibility, analytics, simplified security, curations, synonym management, simplified relevance tuning (and all the rest of the Enterprise Search features), then App Search should be reconsidered as the ideal tool for this particular use case.
are there some plan on your side to speed up things on the processing side ?
We will address that in the near future. Stay tuned!
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
