# Is my response time is ok? **URL:** \ **Category:** Elasticsearch\ **Created:** [June 3, 2018, 4:45pm UTC](https://discuss.elastic.co/t/is-my-response-time-is-ok/134326 "2018-06-03T16:45:33Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![artpolikarpov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/artpolikarpov/32/31875_2.png) [@artpolikarpov](https://discuss.elastic.co/u/artpolikarpov)\ **Post date:** [June 3, 2018, 4:45pm UTC](https://discuss.elastic.co/t/is-my-response-time-is-ok/134326/1 "2018-06-03T16:45:33Z")
Hi there! I am a novice in ES. And I want to know if I’m doing something wrong or trying to achieve impossible things. I think my searches are slow. I have index `messages` (22.9 GB, 77m docs). Mapping is simple: ```auto esClient.indices.create({ index: 'messages', body: { mappings: { messages: { properties: { hostId: {type: 'keyword'}, clientId: {type: 'keyword'}, userId: {type: 'keyword'}, createdAt: {type: 'long'}, message: {type: 'text', fields: { ngrams: { type: 'text', analyzer: 'ngrams' } } } } } }, settings: { analysis: { analyzer: { default: { type: 'custom', filter: ['lowercase'], tokenizer: 'whitespace' }, ngrams: { type: 'custom', filter: ['lowercase', 'custom_edge_ngram'], tokenizer: 'whitespace' } }, filter: { custom_edge_ngram: { type: 'edge_ngram', min_gram: 1, max_gram: 20, token_chars: [ 'letter', 'digit' ] } } } } } }); ``` Query is simple: ```auto esClient.search({ index: 'messages', from: 0, size: 10, terminate_after: 1000, body: { profile, query: { bool: { must: { simple_query_string: { query, fields: ['message', 'message.ngrams'], analyzer: 'whitespace', default_operator: 'and' } }, filter: { term: {hostId} } } }, sort: { _score: {order: 'desc'}, createdAt: {order: 'desc'} } } } ``` But average response time for this index is about 1-3 sec, not quite acceptable for search-as-you-type experience. I am using Elastic Cloud (192/8gb plan). Please help! ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [June 3, 2018, 5:00pm UTC](https://discuss.elastic.co/t/is-my-response-time-is-ok/134326/2 "2018-06-03T17:00:28Z")
Please see [https://www.elastic.co/cloud/as-a-service/support](https://www.elastic.co/cloud/as-a-service/support) on how to raise a support ticket for Elastic Cloud 🙂 How many shards you have? What monitoring is saying? Is it happening only for the first query or always? ---
**Author:** ![artpolikarpov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/artpolikarpov/32/31875_2.png) [@artpolikarpov](https://discuss.elastic.co/u/artpolikarpov)\ **Post date:** [June 3, 2018, 5:11pm UTC](https://discuss.elastic.co/t/is-my-response-time-is-ok/134326/3 "2018-06-03T17:11:49Z")
5 shards for that index - default. It happens every time. I have another index (clients) with quite the same mapping and query but smaller in size, and searches for that index is a bit quicker - about 0.5-1.5 sec. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [June 3, 2018, 6:54pm UTC](https://discuss.elastic.co/t/is-my-response-time-is-ok/134326/4 "2018-06-03T18:54:30Z")
I guess you have at least 2 nodes? ---
**Author:** ![artpolikarpov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/artpolikarpov/32/31875_2.png) [@artpolikarpov](https://discuss.elastic.co/u/artpolikarpov)\ **Post date:** [June 3, 2018, 7:18pm UTC](https://discuss.elastic.co/t/is-my-response-time-is-ok/134326/5 "2018-06-03T19:18:13Z")
I guess only one at the moment (192/8gb). There is an option for high availability to spread nodes across regions — I’ll try to add more. But now I am only the person how run queries, so there is no load on system, this is not in production yet. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [June 3, 2018, 7:49pm UTC](https://discuss.elastic.co/t/is-my-response-time-is-ok/134326/6 "2018-06-03T19:49:20Z")
Could you run the query with profile: true so we can better understand where the time is spent? ---
**Author:** ![artpolikarpov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/artpolikarpov/32/31875_2.png) [@artpolikarpov](https://discuss.elastic.co/u/artpolikarpov)\ **Post date:** [June 4, 2018, 7:36am UTC](https://discuss.elastic.co/t/is-my-response-time-is-ok/134326/7 "2018-06-04T07:36:46Z")
David, thank you. Here is example of query: ```auto { "query":{ "bool": { "must": { "simple_query_string": { "query": "привет", "fields": ["message"], "analyzer": "whitespace", "default_operator": "and" } }, "filter": { "term": {"hostId": "hX8ihkAcyHK93ue99"} } } } } ``` Screenshots of Kibana profiler: [https://d.pr/i/4g5pe1](https://d.pr/i/4g5pe1), [https://d.pr/i/xLLUcS](https://d.pr/i/xLLUcS) Here is another query, this time faster: [https://d.pr/i/JZUVvH](https://d.pr/i/JZUVvH) Few json-profiles for queries limited to 10 hits: - "ok fine": [https://pastebin.com/Ejtth15d](https://pastebin.com/Ejtth15d) - "привет": [https://pastebin.com/SiJ6Qmwk](https://pastebin.com/SiJ6Qmwk) - "david": [https://pastebin.com/LtrA5Kc6](https://pastebin.com/LtrA5Kc6) When querying another index (clients) timing is ok, screenshot: [https://d.pr/i/Xui19E](https://d.pr/i/Xui19E) ---
**Author:** ![artpolikarpov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/artpolikarpov/32/31875_2.png) [@artpolikarpov](https://discuss.elastic.co/u/artpolikarpov)\ **Post date:** [June 4, 2018, 8:01am UTC](https://discuss.elastic.co/t/is-my-response-time-is-ok/134326/8 "2018-06-04T08:01:17Z")
Another query: ```auto { "query":{ "bool": { "must": { "simple_query_string": { "query": "да", "fields": ["message", "message.ngrams"], "analyzer": "whitespace", "default_operator": "and" } }, "filter": { "term": {"hostId": "JJkh6K8yFERoyvQbx"} } } } } ``` Profiler screen, 4.5 sec response: [https://d.pr/i/R5zZYI](https://d.pr/i/R5zZYI) ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [June 4, 2018, 8:13am UTC](https://discuss.elastic.co/t/is-my-response-time-is-ok/134326/9 "2018-06-04T08:13:31Z")
Interesting. So anytime you are using non ASCII characters it's slow, otherwise it's "fast". @jpountz What do you think? ---
**Author:** ![artpolikarpov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/artpolikarpov/32/31875_2.png) [@artpolikarpov](https://discuss.elastic.co/u/artpolikarpov)\ **Post date:** [June 4, 2018, 9:52am UTC](https://discuss.elastic.co/t/is-my-response-time-is-ok/134326/10 "2018-06-04T09:52:08Z")
Not really, it can be slow and for english query, "yes" for example: [https://d.pr/i/0eF6jD](https://d.pr/i/0eF6jD) ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [June 4, 2018, 4:12pm UTC](https://discuss.elastic.co/t/is-my-response-time-is-ok/134326/11 "2018-06-04T16:12:01Z")
Your approach looks ok to me. The profiler output suggests that most time is spent in weight creation, whose main task is to look up terms in the terms dictionary to have access to terms statistics. So this might be caused by a busy disk? Would you be able to run some queries with `"sort": ["_doc"]` and share the output of a slow query? ---
**Author:** ![artpolikarpov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/artpolikarpov/32/31875_2.png) [@artpolikarpov](https://discuss.elastic.co/u/artpolikarpov)\ **Post date:** [June 4, 2018, 4:33pm UTC](https://discuss.elastic.co/t/is-my-response-time-is-ok/134326/12 "2018-06-04T16:33:01Z")
@jpountz, thanks for joining! Hmm, I believe disk is not busy. I have now 2 nodes (192gb/8gb) - and I am the only client who run queries against this Cluster. Few screens from dashboard: [https://d.pr/i/NQrNNY](https://d.pr/i/NQrNNY), [https://d.pr/i/6pr7ms](https://d.pr/i/6pr7ms) (hmm, but I don’t understand why so many search requests on picture) Here is few profile screens with `sort: ['_doc']`: [https://d.pr/i/UgGg28](https://d.pr/i/UgGg28), [https://d.pr/i/F1HPkL](https://d.pr/i/F1HPkL) When I run almost the same query for another index (clients), speed is better: [https://d.pr/i/5eV4pi](https://d.pr/i/5eV4pi) Looks like size of the indices matters: messages — 71m (22gb), clients — 8m (5.3) ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [June 4, 2018, 4:50pm UTC](https://discuss.elastic.co/t/is-my-response-time-is-ok/134326/13 "2018-06-04T16:50:12Z")
Thanks for sharing. This added sort moved the terms lookup from `create_weight` to `build_scorer` and it turns out that the latter is now the bottleneck, which strongly suggests that looking up terms is what makes your queries slow. Are these suggestions indices read-only? If yes then a force-merge would likely help as it would decrease the number of segments that need to be looked up. If no, then can you share the number of segments that you have in your shards? If the disk is not busy, then there is still a possibility that it is just slow. On spinning disks, random access might take ~10ms if the filesystem cache is cold. So if you have 50 segments, that could be 50x10=500ms only to look up terms in the terms dictionary. If the size of your filesystem cache is expected to be larger than the size of your terms dictionary (accessible via the node stats API by passing `include_segment_file_sizes=true`) then we could look into forcing the terms dict to be loaded into the filesystem cache. ---
**Author:** ![artpolikarpov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/artpolikarpov/32/31875_2.png) [@artpolikarpov](https://discuss.elastic.co/u/artpolikarpov)\ **Post date:** [June 6, 2018, 6:07am UTC](https://discuss.elastic.co/t/is-my-response-time-is-ok/134326/14 "2018-06-06T06:07:30Z")
> Are these suggestions indices read-only? No. It's a dynamic index. > On spinning disks... Hmm, I’m not sure what disks uses Elastic Cloud. > So if you have 50 segments... Please have a look to the API response for `/messages/_segments`: [{ "\_shards": { "total": 10, "successful": 5, "failed": 0 }, - Pastebin.com](https://pastebin.com/3RgTniEU). Any optimising ideas? Or should I move to a more powerful hardware? ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [June 6, 2018, 8:38am UTC](https://discuss.elastic.co/t/is-my-response-time-is-ok/134326/15 "2018-06-06T08:38:51Z")
Sorry, I had forgotten this was Elastic Cloud, so you are on SSDs. Thanks for sharing the segments output. Everything looks very sane. More powerful hardware would certainly help but I'm a bit surprised your current plan doesn't perform better than that. One last thing I'd like to look at: could you run some slow queries in a loop and concurrently get the output of the nodes hot threads API a couple times at a couple seconds interval? This might give us indications about the bottleneck of these queries. ---
**Author:** ![artpolikarpov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/artpolikarpov/32/31875_2.png) [@artpolikarpov](https://discuss.elastic.co/u/artpolikarpov)\ **Post date:** [June 8, 2018, 10:46am UTC](https://discuss.elastic.co/t/is-my-response-time-is-ok/134326/16 "2018-06-08T10:46:07Z")
Here few hot\_threads requests: [https://pastebin.com/aytRt7xK](https://pastebin.com/aytRt7xK), [https://pastebin.com/DV3VYsTq](https://pastebin.com/DV3VYsTq), [https://pastebin.com/k6uJewU5](https://pastebin.com/k6uJewU5), [https://pastebin.com/s29PMDan](https://pastebin.com/s29PMDan) Finally, we decided to move away from Elastic Cloud. On our server with same indices requests took 3-9ms, not 5000-10000 😕 Thank you for discussion! ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [June 8, 2018, 11:24am UTC](https://discuss.elastic.co/t/is-my-response-time-is-ok/134326/17 "2018-06-08T11:24:34Z")
Oh. That's very good to know. Sometimes it could happen that clicking on restart the service helps as it allocates the nodes elsewhere. In all cases I'd definitely open a ticket to the cloud support team. ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [June 8, 2018, 2:58pm UTC](https://discuss.elastic.co/t/is-my-response-time-is-ok/134326/18 "2018-06-08T14:58:53Z")
Sorry to hear that, I'm disappointed by these response times too given that documents and queries are well designed. Hot threads look almost idle, only one of them shows an indexing request. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2018, 2:59pm UTC](https://discuss.elastic.co/t/is-my-response-time-is-ok/134326/19 "2018-07-06T14:59:12Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.