# Our elastic search query performance is VERY low
**URL:** https://discuss.elastic.co/t/our-elastic-search-query-performance-is-very-low/82108
**Category:** Elasticsearch
**Created:** [April 12, 2017, 7:44am UTC](https://discuss.elastic.co/t/our-elastic-search-query-performance-is-very-low/82108 "2017-04-12T07:44:25Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@Indyan\_Minerva](https://discuss.elastic.co/u/Indyan_Minerva)
#### Post date: [April 12, 2017, 7:44am UTC](https://discuss.elastic.co/t/our-elastic-search-query-performance-is-very-low/82108/1 "2017-04-12T07:44:25Z")
Hi,
I have a 5 node ES cluster (each node is single core and 4Gig RAM) which is receiving data from metricbeat and winbeat via logstash. The data generally amounts to 175 GB and is stored in a per-day index.
Even when I search for a data for an hour, our queries are taking very long time.
Below is our config :
[cluster.name](http://cluster.name): clustername
[node.name](http://node.name): ${HOSTNAME}
path.data: /apps/elasticsearch-5.2.2/data,/data1,/data2
bootstrap.memory\_lock: true
node.data: true
node.master: true
node.ingest: true
network.host: 0.0.0.0
discovery.zen.ping.unicast.hosts: ["node1", "node2","node3", "node4", "node5"]
discovery.zen.ping\_timeout: 30s
discovery.zen.minimum\_master\_nodes: 3
thread\_pool.bulk.queue\_size: 1000
xpack.security.enabled: false
indices.memory.index\_buffer\_size: 30%
indices.memory.min\_index\_buffer\_size: 512mb
Am I doing something wrong?
Do I need to customise my mapping? Do I need to store a week/month's data per index?
Thanks
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [April 12, 2017, 8:03am UTC](https://discuss.elastic.co/t/our-elastic-search-query-performance-is-very-low/82108/2 "2017-04-12T08:03:37Z")
4gb of RAM ? So something like 2gb of Java HEAP?
I believe you are seeing some few things in your logs, don't you?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 12, 2017, 11:59am UTC](https://discuss.elastic.co/t/our-elastic-search-query-performance-is-very-low/82108/3 "2017-04-12T11:59:11Z")
> [@Indyan\_Minerva](#):
>
> each node is single core and 4Gig RAM
Indexing as well as querying can be CPU intensive, so I suspect you may be limited by the amount of CPU available.
---
### Author:  [@Indyan\_Minerva](https://discuss.elastic.co/u/Indyan_Minerva)
#### Post date: [April 13, 2017, 1:59am UTC](https://discuss.elastic.co/t/our-elastic-search-query-performance-is-very-low/82108/4 "2017-04-13T01:59:50Z")
Except for an occasional error about some node not able to reach any of the masters, I don't see any errros. Well! I would expect to see a lot of OOMs but no trace of such errors.
My heap size is 3Gig.
---
### Author:  [@Indyan\_Minerva](https://discuss.elastic.co/u/Indyan_Minerva)
#### Post date: [April 13, 2017, 2:03am UTC](https://discuss.elastic.co/t/our-elastic-search-query-performance-is-very-low/82108/5 "2017-04-13T02:03:34Z")
I am experimenting with 64 Gig (planning to have a JVM heap of 31 Gig and 64 core node. Although my indexing rates went upto docs 20k/s, my search (running some 10 queries parallelly from a visualisation) is taking morethan 30 seconds.
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [April 13, 2017, 2:51am UTC](https://discuss.elastic.co/t/our-elastic-search-query-performance-is-very-low/82108/6 "2017-04-13T02:51:30Z")
But nothing related to GC in logs?
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [April 13, 2017, 2:52am UTC](https://discuss.elastic.co/t/our-elastic-search-query-performance-is-very-low/82108/7 "2017-04-13T02:52:06Z")
BTW what is a typical query is looking like?
---
### Author:  [@Indyan\_Minerva](https://discuss.elastic.co/u/Indyan_Minerva)
#### Post date: [April 13, 2017, 4:50am UTC](https://discuss.elastic.co/t/our-elastic-search-query-performance-is-very-low/82108/8 "2017-04-13T04:50:21Z")
I was expecting the logs to be full of GCs with the slowness we are seeing but GC not is happening that often.
Query-Type1:
{"query":{"bool":{"must":[{"query\_string":{"query":"tags:flame","analyze\_wildcard":true}},{"query\_string":{"analyze\_wildcard":true,"query":"\*"}},{"range":{"@timestamp":{"gte":1491676200000,"lte":1492280999999,"format":"epoch\_millis"}}}],"must\_not":[]}},"size":0,"\_source":{"excludes":[]},"aggs":{"2":{"date\_histogram":{"field":"@timestamp","interval":"3h","time\_zone":"Asia/Kolkata","min\_doc\_count":1},"aggs":{"1":{"avg":{"field":"system.cpu.user.pct"}},"3":{"avg":{"field":"system.cpu.system.pct"}}}}}}
Query-Type2 :
{"query":{"bool":{"must":[{"query\_string":{"query":"tags: hadoop","analyze\_wildcard":true}},{"query\_string":{"analyze\_wildcard":true,"query":"\*"}},{"range":{"@timestamp":{"gte":1491676200000,"lte":1492280999999,"format":"epoch\_millis"}}}],"must\_not":[]}},"size":0,"\_source":{"excludes":[]},"aggs":{"1":{"percentiles":{"field":"score","percents":[50],"keyed":false}},"2":{"min":{"field":"score"}},"3":{"max":{"field":"score"}}}}
We have a 9 of these queries being fired from a dashboard parallelly and that is dashboard is taking some 45 seconds to load.
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [April 13, 2017, 7:21am UTC](https://discuss.elastic.co/t/our-elastic-search-query-performance-is-very-low/82108/9 "2017-04-13T07:21:43Z")
Do you have the same response time when running the same query outside Kibana?
For sure you don't have enough memory for the file system cache so you are probably always reading data from disk. Are you using SSD drives?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 13, 2017, 7:40am UTC](https://discuss.elastic.co/t/our-elastic-search-query-performance-is-very-low/82108/10 "2017-04-13T07:40:25Z")
Querying and indexing shares/competes for the same resources, so I would recommend monitoring CPU, disk I/O and GC while you are indexing and querying simultaneously. If you are running indexing at full speed, try increasing indexing throughput gradually, e.g. by altering the number of indexing threads, and see how increased indexing throughput affects query latency. Start without any indexing at all so you have a baseline for your query performance.
---
### Author:  [@Indyan\_Minerva](https://discuss.elastic.co/u/Indyan_Minerva)
#### Post date: [April 13, 2017, 8:46am UTC](https://discuss.elastic.co/t/our-elastic-search-query-performance-is-very-low/82108/11 "2017-04-13T08:46:56Z")
Each query is taking only 80 milli-seconds (it initially takes 4 seconds though). But even when I run the queries paralelly, the ensemble of queries is taking around 55 seconds (this is same as when we run them serially). Initially I though this must be related to the queue sizes. But it turned out to be not.
No we are not using SDDs.
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [April 13, 2017, 10:55am UTC](https://discuss.elastic.co/t/our-elastic-search-query-performance-is-very-low/82108/12 "2017-04-13T10:55:57Z")
It can't be fast with so little RAM and spinning disks IMO. The dataset is 175gb! So you read everything from disk here I think.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 11, 2017, 11:03am UTC](https://discuss.elastic.co/t/our-elastic-search-query-performance-is-very-low/82108/13 "2017-05-11T11:03:39Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.