# Query duration too high for ~200G logs a day **URL:** \ **Category:** Elasticsearch\ **Created:** [August 23, 2017, 6:26pm UTC](https://discuss.elastic.co/t/query-duration-too-high-for-200g-logs-a-day/98117 "2017-08-23T18:26:46Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Moshe\_Saada](https://avatars.discourse-cdn.com/v4/letter/m/f07891/32.png) [@Moshe\_Saada](https://discuss.elastic.co/u/Moshe_Saada)\ **Post date:** [August 23, 2017, 6:26pm UTC](https://discuss.elastic.co/t/query-duration-too-high-for-200g-logs-a-day/98117/1 "2017-08-23T18:26:46Z")
Hi, We're using ES cluster (version 5.4.1) with 4 data nodes, 3 master, one client node (kibana). The data nodes are r4.2xlarge aws instance (61g memory, 8vCPU) with 30G memory allocated for the ES JAVA. We have writing of around 200G of logs every day and keep it for the last 14 days. A one big index of 160G-170G a day (6 shards, 1 replica) and other smaller indices of 1-3G (2 shards, 1 replica) We're dealing with performance latency in the query duration and I'm looking for recommendations to our cluster to improve the cluster performance, especially the search performance - query duration (kibana). For example, searching for the last 6 days on the big index takes: > Query Duration 51498ms > Request Duration 52706ms More data nodes? more client nodes? bigger nodes? more replica's? maybe improve the queries duration at the expense of writes speed(?) - anything that can improve the performance is an option. Is there anyone with something close to this design or loads? I'll be glad to hear about other designs, loads and query stats. Thanks, M ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [August 23, 2017, 9:30pm UTC](https://discuss.elastic.co/t/query-duration-too-high-for-200g-logs-a-day/98117/2 "2017-08-23T21:30:49Z")
What does the query look like? ---
**Author:** ![Moshe\_Saada](https://avatars.discourse-cdn.com/v4/letter/m/f07891/32.png) [@Moshe\_Saada](https://discuss.elastic.co/u/Moshe_Saada)\ **Post date:** [August 24, 2017, 6:09am UTC](https://discuss.elastic.co/t/query-duration-too-high-for-200g-logs-a-day/98117/3 "2017-08-24T06:09:42Z")
the default query of kibana "\*" : ``` "query": { "bool": { "must": [ { "query_string": { "analyze_wildcard": true, "query": "*" } }, { "range": { "@timestamp": { "gte": 1502994436234, "lte": 1503512836234, "format": "epoch_millis" } } } ], "must_not": [] } } ``` ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [August 24, 2017, 8:27am UTC](https://discuss.elastic.co/t/query-duration-too-high-for-200g-logs-a-day/98117/4 "2017-08-24T08:27:07Z")
I suspect you are hitting this bug: [https://github.com/elastic/kibana/pull/13047](https://github.com/elastic/kibana/pull/13047) whose fix will soon be released. You can work around it by setting `*:*` as a query on the Kibana side or configuring a default search field. ---
**Author:** ![Moshe\_Saada](https://avatars.discourse-cdn.com/v4/letter/m/f07891/32.png) [@Moshe\_Saada](https://discuss.elastic.co/u/Moshe_Saada)\ **Post date:** [August 24, 2017, 9:21am UTC](https://discuss.elastic.co/t/query-duration-too-high-for-200g-logs-a-day/98117/5 "2017-08-24T09:21:32Z")
I'm not sure it is, I'm getting the "discover: gateway timeout" error when I'm searching `*:*` on kibana: ``` Error: Gateway Timeout at respond (https://kibana.prod.caazz.com/bundles/kibana.bundle.js?v=15104:12:2730) at checkRespForFailure (https://kibana.prod.caazz.com/bundles/kibana.bundle.js?v=15104:12:1959) at https://kibana.prod.caazz.com/bundles/kibana.bundle.js?v=15104:1:9200 at processQueue (https://kibana.prod.caazz.com/bundles/commons.bundle.js?v=15104:38:23621) at https://kibana.prod.caazz.com/bundles/commons.bundle.js?v=15104:38:23888 at Scope.$eval (https://kibana.prod.caazz.com/bundles/commons.bundle.js?v=15104:39:4619) at Scope.$digest (https://kibana.prod.caazz.com/bundles/commons.bundle.js?v=15104:39:2359) at Scope.$apply (https://kibana.prod.caazz.com/bundles/commons.bundle.js?v=15104:39:5037) at done (https://kibana.prod.caazz.com/bundles/commons.bundle.js?v=15104:37:25027) at completeRequest (https://kibana.prod.caazz.com/bundles/commons.bundle.js?v=15104:37:28702) ``` ---
**Author:** ![Moshe\_Saada](https://avatars.discourse-cdn.com/v4/letter/m/f07891/32.png) [@Moshe\_Saada](https://discuss.elastic.co/u/Moshe_Saada)\ **Post date:** [August 24, 2017, 9:27am UTC](https://discuss.elastic.co/t/query-duration-too-high-for-200g-logs-a-day/98117/6 "2017-08-24T09:27:45Z")
Maybe more information will put some light on it, the parameters I'm using on elasticsearch.yml on the data nodes are: ``` cluster.name: prod node.name: data1.us-west-2a.prod network.host: ["127.0.0.1", " ******"] discovery.zen.ping.unicast.hosts: [" *****", "*****", " *****"] discovery.zen.minimum_master_nodes: 2 cloud.aws.s3.access_key: **** cloud.aws.s3.secret_key: ***** node.master: false node.data: true node.attr.zone: us-west-2a path.data: [/es-data-1] bootstrap.system_call_filter: false bootstrap.memory_lock: true ``` ---
**Author:** ![Moshe\_Saada](https://avatars.discourse-cdn.com/v4/letter/m/f07891/32.png) [@Moshe\_Saada](https://discuss.elastic.co/u/Moshe_Saada)\ **Post date:** [August 28, 2017, 10:46am UTC](https://discuss.elastic.co/t/query-duration-too-high-for-200g-logs-a-day/98117/7 "2017-08-28T10:46:03Z")
anyone? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 25, 2017, 10:46am UTC](https://discuss.elastic.co/t/query-duration-too-high-for-200g-logs-a-day/98117/8 "2017-09-25T10:46:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.