# Elastic Search Goes Red **URL:** \ **Category:** Elasticsearch\ **Created:** [July 14, 2018, 5:33pm UTC](https://discuss.elastic.co/t/elastic-search-goes-red/140037 "2018-07-14T17:33:37Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![adwaitjoshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adwaitjoshi/32/35098_2.png) [@adwaitjoshi](https://discuss.elastic.co/u/adwaitjoshi)\ **Post date:** [July 14, 2018, 5:33pm UTC](https://discuss.elastic.co/t/elastic-search-goes-red/140037/1 "2018-07-14T17:33:38Z")
I have a single node elastic cluster with replication turned off. I have significant amount of data in the system (about 300MM records). When I load my Visualizations they load but are slow. However when I try to open dashboards, they not only take much longer (I understand due to multiple connections to the underlying visualizations) but they also error out and put ElasticSearch in a red status. I have a pretty powerful system with 8C/64GB/4.8TB SSD. What part of configuration would I have to change in order to fix this issue, I am assuming that 300MM is not too much for Elastic to handle on one node. So something else must be wrong. Any advice is welcome. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 14, 2018, 7:07pm UTC](https://discuss.elastic.co/t/elastic-search-goes-red/140037/2 "2018-07-14T19:07:23Z")
What is the output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/6.3/cluster-stats.html)? ---
**Author:** ![adwaitjoshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adwaitjoshi/32/35098_2.png) [@adwaitjoshi](https://discuss.elastic.co/u/adwaitjoshi)\ **Post date:** [July 14, 2018, 8:52pm UTC](https://discuss.elastic.co/t/elastic-search-goes-red/140037/3 "2018-07-14T20:52:28Z")
{ "\_nodes": { "total": 1, "successful": 1, "failed": 0 }, "cluster\_name": "dataseers", "timestamp": 1531601516167, "status": "yellow", "indices": { "count": 33, "shards": { "total": 45, "primaries": 45, "replication": 0, "index": { "shards": { "min": 1, "max": 5, "avg": 1.3636363636363635 }, "primaries": { "min": 1, "max": 5, "avg": 1.3636363636363635 }, "replication": { "min": 0, "max": 0, "avg": 0 } } }, "docs": { "count": 390889190, "deleted": 61556174 }, "store": { "size": "337.8gb", "size\_in\_bytes": 362790191389 }, "fielddata": { "memory\_size": "5.5mb", "memory\_size\_in\_bytes": 5794136, "evictions": 0 }, "query\_cache": { "memory\_size": "0b", "memory\_size\_in\_bytes": 0, "total\_count": 0, "hit\_count": 0, "miss\_count": 0, "cache\_size": 0, "cache\_count": 0, "evictions": 0 }, "completion": { "size": "0b", "size\_in\_bytes": 0 }, "segments": { "count": 566, "memory": "809.3mb", "memory\_in\_bytes": 848656687, "terms\_memory": "650mb", "terms\_memory\_in\_bytes": 681591121, "stored\_fields\_memory": "118.8mb", "stored\_fields\_memory\_in\_bytes": 124644680, "term\_vectors\_memory": "0b", "term\_vectors\_memory\_in\_bytes": 0, "norms\_memory": "1mb", "norms\_memory\_in\_bytes": 1053824, "points\_memory": "34.9mb", "points\_memory\_in\_bytes": 36686758, "doc\_values\_memory": "4.4mb", "doc\_values\_memory\_in\_bytes": 4680304, "index\_writer\_memory": "4.9mb", "index\_writer\_memory\_in\_bytes": 5165080, "version\_map\_memory": "1.5mb", "version\_map\_memory\_in\_bytes": 1654897, "fixed\_bit\_set": "12.6kb", "fixed\_bit\_set\_memory\_in\_bytes": 12960, "max\_unsafe\_auto\_id\_timestamp": 1531531162502, "file\_sizes": {} } }, "nodes": { "count": { "total": 1, "data": 1, "coordinating\_only": 0, "master": 1, "ingest": 1 }, "versions": [ "6.3.1" ], "os": { "available\_processors": 16, "allocated\_processors": 16, "names": [ { "name": "Linux", "count": 1 } ], "mem": { "total": "62.5gb", "total\_in\_bytes": 67126927360, "free": "37.9gb", "free\_in\_bytes": 40721190912, "used": "24.5gb", "used\_in\_bytes": 26405736448, "free\_percent": 61, "used\_percent": 39 } }, "process": { "cpu": { "percent": 0 }, "open\_file\_descriptors": { "min": 927, "max": 927, "avg": 927 } }, "jvm": { "max\_uptime": "19.5h", "max\_uptime\_in\_millis": 70368444, "versions": [ { "version": "1.8.0\_171", "vm\_name": "OpenJDK 64-Bit Server VM", "vm\_version": "25.171-b10", "vm\_vendor": "Oracle Corporation", "count": 1 } ], "mem": { "heap\_used": "4.6gb", "heap\_used\_in\_bytes": 4962910312, "heap\_max": "7.8gb", "heap\_max\_in\_bytes": 8476557312 }, "threads": 229 }, "fs": { "total": "4.2tb", "total\_in\_bytes": 4708418715648, "free": "3.7tb", "free\_in\_bytes": 4165199818752, "available": "3.7tb", "available\_in\_bytes": 4165199818752 }, "plugins": [], "network\_types": { "transport\_types": { "security4": 1 }, "http\_types": { "security4": 1 } } } } ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 14, 2018, 9:36pm UTC](https://discuss.elastic.co/t/elastic-search-goes-red/140037/4 "2018-07-14T21:36:26Z")
That looks fine. Is there anything in the Elasticsearch logs? ---
**Author:** ![adwaitjoshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adwaitjoshi/32/35098_2.png) [@adwaitjoshi](https://discuss.elastic.co/u/adwaitjoshi)\ **Post date:** [July 15, 2018, 2:37pm UTC](https://discuss.elastic.co/t/elastic-search-goes-red/140037/5 "2018-07-15T14:37:15Z")
Unfortunately it has not broken since I posted this. I am trying to see if multiple users hitting the system at the same time is causing it. I will post the logs as soon as it fails. ---
**Author:** ![adwaitjoshi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adwaitjoshi/32/35098_2.png) [@adwaitjoshi](https://discuss.elastic.co/u/adwaitjoshi)\ **Post date:** [July 25, 2018, 2:27pm UTC](https://discuss.elastic.co/t/elastic-search-goes-red/140037/6 "2018-07-25T14:27:52Z")
Broke again [2018-07-25T10:02:46,072][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node-1] failed to execute [search] input for watch [J8OVNfqLSe2NW0TOJPOhzw\_logstash\_version\_mismatch], reason [java.util.concurrent.TimeoutException: Timeout waiting for task.] [2018-07-25T10:02:46,072][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node-1] failed to execute [search] input for watch [J8OVNfqLSe2NW0TOJPOhzw\_kibana\_version\_mismatch], reason [java.util.concurrent.TimeoutException: Timeout waiting for task.] [2018-07-25T10:02:46,072][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node-1] failed to execute [search] input for watch [J8OVNfqLSe2NW0TOJPOhzw\_elasticsearch\_nodes], reason [java.util.concurrent.TimeoutException: Timeout waiting for task.] [2018-07-25T10:02:46,072][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node-1] failed to execute [search] input for watch [J8OVNfqLSe2NW0TOJPOhzw\_xpack\_license\_expiration], reason [java.util.concurrent.TimeoutException: Timeout waiting for task.] [2018-07-25T10:02:46,072][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node-1] failed to execute [search] input for watch [J8OVNfqLSe2NW0TOJPOhzw\_elasticsearch\_cluster\_status], reason [java.util.concurrent.TimeoutException: Timeout waiting for task.] [2018-07-25T10:02:46,072][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node-1] failed to execute [search] input for watch [J8OVNfqLSe2NW0TOJPOhzw\_elasticsearch\_version\_mismatch], reason [java.util.concurrent.TimeoutException: Timeout waiting for task.] [2018-07-25T10:02:46,072][ERROR][o.e.x.m.c.n.NodeStatsCollector] [node-1] collector [node\_stats] timed out when collecting data [2018-07-25T10:02:46,076][WARN][o.e.x.w.e.ExecutionService] [node-1] failed to execute watch [J8OVNfqLSe2NW0TOJPOhzw\_logstash\_version\_mismatch] [2018-07-25T10:02:46,086][WARN][o.e.m.j.JvmGcMonitorService] [node-1] [gc][old][1050][14] duration [26.5s], collections [1]/[27.1s], total [26.5s]/[4.8m], memory [15.8gb]-\>[15.8gb]/[15.8gb], all\_pools {[young] [865.3mb]-\>[865.3mb]/[865.3mb]}{[survivor] [51.3mb]-\>[57.2mb]/[108.1mb]}{[old] [14.9gb]-\>[14.9gb]/[14.9gb]} [2018-07-25T10:02:46,086][WARN][o.e.m.j.JvmGcMonitorService] [node-1] [gc][1050] overhead, spent [26.5s] collecting in the last [27.1s] [2018-07-25T10:03:09,939][ERROR][o.e.x.m.c.i.IndexStatsCollector] [node-1] collector [index-stats] timed out when collecting data [2018-07-25T10:03:09,946][WARN][o.e.m.j.JvmGcMonitorService] [node-1] [gc][old][1051][15] duration [23.5s], collections [1]/[23.8s], total [23.5s]/[5.2m], memory [15.8gb]-\>[15.8gb]/[15.8gb], all\_pools {[young] [865.3mb]-\>[865.3mb]/[865.3mb]}{[survivor] [57.2mb]-\>[74.2mb]/[108.1mb]}{[old] [14.9gb]-\>[14.9gb]/[14.9gb]} [2018-07-25T10:03:09,946][WARN][o.e.m.j.JvmGcMonitorService] [node-1] [gc][1051] overhead, spent [23.5s] collecting in the last [23.8s] [2018-07-25T10:03:36,701][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node-1] failed to execute [search] input for watch [J8OVNfqLSe2NW0TOJPOhzw\_kibana\_version\_mismatch], reason [java.util.concurrent.TimeoutException: Timeout waiting for task.] [2018-07-25T10:03:36,701][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node-1] failed to execute [search] input for watch [J8OVNfqLSe2NW0TOJPOhzw\_logstash\_version\_mismatch], reason [java.util.concurrent.TimeoutException: Timeout waiting for task.] [2018-07-25T10:03:36,701][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node-1] failed to execute [search] input for watch [J8OVNfqLSe2NW0TOJPOhzw\_xpack\_license\_expiration], reason [java.util.concurrent.TimeoutException: Timeout waiting for task.] [2018-07-25T10:03:36,702][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node-1] failed to execute [search] input for watch [J8OVNfqLSe2NW0TOJPOhzw\_elasticsearch\_nodes], reason [java.util.concurrent.TimeoutException: Timeout waiting for task.] [2018-07-25T10:03:36,702][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node-1] failed to execute [search] input for watch [J8OVNfqLSe2NW0TOJPOhzw\_elasticsearch\_version\_mismatch], reason [java.util.concurrent.TimeoutException: Timeout waiting for task.] [2018-07-25T10:03:36,703][ERROR][o.e.x.m.c.c.ClusterStatsCollector] [node-1] collector [cluster\_stats] timed out when collecting data [2018-07-25T10:03:36,702][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node-1] failed to execute [search] input for watch [J8OVNfqLSe2NW0TOJPOhzw\_elasticsearch\_cluster\_status], reason [java.util.concurrent.TimeoutException: Timeout waiting for task.] [2018-07-25T10:03:36,704][WARN][o.e.m.j.JvmGcMonitorService] [node-1] [gc][old][1052][16] duration [26.5s], collections [1]/[26.7s], total [26.5s]/[5.6m], memory [15.8gb]-\>[15.8gb]/[15.8gb], all\_pools {[young] [865.3mb]-\>[865.3mb]/[865.3mb]}{[survivor] [74.2mb]-\>[84.8mb]/[108.1mb]}{[old] [14.9gb]-\>[14.9gb]/[14.9gb]} [2018-07-25T10:03:36,705][WARN][o.e.m.j.JvmGcMonitorService] [node-1] [gc][1052] overhead, spent [26.5s] collecting in the last [26.7s] ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 22, 2018, 2:27pm UTC](https://discuss.elastic.co/t/elastic-search-goes-red/140037/7 "2018-08-22T14:27:53Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.