# ES Stops Works Randomly **URL:** \ **Category:** Elasticsearch\ **Created:** [March 7, 2014, 3:12pm UTC](https://discuss.elastic.co/t/es-stops-works-randomly/16231 "2014-03-07T15:12:54Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![Eric\_Luellen](https://avatars.discourse-cdn.com/v4/letter/e/ce73a5/32.png) [@Eric\_Luellen](https://discuss.elastic.co/u/Eric_Luellen)\ **Post date:** [March 7, 2014, 3:12pm UTC](https://discuss.elastic.co/t/es-stops-works-randomly/16231/1 "2014-03-07T15:12:54Z")
Hello, I have 2 ES servers that are being fed by 1 logstash server and viewing the logs in Kibana. This is a POC to work out any issues before going into production. The system has ran for ~1 month and every few days, Kibana will stop showing logs at some random time in the middle of the night. Last night, the last log entry I received in Kibana was around 18:30. When I checked on the ES servers, it showed the master running and the secondary not running (from /sbin/service elasticsearch status), but I was able to do a curl on the localhost and it returned information. So not sure what's up with that. Anyway, when I do a status on the master node, I get this: curl -XGET '[http://localhost:9200/\_cluster/health?pretty=true](http://localhost:9200/_cluster/health?pretty=true)' { "cluster\_name" : "gis-elasticsearch", "status" : "red", "timed\_out" : false, "number\_of\_nodes" : 6, "number\_of\_data\_nodes" : 2, "active\_primary\_shards" : 186, "active\_shards" : 194, "relocating\_shards" : 0, "initializing\_shards" : 7, "unassigned\_shards" : 249 } When I view the indexes, via "ls ...nodes/0/indeces/" it shows all indexes being modified today for some reason and there are new file for today's date.So I think I'm starting to catch back up after I restarted both servers but not sure why it failed in the first place. When I look at the logs on the master, I only see 4 warning errors at 18:57 and then the 2ndary leaving the cluster. I don't see any logs on the secondary (Pistol) on why it stopped working or what truly happened. [2014-03-06 18:57:04,121][WARN][transport] [ElasticSearch Server1] Transport response handler not found of id [64147630] [2014-03-06 18:57:04,124][WARN][transport] [ElasticSearch Server1] Transport response handler not found of id [64147717] [2014-03-06 18:57:04,124][WARN][transport] [ElasticSearch Server1] Transport response handler not found of id [64147718] [2014-03-06 18:57:04,124][WARN][transport] [ElasticSearch Server1] Transport response handler not found of id [64147721] [2014-03-06 19:56:08,467][INFO][cluster.service] [ElasticSearch Server1] removed {[Pistol][sIAMHNj6TMCmrMJGW7u97A][inet[/10.1.1.10:9301]]{client=true, data=false},}, reason: zen-disco-node\_failed([Pistol][sIAMHNj6TMCmrMJGW7u97A][inet[/10.13.3.46:9301]]{client=true, data=false}), reason failed to ping, tried [3] times, each with maximum [30s] timeout [2014-03-06 19:56:12,304][INFO][cluster.service] [ElasticSearch Server1] added {[Pistol][sIAMHNj6TMCmrMJGW7u97A][inet[/10.1.1.10:9301]]{client=true, data=false},}, reason: zen-disco-receive(join from node[[Pistol][sIAMHNj6TMCmrMJGW7u97A][inet[/10.13.3.46:9301]]{client=true, data=false}]) Any idea on additional logging or troubleshooting I can turn on to keep this from happening in the future? Since the shards are not caught up, right now I"m just seeing a lot o debug messages about failed to parse. I'm assuming that will be corrected once we catch up. [2014-03-07 10:06:52,235][DEBUG][action.search.type] [ElasticSearch Server1] All shards failed for phase: [query] [2014-03-07 10:06:52,223][DEBUG][action.search.type] [ElasticSearch Server1] [windows-2014.03.07][3], node[W6aEFbimR5G712ddG\_G5yQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@74ecbbc6] lastShard [true] org.elasticsearch.search.SearchParseException: [windows-2014.03.07][3]: from[-1],size[-1]: Parse Failure [Failed to parse source [{"facets":{"0":{"date\_histogram":{"field":"@timestamp","interval":"10m"},"global":true,"facet\_filter":{"fquery":{"query":{"filtered":{"query":{"query\_string":{"query":"(ASA AND Deny)"}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1394118412373,"to":"now"}}}]}}}}}}}},"size":0}]] -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/0a693898-16e3-449f-9bf5-6adc97251e09%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0a693898-16e3-449f-9bf5-6adc97251e09%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:44am UTC](https://discuss.elastic.co/t/es-stops-works-randomly/16231/2 "2017-07-06T01:44:54Z")
Closed on July 6, 2017, 1:44am UTC