# Status: red; Crashes with java.nio.channels.ClosedChannelException **URL:** \ **Category:** Elasticsearch\ **Created:** [March 26, 2013, 2:12pm UTC](https://discuss.elastic.co/t/status-red-crashes-with-java-nio-channels-closedchannelexception/11318 "2013-03-26T14:12:37Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Andreas\_Jansson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andreas_jansson/32/2425_2.png) [@Andreas\_Jansson](https://discuss.elastic.co/u/Andreas_Jansson)\ **Post date:** [March 26, 2013, 2:12pm UTC](https://discuss.elastic.co/t/status-red-crashes-with-java-nio-channels-closedchannelexception/11318/1 "2013-03-26T14:12:37Z")
Hi list, I've been running ES for about a week (I'm a complete noob, I should add), but recently it started behaving weirdly. I start elasticsearch, indices are recovered, then it hangs for a minute or so, refusing connections, and finally crashes with Caught exception while handling client http traffic, closing connection [id: 0xd6f24dd4, /10.114.30.31:58582 :\> /10.204.95.154:9200] [full stack trace] [https://gist.github.com/andreasjansson/5245533](https://gist.github.com/andreasjansson/5245533) While it's hanging, this is the cluster health: $ curl -XGET '[http://127.0.0.1:9200/\_cluster/health?pretty=true](http://127.0.0.1:9200/_cluster/health?pretty=true)' { "cluster\_name" : "elasticsearch", "status" : "red", "timed\_out" : false, "number\_of\_nodes" : 1, "number\_of\_data\_nodes" : 1, "active\_primary\_shards" : 49, "active\_shards" : 49, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 51 } I use ES to store logs, using a home-built tool that sends data to ES in logstash format, using pyes. I run it on a medium EC2 instance, with the actual data on a mounted EBS volume. The OS is 64 bit Ubuntu 12.04, OpenJDK Java 1.6.0\_24, Elasticsearch version 0.90.0.RC1 (although the same error occurred on 0.20.6). The only non-default config options in elasticsearch.yml are index.number\_of\_shards: 1 index.number\_of\_replicas: 0 gateway.expected\_nodes: 1 $ ulimit -Hn 16384 Full DEBUG log here [https://gist.github.com/andreasjansson/5245603](https://gist.github.com/andreasjansson/5245603) Thanks! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [March 26, 2013, 2:45pm UTC](https://discuss.elastic.co/t/status-red-crashes-with-java-nio-channels-closedchannelexception/11318/2 "2013-03-26T14:45:00Z")
It does not look like a crash. Start ES and wait for status green. This can take some minutes if many indices must be recovered. After that, you can connect logstash/kibana etc to ES. It seems you started the clients too early and the clients ran into a timeout (60s), they closed the HTTP connection, that's all. Look if there is something in logstash/kibana logs. You may also try to speed up recovery with node\_concurrent\_recoveries > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. Jörg Am 26.03.13 15:12, schrieb Andreas Jansson: > Hi list, > > I've been running ES for about a week (I'm a complete noob, I should > add), but recently it started behaving weirdly. I start elasticsearch, > indices are recovered, then it hangs for a minute or so, refusing > connections, and finally crashes with > > Caught exception while handling client http traffic, closing > connection [id: 0xd6f24dd4, /10.114.30.31:58582 :\> /10.204.95.154:9200] > [full stack trace] [https://gist.github.com/andreasjansson/5245533](https://gist.github.com/andreasjansson/5245533) > > While it's hanging, this is the cluster health: > > $ curl -XGET '[http://127.0.0.1:9200/\_cluster/health?pretty=true](http://127.0.0.1:9200/_cluster/health?pretty=true)' > { > "cluster\_name" : "elasticsearch", > "status" : "red", > "timed\_out" : false, > "number\_of\_nodes" : 1, > "number\_of\_data\_nodes" : 1, > "active\_primary\_shards" : 49, > "active\_shards" : 49, > "relocating\_shards" : 0, > "initializing\_shards" : 0, > "unassigned\_shards" : 51 > } > > I use ES to store logs, using a home-built tool that sends data to ES > in logstash format, using pyes. I run it on a medium EC2 instance, > with the actual data on a mounted EBS volume. The OS is 64 bit Ubuntu > 12.04, OpenJDK Java 1.6.0\_24, Elasticsearch version 0.90.0.RC1 > (although the same error occurred on 0.20.6). > > The only non-default config options in elasticsearch.yml are > > index.number\_of\_shards: 1 > index.number\_of\_replicas: 0 > gateway.expected\_nodes: 1 > > $ ulimit -Hn > 16384 > > Full DEBUG log here [https://gist.github.com/andreasjansson/5245603](https://gist.github.com/andreasjansson/5245603) > > ## Thanks! > > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Andreas\_Jansson](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andreas_jansson/32/2425_2.png) [@Andreas\_Jansson](https://discuss.elastic.co/u/Andreas_Jansson)\ **Post date:** [March 26, 2013, 3:34pm UTC](https://discuss.elastic.co/t/status-red-crashes-with-java-nio-channels-closedchannelexception/11318/3 "2013-03-26T15:34:44Z")
Thanks Jörg, you were right, the exception was a red herring. I've waited for a good 20 minutes now, and status is still red. When I manually post to Elasticsearch, around one in ten requests hang for a minute, then fail. Curl output: $ curl -v -XPOST '[http://10.204.95.154:9200/logstash-2013.03.26/preprod-web-access?pretty=true](http://10.204.95.154:9200/logstash-2013.03.26/preprod-web-access?pretty=true)' -d '{ [...] }' - About to connect() to 10.204.95.154 port 9200 (#0) - Trying 10.204.95.154... - connected - Connected to 10.204.95.154 (10.204.95.154) port 9200 (#0) > POST /logstash-2013.03.26/preprod-web-access?pretty=true HTTP/1.1 > User-Agent: curl/7.24.0 (x86\_64-redhat-linux-gnu) libcurl/7.24.0 > NSS/3.12.10.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2 > Host: 10.204.95.154:9200 > Accept: _/_ > Content-Length: 1487 > Content-Type: application/x-www-form-urlencoded > Expect: 100-continue \< HTTP/1.1 100 Continue \< HTTP/1.1 503 Service Unavailable \< Content-Type: application/json; charset=UTF-8 \< Content-Length: 1866 - HTTP error before end of send, stop sending \< { "error" : "UnavailableShardsException[[logstash-2013.03.26][4] [2] shardIt, [0] active : Timeout waiting for [1m], request: index {[logstash-2013.03.26][preprod-web-access][t-qj5GqnQn6p1JYz6V0nPw], source[{ [...] }]", "status" : 503 - Closing connection #0 On Tuesday, 26 March 2013 14:45:00 UTC, Jörg Prante wrote: > It does not look like a crash. Start ES and wait for status green. This > can take some minutes if many indices must be recovered. > > After that, you can connect logstash/kibana etc to ES. It seems you > started the clients too early and the clients ran into a timeout (60s), > they closed the HTTP connection, that's all. Look if there is something > in logstash/kibana logs. > > You may also try to speed up recovery with node\_concurrent\_recoveries > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/cluster.html) > > Jörg > > Am 26.03.13 15:12, schrieb Andreas Jansson: > > > Hi list, > > > > I've been running ES for about a week (I'm a complete noob, I should > > add), but recently it started behaving weirdly. I start elasticsearch, > > indices are recovered, then it hangs for a minute or so, refusing > > connections, and finally crashes with > > > > Caught exception while handling client http traffic, closing > > connection [id: 0xd6f24dd4, /10.114.30.31:58582 :\> /10.204.95.154:9200] > > [full stack trace] [https://gist.github.com/andreasjansson/5245533](https://gist.github.com/andreasjansson/5245533) > > > > While it's hanging, this is the cluster health: > > > > $ curl -XGET '[http://127.0.0.1:9200/\_cluster/health?pretty=true](http://127.0.0.1:9200/_cluster/health?pretty=true)' > > { > > "cluster\_name" : "elasticsearch", > > "status" : "red", > > "timed\_out" : false, > > "number\_of\_nodes" : 1, > > "number\_of\_data\_nodes" : 1, > > "active\_primary\_shards" : 49, > > "active\_shards" : 49, > > "relocating\_shards" : 0, > > "initializing\_shards" : 0, > > "unassigned\_shards" : 51 > > } > > > > I use ES to store logs, using a home-built tool that sends data to ES > > in logstash format, using pyes. I run it on a medium EC2 instance, > > with the actual data on a mounted EBS volume. The OS is 64 bit Ubuntu > > 12.04, OpenJDK Java 1.6.0\_24, Elasticsearch version 0.90.0.RC1 > > (although the same error occurred on 0.20.6). > > > > The only non-default config options in elasticsearch.yml are > > > > index.number\_of\_shards: 1 > > index.number\_of\_replicas: 0 > > gateway.expected\_nodes: 1 > > > > $ ulimit -Hn > > 16384 > > > > Full DEBUG log here [https://gist.github.com/andreasjansson/5245603](https://gist.github.com/andreasjansson/5245603) > > > > ## Thanks! > > > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Derry\_O\_Sullivan](https://avatars.discourse-cdn.com/v4/letter/d/7bcc69/32.png) [@Derry\_O\_Sullivan](https://discuss.elastic.co/u/Derry_O_Sullivan)\ **Post date:** [March 27, 2013, 12:23pm UTC](https://discuss.elastic.co/t/status-red-crashes-with-java-nio-channels-closedchannelexception/11318/4 "2013-03-27T12:23:15Z")
Hi Andreas, Jorg, I think i've been seeing this (or something similar) in my testing: [https://groups.google.com/forum/?fromgroups=#!topic/elasticsearch/o5zlfTAuAqI[1-25-false]](https://groups.google.com/forum/?fromgroups=#!topic/elasticsearch/o5zlfTAuAqI%5B1-25-false%5D) I create 3 indexes based on mappings and can't do anything with them. Just thought it was worth mentioning here as well.. Derry On Tuesday, 26 March 2013 15:34:44 UTC, Andreas Jansson wrote: > Thanks Jörg, you were right, the exception was a red herring. > > I've waited for a good 20 minutes now, and status is still red. When I > manually post to Elasticsearch, around one in ten requests hang for a > minute, then fail. Curl output: > > $ curl -v -XPOST ' > [http://10.204.95.154:9200/logstash-2013.03.26/preprod-web-access?pretty=true](http://10.204.95.154:9200/logstash-2013.03.26/preprod-web-access?pretty=true)' > -d '{ [...] }' > > - About to connect() to 10.204.95.154 port 9200 (#0) > - Trying 10.204.95.154... > - connected > - Connected to 10.204.95.154 (10.204.95.154) port 9200 (#0) > > > POST /logstash-2013.03.26/preprod-web-access?pretty=true HTTP/1.1 > > User-Agent: curl/7.24.0 (x86\_64-redhat-linux-gnu) libcurl/7.24.0 NSS/ > > 3.12.10.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2 > > Host: 10.204.95.154:9200 > > Accept: _/_ > > Content-Length: 1487 > > Content-Type: application/x-www-form-urlencoded > > Expect: 100-continue > > \< HTTP/1.1 100 Continue > \< HTTP/1.1 503 Service Unavailable > \< Content-Type: application/json; charset=UTF-8 > \< Content-Length: 1866 > > - HTTP error before end of send, stop sending > \< > { > "error" : "UnavailableShardsException[[logstash-2013.03.26][4] [2] > shardIt, [0] active : Timeout waiting for [1m], request: index > {[logstash-2013.03.26][preprod-web-access][t-qj5GqnQn6p1JYz6V0nPw], > source[{ [...] }]", > "status" : 503 > - Closing connection #0 > > On Tuesday, 26 March 2013 14:45:00 UTC, Jörg Prante wrote: > > > It does not look like a crash. Start ES and wait for status green. This > > can take some minutes if many indices must be recovered. > > > > After that, you can connect logstash/kibana etc to ES. It seems you > > started the clients too early and the clients ran into a timeout (60s), > > they closed the HTTP connection, that's all. Look if there is something > > in logstash/kibana logs. > > > > You may also try to speed up recovery with node\_concurrent\_recoveries > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/cluster.html) > > > > Jörg > > > > Am 26.03.13 15:12, schrieb Andreas Jansson: > > > > > Hi list, > > > > > > I've been running ES for about a week (I'm a complete noob, I should > > > add), but recently it started behaving weirdly. I start elasticsearch, > > > indices are recovered, then it hangs for a minute or so, refusing > > > connections, and finally crashes with > > > > > > Caught exception while handling client http traffic, closing > > > connection [id: 0xd6f24dd4, /10.114.30.31:58582 :\> /10.204.95.154:9200] > > > > > [full stack trace] [https://gist.github.com/andreasjansson/5245533](https://gist.github.com/andreasjansson/5245533) > > > > > > While it's hanging, this is the cluster health: > > > > > > $ curl -XGET '[http://127.0.0.1:9200/\_cluster/health?pretty=true](http://127.0.0.1:9200/_cluster/health?pretty=true)' > > > { > > > "cluster\_name" : "elasticsearch", > > > "status" : "red", > > > "timed\_out" : false, > > > "number\_of\_nodes" : 1, > > > "number\_of\_data\_nodes" : 1, > > > "active\_primary\_shards" : 49, > > > "active\_shards" : 49, > > > "relocating\_shards" : 0, > > > "initializing\_shards" : 0, > > > "unassigned\_shards" : 51 > > > } > > > > > > I use ES to store logs, using a home-built tool that sends data to ES > > > in logstash format, using pyes. I run it on a medium EC2 instance, > > > with the actual data on a mounted EBS volume. The OS is 64 bit Ubuntu > > > 12.04, OpenJDK Java 1.6.0\_24, Elasticsearch version 0.90.0.RC1 > > > (although the same error occurred on 0.20.6). > > > > > > The only non-default config options in elasticsearch.yml are > > > > > > index.number\_of\_shards: 1 > > > index.number\_of\_replicas: 0 > > > gateway.expected\_nodes: 1 > > > > > > $ ulimit -Hn > > > 16384 > > > > > > Full DEBUG log here [https://gist.github.com/andreasjansson/5245603](https://gist.github.com/andreasjansson/5245603) > > > > > > ## Thanks! > > > > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:44am UTC](https://discuss.elastic.co/t/status-red-crashes-with-java-nio-channels-closedchannelexception/11318/5 "2017-07-06T02:44:13Z")