Status: red; Crashes with java.nio.channels.ClosedChannelException

Elastic Stack Elasticsearch
1

Hi list,

I've been running ES for about a week (I'm a complete noob, I should add),
but recently it started behaving weirdly. I start elasticsearch, indices
are recovered, then it hangs for a minute or so, refusing connections, and
finally crashes with

Caught exception while handling client http traffic, closing connection
[id: 0xd6f24dd4, /10.114.30.31:58582 :> /10.204.95.154:9200]
[full stack trace] https://gist.github.com/andreasjansson/5245533

While it's hanging, this is the cluster health:

$ curl -XGET 'http://127.0.0.1:9200/_cluster/health?pretty=true'
{
"cluster_name" : "elasticsearch",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 49,
"active_shards" : 49,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 51
}

I use ES to store logs, using a home-built tool that sends data to ES in
logstash format, using pyes. I run it on a medium EC2 instance, with the
actual data on a mounted EBS volume. The OS is 64 bit Ubuntu 12.04, OpenJDK
Java 1.6.0_24, Elasticsearch version 0.90.0.RC1 (although the same error
occurred on 0.20.6).

The only non-default config options in elasticsearch.yml are

index.number_of_shards: 1
index.number_of_replicas: 0
gateway.expected_nodes: 1

$ ulimit -Hn
16384

Full DEBUG log here https://gist.github.com/andreasjansson/5245603

Thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

It does not look like a crash. Start ES and wait for status green. This
can take some minutes if many indices must be recovered.

After that, you can connect logstash/kibana etc to ES. It seems you
started the clients too early and the clients ran into a timeout (60s),
they closed the HTTP connection, that's all. Look if there is something
in logstash/kibana logs.

You may also try to speed up recovery with node_concurrent_recoveries

Jörg

Am 26.03.13 15:12, schrieb Andreas Jansson:

Hi list,

I've been running ES for about a week (I'm a complete noob, I should
add), but recently it started behaving weirdly. I start elasticsearch,
indices are recovered, then it hangs for a minute or so, refusing
connections, and finally crashes with

Caught exception while handling client http traffic, closing
connection [id: 0xd6f24dd4, /10.114.30.31:58582 :> /10.204.95.154:9200]
[full stack trace] https://gist.github.com/andreasjansson/5245533

While it's hanging, this is the cluster health:

$ curl -XGET 'http://127.0.0.1:9200/_cluster/health?pretty=true'
{
"cluster_name" : "elasticsearch",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 49,
"active_shards" : 49,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 51
}

I use ES to store logs, using a home-built tool that sends data to ES
in logstash format, using pyes. I run it on a medium EC2 instance,
with the actual data on a mounted EBS volume. The OS is 64 bit Ubuntu
12.04, OpenJDK Java 1.6.0_24, Elasticsearch version 0.90.0.RC1
(although the same error occurred on 0.20.6).

The only non-default config options in elasticsearch.yml are

index.number_of_shards: 1
index.number_of_replicas: 0
gateway.expected_nodes: 1

$ ulimit -Hn
16384

Full DEBUG log here https://gist.github.com/andreasjansson/5245603

Thanks!

You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Thanks Jörg, you were right, the exception was a red herring.

I've waited for a good 20 minutes now, and status is still red. When I
manually post to Elasticsearch, around one in ten requests hang for a
minute, then fail. Curl output:

$ curl -v -XPOST
'http://10.204.95.154:9200/logstash-2013.03.26/preprod-web-access?pretty=true'
-d '{ [...] }'

  • About to connect() to 10.204.95.154 port 9200 (#0)
  • Trying 10.204.95.154...
  • connected
  • Connected to 10.204.95.154 (10.204.95.154) port 9200 (#0)

POST /logstash-2013.03.26/preprod-web-access?pretty=true HTTP/1.1
User-Agent: curl/7.24.0 (x86_64-redhat-linux-gnu) libcurl/7.24.0
NSS/3.12.10.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
Host: 10.204.95.154:9200
Accept: /
Content-Length: 1487
Content-Type: application/x-www-form-urlencoded
Expect: 100-continue

< HTTP/1.1 100 Continue
< HTTP/1.1 503 Service Unavailable
< Content-Type: application/json; charset=UTF-8
< Content-Length: 1866

  • HTTP error before end of send, stop sending
    <
    {
    "error" : "UnavailableShardsException[[logstash-2013.03.26][4] [2]
    shardIt, [0] active : Timeout waiting for [1m], request: index
    {[logstash-2013.03.26][preprod-web-access][t-qj5GqnQn6p1JYz6V0nPw],
    source[{ [...] }]",
    "status" : 503
  • Closing connection #0

On Tuesday, 26 March 2013 14:45:00 UTC, Jörg Prante wrote:

It does not look like a crash. Start ES and wait for status green. This
can take some minutes if many indices must be recovered.

After that, you can connect logstash/kibana etc to ES. It seems you
started the clients too early and the clients ran into a timeout (60s),
they closed the HTTP connection, that's all. Look if there is something
in logstash/kibana logs.

You may also try to speed up recovery with node_concurrent_recoveries
Elasticsearch Platform — Find real-time answers at scale | Elastic

Jörg

Am 26.03.13 15:12, schrieb Andreas Jansson:

Hi list,

I've been running ES for about a week (I'm a complete noob, I should
add), but recently it started behaving weirdly. I start elasticsearch,
indices are recovered, then it hangs for a minute or so, refusing
connections, and finally crashes with

Caught exception while handling client http traffic, closing
connection [id: 0xd6f24dd4, /10.114.30.31:58582 :> /10.204.95.154:9200]
[full stack trace] https://gist.github.com/andreasjansson/5245533

While it's hanging, this is the cluster health:

$ curl -XGET 'http://127.0.0.1:9200/_cluster/health?pretty=true'
{
"cluster_name" : "elasticsearch",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 49,
"active_shards" : 49,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 51
}

I use ES to store logs, using a home-built tool that sends data to ES
in logstash format, using pyes. I run it on a medium EC2 instance,
with the actual data on a mounted EBS volume. The OS is 64 bit Ubuntu
12.04, OpenJDK Java 1.6.0_24, Elasticsearch version 0.90.0.RC1
(although the same error occurred on 0.20.6).

The only non-default config options in elasticsearch.yml are

index.number_of_shards: 1
index.number_of_replicas: 0
gateway.expected_nodes: 1

$ ulimit -Hn
16384

Full DEBUG log here https://gist.github.com/andreasjansson/5245603

Thanks!

You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Hi Andreas, Jorg,

I think i've been seeing this (or something similar) in my testing:
https://groups.google.com/forum/?fromgroups=#!topic/elasticsearch/o5zlfTAuAqI[1-25-false]

I create 3 indexes based on mappings and can't do anything with them.

Just thought it was worth mentioning here as well..

Derry

On Tuesday, 26 March 2013 15:34:44 UTC, Andreas Jansson wrote:

Thanks Jörg, you were right, the exception was a red herring.

I've waited for a good 20 minutes now, and status is still red. When I
manually post to Elasticsearch, around one in ten requests hang for a
minute, then fail. Curl output:

$ curl -v -XPOST '
http://10.204.95.154:9200/logstash-2013.03.26/preprod-web-access?pretty=true'
-d '{ [...] }'

  • About to connect() to 10.204.95.154 port 9200 (#0)
  • Trying 10.204.95.154...
  • connected
  • Connected to 10.204.95.154 (10.204.95.154) port 9200 (#0)

POST /logstash-2013.03.26/preprod-web-access?pretty=true HTTP/1.1
User-Agent: curl/7.24.0 (x86_64-redhat-linux-gnu) libcurl/7.24.0 NSS/
3.12.10.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
Host: 10.204.95.154:9200
Accept: /
Content-Length: 1487
Content-Type: application/x-www-form-urlencoded
Expect: 100-continue

< HTTP/1.1 100 Continue
< HTTP/1.1 503 Service Unavailable
< Content-Type: application/json; charset=UTF-8
< Content-Length: 1866

  • HTTP error before end of send, stop sending
    <
    {
    "error" : "UnavailableShardsException[[logstash-2013.03.26][4] [2]
    shardIt, [0] active : Timeout waiting for [1m], request: index
    {[logstash-2013.03.26][preprod-web-access][t-qj5GqnQn6p1JYz6V0nPw],
    source[{ [...] }]",
    "status" : 503
  • Closing connection #0

On Tuesday, 26 March 2013 14:45:00 UTC, Jörg Prante wrote:

It does not look like a crash. Start ES and wait for status green. This
can take some minutes if many indices must be recovered.

After that, you can connect logstash/kibana etc to ES. It seems you
started the clients too early and the clients ran into a timeout (60s),
they closed the HTTP connection, that's all. Look if there is something
in logstash/kibana logs.

You may also try to speed up recovery with node_concurrent_recoveries
Elasticsearch Platform — Find real-time answers at scale | Elastic

Jörg

Am 26.03.13 15:12, schrieb Andreas Jansson:

Hi list,

I've been running ES for about a week (I'm a complete noob, I should
add), but recently it started behaving weirdly. I start elasticsearch,
indices are recovered, then it hangs for a minute or so, refusing
connections, and finally crashes with

Caught exception while handling client http traffic, closing
connection [id: 0xd6f24dd4, /10.114.30.31:58582 :> /10.204.95.154:9200]

[full stack trace] https://gist.github.com/andreasjansson/5245533

While it's hanging, this is the cluster health:

$ curl -XGET 'http://127.0.0.1:9200/_cluster/health?pretty=true'
{
"cluster_name" : "elasticsearch",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 49,
"active_shards" : 49,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 51
}

I use ES to store logs, using a home-built tool that sends data to ES
in logstash format, using pyes. I run it on a medium EC2 instance,
with the actual data on a mounted EBS volume. The OS is 64 bit Ubuntu
12.04, OpenJDK Java 1.6.0_24, Elasticsearch version 0.90.0.RC1
(although the same error occurred on 0.20.6).

The only non-default config options in elasticsearch.yml are

index.number_of_shards: 1
index.number_of_replicas: 0
gateway.expected_nodes: 1

$ ulimit -Hn
16384

Full DEBUG log here https://gist.github.com/andreasjansson/5245603

Thanks!

You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.