Getting Timeout Exceptions with Elasticsearch

shriyansh_jain · November 12, 2014, 7:49pm

Hi all,

I am getting timeout exceptions with elastic search while deleting an index
and while looking for logs in kibana its taking lot of time to load logs.
Following are the log details and setup info.

API Call for deleting an Index:

curl -XDELETE 'http://localhost:9200/logstash-2014.08.21'
{"error":"ProcessClusterEventTimeoutException[failed to process cluster
event (delete-index [logstash-2014.08.21]) within 30s]","status":503}

Logs from master node: (setup is cluster of 2 nodes)

http://pastebin.com/hUZEf26d

Logs from other node

http://pastebin.com/HxWppiYL

curl -XGET 'localhost:9200'
{
"status" : 200,
"name" : "Node1",
"version" : {
"number" : "1.2.2",
"build_hash" : "9902f08efc3ad14ce27882b991c4c56b920c9872",
"build_timestamp" : "2014-07-09T12:02:32Z",
"build_snapshot" : false,
"lucene_version" : "4.8"
},
"tagline" : "You Know, for Search"
}

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "xxxxx",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 270,
"active_shards" : 540,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

Java version on the system

java version "1.7.0_09-icedtea"
OpenJDK Runtime Environment (rhel-2.3.4.1.el6_3-x86_64)
OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)

I am using HQ plugin to visualize the elasticsearch plugin, but that plugin
is taking lot of time to load the cluster info.

what might be causing the timeouts to happen. Any inputs will be greatly
appreciated.
please let me know if any other details are required.

Thanks!
Shriyansh

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/89298f22-ec17-49b2-a7fe-c6f110341146%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

warkolm · November 12, 2014, 7:58pm

How much data is in the cluster? How big are your nodes?

You're running a very old version of Java, recommended is 1.7u55 or
greater, and Oracle not OpenJDK.

On 13 November 2014 06:49, shriyansh jain shriyanshajain@gmail.com wrote:

Hi all,

I am getting timeout exceptions with Elasticsearch while deleting an
index and while looking for logs in kibana its taking lot of time to load
logs.
Following are the log details and setup info.

API Call for deleting an Index:

curl -XDELETE 'http://localhost:9200/logstash-2014.08.21'
{"error":"ProcessClusterEventTimeoutException[failed to process cluster
event (delete-index [logstash-2014.08.21]) within 30s]","status":503}

Logs from master node: (setup is cluster of 2 nodes)

[2014-11-12 11:31:20,699][WARN ][monitor.jvm ] [Node1] [gc][old][13 - Pastebin.com

Logs from other node

[2014-11-12 11:08:13,112][WARN ][transport ] [Node2] Received res - Pastebin.com

curl -XGET 'localhost:9200'
{
"status" : 200,
"name" : "Node1",
"version" : {
"number" : "1.2.2",
"build_hash" : "9902f08efc3ad14ce27882b991c4c56b920c9872",
"build_timestamp" : "2014-07-09T12:02:32Z",
"build_snapshot" : false,
"lucene_version" : "4.8"
},
"tagline" : "You Know, for Search"
}

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "xxxxx",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 270,
"active_shards" : 540,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}

Java version on the system

java version "1.7.0_09-icedtea"
OpenJDK Runtime Environment (rhel-2.3.4.1.el6_3-x86_64)
OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)

I am using HQ plugin to visualize the elasticsearch plugin, but that
plugin is taking lot of time to load the cluster info.

what might be causing the timeouts to happen. Any inputs will be greatly
appreciated.
please let me know if any other details are required.

Thanks!
Shriyansh

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/89298f22-ec17-49b2-a7fe-c6f110341146%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/89298f22-ec17-49b2-a7fe-c6f110341146%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZ%3DNiOEhZ3-d%3DGJoxY1JEi%3DLyF_HxiLuCrGDip3rykENqg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

shriyansh_jain · November 12, 2014, 8:08pm

Hi Mark,

The data is around id 150GB.
The master node holds : 16 GB RAM with ES_HEAP_SIZE = 8GB
The other node holds : 16 GB RAM with ES_HEAP_SIZE = 8 GB

I will get the version updated with Oracle Java, will that solve the issue.?

Thanks!
Shriyansh

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8fc2f714-2c1a-43d4-a67c-2b6869007712%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

warkolm · November 12, 2014, 8:16pm

It won't solve it but it may help.

You have some pretty large GC's happening, some getting close to the 30s
default ping timeout.
You can try increasing discovery.zen.fd.ping_timeout but you may want to
consider adding another node in to generally help relieve heap pressure and
improve performance, 3 nodes is much better than 2!

On 13 November 2014 07:08, shriyansh jain shriyanshajain@gmail.com wrote:

Hi Mark,

The data is around id 150GB.
The master node holds : 16 GB RAM with ES_HEAP_SIZE = 8GB
The other node holds : 16 GB RAM with ES_HEAP_SIZE = 8 GB

I will get the version updated with Oracle Java, will that solve the
issue.?

Thanks!
Shriyansh

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8fc2f714-2c1a-43d4-a67c-2b6869007712%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8fc2f714-2c1a-43d4-a67c-2b6869007712%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZn0ZmJjkR_K5JtqPvU7bPbwB4DdRuf_4ugSxAN%2BN2Adqg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

shriyansh_jain · November 12, 2014, 8:59pm

I have stopped collecting all the data from logstash( as I am using ELK
stack), but it still I am getting pretty large GC's (as you mentioned). Can
you please point me out what might be the reason for that.?

Thank you for the suggestion and pointing me out the error, I appreciate it
and will consider increasing the number of nodes of the cluster.

Thanks!
Shriyansh

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8e005f9a-bb48-4a50-9c94-2c14861dad7d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

shriyansh_jain · November 12, 2014, 9:11pm

I am also getting Out of Memory exception now in the logs

[2014-11-12 13:08:23,157][WARN ][netty.channel.DefaultChannelPipeline] An
exception was thrown by a user handler while handling an exception event
([id: 0x40f3788c, /10.18.144.231:52924 => /10.18.144.229:9300] EXCEPTION:
java.lang.OutOfMemoryError: Java heap space)
java.lang.OutOfMemoryError: Java heap space

what will be way out to get out of this situation.?

Thanks!
Shriyansh

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/de51ffc7-7d40-4ced-b37c-84ee68ca7d7c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

warkolm · November 12, 2014, 9:16pm

You either need to add more nodes, reduce your dataset, increase the heap
or close some old indexes.

On 13 November 2014 08:11, shriyansh jain shriyanshajain@gmail.com wrote:

I am also getting Out of Memory exception now in the logs

[2014-11-12 13:08:23,157][WARN ][netty.channel.DefaultChannelPipeline] An
exception was thrown by a user handler while handling an exception event
([id: 0x40f3788c, /10.18.144.231:52924 => /10.18.144.229:9300] EXCEPTION:
java.lang.OutOfMemoryError: Java heap space)
java.lang.OutOfMemoryError: Java heap space

what will be way out to get out of this situation.?

Thanks!
Shriyansh

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/de51ffc7-7d40-4ced-b37c-84ee68ca7d7c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/de51ffc7-7d40-4ced-b37c-84ee68ca7d7c%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZ%3DaT2YoyTgDaL7nTgoqraGuKG9KGPCP3gqHcVFHDMJbiA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

shriyansh_jain · November 12, 2014, 9:26pm

Thank you mark for pointing me out with the situation. As I am trying to
delete the index I am getting timeouts for that. But you suggested me to
increase discovery.zen.fd.ping_timeout to get rid of that. I am not able to
see any such variable in elasticsearch.yml file.
Can you please point me how to increase discovery.zen.fd.ping_timeout.

Thanks!
Shriyansh

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/db816351-88d2-4e18-9863-3e719f2af1ed%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

warkolm · November 12, 2014, 9:30pm

You can add that in yourself.

On 13 November 2014 08:26, shriyansh jain shriyanshajain@gmail.com wrote:

Thank you mark for pointing me out with the situation. As I am trying to
delete the index I am getting timeouts for that. But you suggested me to
increase discovery.zen.fd.ping_timeout to get rid of that. I am not able to
see any such variable in elasticsearch.yml file.
Can you please point me how to increase discovery.zen.fd.ping_timeout.

Thanks!
Shriyansh

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/db816351-88d2-4e18-9863-3e719f2af1ed%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/db816351-88d2-4e18-9863-3e719f2af1ed%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZkspduJTuezUj_%3DhojkhPcO0CjsWVmYxo4HVWsoFR1JTQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

shriyansh_jain · November 12, 2014, 9:33pm

Thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b9bf42ad-f55f-4091-99c0-e34125eccaf0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

shriyansh_jain · November 12, 2014, 9:38pm

will it need a restart to being that info effect.?

Thank!
Shriyansh

On Wednesday, November 12, 2014 1:30:57 PM UTC-8, Mark Walkom wrote:

You can add that in yourself.

On 13 November 2014 08:26, shriyansh jain <shriyan...@gmail.com
<javascript:>> wrote:

Thank you mark for pointing me out with the situation. As I am trying to
delete the index I am getting timeouts for that. But you suggested me to
increase discovery.zen.fd.ping_timeout to get rid of that. I am not able to
see any such variable in elasticsearch.yml file.
Can you please point me how to increase discovery.zen.fd.ping_timeout.

Thanks!
Shriyansh

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/db816351-88d2-4e18-9863-3e719f2af1ed%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/db816351-88d2-4e18-9863-3e719f2af1ed%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a1b4fb14-a053-4c9b-ac5f-c6bb22fe3fdb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

warkolm · November 12, 2014, 9:39pm

Yes it will, anything in the elasticsearch.yml that changes needs a restart.

is the page that setting came from as well.

On 13 November 2014 08:38, shriyansh jain shriyanshajain@gmail.com wrote:

will it need a restart to being that info effect.?

Thank!
Shriyansh

On Wednesday, November 12, 2014 1:30:57 PM UTC-8, Mark Walkom wrote:

You can add that in yourself.

On 13 November 2014 08:26, shriyansh jain shriyan...@gmail.com wrote:

Thank you mark for pointing me out with the situation. As I am trying to
delete the index I am getting timeouts for that. But you suggested me to
increase discovery.zen.fd.ping_timeout to get rid of that. I am not able to
see any such variable in elasticsearch.yml file.
Can you please point me how to increase discovery.zen.fd.ping_timeout.

Thanks!
Shriyansh

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/db816351-88d2-4e18-9863-3e719f2af1ed%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/db816351-88d2-4e18-9863-3e719f2af1ed%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a1b4fb14-a053-4c9b-ac5f-c6bb22fe3fdb%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/a1b4fb14-a053-4c9b-ac5f-c6bb22fe3fdb%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZm%2BvwsZgR8bvvZifbeojP-hvR96avAQe_zxpAq1omnJSA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.