Elasticsearch gets unresponsive with logstash

Elastic Stack Elasticsearch
1

Hello,

we use a two node elasticsearch (version 1.1.1) cluster in combination with
logstash to centralize our log management.
We currently facing the problem that the cluster gets unusable unresponsive
after a couple of hours. The "head" plugin doesn't get any data as well as
the Kibana web-interface.
We cant see any errors in the logs and a curl Get respond 200 OK on both
nodes.

After restarting the elasticsearch service everything works fine again.

On both nodes we have the following configuration:
grep ^[^#] /etc/elasticsearch/elasticsearch.yml
cluster.name: TEST_Logstash
node.name: "server2"
path.data: /data/elasticsearch
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["server1:9300", "server2:9300"]
monitor.jvm.gc.young.warn: 500ms
monitor.jvm.gc.young.info: 200ms
monitor.jvm.gc.old.warn: 5s
monitor.jvm.gc.old.info: 2s

Cheers
Chris

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4b2aa171-b2cc-4241-bd74-41cb50bb9009%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Check your ES logs, you are probably running into GC issues.

How many nodes, how much heap, how much data is on the nodes - both GB and
index count, how many shards, how many replicas, what java version?

On 26 December 2014 at 17:21, chris85lang@googlemail.com wrote:

Hello,

we use a two node elasticsearch (version 1.1.1) cluster in combination
with logstash to centralize our log management.
We currently facing the problem that the cluster gets unusable
unresponsive after a couple of hours. The "head" plugin doesn't get any
data as well as the Kibana web-interface.
We cant see any errors in the logs and a curl Get respond 200 OK on both
nodes.

After restarting the elasticsearch service everything works fine again.

On both nodes we have the following configuration:
grep [1] /etc/elasticsearch/elasticsearch.yml
cluster.name http://cluster.name: TEST_Logstash
node.name http://node.name: "server2"
path.data: /data/elasticsearch
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["server1:9300", "server2:9300"]
monitor.jvm.gc.young.warn: 500ms
monitor.jvm.gc.young.info http://monitor.jvm.gc.young.info: 200ms
monitor.jvm.gc.old.warn: 5s
monitor.jvm.gc.old.info http://monitor.jvm.gc.old.info: 2s

Cheers
Chris

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4b2aa171-b2cc-4241-bd74-41cb50bb9009%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/4b2aa171-b2cc-4241-bd74-41cb50bb9009%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-2w5sD0u4F1463%2BpD8ZB_O0kKWFdA3MP65F9mGnAURTA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

  1. ^# ↩︎

3

Hello,

sorry for the incomplete information I'm kinda new to elasticsearch.

  • In our setup we have 2x elasticsearch data/client nodes and 4 logstash
    instances just seen as clients.
  • both elsaticsearch servers have 32 GB memory and the heap is
    8GB -Xms8g -Xmx8g -Xss256k
  • 150 total shards
  • 15 Indices
  • 40,100,997 documents
  • 21 GB total size
  • Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
  • One replica

For each day a new index is created with 5 shards an one replica.

I hope this helps, thank you!

Cheers
Chris

On Saturday, December 27, 2014 3:52:30 AM UTC+7, Mark Walkom wrote:

Check your ES logs, you are probably running into GC issues.

How many nodes, how much heap, how much data is on the nodes - both GB and
index count, how many shards, how many replicas, what java version?

On 26 December 2014 at 17:21, <chris...@googlemail.com <javascript:>>
wrote:

Hello,

we use a two node elasticsearch (version 1.1.1) cluster in combination
with logstash to centralize our log management.
We currently facing the problem that the cluster gets unusable
unresponsive after a couple of hours. The "head" plugin doesn't get any
data as well as the Kibana web-interface.
We cant see any errors in the logs and a curl Get respond 200 OK on both
nodes.

After restarting the elasticsearch service everything works fine again.

On both nodes we have the following configuration:
grep [1] /etc/elasticsearch/elasticsearch.yml
cluster.name http://cluster.name: TEST_Logstash
node.name http://node.name: "server2"
path.data: /data/elasticsearch
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["server1:9300", "server2:9300"]
monitor.jvm.gc.young.warn: 500ms
monitor.jvm.gc.young.info http://monitor.jvm.gc.young.info: 200ms
monitor.jvm.gc.old.warn: 5s
monitor.jvm.gc.old.info http://monitor.jvm.gc.old.info: 2s

Cheers
Chris

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4b2aa171-b2cc-4241-bd74-41cb50bb9009%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/4b2aa171-b2cc-4241-bd74-41cb50bb9009%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4aea7c7d-5d13-465d-8891-bb96789be1d7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

  1. ^# ↩︎