Elasticsearch stopped itself from time to time

Elastic Stack Elasticsearch
1

Hi All,

I am currently using logstash 1.1.13 and elasticsearch 0.20.6 with single
node, single cluster. Two of them are sitting on same server CentOS 6.4 I
have already output 3 device logging from logstash to elasticsearch
successfully. After monitoring for a while, I found the service
elasticsearch was stopped itself and cause the logstash process terminated
as well. Here is the log I found from $ELASTICSEARCH_HOME\log. The
workaround solution is I keep monitor port 9300/tcp and restart the service
in case of failure event occurs.

[2013-06-06 00:24:19,891][INFO ][cluster.service ] [FranzKafka]

removed
{[Ara?a][brRuXreYRXik1omW0WOAWg][inet[/192.168.90.45:9301]]{client=true,
data=false},}, reason:
zen-disco-node_failed([Ara?a][brRuXreYRXik1omW0WOAWg][inet[/192.168.90.45:9301]]{client=true,
data=false}), reason transport disconnected (with verified connect)

[2013-06-06 00:24:44,807][INFO ][cluster.service ] [FranzKafka]

added {[Uncle Ben
Parker][c0F6hrd8QIG701a2obkI9Q][inet[/192.168.90.45:9301]]{client=true,
data=false},}, reason: zen-disco-receive(join from node[[Uncle Ben
Parker][c0F6hrd8QIG701a2obkI9Q][inet[/192.168.90.45:9301]]{client=true,
data=false}])

[2013-06-06 00:26:57,032][INFO ][cluster.service ] [FranzKafka]

removed {[Uncle Ben
Parker][c0F6hrd8QIG701a2obkI9Q][inet[/192.168.90.45:9301]]{client=true,
data=false},}, reason: zen-disco-node_failed([Uncle Ben
Parker][c0F6hrd8QIG701a2obkI9Q][inet[/192.168.90.45:9301]]{client=true,
data=false}), reason transport disconnected (with verified connect)

[2013-06-06 00:27:25,084][INFO ][cluster.service ] [FranzKafka]

added {[Blind
Faith][YLL4xgQHRQ2uc7uEmY0riA][inet[/192.168.90.45:9301]]{client=true,
data=false},}, reason: zen-disco-receive(join from node[[Blind
Faith][YLL4xgQHRQ2uc7uEmY0riA][inet[/192.168.90.45:9301]]{client=true,
data=false}])

[2013-06-06 00:29:30,799][INFO ][cluster.service ] [FranzKafka]

removed {[Blind
Faith][YLL4xgQHRQ2uc7uEmY0riA][inet[/192.168.90.45:9301]]{client=true,
data=false},}, reason: zen-disco-node_failed([Blind
Faith][YLL4xgQHRQ2uc7uEmY0riA][inet[/192.168.90.45:9301]]{client=true,
data=false}), reason transport disconnected (with verified connect)

[2013-06-06 00:38:32,038][INFO ][cluster.service ] [FranzKafka]

added
{[Deathwatch][hit-QfOoQaCURqFMwyJXKw][inet[/192.168.90.45:9301]]{client=true,
data=false},}, reason: zen-disco-receive(join from
node[[Deathwatch][hit-QfOoQaCURqFMwyJXKw][inet[/192.168.90.45:9301]]{client=true,
data=false}])

[2013-06-06 00:38:33,092][INFO ][cluster.metadata ] [FranzKafka]

[logstash-2013.06.05] update_mapping [syslog] (dynamic)

[2013-06-06 04:02:47,808][INFO ][cluster.service ] [FranzKafka]

removed
{[Deathwatch][hit-QfOoQaCURqFMwyJXKw][inet[/192.168.90.45:9301]]{client=true,
data=false},}, reason:
zen-disco-node_failed([Deathwatch][hit-QfOoQaCURqFMwyJXKw][inet[/192.168.90.45:9301]]{client=true,
data=false}), reason transport disconnected (with verified connect)

It would be appreciate if someone could help. Thanks.

Rgds,
Jacky

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Can you see something in "Deathwatch" node's logs?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 6 juin 2013 à 03:40, Jacky Chu cksjackey@gmail.com a écrit :

Deathwatch

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Hi David,

Thanks for reply but may know where is the "deathwatch" node's logs. I have
searched the elasticsearch log and data log but cannot find it.

[root@indexer config]# more elasticsearch.yml | grep log

You can exploit these settings to design advanced cluster topologies.

or analyzer definitions, translog settings, ...) for indices globally,

Path to directory containing configuration (this file and logging.yml):

Path to log files:

path.logs: /path/to/logs

path.data: /var/log/elasticsearch

path.logs: /usr/local/elasticsearch/logs

[root@indexer logs]# ls -l

total 1996

-rw-r--r--. 1 elasticsearch elasticsearch 0 May 30 11:11

elasticsearch_index_indexing_slowlog.log

-rw-r--r--. 1 elasticsearch elasticsearch 0 May 30 11:11

elasticsearch_index_search_slowlog.log

-rw-r--r--. 1 root root 0 May 30 12:13

elasticsearch-kibana_index_indexing_slowlog.log

-rw-r--r--. 1 root root 0 May 30 12:13

elasticsearch-kibana_index_search_slowlog.log

-rw-r--r--. 1 root root 1124595 Jun 6 21:17

elasticsearch-kibana.log

-rw-r--r--. 1 root root 11350 May 30 23:46

elasticsearch-kibana.log.2013-05-30

-rw-r--r--. 1 root root 277092 May 31 15:28

elasticsearch-kibana.log.2013-05-31

-rw-r--r--. 1 root root 1936 Jun 1 18:38

elasticsearch-kibana.log.2013-06-01

-rw-r--r--. 1 root root 293 Jun 2 08:00

elasticsearch-kibana.log.2013-06-02

-rw-r--r--. 1 root root 5672 Jun 3 18:50

elasticsearch-kibana.log.2013-06-03

-rw-r--r--. 1 root root 19202 Jun 4 20:39

elasticsearch-kibana.log.2013-06-04

-rw-r--r--. 1 root root 540693 Jun 5 18:43

elasticsearch-kibana.log.2013-06-05

-rw-r--r--. 1 elasticsearch elasticsearch 2230 May 30 11:31

elasticsearch.log

-rw-r--r--. 1 root root 23586 Jun 6 09:41 service.log

Rgds,
Jacky

On Thursday, June 6, 2013 1:17:18 PM UTC+8, David Pilato wrote:

Can you see something in "Deathwatch" node's logs?

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 6 juin 2013 à 03:40, Jacky Chu <cksj...@gmail.com <javascript:>> a
écrit :

Deathwatch

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

HI Jacky,

Did you get the solution for the same? If yes, please share as i am also
getting similar error in my environment.
Thanks in advance.

Regards,
Amit

On Thursday, June 6, 2013 2:40:56 AM UTC+1, Jacky Chu wrote:

Hi All,

I am currently using logstash 1.1.13 and elasticsearch 0.20.6 with single
node, single cluster. Two of them are sitting on same server CentOS 6.4 I
have already output 3 device logging from logstash to elasticsearch
successfully. After monitoring for a while, I found the service
elasticsearch was stopped itself and cause the logstash process terminated
as well. Here is the log I found from $ELASTICSEARCH_HOME\log. The
workaround solution is I keep monitor port 9300/tcp and restart the service
in case of failure event occurs.

[2013-06-06 00:24:19,891][INFO ][cluster.service ] [FranzKafka]

removed
{[Ara?a][brRuXreYRXik1omW0WOAWg][inet[/192.168.90.45:9301]]{client=true,
data=false},}, reason:
zen-disco-node_failed([Ara?a][brRuXreYRXik1omW0WOAWg][inet[/192.168.90.45:9301]]{client=true,
data=false}), reason transport disconnected (with verified connect)

[2013-06-06 00:24:44,807][INFO ][cluster.service ] [FranzKafka]

added {[Uncle Ben
Parker][c0F6hrd8QIG701a2obkI9Q][inet[/192.168.90.45:9301]]{client=true,
data=false},}, reason: zen-disco-receive(join from node[[Uncle Ben
Parker][c0F6hrd8QIG701a2obkI9Q][inet[/192.168.90.45:9301]]{client=true,
data=false}])

[2013-06-06 00:26:57,032][INFO ][cluster.service ] [FranzKafka]

removed {[Uncle Ben
Parker][c0F6hrd8QIG701a2obkI9Q][inet[/192.168.90.45:9301]]{client=true,
data=false},}, reason: zen-disco-node_failed([Uncle Ben
Parker][c0F6hrd8QIG701a2obkI9Q][inet[/192.168.90.45:9301]]{client=true,
data=false}), reason transport disconnected (with verified connect)

[2013-06-06 00:27:25,084][INFO ][cluster.service ] [FranzKafka]

added {[Blind
Faith][YLL4xgQHRQ2uc7uEmY0riA][inet[/192.168.90.45:9301]]{client=true,
data=false},}, reason: zen-disco-receive(join from node[[Blind
Faith][YLL4xgQHRQ2uc7uEmY0riA][inet[/192.168.90.45:9301]]{client=true,
data=false}])

[2013-06-06 00:29:30,799][INFO ][cluster.service ] [FranzKafka]

removed {[Blind
Faith][YLL4xgQHRQ2uc7uEmY0riA][inet[/192.168.90.45:9301]]{client=true,
data=false},}, reason: zen-disco-node_failed([Blind
Faith][YLL4xgQHRQ2uc7uEmY0riA][inet[/192.168.90.45:9301]]{client=true,
data=false}), reason transport disconnected (with verified connect)

[2013-06-06 00:38:32,038][INFO ][cluster.service ] [FranzKafka]

added
{[Deathwatch][hit-QfOoQaCURqFMwyJXKw][inet[/192.168.90.45:9301]]{client=true,
data=false},}, reason: zen-disco-receive(join from
node[[Deathwatch][hit-QfOoQaCURqFMwyJXKw][inet[/192.168.90.45:9301]]{client=true,
data=false}])

[2013-06-06 00:38:33,092][INFO ][cluster.metadata ] [FranzKafka]

[logstash-2013.06.05] update_mapping [syslog] (dynamic)

[2013-06-06 04:02:47,808][INFO ][cluster.service ] [FranzKafka]

removed
{[Deathwatch][hit-QfOoQaCURqFMwyJXKw][inet[/192.168.90.45:9301]]{client=true,
data=false},}, reason:
zen-disco-node_failed([Deathwatch][hit-QfOoQaCURqFMwyJXKw][inet[/192.168.90.45:9301]]{client=true,
data=false}), reason transport disconnected (with verified connect)

It would be appreciate if someone could help. Thanks.

Rgds,
Jacky

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.