# Elasticsearch stopped itself from time to time
**URL:** https://discuss.elastic.co/t/elasticsearch-stopped-itself-from-time-to-time/12290
**Category:** Elasticsearch
**Created:** [June 6, 2013, 1:40am UTC](https://discuss.elastic.co/t/elasticsearch-stopped-itself-from-time-to-time/12290 "2013-06-06T01:40:56Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@Jacky\_Chu](https://discuss.elastic.co/u/Jacky_Chu)
#### Post date: [June 6, 2013, 1:40am UTC](https://discuss.elastic.co/t/elasticsearch-stopped-itself-from-time-to-time/12290/1 "2013-06-06T01:40:56Z")
Hi All,
I am currently using logstash 1.1.13 and elasticsearch 0.20.6 with single
node, single cluster. Two of them are sitting on same server CentOS 6.4 I
have already output 3 device logging from logstash to elasticsearch
successfully. After monitoring for a while, I found the service
elasticsearch was stopped itself and cause the logstash process terminated
as well. Here is the log I found from $ELASTICSEARCH\_HOME\log. The
workaround solution is I keep monitor port 9300/tcp and restart the service
in case of failure event occurs.
[2013-06-06 00:24:19,891][INFO][cluster.service] [FranzKafka]
> > removed
> > {[Ara?a][brRuXreYRXik1omW0WOAWg][inet[/192.168.90.45:9301]]{client=true,
> > data=false},}, reason:
> > zen-disco-node\_failed([Ara?a][brRuXreYRXik1omW0WOAWg][inet[/192.168.90.45:9301]]{client=true,
> > data=false}), reason transport disconnected (with verified connect)
>
> [2013-06-06 00:24:44,807][INFO][cluster.service] [FranzKafka]
>
> > added {[Uncle Ben
> > Parker][c0F6hrd8QIG701a2obkI9Q][inet[/192.168.90.45:9301]]{client=true,
> > data=false},}, reason: zen-disco-receive(join from node[[Uncle Ben
> > Parker][c0F6hrd8QIG701a2obkI9Q][inet[/192.168.90.45:9301]]{client=true,
> > data=false}])
>
> [2013-06-06 00:26:57,032][INFO][cluster.service] [FranzKafka]
>
> > removed {[Uncle Ben
> > Parker][c0F6hrd8QIG701a2obkI9Q][inet[/192.168.90.45:9301]]{client=true,
> > data=false},}, reason: zen-disco-node\_failed([Uncle Ben
> > Parker][c0F6hrd8QIG701a2obkI9Q][inet[/192.168.90.45:9301]]{client=true,
> > data=false}), reason transport disconnected (with verified connect)
>
> [2013-06-06 00:27:25,084][INFO][cluster.service] [FranzKafka]
>
> > added {[Blind
> > Faith][YLL4xgQHRQ2uc7uEmY0riA][inet[/192.168.90.45:9301]]{client=true,
> > data=false},}, reason: zen-disco-receive(join from node[[Blind
> > Faith][YLL4xgQHRQ2uc7uEmY0riA][inet[/192.168.90.45:9301]]{client=true,
> > data=false}])
>
> [2013-06-06 00:29:30,799][INFO][cluster.service] [FranzKafka]
>
> > removed {[Blind
> > Faith][YLL4xgQHRQ2uc7uEmY0riA][inet[/192.168.90.45:9301]]{client=true,
> > data=false},}, reason: zen-disco-node\_failed([Blind
> > Faith][YLL4xgQHRQ2uc7uEmY0riA][inet[/192.168.90.45:9301]]{client=true,
> > data=false}), reason transport disconnected (with verified connect)
>
> [2013-06-06 00:38:32,038][INFO][cluster.service] [FranzKafka]
>
> > added
> > {[Deathwatch][hit-QfOoQaCURqFMwyJXKw][inet[/192.168.90.45:9301]]{client=true,
> > data=false},}, reason: zen-disco-receive(join from
> > node[[Deathwatch][hit-QfOoQaCURqFMwyJXKw][inet[/192.168.90.45:9301]]{client=true,
> > data=false}])
>
> [2013-06-06 00:38:33,092][INFO][cluster.metadata] [FranzKafka]
>
> > [logstash-2013.06.05] update\_mapping [syslog] (dynamic)
>
> [2013-06-06 04:02:47,808][INFO][cluster.service] [FranzKafka]
>
> > removed
> > {[Deathwatch][hit-QfOoQaCURqFMwyJXKw][inet[/192.168.90.45:9301]]{client=true,
> > data=false},}, reason:
> > zen-disco-node\_failed([Deathwatch][hit-QfOoQaCURqFMwyJXKw][inet[/192.168.90.45:9301]]{client=true,
> > data=false}), reason transport disconnected (with verified connect)
It would be appreciate if someone could help. Thanks.
Rgds,
Jacky
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [June 6, 2013, 5:17am UTC](https://discuss.elastic.co/t/elasticsearch-stopped-itself-from-time-to-time/12290/2 "2013-06-06T05:17:18Z")
Can you see something in "Deathwatch" node's logs?
--
David 😉
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
Le 6 juin 2013 à 03:40, Jacky Chu [cksjackey@gmail.com](mailto:cksjackey@gmail.com) a écrit :
> Deathwatch
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Jacky\_Chu](https://discuss.elastic.co/u/Jacky_Chu)
#### Post date: [June 6, 2013, 2:50pm UTC](https://discuss.elastic.co/t/elasticsearch-stopped-itself-from-time-to-time/12290/3 "2013-06-06T14:50:22Z")
Hi David,
Thanks for reply but may know where is the "deathwatch" node's logs. I have
searched the elasticsearch log and data log but cannot find it.
[root@indexer config]# more elasticsearch.yml | grep log
> # You can exploit these settings to design advanced cluster topologies.
>
> # or analyzer definitions, translog settings, ...) for indices globally,
>
> # Path to directory containing configuration (this file and logging.yml):
>
> # Path to log files:
>
> # path.logs: /path/to/logs
>
> path.data: /var/log/elasticsearch
>
> path.logs: /usr/local/elasticsearch/logs
[root@indexer logs]# ls -l
total 1996
-rw-r--r--. 1 elasticsearch elasticsearch 0 May 30 11:11
> elasticsearch\_index\_indexing\_slowlog.log
-rw-r--r--. 1 elasticsearch elasticsearch 0 May 30 11:11
> elasticsearch\_index\_search\_slowlog.log
-rw-r--r--. 1 root root 0 May 30 12:13
> elasticsearch-kibana\_index\_indexing\_slowlog.log
-rw-r--r--. 1 root root 0 May 30 12:13
> elasticsearch-kibana\_index\_search\_slowlog.log
-rw-r--r--. 1 root root 1124595 Jun 6 21:17
> elasticsearch-kibana.log
-rw-r--r--. 1 root root 11350 May 30 23:46
> elasticsearch-kibana.log.2013-05-30
-rw-r--r--. 1 root root 277092 May 31 15:28
> elasticsearch-kibana.log.2013-05-31
-rw-r--r--. 1 root root 1936 Jun 1 18:38
> elasticsearch-kibana.log.2013-06-01
-rw-r--r--. 1 root root 293 Jun 2 08:00
> elasticsearch-kibana.log.2013-06-02
-rw-r--r--. 1 root root 5672 Jun 3 18:50
> elasticsearch-kibana.log.2013-06-03
-rw-r--r--. 1 root root 19202 Jun 4 20:39
> elasticsearch-kibana.log.2013-06-04
-rw-r--r--. 1 root root 540693 Jun 5 18:43
> elasticsearch-kibana.log.2013-06-05
-rw-r--r--. 1 elasticsearch elasticsearch 2230 May 30 11:31
> elasticsearch.log
-rw-r--r--. 1 root root 23586 Jun 6 09:41 service.log
Rgds,
Jacky
On Thursday, June 6, 2013 1:17:18 PM UTC+8, David Pilato wrote:
> Can you see something in "Deathwatch" node's logs?
>
> --
> David 😉
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>
> Le 6 juin 2013 Ã 03:40, Jacky Chu \<[cksj...@gmail.com](mailto:cksj...@gmail.com) \\> a
> écrit :
>
> > Deathwatch
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Amit\_Jain](https://discuss.elastic.co/u/Amit_Jain)
#### Post date: [October 28, 2013, 9:31am UTC](https://discuss.elastic.co/t/elasticsearch-stopped-itself-from-time-to-time/12290/4 "2013-10-28T09:31:34Z")
HI Jacky,
Did you get the solution for the same? If yes, please share as i am also
getting similar error in my environment.
Thanks in advance.
Regards,
Amit
On Thursday, June 6, 2013 2:40:56 AM UTC+1, Jacky Chu wrote:
> Hi All,
>
> I am currently using logstash 1.1.13 and elasticsearch 0.20.6 with single
> node, single cluster. Two of them are sitting on same server CentOS 6.4 I
> have already output 3 device logging from logstash to elasticsearch
> successfully. After monitoring for a while, I found the service
> elasticsearch was stopped itself and cause the logstash process terminated
> as well. Here is the log I found from $ELASTICSEARCH\_HOME\log. The
> workaround solution is I keep monitor port 9300/tcp and restart the service
> in case of failure event occurs.
>
> [2013-06-06 00:24:19,891][INFO][cluster.service] [FranzKafka]
>
> > > removed
> > > {[Ara?a][brRuXreYRXik1omW0WOAWg][inet[/192.168.90.45:9301]]{client=true,
> > > data=false},}, reason:
> > > zen-disco-node\_failed([Ara?a][brRuXreYRXik1omW0WOAWg][inet[/192.168.90.45:9301]]{client=true,
> > > data=false}), reason transport disconnected (with verified connect)
> >
> > [2013-06-06 00:24:44,807][INFO][cluster.service] [FranzKafka]
> >
> > > added {[Uncle Ben
> > > Parker][c0F6hrd8QIG701a2obkI9Q][inet[/192.168.90.45:9301]]{client=true,
> > > data=false},}, reason: zen-disco-receive(join from node[[Uncle Ben
> > > Parker][c0F6hrd8QIG701a2obkI9Q][inet[/192.168.90.45:9301]]{client=true,
> > > data=false}])
> >
> > [2013-06-06 00:26:57,032][INFO][cluster.service] [FranzKafka]
> >
> > > removed {[Uncle Ben
> > > Parker][c0F6hrd8QIG701a2obkI9Q][inet[/192.168.90.45:9301]]{client=true,
> > > data=false},}, reason: zen-disco-node\_failed([Uncle Ben
> > > Parker][c0F6hrd8QIG701a2obkI9Q][inet[/192.168.90.45:9301]]{client=true,
> > > data=false}), reason transport disconnected (with verified connect)
> >
> > [2013-06-06 00:27:25,084][INFO][cluster.service] [FranzKafka]
> >
> > > added {[Blind
> > > Faith][YLL4xgQHRQ2uc7uEmY0riA][inet[/192.168.90.45:9301]]{client=true,
> > > data=false},}, reason: zen-disco-receive(join from node[[Blind
> > > Faith][YLL4xgQHRQ2uc7uEmY0riA][inet[/192.168.90.45:9301]]{client=true,
> > > data=false}])
> >
> > [2013-06-06 00:29:30,799][INFO][cluster.service] [FranzKafka]
> >
> > > removed {[Blind
> > > Faith][YLL4xgQHRQ2uc7uEmY0riA][inet[/192.168.90.45:9301]]{client=true,
> > > data=false},}, reason: zen-disco-node\_failed([Blind
> > > Faith][YLL4xgQHRQ2uc7uEmY0riA][inet[/192.168.90.45:9301]]{client=true,
> > > data=false}), reason transport disconnected (with verified connect)
> >
> > [2013-06-06 00:38:32,038][INFO][cluster.service] [FranzKafka]
> >
> > > added
> > > {[Deathwatch][hit-QfOoQaCURqFMwyJXKw][inet[/192.168.90.45:9301]]{client=true,
> > > data=false},}, reason: zen-disco-receive(join from
> > > node[[Deathwatch][hit-QfOoQaCURqFMwyJXKw][inet[/192.168.90.45:9301]]{client=true,
> > > data=false}])
> >
> > [2013-06-06 00:38:33,092][INFO][cluster.metadata] [FranzKafka]
> >
> > > [logstash-2013.06.05] update\_mapping [syslog] (dynamic)
> >
> > [2013-06-06 04:02:47,808][INFO][cluster.service] [FranzKafka]
> >
> > > removed
> > > {[Deathwatch][hit-QfOoQaCURqFMwyJXKw][inet[/192.168.90.45:9301]]{client=true,
> > > data=false},}, reason:
> > > zen-disco-node\_failed([Deathwatch][hit-QfOoQaCURqFMwyJXKw][inet[/192.168.90.45:9301]]{client=true,
> > > data=false}), reason transport disconnected (with verified connect)
>
> It would be appreciate if someone could help. Thanks.
>
> Rgds,
> Jacky
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 2:10am UTC](https://discuss.elastic.co/t/elasticsearch-stopped-itself-from-time-to-time/12290/5 "2017-07-06T02:10:17Z")