Elasticsearch randomly restarts


(yarin) #1

Hey all,

I have a testing machine that queries an elasticsearch node that is locally
on the very same machine.
The node has a single index with 2 primary shards and no replicas.

We've recently started running simultaneous tests on the machine and we
noticed that the elasticsearch node randomly restarts during searches.
On the elasticsearch log there's no indication of any errors

Here's a small example of the log

[2014-01-25 12:06:52,432][INFO ][node ] [Jenkins]
starting ...
[2014-01-25 12:06:52,600][INFO ][transport ] [Jenkins]
bound_address {inet[/0.0.0.0:9300]}, publish_address
{inet[/192.168.1.61:9300]}
[2014-01-25 12:06:55,685][INFO ][cluster.service ] [Jenkins]
new_master
[Jenkins][WEbUh6FNRmC6MhNjJKDNWQ][inet[/192.168.1.61:9300]]{max_local_storage_nodes=1},
reason: zen-disco-join (elected_as_master)
[2014-01-25 12:06:55,721][INFO ][discovery ] [Jenkins]
cluster_Jenkins/WEbUh6FNRmC6MhNjJKDNWQ
[2014-01-25 12:06:55,764][INFO ][http ] [Jenkins]
bound_address {inet[/127.0.0.1:9200]}, publish_address
{inet[/127.0.0.1:9200]}
[2014-01-25 12:06:55,765][INFO ][node ] [Jenkins]
started
[2014-01-25 12:06:56,708][INFO ][gateway ] [Jenkins]
recovered [1] indices into cluster_state
[2014-01-25 12:40:48,969][INFO ][node ] [Jenkins]
stopping ...
[2014-01-25 12:40:49,061][INFO ][node ] [Jenkins]
stopped
[2014-01-25 12:40:49,062][INFO ][node ] [Jenkins]
closing ...
[2014-01-25 12:40:49,072][INFO ][node ] [Jenkins] closed
[2014-01-25 12:40:59,274][INFO ][node ] [Jenkins]
version[0.90.5], pid[25402], build[c8714e8/2013-09-17T12:50:20Z]
[2014-01-25 12:40:59,274][INFO ][node ] [Jenkins]
initializing ...
[2014-01-25 12:40:59,294][INFO ][plugins ] [Jenkins]
loaded [inout], sites [bigdesk]
[2014-01-25 12:41:02,747][INFO ][node ] [Jenkins]
initialized
[2014-01-25 12:41:02,747][INFO ][node ] [Jenkins]
starting ...
[2014-01-25 12:41:02,899][INFO ][transport ] [Jenkins]
bound_address {inet[/0.0.0.0:9300]}, publish_address
{inet[/192.168.1.61:9300]}
[2014-01-25 12:41:05,981][INFO ][cluster.service ] [Jenkins]
new_master
[Jenkins][K2hM2A-DRwulWrMHuTz1Jw][inet[/192.168.1.61:9300]]{max_local_storage_nodes=1},
reason: zen-disco-join (elected_as_master)
[2014-01-25 12:41:06,018][INFO ][discovery ] [Jenkins]
cluster_Jenkins/K2hM2A-DRwulWrMHuTz1Jw
[2014-01-25 12:41:06,062][INFO ][http ] [Jenkins]
bound_address {inet[/127.0.0.1:9200]}, publish_address
{inet[/127.0.0.1:9200]}
[2014-01-25 12:41:06,062][INFO ][node ] [Jenkins]
started
[2014-01-25 12:41:06,812][INFO ][gateway ] [Jenkins]
recovered [1] indices into cluster_state
[2014-01-25 13:02:28,648][DEBUG][action.index ] [Jenkins]
Sending mapping updated to master: index [tenant] type [audit]
[2014-01-25 13:14:57,650][INFO ][node ] [Jenkins]
stopping ...
[2014-01-25 13:14:57,745][INFO ][node ] [Jenkins]
stopped
[2014-01-25 13:14:57,746][INFO ][node ] [Jenkins]
closing ...
[2014-01-25 13:14:57,762][INFO ][node ] [Jenkins] closed
[2014-01-25 13:15:09,126][INFO ][node ] [Jenkins]
version[0.90.5], pid[4000], build[c8714e8/2013-09-17T12:50:20Z]
[2014-01-25 13:15:09,127][INFO ][node ] [Jenkins]
initializing ...
[2014-01-25 13:15:09,142][INFO ][plugins ] [Jenkins]
loaded [inout], sites [bigdesk]
[2014-01-25 13:15:12,913][INFO ][node ] [Jenkins]
initialized
[2014-01-25 13:15:12,914][INFO ][node ] [Jenkins]
starting ...
[2014-01-25 13:15:13,162][INFO ][transport ] [Jenkins]
bound_address {inet[/0.0.0.0:9300]}, publish_address
{inet[/192.168.1.61:9300]}
[2014-01-25 13:15:16,268][INFO ][cluster.service ] [Jenkins]
new_master
[Jenkins][M1nNu0uvS1GGMPqHfJMyFA][inet[/192.168.1.61:9300]]{max_local_storage_nodes=1},
reason: zen-disco-join (elected_as_master)
[2014-01-25 13:15:16,326][INFO ][discovery ] [Jenkins]
cluster_Jenkins/M1nNu0uvS1GGMPqHfJMyFA
[2014-01-25 13:15:16,363][INFO ][http ] [Jenkins]
bound_address {inet[/127.0.0.1:9200]}, publish_address
{inet[/127.0.0.1:9200]}
[2014-01-25 13:15:16,364][INFO ][node ] [Jenkins]
started
[2014-01-25 13:15:17,264][INFO ][gateway ] [Jenkins]
recovered [1] indices into cluster_state
[2014-01-25 13:49:04,861][INFO ][node ] [Jenkins]
stopping ...
[2014-01-25 13:49:05,002][INFO ][node ] [Jenkins]
stopped
[2014-01-25 13:49:05,003][INFO ][node ] [Jenkins]
closing ...
[2014-01-25 13:49:05,033][INFO ][node ] [Jenkins] closed
[2014-01-25 13:49:16,651][INFO ][node ] [Jenkins]
version[0.90.5], pid[16773], build[c8714e8/2013-09-17T12:50:20Z]
[2014-01-25 13:49:16,652][INFO ][node ] [Jenkins]
initializing ...
[2014-01-25 13:49:16,677][INFO ][plugins ] [Jenkins]
loaded [inout], sites [bigdesk]
[2014-01-25 13:49:20,567][INFO ][node ] [Jenkins]
initialized
[2014-01-25 13:49:20,567][INFO ][node ] [Jenkins]
starting ...
[2014-01-25 13:49:20,802][INFO ][transport ] [Jenkins]
bound_address {inet[/0.0.0.0:9300]}, publish_address
{inet[/192.168.1.61:9300]}
[2014-01-25 13:49:23,904][INFO ][cluster.service ] [Jenkins]
new_master
[Jenkins][FDaqAZjQRsaDE7zlueNMBg][inet[/192.168.1.61:9300]]{max_local_storage_nodes=1},
reason: zen-disco-join (elected_as_master)
[2014-01-25 13:49:23,927][INFO ][discovery ] [Jenkins]
cluster_Jenkins/FDaqAZjQRsaDE7zlueNMBg
[2014-01-25 13:49:23,979][INFO ][http ] [Jenkins]
bound_address {inet[/127.0.0.1:9200]}, publish_address
{inet[/127.0.0.1:9200]}
[2014-01-25 13:49:23,980][INFO ][node ] [Jenkins]
started
[2014-01-25 13:49:25,120][INFO ][gateway ] [Jenkins]
recovered [1] indices into cluster_state

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b49ef9be-8f27-48ab-bdb6-ad4266d9eb07%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Mark Walkom) #2

It looks like it's happening every 33-34 minutes.
You might want to check if there is a scheduled task that is causing this
(eg cron).

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 26 January 2014 00:59, Yarin Miran yarin.miran@gmail.com wrote:

Hey all,

I have a testing machine that queries an elasticsearch node that is
locally on the very same machine.
The node has a single index with 2 primary shards and no replicas.

We've recently started running simultaneous tests on the machine and we
noticed that the elasticsearch node randomly restarts during searches.
On the elasticsearch log there's no indication of any errors

Here's a small example of the log

[2014-01-25 12:06:52,432][INFO ][node ] [Jenkins]
starting ...
[2014-01-25 12:06:52,600][INFO ][transport ] [Jenkins]
bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/
192.168.1.61:9300]}
[2014-01-25 12:06:55,685][INFO ][cluster.service ] [Jenkins]
new_master [Jenkins][WEbUh6FNRmC6MhNjJKDNWQ][inet[/192.168.1.61:9300]]{max_local_storage_nodes=1},
reason: zen-disco-join (elected_as_master)
[2014-01-25 12:06:55,721][INFO ][discovery ] [Jenkins]
cluster_Jenkins/WEbUh6FNRmC6MhNjJKDNWQ
[2014-01-25 12:06:55,764][INFO ][http ] [Jenkins]
bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[/
127.0.0.1:9200]}
[2014-01-25 12:06:55,765][INFO ][node ] [Jenkins]
started
[2014-01-25 12:06:56,708][INFO ][gateway ] [Jenkins]
recovered [1] indices into cluster_state
[2014-01-25 12:40:48,969][INFO ][node ] [Jenkins]
stopping ...
[2014-01-25 12:40:49,061][INFO ][node ] [Jenkins]
stopped
[2014-01-25 12:40:49,062][INFO ][node ] [Jenkins]
closing ...
[2014-01-25 12:40:49,072][INFO ][node ] [Jenkins]
closed
[2014-01-25 12:40:59,274][INFO ][node ] [Jenkins]
version[0.90.5], pid[25402], build[c8714e8/2013-09-17T12:50:20Z]
[2014-01-25 12:40:59,274][INFO ][node ] [Jenkins]
initializing ...
[2014-01-25 12:40:59,294][INFO ][plugins ] [Jenkins]
loaded [inout], sites [bigdesk]
[2014-01-25 12:41:02,747][INFO ][node ] [Jenkins]
initialized
[2014-01-25 12:41:02,747][INFO ][node ] [Jenkins]
starting ...
[2014-01-25 12:41:02,899][INFO ][transport ] [Jenkins]
bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/
192.168.1.61:9300]}
[2014-01-25 12:41:05,981][INFO ][cluster.service ] [Jenkins]
new_master [Jenkins][K2hM2A-DRwulWrMHuTz1Jw][inet[/192.168.1.61:9300]]{max_local_storage_nodes=1},
reason: zen-disco-join (elected_as_master)
[2014-01-25 12:41:06,018][INFO ][discovery ] [Jenkins]
cluster_Jenkins/K2hM2A-DRwulWrMHuTz1Jw
[2014-01-25 12:41:06,062][INFO ][http ] [Jenkins]
bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[/
127.0.0.1:9200]}
[2014-01-25 12:41:06,062][INFO ][node ] [Jenkins]
started
[2014-01-25 12:41:06,812][INFO ][gateway ] [Jenkins]
recovered [1] indices into cluster_state
[2014-01-25 13:02:28,648][DEBUG][action.index ] [Jenkins]
Sending mapping updated to master: index [tenant] type [audit]
[2014-01-25 13:14:57,650][INFO ][node ] [Jenkins]
stopping ...
[2014-01-25 13:14:57,745][INFO ][node ] [Jenkins]
stopped
[2014-01-25 13:14:57,746][INFO ][node ] [Jenkins]
closing ...
[2014-01-25 13:14:57,762][INFO ][node ] [Jenkins]
closed
[2014-01-25 13:15:09,126][INFO ][node ] [Jenkins]
version[0.90.5], pid[4000], build[c8714e8/2013-09-17T12:50:20Z]
[2014-01-25 13:15:09,127][INFO ][node ] [Jenkins]
initializing ...
[2014-01-25 13:15:09,142][INFO ][plugins ] [Jenkins]
loaded [inout], sites [bigdesk]
[2014-01-25 13:15:12,913][INFO ][node ] [Jenkins]
initialized
[2014-01-25 13:15:12,914][INFO ][node ] [Jenkins]
starting ...
[2014-01-25 13:15:13,162][INFO ][transport ] [Jenkins]
bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/
192.168.1.61:9300]}
[2014-01-25 13:15:16,268][INFO ][cluster.service ] [Jenkins]
new_master [Jenkins][M1nNu0uvS1GGMPqHfJMyFA][inet[/192.168.1.61:9300]]{max_local_storage_nodes=1},
reason: zen-disco-join (elected_as_master)
[2014-01-25 13:15:16,326][INFO ][discovery ] [Jenkins]
cluster_Jenkins/M1nNu0uvS1GGMPqHfJMyFA
[2014-01-25 13:15:16,363][INFO ][http ] [Jenkins]
bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[/
127.0.0.1:9200]}
[2014-01-25 13:15:16,364][INFO ][node ] [Jenkins]
started
[2014-01-25 13:15:17,264][INFO ][gateway ] [Jenkins]
recovered [1] indices into cluster_state
[2014-01-25 13:49:04,861][INFO ][node ] [Jenkins]
stopping ...
[2014-01-25 13:49:05,002][INFO ][node ] [Jenkins]
stopped
[2014-01-25 13:49:05,003][INFO ][node ] [Jenkins]
closing ...
[2014-01-25 13:49:05,033][INFO ][node ] [Jenkins]
closed
[2014-01-25 13:49:16,651][INFO ][node ] [Jenkins]
version[0.90.5], pid[16773], build[c8714e8/2013-09-17T12:50:20Z]
[2014-01-25 13:49:16,652][INFO ][node ] [Jenkins]
initializing ...
[2014-01-25 13:49:16,677][INFO ][plugins ] [Jenkins]
loaded [inout], sites [bigdesk]
[2014-01-25 13:49:20,567][INFO ][node ] [Jenkins]
initialized
[2014-01-25 13:49:20,567][INFO ][node ] [Jenkins]
starting ...
[2014-01-25 13:49:20,802][INFO ][transport ] [Jenkins]
bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/
192.168.1.61:9300]}
[2014-01-25 13:49:23,904][INFO ][cluster.service ] [Jenkins]
new_master [Jenkins][FDaqAZjQRsaDE7zlueNMBg][inet[/192.168.1.61:9300]]{max_local_storage_nodes=1},
reason: zen-disco-join (elected_as_master)
[2014-01-25 13:49:23,927][INFO ][discovery ] [Jenkins]
cluster_Jenkins/FDaqAZjQRsaDE7zlueNMBg
[2014-01-25 13:49:23,979][INFO ][http ] [Jenkins]
bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[/
127.0.0.1:9200]}
[2014-01-25 13:49:23,980][INFO ][node ] [Jenkins]
started
[2014-01-25 13:49:25,120][INFO ][gateway ] [Jenkins]
recovered [1] indices into cluster_state

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b49ef9be-8f27-48ab-bdb6-ad4266d9eb07%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624bP8ouWV3YucNEpX6z1ysn40uk6sk_WakE-gKG9oQKGLw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(yarin) #3

Yeah that's the interval this is happening,

I've checked the cronjobs on the machine, there's nothing there.

On 26 January 2014 00:35, Mark Walkom markw@campaignmonitor.com wrote:

It looks like it's happening every 33-34 minutes.
You might want to check if there is a scheduled task that is causing this
(eg cron).

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 26 January 2014 00:59, Yarin Miran yarin.miran@gmail.com wrote:

Hey all,

I have a testing machine that queries an elasticsearch node that is
locally on the very same machine.
The node has a single index with 2 primary shards and no replicas.

We've recently started running simultaneous tests on the machine and we
noticed that the elasticsearch node randomly restarts during searches.
On the elasticsearch log there's no indication of any errors

Here's a small example of the log

[2014-01-25 12:06:52,432][INFO ][node ] [Jenkins]
starting ...
[2014-01-25 12:06:52,600][INFO ][transport ] [Jenkins]
bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/
192.168.1.61:9300]}
[2014-01-25 12:06:55,685][INFO ][cluster.service ] [Jenkins]
new_master [Jenkins][WEbUh6FNRmC6MhNjJKDNWQ][inet[/192.168.1.61:9300]]{max_local_storage_nodes=1},
reason: zen-disco-join (elected_as_master)
[2014-01-25 12:06:55,721][INFO ][discovery ] [Jenkins]
cluster_Jenkins/WEbUh6FNRmC6MhNjJKDNWQ
[2014-01-25 12:06:55,764][INFO ][http ] [Jenkins]
bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[/
127.0.0.1:9200]}
[2014-01-25 12:06:55,765][INFO ][node ] [Jenkins]
started
[2014-01-25 12:06:56,708][INFO ][gateway ] [Jenkins]
recovered [1] indices into cluster_state
[2014-01-25 12:40:48,969][INFO ][node ] [Jenkins]
stopping ...
[2014-01-25 12:40:49,061][INFO ][node ] [Jenkins]
stopped
[2014-01-25 12:40:49,062][INFO ][node ] [Jenkins]
closing ...
[2014-01-25 12:40:49,072][INFO ][node ] [Jenkins]
closed
[2014-01-25 12:40:59,274][INFO ][node ] [Jenkins]
version[0.90.5], pid[25402], build[c8714e8/2013-09-17T12:50:20Z]
[2014-01-25 12:40:59,274][INFO ][node ] [Jenkins]
initializing ...
[2014-01-25 12:40:59,294][INFO ][plugins ] [Jenkins]
loaded [inout], sites [bigdesk]
[2014-01-25 12:41:02,747][INFO ][node ] [Jenkins]
initialized
[2014-01-25 12:41:02,747][INFO ][node ] [Jenkins]
starting ...
[2014-01-25 12:41:02,899][INFO ][transport ] [Jenkins]
bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/
192.168.1.61:9300]}
[2014-01-25 12:41:05,981][INFO ][cluster.service ] [Jenkins]
new_master [Jenkins][K2hM2A-DRwulWrMHuTz1Jw][inet[/192.168.1.61:9300]]{max_local_storage_nodes=1},
reason: zen-disco-join (elected_as_master)
[2014-01-25 12:41:06,018][INFO ][discovery ] [Jenkins]
cluster_Jenkins/K2hM2A-DRwulWrMHuTz1Jw
[2014-01-25 12:41:06,062][INFO ][http ] [Jenkins]
bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[/
127.0.0.1:9200]}
[2014-01-25 12:41:06,062][INFO ][node ] [Jenkins]
started
[2014-01-25 12:41:06,812][INFO ][gateway ] [Jenkins]
recovered [1] indices into cluster_state
[2014-01-25 13:02:28,648][DEBUG][action.index ] [Jenkins]
Sending mapping updated to master: index [tenant] type [audit]
[2014-01-25 13:14:57,650][INFO ][node ] [Jenkins]
stopping ...
[2014-01-25 13:14:57,745][INFO ][node ] [Jenkins]
stopped
[2014-01-25 13:14:57,746][INFO ][node ] [Jenkins]
closing ...
[2014-01-25 13:14:57,762][INFO ][node ] [Jenkins]
closed
[2014-01-25 13:15:09,126][INFO ][node ] [Jenkins]
version[0.90.5], pid[4000], build[c8714e8/2013-09-17T12:50:20Z]
[2014-01-25 13:15:09,127][INFO ][node ] [Jenkins]
initializing ...
[2014-01-25 13:15:09,142][INFO ][plugins ] [Jenkins]
loaded [inout], sites [bigdesk]
[2014-01-25 13:15:12,913][INFO ][node ] [Jenkins]
initialized
[2014-01-25 13:15:12,914][INFO ][node ] [Jenkins]
starting ...
[2014-01-25 13:15:13,162][INFO ][transport ] [Jenkins]
bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/
192.168.1.61:9300]}
[2014-01-25 13:15:16,268][INFO ][cluster.service ] [Jenkins]
new_master [Jenkins][M1nNu0uvS1GGMPqHfJMyFA][inet[/192.168.1.61:9300]]{max_local_storage_nodes=1},
reason: zen-disco-join (elected_as_master)
[2014-01-25 13:15:16,326][INFO ][discovery ] [Jenkins]
cluster_Jenkins/M1nNu0uvS1GGMPqHfJMyFA
[2014-01-25 13:15:16,363][INFO ][http ] [Jenkins]
bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[/
127.0.0.1:9200]}
[2014-01-25 13:15:16,364][INFO ][node ] [Jenkins]
started
[2014-01-25 13:15:17,264][INFO ][gateway ] [Jenkins]
recovered [1] indices into cluster_state
[2014-01-25 13:49:04,861][INFO ][node ] [Jenkins]
stopping ...
[2014-01-25 13:49:05,002][INFO ][node ] [Jenkins]
stopped
[2014-01-25 13:49:05,003][INFO ][node ] [Jenkins]
closing ...
[2014-01-25 13:49:05,033][INFO ][node ] [Jenkins]
closed
[2014-01-25 13:49:16,651][INFO ][node ] [Jenkins]
version[0.90.5], pid[16773], build[c8714e8/2013-09-17T12:50:20Z]
[2014-01-25 13:49:16,652][INFO ][node ] [Jenkins]
initializing ...
[2014-01-25 13:49:16,677][INFO ][plugins ] [Jenkins]
loaded [inout], sites [bigdesk]
[2014-01-25 13:49:20,567][INFO ][node ] [Jenkins]
initialized
[2014-01-25 13:49:20,567][INFO ][node ] [Jenkins]
starting ...
[2014-01-25 13:49:20,802][INFO ][transport ] [Jenkins]
bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/
192.168.1.61:9300]}
[2014-01-25 13:49:23,904][INFO ][cluster.service ] [Jenkins]
new_master [Jenkins][FDaqAZjQRsaDE7zlueNMBg][inet[/192.168.1.61:9300]]{max_local_storage_nodes=1},
reason: zen-disco-join (elected_as_master)
[2014-01-25 13:49:23,927][INFO ][discovery ] [Jenkins]
cluster_Jenkins/FDaqAZjQRsaDE7zlueNMBg
[2014-01-25 13:49:23,979][INFO ][http ] [Jenkins]
bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[/
127.0.0.1:9200]}
[2014-01-25 13:49:23,980][INFO ][node ] [Jenkins]
started
[2014-01-25 13:49:25,120][INFO ][gateway ] [Jenkins]
recovered [1] indices into cluster_state

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b49ef9be-8f27-48ab-bdb6-ad4266d9eb07%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/jYadXqdV45U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624bP8ouWV3YucNEpX6z1ysn40uk6sk_WakE-gKG9oQKGLw%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
Best regards,

Yarin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOVdZTd0%2BcWuVV4rMZJYaFR71TiVRjTyvaHEvcH%3D6V1w2JyUyQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Mark Walkom) #4

There might be something in the various /var/log/ files, have a look there
around the time and see if you can correlate anything.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 26 January 2014 18:44, Yarin Miran yarin.miran@gmail.com wrote:

Yeah that's the interval this is happening,

I've checked the cronjobs on the machine, there's nothing there.

On 26 January 2014 00:35, Mark Walkom markw@campaignmonitor.com wrote:

It looks like it's happening every 33-34 minutes.
You might want to check if there is a scheduled task that is causing this
(eg cron).

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 26 January 2014 00:59, Yarin Miran yarin.miran@gmail.com wrote:

Hey all,

I have a testing machine that queries an elasticsearch node that is
locally on the very same machine.
The node has a single index with 2 primary shards and no replicas.

We've recently started running simultaneous tests on the machine and we
noticed that the elasticsearch node randomly restarts during searches.
On the elasticsearch log there's no indication of any errors

Here's a small example of the log

[2014-01-25 12:06:52,432][INFO ][node ] [Jenkins]
starting ...
[2014-01-25 12:06:52,600][INFO ][transport ] [Jenkins]
bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/
192.168.1.61:9300]}
[2014-01-25 12:06:55,685][INFO ][cluster.service ] [Jenkins]
new_master [Jenkins][WEbUh6FNRmC6MhNjJKDNWQ][inet[/192.168.1.61:9300]]{max_local_storage_nodes=1},
reason: zen-disco-join (elected_as_master)
[2014-01-25 12:06:55,721][INFO ][discovery ] [Jenkins]
cluster_Jenkins/WEbUh6FNRmC6MhNjJKDNWQ
[2014-01-25 12:06:55,764][INFO ][http ] [Jenkins]
bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[/
127.0.0.1:9200]}
[2014-01-25 12:06:55,765][INFO ][node ] [Jenkins]
started
[2014-01-25 12:06:56,708][INFO ][gateway ] [Jenkins]
recovered [1] indices into cluster_state
[2014-01-25 12:40:48,969][INFO ][node ] [Jenkins]
stopping ...
[2014-01-25 12:40:49,061][INFO ][node ] [Jenkins]
stopped
[2014-01-25 12:40:49,062][INFO ][node ] [Jenkins]
closing ...
[2014-01-25 12:40:49,072][INFO ][node ] [Jenkins]
closed
[2014-01-25 12:40:59,274][INFO ][node ] [Jenkins]
version[0.90.5], pid[25402], build[c8714e8/2013-09-17T12:50:20Z]
[2014-01-25 12:40:59,274][INFO ][node ] [Jenkins]
initializing ...
[2014-01-25 12:40:59,294][INFO ][plugins ] [Jenkins]
loaded [inout], sites [bigdesk]
[2014-01-25 12:41:02,747][INFO ][node ] [Jenkins]
initialized
[2014-01-25 12:41:02,747][INFO ][node ] [Jenkins]
starting ...
[2014-01-25 12:41:02,899][INFO ][transport ] [Jenkins]
bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/
192.168.1.61:9300]}
[2014-01-25 12:41:05,981][INFO ][cluster.service ] [Jenkins]
new_master [Jenkins][K2hM2A-DRwulWrMHuTz1Jw][inet[/192.168.1.61:9300]]{max_local_storage_nodes=1},
reason: zen-disco-join (elected_as_master)
[2014-01-25 12:41:06,018][INFO ][discovery ] [Jenkins]
cluster_Jenkins/K2hM2A-DRwulWrMHuTz1Jw
[2014-01-25 12:41:06,062][INFO ][http ] [Jenkins]
bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[/
127.0.0.1:9200]}
[2014-01-25 12:41:06,062][INFO ][node ] [Jenkins]
started
[2014-01-25 12:41:06,812][INFO ][gateway ] [Jenkins]
recovered [1] indices into cluster_state
[2014-01-25 13:02:28,648][DEBUG][action.index ] [Jenkins]
Sending mapping updated to master: index [tenant] type [audit]
[2014-01-25 13:14:57,650][INFO ][node ] [Jenkins]
stopping ...
[2014-01-25 13:14:57,745][INFO ][node ] [Jenkins]
stopped
[2014-01-25 13:14:57,746][INFO ][node ] [Jenkins]
closing ...
[2014-01-25 13:14:57,762][INFO ][node ] [Jenkins]
closed
[2014-01-25 13:15:09,126][INFO ][node ] [Jenkins]
version[0.90.5], pid[4000], build[c8714e8/2013-09-17T12:50:20Z]
[2014-01-25 13:15:09,127][INFO ][node ] [Jenkins]
initializing ...
[2014-01-25 13:15:09,142][INFO ][plugins ] [Jenkins]
loaded [inout], sites [bigdesk]
[2014-01-25 13:15:12,913][INFO ][node ] [Jenkins]
initialized
[2014-01-25 13:15:12,914][INFO ][node ] [Jenkins]
starting ...
[2014-01-25 13:15:13,162][INFO ][transport ] [Jenkins]
bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/
192.168.1.61:9300]}
[2014-01-25 13:15:16,268][INFO ][cluster.service ] [Jenkins]
new_master [Jenkins][M1nNu0uvS1GGMPqHfJMyFA][inet[/192.168.1.61:9300]]{max_local_storage_nodes=1},
reason: zen-disco-join (elected_as_master)
[2014-01-25 13:15:16,326][INFO ][discovery ] [Jenkins]
cluster_Jenkins/M1nNu0uvS1GGMPqHfJMyFA
[2014-01-25 13:15:16,363][INFO ][http ] [Jenkins]
bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[/
127.0.0.1:9200]}
[2014-01-25 13:15:16,364][INFO ][node ] [Jenkins]
started
[2014-01-25 13:15:17,264][INFO ][gateway ] [Jenkins]
recovered [1] indices into cluster_state
[2014-01-25 13:49:04,861][INFO ][node ] [Jenkins]
stopping ...
[2014-01-25 13:49:05,002][INFO ][node ] [Jenkins]
stopped
[2014-01-25 13:49:05,003][INFO ][node ] [Jenkins]
closing ...
[2014-01-25 13:49:05,033][INFO ][node ] [Jenkins]
closed
[2014-01-25 13:49:16,651][INFO ][node ] [Jenkins]
version[0.90.5], pid[16773], build[c8714e8/2013-09-17T12:50:20Z]
[2014-01-25 13:49:16,652][INFO ][node ] [Jenkins]
initializing ...
[2014-01-25 13:49:16,677][INFO ][plugins ] [Jenkins]
loaded [inout], sites [bigdesk]
[2014-01-25 13:49:20,567][INFO ][node ] [Jenkins]
initialized
[2014-01-25 13:49:20,567][INFO ][node ] [Jenkins]
starting ...
[2014-01-25 13:49:20,802][INFO ][transport ] [Jenkins]
bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/
192.168.1.61:9300]}
[2014-01-25 13:49:23,904][INFO ][cluster.service ] [Jenkins]
new_master [Jenkins][FDaqAZjQRsaDE7zlueNMBg][inet[/192.168.1.61:9300]]{max_local_storage_nodes=1},
reason: zen-disco-join (elected_as_master)
[2014-01-25 13:49:23,927][INFO ][discovery ] [Jenkins]
cluster_Jenkins/FDaqAZjQRsaDE7zlueNMBg
[2014-01-25 13:49:23,979][INFO ][http ] [Jenkins]
bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[/
127.0.0.1:9200]}
[2014-01-25 13:49:23,980][INFO ][node ] [Jenkins]
started
[2014-01-25 13:49:25,120][INFO ][gateway ] [Jenkins]
recovered [1] indices into cluster_state

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b49ef9be-8f27-48ab-bdb6-ad4266d9eb07%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/jYadXqdV45U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624bP8ouWV3YucNEpX6z1ysn40uk6sk_WakE-gKG9oQKGLw%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
Best regards,

Yarin

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAOVdZTd0%2BcWuVV4rMZJYaFR71TiVRjTyvaHEvcH%3D6V1w2JyUyQ%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624aLCGGrzAR6x4gbNCoe%2Bffur5191SbuPf5yYWL%3D46Ut4A%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(yarin) #5

Thanks mark, I've dug through the logs and finally found a legacy
service that caused the restarts.

Thanks!

On 27/01/2014, Mark Walkom markw@campaignmonitor.com wrote:

There might be something in the various /var/log/ files, have a look there
around the time and see if you can correlate anything.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 26 January 2014 18:44, Yarin Miran yarin.miran@gmail.com wrote:

Yeah that's the interval this is happening,

I've checked the cronjobs on the machine, there's nothing there.

On 26 January 2014 00:35, Mark Walkom markw@campaignmonitor.com wrote:

It looks like it's happening every 33-34 minutes.
You might want to check if there is a scheduled task that is causing
this
(eg cron).

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 26 January 2014 00:59, Yarin Miran yarin.miran@gmail.com wrote:

Hey all,

I have a testing machine that queries an elasticsearch node that is
locally on the very same machine.
The node has a single index with 2 primary shards and no replicas.

We've recently started running simultaneous tests on the machine and we
noticed that the elasticsearch node randomly restarts during searches.
On the elasticsearch log there's no indication of any errors

Here's a small example of the log

[2014-01-25 12:06:52,432][INFO ][node ] [Jenkins]
starting ...
[2014-01-25 12:06:52,600][INFO ][transport ] [Jenkins]
bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/
192.168.1.61:9300]}
[2014-01-25 12:06:55,685][INFO ][cluster.service ] [Jenkins]
new_master
[Jenkins][WEbUh6FNRmC6MhNjJKDNWQ][inet[/192.168.1.61:9300]]{max_local_storage_nodes=1},
reason: zen-disco-join (elected_as_master)
[2014-01-25 12:06:55,721][INFO ][discovery ] [Jenkins]
cluster_Jenkins/WEbUh6FNRmC6MhNjJKDNWQ
[2014-01-25 12:06:55,764][INFO ][http ] [Jenkins]
bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[/
127.0.0.1:9200]}
[2014-01-25 12:06:55,765][INFO ][node ] [Jenkins]
started
[2014-01-25 12:06:56,708][INFO ][gateway ] [Jenkins]
recovered [1] indices into cluster_state
[2014-01-25 12:40:48,969][INFO ][node ] [Jenkins]
stopping ...
[2014-01-25 12:40:49,061][INFO ][node ] [Jenkins]
stopped
[2014-01-25 12:40:49,062][INFO ][node ] [Jenkins]
closing ...
[2014-01-25 12:40:49,072][INFO ][node ] [Jenkins]
closed
[2014-01-25 12:40:59,274][INFO ][node ] [Jenkins]
version[0.90.5], pid[25402], build[c8714e8/2013-09-17T12:50:20Z]
[2014-01-25 12:40:59,274][INFO ][node ] [Jenkins]
initializing ...
[2014-01-25 12:40:59,294][INFO ][plugins ] [Jenkins]
loaded [inout], sites [bigdesk]
[2014-01-25 12:41:02,747][INFO ][node ] [Jenkins]
initialized
[2014-01-25 12:41:02,747][INFO ][node ] [Jenkins]
starting ...
[2014-01-25 12:41:02,899][INFO ][transport ] [Jenkins]
bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/
192.168.1.61:9300]}
[2014-01-25 12:41:05,981][INFO ][cluster.service ] [Jenkins]
new_master
[Jenkins][K2hM2A-DRwulWrMHuTz1Jw][inet[/192.168.1.61:9300]]{max_local_storage_nodes=1},
reason: zen-disco-join (elected_as_master)
[2014-01-25 12:41:06,018][INFO ][discovery ] [Jenkins]
cluster_Jenkins/K2hM2A-DRwulWrMHuTz1Jw
[2014-01-25 12:41:06,062][INFO ][http ] [Jenkins]
bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[/
127.0.0.1:9200]}
[2014-01-25 12:41:06,062][INFO ][node ] [Jenkins]
started
[2014-01-25 12:41:06,812][INFO ][gateway ] [Jenkins]
recovered [1] indices into cluster_state
[2014-01-25 13:02:28,648][DEBUG][action.index ] [Jenkins]
Sending mapping updated to master: index [tenant] type [audit]
[2014-01-25 13:14:57,650][INFO ][node ] [Jenkins]
stopping ...
[2014-01-25 13:14:57,745][INFO ][node ] [Jenkins]
stopped
[2014-01-25 13:14:57,746][INFO ][node ] [Jenkins]
closing ...
[2014-01-25 13:14:57,762][INFO ][node ] [Jenkins]
closed
[2014-01-25 13:15:09,126][INFO ][node ] [Jenkins]
version[0.90.5], pid[4000], build[c8714e8/2013-09-17T12:50:20Z]
[2014-01-25 13:15:09,127][INFO ][node ] [Jenkins]
initializing ...
[2014-01-25 13:15:09,142][INFO ][plugins ] [Jenkins]
loaded [inout], sites [bigdesk]
[2014-01-25 13:15:12,913][INFO ][node ] [Jenkins]
initialized
[2014-01-25 13:15:12,914][INFO ][node ] [Jenkins]
starting ...
[2014-01-25 13:15:13,162][INFO ][transport ] [Jenkins]
bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/
192.168.1.61:9300]}
[2014-01-25 13:15:16,268][INFO ][cluster.service ] [Jenkins]
new_master
[Jenkins][M1nNu0uvS1GGMPqHfJMyFA][inet[/192.168.1.61:9300]]{max_local_storage_nodes=1},
reason: zen-disco-join (elected_as_master)
[2014-01-25 13:15:16,326][INFO ][discovery ] [Jenkins]
cluster_Jenkins/M1nNu0uvS1GGMPqHfJMyFA
[2014-01-25 13:15:16,363][INFO ][http ] [Jenkins]
bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[/
127.0.0.1:9200]}
[2014-01-25 13:15:16,364][INFO ][node ] [Jenkins]
started
[2014-01-25 13:15:17,264][INFO ][gateway ] [Jenkins]
recovered [1] indices into cluster_state
[2014-01-25 13:49:04,861][INFO ][node ] [Jenkins]
stopping ...
[2014-01-25 13:49:05,002][INFO ][node ] [Jenkins]
stopped
[2014-01-25 13:49:05,003][INFO ][node ] [Jenkins]
closing ...
[2014-01-25 13:49:05,033][INFO ][node ] [Jenkins]
closed
[2014-01-25 13:49:16,651][INFO ][node ] [Jenkins]
version[0.90.5], pid[16773], build[c8714e8/2013-09-17T12:50:20Z]
[2014-01-25 13:49:16,652][INFO ][node ] [Jenkins]
initializing ...
[2014-01-25 13:49:16,677][INFO ][plugins ] [Jenkins]
loaded [inout], sites [bigdesk]
[2014-01-25 13:49:20,567][INFO ][node ] [Jenkins]
initialized
[2014-01-25 13:49:20,567][INFO ][node ] [Jenkins]
starting ...
[2014-01-25 13:49:20,802][INFO ][transport ] [Jenkins]
bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/
192.168.1.61:9300]}
[2014-01-25 13:49:23,904][INFO ][cluster.service ] [Jenkins]
new_master
[Jenkins][FDaqAZjQRsaDE7zlueNMBg][inet[/192.168.1.61:9300]]{max_local_storage_nodes=1},
reason: zen-disco-join (elected_as_master)
[2014-01-25 13:49:23,927][INFO ][discovery ] [Jenkins]
cluster_Jenkins/FDaqAZjQRsaDE7zlueNMBg
[2014-01-25 13:49:23,979][INFO ][http ] [Jenkins]
bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[/
127.0.0.1:9200]}
[2014-01-25 13:49:23,980][INFO ][node ] [Jenkins]
started
[2014-01-25 13:49:25,120][INFO ][gateway ] [Jenkins]
recovered [1] indices into cluster_state

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.

To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b49ef9be-8f27-48ab-bdb6-ad4266d9eb07%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/jYadXqdV45U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624bP8ouWV3YucNEpX6z1ysn40uk6sk_WakE-gKG9oQKGLw%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
Best regards,

Yarin

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAOVdZTd0%2BcWuVV4rMZJYaFR71TiVRjTyvaHEvcH%3D6V1w2JyUyQ%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/jYadXqdV45U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEM624aLCGGrzAR6x4gbNCoe%2Bffur5191SbuPf5yYWL%3D46Ut4A%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Best regards,

Yarin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAOVdZTfgS-6c%3D0iZt%3DkWdTJQJO-jVd2SDgx5oKT4a92T-myxMA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #6