Automatic partial shutdown of cluster

krishnan · August 14, 2014, 4:23am

Hi All,

I have cluster of one master only nodes and 7 data only nodes. Some nodes
are getting shutdown automatically. Could you please advise what is the
possible cause of this behaviour or how do I investigate the issue.

For you perusal the log entry is below :

Data node logs :

[2014-08-13 17:10:22,882][INFO ][action.admin.cluster.node.shutdown]
[search7] shutting down in [200ms]
[2014-08-13 17:10:23,102][INFO ][action.admin.cluster.node.shutdown]
[search7] initiating requested shutdown...
[2014-08-13 17:10:23,106][INFO ][node ] [search7]
stopping ...
[2014-08-13 17:10:52,290][INFO ][node ] [search7]
stopped
[2014-08-13 17:10:52,290][INFO ][node ] [search7]
closing ...
[2014-08-13 17:10:52,323][INFO ][node ] [search7] closed

Server node logs :
[2014-08-13 17:10:21,879][INFO ][action.admin.cluster.node.shutdown]
[search1] [partial_cluster_shutdown]: requested, shutting down
[[QowbqobdTyqrMZ00EVSvbw]] in [1s]
[2014-08-13 17:10:21,991][INFO ][action.admin.cluster.node.shutdown]
[search1] [partial_cluster_shutdown]: requested, shutting down
[[hRJliGmSSTuXX1Kmmjz94Q]] in [1s]
[2014-08-13 17:10:22,206][INFO ][action.admin.cluster.node.shutdown]
[search1] [partial_cluster_shutdown]: requested, shutting down
[[dqnFXTV9RoKeJpSy7zNqPA]] in [1s]
[2014-08-13 17:10:22,901][INFO ][action.admin.cluster.node.shutdown]
[search1] [partial_cluster_shutdown]: done shutting down
[[QowbqobdTyqrMZ00EVSvbw]]
[2014-08-13 17:10:22,999][INFO ][action.admin.cluster.node.shutdown]
[search1] [partial_cluster_shutdown]: done shutting down
[[hRJliGmSSTuXX1Kmmjz94Q]]
[2014-08-13 17:10:23,290][INFO ][action.admin.cluster.node.shutdown]
[search1] [partial_cluster_shutdown]: done shutting down
[[dqnFXTV9RoKeJpSy7zNqPA]]

Thank You,
Krishna

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1cf8f1ca-6388-4ffa-b4aa-805fc4254b8e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

warkolm · August 14, 2014, 4:32am

Something is shutting your node down using the API.

You might want to look at

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 14 August 2014 14:23, Krishnan krishna.dave@gmail.com wrote:

Hi All,

I have cluster of one master only nodes and 7 data only nodes. Some nodes
are getting shutdown automatically. Could you please advise what is the
possible cause of this behaviour or how do I investigate the issue.

For you perusal the log entry is below :

Data node logs :

[2014-08-13 17:10:22,882][INFO ][action.admin.cluster.node.shutdown]
[search7] shutting down in [200ms]
[2014-08-13 17:10:23,102][INFO ][action.admin.cluster.node.shutdown]
[search7] initiating requested shutdown...
[2014-08-13 17:10:23,106][INFO ][node ] [search7]
stopping ...
[2014-08-13 17:10:52,290][INFO ][node ] [search7]
stopped
[2014-08-13 17:10:52,290][INFO ][node ] [search7]
closing ...
[2014-08-13 17:10:52,323][INFO ][node ] [search7]
closed

Server node logs :
[2014-08-13 17:10:21,879][INFO ][action.admin.cluster.node.shutdown]
[search1] [partial_cluster_shutdown]: requested, shutting down
[[QowbqobdTyqrMZ00EVSvbw]] in [1s]
[2014-08-13 17:10:21,991][INFO ][action.admin.cluster.node.shutdown]
[search1] [partial_cluster_shutdown]: requested, shutting down
[[hRJliGmSSTuXX1Kmmjz94Q]] in [1s]
[2014-08-13 17:10:22,206][INFO ][action.admin.cluster.node.shutdown]
[search1] [partial_cluster_shutdown]: requested, shutting down
[[dqnFXTV9RoKeJpSy7zNqPA]] in [1s]
[2014-08-13 17:10:22,901][INFO ][action.admin.cluster.node.shutdown]
[search1] [partial_cluster_shutdown]: done shutting down
[[QowbqobdTyqrMZ00EVSvbw]]
[2014-08-13 17:10:22,999][INFO ][action.admin.cluster.node.shutdown]
[search1] [partial_cluster_shutdown]: done shutting down
[[hRJliGmSSTuXX1Kmmjz94Q]]
[2014-08-13 17:10:23,290][INFO ][action.admin.cluster.node.shutdown]
[search1] [partial_cluster_shutdown]: done shutting down
[[dqnFXTV9RoKeJpSy7zNqPA]]

Thank You,
Krishna

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1cf8f1ca-6388-4ffa-b4aa-805fc4254b8e%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1cf8f1ca-6388-4ffa-b4aa-805fc4254b8e%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624beY-NaK15DqeBDM3N5Ayfe%2BJZMTpBdkG3_Z6NNcjAkdQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

krishnan · August 14, 2014, 8:04am

Thank you so much for the suggestion, Walkom.
Now how do I find out what is shutting down my nodes? I checked /var/logs,
did not find any thing suspicious. is this related to out of memory system
states.

Regards,
Krishna

On Thursday, August 14, 2014 10:03:10 AM UTC+5:30, Mark Walkom wrote:

Something is shutting your node down using the API.

You might want to look at
Elasticsearch Platform — Find real-time answers at scale | Elastic

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 14 August 2014 14:23, Krishnan <krishn...@gmail.com <javascript:>>
wrote:

Hi All,

I have cluster of one master only nodes and 7 data only nodes. Some nodes
are getting shutdown automatically. Could you please advise what is the
possible cause of this behaviour or how do I investigate the issue.

For you perusal the log entry is below :

Data node logs :

[2014-08-13 17:10:22,882][INFO ][action.admin.cluster.node.shutdown]
[search7] shutting down in [200ms]
[2014-08-13 17:10:23,102][INFO ][action.admin.cluster.node.shutdown]
[search7] initiating requested shutdown...
[2014-08-13 17:10:23,106][INFO ][node ] [search7]
stopping ...
[2014-08-13 17:10:52,290][INFO ][node ] [search7]
stopped
[2014-08-13 17:10:52,290][INFO ][node ] [search7]
closing ...
[2014-08-13 17:10:52,323][INFO ][node ] [search7]
closed

Server node logs :
[2014-08-13 17:10:21,879][INFO ][action.admin.cluster.node.shutdown]
[search1] [partial_cluster_shutdown]: requested, shutting down
[[QowbqobdTyqrMZ00EVSvbw]] in [1s]
[2014-08-13 17:10:21,991][INFO ][action.admin.cluster.node.shutdown]
[search1] [partial_cluster_shutdown]: requested, shutting down
[[hRJliGmSSTuXX1Kmmjz94Q]] in [1s]
[2014-08-13 17:10:22,206][INFO ][action.admin.cluster.node.shutdown]
[search1] [partial_cluster_shutdown]: requested, shutting down
[[dqnFXTV9RoKeJpSy7zNqPA]] in [1s]
[2014-08-13 17:10:22,901][INFO ][action.admin.cluster.node.shutdown]
[search1] [partial_cluster_shutdown]: done shutting down
[[QowbqobdTyqrMZ00EVSvbw]]
[2014-08-13 17:10:22,999][INFO ][action.admin.cluster.node.shutdown]
[search1] [partial_cluster_shutdown]: done shutting down
[[hRJliGmSSTuXX1Kmmjz94Q]]
[2014-08-13 17:10:23,290][INFO ][action.admin.cluster.node.shutdown]
[search1] [partial_cluster_shutdown]: done shutting down
[[dqnFXTV9RoKeJpSy7zNqPA]]

Thank You,
Krishna

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1cf8f1ca-6388-4ffa-b4aa-805fc4254b8e%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1cf8f1ca-6388-4ffa-b4aa-805fc4254b8e%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b049afa9-582f-4ffe-bf0e-9150a816611b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.