Elasticsearch cluster occasional shutdown by itself


(Tfasun) #1

Usually in the morning my elasticsearch cluster shutdown by itself. My elasticsearch log is below.
Why Korvus node shutdown?
After I restart my cluster .it work.My new node name is Master Khan

-21 06:41:57,889][INFO ][discovery.zen ] [Korvus] master_left [{Korvus}{iMJI5jdDRvG3_W6NDU1Jzg}{127.0.0.1}{127.0.0.1:9300}], reason [transport disconnected]
[2016-12-21 06:41:57,900][WARN ][discovery.zen ] [Korvus] master left (reason = transport disconnected), current nodes: {{Korvus}{prsC2OwYThS1pqFwpyYNzA}{127.0.0.1}{127.0.0.1:9301},}
[2016-12-21 06:41:57,901][INFO ][cluster.service ] [Korvus] removed {{Korvus}{iMJI5jdDRvG3_W6NDU1Jzg}{127.0.0.1}{127.0.0.1:9300},}, reason: zen-disco-master_failed ({Korvus}{iMJI5jdDRvG3_W6NDU1Jzg}{127.0.0.1}{127.0.0.1:9300})
[2016-12-21 06:42:00,971][INFO ][cluster.routing.allocation] [Korvus] Cluster health status changed from [GREEN] to [RED] (reason: [nodes joined]).
[2016-12-21 06:42:00,972][INFO ][cluster.service ] [Korvus] new_master {Korvus}{prsC2OwYThS1pqFwpyYNzA}{127.0.0.1}{127.0.0.1:9301}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2016-12-21 06:47:38,987][INFO ][node ] [Master Khan] version[2.3.1], pid[31686], build[bd98092/2016-04-04T12:25:05Z]
[2016-12-21 06:47:38,987][INFO ][node ] [Master Khan] initializing ...
[2016-12-21 06:47:40,626][INFO ][plugins ] [Master Khan] modules [lang-groovy, reindex, lang-expression], plugins [cloud-aws, head], sites [head]
[2016-12-21 06:47:40,670][INFO ][env ] [Master Khan] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [17gb], net total_space [19.5gb], spins? [no], types [ext4]
[2016-12-21 06:47:40,670][INFO ][env ] [Master Khan] heap size [1.9gb], compressed ordinary object pointers [true]
[2016-12-21 06:47:43,831][INFO ][node ] [Master Khan] initialized
[2016-12-21 06:47:43,831][INFO ][node ] [Master Khan] starting ...
[2016-12-21 06:47:43,997][INFO ][transport ] [Master Khan] publish_address {127.0.0.1:9300}, bound_addresses {[::]:9300}
[2016-12-21 06:47:44,004][INFO ][discovery ] [Master Khan] elasticsearch/Ma7OSi1BRXKeCRTst7mv-Q
[2016-12-21 06:47:47,119][INFO ][cluster.service ] [Master Khan] detected_master {Master Khan}{LE3_4tKVTzuYRSKnH0T9Xg}{127.0.0.1}{127.0.0.1:9301}, added {{Master Khan}{LE3_4tKVTzuYRSKnH0T9Xg}{127.0.0.1}{127.0.0.1:9301},}, reason: zen-disco-receive(from master [{Master Khan}{LE3_4tKVTzuYRSKnH0T9Xg}{127.0.0.1}{127.0.0.1:9301}])
[2016-12-21 06:47:47,135][INFO ][http ] [Master Khan] publish_address {127.0.0.1:9200}, bound_addresses {[::]:9200}
[2016-12-21 06:47:47,135][INFO ][node ] [Master Khan] started


(Jason Tedor) #2

Since there is nothing in the logs indicating a clean shutdown, I suspect something like either a JVM crash or the OS OOM killer. For the former, check the working directory for Elasticsearch for the presence of an hs_err_<pid>.log file. For the latter, check your OS logs like /var/log/messages and other similar infrastructure.


(Tfasun) #3

Thank you very much.


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.