Elasticsearch cluster occasional shutdown by itself

Usually in the morning my elasticsearch cluster shutdown by itself. My elasticsearch log is below.
Why Korvus node shutdown?
After I restart my cluster .it work.My new node name is Master Khan

-21 06:41:57,889][INFO ][discovery.zen ] [Korvus] master_left [{Korvus}{iMJI5jdDRvG3_W6NDU1Jzg}{}{}], reason [transport disconnected]
[2016-12-21 06:41:57,900][WARN ][discovery.zen ] [Korvus] master left (reason = transport disconnected), current nodes: {{Korvus}{prsC2OwYThS1pqFwpyYNzA}{}{},}
[2016-12-21 06:41:57,901][INFO ][cluster.service ] [Korvus] removed {{Korvus}{iMJI5jdDRvG3_W6NDU1Jzg}{}{},}, reason: zen-disco-master_failed ({Korvus}{iMJI5jdDRvG3_W6NDU1Jzg}{}{})
[2016-12-21 06:42:00,971][INFO ][cluster.routing.allocation] [Korvus] Cluster health status changed from [GREEN] to [RED] (reason: [nodes joined]).
[2016-12-21 06:42:00,972][INFO ][cluster.service ] [Korvus] new_master {Korvus}{prsC2OwYThS1pqFwpyYNzA}{}{}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2016-12-21 06:47:38,987][INFO ][node ] [Master Khan] version[2.3.1], pid[31686], build[bd98092/2016-04-04T12:25:05Z]
[2016-12-21 06:47:38,987][INFO ][node ] [Master Khan] initializing ...
[2016-12-21 06:47:40,626][INFO ][plugins ] [Master Khan] modules [lang-groovy, reindex, lang-expression], plugins [cloud-aws, head], sites [head]
[2016-12-21 06:47:40,670][INFO ][env ] [Master Khan] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [17gb], net total_space [19.5gb], spins? [no], types [ext4]
[2016-12-21 06:47:40,670][INFO ][env ] [Master Khan] heap size [1.9gb], compressed ordinary object pointers [true]
[2016-12-21 06:47:43,831][INFO ][node ] [Master Khan] initialized
[2016-12-21 06:47:43,831][INFO ][node ] [Master Khan] starting ...
[2016-12-21 06:47:43,997][INFO ][transport ] [Master Khan] publish_address {}, bound_addresses {[::]:9300}
[2016-12-21 06:47:44,004][INFO ][discovery ] [Master Khan] elasticsearch/Ma7OSi1BRXKeCRTst7mv-Q
[2016-12-21 06:47:47,119][INFO ][cluster.service ] [Master Khan] detected_master {Master Khan}{LE3_4tKVTzuYRSKnH0T9Xg}{}{}, added {{Master Khan}{LE3_4tKVTzuYRSKnH0T9Xg}{}{},}, reason: zen-disco-receive(from master [{Master Khan}{LE3_4tKVTzuYRSKnH0T9Xg}{}{}])
[2016-12-21 06:47:47,135][INFO ][http ] [Master Khan] publish_address {}, bound_addresses {[::]:9200}
[2016-12-21 06:47:47,135][INFO ][node ] [Master Khan] started

Since there is nothing in the logs indicating a clean shutdown, I suspect something like either a JVM crash or the OS OOM killer. For the former, check the working directory for Elasticsearch for the presence of an hs_err_<pid>.log file. For the latter, check your OS logs like /var/log/messages and other similar infrastructure.

Thank you very much.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.