Elasticsearch gets shut down automatically

Elastic Stack Elasticsearch
1

I am running elasticsearch on a 2GB RAM server. But sometimes the elasticsearch server automatically gets shuts down. How can I sort this problem?

2

What does your node configuration look like? Is there anything in the logs? Which version of Elasticsearch are you using? What operating system?

3

I am using Elasticsearch 1.7.5 on Ubuntu 16.04.2. I have not enabled logging manually. Is there any pre-enabled logging system? How can I check the logs if there is?

4

The location of the log directory will depend on how you installed it, but should be specified in the elasticsearch.yml file.

How much data do you have in the cluster? How many indices and shards? What is the heap size set to?

5

Everything is set to default. I have not created clusters manually. I am using elasticsearch in a Django project for indexing with django-haystack as front-end for querying in the indexes.

1 Like
6

How much data do you have in the cluster? How many indices and shards? You can check this with the cat indices API and cat nodes API.

7

I get the following single row when I run the command curl -XGET 'localhost:9200/_cat/indices/' :

yellow open haystack 5 1 3693 26 19.1mb 19.1mb

8

And the following row when I run curl -XGET 'localhost:9200/_cat/nodes/ :

Thakurani 127.0.1.1 12 86 0.02 d * Zero-G
9

That does not look like a lot of data that would cause an OOM, even given the small heap. You need to have a look in the logs around the times it went down to see if there are any clues.

10

I am posting the contents of the log file on one of the days the server got shut down. Can you please check through the log if there is a hint.

[2017-07-13 07:14:07,304][INFO ][node ] [Taurus] version[1.7.5], pid[973], build[00f95f4/2016-02-02T09:55:30Z]
[2017-07-13 07:14:07,306][INFO ][node ] [Taurus] initializing ...
[2017-07-13 07:14:07,404][INFO ][plugins ] [Taurus] loaded , sites
[2017-07-13 07:14:07,455][INFO ][env ] [Taurus] using [1] data paths, mounts [[/ (/dev/vda1)]], net usable_space [23.1gb], net total_space [39.2gb], types [ext4]
[2017-07-13 07:14:10,193][INFO ][node ] [Taurus] initialized
[2017-07-13 07:14:10,193][INFO ][node ] [Taurus] starting ...
[2017-07-13 07:14:10,369][INFO ][transport ] [Taurus] bound_address {inet[/127.0.0.1:9300]}, publish_address {inet[localhost/127.0.0.1:9300]}
[2017-07-13 07:14:10,401][INFO ][discovery ] [Taurus] elasticsearch/W9-u8JbdRxe1rTXMpoeimA
[2017-07-13 07:14:14,204][INFO ][cluster.service ] [Taurus] new_master [Taurus][W9-u8JbdRxe1rTXMpoeimA][Thakurani][inet[localhost/127.0.0.1:9300]], reason: zen-disco-join (elected_as_master)
[2017-07-13 07:14:14,263][INFO ][http ] [Taurus] bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[localhost/127.0.0.1:9200]}
[2017-07-13 07:14:14,264][INFO ][node ] [Taurus] started
[2017-07-13 07:14:14,373][INFO ][gateway ] [Taurus] recovered [1] indices into cluster_state
[2017-07-13 07:14:22,917][INFO ][node ] [Taurus] stopping ...
[2017-07-13 07:14:23,007][INFO ][node ] [Taurus] stopped
[2017-07-13 07:14:23,007][INFO ][node ] [Taurus] closing ...
[2017-07-13 07:14:23,012][INFO ][node ] [Taurus] closed
[2017-07-13 07:14:23,755][INFO ][node ] [Suicide] version[1.7.5], pid[1050], build[00f95f4/2016-02-02T09:55:30Z]
[2017-07-13 07:14:23,756][INFO ][node ] [Suicide] initializing ...
[2017-07-13 07:14:23,851][INFO ][plugins ] [Suicide] loaded , sites
[2017-07-13 07:14:23,894][INFO ][env ] [Suicide] using [1] data paths, mounts [[/ (/dev/vda1)]], net usable_space [23.1gb], net total_space [39.2gb], types [ext4]
[2017-07-13 07:14:26,490][INFO ][node ] [Suicide] initialized
[2017-07-13 07:14:26,491][INFO ][node ] [Suicide] starting ...
[2017-07-13 07:14:26,744][INFO ][transport ] [Suicide] bound_address {inet[/127.0.0.1:9300]}, publish_address {inet[localhost/127.0.0.1:9300]}
[2017-07-13 07:14:26,761][INFO ][discovery ] [Suicide] elasticsearch/XXx72m_LS3acX1jDDRZlyA
[2017-07-13 07:14:30,536][INFO ][cluster.service ] [Suicide] new_master [Suicide][XXx72m_LS3acX1jDDRZlyA][Thakurani][inet[localhost/127.0.0.1:9300]], reason: zen-disco-join (elected_as_master)
[2017-07-13 07:14:30,567][INFO ][http ] [Suicide] bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[localhost/127.0.0.1:9200]}
[2017-07-13 07:14:30,568][INFO ][node ] [Suicide] started
[2017-07-13 07:14:30,616][INFO ][gateway ] [Suicide] recovered [1] indices into cluster_state
[2017-07-13 07:14:57,212][INFO ][node ] [Suicide] stopping ...
[2017-07-13 07:14:57,309][INFO ][node ] [Suicide] stopped
[2017-07-13 07:14:57,310][INFO ][node ] [Suicide] closing ...
[2017-07-13 07:14:57,318][INFO ][node ] [Suicide] closed
[2017-07-13 07:14:57,947][INFO ][node ] [Michael Twoyoungmen] version[1.7.5], pid[1138], build[00f95f4/2016-02-02T09:55:30Z]
[2017-07-13 07:14:57,948][INFO ][node ] [Michael Twoyoungmen] initializing ...
[2017-07-13 07:14:58,030][INFO ][plugins ] [Michael Twoyoungmen] loaded , sites
[2017-07-13 07:14:58,071][INFO ][env ] [Michael Twoyoungmen] using [1] data paths, mounts [[/ (/dev/vda1)]], net usable_space [23.1gb], net total_space [39.2gb], types [ext4]
[2017-07-13 07:15:00,925][INFO ][node ] [Michael Twoyoungmen] initialized
[2017-07-13 07:15:00,926][INFO ][node ] [Michael Twoyoungmen] starting ...
[2017-07-13 07:15:01,158][INFO ][transport ] [Michael Twoyoungmen] bound_address {inet[/127.0.0.1:9300]}, publish_address {inet[localhost/127.0.0.1:9300]}
[2017-07-13 07:15:01,176][INFO ][discovery ] [Michael Twoyoungmen] elasticsearch/Lw_Y9BlrTa6wmCJYKBO7-w
[2017-07-13 07:15:04,962][INFO ][cluster.service ] [Michael Twoyoungmen] new_master [Michael Twoyoungmen][Lw_Y9BlrTa6wmCJYKBO7-w][Thakurani][inet[localhost/127.0.0.1:9300]], reason: zen-disco-join (elected_as_master)
[2017-07-13 07:15:04,989][INFO ][http ] [Michael Twoyoungmen] bound_address {inet[/127.0.0.1:9200]}, publish_address {inet[localhost/127.0.0.1:9200]}
[2017-07-13 07:15:04,990][INFO ][node ] [Michael Twoyoungmen] started
[2017-07-13 07:15:05,107][INFO ][gateway ] [Michael Twoyoungmen] recovered [1] indices into cluster_state

11

There are no errors or anything indicating a crash, so it looks like it may have been deliberately restarted, but I can not tell for sure.

12

There are a few rows with the word [Suicide]. Do these rows give a hint to the problem?

13

No, that is the name of the node, which gets randomly assigned on startup in Elasticsearch 1.7.

14

Okay. Can increasing the RAM size from 2 GB to more be of help? The server is not dedicated to Elasticsearch only. We are running a full website on the server with elasticsearch as a service.

15

I don't see any indication that memory in Elasticsearch is a problem. If other applications share the host, their memory usage may have had an impact and cause d the OS to shut down the node. You may need to look for that kind of information in the system logs.

16

Okay.

17

Given the low amount of data in the cluster, you might also be able to reduce the heap Elasticsearch uses from 1GB (believe this is the default) to 512MB.

18

Okay

19

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.