# Elasticsearch 5.1.1 keep dying after 20 minutes **URL:** \ **Category:** Elasticsearch\ **Created:** [January 3, 2017, 3:58am UTC](https://discuss.elastic.co/t/elasticsearch-5-1-1-keep-dying-after-20-minutes/70384 "2017-01-03T03:58:10Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![nathan.tivaci](https://avatars.discourse-cdn.com/v4/letter/n/c89c15/32.png) [@nathan.tivaci](https://discuss.elastic.co/u/nathan.tivaci)\ **Post date:** [January 3, 2017, 3:58am UTC](https://discuss.elastic.co/t/elasticsearch-5-1-1-keep-dying-after-20-minutes/70384/1 "2017-01-03T03:58:10Z")
Hi there, I hope somebody can help me. I have an instance of ELK in Ubuntu 16.04 (2GB of RAM, 30GB of HDD). I can setup visualisation, dashboard and all. BUT, the elasticsearch instance keep dying after 20 minutes or so (Around 6000 row of input) I've tried to add the memlock on `/etc/security/limits.conf` ```auto elasticsearch soft memlock unlimited elasticsearch hard memlock unlimited ``` still no luck. Anyone can help me where can I start debugging things out? I've tried checking `/var/log/elasticsearch/elasticsearch.log` but nothing much on the "dying" part, I can only see that it started. Cheers, Nathan ---
**Author:** ![JKhondhu](https://avatars.discourse-cdn.com/v4/letter/j/ed655f/32.png) [@JKhondhu](https://discuss.elastic.co/u/JKhondhu)\ **Post date:** [January 3, 2017, 2:25pm UTC](https://discuss.elastic.co/t/elasticsearch-5-1-1-keep-dying-after-20-minutes/70384/2 "2017-01-03T14:25:15Z")
Hi, How much of that 2GB RAM is allocated in the /etc/elasticsearch/jvm.options? ..elasticsearch instance keep dying.. What exactly are you experiencing on the operating system itself? Cheers, ---
**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\ **Post date:** [January 3, 2017, 2:27pm UTC](https://discuss.elastic.co/t/elasticsearch-5-1-1-keep-dying-after-20-minutes/70384/3 "2017-01-03T14:27:12Z")
By dying you mean the process is killed or unresponsive? Does it only fail when you are feeding it new docs? Are you using any unusual plugins? (e.g. I remember reading Zookeeper can call System.exit when unhappy). ---
**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\ **Post date:** [January 3, 2017, 5:37pm UTC](https://discuss.elastic.co/t/elasticsearch-5-1-1-keep-dying-after-20-minutes/70384/4 "2017-01-03T17:37:37Z")
Plugins can not call `System#exit`, we have the permissions for that [locked down](https://github.com/elastic/securesm/commit/db51630867fc5f59c05f34706cf7a345e70d0b84) now: > > ---
**Author:** ![nathan.tivaci](https://avatars.discourse-cdn.com/v4/letter/n/c89c15/32.png) [@nathan.tivaci](https://discuss.elastic.co/u/nathan.tivaci)\ **Post date:** [January 4, 2017, 3:37am UTC](https://discuss.elastic.co/t/elasticsearch-5-1-1-keep-dying-after-20-minutes/70384/5 "2017-01-04T03:37:15Z")
> [@JKhondhu](#): > > Hi, How much of that 2GB RAM is allocated in the /etc/elasticsearch/jvm.options? ..elasticsearch instance keep dying.. What exactly are you experiencing on the operating system itself? > > Cheers, Hi Jymit, I've checked the `/etc/elasticsearch/jvm.options` it shown: ```auto -Xms1g -Xmx1g ``` is it enough? The process of elasticsearch just stopped running after a few minutes (15-20minutes). The other processes like Kibana, Nginx, Logstash is still running fine. Here is the screenshot of the Kibana: ![](https://us1.discourse-cdn.com/elastic/original/2X/c/c38182d318a990898b31a40b1c3eb3b85272a471.png) ---
**Author:** ![nathan.tivaci](https://avatars.discourse-cdn.com/v4/letter/n/c89c15/32.png) [@nathan.tivaci](https://discuss.elastic.co/u/nathan.tivaci)\ **Post date:** [January 4, 2017, 3:59am UTC](https://discuss.elastic.co/t/elasticsearch-5-1-1-keep-dying-after-20-minutes/70384/6 "2017-01-04T03:59:13Z")
> [@Mark\_Harwood](#): > > By dying you mean the process is killed or unresponsive? > Does it only fail when you are feeding it new docs? > Are you using any unusual plugins? (e.g. I remember reading Zookeeper can call System.exit when unhappy). Hi Mark, CMIIW, but it seems to be killed: ```auto ● elasticsearch.service - Elasticsearch Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendo Active: failed (Result: signal) since Tue 2017-01-03 05:01:55 UTC; 22h ago Docs: http://www.elastic.co Process: 1461 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DI Process: 1422 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd- Main PID: 1461 (code=killed, signal=KILL) ``` Yes, it only fails when I am feeding new docs. The plugin I have is just `Timelion` which was installed by default. ---
**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\ **Post date:** [January 4, 2017, 4:25am UTC](https://discuss.elastic.co/t/elasticsearch-5-1-1-keep-dying-after-20-minutes/70384/7 "2017-01-04T04:25:11Z")
> [@nathan.tivaci](#): > > 2GB of RAM > [@nathan.tivaci](#): > > I've checked the /etc/elasticsearch/jvm.options it shown: > > ```auto > -Xms1g > > ``` > [@nathan.tivaci](#): > > The process of elasticsearch just stopped running after a few minutes (15-20minutes). The other processes like Kibana, Nginx, Logstash is still running fine. > [@nathan.tivaci](#): > > ```auto > Main PID: 1461 (code=killed, signal=KILL) > > ``` Since you're running Elasticsearch with a 1 GB heap on a machine with 2 GB of RAM, I suspect that your instance is being killed by the OS OOM killer. Check your kernel logs. ---
**Author:** ![nathan.tivaci](https://avatars.discourse-cdn.com/v4/letter/n/c89c15/32.png) [@nathan.tivaci](https://discuss.elastic.co/u/nathan.tivaci)\ **Post date:** [January 4, 2017, 5:25am UTC](https://discuss.elastic.co/t/elasticsearch-5-1-1-keep-dying-after-20-minutes/70384/8 "2017-01-04T05:25:16Z")
> [@jasontedor](#): > > Since you're running Elasticsearch with a 1 GB heap on a machine with 2 GB of RAM, I suspect that you're instance is being killed by the OS OOM killer. Check your kernel logs. Hi Jason, I checked my `/var/log/kern.log`: ```auto [1295.629750] node invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0 [1295.629876] [] oom_kill_process+0x202/0x3c0 [1295.630149] Out of memory: Kill process 1461 (java) score 679 or sacrifice child [1295.631466] Killed process 1461 (java) total-vm:3642564kB, anon-rss:1367924kB, file-rss:21488kB ``` Looks like you are right. Do you have any suggestion on what should I do? Decreasing the heap or increasing my RAM? Cheers! ---
**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\ **Post date:** [January 4, 2017, 5:45am UTC](https://discuss.elastic.co/t/elasticsearch-5-1-1-keep-dying-after-20-minutes/70384/9 "2017-01-04T05:45:12Z")
The immediate problem is running Elasticsearch, Logstash, Kibana, and nginx in a machine with 2 GB of RAM. Even if you drop the heap by half you're likely to still have trouble, and then you're more likely to run into heap space issues in Elasticsearch. I think you need to either get sone of those other processes off this host, or get more RAM. ---
**Author:** ![nathan.tivaci](https://avatars.discourse-cdn.com/v4/letter/n/c89c15/32.png) [@nathan.tivaci](https://discuss.elastic.co/u/nathan.tivaci)\ **Post date:** [January 4, 2017, 6:10am UTC](https://discuss.elastic.co/t/elasticsearch-5-1-1-keep-dying-after-20-minutes/70384/10 "2017-01-04T06:10:32Z")
Hi Jason, I am going to bump it to a 4GB RAM, i hope this will help. Would you know what is the normal RAM for ELK stack in a machine? Cheers! ---
**Author:** ![JKhondhu](https://avatars.discourse-cdn.com/v4/letter/j/ed655f/32.png) [@JKhondhu](https://discuss.elastic.co/u/JKhondhu)\ **Post date:** [January 4, 2017, 9:54am UTC](https://discuss.elastic.co/t/elasticsearch-5-1-1-keep-dying-after-20-minutes/70384/11 "2017-01-04T09:54:28Z")
> [@nathan.tivaci](#): > > Would you know what is the normal RAM for ELK stack in a machine? Hi, this would very much depend on what you are expecting use this server for. If this is a server purpose built for testing 5.1.1 then of course the resources you will look to have may suffice. This all goes hand in hand with what you are looking to achieve here. ---
**Author:** ![nathan.tivaci](https://avatars.discourse-cdn.com/v4/letter/n/c89c15/32.png) [@nathan.tivaci](https://discuss.elastic.co/u/nathan.tivaci)\ **Post date:** [January 5, 2017, 1:16am UTC](https://discuss.elastic.co/t/elasticsearch-5-1-1-keep-dying-after-20-minutes/70384/12 "2017-01-05T01:16:00Z")
> [@JKhondhu](#): > > Hi, this would very much depend on what you are expecting use this server for. > If this is a server purpose built for testing 5.1.1 then of course the resources you will look to have may suffice. This all goes hand in hand with what you are looking to achieve here. Hi Jymit, In fact, I was thinking to make this a production ELK server. How do you guys normally judge the server requirement for ELK? based on number of docs coming in or? Cheers! ---
**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\ **Post date:** [January 5, 2017, 2:46am UTC](https://discuss.elastic.co/t/elasticsearch-5-1-1-keep-dying-after-20-minutes/70384/13 "2017-01-05T02:46:23Z")
Running all three on a single machine with only 4 GB might be too much, especially combined with an nginx server (it really depends on your use-case though). Elasticsearch loves the filesystem cache, but if all the memory not dedicated to the Elasticsearch heap is going to other processes, there is not going to be any room left over for the filesystem cache. ---
**Author:** ![nathan.tivaci](https://avatars.discourse-cdn.com/v4/letter/n/c89c15/32.png) [@nathan.tivaci](https://discuss.elastic.co/u/nathan.tivaci)\ **Post date:** [January 9, 2017, 5:05am UTC](https://discuss.elastic.co/t/elasticsearch-5-1-1-keep-dying-after-20-minutes/70384/14 "2017-01-09T05:05:47Z")
> [@jasontedor](#): > > Running all three on a single machine with only 4 GB might be too much, especially combined with an nginx server (it really depends on your use-case though). Elasticsearch loves the filesystem cache, but if all the memory not dedicated to the Elasticsearch heap is going to other processes, there is not going to be any room left over for the filesystem cache. Hello Jason, I've just trying to reduce the xms to 750mb and fortunately (finger crossed) the server has been running fine for a few days now. I am feeding it around 40k hits every 15mins or so. Thank you for your help! ---
**Author:** ![jasontedor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jasontedor/32/66992_2.png) [@jasontedor](https://discuss.elastic.co/u/jasontedor)\ **Post date:** [January 9, 2017, 12:08pm UTC](https://discuss.elastic.co/t/elasticsearch-5-1-1-keep-dying-after-20-minutes/70384/15 "2017-01-09T12:08:17Z")
> [@nathan.tivaci](#): > > Thank you for your help! You're very welcome. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 6, 2017, 12:08pm UTC](https://discuss.elastic.co/t/elasticsearch-5-1-1-keep-dying-after-20-minutes/70384/16 "2017-02-06T12:08:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.