### Author:  [@nik9000](https://discuss.elastic.co/u/nik9000)
#### Post date: [July 23, 2014, 3:11pm UTC](https://discuss.elastic.co/t/sudden-high-os-load-then-es-vm-disappears/18823/5 "2014-07-23T15:11:37Z")
On Wed, Jul 23, 2014 at 10:19 AM, [michael@modernmast.com](mailto:michael@modernmast.com) wrote:
> Looking at the JVM GC graphs, I do see some increases there, but not sure
> those are enough to cause this storm?
>
> [https://lh6.googleusercontent.com/-4wVrdN5UNRY/U8\_DuSsh15I/AAAAAAAAAAk/prHDyOwB\_gE/s1600/Screenshot+2014-07-23+10.05.10.png](https://lh6.googleusercontent.com/-4wVrdN5UNRY/U8_DuSsh15I/AAAAAAAAAAk/prHDyOwB_gE/s1600/Screenshot+2014-07-23+10.05.10.png)
That looks like it wasn't the problem.
> [https://lh6.googleusercontent.com/-4wVrdN5UNRY/U8\_DuSsh15I/AAAAAAAAAAk/prHDyOwB\_gE/s1600/Screenshot+2014-07-23+10.05.10.png](https://lh6.googleusercontent.com/-4wVrdN5UNRY/U8_DuSsh15I/AAAAAAAAAAk/prHDyOwB_gE/s1600/Screenshot+2014-07-23+10.05.10.png)
> The disk graphs in Marvel don't show anything out of the ordinary. I'm not
> sure how to check on those write\_bytes and read\_bytes... Where does ES
> report those? I'm using Google Compute Engine, and according to their
> minimal graphs, while there was a small spike in the disk I/Os, it wasn't
> anything insane.
Elasticsearch reports them in the same API as the number of reads. I
imagine they are right next to each other in marvel but I'm not sure.
> Right after the spike happened, the indexing rate spiked to 6.6k / second.
> Notice that the first tag notes the VM that left the cluster, the second
> tag shows the cluster went back to "green". Considering this happened after
> the node "left", does this give us any clues as for the reason?
>
> [https://lh3.googleusercontent.com/-8hmcRo-kH6k/U8\_D4QliadI/AAAAAAAAAAs/U69wNWbvbcA/s1600/Screenshot+2014-07-23+10.08.01.png](https://lh3.googleusercontent.com/-8hmcRo-kH6k/U8_D4QliadI/AAAAAAAAAAs/U69wNWbvbcA/s1600/Screenshot+2014-07-23+10.08.01.png)
>
> A few observations:
>
> - The es process is still running on the "dead machine". I can see it when
> I ssh into the VM (thru ps aux, and docker)
> - The VM doesn't show anything in the error logs, etc
> - Running a "ps aux" on that VM actually freezes (after showing the es
> process)
That seems pretty nasty. Does the node respond to stuff like `curl localhost:9200`? Sounds like the hardware/docker is just angry.
Nik
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0v%2B4iNmissyNcW50OmpD1yBVoJDTExCepawA1HDms-8w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0v%2B4iNmissyNcW50OmpD1yBVoJDTExCepawA1HDms-8w%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@michael\_4](https://discuss.elastic.co/u/michael_4)
#### Post date: [July 23, 2014, 4:15pm UTC](https://discuss.elastic.co/t/sudden-high-os-load-then-es-vm-disappears/18823/6 "2014-07-23T16:15:51Z")
No, the VM does not response to curl requests. Closest thing I found to
that read bytes in the API was the \_cluster/stats endpoint
--\> [GET \_cluster/stats · GitHub](https://gist.github.com/schonfeld/d45401e44f5961c38502)
Were you referring to a different endpoint?
What're your thoughts re "angry hardware"? Insufficient resources? Are
there any known issues with CoreOS + ES?
On Wednesday, July 23, 2014 11:12:10 AM UTC-4, Nikolas Everett wrote:
> On Wed, Jul 23, 2014 at 10:19 AM, \<[mic...@modernmast.com](mailto:mic...@modernmast.com) \\>
> wrote:
>
> > Looking at the JVM GC graphs, I do see some increases there, but not sure
> > those are enough to cause this storm?
> >
> > [https://lh6.googleusercontent.com/-4wVrdN5UNRY/U8\_DuSsh15I/AAAAAAAAAAk/prHDyOwB\_gE/s1600/Screenshot+2014-07-23+10.05.10.png](https://lh6.googleusercontent.com/-4wVrdN5UNRY/U8_DuSsh15I/AAAAAAAAAAk/prHDyOwB_gE/s1600/Screenshot+2014-07-23+10.05.10.png)
>
> That looks like it wasn't the problem.
>
> > [https://lh6.googleusercontent.com/-4wVrdN5UNRY/U8\_DuSsh15I/AAAAAAAAAAk/prHDyOwB\_gE/s1600/Screenshot+2014-07-23+10.05.10.png](https://lh6.googleusercontent.com/-4wVrdN5UNRY/U8_DuSsh15I/AAAAAAAAAAk/prHDyOwB_gE/s1600/Screenshot+2014-07-23+10.05.10.png)
> > The disk graphs in Marvel don't show anything out of the ordinary. I'm
> > not sure how to check on those write\_bytes and read\_bytes... Where does ES
> > report those? I'm using Google Compute Engine, and according to their
> > minimal graphs, while there was a small spike in the disk I/Os, it wasn't
> > anything insane.
>
> Elasticsearch reports them in the same API as the number of reads. I
> imagine they are right next to each other in marvel but I'm not sure.
>
> > Right after the spike happened, the indexing rate spiked to 6.6k /
> > second. Notice that the first tag notes the VM that left the cluster, the
> > second tag shows the cluster went back to "green". Considering this
> > happened after the node "left", does this give us any clues as for the
> > reason?
> >
> > [https://lh3.googleusercontent.com/-8hmcRo-kH6k/U8\_D4QliadI/AAAAAAAAAAs/U69wNWbvbcA/s1600/Screenshot+2014-07-23+10.08.01.png](https://lh3.googleusercontent.com/-8hmcRo-kH6k/U8_D4QliadI/AAAAAAAAAAs/U69wNWbvbcA/s1600/Screenshot+2014-07-23+10.08.01.png)
> >
> > A few observations:
> >
> > - The es process is still running on the "dead machine". I can see it
> > when I ssh into the VM (thru ps aux, and docker)
> > - The VM doesn't show anything in the error logs, etc
> > - Running a "ps aux" on that VM actually freezes (after showing the es
> > process)
>
> That seems pretty nasty. Does the node respond to stuff like `curl localhost:9200`? Sounds like the hardware/docker is just angry.
>
> Nik
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1243b181-3ec1-410c-8091-d561ad2b8bed%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1243b181-3ec1-410c-8091-d561ad2b8bed%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Daniel\_Schonfeld](https://discuss.elastic.co/u/Daniel_Schonfeld)
#### Post date: [July 23, 2014, 4:49pm UTC](https://discuss.elastic.co/t/sudden-high-os-load-then-es-vm-disappears/18823/7 "2014-07-23T16:49:31Z")
I wanted to add another five cents worth to what Michael already described.
Before, when this happened, we used to run the docker container without a
memory limit and giving ES a HEAP\_SIZE of 10GB (10240M to be exact). When
it happened the first time, doing `free -m` revealed that the system was
left with barely a few hundreds of megabytes left. Second time around we
figured we'd tweak it a bit. Give the docker container 10G memory limit
and limit the ES\_HEAP\_SIZE to 7.5GB (7168M). This time when it crashed and
is still hung as we speak (been trying to get a thread dump, but that's not
happening... jstack can't even connect to that process its so deeply hung)
doing 'top' revealed that the memory usage is at 9.9GB for that process.
Basically the system seems bogged out of memory.
Question is... how did it exceed the heap size given? And more to the
point, just how much memory should we be allocating? If this is just a
problem of shortage of memory, spawning up a new machine with a bunch more
RAM is no problem. But we just wish we weren't shooting in the dark.
We've already tried increasing the SSD size to get more IOPS (GCE ties
size to IOPS bandwidth).
Last but not least, is there something we should be doing to tweak the
lucene segment size?
Thanks for all your thoughts!
On Wednesday, July 23, 2014 12:15:51 PM UTC-4, [mic...@modernmast.com](mailto:mic...@modernmast.com) wrote:
> No, the VM does not response to curl requests. Closest thing I found to
> that read bytes in the API was the \_cluster/stats endpoint --\>
> [GET \_cluster/stats · GitHub](https://gist.github.com/schonfeld/d45401e44f5961c38502)
>
> Were you referring to a different endpoint?
>
> What're your thoughts re "angry hardware"? Insufficient resources? Are
> there any known issues with CoreOS + ES?
>
> On Wednesday, July 23, 2014 11:12:10 AM UTC-4, Nikolas Everett wrote:
>
> > On Wed, Jul 23, 2014 at 10:19 AM, [mic...@modernmast.com](mailto:mic...@modernmast.com) wrote:
> >
> > > Looking at the JVM GC graphs, I do see some increases there, but not
> > > sure those are enough to cause this storm?
> > >
> > > [https://lh6.googleusercontent.com/-4wVrdN5UNRY/U8\_DuSsh15I/AAAAAAAAAAk/prHDyOwB\_gE/s1600/Screenshot+2014-07-23+10.05.10.png](https://lh6.googleusercontent.com/-4wVrdN5UNRY/U8_DuSsh15I/AAAAAAAAAAk/prHDyOwB_gE/s1600/Screenshot+2014-07-23+10.05.10.png)
> >
> > That looks like it wasn't the problem.
> >
> > > [https://lh6.googleusercontent.com/-4wVrdN5UNRY/U8\_DuSsh15I/AAAAAAAAAAk/prHDyOwB\_gE/s1600/Screenshot+2014-07-23+10.05.10.png](https://lh6.googleusercontent.com/-4wVrdN5UNRY/U8_DuSsh15I/AAAAAAAAAAk/prHDyOwB_gE/s1600/Screenshot+2014-07-23+10.05.10.png)
> > > The disk graphs in Marvel don't show anything out of the ordinary. I'm
> > > not sure how to check on those write\_bytes and read\_bytes... Where does ES
> > > report those? I'm using Google Compute Engine, and according to their
> > > minimal graphs, while there was a small spike in the disk I/Os, it wasn't
> > > anything insane.
> >
> > Elasticsearch reports them in the same API as the number of reads. I
> > imagine they are right next to each other in marvel but I'm not sure.
> >
> > > Right after the spike happened, the indexing rate spiked to 6.6k /
> > > second. Notice that the first tag notes the VM that left the cluster, the
> > > second tag shows the cluster went back to "green". Considering this
> > > happened after the node "left", does this give us any clues as for the
> > > reason?
> > >
> > > [https://lh3.googleusercontent.com/-8hmcRo-kH6k/U8\_D4QliadI/AAAAAAAAAAs/U69wNWbvbcA/s1600/Screenshot+2014-07-23+10.08.01.png](https://lh3.googleusercontent.com/-8hmcRo-kH6k/U8_D4QliadI/AAAAAAAAAAs/U69wNWbvbcA/s1600/Screenshot+2014-07-23+10.08.01.png)
> > >
> > > A few observations:
> > >
> > > - The es process is still running on the "dead machine". I can see it
> > > when I ssh into the VM (thru ps aux, and docker)
> > > - The VM doesn't show anything in the error logs, etc
> > > - Running a "ps aux" on that VM actually freezes (after showing the es
> > > process)
> >
> > That seems pretty nasty. Does the node respond to stuff like `curl localhost:9200`? Sounds like the hardware/docker is just angry.
> >
> > Nik
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/918f6972-60be-4bff-81e6-939f4840cde0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/918f6972-60be-4bff-81e6-939f4840cde0%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Daniel\_Schonfeld](https://discuss.elastic.co/u/Daniel_Schonfeld)
#### Post date: [July 23, 2014, 4:56pm UTC](https://discuss.elastic.co/t/sudden-high-os-load-then-es-vm-disappears/18823/8 "2014-07-23T16:56:54Z")
Here's TOP and df -h... that's as best as I can get for now from inside
that container.
[https://gist.github.com/danielschonfeld/d75c43cce34a16a57926](https://gist.github.com/danielschonfeld/d75c43cce34a16a57926)
On Wednesday, July 23, 2014 12:49:32 PM UTC-4, Daniel Schonfeld wrote:
> I wanted to add another five cents worth to what Michael already described.
>
> Before, when this happened, we used to run the docker container without a
> memory limit and giving ES a HEAP\_SIZE of 10GB (10240M to be exact). When
> it happened the first time, doing `free -m` revealed that the system was
> left with barely a few hundreds of megabytes left. Second time around we
> figured we'd tweak it a bit. Give the docker container 10G memory limit
> and limit the ES\_HEAP\_SIZE to 7.5GB (7168M). This time when it crashed and
> is still hung as we speak (been trying to get a thread dump, but that's not
> happening... jstack can't even connect to that process its so deeply hung)
> doing 'top' revealed that the memory usage is at 9.9GB for that process.
> Basically the system seems bogged out of memory.
>
> Question is... how did it exceed the heap size given? And more to the
> point, just how much memory should we be allocating? If this is just a
> problem of shortage of memory, spawning up a new machine with a bunch more
> RAM is no problem. But we just wish we weren't shooting in the dark.
> We've already tried increasing the SSD size to get more IOPS (GCE ties
> size to IOPS bandwidth).
>
> Last but not least, is there something we should be doing to tweak the
> lucene segment size?
>
> Thanks for all your thoughts!
>
> On Wednesday, July 23, 2014 12:15:51 PM UTC-4, [mic...@modernmast.com](mailto:mic...@modernmast.com)
> wrote:
>
> > No, the VM does not response to curl requests. Closest thing I found to
> > that read bytes in the API was the \_cluster/stats endpoint --\>
> > [GET \_cluster/stats · GitHub](https://gist.github.com/schonfeld/d45401e44f5961c38502)
> >
> > Were you referring to a different endpoint?
> >
> > What're your thoughts re "angry hardware"? Insufficient resources? Are
> > there any known issues with CoreOS + ES?
> >
> > On Wednesday, July 23, 2014 11:12:10 AM UTC-4, Nikolas Everett wrote:
> >
> > > On Wed, Jul 23, 2014 at 10:19 AM, [mic...@modernmast.com](mailto:mic...@modernmast.com) wrote:
> > >
> > > > Looking at the JVM GC graphs, I do see some increases there, but not
> > > > sure those are enough to cause this storm?
> > > >
> > > > [https://lh6.googleusercontent.com/-4wVrdN5UNRY/U8\_DuSsh15I/AAAAAAAAAAk/prHDyOwB\_gE/s1600/Screenshot+2014-07-23+10.05.10.png](https://lh6.googleusercontent.com/-4wVrdN5UNRY/U8_DuSsh15I/AAAAAAAAAAk/prHDyOwB_gE/s1600/Screenshot+2014-07-23+10.05.10.png)
> > >
> > > That looks like it wasn't the problem.
> > >
> > > > [https://lh6.googleusercontent.com/-4wVrdN5UNRY/U8\_DuSsh15I/AAAAAAAAAAk/prHDyOwB\_gE/s1600/Screenshot+2014-07-23+10.05.10.png](https://lh6.googleusercontent.com/-4wVrdN5UNRY/U8_DuSsh15I/AAAAAAAAAAk/prHDyOwB_gE/s1600/Screenshot+2014-07-23+10.05.10.png)
> > > > The disk graphs in Marvel don't show anything out of the ordinary. I'm
> > > > not sure how to check on those write\_bytes and read\_bytes... Where does ES
> > > > report those? I'm using Google Compute Engine, and according to their
> > > > minimal graphs, while there was a small spike in the disk I/Os, it wasn't
> > > > anything insane.
> > >
> > > Elasticsearch reports them in the same API as the number of reads. I
> > > imagine they are right next to each other in marvel but I'm not sure.
> > >
> > > > Right after the spike happened, the indexing rate spiked to 6.6k /
> > > > second. Notice that the first tag notes the VM that left the cluster, the
> > > > second tag shows the cluster went back to "green". Considering this
> > > > happened after the node "left", does this give us any clues as for the
> > > > reason?
> > > >
> > > > [https://lh3.googleusercontent.com/-8hmcRo-kH6k/U8\_D4QliadI/AAAAAAAAAAs/U69wNWbvbcA/s1600/Screenshot+2014-07-23+10.08.01.png](https://lh3.googleusercontent.com/-8hmcRo-kH6k/U8_D4QliadI/AAAAAAAAAAs/U69wNWbvbcA/s1600/Screenshot+2014-07-23+10.08.01.png)
> > > >
> > > > A few observations:
> > > >
> > > > - The es process is still running on the "dead machine". I can see it
> > > > when I ssh into the VM (thru ps aux, and docker)
> > > > - The VM doesn't show anything in the error logs, etc
> > > > - Running a "ps aux" on that VM actually freezes (after showing the es
> > > > process)
> > >
> > > That seems pretty nasty. Does the node respond to stuff like `curl localhost:9200`? Sounds like the hardware/docker is just angry.
> > >
> > > Nik
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/11474c65-4443-4851-b109-75f092b49d04%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/11474c65-4443-4851-b109-75f092b49d04%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@nik9000](https://discuss.elastic.co/u/nik9000)
#### Post date: [July 23, 2014, 5:25pm UTC](https://discuss.elastic.co/t/sudden-high-os-load-then-es-vm-disappears/18823/9 "2014-07-23T17:25:33Z")
Heap size isn't total memory size. Its size for java to allocate stuff.
There are tons of other memory costs but the rule of thumb is to set heap
to no more then 30GB and around half of physical memory. I imagine docker
is complicating things.
I'm not sure what docker does with memory mapped files, for instance.
Elasticsearch uses tons of them so it uses tons of virtual memory. But
that shouldn't be a problem because its virtual. Though I don't know what
docker thinks of that.
On Wed, Jul 23, 2014 at 12:56 PM, Daniel Schonfeld [downwindabeam@gmail.com](mailto:downwindabeam@gmail.com)
wrote:
> Here's TOP and df -h... that's as best as I can get for now from inside
> that container.
>
> [https://gist.github.com/danielschonfeld/d75c43cce34a16a57926](https://gist.github.com/danielschonfeld/d75c43cce34a16a57926)
>
> On Wednesday, July 23, 2014 12:49:32 PM UTC-4, Daniel Schonfeld wrote:
>
> > I wanted to add another five cents worth to what Michael already
> > described.
> >
> > Before, when this happened, we used to run the docker container without a
> > memory limit and giving ES a HEAP\_SIZE of 10GB (10240M to be exact). When
> > it happened the first time, doing `free -m` revealed that the system was
> > left with barely a few hundreds of megabytes left. Second time around we
> > figured we'd tweak it a bit. Give the docker container 10G memory limit
> > and limit the ES\_HEAP\_SIZE to 7.5GB (7168M). This time when it crashed and
> > is still hung as we speak (been trying to get a thread dump, but that's not
> > happening... jstack can't even connect to that process its so deeply hung)
> > doing 'top' revealed that the memory usage is at 9.9GB for that process.
> > Basically the system seems bogged out of memory.
> >
> > Question is... how did it exceed the heap size given? And more to the
> > point, just how much memory should we be allocating? If this is just a
> > problem of shortage of memory, spawning up a new machine with a bunch more
> > RAM is no problem. But we just wish we weren't shooting in the dark.
> > We've already tried increasing the SSD size to get more IOPS (GCE ties
> > size to IOPS bandwidth).
> >
> > Last but not least, is there something we should be doing to tweak the
> > lucene segment size?
> >
> > Thanks for all your thoughts!
> >
> > On Wednesday, July 23, 2014 12:15:51 PM UTC-4, [mic...@modernmast.com](mailto:mic...@modernmast.com)
> > wrote:
> >
> > > No, the VM does not response to curl requests. Closest thing I found to
> > > that read bytes in the API was the \_cluster/stats endpoint --\>
> > > [GET \_cluster/stats · GitHub](https://gist.github.com/schonfeld/d45401e44f5961c38502)
> > >
> > > Were you referring to a different endpoint?
> > >
> > > What're your thoughts re "angry hardware"? Insufficient resources? Are
> > > there any known issues with CoreOS + ES?
> > >
> > > On Wednesday, July 23, 2014 11:12:10 AM UTC-4, Nikolas Everett wrote:
> > >
> > > > On Wed, Jul 23, 2014 at 10:19 AM, [mic...@modernmast.com](mailto:mic...@modernmast.com) wrote:
> > > >
> > > > > Looking at the JVM GC graphs, I do see some increases there, but not
> > > > > sure those are enough to cause this storm?
> > > > >
> > > > > [https://lh6.googleusercontent.com/-4wVrdN5UNRY/U8\_DuSsh15I/AAAAAAAAAAk/prHDyOwB\_gE/s1600/Screenshot+2014-07-23+10.05.10.png](https://lh6.googleusercontent.com/-4wVrdN5UNRY/U8_DuSsh15I/AAAAAAAAAAk/prHDyOwB_gE/s1600/Screenshot+2014-07-23+10.05.10.png)
> > > >
> > > > That looks like it wasn't the problem.
> > > >
> > > > > [https://lh6.googleusercontent.com/-4wVrdN5UNRY/U8\_DuSsh15I/AAAAAAAAAAk/prHDyOwB\_gE/s1600/Screenshot+2014-07-23+10.05.10.png](https://lh6.googleusercontent.com/-4wVrdN5UNRY/U8_DuSsh15I/AAAAAAAAAAk/prHDyOwB_gE/s1600/Screenshot+2014-07-23+10.05.10.png)
> > > > > The disk graphs in Marvel don't show anything out of the ordinary. I'm
> > > > > not sure how to check on those write\_bytes and read\_bytes... Where does ES
> > > > > report those? I'm using Google Compute Engine, and according to their
> > > > > minimal graphs, while there was a small spike in the disk I/Os, it wasn't
> > > > > anything insane.
> > > >
> > > > Elasticsearch reports them in the same API as the number of reads. I
> > > > imagine they are right next to each other in marvel but I'm not sure.
> > > >
> > > > > Right after the spike happened, the indexing rate spiked to 6.6k /
> > > > > second. Notice that the first tag notes the VM that left the cluster, the
> > > > > second tag shows the cluster went back to "green". Considering this
> > > > > happened after the node "left", does this give us any clues as for the
> > > > > reason?
> > > > >
> > > > > [https://lh3.googleusercontent.com/-8hmcRo-kH6k/U8\_D4QliadI/AAAAAAAAAAs/U69wNWbvbcA/s1600/Screenshot+2014-07-23+10.08.01.png](https://lh3.googleusercontent.com/-8hmcRo-kH6k/U8_D4QliadI/AAAAAAAAAAs/U69wNWbvbcA/s1600/Screenshot+2014-07-23+10.08.01.png)
> > > > >
> > > > > A few observations:
> > > > >
> > > > > - The es process is still running on the "dead machine". I can see it
> > > > > when I ssh into the VM (thru ps aux, and docker)
> > > > > - The VM doesn't show anything in the error logs, etc
> > > > > - Running a "ps aux" on that VM actually freezes (after showing the es
> > > > > process)
> > > >
> > > > That seems pretty nasty. Does the node respond to stuff like `curl localhost:9200`? Sounds like the hardware/docker is just angry.
> > > >
> > > > Nik
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an
> > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/11474c65-4443-4851-b109-75f092b49d04%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/11474c65-4443-4851-b109-75f092b49d04%40googlegroups.com)
> > > [https://groups.google.com/d/msgid/elasticsearch/11474c65-4443-4851-b109-75f092b49d04%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/11474c65-4443-4851-b109-75f092b49d04%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > .
>
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2-nhicQa0gwtY%2BmCO6ir%2Ba4n-gtAXbzggTHnuwqLdsfQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2-nhicQa0gwtY%2BmCO6ir%2Ba4n-gtAXbzggTHnuwqLdsfQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [July 23, 2014, 5:42pm UTC](https://discuss.elastic.co/t/sudden-high-os-load-then-es-vm-disappears/18823/10 "2014-07-23T17:42:36Z")
How do you estimate the memory to configure?
Here is my rough estimation:
Docker limit = 10G
Kernel, OS services etc. ~ 1G
OS filesystem cache for ES ~50% of 10G ~ 5G
ES JVM + direct buffer + heap = 10G - 1G - 5G ~ 4G
So when you estimate for ES JVM + direct buffers ~1G, you have left 3G
(maybe 4G) for ES\_HEAP\_SIZE. So I recommend to start with 3G and increase
slowly, but no more than 5G.
With mlockall=true, you force Linux to abort processes that are not mlocked
(see OOM killer, it may be responsible for rendering the Docker LXC
unusable) so it is not a good idea to start with mlockall=true, as long as
the overall memory allocation in the Docker LXC is only vague. If you find
a save configuration where everything is balanced out, then you can enable
mlockall=true.
Jörg
On Wed, Jul 23, 2014 at 6:56 PM, Daniel Schonfeld [downwindabeam@gmail.com](mailto:downwindabeam@gmail.com)
wrote:
> Here's TOP and df -h... that's as best as I can get for now from inside
> that container.
>
> [https://gist.github.com/danielschonfeld/d75c43cce34a16a57926](https://gist.github.com/danielschonfeld/d75c43cce34a16a57926)
>
> On Wednesday, July 23, 2014 12:49:32 PM UTC-4, Daniel Schonfeld wrote:
>
> > I wanted to add another five cents worth to what Michael already
> > described.
> >
> > Before, when this happened, we used to run the docker container without a
> > memory limit and giving ES a HEAP\_SIZE of 10GB (10240M to be exact). When
> > it happened the first time, doing `free -m` revealed that the system was
> > left with barely a few hundreds of megabytes left. Second time around we
> > figured we'd tweak it a bit. Give the docker container 10G memory limit
> > and limit the ES\_HEAP\_SIZE to 7.5GB (7168M). This time when it crashed and
> > is still hung as we speak (been trying to get a thread dump, but that's not
> > happening... jstack can't even connect to that process its so deeply hung)
> > doing 'top' revealed that the memory usage is at 9.9GB for that process.
> > Basically the system seems bogged out of memory.
> >
> > Question is... how did it exceed the heap size given? And more to the
> > point, just how much memory should we be allocating? If this is just a
> > problem of shortage of memory, spawning up a new machine with a bunch more
> > RAM is no problem. But we just wish we weren't shooting in the dark.
> > We've already tried increasing the SSD size to get more IOPS (GCE ties
> > size to IOPS bandwidth).
> >
> > Last but not least, is there something we should be doing to tweak the
> > lucene segment size?
> >
> > Thanks for all your thoughts!
> >
> > On Wednesday, July 23, 2014 12:15:51 PM UTC-4, [mic...@modernmast.com](mailto:mic...@modernmast.com)
> > wrote:
> >
> > > No, the VM does not response to curl requests. Closest thing I found to
> > > that read bytes in the API was the \_cluster/stats endpoint --\>
> > > [GET \_cluster/stats · GitHub](https://gist.github.com/schonfeld/d45401e44f5961c38502)
> > >
> > > Were you referring to a different endpoint?
> > >
> > > What're your thoughts re "angry hardware"? Insufficient resources? Are
> > > there any known issues with CoreOS + ES?
> > >
> > > On Wednesday, July 23, 2014 11:12:10 AM UTC-4, Nikolas Everett wrote:
> > >
> > > > On Wed, Jul 23, 2014 at 10:19 AM, [mic...@modernmast.com](mailto:mic...@modernmast.com) wrote:
> > > >
> > > > > Looking at the JVM GC graphs, I do see some increases there, but not
> > > > > sure those are enough to cause this storm?
> > > > >
> > > > > [https://lh6.googleusercontent.com/-4wVrdN5UNRY/U8\_DuSsh15I/AAAAAAAAAAk/prHDyOwB\_gE/s1600/Screenshot+2014-07-23+10.05.10.png](https://lh6.googleusercontent.com/-4wVrdN5UNRY/U8_DuSsh15I/AAAAAAAAAAk/prHDyOwB_gE/s1600/Screenshot+2014-07-23+10.05.10.png)
> > > >
> > > > That looks like it wasn't the problem.
> > > >
> > > > > [https://lh6.googleusercontent.com/-4wVrdN5UNRY/U8\_DuSsh15I/AAAAAAAAAAk/prHDyOwB\_gE/s1600/Screenshot+2014-07-23+10.05.10.png](https://lh6.googleusercontent.com/-4wVrdN5UNRY/U8_DuSsh15I/AAAAAAAAAAk/prHDyOwB_gE/s1600/Screenshot+2014-07-23+10.05.10.png)
> > > > > The disk graphs in Marvel don't show anything out of the ordinary. I'm
> > > > > not sure how to check on those write\_bytes and read\_bytes... Where does ES
> > > > > report those? I'm using Google Compute Engine, and according to their
> > > > > minimal graphs, while there was a small spike in the disk I/Os, it wasn't
> > > > > anything insane.
> > > >
> > > > Elasticsearch reports them in the same API as the number of reads. I
> > > > imagine they are right next to each other in marvel but I'm not sure.
> > > >
> > > > > Right after the spike happened, the indexing rate spiked to 6.6k /
> > > > > second. Notice that the first tag notes the VM that left the cluster, the
> > > > > second tag shows the cluster went back to "green". Considering this
> > > > > happened after the node "left", does this give us any clues as for the
> > > > > reason?
> > > > >
> > > > > [https://lh3.googleusercontent.com/-8hmcRo-kH6k/U8\_D4QliadI/AAAAAAAAAAs/U69wNWbvbcA/s1600/Screenshot+2014-07-23+10.08.01.png](https://lh3.googleusercontent.com/-8hmcRo-kH6k/U8_D4QliadI/AAAAAAAAAAs/U69wNWbvbcA/s1600/Screenshot+2014-07-23+10.08.01.png)
> > > > >
> > > > > A few observations:
> > > > >
> > > > > - The es process is still running on the "dead machine". I can see it
> > > > > when I ssh into the VM (thru ps aux, and docker)
> > > > > - The VM doesn't show anything in the error logs, etc
> > > > > - Running a "ps aux" on that VM actually freezes (after showing the es
> > > > > process)
> > > >
> > > > That seems pretty nasty. Does the node respond to stuff like `curl localhost:9200`? Sounds like the hardware/docker is just angry.
> > > >
> > > > Nik
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an
> > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/11474c65-4443-4851-b109-75f092b49d04%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/11474c65-4443-4851-b109-75f092b49d04%40googlegroups.com)
> > > [https://groups.google.com/d/msgid/elasticsearch/11474c65-4443-4851-b109-75f092b49d04%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/11474c65-4443-4851-b109-75f092b49d04%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > .
>
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFKTuK92UMYjMJCBwibAH7CJ-1S5W2yMeNbZQK9\_y\_Wvg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFKTuK92UMYjMJCBwibAH7CJ-1S5W2yMeNbZQK9_y_Wvg%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:13am UTC](https://discuss.elastic.co/t/sudden-high-os-load-then-es-vm-disappears/18823/11 "2017-07-06T01:13:50Z")