# Direct buffer memory problem on master Discovery
**URL:** https://discuss.elastic.co/t/direct-buffer-memory-problem-on-master-discovery/18676
**Category:** Elasticsearch
**Created:** [July 15, 2014, 12:33pm UTC](https://discuss.elastic.co/t/direct-buffer-memory-problem-on-master-discovery/18676 "2014-07-15T12:33:54Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@Pedro\_Jeronimo](https://discuss.elastic.co/u/Pedro_Jeronimo)
#### Post date: [July 15, 2014, 12:33pm UTC](https://discuss.elastic.co/t/direct-buffer-memory-problem-on-master-discovery/18676/1 "2014-07-15T12:33:54Z")
I have a cluster of 2 ES machines with a lot of indexing, not so much
searching. I'm using 2 EC2 machines with 30gb of RAM and I'm running ES on
each with 12gb heap (ES\_HEAP\_SIZE) and one of them (let's call it logs1) is
running logstash as well, with 2gb heap. The master node is logs1 and the
other instance is logs2. I start the cluster and every things looks fine,
but after a while (1-3 days) I get the following error on logs1:
[2014-07-15 12:26:39,867][WARN][transport.netty] [Keen Marlow] exception
caught on transport layer [[id: 0x16801a48, /XX.XX.XXX.XX:36314 =\>
/XX.XXX.XX.XX:9300]], closing connection
java.lang.OutOfMemoryError: Direct buffer memory
...Stack Trace...
And then the cluster is no longer connected and if I try to restart logs2,
I get the same error above for logs1 and this one for logs2:
[2014-07-15 12:27:39,282][INFO][discovery.ec2] [Betty Ross Banner] failed
to send join request to master [[Keen
Marlow][9a7FIRpBSrKQcdcV\_sjSTw][ip-XX-XX-XXX-XX][inet[/XX.XX.XXX.XX:9300]]{aws\_availability\_zone=us-west-2a,
master=true}], reason
[org.elasticsearch.transport.RemoteTransportException: [Keen
Marlow][inet[/XX.XX.XXX.XX:9300]][discovery/zen/join];
org.elasticsearch.transport.NodeDisconnectedException: [Betty Ross
Banner][inet[/XX.XXX.XX.XX:9300]][discovery/zen/join/validate] disconnected]
Is there any memory configuration I should tune up a bit? I'm kind of new
to ElasticSearch so I'd love some help! :).
Thanks!
Pedro
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [July 15, 2014, 10:46pm UTC](https://discuss.elastic.co/t/direct-buffer-memory-problem-on-master-discovery/18676/2 "2014-07-15T22:46:02Z")
How much data do you have in ES, index count and total size of all indexes?
OutOfMemoryError means you ran out of heap, which could mean a few things.
Regards,
Mark Walkom
Infrastructure Engineer
Campaign Monitor
email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)
web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
On 15 July 2014 22:33, Pedro Jerónimo [pedropregueiro@gmail.com](mailto:pedropregueiro@gmail.com) wrote:
> I have a cluster of 2 ES machines with a lot of indexing, not so much
> searching. I'm using 2 EC2 machines with 30gb of RAM and I'm running ES on
> each with 12gb heap (ES\_HEAP\_SIZE) and one of them (let's call it logs1)
> is running logstash as well, with 2gb heap. The master node is logs1 and
> the other instance is logs2. I start the cluster and every things looks
> fine, but after a while (1-3 days) I get the following error on logs1:
>
> [2014-07-15 12:26:39,867][WARN][transport.netty] [Keen Marlow] exception
> caught on transport layer [[id: 0x16801a48, /XX.XX.XXX.XX:36314 =\>
> /XX.XXX.XX.XX:9300]], closing connection
> java.lang.OutOfMemoryError: Direct buffer memory
> ...Stack Trace...
>
> And then the cluster is no longer connected and if I try to restart logs2,
> I get the same error above for logs1 and this one for logs2:
>
> [2014-07-15 12:27:39,282][INFO][discovery.ec2] [Betty Ross Banner]
> failed to send join request to master [[Keen
> Marlow][9a7FIRpBSrKQcdcV\_sjSTw][ip-XX-XX-XXX-XX][inet[/XX.XX.XXX.XX:9300]]{aws\_availability\_zone=us-west-2a,
> master=true}], reason
> [org.elasticsearch.transport.RemoteTransportException: [Keen
> Marlow][inet[/XX.XX.XXX.XX:9300]][discovery/zen/join];
> org.elasticsearch.transport.NodeDisconnectedException: [Betty Ross
> Banner][inet[/XX.XXX.XX.XX:9300]][discovery/zen/join/validate] disconnected]
>
> Is there any memory configuration I should tune up a bit? I'm kind of new
> to Elasticsearch so I'd love some help! :).
>
> Thanks!
>
> Pedro
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com)
> [https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com?utm_medium=email&utm_source=footer)
> .
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624Yb9mhh3-E9jYvFFaY\_dB8rF-R5yZCGYCHrvWNsur-MwQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624Yb9mhh3-E9jYvFFaY_dB8rF-R5yZCGYCHrvWNsur-MwQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Ivan](https://discuss.elastic.co/u/Ivan)
#### Post date: [July 16, 2014, 3:40am UTC](https://discuss.elastic.co/t/direct-buffer-memory-problem-on-master-discovery/18676/3 "2014-07-16T03:40:12Z")
Direct memory is off heap memory. Are elasticsearch and logstash the only
processes on those servers? Did you set an explicit direct memory value?
--
Ivan
On Jul 15, 2014 3:46 PM, "Mark Walkom" [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) wrote:
> How much data do you have in ES, index count and total size of all indexes?
>
> OutOfMemoryError means you ran out of heap, which could mean a few things.
>
> Regards,
> Mark Walkom
>
> Infrastructure Engineer
> Campaign Monitor
> email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)
> web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
>
> On 15 July 2014 22:33, Pedro Jerónimo [pedropregueiro@gmail.com](mailto:pedropregueiro@gmail.com) wrote:
>
> > I have a cluster of 2 ES machines with a lot of indexing, not so much
> > searching. I'm using 2 EC2 machines with 30gb of RAM and I'm running ES on
> > each with 12gb heap (ES\_HEAP\_SIZE) and one of them (let's call it logs1)
> > is running logstash as well, with 2gb heap. The master node is logs1 and
> > the other instance is logs2. I start the cluster and every things looks
> > fine, but after a while (1-3 days) I get the following error on logs1:
> >
> > [2014-07-15 12:26:39,867][WARN][transport.netty] [Keen Marlow]
> > exception caught on transport layer [[id: 0x16801a48, /XX.XX.XXX.XX:36314
> > =\> /XX.XXX.XX.XX:9300]], closing connection
> > java.lang.OutOfMemoryError: Direct buffer memory
> > ...Stack Trace...
> >
> > And then the cluster is no longer connected and if I try to restart
> > logs2, I get the same error above for logs1 and this one for logs2:
> >
> > [2014-07-15 12:27:39,282][INFO][discovery.ec2] [Betty Ross Banner]
> > failed to send join request to master [[Keen
> > Marlow][9a7FIRpBSrKQcdcV\_sjSTw][ip-XX-XX-XXX-XX][inet[/XX.XX.XXX.XX:9300]]{aws\_availability\_zone=us-west-2a,
> > master=true}], reason
> > [org.elasticsearch.transport.RemoteTransportException: [Keen
> > Marlow][inet[/XX.XX.XXX.XX:9300]][discovery/zen/join];
> > org.elasticsearch.transport.NodeDisconnectedException: [Betty Ross
> > Banner][inet[/XX.XXX.XX.XX:9300]][discovery/zen/join/validate] disconnected]
> >
> > Is there any memory configuration I should tune up a bit? I'm kind of new
> > to Elasticsearch so I'd love some help! :).
> >
> > Thanks!
> >
> > Pedro
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com)
> > [https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/CAEM624Yb9mhh3-E9jYvFFaY\_dB8rF-R5yZCGYCHrvWNsur-MwQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624Yb9mhh3-E9jYvFFaY_dB8rF-R5yZCGYCHrvWNsur-MwQ%40mail.gmail.com)
> [https://groups.google.com/d/msgid/elasticsearch/CAEM624Yb9mhh3-E9jYvFFaY\_dB8rF-R5yZCGYCHrvWNsur-MwQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEM624Yb9mhh3-E9jYvFFaY_dB8rF-R5yZCGYCHrvWNsur-MwQ%40mail.gmail.com?utm_medium=email&utm_source=footer)
> .
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDXTAG7%3Dz66Ghs7H3hfSTmLuEprUjoKK-4XEv\_tSbENBA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDXTAG7%3Dz66Ghs7H3hfSTmLuEprUjoKK-4XEv_tSbENBA%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Pedro\_Jeronimo](https://discuss.elastic.co/u/Pedro_Jeronimo)
#### Post date: [July 16, 2014, 8:04am UTC](https://discuss.elastic.co/t/direct-buffer-memory-problem-on-master-discovery/18676/4 "2014-07-16T08:04:14Z")
@Mark, I'm using this cluster only for logstash, so I have 1 new index
p/day, each index has between 30k and 100k documents. I use curator in a
crontab to close old logs older than X days so I don't have them all loaded
into memory at the same time. Right now I have 8 indices opened with a
total of ~500k documents occupying ~4.5Gb of disk space. I read a bit about
OOM with Elasticsearch and I couldn't figure out my problem yet. I followed
ES docs memory tips:
> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.
And I see that mlockall is set to true on both instances.
@Ivan, yes, ES and Logstash are the only processes running on the servers.
No, I didn't set direct memory values. Should I? If so, which sort of
values and how do you determine those?
It's been a while since I worked with JVMs so my knowledge on Java Memory
stuff is kind of rusty :). Thanks for the help!
On Wednesday, July 16, 2014 5:40:27 AM UTC+2, Ivan Brusic wrote:
> Direct memory is off heap memory. Are elasticsearch and logstash the only
> processes on those servers? Did you set an explicit direct memory value?
>
> --
> Ivan
> On Jul 15, 2014 3:46 PM, "Mark Walkom" \<[ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com)
> \\> wrote:
>
> > How much data do you have in ES, index count and total size of all
> > indexes?
> >
> > OutOfMemoryError means you ran out of heap, which could mean a few things.
> >
> > Regards,
> > Mark Walkom
> >
> > Infrastructure Engineer
> > Campaign Monitor
> > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \
> > web: [www.campaignmonitor.com](http://www.campaignmonitor.com)
> >
> > On 15 July 2014 22:33, Pedro Jerónimo \<[pedropr...@gmail.com](mailto:pedropr...@gmail.com) \
> >
> > > wrote:
> >
> > > I have a cluster of 2 ES machines with a lot of indexing, not so much
> > > searching. I'm using 2 EC2 machines with 30gb of RAM and I'm running ES on
> > > each with 12gb heap (ES\_HEAP\_SIZE) and one of them (let's call it
> > > logs1) is running logstash as well, with 2gb heap. The master node is logs1
> > > and the other instance is logs2. I start the cluster and every things looks
> > > fine, but after a while (1-3 days) I get the following error on logs1:
> > >
> > > [2014-07-15 12:26:39,867][WARN][transport.netty] [Keen Marlow]
> > > exception caught on transport layer [[id: 0x16801a48, /XX.XX.XXX.XX:36314
> > > =\> /XX.XXX.XX.XX:9300]], closing connection
> > > java.lang.OutOfMemoryError: Direct buffer memory
> > > ...Stack Trace...
> > >
> > > And then the cluster is no longer connected and if I try to restart
> > > logs2, I get the same error above for logs1 and this one for logs2:
> > >
> > > [2014-07-15 12:27:39,282][INFO][discovery.ec2] [Betty Ross Banner]
> > > failed to send join request to master [[Keen
> > > Marlow][9a7FIRpBSrKQcdcV\_sjSTw][ip-XX-XX-XXX-XX][inet[/XX.XX.XXX.XX:9300]]{aws\_availability\_zone=us-west-2a,
> > > master=true}], reason
> > > [org.elasticsearch.transport.RemoteTransportException: [Keen
> > > Marlow][inet[/XX.XX.XXX.XX:9300]][discovery/zen/join];
> > > org.elasticsearch.transport.NodeDisconnectedException: [Betty Ross
> > > Banner][inet[/XX.XXX.XX.XX:9300]][discovery/zen/join/validate] disconnected]
> > >
> > > Is there any memory configuration I should tune up a bit? I'm kind of
> > > new to Elasticsearch so I'd love some help! :).
> > >
> > > Thanks!
> > >
> > > Pedro
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com)
> > > [https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > .
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/CAEM624Yb9mhh3-E9jYvFFaY\_dB8rF-R5yZCGYCHrvWNsur-MwQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624Yb9mhh3-E9jYvFFaY_dB8rF-R5yZCGYCHrvWNsur-MwQ%40mail.gmail.com)
> > [https://groups.google.com/d/msgid/elasticsearch/CAEM624Yb9mhh3-E9jYvFFaY\_dB8rF-R5yZCGYCHrvWNsur-MwQ%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAEM624Yb9mhh3-E9jYvFFaY_dB8rF-R5yZCGYCHrvWNsur-MwQ%40mail.gmail.com?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/bb843355-0c29-4ec1-803a-9149b89fc36a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/bb843355-0c29-4ec1-803a-9149b89fc36a%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [July 16, 2014, 8:54am UTC](https://discuss.elastic.co/t/direct-buffer-memory-problem-on-master-discovery/18676/5 "2014-07-16T08:54:58Z")
What ES, and what Java version is this?
Jörg
On Tue, Jul 15, 2014 at 2:33 PM, Pedro Jerónimo [pedropregueiro@gmail.com](mailto:pedropregueiro@gmail.com)
wrote:
> I have a cluster of 2 ES machines with a lot of indexing, not so much
> searching. I'm using 2 EC2 machines with 30gb of RAM and I'm running ES on
> each with 12gb heap (ES\_HEAP\_SIZE) and one of them (let's call it logs1)
> is running logstash as well, with 2gb heap. The master node is logs1 and
> the other instance is logs2. I start the cluster and every things looks
> fine, but after a while (1-3 days) I get the following error on logs1:
>
> [2014-07-15 12:26:39,867][WARN][transport.netty] [Keen Marlow] exception
> caught on transport layer [[id: 0x16801a48, /XX.XX.XXX.XX:36314 =\>
> /XX.XXX.XX.XX:9300]], closing connection
> java.lang.OutOfMemoryError: Direct buffer memory
> ...Stack Trace...
>
> And then the cluster is no longer connected and if I try to restart logs2,
> I get the same error above for logs1 and this one for logs2:
>
> [2014-07-15 12:27:39,282][INFO][discovery.ec2] [Betty Ross Banner]
> failed to send join request to master [[Keen
> Marlow][9a7FIRpBSrKQcdcV\_sjSTw][ip-XX-XX-XXX-XX][inet[/XX.XX.XXX.XX:9300]]{aws\_availability\_zone=us-west-2a,
> master=true}], reason
> [org.elasticsearch.transport.RemoteTransportException: [Keen
> Marlow][inet[/XX.XX.XXX.XX:9300]][discovery/zen/join];
> org.elasticsearch.transport.NodeDisconnectedException: [Betty Ross
> Banner][inet[/XX.XXX.XX.XX:9300]][discovery/zen/join/validate] disconnected]
>
> Is there any memory configuration I should tune up a bit? I'm kind of new
> to Elasticsearch so I'd love some help! :).
>
> Thanks!
>
> Pedro
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com)
> [https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com?utm_medium=email&utm_source=footer)
> .
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGZauOEQXa6ybBTqgSPMuVR4AS-Tdio7X61Wfkobc7sJQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGZauOEQXa6ybBTqgSPMuVR4AS-Tdio7X61Wfkobc7sJQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Pedro\_Jeronimo](https://discuss.elastic.co/u/Pedro_Jeronimo)
#### Post date: [July 16, 2014, 9:40am UTC](https://discuss.elastic.co/t/direct-buffer-memory-problem-on-master-discovery/18676/6 "2014-07-16T09:40:23Z")
\*Java: \*java version "1.7.0\_55"
\*Elasticsearch: \*1.2.1
On Wednesday, July 16, 2014 10:55:10 AM UTC+2, Jörg Prante wrote:
> What ES, and what Java version is this?
>
> Jörg
>
> On Tue, Jul 15, 2014 at 2:33 PM, Pedro Jerónimo \<[pedropr...@gmail.com](mailto:pedropr...@gmail.com)
> \\> wrote:
>
> > I have a cluster of 2 ES machines with a lot of indexing, not so much
> > searching. I'm using 2 EC2 machines with 30gb of RAM and I'm running ES on
> > each with 12gb heap (ES\_HEAP\_SIZE) and one of them (let's call it logs1)
> > is running logstash as well, with 2gb heap. The master node is logs1 and
> > the other instance is logs2. I start the cluster and every things looks
> > fine, but after a while (1-3 days) I get the following error on logs1:
> >
> > [2014-07-15 12:26:39,867][WARN][transport.netty] [Keen Marlow]
> > exception caught on transport layer [[id: 0x16801a48, /XX.XX.XXX.XX:36314
> > =\> /XX.XXX.XX.XX:9300]], closing connection
> > java.lang.OutOfMemoryError: Direct buffer memory
> > ...Stack Trace...
> >
> > And then the cluster is no longer connected and if I try to restart
> > logs2, I get the same error above for logs1 and this one for logs2:
> >
> > [2014-07-15 12:27:39,282][INFO][discovery.ec2] [Betty Ross Banner]
> > failed to send join request to master [[Keen
> > Marlow][9a7FIRpBSrKQcdcV\_sjSTw][ip-XX-XX-XXX-XX][inet[/XX.XX.XXX.XX:9300]]{aws\_availability\_zone=us-west-2a,
> > master=true}], reason
> > [org.elasticsearch.transport.RemoteTransportException: [Keen
> > Marlow][inet[/XX.XX.XXX.XX:9300]][discovery/zen/join];
> > org.elasticsearch.transport.NodeDisconnectedException: [Betty Ross
> > Banner][inet[/XX.XXX.XX.XX:9300]][discovery/zen/join/validate] disconnected]
> >
> > Is there any memory configuration I should tune up a bit? I'm kind of new
> > to Elasticsearch so I'd love some help! :).
> >
> > Thanks!
> >
> > Pedro
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com)
> > [https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3dc6803d-618c-4c2d-84cc-1fa9e3a7d6a3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3dc6803d-618c-4c2d-84cc-1fa9e3a7d6a3%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Ivan](https://discuss.elastic.co/u/Ivan)
#### Post date: [July 16, 2014, 4:55pm UTC](https://discuss.elastic.co/t/direct-buffer-memory-problem-on-master-discovery/18676/7 "2014-07-16T16:55:31Z")
Most users do not set the direct memory setting. mlockall is set, but does
the server allow it? You would see an error on startup if it didn't. Did
you change the vm swapiness on the server?
--
Ivan
On Wed, Jul 16, 2014 at 2:40 AM, Pedro Jerónimo [pedropregueiro@gmail.com](mailto:pedropregueiro@gmail.com)
wrote:
> \*Java: \*java version "1.7.0\_55"
> \*Elasticsearch: \*1.2.1
>
> On Wednesday, July 16, 2014 10:55:10 AM UTC+2, Jörg Prante wrote:
>
> > What ES, and what Java version is this?
> >
> > Jörg
> >
> > On Tue, Jul 15, 2014 at 2:33 PM, Pedro Jerónimo [pedropr...@gmail.com](mailto:pedropr...@gmail.com)
> > wrote:
> >
> > > I have a cluster of 2 ES machines with a lot of indexing, not so much
> > > searching. I'm using 2 EC2 machines with 30gb of RAM and I'm running ES on
> > > each with 12gb heap (ES\_HEAP\_SIZE) and one of them (let's call it
> > > logs1) is running logstash as well, with 2gb heap. The master node is logs1
> > > and the other instance is logs2. I start the cluster and every things looks
> > > fine, but after a while (1-3 days) I get the following error on logs1:
> > >
> > > [2014-07-15 12:26:39,867][WARN][transport.netty] [Keen Marlow]
> > > exception caught on transport layer [[id: 0x16801a48, /XX.XX.XXX.XX:36314
> > > =\> /XX.XXX.XX.XX:9300]], closing connection
> > > java.lang.OutOfMemoryError: Direct buffer memory
> > > ...Stack Trace...
> > >
> > > And then the cluster is no longer connected and if I try to restart
> > > logs2, I get the same error above for logs1 and this one for logs2:
> > >
> > > [2014-07-15 12:27:39,282][INFO][discovery.ec2] [Betty Ross Banner]
> > > failed to send join request to master [[Keen Marlow][9a7FIRpBSrKQcdcV\_
> > > sjSTw][ip-XX-XX-XXX-XX][inet[/XX.XX.XXX.XX:9300]]{aws\_availability\_zone=us-west-2a,
> > > master=true}], reason [org.elasticsearch.transport.RemoteTransportException:
> > > [Keen Marlow][inet[/XX.XX.XXX.XX:9300]][discovery/zen/join];
> > > org.elasticsearch.transport.NodeDisconnectedException: [Betty Ross
> > > Banner][inet[/XX.XXX.XX.XX:9300]][discovery/zen/join/validate]
> > > disconnected]
> > >
> > > Is there any memory configuration I should tune up a bit? I'm kind of
> > > new to Elasticsearch so I'd love some help! :).
> > >
> > > Thanks!
> > >
> > > Pedro
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > >
> > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%
> > > [40googlegroups.com](http://40googlegroups.com)
> > > [https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > .
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/3dc6803d-618c-4c2d-84cc-1fa9e3a7d6a3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3dc6803d-618c-4c2d-84cc-1fa9e3a7d6a3%40googlegroups.com)
> > [https://groups.google.com/d/msgid/elasticsearch/3dc6803d-618c-4c2d-84cc-1fa9e3a7d6a3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/3dc6803d-618c-4c2d-84cc-1fa9e3a7d6a3%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
>
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQD6Zp%2BHMvFTaUQ3L5FXV7sKaR4MYSVOVQa%3DZzfsBrB7Lg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQD6Zp%2BHMvFTaUQ3L5FXV7sKaR4MYSVOVQa%3DZzfsBrB7Lg%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Pedro\_Jeronimo](https://discuss.elastic.co/u/Pedro_Jeronimo)
#### Post date: [July 16, 2014, 5:55pm UTC](https://discuss.elastic.co/t/direct-buffer-memory-problem-on-master-discovery/18676/8 "2014-07-16T17:55:30Z")
Ivan, I don't see any error on startup and, besides that, I see that it is
set on the GET \_nodes/process?pretty:
process: {
refresh\_interval\_in\_millis: 1000,
id: 17132,
max\_file\_descriptors: 4096,
- mlockall: true\*
}
I didn't make any other changes related to memory. According to ES docs
([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/setup-configuration.html))
I got under the impression that enabling this mlockall would solve memory
issues. Apparently I was wrong 🙂
Any hints on this or the direct memory?
On Wednesday, July 16, 2014 6:55:41 PM UTC+2, Ivan Brusic wrote:
> Most users do not set the direct memory setting. mlockall is set, but does
> the server allow it? You would see an error on startup if it didn't. Did
> you change the vm swapiness on the server?
>
> --
> Ivan
>
> On Wed, Jul 16, 2014 at 2:40 AM, Pedro Jerónimo \<[pedropr...@gmail.com](mailto:pedropr...@gmail.com)
> \\> wrote:
>
> > \*Java: \*java version "1.7.0\_55"
> > \*Elasticsearch: \*1.2.1
> >
> > On Wednesday, July 16, 2014 10:55:10 AM UTC+2, Jörg Prante wrote:
> >
> > > What ES, and what Java version is this?
> > >
> > > Jörg
> > >
> > > On Tue, Jul 15, 2014 at 2:33 PM, Pedro Jerónimo [pedropr...@gmail.com](mailto:pedropr...@gmail.com)
> > > wrote:
> > >
> > > > I have a cluster of 2 ES machines with a lot of indexing, not so much
> > > > searching. I'm using 2 EC2 machines with 30gb of RAM and I'm running ES on
> > > > each with 12gb heap (ES\_HEAP\_SIZE) and one of them (let's call it
> > > > logs1) is running logstash as well, with 2gb heap. The master node is logs1
> > > > and the other instance is logs2. I start the cluster and every things looks
> > > > fine, but after a while (1-3 days) I get the following error on logs1:
> > > >
> > > > [2014-07-15 12:26:39,867][WARN][transport.netty] [Keen Marlow]
> > > > exception caught on transport layer [[id: 0x16801a48, /XX.XX.XXX.XX:36314
> > > > =\> /XX.XXX.XX.XX:9300]], closing connection
> > > > java.lang.OutOfMemoryError: Direct buffer memory
> > > > ...Stack Trace...
> > > >
> > > > And then the cluster is no longer connected and if I try to restart
> > > > logs2, I get the same error above for logs1 and this one for logs2:
> > > >
> > > > [2014-07-15 12:27:39,282][INFO][discovery.ec2] [Betty Ross Banner]
> > > > failed to send join request to master [[Keen Marlow][9a7FIRpBSrKQcdcV\_
> > > > sjSTw][ip-XX-XX-XXX-XX][inet[/XX.XX.XXX.XX:9300]]{aws\_availability\_zone=us-west-2a,
> > > > master=true}], reason [org.elasticsearch.transport.RemoteTransportException:
> > > > [Keen Marlow][inet[/XX.XX.XXX.XX:9300]][discovery/zen/join];
> > > > org.elasticsearch.transport.NodeDisconnectedException: [Betty Ross
> > > > Banner][inet[/XX.XXX.XX.XX:9300]][discovery/zen/join/validate]
> > > > disconnected]
> > > >
> > > > Is there any memory configuration I should tune up a bit? I'm kind of
> > > > new to Elasticsearch so I'd love some help! :).
> > > >
> > > > Thanks!
> > > >
> > > > Pedro
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google
> > > > Groups "elasticsearch" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > >
> > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > > msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%
> > > > [40googlegroups.com](http://40googlegroups.com)
> > > > [https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/7d155e16-c8ff-40b0-ac0a-e3719e8296c3%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > .
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an
> > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/3dc6803d-618c-4c2d-84cc-1fa9e3a7d6a3%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3dc6803d-618c-4c2d-84cc-1fa9e3a7d6a3%40googlegroups.com)
> > > [https://groups.google.com/d/msgid/elasticsearch/3dc6803d-618c-4c2d-84cc-1fa9e3a7d6a3%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/3dc6803d-618c-4c2d-84cc-1fa9e3a7d6a3%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > .
> >
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/29ea7a97-50af-4f75-beb6-f6cbb18d266d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/29ea7a97-50af-4f75-beb6-f6cbb18d266d%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 1:15am UTC](https://discuss.elastic.co/t/direct-buffer-memory-problem-on-master-discovery/18676/9 "2017-07-06T01:15:22Z")