I have a cluster of 2 ES machines with a lot of indexing, not so much
searching. I'm using 2 EC2 machines with 30gb of RAM and I'm running ES on
each with 12gb heap (ES_HEAP_SIZE) and one of them (let's call it logs1) is
running logstash as well, with 2gb heap. The master node is logs1 and the
other instance is logs2. I start the cluster and every things looks fine,
but after a while (1-3 days) I get the following error on logs1:
[2014-07-15 12:26:39,867][WARN ][transport.netty ] [Keen Marlow] exception
caught on transport layer [[id: 0x16801a48, /XX.XX.XXX.XX:36314 =>
/XX.XXX.XX.XX:9300]], closing connection
java.lang.OutOfMemoryError: Direct buffer memory
...Stack Trace...
And then the cluster is no longer connected and if I try to restart logs2,
I get the same error above for logs1 and this one for logs2:
[2014-07-15 12:27:39,282][INFO ][discovery.ec2 ] [Betty Ross Banner] failed
to send join request to master [[Keen
Marlow][9a7FIRpBSrKQcdcV_sjSTw][ip-XX-XX-XXX-XX][inet[/XX.XX.XXX.XX:9300]]{aws_availability_zone=us-west-2a,
master=true}], reason
[org.elasticsearch.transport.RemoteTransportException: [Keen
Marlow][inet[/XX.XX.XXX.XX:9300]][discovery/zen/join];
org.elasticsearch.transport.NodeDisconnectedException: [Betty Ross
Banner][inet[/XX.XXX.XX.XX:9300]][discovery/zen/join/validate] disconnected]
Is there any memory configuration I should tune up a bit? I'm kind of new
to ElasticSearch so I'd love some help! :).
I have a cluster of 2 ES machines with a lot of indexing, not so much
searching. I'm using 2 EC2 machines with 30gb of RAM and I'm running ES on
each with 12gb heap (ES_HEAP_SIZE) and one of them (let's call it logs1)
is running logstash as well, with 2gb heap. The master node is logs1 and
the other instance is logs2. I start the cluster and every things looks
fine, but after a while (1-3 days) I get the following error on logs1:
[2014-07-15 12:26:39,867][WARN ][transport.netty ] [Keen Marlow] exception
caught on transport layer [[id: 0x16801a48, /XX.XX.XXX.XX:36314 =>
/XX.XXX.XX.XX:9300]], closing connection
java.lang.OutOfMemoryError: Direct buffer memory
...Stack Trace...
And then the cluster is no longer connected and if I try to restart logs2,
I get the same error above for logs1 and this one for logs2:
[2014-07-15 12:27:39,282][INFO ][discovery.ec2 ] [Betty Ross Banner]
failed to send join request to master [[Keen
Marlow][9a7FIRpBSrKQcdcV_sjSTw][ip-XX-XX-XXX-XX][inet[/XX.XX.XXX.XX:9300]]{aws_availability_zone=us-west-2a,
master=true}], reason
[org.elasticsearch.transport.RemoteTransportException: [Keen
Marlow][inet[/XX.XX.XXX.XX:9300]][discovery/zen/join];
org.elasticsearch.transport.NodeDisconnectedException: [Betty Ross
Banner][inet[/XX.XXX.XX.XX:9300]][discovery/zen/join/validate] disconnected]
Is there any memory configuration I should tune up a bit? I'm kind of new
to Elasticsearch so I'd love some help! :).
I have a cluster of 2 ES machines with a lot of indexing, not so much
searching. I'm using 2 EC2 machines with 30gb of RAM and I'm running ES on
each with 12gb heap (ES_HEAP_SIZE) and one of them (let's call it logs1)
is running logstash as well, with 2gb heap. The master node is logs1 and
the other instance is logs2. I start the cluster and every things looks
fine, but after a while (1-3 days) I get the following error on logs1:
[2014-07-15 12:26:39,867][WARN ][transport.netty ] [Keen Marlow]
exception caught on transport layer [[id: 0x16801a48, /XX.XX.XXX.XX:36314
=> /XX.XXX.XX.XX:9300]], closing connection
java.lang.OutOfMemoryError: Direct buffer memory
...Stack Trace...
And then the cluster is no longer connected and if I try to restart
logs2, I get the same error above for logs1 and this one for logs2:
[2014-07-15 12:27:39,282][INFO ][discovery.ec2 ] [Betty Ross Banner]
failed to send join request to master [[Keen
Marlow][9a7FIRpBSrKQcdcV_sjSTw][ip-XX-XX-XXX-XX][inet[/XX.XX.XXX.XX:9300]]{aws_availability_zone=us-west-2a,
master=true}], reason
[org.elasticsearch.transport.RemoteTransportException: [Keen
Marlow][inet[/XX.XX.XXX.XX:9300]][discovery/zen/join];
org.elasticsearch.transport.NodeDisconnectedException: [Betty Ross
Banner][inet[/XX.XXX.XX.XX:9300]][discovery/zen/join/validate] disconnected]
Is there any memory configuration I should tune up a bit? I'm kind of new
to Elasticsearch so I'd love some help! :).
@Mark, I'm using this cluster only for logstash, so I have 1 new index
p/day, each index has between 30k and 100k documents. I use curator in a
crontab to close old logs older than X days so I don't have them all loaded
into memory at the same time. Right now I have 8 indices opened with a
total of ~500k documents occupying ~4.5Gb of disk space. I read a bit about
OOM with Elasticsearch and I couldn't figure out my problem yet. I followed
ES docs memory tips:
And I see that mlockall is set to true on both instances.
@Ivan, yes, ES and Logstash are the only processes running on the servers.
No, I didn't set direct memory values. Should I? If so, which sort of
values and how do you determine those?
It's been a while since I worked with JVMs so my knowledge on Java Memory
stuff is kind of rusty :). Thanks for the help!
On Wednesday, July 16, 2014 5:40:27 AM UTC+2, Ivan Brusic wrote:
Direct memory is off heap memory. Are elasticsearch and logstash the only
processes on those servers? Did you set an explicit direct memory value?
I have a cluster of 2 ES machines with a lot of indexing, not so much
searching. I'm using 2 EC2 machines with 30gb of RAM and I'm running ES on
each with 12gb heap (ES_HEAP_SIZE) and one of them (let's call it
logs1) is running logstash as well, with 2gb heap. The master node is logs1
and the other instance is logs2. I start the cluster and every things looks
fine, but after a while (1-3 days) I get the following error on logs1:
[2014-07-15 12:26:39,867][WARN ][transport.netty ] [Keen Marlow]
exception caught on transport layer [[id: 0x16801a48, /XX.XX.XXX.XX:36314
=> /XX.XXX.XX.XX:9300]], closing connection
java.lang.OutOfMemoryError: Direct buffer memory
...Stack Trace...
And then the cluster is no longer connected and if I try to restart
logs2, I get the same error above for logs1 and this one for logs2:
[2014-07-15 12:27:39,282][INFO ][discovery.ec2 ] [Betty Ross Banner]
failed to send join request to master [[Keen
Marlow][9a7FIRpBSrKQcdcV_sjSTw][ip-XX-XX-XXX-XX][inet[/XX.XX.XXX.XX:9300]]{aws_availability_zone=us-west-2a,
master=true}], reason
[org.elasticsearch.transport.RemoteTransportException: [Keen
Marlow][inet[/XX.XX.XXX.XX:9300]][discovery/zen/join];
org.elasticsearch.transport.NodeDisconnectedException: [Betty Ross
Banner][inet[/XX.XXX.XX.XX:9300]][discovery/zen/join/validate] disconnected]
Is there any memory configuration I should tune up a bit? I'm kind of
new to Elasticsearch so I'd love some help! :).
I have a cluster of 2 ES machines with a lot of indexing, not so much
searching. I'm using 2 EC2 machines with 30gb of RAM and I'm running ES on
each with 12gb heap (ES_HEAP_SIZE) and one of them (let's call it logs1)
is running logstash as well, with 2gb heap. The master node is logs1 and
the other instance is logs2. I start the cluster and every things looks
fine, but after a while (1-3 days) I get the following error on logs1:
[2014-07-15 12:26:39,867][WARN ][transport.netty ] [Keen Marlow] exception
caught on transport layer [[id: 0x16801a48, /XX.XX.XXX.XX:36314 =>
/XX.XXX.XX.XX:9300]], closing connection
java.lang.OutOfMemoryError: Direct buffer memory
...Stack Trace...
And then the cluster is no longer connected and if I try to restart logs2,
I get the same error above for logs1 and this one for logs2:
[2014-07-15 12:27:39,282][INFO ][discovery.ec2 ] [Betty Ross Banner]
failed to send join request to master [[Keen
Marlow][9a7FIRpBSrKQcdcV_sjSTw][ip-XX-XX-XXX-XX][inet[/XX.XX.XXX.XX:9300]]{aws_availability_zone=us-west-2a,
master=true}], reason
[org.elasticsearch.transport.RemoteTransportException: [Keen
Marlow][inet[/XX.XX.XXX.XX:9300]][discovery/zen/join];
org.elasticsearch.transport.NodeDisconnectedException: [Betty Ross
Banner][inet[/XX.XXX.XX.XX:9300]][discovery/zen/join/validate] disconnected]
Is there any memory configuration I should tune up a bit? I'm kind of new
to Elasticsearch so I'd love some help! :).
*Java: *java version "1.7.0_55"
*Elasticsearch: *1.2.1
On Wednesday, July 16, 2014 10:55:10 AM UTC+2, Jörg Prante wrote:
What ES, and what Java version is this?
Jörg
On Tue, Jul 15, 2014 at 2:33 PM, Pedro Jerónimo <pedropr...@gmail.com
<javascript:>> wrote:
I have a cluster of 2 ES machines with a lot of indexing, not so much
searching. I'm using 2 EC2 machines with 30gb of RAM and I'm running ES on
each with 12gb heap (ES_HEAP_SIZE) and one of them (let's call it logs1)
is running logstash as well, with 2gb heap. The master node is logs1 and
the other instance is logs2. I start the cluster and every things looks
fine, but after a while (1-3 days) I get the following error on logs1:
[2014-07-15 12:26:39,867][WARN ][transport.netty ] [Keen Marlow]
exception caught on transport layer [[id: 0x16801a48, /XX.XX.XXX.XX:36314
=> /XX.XXX.XX.XX:9300]], closing connection
java.lang.OutOfMemoryError: Direct buffer memory
...Stack Trace...
And then the cluster is no longer connected and if I try to restart
logs2, I get the same error above for logs1 and this one for logs2:
[2014-07-15 12:27:39,282][INFO ][discovery.ec2 ] [Betty Ross Banner]
failed to send join request to master [[Keen
Marlow][9a7FIRpBSrKQcdcV_sjSTw][ip-XX-XX-XXX-XX][inet[/XX.XX.XXX.XX:9300]]{aws_availability_zone=us-west-2a,
master=true}], reason
[org.elasticsearch.transport.RemoteTransportException: [Keen
Marlow][inet[/XX.XX.XXX.XX:9300]][discovery/zen/join];
org.elasticsearch.transport.NodeDisconnectedException: [Betty Ross
Banner][inet[/XX.XXX.XX.XX:9300]][discovery/zen/join/validate] disconnected]
Is there any memory configuration I should tune up a bit? I'm kind of new
to Elasticsearch so I'd love some help! :).
Most users do not set the direct memory setting. mlockall is set, but does
the server allow it? You would see an error on startup if it didn't. Did
you change the vm swapiness on the server?
I have a cluster of 2 ES machines with a lot of indexing, not so much
searching. I'm using 2 EC2 machines with 30gb of RAM and I'm running ES on
each with 12gb heap (ES_HEAP_SIZE) and one of them (let's call it
logs1) is running logstash as well, with 2gb heap. The master node is logs1
and the other instance is logs2. I start the cluster and every things looks
fine, but after a while (1-3 days) I get the following error on logs1:
[2014-07-15 12:26:39,867][WARN ][transport.netty ] [Keen Marlow]
exception caught on transport layer [[id: 0x16801a48, /XX.XX.XXX.XX:36314
=> /XX.XXX.XX.XX:9300]], closing connection
java.lang.OutOfMemoryError: Direct buffer memory
...Stack Trace...
And then the cluster is no longer connected and if I try to restart
logs2, I get the same error above for logs1 and this one for logs2:
[2014-07-15 12:27:39,282][INFO ][discovery.ec2 ] [Betty Ross Banner]
failed to send join request to master [[Keen Marlow][9a7FIRpBSrKQcdcV_
sjSTw][ip-XX-XX-XXX-XX][inet[/XX.XX.XXX.XX:9300]]{aws_availability_zone=us-west-2a,
master=true}], reason [org.elasticsearch.transport.RemoteTransportException:
[Keen Marlow][inet[/XX.XX.XXX.XX:9300]][discovery/zen/join];
org.elasticsearch.transport.NodeDisconnectedException: [Betty Ross
Banner][inet[/XX.XXX.XX.XX:9300]][discovery/zen/join/validate]
disconnected]
Is there any memory configuration I should tune up a bit? I'm kind of
new to Elasticsearch so I'd love some help! :).
Thanks!
Pedro
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
On Wednesday, July 16, 2014 6:55:41 PM UTC+2, Ivan Brusic wrote:
Most users do not set the direct memory setting. mlockall is set, but does
the server allow it? You would see an error on startup if it didn't. Did
you change the vm swapiness on the server?
--
Ivan
On Wed, Jul 16, 2014 at 2:40 AM, Pedro Jerónimo <pedropr...@gmail.com
<javascript:>> wrote:
*Java: *java version "1.7.0_55"
*Elasticsearch: *1.2.1
On Wednesday, July 16, 2014 10:55:10 AM UTC+2, Jörg Prante wrote:
I have a cluster of 2 ES machines with a lot of indexing, not so much
searching. I'm using 2 EC2 machines with 30gb of RAM and I'm running ES on
each with 12gb heap (ES_HEAP_SIZE) and one of them (let's call it
logs1) is running logstash as well, with 2gb heap. The master node is logs1
and the other instance is logs2. I start the cluster and every things looks
fine, but after a while (1-3 days) I get the following error on logs1:
[2014-07-15 12:26:39,867][WARN ][transport.netty ] [Keen Marlow]
exception caught on transport layer [[id: 0x16801a48, /XX.XX.XXX.XX:36314
=> /XX.XXX.XX.XX:9300]], closing connection
java.lang.OutOfMemoryError: Direct buffer memory
...Stack Trace...
And then the cluster is no longer connected and if I try to restart
logs2, I get the same error above for logs1 and this one for logs2:
[2014-07-15 12:27:39,282][INFO ][discovery.ec2 ] [Betty Ross Banner]
failed to send join request to master [[Keen Marlow][9a7FIRpBSrKQcdcV_
sjSTw][ip-XX-XX-XXX-XX][inet[/XX.XX.XXX.XX:9300]]{aws_availability_zone=us-west-2a,
master=true}], reason [org.elasticsearch.transport.RemoteTransportException:
[Keen Marlow][inet[/XX.XX.XXX.XX:9300]][discovery/zen/join];
org.elasticsearch.transport.NodeDisconnectedException: [Betty Ross
Banner][inet[/XX.XXX.XX.XX:9300]][discovery/zen/join/validate]
disconnected]
Is there any memory configuration I should tune up a bit? I'm kind of
new to Elasticsearch so I'd love some help! :).
Thanks!
Pedro
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
