elasticsearch is set as a single node instance on a 60G RAM and 32*2.6GHz
machine. I am actively indexing historic data with logstash. It worked well
with ~300 million documents (search and indexing were doing ok) , but all
of a sudden es fails to starts and keep itself up. It starts for few
minutes and I can query but fails with out of memory error. I monitor the
memory and atleast 12G of memory is available when it fails. I had set the
es_heap_size to 31G and then reduced it to 28, 24 and 18 and the same error
every time (see dump below)
*My security limits are as under (this is a test/POC server thus "root"
user) *
root soft nofile 65536
root hard nofile 65536
root - memlock unlimited
*ES settings *
config]# grep -v "^#" elasticsearch.yml | grep -v "^$"
bootstrap.mlockall: true
echo $ES_HEAP_SIZE
18432m
---DUMP----
bin/elasticsearch
[2014-05-04 13:30:12,653][INFO ][node ] [Sabretooth]
version[1.1.1], pid[19309], build[f1585f0/2014-04-16T14:27:12Z]
[2014-05-04 13:30:12,653][INFO ][node ] [Sabretooth]
initializing ...
[2014-05-04 13:30:12,669][INFO ][plugins ] [Sabretooth]
loaded [], sites []
[2014-05-04 13:30:15,390][INFO ][node ] [Sabretooth]
initialized
[2014-05-04 13:30:15,390][INFO ][node ] [Sabretooth]
starting ...
[2014-05-04 13:30:15,531][INFO ][transport ] [Sabretooth]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/10.109.136.59:9300]}
[2014-05-04 13:30:18,553][INFO ][cluster.service ] [Sabretooth]
new_master
[Sabretooth][eocFkTYMQnSTUar94A2vHw][ip-10-109-136-59][inet[/10.109.136.59:9300]],
reason: zen-disco-join (elected_as_master)
[2014-05-04 13:30:18,579][INFO ][discovery ] [Sabretooth]
elasticsearch/eocFkTYMQnSTUar94A2vHw
[2014-05-04 13:30:18,790][INFO ][http ] [Sabretooth]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/10.109.136.59:9200]}
[2014-05-04 13:30:19,976][INFO ][gateway ] [Sabretooth]
recovered [278] indices into cluster_state
[2014-05-04 13:30:19,984][INFO ][node ] [Sabretooth]
started
OpenJDK 64-Bit Server VM warning: Attempt to protect stack guard pages
failed.
OpenJDK 64-Bit Server VM warning: Attempt to deallocate stack guard pages
failed.
OpenJDK 64-Bit Server VM warning: INFO:
os::commit_memory(0x00000007f7c70000, 196608, 0) failed; error='Cannot
allocate memory' (errno=12)
There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (malloc) failed to allocate 196608 bytes for
committing reserved memory.
An error report file with more information is saved as:
/tmp/jvm-19309/hs_error.log
*user untergeek on #logstash told me that I have reached a max number of
indices on a single node. Here are my questions: *
- Can I move half of my indexes to a new node ? If yes, how to do that
without compromising indexes - Logstash makes 1 index per day and I want to have 2 years of data
indexable ; Can I combine multiple indexes into one ? Like one month per
month : this will mean I will not have more than 24 indexes. - How many nodes are ideal for 24 moths of data ~1.5G a day
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f1d8685d-b230-47c8-b52c-71808545059c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.