Hi all,
We've been running an ElasticSearch cluster of three nodes since last
December. We're running them on Debian Wheezy. Due to the size of our
network, we're getting about 600 messages/s (800 at peak times). Using
logstash and daily indices, we're currently at about 3TB of data spread
over 158 indices. We use 2 replicas, so all machines have all data.
I'm struggling to understand what expected working limits are for
ElasticSearch in a centralized log situation and would really appreciate
input from others. What we eventually try to do is provide about 13 months
of searchable logs via Kibana, but we're mostly running into RAM
constraints. Each node has 3TB of data, but the heap usage is almost
constantly up at 27-29GB.
We did have a problem with garbage collects earlier, which took so long
that a node would drop from the cluster. To fix this, we switched to the G1
gc, which seems very well suited for this operation. The machines we're
using have 12 cores, so the added CPU overhead is negligible (general usage
of the cores is less than 30%). But I'm not enough of a Java dev to judge
if this switch could be the cause of the constant high heap usage. We're
currently at about 1GB RAM per 100GB of disk usage.
My questions:
1- Is 1GB RAM usage per 100GB disk usage an expected usage pattern for an
index heavy cluster?
2- Aside from closing indices, are there other ways of lowering this?
2.5- Should I worry about it?
3- Are we approaching this the wrong way and should we change our setup?
4- Would upgrading to 1.3.1 change the usage significantly, due to the fix
in issue 6856 or is it unrelated?
The numbers:
3 nodes with the following config:
- 92GB of RAM
- 12 cores with HT
- 6x 3.6TB spinning disks (we're contemplating adding CacheCade, since the
controller supports it)- We expose the disks to elasticsearch via 6 mount points
- Debian Wheezy
- Java 1.7.0_65 OpenJDK JRE with IcedTea 2.5.1 (Debian package with version
7u65-2.5.1-2~deb7u1) - ElasticSearch 1.2.1 from the ElasticSearch repositories
Config snippets (leaving out ips and the like):
bootstrap:
mlockall: true
cluster:
name: logging
routing:
allocation:
node_concurrent_recoveries: 4
discovery:
zen:
minimum_master_nodes: 2
ping:
unicast:
hosts:
[snip]
index:
number_of_replicas: 2
number_of_shards: 6
indices:
memory:
index_buffer_size: 50%
recovery:
concurrent_streams: 5
max_bytes_per_sec: 100mb
node:
concurrent_recoveries: 6
name: stor1-stor
path:
data:
- /srv/elasticsearch/a
- /srv/elasticsearch/b
- /srv/elasticsearch/c
- /srv/elasticsearch/d
- /srv/elasticsearch/f
- /srv/elasticsearch/e
Java options: -Xms30g -Xmx30g -Xss256k -Djava.awt.headless=true
-XX:+UseG1GC -XX:+HeapDumpOnOutOfMemoryError
Thanks for your help! Please let me know if you need more information
--
Kind regards,
Tim Stoop
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2846daa5-a75c-4d04-ab34-957984c9e05e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.