I'm having troubles with a my GrayLog2 + ES 0.19.0 installation. It
seems I've hit a wall. Java is CPU bound and there's next to no disk
As I believe that the cure could be tuning ES or adding hardware
resources, I'm posting to this list.
The indexing capacity I would like is:
- Documents: long syslog mesages (160 chars avg) from a busy mail
- Message load exceeding 200 messages/second
- Capacity for storing 60 days worth of log messages, that's about 1
- I currently have some 800 million mesages on disk, using about 5x140
gigs of disk.
My setup is ( I'm using 64 bit Redhat Linux)
- One VM running GrayLog2 server + a ES server with no local shards.
- Two VMs with a total of 5 shards distributed with 3 / 2 on each.
The two VMs with shards have 32 gigs of memory and 4 cores in a VMware
I've applied the following changes to the bin/elasticsearch script:
ulimit -n 60000 # fs.file-max = 131000
ulimit -l unlimited
export JAVA_HOME=/usr/java/latest # SUN JRE 1.6.31
and disabled the swap area to prevent swapping.
Java on the two VMs containing shards, use about 12 gigs out memory
and all CPU resources.
The first VM, is not significantly loaded.
What could a solution be -- "Kiwi" (kill it with Iron) or should I
rather change the ES configuration?