Hello all, I am using ES with Graylog2 indexer to index all my machines'
logs which are around 500 million a day. The problem manifests itself when
Graylog server stops indexing any new message into ES. The Graylog server
seems to be just fine and it feels like I am hitting somekind of threshold
on ES side. Here is my configuration :
I have 14 x ( 8 CPU, 64GB RAM, 1TB RAID 50 Array ) machines. Entire RAM is
dedicated to ES exclusively. I have 14 shards with 1 replica and with just
one Graylog2 Index and it's 99.99% write only index. With 1.5 billion
messages the disks were around 31% utilized. The CPUs were around 25%
utilized and Virtual memory was 62G but the resident memory was 40G only.
I am running 0.19.1 ES. And last week I moved to Graylog2 0.9.7 version
which has embedded ES client. My log messages throughput is around 500
million a day and I would like to keep atleast 1 week of data into the
I have not changed any JVM settings on ES except the Heap memory.
To resolve the problem, I have to delete the entire index everytime and
then things start working fine again until it the index reaches around 1.5
billion messages again. I have tried with more shards but same results
Any pointers what I should look at ?