# Hitting some limit on ElasticSearch **URL:** \ **Category:** Elasticsearch\ **Created:** [April 13, 2012, 2:01am UTC](https://discuss.elastic.co/t/hitting-some-limit-on-elasticsearch/7320 "2012-04-13T02:01:56Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Amit\_Mohan](https://avatars.discourse-cdn.com/v4/letter/a/9fc29f/32.png) [@Amit\_Mohan](https://discuss.elastic.co/u/Amit_Mohan)\ **Post date:** [April 13, 2012, 2:01am UTC](https://discuss.elastic.co/t/hitting-some-limit-on-elasticsearch/7320/1 "2012-04-13T02:01:56Z")
Hello all, I am using ES with Graylog2 indexer to index all my machines' logs which are around 500 million a day. The problem manifests itself when Graylog server stops indexing any new message into ES. The Graylog server seems to be just fine and it feels like I am hitting somekind of threshold on ES side. Here is my configuration : I have 14 x ( 8 CPU, 64GB RAM, 1TB RAID 50 Array ) machines. Entire RAM is dedicated to ES exclusively. I have 14 shards with 1 replica and with just one Graylog2 Index and it's 99.99% write only index. With 1.5 billion messages the disks were around 31% utilized. The CPUs were around 25% utilized and Virtual memory was 62G but the resident memory was 40G only. I am running 0.19.1 ES. And last week I moved to Graylog2 0.9.7 version which has embedded ES client. My log messages throughput is around 500 million a day and I would like to keep atleast 1 week of data into the cluster. I have not changed any JVM settings on ES except the Heap memory. To resolve the problem, I have to delete the entire index everytime and then things start working fine again until it the index reaches around 1.5 billion messages again. I have tried with more shards but same results everytime. Any pointers what I should look at ? Thanks, -Amit Mohan ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [April 13, 2012, 2:52am UTC](https://discuss.elastic.co/t/hitting-some-limit-on-elasticsearch/7320/2 "2012-04-13T02:52:01Z")
Hi Amit, > Entire RAM is dedicated to ES exclusively. If the above is true, then that's the problem. Lower your heap to, say, 8GB. And you grab SPM for Elasticsearch from [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) you'll see your shards bounce around the cluster after you restart your nodes. 🙂 ## Otis Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) Scalable Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) On Thursday, April 12, 2012 10:01:56 PM UTC-4, Amit Mohan wrote: > Hello all, I am using ES with Graylog2 indexer to index all my machines' > logs which are around 500 million a day. The problem manifests itself when > Graylog server stops indexing any new message into ES. The Graylog server > seems to be just fine and it feels like I am hitting somekind of threshold > on ES side. Here is my configuration : > > I have 14 x ( 8 CPU, 64GB RAM, 1TB RAID 50 Array ) machines. Entire RAM is > dedicated to ES exclusively. I have 14 shards with 1 replica and with just > one Graylog2 Index and it's 99.99% write only index. With 1.5 billion > messages the disks were around 31% utilized. The CPUs were around 25% > utilized and Virtual memory was 62G but the resident memory was 40G only. > > I am running 0.19.1 ES. And last week I moved to Graylog2 0.9.7 version > which has embedded ES client. My log messages throughput is around 500 > million a day and I would like to keep atleast 1 week of data into the > cluster. > > I have not changed any JVM settings on ES except the Heap memory. > > To resolve the problem, I have to delete the entire index everytime and > then things start working fine again until it the index reaches around 1.5 > billion messages again. I have tried with more shards but same results > everytime. > > Any pointers what I should look at ? > > Thanks, > -Amit Mohan ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [April 13, 2012, 12:45pm UTC](https://discuss.elastic.co/t/hitting-some-limit-on-elasticsearch/7320/3 "2012-04-13T12:45:19Z")
As Otis mentioned, I would recommend allocating around 22gb to start with to ES (and make sure you use latest 1.6/6 Java version). The reason I recommend to start with 22gb heap allocation is because in this case, the JVM can do memory optimizations to effectively take less memory (pointer compression). Next, I would recommend using rolling indices. I am not sure how graylog index the data, but for logging type data, its best to create an index per "time span", like day for example. The reason for that is the fact that deleting old indices is a snap, while deleting docs from an index will be considerably more expensive. You can always search over more than one index. But, it boils down to graylog and seeing if it supports it or not. On Fri, Apr 13, 2012 at 5:01 AM, Amit Mohan [amisakrenvan@gmail.com](mailto:amisakrenvan@gmail.com) wrote: > Hello all, I am using ES with Graylog2 indexer to index all my machines' > logs which are around 500 million a day. The problem manifests itself when > Graylog server stops indexing any new message into ES. The Graylog server > seems to be just fine and it feels like I am hitting somekind of threshold > on ES side. Here is my configuration : > > I have 14 x ( 8 CPU, 64GB RAM, 1TB RAID 50 Array ) machines. Entire RAM is > dedicated to ES exclusively. I have 14 shards with 1 replica and with just > one Graylog2 Index and it's 99.99% write only index. With 1.5 billion > messages the disks were around 31% utilized. The CPUs were around 25% > utilized and Virtual memory was 62G but the resident memory was 40G only. > > I am running 0.19.1 ES. And last week I moved to Graylog2 0.9.7 version > which has embedded ES client. My log messages throughput is around 500 > million a day and I would like to keep atleast 1 week of data into the > cluster. > > I have not changed any JVM settings on ES except the Heap memory. > > To resolve the problem, I have to delete the entire index everytime and > then things start working fine again until it the index reaches around 1.5 > billion messages again. I have tried with more shards but same results > everytime. > > Any pointers what I should look at ? > > Thanks, > -Amit Mohan ---
**Author:** ![Amit\_Mohan](https://avatars.discourse-cdn.com/v4/letter/a/9fc29f/32.png) [@Amit\_Mohan](https://discuss.elastic.co/u/Amit_Mohan)\ **Post date:** [April 13, 2012, 1:54pm UTC](https://discuss.elastic.co/t/hitting-some-limit-on-elasticsearch/7320/4 "2012-04-13T13:54:23Z")
Thanks guys! I am going to give 22GB RAM to ES and see how it behaves over next couple days. Will post the results here as and when I have those. Otis : Thanks for pointing me to Sematext ! It certainly looks something very interesting and I am going to give it a try. -Amit Mohan On Friday, April 13, 2012 8:45:19 AM UTC-4, kimchy wrote: > As Otis mentioned, I would recommend allocating around 22gb to start with > to ES (and make sure you use latest 1.6/6 Java version). The reason I > recommend to start with 22gb heap allocation is because in this case, the > JVM can do memory optimizations to effectively take less memory (pointer > compression). > > Next, I would recommend using rolling indices. I am not sure how graylog > index the data, but for logging type data, its best to create an index per > "time span", like day for example. The reason for that is the fact that > deleting old indices is a snap, while deleting docs from an index will be > considerably more expensive. You can always search over more than one > index. But, it boils down to graylog and seeing if it supports it or not. > > On Fri, Apr 13, 2012 at 5:01 AM, Amit Mohan [amisakrenvan@gmail.com](mailto:amisakrenvan@gmail.com)wrote: > > > Hello all, I am using ES with Graylog2 indexer to index all my machines' > > logs which are around 500 million a day. The problem manifests itself when > > Graylog server stops indexing any new message into ES. The Graylog server > > seems to be just fine and it feels like I am hitting somekind of threshold > > on ES side. Here is my configuration : > > > > I have 14 x ( 8 CPU, 64GB RAM, 1TB RAID 50 Array ) machines. Entire RAM > > is dedicated to ES exclusively. I have 14 shards with 1 replica and with > > just one Graylog2 Index and it's 99.99% write only index. With 1.5 billion > > messages the disks were around 31% utilized. The CPUs were around 25% > > utilized and Virtual memory was 62G but the resident memory was 40G only. > > > > I am running 0.19.1 ES. And last week I moved to Graylog2 0.9.7 version > > which has embedded ES client. My log messages throughput is around 500 > > million a day and I would like to keep atleast 1 week of data into the > > cluster. > > > > I have not changed any JVM settings on ES except the Heap memory. > > > > To resolve the problem, I have to delete the entire index everytime and > > then things start working fine again until it the index reaches around 1.5 > > billion messages again. I have tried with more shards but same results > > everytime. > > > > Any pointers what I should look at ? > > > > Thanks, > > -Amit Mohan ---
**Author:** ![Radu\_Gheorghe1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe1/32/2688_2.png) [@Radu\_Gheorghe1](https://discuss.elastic.co/u/Radu_Gheorghe1)\ **Post date:** [April 14, 2012, 4:20pm UTC](https://discuss.elastic.co/t/hitting-some-limit-on-elasticsearch/7320/5 "2012-04-14T16:20:47Z")
Hi Amit, You might also find something helpful from here: [http://elasticsearch-users.115913.n3.nabble.com/Problems-with-GrayLog2-ES-setup-long-td3859362.html](http://elasticsearch-users.115913.n3.nabble.com/Problems-with-GrayLog2-ES-setup-long-td3859362.html) On Apr 13, 3:54 pm, Amit Mohan [amisakren...@gmail.com](mailto:amisakren...@gmail.com) wrote: > Thanks guys! I am going to give 22GB RAM to ES and see how it behaves over > next couple days. Will post the results here as and when I have those. > > Otis : Thanks for pointing me to Sematext ! It certainly looks something > very interesting and I am going to give it a try. > > -Amit Mohan > > On Friday, April 13, 2012 8:45:19 AM UTC-4, kimchy wrote: > > > As Otis mentioned, I would recommend allocating around 22gb to start with > > to ES (and make sure you use latest 1.6/6 Java version). The reason I > > recommend to start with 22gb heap allocation is because in this case, the > > JVM can do memory optimizations to effectively take less memory (pointer > > compression). > > > Next, I would recommend using rolling indices. I am not sure how graylog > > index the data, but for logging type data, its best to create an index per > > "time span", like day for example. The reason for that is the fact that > > deleting old indices is a snap, while deleting docs from an index will be > > considerably more expensive. You can always search over more than one > > index. But, it boils down to graylog and seeing if it supports it or not. > > > On Fri, Apr 13, 2012 at 5:01 AM, Amit Mohan [amisakren...@gmail.com](mailto:amisakren...@gmail.com)wrote: > > > > Hello all, I am using ES with Graylog2 indexer to index all my machines' > > > logs which are around 500 million a day. The problem manifests itself when > > > Graylog server stops indexing any new message into ES. The Graylog server > > > seems to be just fine and it feels like I am hitting somekind of threshold > > > on ES side. Here is my configuration : > > > > I have 14 x ( 8 CPU, 64GB RAM, 1TB RAID 50 Array ) machines. Entire RAM > > > is dedicated to ES exclusively. I have 14 shards with 1 replica and with > > > just one Graylog2 Index and it's 99.99% write only index. With 1.5 billion > > > messages the disks were around 31% utilized. The CPUs were around 25% > > > utilized and Virtual memory was 62G but the resident memory was 40G only. > > > > I am running 0.19.1 ES. And last week I moved to Graylog2 0.9.7 version > > > which has embedded ES client. My log messages throughput is around 500 > > > million a day and I would like to keep atleast 1 week of data into the > > > cluster. > > > > I have not changed any JVM settings on ES except the Heap memory. > > > > To resolve the problem, I have to delete the entire index everytime and > > > then things start working fine again until it the index reaches around 1.5 > > > billion messages again. I have tried with more shards but same results > > > everytime. > > > > Any pointers what I should look at ? > > > > Thanks, > > > -Amit Mohan ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:32am UTC](https://discuss.elastic.co/t/hitting-some-limit-on-elasticsearch/7320/6 "2017-07-06T03:32:34Z")