# Large Scale elastic Search Logstash collection system **URL:** \ **Category:** Elasticsearch\ **Created:** [August 13, 2014, 3:18pm UTC](https://discuss.elastic.co/t/large-scale-elastic-search-logstash-collection-system/19261 "2014-08-13T15:18:55Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Robert\_Gardam](https://avatars.discourse-cdn.com/v4/letter/r/919ad9/32.png) [@Robert\_Gardam](https://discuss.elastic.co/u/Robert_Gardam)\ **Post date:** [August 13, 2014, 3:18pm UTC](https://discuss.elastic.co/t/large-scale-elastic-search-logstash-collection-system/19261/1 "2014-08-13T15:18:55Z")
Hello We have a 10 node elasticsearch cluster which is receieving roughly 10k/s worth of logs lines from our application. Each elasticsearch node has 132gb of memory - 48gb heap size, the disk subsystem is not great, but it seems to be keeping up. (This could be an issue, but i'm not sure that it is) The logs path is: app server -\> redis (via logstash) -\> logstash filters (3 dedicated boxes) -\> elasticsearch\_http We currently bulk import from logstash at 5k documents per flush to keep up with the volume of data that comes in. Here are the es non standard configs. indices.memory.index\_buffer\_size: 50% index.translog.flush\_threshold\_ops: 50000 # Refresh tuning. index.refresh\_interval: 15s # Field Data cache tuning indices.fielddata.cache.size: 24g indices.fielddata.cache.expire: 10m #Segment Merging Tuning index.merge.policy.max\_merged\_segment: 15g # Thread Tuning threadpool: bulk: type: fixed queue\_size: -1 We have not had this cluster stay up for more than a week, but it also seems to crash for no real reason. It seems like one node starts having issues and then it takes the entire cluster down. Does anyone from the community have any experience with this kind of setup? Thanks in Advance, Rob -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d04a643e-990b-40b0-b230-2ba560f08eea%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d04a643e-990b-40b0-b230-2ba560f08eea%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [August 13, 2014, 4:09pm UTC](https://discuss.elastic.co/t/large-scale-elastic-search-logstash-collection-system/19261/2 "2014-08-13T16:09:39Z")
Because you set queue\_size: -1 in the bulk thread pool, you explicitly allowed the node to crash. You should use reasonable resource limits. Default settings, which are reasonable, are sufficient in most cases. Jörg On Wed, Aug 13, 2014 at 5:18 PM, Robert Gardam [robert.gardam@fyber.com](mailto:robert.gardam@fyber.com) wrote: > Hello > > We have a 10 node elasticsearch cluster which is receieving roughly 10k/s > worth of logs lines from our application. > > Each elasticsearch node has 132gb of memory - 48gb heap size, the disk > subsystem is not great, but it seems to be keeping up. (This could be an > issue, but i'm not sure that it is) > > The logs path is: > > app server -\> redis (via logstash) -\> logstash filters (3 dedicated boxes) > -\> elasticsearch\_http > > We currently bulk import from logstash at 5k documents per flush to keep > up with the volume of data that comes in. > > Here are the es non standard configs. > > indices.memory.index\_buffer\_size: 50% > index.translog.flush\_threshold\_ops: 50000 > > # Refresh tuning. > > index.refresh\_interval: 15s > > # Field Data cache tuning > > indices.fielddata.cache.size: 24g > indices.fielddata.cache.expire: 10m > #Segment Merging Tuning > index.merge.policy.max\_merged\_segment: 15g > > # Thread Tuning > > threadpool: > bulk: > type: fixed > queue\_size: -1 > > We have not had this cluster stay up for more than a week, but it also > seems to crash for no real reason. > > It seems like one node starts having issues and then it takes the entire > cluster down. > > Does anyone from the community have any experience with this kind of setup? > > Thanks in Advance, > Rob > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/d04a643e-990b-40b0-b230-2ba560f08eea%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d04a643e-990b-40b0-b230-2ba560f08eea%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/d04a643e-990b-40b0-b230-2ba560f08eea%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/d04a643e-990b-40b0-b230-2ba560f08eea%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEU84GhzjUuuo69K2xeu1N2-nQPCJXXCj3wu20Mz0R4VA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEU84GhzjUuuo69K2xeu1N2-nQPCJXXCj3wu20Mz0R4VA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Robert\_Gardam](https://avatars.discourse-cdn.com/v4/letter/r/919ad9/32.png) [@Robert\_Gardam](https://discuss.elastic.co/u/Robert_Gardam)\ **Post date:** [August 13, 2014, 4:50pm UTC](https://discuss.elastic.co/t/large-scale-elastic-search-logstash-collection-system/19261/3 "2014-08-13T16:50:52Z")
Hi, The reason this is set is because without it we reject messages and there fore don't have all the log entries. I'm happy to be told this isn't required, but i'm pretty sure it is. We are constantly bulk indexing large numbers of events. On Wednesday, August 13, 2014 6:09:46 PM UTC+2, Jörg Prante wrote: > Because you set queue\_size: -1 in the bulk thread pool, you explicitly > allowed the node to crash. > > You should use reasonable resource limits. Default settings, which are > reasonable, are sufficient in most cases. > > Jörg > > On Wed, Aug 13, 2014 at 5:18 PM, Robert Gardam \<[robert...@fyber.com](mailto:robert...@fyber.com) > \\> wrote: > > > Hello > > > > We have a 10 node elasticsearch cluster which is receieving roughly 10k/s > > worth of logs lines from our application. > > > > Each elasticsearch node has 132gb of memory - 48gb heap size, the disk > > subsystem is not great, but it seems to be keeping up. (This could be an > > issue, but i'm not sure that it is) > > > > The logs path is: > > > > app server -\> redis (via logstash) -\> logstash filters (3 dedicated > > boxes) -\> elasticsearch\_http > > > > We currently bulk import from logstash at 5k documents per flush to keep > > up with the volume of data that comes in. > > > > Here are the es non standard configs. > > > > indices.memory.index\_buffer\_size: 50% > > index.translog.flush\_threshold\_ops: 50000 > > > > # Refresh tuning. > > > > index.refresh\_interval: 15s > > > > # Field Data cache tuning > > > > indices.fielddata.cache.size: 24g > > indices.fielddata.cache.expire: 10m > > #Segment Merging Tuning > > index.merge.policy.max\_merged\_segment: 15g > > > > # Thread Tuning > > > > threadpool: > > bulk: > > type: fixed > > queue\_size: -1 > > > > We have not had this cluster stay up for more than a week, but it also > > seems to crash for no real reason. > > > > It seems like one node starts having issues and then it takes the entire > > cluster down. > > > > Does anyone from the community have any experience with this kind of > > setup? > > > > Thanks in Advance, > > Rob > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/d04a643e-990b-40b0-b230-2ba560f08eea%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d04a643e-990b-40b0-b230-2ba560f08eea%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/d04a643e-990b-40b0-b230-2ba560f08eea%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/d04a643e-990b-40b0-b230-2ba560f08eea%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/be02ed79-31de-4002-9144-d124370d0c31%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/be02ed79-31de-4002-9144-d124370d0c31%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [August 13, 2014, 5:07pm UTC](https://discuss.elastic.co/t/large-scale-elastic-search-logstash-collection-system/19261/4 "2014-08-13T17:07:52Z")
If Elasticsearch rejects bulk actions, this is serious and you should examine the cluster to find out why this is so. From slow disks, cluster health, or capacity problems, everything comes to mind. But if you ignore problem solution and merely disable bulk resource control instead, you open the gate wide to unpredictable node crashes, and you won't be able to control the cluster at a certain point. To reduce the number of active bulk requests per timeframe, for example, you could increase the bulk request actions per request. Or simply increase the number of nodes. Or think about the shard/replica organization while indexing - it can be an advantage to bulk index to replica level 0 index only and increase the replica level later. Jörg On Wed, Aug 13, 2014 at 6:50 PM, Robert Gardam [robert.gardam@fyber.com](mailto:robert.gardam@fyber.com) wrote: > Hi, > The reason this is set is because without it we reject messages and there > fore don't have all the log entries. > > I'm happy to be told this isn't required, but i'm pretty sure it is. We > are constantly bulk indexing large numbers of events. > > On Wednesday, August 13, 2014 6:09:46 PM UTC+2, Jörg Prante wrote: > > > Because you set queue\_size: -1 in the bulk thread pool, you explicitly > > allowed the node to crash. > > > > You should use reasonable resource limits. Default settings, which are > > reasonable, are sufficient in most cases. > > > > Jörg > > > > On Wed, Aug 13, 2014 at 5:18 PM, Robert Gardam [robert...@fyber.com](mailto:robert...@fyber.com) > > wrote: > > > > > Hello > > > > > > We have a 10 node elasticsearch cluster which is receieving roughly > > > 10k/s worth of logs lines from our application. > > > > > > Each elasticsearch node has 132gb of memory - 48gb heap size, the disk > > > subsystem is not great, but it seems to be keeping up. (This could be an > > > issue, but i'm not sure that it is) > > > > > > The logs path is: > > > > > > app server -\> redis (via logstash) -\> logstash filters (3 dedicated > > > boxes) -\> elasticsearch\_http > > > > > > We currently bulk import from logstash at 5k documents per flush to keep > > > up with the volume of data that comes in. > > > > > > Here are the es non standard configs. > > > > > > indices.memory.index\_buffer\_size: 50% > > > index.translog.flush\_threshold\_ops: 50000 > > > > > > # Refresh tuning. > > > > > > index.refresh\_interval: 15s > > > > > > # Field Data cache tuning > > > > > > indices.fielddata.cache.size: 24g > > > indices.fielddata.cache.expire: 10m > > > #Segment Merging Tuning > > > index.merge.policy.max\_merged\_segment: 15g > > > > > > # Thread Tuning > > > > > > threadpool: > > > bulk: > > > type: fixed > > > queue\_size: -1 > > > > > > We have not had this cluster stay up for more than a week, but it also > > > seems to crash for no real reason. > > > > > > It seems like one node starts having issues and then it takes the entire > > > cluster down. > > > > > > Does anyone from the community have any experience with this kind of > > > setup? > > > > > > Thanks in Advance, > > > Rob > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/d04a643e-990b-40b0-b230-2ba560f08eea% > > > [40googlegroups.com](http://40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/d04a643e-990b-40b0-b230-2ba560f08eea%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/d04a643e-990b-40b0-b230-2ba560f08eea%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/be02ed79-31de-4002-9144-d124370d0c31%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/be02ed79-31de-4002-9144-d124370d0c31%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/be02ed79-31de-4002-9144-d124370d0c31%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/be02ed79-31de-4002-9144-d124370d0c31%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFUPQBUvrsFYtQcWZ0W%3DnLZz\_LasQ1T9p0XTx%3DqoesNMw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFUPQBUvrsFYtQcWZ0W%3DnLZz_LasQ1T9p0XTx%3DqoesNMw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Robert\_Gardam](https://avatars.discourse-cdn.com/v4/letter/r/919ad9/32.png) [@Robert\_Gardam](https://discuss.elastic.co/u/Robert_Gardam)\ **Post date:** [August 13, 2014, 5:24pm UTC](https://discuss.elastic.co/t/large-scale-elastic-search-logstash-collection-system/19261/5 "2014-08-13T17:24:09Z")
I appreciate your answers. I think IO could be a contributing factor. I'm thinking of splitting the index into an hourly index with no replicas for bulk importing and then switch it on afterwards. I think the risk of loosing data would be too high if it was any longer than that. Also Does the async replication from the logstash side of things cause unknown issues? On Wednesday, August 13, 2014 7:08:05 PM UTC+2, Jörg Prante wrote: > If Elasticsearch rejects bulk actions, this is serious and you should > examine the cluster to find out why this is so. From slow disks, cluster > health, or capacity problems, everything comes to mind. But if you ignore > problem solution and merely disable bulk resource control instead, you open > the gate wide to unpredictable node crashes, and you won't be able to > control the cluster at a certain point. > > To reduce the number of active bulk requests per timeframe, for example, > you could increase the bulk request actions per request. Or simply increase > the number of nodes. Or think about the shard/replica organization while > indexing - it can be an advantage to bulk index to replica level 0 index > only and increase the replica level later. > > Jörg > > On Wed, Aug 13, 2014 at 6:50 PM, Robert Gardam \<[robert...@fyber.com](mailto:robert...@fyber.com) > \\> wrote: > > > Hi, > > The reason this is set is because without it we reject messages and there > > fore don't have all the log entries. > > > > I'm happy to be told this isn't required, but i'm pretty sure it is. We > > are constantly bulk indexing large numbers of events. > > > > On Wednesday, August 13, 2014 6:09:46 PM UTC+2, Jörg Prante wrote: > > > > > Because you set queue\_size: -1 in the bulk thread pool, you explicitly > > > allowed the node to crash. > > > > > > You should use reasonable resource limits. Default settings, which are > > > reasonable, are sufficient in most cases. > > > > > > Jörg > > > > > > On Wed, Aug 13, 2014 at 5:18 PM, Robert Gardam [robert...@fyber.com](mailto:robert...@fyber.com) > > > wrote: > > > > > > > Hello > > > > > > > > We have a 10 node elasticsearch cluster which is receieving roughly > > > > 10k/s worth of logs lines from our application. > > > > > > > > Each elasticsearch node has 132gb of memory - 48gb heap size, the disk > > > > subsystem is not great, but it seems to be keeping up. (This could be an > > > > issue, but i'm not sure that it is) > > > > > > > > The logs path is: > > > > > > > > app server -\> redis (via logstash) -\> logstash filters (3 dedicated > > > > boxes) -\> elasticsearch\_http > > > > > > > > We currently bulk import from logstash at 5k documents per flush to > > > > keep up with the volume of data that comes in. > > > > > > > > Here are the es non standard configs. > > > > > > > > indices.memory.index\_buffer\_size: 50% > > > > index.translog.flush\_threshold\_ops: 50000 > > > > > > > > # Refresh tuning. > > > > > > > > index.refresh\_interval: 15s > > > > > > > > # Field Data cache tuning > > > > > > > > indices.fielddata.cache.size: 24g > > > > indices.fielddata.cache.expire: 10m > > > > #Segment Merging Tuning > > > > index.merge.policy.max\_merged\_segment: 15g > > > > > > > > # Thread Tuning > > > > > > > > threadpool: > > > > bulk: > > > > type: fixed > > > > queue\_size: -1 > > > > > > > > We have not had this cluster stay up for more than a week, but it also > > > > seems to crash for no real reason. > > > > > > > > It seems like one node starts having issues and then it takes the > > > > entire cluster down. > > > > > > > > Does anyone from the community have any experience with this kind of > > > > setup? > > > > > > > > Thanks in Advance, > > > > Rob > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/d04a643e-990b-40b0-b230-2ba560f08eea% > > > > [40googlegroups.com](http://40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/d04a643e-990b-40b0-b230-2ba560f08eea%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/d04a643e-990b-40b0-b230-2ba560f08eea%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/be02ed79-31de-4002-9144-d124370d0c31%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/be02ed79-31de-4002-9144-d124370d0c31%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/be02ed79-31de-4002-9144-d124370d0c31%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/be02ed79-31de-4002-9144-d124370d0c31%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/29de6d50-0798-490d-903f-631e4d47a7a5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/29de6d50-0798-490d-903f-631e4d47a7a5%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [August 17, 2014, 12:03pm UTC](https://discuss.elastic.co/t/large-scale-elastic-search-logstash-collection-system/19261/6 "2014-08-17T12:03:12Z")
Hi Robert, Or maybe it's worth rethinking the architecture to avoid having to do tricks like no-replicas for 1h. Kafka in front of ES comes to mind. We use this setup for Logsene [http://sematext.com/logsene/](http://sematext.com/logsene/) and don't have the problem with log loss, so it may work well for you, too. I think you could also replace Redis + 3 Logstash servers with 1 rsyslog server with omelasticsearch, which has built-in buffering in memory and on disk (see links below for config examples). Some pointers that may be helpful: - [Blog - Sematext Community](http://blog.sematext.com/?s=omelasticsearch) - [Sematext ElasticSearch Services](http://blog.sematext.com/2013/07/01/recipe-rsyslog-elasticsearch-kibana/) ## Otis Elasticsearch Performance Monitoring \* Log Analytics \* Search Analytics Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) On Wednesday, August 13, 2014 7:24:09 PM UTC+2, Robert Gardam wrote: > I appreciate your answers. I think IO could be a contributing factor. I'm > thinking of splitting the index into an hourly index with no replicas for > bulk importing and then switch it on afterwards. > > I think the risk of loosing data would be too high if it was any longer > than that. Also Does the async replication from the logstash side of things > cause unknown issues? > > On Wednesday, August 13, 2014 7:08:05 PM UTC+2, Jörg Prante wrote: > > > If Elasticsearch rejects bulk actions, this is serious and you should > > examine the cluster to find out why this is so. From slow disks, cluster > > health, or capacity problems, everything comes to mind. But if you ignore > > problem solution and merely disable bulk resource control instead, you open > > the gate wide to unpredictable node crashes, and you won't be able to > > control the cluster at a certain point. > > > > To reduce the number of active bulk requests per timeframe, for example, > > you could increase the bulk request actions per request. Or simply increase > > the number of nodes. Or think about the shard/replica organization while > > indexing - it can be an advantage to bulk index to replica level 0 index > > only and increase the replica level later. > > > > Jörg > > > > On Wed, Aug 13, 2014 at 6:50 PM, Robert Gardam [robert...@fyber.com](mailto:robert...@fyber.com) > > wrote: > > > > > Hi, > > > The reason this is set is because without it we reject messages and > > > there fore don't have all the log entries. > > > > > > I'm happy to be told this isn't required, but i'm pretty sure it is. We > > > are constantly bulk indexing large numbers of events. > > > > > > On Wednesday, August 13, 2014 6:09:46 PM UTC+2, Jörg Prante wrote: > > > > > > > Because you set queue\_size: -1 in the bulk thread pool, you explicitly > > > > allowed the node to crash. > > > > > > > > You should use reasonable resource limits. Default settings, which are > > > > reasonable, are sufficient in most cases. > > > > > > > > Jörg > > > > > > > > On Wed, Aug 13, 2014 at 5:18 PM, Robert Gardam [robert...@fyber.com](mailto:robert...@fyber.com) > > > > wrote: > > > > > > > > > Hello > > > > > > > > > > We have a 10 node elasticsearch cluster which is receieving roughly > > > > > 10k/s worth of logs lines from our application. > > > > > > > > > > Each elasticsearch node has 132gb of memory - 48gb heap size, the disk > > > > > subsystem is not great, but it seems to be keeping up. (This could be an > > > > > issue, but i'm not sure that it is) > > > > > > > > > > The logs path is: > > > > > > > > > > app server -\> redis (via logstash) -\> logstash filters (3 dedicated > > > > > boxes) -\> elasticsearch\_http > > > > > > > > > > We currently bulk import from logstash at 5k documents per flush to > > > > > keep up with the volume of data that comes in. > > > > > > > > > > Here are the es non standard configs. > > > > > > > > > > indices.memory.index\_buffer\_size: 50% > > > > > index.translog.flush\_threshold\_ops: 50000 > > > > > > > > > > # Refresh tuning. > > > > > > > > > > index.refresh\_interval: 15s > > > > > > > > > > # Field Data cache tuning > > > > > > > > > > indices.fielddata.cache.size: 24g > > > > > indices.fielddata.cache.expire: 10m > > > > > #Segment Merging Tuning > > > > > index.merge.policy.max\_merged\_segment: 15g > > > > > > > > > > # Thread Tuning > > > > > > > > > > threadpool: > > > > > bulk: > > > > > type: fixed > > > > > queue\_size: -1 > > > > > > > > > > We have not had this cluster stay up for more than a week, but it also > > > > > seems to crash for no real reason. > > > > > > > > > > It seems like one node starts having issues and then it takes the > > > > > entire cluster down. > > > > > > > > > > Does anyone from the community have any experience with this kind of > > > > > setup? > > > > > > > > > > Thanks in Advance, > > > > > Rob > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/d04a643e-990b-40b0-b230-2ba560f08eea% > > > > > [40googlegroups.com](http://40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/d04a643e-990b-40b0-b230-2ba560f08eea%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/d04a643e-990b-40b0-b230-2ba560f08eea%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/be02ed79-31de-4002-9144-d124370d0c31%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/be02ed79-31de-4002-9144-d124370d0c31%40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/be02ed79-31de-4002-9144-d124370d0c31%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/be02ed79-31de-4002-9144-d124370d0c31%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5ab612b8-31c3-47bd-b8a5-8687c7fc9108%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5ab612b8-31c3-47bd-b8a5-8687c7fc9108%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:08am UTC](https://discuss.elastic.co/t/large-scale-elastic-search-logstash-collection-system/19261/7 "2017-07-06T01:08:07Z")