# Logstash limitting ElasticSearch heap **URL:** \ **Category:** Elasticsearch\ **Created:** [June 18, 2014, 12:53pm UTC](https://discuss.elastic.co/t/logstash-limitting-elasticsearch-heap/18180 "2014-06-18T12:53:50Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Antonio\_Augusto\_Sant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/antonio_augusto_sant/32/82851_2.png) [@Antonio\_Augusto\_Sant](https://discuss.elastic.co/u/Antonio_Augusto_Sant)\ **Post date:** [June 18, 2014, 12:53pm UTC](https://discuss.elastic.co/t/logstash-limitting-elasticsearch-heap/18180/1 "2014-06-18T12:53:50Z")
Hello, I think I'm hitting some kind of wall here... I'm running logstash on a syslog server. It receives logs from about 150 machines and also a LOT of iptables logs, and sending it to ElasticSearch. But, I think I'm not hitting all speed that I should. My Logstash throughput tops at about 1.000 events/s, and it looks like my ES servers (I've 2) are really light. On logstash I've three configs (syslog, ossec and iptables), so I get three new nodes on my cluster. I've set up LS Heap Size to be 2G, but according to bigdesk, the ES module is getting only about 150MB, and its generating a LOT of GC. Bellow the screenshot for big desk: [image: bigdesk] [https://cloud.githubusercontent.com/assets/6423413/3261580/5bc4faf2-f25b-11e3-8529-df0eee61b1e5.png](https://cloud.githubusercontent.com/assets/6423413/3261580/5bc4faf2-f25b-11e3-8529-df0eee61b1e5.png) And here the logstash process I'm running: _#ps -ef | grep logstash logstash 13371 1 99 14:42 pts/0 00:29:37 /usr/bin/java -Djava.io.tmpdir=/opt/logstash/tmp -Xmx2g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintClassHistogram -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:./logstash-gc.log -jar /opt/logstash/vendor/jar/jruby-complete-1.7.11.jar -I/opt/logstash/lib /opt/logstash/lib/logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log_ My Syslog/LS memory usage seens very light as well (its a 4 core VM), but the logstash process is always topping in about 150% - 200% \*# free -m total used free shared buffers cached Mem: 7872 2076 5795 0 39 1502 -/+ buffers/cache: 534 7337 Swap: 1023 8 1015 # uptime 15:02:04 up 23:52, 1 user, load average: 1.39, 1.12, 0.96\* Any ideas what I can do to increase the indexing performance? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b6346f68-1c17-4699-8ad0-cb9121b5c7cb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b6346f68-1c17-4699-8ad0-cb9121b5c7cb%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 19, 2014, 8:32am UTC](https://discuss.elastic.co/t/logstash-limitting-elasticsearch-heap/18180/2 "2014-06-19T08:32:28Z")
Lots of GC isn't bad, you want to see a lot of small GCs rather than stop-the-world sort of ones which can bring your cluster down. You can try increasing the index refresh interval - index.refresh\_interval. If you don't require "live" access, then increasing it to 60 seconds or more will help. If you can gist/pastebin a bit more info on your cluster, node specs, versions, total indexes and size etc it may help. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 18 June 2014 22:53, Antonio Augusto Santos [mkhaos7@gmail.com](mailto:mkhaos7@gmail.com) wrote: > Hello, > > I think I'm hitting some kind of wall here... I'm running logstash on a > syslog server. It receives logs from about 150 machines and also a LOT of > iptables logs, and sending it to Elasticsearch. But, I think I'm not > hitting all speed that I should. My Logstash throughput tops at about 1.000 > events/s, and it looks like my ES servers (I've 2) are really light. > > On logstash I've three configs (syslog, ossec and iptables), so I get > three new nodes on my cluster. I've set up LS Heap Size to be 2G, but > according to bigdesk, the ES module is getting only about 150MB, and its > generating a LOT of GC. > > Bellow the screenshot for big desk: > > [image: bigdesk] > [https://cloud.githubusercontent.com/assets/6423413/3261580/5bc4faf2-f25b-11e3-8529-df0eee61b1e5.png](https://cloud.githubusercontent.com/assets/6423413/3261580/5bc4faf2-f25b-11e3-8529-df0eee61b1e5.png) > > And here the logstash process I'm running: > > _#ps -ef | grep logstash > logstash 13371 1 99 14:42 pts/0 00:29:37 /usr/bin/java -Djava.io.tmpdir=/opt/logstash/tmp -Xmx2g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintClassHistogram -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:./logstash-gc.log -jar /opt/logstash/vendor/jar/jruby-complete-1.7.11.jar -I/opt/logstash/lib /opt/logstash/lib/logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log_ > > My Syslog/LS memory usage seens very light as well (its a 4 core VM), but > the logstash process is always topping in about 150% - 200% > > \*# free -m > total used free shared buffers cached > Mem: 7872 2076 5795 0 39 1502 > -/+ buffers/cache: 534 7337 > Swap: 1023 8 1015 > > # uptime > > 15:02:04 up 23:52, 1 user, load average: 1.39, 1.12, 0.96\* > > Any ideas what I can do to increase the indexing performance? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/b6346f68-1c17-4699-8ad0-cb9121b5c7cb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b6346f68-1c17-4699-8ad0-cb9121b5c7cb%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/b6346f68-1c17-4699-8ad0-cb9121b5c7cb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b6346f68-1c17-4699-8ad0-cb9121b5c7cb%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624Zga8ixq9jVMvsvL9%2BO%2BPQNDUH3VCyubwdBrGLSRBE5eQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624Zga8ixq9jVMvsvL9%2BO%2BPQNDUH3VCyubwdBrGLSRBE5eQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Antonio\_Augusto\_Sant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/antonio_augusto_sant/32/82851_2.png) [@Antonio\_Augusto\_Sant](https://discuss.elastic.co/u/Antonio_Augusto_Sant)\ **Post date:** [June 20, 2014, 1:43am UTC](https://discuss.elastic.co/t/logstash-limitting-elasticsearch-heap/18180/3 "2014-06-20T01:43:35Z")
Thanks for you response Mark. I think I've finally fine tuned my scenario... For starters, it helped me A LOT to set xms on Logstash to the same value as LS\_HEAP\_SIZE. It really reduced the GC. Second, I followed some tips form [ElasticSearch and Logstash Tuning – Vaidas Jablonskis](http://jablonskis.org/2013/elasticsearch-and-logstash-tuning/index.html) and [Elasticsearch Indexing Performance Cheatsheet](https://blog.codecentric.de/en/2014/05/elasticsearch-indexing-performance-cheatsheet/), for increasing my indexing speed (search is second here). After that I increased the number of workers on LS (had to change /etc/init.d/logstash, since it was not respecting LS\_WORKERS on /etc/sysconfig/logstash). This made a big difference, and I could finally see that the workers were being my bottleneck (with 3 workers my 4 cores were hitting 100% usage all the time). So I increased my VM cores to 8, set LS\_WORKERS to 6, and set workers to 3 on the elasticsearch output. The major boost came form these changes. And I could see LS is heavily CPU dependent. Last, but not least, I changed my log strategy. Instead of saving the logs to disk with syslog and reading it back with LS, I got a setup a scenario like [http://cookbook.logstash.net/recipes/central-syslog/](http://cookbook.logstash.net/recipes/central-syslog/) and got my self a redis server as a temp storage (for these logs I don't need logs on file, ES will do just fine). After that I've bumped my indexing speed from about 500 tps to about 4k TPS. Not bad 😉 On Thursday, June 19, 2014 5:32:56 AM UTC-3, Mark Walkom wrote: > Lots of GC isn't bad, you want to see a lot of small GCs rather than > stop-the-world sort of ones which can bring your cluster down. > > You can try increasing the index refresh interval > > - index.refresh\_interval. If you don't require "live" access, then > increasing it to 60 seconds or more will help. > If you can gist/pastebin a bit more info on your cluster, node specs, > versions, total indexes and size etc it may help. > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \ > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 18 June 2014 22:53, Antonio Augusto Santos \<[mkh...@gmail.com](mailto:mkh...@gmail.com) > \\> wrote: > > > Hello, > > > > I think I'm hitting some kind of wall here... I'm running logstash on a > > syslog server. It receives logs from about 150 machines and also a LOT of > > iptables logs, and sending it to Elasticsearch. But, I think I'm not > > hitting all speed that I should. My Logstash throughput tops at about 1.000 > > events/s, and it looks like my ES servers (I've 2) are really light. > > > > On logstash I've three configs (syslog, ossec and iptables), so I get > > three new nodes on my cluster. I've set up LS Heap Size to be 2G, but > > according to bigdesk, the ES module is getting only about 150MB, and its > > generating a LOT of GC. > > > > Bellow the screenshot for big desk: > > > > [image: bigdesk] > > [https://cloud.githubusercontent.com/assets/6423413/3261580/5bc4faf2-f25b-11e3-8529-df0eee61b1e5.png](https://cloud.githubusercontent.com/assets/6423413/3261580/5bc4faf2-f25b-11e3-8529-df0eee61b1e5.png) > > > > And here the logstash process I'm running: > > > > _#ps -ef | grep logstash > > logstash 13371 1 99 14:42 pts/0 00:29:37 /usr/bin/java -Djava.io.tmpdir=/opt/logstash/tmp -Xmx2g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintClassHistogram -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:./logstash-gc.log -jar /opt/logstash/vendor/jar/jruby-complete-1.7.11.jar -I/opt/logstash/lib /opt/logstash/lib/logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log_ > > > > My Syslog/LS memory usage seens very light as well (its a 4 core VM), but > > the logstash process is always topping in about 150% - 200% > > > > \*# free -m > > total used free shared buffers cached > > Mem: 7872 2076 5795 0 39 1502 > > -/+ buffers/cache: 534 7337 > > Swap: 1023 8 1015 > > > > # uptime > > > > 15:02:04 up 23:52, 1 user, load average: 1.39, 1.12, 0.96\* > > > > Any ideas what I can do to increase the indexing performance? > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/b6346f68-1c17-4699-8ad0-cb9121b5c7cb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b6346f68-1c17-4699-8ad0-cb9121b5c7cb%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/b6346f68-1c17-4699-8ad0-cb9121b5c7cb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b6346f68-1c17-4699-8ad0-cb9121b5c7cb%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6727570c-085d-4e85-b3a3-f965b8c08d87%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6727570c-085d-4e85-b3a3-f965b8c08d87%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Erica](https://avatars.discourse-cdn.com/v4/letter/e/9dc877/32.png) [@Erica](https://discuss.elastic.co/u/Erica)\ **Post date:** [July 3, 2014, 7:05pm UTC](https://discuss.elastic.co/t/logstash-limitting-elasticsearch-heap/18180/4 "2014-07-03T19:05:03Z")
Antonio, I have heard many people talking about setting the heap size in elasticsearch, but I can't seem to figure out where to do this--I have tried multiple ways, none of which seem to change the performance and throughput, so I am assuming I have maybe implemented them incorrectly. If you could point me in the right direction that would be great. I am using a windows machine with 4GB RAM if that makes any difference. Erica On Thursday, June 19, 2014 9:43:36 PM UTC-4, Antonio Augusto Santos wrote: > Thanks for you response Mark. > > I think I've finally fine tuned my scenario... > For starters, it helped me A LOT to set xms on Logstash to the same value > as LS\_HEAP\_SIZE. It really reduced the GC. > > Second, I followed some tips form > [ElasticSearch and Logstash Tuning – Vaidas Jablonskis](http://jablonskis.org/2013/elasticsearch-and-logstash-tuning/index.html) > and > [Elasticsearch Indexing Performance Cheatsheet](https://blog.codecentric.de/en/2014/05/elasticsearch-indexing-performance-cheatsheet/), > for increasing my indexing speed (search is second here). > > After that I increased the number of workers on LS (had to change > /etc/init.d/logstash, since it was not respecting LS\_WORKERS on > /etc/sysconfig/logstash). This made a big difference, and I could finally > see that the workers were being my bottleneck (with 3 workers my 4 cores > were hitting 100% usage all the time). So I increased my VM cores to 8, set > LS\_WORKERS to 6, and set workers to 3 on the elasticsearch output. The > major boost came form these changes. And I could see LS is heavily CPU > dependent. > > Last, but not least, I changed my log strategy. Instead of saving the logs > to disk with syslog and reading it back with LS, I got a setup a scenario > like [http://cookbook.logstash.net/recipes/central-syslog/](http://cookbook.logstash.net/recipes/central-syslog/) and got my self > a redis server as a temp storage (for these logs I don't need logs on file, > ES will do just fine). > > After that I've bumped my indexing speed from about 500 tps to about 4k > TPS. > > Not bad 😉 > > On Thursday, June 19, 2014 5:32:56 AM UTC-3, Mark Walkom wrote: > > > Lots of GC isn't bad, you want to see a lot of small GCs rather than > > stop-the-world sort of ones which can bring your cluster down. > > > > You can try increasing the index refresh interval > > > > - index.refresh\_interval. If you don't require "live" access, then > > increasing it to 60 seconds or more will help. > > If you can gist/pastebin a bit more info on your cluster, node specs, > > versions, total indexes and size etc it may help. > > > > Regards, > > Mark Walkom > > > > Infrastructure Engineer > > Campaign Monitor > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > On 18 June 2014 22:53, Antonio Augusto Santos [mkh...@gmail.com](mailto:mkh...@gmail.com) wrote: > > > > > Hello, > > > > > > I think I'm hitting some kind of wall here... I'm running logstash on a > > > syslog server. It receives logs from about 150 machines and also a LOT of > > > iptables logs, and sending it to Elasticsearch. But, I think I'm not > > > hitting all speed that I should. My Logstash throughput tops at about 1.000 > > > events/s, and it looks like my ES servers (I've 2) are really light. > > > > > > On logstash I've three configs (syslog, ossec and iptables), so I get > > > three new nodes on my cluster. I've set up LS Heap Size to be 2G, but > > > according to bigdesk, the ES module is getting only about 150MB, and its > > > generating a LOT of GC. > > > > > > Bellow the screenshot for big desk: > > > > > > [image: bigdesk] > > > [https://cloud.githubusercontent.com/assets/6423413/3261580/5bc4faf2-f25b-11e3-8529-df0eee61b1e5.png](https://cloud.githubusercontent.com/assets/6423413/3261580/5bc4faf2-f25b-11e3-8529-df0eee61b1e5.png) > > > > > > And here the logstash process I'm running: > > > > > > _#ps -ef | grep logstash > > > logstash 13371 1 99 14:42 pts/0 00:29:37 /usr/bin/java -Djava.io.tmpdir=/opt/logstash/tmp -Xmx2g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintClassHistogram -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:./logstash-gc.log -jar /opt/logstash/vendor/jar/jruby-complete-1.7.11.jar -I/opt/logstash/lib /opt/logstash/lib/logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log_ > > > > > > My Syslog/LS memory usage seens very light as well (its a 4 core VM), > > > but the logstash process is always topping in about 150% - 200% > > > > > > \*# free -m > > > total used free shared buffers cached > > > Mem: 7872 2076 5795 0 39 1502 > > > -/+ buffers/cache: 534 7337 > > > Swap: 1023 8 1015 > > > > > > # uptime > > > > > > 15:02:04 up 23:52, 1 user, load average: 1.39, 1.12, 0.96\* > > > > > > Any ideas what I can do to increase the indexing performance? > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/b6346f68-1c17-4699-8ad0-cb9121b5c7cb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b6346f68-1c17-4699-8ad0-cb9121b5c7cb%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/b6346f68-1c17-4699-8ad0-cb9121b5c7cb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b6346f68-1c17-4699-8ad0-cb9121b5c7cb%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/30e59c24-44fb-458f-8689-1d8ecf3f84c1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/30e59c24-44fb-458f-8689-1d8ecf3f84c1%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Antonio\_Augusto\_Sant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/antonio_augusto_sant/32/82851_2.png) [@Antonio\_Augusto\_Sant](https://discuss.elastic.co/u/Antonio_Augusto_Sant)\ **Post date:** [July 3, 2014, 11:16pm UTC](https://discuss.elastic.co/t/logstash-limitting-elasticsearch-heap/18180/5 "2014-07-03T23:16:19Z")
You should have to pass -Xmx2g to the .bat you are calling. If this doesn't work you may have to set an environment variable called ES\_JAVA\_OPTS, and set the Xmx there. You can confirm any of these worked by looking at the resulting process and see the value of Xmx. Also it might be work taking a look at > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. On Thursday, July 3, 2014 4:05:03 PM UTC-3, Erica wrote: > Antonio, > > I have heard many people talking about setting the heap size in > elasticsearch, but I can't seem to figure out where to do this--I have > tried multiple ways, none of which seem to change the performance and > throughput, so I am assuming I have maybe implemented them incorrectly. If > you could point me in the right direction that would be great. I am using a > windows machine with 4GB RAM if that makes any difference. > > Erica > > On Thursday, June 19, 2014 9:43:36 PM UTC-4, Antonio Augusto Santos wrote: > > > Thanks for you response Mark. > > > > I think I've finally fine tuned my scenario... > > For starters, it helped me A LOT to set xms on Logstash to the same value > > as LS\_HEAP\_SIZE. It really reduced the GC. > > > > Second, I followed some tips form > > [ElasticSearch and Logstash Tuning – Vaidas Jablonskis](http://jablonskis.org/2013/elasticsearch-and-logstash-tuning/index.html) > > and > > [Elasticsearch Indexing Performance Cheatsheet](https://blog.codecentric.de/en/2014/05/elasticsearch-indexing-performance-cheatsheet/), > > for increasing my indexing speed (search is second here). > > > > After that I increased the number of workers on LS (had to change > > /etc/init.d/logstash, since it was not respecting LS\_WORKERS on > > /etc/sysconfig/logstash). This made a big difference, and I could finally > > see that the workers were being my bottleneck (with 3 workers my 4 cores > > were hitting 100% usage all the time). So I increased my VM cores to 8, set > > LS\_WORKERS to 6, and set workers to 3 on the elasticsearch output. The > > major boost came form these changes. And I could see LS is heavily CPU > > dependent. > > > > Last, but not least, I changed my log strategy. Instead of saving the > > logs to disk with syslog and reading it back with LS, I got a setup a > > scenario like [http://cookbook.logstash.net/recipes/central-syslog/](http://cookbook.logstash.net/recipes/central-syslog/) and > > got my self a redis server as a temp storage (for these logs I don't need > > logs on file, ES will do just fine). > > > > After that I've bumped my indexing speed from about 500 tps to about 4k > > TPS. > > > > Not bad 😉 > > > > On Thursday, June 19, 2014 5:32:56 AM UTC-3, Mark Walkom wrote: > > > > > Lots of GC isn't bad, you want to see a lot of small GCs rather than > > > stop-the-world sort of ones which can bring your cluster down. > > > > > > You can try increasing the index refresh interval > > > > > > - index.refresh\_interval. If you don't require "live" access, then > > > increasing it to 60 seconds or more will help. > > > If you can gist/pastebin a bit more info on your cluster, node specs, > > > versions, total indexes and size etc it may help. > > > > > > Regards, > > > Mark Walkom > > > > > > Infrastructure Engineer > > > Campaign Monitor > > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > > > On 18 June 2014 22:53, Antonio Augusto Santos [mkh...@gmail.com](mailto:mkh...@gmail.com) wrote: > > > > > > > Hello, > > > > > > > > I think I'm hitting some kind of wall here... I'm running logstash on a > > > > syslog server. It receives logs from about 150 machines and also a LOT of > > > > iptables logs, and sending it to Elasticsearch. But, I think I'm not > > > > hitting all speed that I should. My Logstash throughput tops at about 1.000 > > > > events/s, and it looks like my ES servers (I've 2) are really light. > > > > > > > > On logstash I've three configs (syslog, ossec and iptables), so I get > > > > three new nodes on my cluster. I've set up LS Heap Size to be 2G, but > > > > according to bigdesk, the ES module is getting only about 150MB, and its > > > > generating a LOT of GC. > > > > > > > > Bellow the screenshot for big desk: > > > > > > > > [image: bigdesk] > > > > [https://cloud.githubusercontent.com/assets/6423413/3261580/5bc4faf2-f25b-11e3-8529-df0eee61b1e5.png](https://cloud.githubusercontent.com/assets/6423413/3261580/5bc4faf2-f25b-11e3-8529-df0eee61b1e5.png) > > > > > > > > And here the logstash process I'm running: > > > > > > > > _#ps -ef | grep logstash > > > > logstash 13371 1 99 14:42 pts/0 00:29:37 /usr/bin/java -Djava.io.tmpdir=/opt/logstash/tmp -Xmx2g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintClassHistogram -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:./logstash-gc.log -jar /opt/logstash/vendor/jar/jruby-complete-1.7.11.jar -I/opt/logstash/lib /opt/logstash/lib/logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log_ > > > > > > > > My Syslog/LS memory usage seens very light as well (its a 4 core VM), > > > > but the logstash process is always topping in about 150% - 200% > > > > > > > > \*# free -m > > > > total used free shared buffers cached > > > > Mem: 7872 2076 5795 0 39 1502 > > > > -/+ buffers/cache: 534 7337 > > > > Swap: 1023 8 1015 > > > > > > > > # uptime > > > > > > > > 15:02:04 up 23:52, 1 user, load average: 1.39, 1.12, 0.96\* > > > > > > > > Any ideas what I can do to increase the indexing performance? > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/b6346f68-1c17-4699-8ad0-cb9121b5c7cb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b6346f68-1c17-4699-8ad0-cb9121b5c7cb%40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/b6346f68-1c17-4699-8ad0-cb9121b5c7cb%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b6346f68-1c17-4699-8ad0-cb9121b5c7cb%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/bb091445-2d95-4a79-878c-9e4bd41d2bad%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/bb091445-2d95-4a79-878c-9e4bd41d2bad%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:17am UTC](https://discuss.elastic.co/t/logstash-limitting-elasticsearch-heap/18180/6 "2017-07-06T01:17:58Z")