# Elasticsearch with logstash tuning **URL:** \ **Category:** Elasticsearch\ **Created:** [February 25, 2019, 9:36am UTC](https://discuss.elastic.co/t/elasticsearch-with-logstash-tuning/169788 "2019-02-25T09:36:15Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![fadihaddad](https://avatars.discourse-cdn.com/v4/letter/f/4bbf92/32.png) [@fadihaddad](https://discuss.elastic.co/u/fadihaddad)\ **Post date:** [February 25, 2019, 9:36am UTC](https://discuss.elastic.co/t/elasticsearch-with-logstash-tuning/169788/1 "2019-02-25T09:36:15Z")
hello, i have been using configuring logstash for tuning and i need help. i set the JVM heap of elasticsearch to 8gb and logstash JVM heap to 6gb my server has 4 cpus, 4 cores each 2.2ghz and 16gb RAM. i set the number of workers of logstash to 16 and batch size to 8000 and batch delay to 50 300K events took like 3.45 mins. i noticed it is jumping to about 4800 events/s then falling back to 500- 900 . like in this picture ![Capture](https://us1.discourse-cdn.com/elastic/original/3X/0/e/0eafed48450de8c6d6ae68e8bac544e753598907.png) and here are stats from unix and metrics ![cpustat](https://us1.discourse-cdn.com/elastic/original/3X/0/5/0533cbeb2d57d6f7bf5e20bfc3f800cbe5407dd8.png) ![iostAT](https://us1.discourse-cdn.com/elastic/original/3X/a/a/aa24dc00dec94589ccb9f58cfe7cb1e6babb4040.png) ![metrocs](https://us1.discourse-cdn.com/elastic/original/3X/3/e/3e553c9dbb6bfaa0c39ef3152d9e21e9668410f8.png) ![network](https://us1.discourse-cdn.com/elastic/original/3X/7/d/7df0f76289f9f87017cca81e46f3b944191adb04.png) can I keep the performance around 4800events/s or even more and I am running logstash and elasticsearch on the same node and here is mu logstash conf input { file { path =\> "/data/elasticsearch/data/\*\_0004\_OCC.csv" start\_position =\> "beginning" sincedb\_path =\> "/dev/null" } } filter { csv { separator =\> "," columns =\> ["100 column headers"] } mutate { add\_field =\> { "Calldatetime" =\> "%{calldate} %{calltime}" } } date { match =\> ["Calldatetime", "YYYYMd HHmmss", "YYYYMdd HHmmss", "YYYYMMd HHmmss", "YYYYMMdd HHmmss"] target =\> "Calldatetime" locale =\> "en" timezone =\> "UTC" } } output { stdout { codec =\> dots ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 25, 2019, 10:21am UTC](https://discuss.elastic.co/t/elasticsearch-with-logstash-tuning/169788/2 "2019-02-25T10:21:35Z")
Given that you seem to be running both Elasticsearch and Logstash on the same host you have allocated too much heap as Elasticsearch generally need as much off heap memory as heap. Is there anything in the Elasticsearch logs or stats about merging being throttled or slow? What type of storage do you have? ---
**Author:** ![fadihaddad](https://avatars.discourse-cdn.com/v4/letter/f/4bbf92/32.png) [@fadihaddad](https://discuss.elastic.co/u/fadihaddad)\ **Post date:** [February 25, 2019, 10:48am UTC](https://discuss.elastic.co/t/elasticsearch-with-logstash-tuning/169788/3 "2019-02-25T10:48:11Z")
I am using virtual storage and I will check about the merging in the logs ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 25, 2019, 1:22pm UTC](https://discuss.elastic.co/t/elasticsearch-with-logstash-tuning/169788/4 "2019-02-25T13:22:17Z")
What type of virtual storage? ---
**Author:** ![fadihaddad](https://avatars.discourse-cdn.com/v4/letter/f/4bbf92/32.png) [@fadihaddad](https://discuss.elastic.co/u/fadihaddad)\ **Post date:** [February 25, 2019, 1:47pm UTC](https://discuss.elastic.co/t/elasticsearch-with-logstash-tuning/169788/5 "2019-02-25T13:47:02Z")
![storage%20type](https://us1.discourse-cdn.com/elastic/original/3X/8/2/8265063169bc5e43fb165a2883ca43757e8e01b1.png) ---
**Author:** ![fadihaddad](https://avatars.discourse-cdn.com/v4/letter/f/4bbf92/32.png) [@fadihaddad](https://discuss.elastic.co/u/fadihaddad)\ **Post date:** [February 26, 2019, 8:23am UTC](https://discuss.elastic.co/t/elasticsearch-with-logstash-tuning/169788/6 "2019-02-26T08:23:11Z")
I'm not sure if this helps i don't know how to get the physical type of a virtual storage ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 26, 2019, 9:12am UTC](https://discuss.elastic.co/t/elasticsearch-with-logstash-tuning/169788/7 "2019-02-26T09:12:08Z")
Then look at disk I/O and iowait to see if storage might be a bottleneck. ---
**Author:** ![fadihaddad](https://avatars.discourse-cdn.com/v4/letter/f/4bbf92/32.png) [@fadihaddad](https://discuss.elastic.co/u/fadihaddad)\ **Post date:** [February 26, 2019, 9:59am UTC](https://discuss.elastic.co/t/elasticsearch-with-logstash-tuning/169788/8 "2019-02-26T09:59:25Z")
Well I'm having a problem I think the bottleneck is I/O because suddenly logstash performance stops to 0 and waits ---
**Author:** ![fadihaddad](https://avatars.discourse-cdn.com/v4/letter/f/4bbf92/32.png) [@fadihaddad](https://discuss.elastic.co/u/fadihaddad)\ **Post date:** [February 26, 2019, 10:02am UTC](https://discuss.elastic.co/t/elasticsearch-with-logstash-tuning/169788/9 "2019-02-26T10:02:05Z")
does elastic search allocate shards while indexing? and I have my settings to 1 shard and 0 replicas ---
**Author:** ![fadihaddad](https://avatars.discourse-cdn.com/v4/letter/f/4bbf92/32.png) [@fadihaddad](https://discuss.elastic.co/u/fadihaddad)\ **Post date:** [February 26, 2019, 10:14am UTC](https://discuss.elastic.co/t/elasticsearch-with-logstash-tuning/169788/10 "2019-02-26T10:14:20Z")
and i am getting this error on logstash :CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -XX:+HeapDumpOnOutOfMemoryError ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 26, 2019, 10:14am UTC](https://discuss.elastic.co/t/elasticsearch-with-logstash-tuning/169788/11 "2019-03-26T10:14:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.