Elasticsearch with logstash tuning

[fadihaddad (fadi)]

hello, i have been using configuring logstash for tuning and i need help. i set the JVM heap of elasticsearch to 8gb and logstash JVM heap to 6gb
my server has 4 cpus, 4 cores each 2.2ghz and 16gb RAM.
i set the number of workers of logstash to 16 and batch size to 8000 and batch delay to 50
300K events took like 3.45 mins. i noticed it is jumping to about 4800 events/s then falling back to 500- 900 . like in this picture

and here are stats from unix and metrics

can I keep the performance around 4800events/s or even more and I am running logstash and elasticsearch on the same node and here is mu logstash conf

input {

file {

path => "/data/elasticsearch/data/*_0004_OCC.csv"

start_position => "beginning"

sincedb_path => "/dev/null"

}

}

filter {

csv {

separator => ","

columns =>

["100 column headers"]

}

mutate {

add_field => {

"Calldatetime" => "%{calldate} %{calltime}"

}

}

date {

match => ["Calldatetime", "YYYYMd HHmmss", "YYYYMdd HHmmss", "YYYYMMd HHmmss", "YYYYMMdd HHmmss"]

target => "Calldatetime"

locale => "en"

timezone => "UTC"

}

}

output {

stdout {

codec => dots

[Christian_Dahlqvist]

Given that you seem to be running both Elasticsearch and Logstash on the same host you have allocated too much heap as Elasticsearch generally need as much off heap memory as heap. Is there anything in the Elasticsearch logs or stats about merging being throttled or slow? What type of storage do you have?

[fadihaddad (fadi)]

I am using virtual storage and I will check about the merging in the logs

[Christian_Dahlqvist]

What type of virtual storage?

[fadihaddad (fadi)]

[fadihaddad (fadi)]

I'm not sure if this helps i don't know how to get the physical type of a virtual storage

[Christian_Dahlqvist]

Then look at disk I/O and iowait to see if storage might be a bottleneck.

[fadihaddad (fadi)]

Well I'm having a problem I think the bottleneck is I/O because suddenly logstash performance stops to 0 and waits

[fadihaddad (fadi)]

does elastic search allocate shards while indexing? and I have my settings to 1 shard and 0 replicas

[fadihaddad (fadi)]

and i am getting this error on logstash

:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -XX:+HeapDumpOnOutOfMemoryError