Elasticsearch with logstash tuning

hello, i have been using configuring logstash for tuning and i need help. i set the JVM heap of elasticsearch to 8gb and logstash JVM heap to 6gb
my server has 4 cpus, 4 cores each 2.2ghz and 16gb RAM.
i set the number of workers of logstash to 16 and batch size to 8000 and batch delay to 50
300K events took like 3.45 mins. i noticed it is jumping to about 4800 events/s then falling back to 500- 900 . like in this picture

and here are stats from unix and metrics

can I keep the performance around 4800events/s or even more and I am running logstash and elasticsearch on the same node and here is mu logstash conf

input {

file {

path => "/data/elasticsearch/data/*_0004_OCC.csv"

start_position => "beginning"

sincedb_path => "/dev/null"



filter {

csv {

separator => ","

columns =>

["100 column headers"]


mutate {

add_field => {

"Calldatetime" => "%{calldate} %{calltime}"



date {

match => ["Calldatetime", "YYYYMd HHmmss", "YYYYMdd HHmmss", "YYYYMMd HHmmss", "YYYYMMdd HHmmss"]

target => "Calldatetime"

locale => "en"

timezone => "UTC"



output {

stdout {

codec => dots

Given that you seem to be running both Elasticsearch and Logstash on the same host you have allocated too much heap as Elasticsearch generally need as much off heap memory as heap. Is there anything in the Elasticsearch logs or stats about merging being throttled or slow? What type of storage do you have?

I am using virtual storage and I will check about the merging in the logs

What type of virtual storage?

I'm not sure if this helps i don't know how to get the physical type of a virtual storage

Then look at disk I/O and iowait to see if storage might be a bottleneck.

Well I'm having a problem I think the bottleneck is I/O because suddenly logstash performance stops to 0 and waits

does elastic search allocate shards while indexing? and I have my settings to 1 shard and 0 replicas

and i am getting this error on logstash

:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -XX:+HeapDumpOnOutOfMemoryError

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.