Logstash/Elastic slow indexing

Hello,
I am new to ELK and maybe my problem is dumm, but i cant get over it.
I have very slow indexing( approx: 4k docs per min. sometimes its 8k)

My setup - Centos 7 - Logstash 7.2 - Elastic 7.2 - Kibana 7.2 - (ELK all on one VM)
Centos is 6 CPU 24G Memory

Logstash input - just a directory with file (file is pretty big ~7Gb)
Codec - multiline
Filter - grok
Output into Elastic with template overwrite

shards: 2
refresh_interval: -1
replicas: 0

pipeline.workers: 4 (i tried more)
pipeline.batch.size: 1000

no errors in both logs(logstash or elastic)
no grok failures

Here is my logstash config:
input {
file {
codec => multiline {
pattern => "^%{DATESTAMP_EVENTLOG}"
what => "previous"
negate => true
}
path => "my-path"
sincedb_path => "my-path-sincedb"
start_position => "beginning"
stat_interval => 120
type => "my-db-name"
}

  }

filter {
mutate {}
mutate {}
ruby {}
grok {}

date {}
}
output {
elasticsearch {
hosts => ["127.0.0.1:9200"]
manage_template => true
template_overwrite => true
template_name => "template"
template => "path/template"
index => "my index"
}
}

What can i do to understand the reason of slow work?
Could it be the file size?

Looks like i found my problem.
its unique for my type of logs.
seems like pattern in multiline was parsing every symbol of 7GB file.. changed pattern so he would work only after separator.
and now i have 200k docs per min.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.