Hello,
I am new to ELK and maybe my problem is dumm, but i cant get over it.
I have very slow indexing( approx: 4k docs per min. sometimes its 8k)
My setup - Centos 7 - Logstash 7.2 - Elastic 7.2 - Kibana 7.2 - (ELK all on one VM)
Centos is 6 CPU 24G Memory
Logstash input - just a directory with file (file is pretty big ~7Gb)
Codec - multiline
Filter - grok
Output into Elastic with template overwrite
shards: 2
refresh_interval: -1
replicas: 0
pipeline.workers: 4 (i tried more)
pipeline.batch.size: 1000
no errors in both logs(logstash or elastic)
no grok failures
Here is my logstash config:
input {
file {
codec => multiline {
pattern => "^%{DATESTAMP_EVENTLOG}"
what => "previous"
negate => true
}
path => "my-path"
sincedb_path => "my-path-sincedb"
start_position => "beginning"
stat_interval => 120
type => "my-db-name"
}
}
filter {
mutate {}
mutate {}
ruby {}
grok {}
date {}
}
output {
elasticsearch {
hosts => ["127.0.0.1:9200"]
manage_template => true
template_overwrite => true
template_name => "template"
template => "path/template"
index => "my index"
}
}
What can i do to understand the reason of slow work?
Could it be the file size?