HI,
I am trying to load data into elasticsearch using logstash, but its loading very slow.
80000/minute into elasticsearch.
So, its taking 10 mins for 11GB data. I have requirement to reduce this to 10 secs.
Please help me what all changes I need to do in ELK and in system Hardware .
The following is my Logstash config
input {
file {
type => "engine"
path => "Path/*"
start_position => "beginning"
sincedb_path => "/dev/null"
ignore_older => 0
codec => multiline {
pattern => "^qns|^lb0"
negate => true
what => "previous"
}
}
}
filter {
if [type] == "engine" {
mutate
{
gsub => ["message", "\n", ""]
}
mutate
{
gsub => ["message", "\r", ""]
}
mutate
{
gsub => ["message", "\t", ""]
}
mutate
{
gsub => ["message", "=", ""]
}
grok {
break_on_match => false
match => ["message", "%{WORD:VM} [%{TIMESTAMP_ISO8601:Timestamp}] %{GREEDYDATA:Record}",
"Record", "POLICY RESULT ERROR: %{DATA:Policy_Result_Error}session action %{GREEDYDATA:Remaining}"]
}
mutate { add_field => { "Policy_Result_Error_Analyzed" => "%{Policy_Result_Error}" }
remove_field => ["message","Remaining"] }
date {
match => [ "Timestamp" , "yyyy-MM-dd HH:mm:ss,SSS" ]
}
}
}
output {
if [type] == "engine" {
stdout { codec => rubydebug }
elasticsearch {
hosts => "IP:9200"
index => "hathway_test_engine"
workers => 40
flush_size => 1000
}
}
}
I have 3-node cluster in AWS
My system hardware config of each node as follows
224 gb RAM, 30 GB Hrd disk with 32-core
I did following changes in /etc/default/logstash
Arguments to pass to logstash agent
LS_OPTS="-b 1000 -w 40"
Arguments to pass to java
LS_HEAP_SIZE="128g"
LS_JAVA_OPTS="-Djava.io.tmpdir=$HOME"
LS_JAVA_OPTS="-Xmx100g -Xms4G"
also in bin/logstash
LS_HEAP_SIZE="xxx" size for the -Xmx${LS_HEAP_SIZE} maximum Java heap size option, default is "1g"
LS_HEAP_SIZE="128g"
LS_JAVA_OPTS="xxx" to append extra options to the defaults JAVA_OPTS provided by logstash
LS_JAVA_OPTS="-Xmx100g -Xms4g"
but still i don't see any improvement in logstash performance.
The following are screen shots of CPU/Memory/IO when I was running Logstash,
Instance-1
Instance-2
Instance-3
Instance-1/CPU
Instance-3/CPU
Instance-2/CPU
Instance-1/Memory
Instance-3/memory
Instance-2/Memory
Instance-1/IOSTAT
Instance-2/IOSTAT
Instance-3/IOSTAT
