Hello,
I have a strange problem with my ELK setup. I'm using this docker container https://hub.docker.com/r/sebp/elk/ with the current version 651.
So I'm using logstash version 6.5.1
I wrote a small script which copies the logfiles each hour over ssh to the ELK host. After the log is completly imported I move the logfile to an archive.
The logfiles have the following structure:
"a93f1bd7-32fc-43a2-bf7e-929c1e539319";"2019-01-24T00:59:56+01:00";"perf;firstpaint;273";""
"a93f1bd7-32fc-43a2-bf7e-929c1e539319";"2019-01-24T00:59:56+01:00";"perf;jsinitialized;648";""
"1434c999-a818-458d-c2d1-6f3db7e1c19c";"2019-01-24T00:59:56+01:00";"productSliders;lastViewed;showProduct";""
"75e508ae-ac56-4a5b-f51e-caee70444f68";"2019-01-24T00:59:56+01:00";"abtest;PGAJA00;init";""
"957f5d1c-a927-43bd-d094-8539511a8e5b";"2019-01-24T00:59:56+01:00";"abtest;PGAJA00;init";""
"1434c999-a818-458d-c2d1-6f3db7e1c19c";"2019-01-24T00:59:57+01:00";"abtest;AZKQA00;controlled";""
"3cacfc25-ca22-4699-9865-d3e32c255d3a";"2019-01-24T00:59:57+01:00";"abtest;AZKQA00;pre-controlled";""
"362c7a33-ba7d-41a0-8800-e7b78f911c4d";"2019-01-24T00:59:57+01:00";"pwa;install;canShow";""
"5c8900bc-d5e1-4b0d-c1cd-9ec3269b5c66";"2019-01-24T00:59:58+01:00";"abtest;PGAJA03;init";""
"c0466a41-b531-4ca8-98ce-ac59b08e3cf7";"2019-01-24T00:59:59+01:00";"abtest;PGAJA00;init";""And that's my logstash config file:
input {
file {
path => '/var/elk-logs/queue/*lfa_event_20*.log'
start_position => "beginning"
sincedb_path => "/dev/null"
add_field => {
"[@metadata][indexType]" => "lfaEvent"
}
}
}
filter {
if [@metadata][indexType] == "lfaEvent" {
if [message] in ["id;timestamp;event;test_variant", ""] {
drop{ }
} else {
grok {
match => { "message" => '"%{UUID:uuid}";"%{TIMESTAMP_ISO8601:[@metadata][timestamp]}";"%{DATA:event}";"%{DATA:test_variant}"' }
}
date {
match => [ "[@metadata][timestamp]" , "ISO8601" ]
}
}
}
if "_grokparsefailure" in [tags] {
#drop { }
} else if [@metadata][indexType] == "lfaEvent" {
fingerprint {
source => "message"
target => "[@metadata][fingerprint]"
method => "SHA1"
key => "budeLogfile"
}
mutate {
remove_field => ["host", "path", "message"]
}
}
}
output {
if "_grokparsefailure" in [tags] {
elasticsearch {
index => "logstash-grokparsefailure"
}
} else if [@metadata][indexType] == "lfaEvent" {
elasticsearch {
index => "logstash-lfa-event-%{+YYYY.MM.dd}"
document_id => "%{[@metadata][fingerprint]}"
}
}
}It works very well but sometimes a grokparsefailure is thrown. Here are some examples:
|Time |path |message |
|January 29th 2019, 16:20:42.891|/var/elk-logs/queue/bude-web2.priv-lfa_event_20190124_01_00.log|rf;firstpaint;709";""|
|January 29th 2019, 16:09:09.948|/var/elk-logs/queue/bude-web2.priv-lfa_event_20190124_00_02.log|d";""|
|January 29th 2019, 16:08:37.539|/var/elk-logs/queue/bude-web1.priv-lfa_event_20190129_15_02.log|"abtest;PGAJA00;init";""|
|January 29th 2019, 16:08:37.536|/var/elk-logs/queue/bude-web1.priv-lfa_event_20190129_15_01.log|7+01:00";"abtest;AZKQA00;unsupported";""|
|January 29th 2019, 16:08:37.519|/var/elk-logs/queue/bude-web1.priv-lfa_event_20190129_15_03.log|53+01:00";"abtest;PGAJA00;init";""|
|January 29th 2019, 16:08:22.253|/var/elk-logs/queue/bude-web1.priv-lfa_event_20190124_23_02.log|9-01-24T23:40:02+01:00";"abtest;PGAJA00;init";""|
|January 29th 2019, 16:08:21.635|/var/elk-logs/queue/bude-web1.priv-lfa_event_20190124_23_00.log|:19+01:00";"addToCart;click;41981735";""|I have this problem since months and my dirty solution is for checking the grokparsefailures. If I detect this failure the logfile is deleted and after a few minutes the logfile is copied again to the logstash import folder. That's the reason why I set the sincedb_path to /dev/null
Normaly after the second try all rows are imported. Sometime it needs 4 or 5 tries until the file is completly imported.
I have absolute no idea why this happens. I seems like logstash stops reading the new file randomly after n bytes. There are no rows which contain only the message part from the grokparsefailure. The string at the message is every time a part of a valid row.
If there are for example 10 files which aren't imported completly I can restart the elk docker container and then all files are imported completly after logstash is started! I tried this a few times and it worked each time.
