Hi,
I am trying to index my mail-relay log files to Elastic search. All the log entries are being indexed into a field named message . @timestamp field shows the time the entry was indexed and not the timestamp from the log entry. The structure of the data varies from line to line.
I have tried using dissect filter and is working fine for the above source log message line 1. However in order to parse the second line, it's failing as the timestamp is in between of the message.
Dissect Filter working for line1:
input {
beats {
port => 5044
}
}
filter {
if "beats_input_codec_plain_applied" in [tags] {
mutate {
remove_tag => ["beats_input_codec_plain_applied"]
}
}
if "mailrelayUAT" in [tags] {
dissect { mapping => { "message" => "%{[@metadata][timestamp]} %{+[@metadata][timestamp]} %{}" } }
date { match => [ "[@metadata][timestamp]", "yyyy-MM-dd HH:mm:ss" ] }
}
}
Can we still use the dissect filter to achieve this or I need to look into grok filter.
I have also used grok as below.
Grok filter:
input {
beats {
port => 5044
}
}
filter {
grok {
match => [ "message", "(?<sourcestamp>(\d){4}-(\d){2}-(\d){2} (\d){2}:(\d){2}:(\d){2},(\d){3})" ]
}
date {
match => [ "sourcestamp" , "yyyy-MM-dd HH:mm:ss,SSS" ]
target => "@timestamp"
timezone => "America/Chicago"
}
}
filter {
if "beats_input_codec_plain_applied" in [tags] {
mutate {
remove_tag => ["beats_input_codec_plain_applied"]
}
}
grok {
match => { "message" => "^%{TIMESTAMP_ISO8601}" }
add_tag => [ "startsWithTimestamp" ]
}
if "startsWithTimestamp" in [tags] {
dissect { mapping => { "message" => "%{[@metadata][timestamp]} %{+[@metadata][timestamp]} %{}" } }
date { match => [ "[@metadata][timestamp]", "yyyy-MM-dd HH:mm:ss" ] }
}
}
grok {
match => { "message" => "^%{IPV4}" }
add_tag => [ "startsWithIp" ]
}
if "startsWithIp" in [tags] {
dissect { mapping => { "message" => "%{ip} %{?-} %{hostname} %{[@metadata][timestamp]} %{+[@metadata][timestamp]} %{}" } }
date { match => [ "[@metadata][timestamp]", "yyyy-MM-dd HH:mm:ss" ] }
}
}
mutate { remove_tag => [ "_grokparsefailure" ] }
Let me know if it looks good?
Also please let me know If I can do the above same by only using GROK filter as I dont want to disturb the exisitng GROK logic for other formats of logs
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.