Dear All,
Have few queries as follows,
-
Requirement is to remove the tag/field Bwpayload and to add message, program and path as tags in case of grokparsefailure. i am confused about either to use remove_tag or remove_field. Sad part is either of one is not working. I am able to view Bwpayload tag with values in Kibana.
-
Also date plugin having timestamp, will point to which format [Generated internally by logstash or from my logs], because facing date mismatch error when not commented.
2017 Feb 14 17:31:28:389 GMT +5 - which is from my logs with tag timestamp in kibana
February 14th 2017, 17:31:28.455 - seen in kibana with @timestamp tag.
- In case of grokprasefailure i would like to add few fields as given in below config. These fields are from my pattern and event. Is this the right procedure to add the same.
Kindly guide,
My filter config looks like below,
filter {
grok {
patterns_dir => "ELK/logstash-5.1.1/patterns/ingdevbw"
match => { "message" => "%{BWLOG}" }
remove_tag => [ "%{Bwpayload}" ]
# remove_tag => [ "%{Bwpayload}" ]
}
date {
match => [ "timestamp" , "YYYY MMM DD HH:mm:ss:SSS" ]
# remove_field => [ "timestamp" ]
}
if "_grokparsefailure" in [tags] {
drop {
add_field => {
parsefail_path => "%{path}"
parsefail_prog => "%{program}"
parsefail_message => "%{@message}"
}
}
}
}
and My pattern is as below
Logstash pattern configuration
BWTIME %{YEAR} %{MONTH} %{MONTHDAY} %{TIME} GMT +%{INT}
BWLOG %{BWTIME:timestamp} %{PROG:program} %{WORD:loglevel} [%{USER:auth}] %{GREEDYDATA:Bwpayload}