Logstash Grok Filter _tags vs _fields

Elastic Stack Logstash
3

Dear Magnus,

Thanks for the reply,

Bwpayload is a field so you should use remove_field.
That's because you're using %{Bwpayload} which expands to the contents of the Bwpayload field, and that's not what you want. Just use Bwpayload.

Used as suggested by you. Working fine now.

I don't understand this sentence, but if the date filter fails it'll log details about the failure. In this case the problem is that your date pattern doesn't match the contents of the timestamp string.

In Kibana i have two timestamp, as highlighted in below json output. whereas the date plugin in config is not matching the same. Query here is, i should form date{match [ field, formats... ]} pattern in config for which timestamp.

In case of grokprasefailure i would like to add few fields as given in below config. These fields are from my pattern and event. Is this the right procedure to add the same.
Using add_field with the drop filter doesn't make any sense since drop deletes events. Use a mutate filter instead.

I Used Mutate, but for parsefail_path field alone getting desired output, for other two fields not getting the actual value, instead getting the same names mention in config. highlighted below in output. (Tried liket his as well parsefail_prog => [ "program" ]).
Also i dont't want other tags, is it okay if i use remove_field in the same mutate plugin.

Kindly help.

Kibana JSON Output:

{
"_index": "logstash-2017.02.15",
"_type": "esbbwlog",
"_id": "AVpBhE3RooOYGdG3tn-Y",
"_score": null,
"_source": {
"path": "D:/logs/TestService-TestService.log",
"@timestamp": "2017-02-15T11:24:34.113Z",
"parsefail_prog": "program",
"@version": "1",
"host": "SDINCNB00010",
"message": "2017 Feb 15 16:54:32:910 GMT +5 TestService-TestService User [User] - \r\n ",
"type": "esbbwlog",
"parsefail_path": "D:/logs/TestService-TestService.log",
"parsefail_message": "message",
"tags": [
"multiline",
"_grokparsefailure"
]
},
"fields": {
"@timestamp": [
** 1487157874113**
** ]**
},
}

My Config File:

filter {
grok {
patterns_dir => "ELK/logstash-5.1.1/patterns/ingdevbw"
match => { "message" => "%{BWLOG}" }
remove_tag => [ "Bwpayload" ]

}
date {
match => [ "timestamp" , "YYYY MMM DD HH:mm:ss:SSS" ]
remove_field => [ "timestamp" ]
}

if "_grokparsefailure" in [tags] {
drop {
add_field => {
parsefail_path => "%{path}"
parsefail_prog => "%{program}"
parsefail_message => "%{@message}"

}
}
}
}