Grok tag_on_failure

[nin77]

Hey, I am using:

grok {
match => [ "ip_filter" , " %{IPV4:clientip}" ]
tag_on_failure => [ "_todelete" ]
}

So every line with a non valid IP should be tagged _todelete, but on Kibana I see every line tagged with it. Any ideas why?

[magnusbaeck (Magnus Bäck)]

Please show an example message (copy/paste the event text from Kibana's JSON tab) and the rest of your Logstash configuration.

[nin77]

Thank you for your replay! Here is the Json output:

{
"_index": "log_analyzer",
"_type": "doc",
"_id": "y57OxmMByI6RAjAO1JOX",
"_version": 1,
"_score": 1.2111092,
"_source": {
"bytes": 209,
"ident": "-",
"message": "192.168.1.71 - - [08/May/2018:12:56:08 +0200] "GET /favicon.ico HTTP/1.1" 404 209 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0"",
"type": "apache_access",
"major": "59",
"request": "/favicon.ico",
"name": "Firefox",
"auth": "-",
"@timestamp": "2018-05-08T10:56:08.000Z",
"os": "Windows 7",
"clientip": "192.168.1.71",
"referrer": ""-"",
"port": 59370,
"response": 404,
"agent": ""Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0"",
"httpversion": "1.1",
"os_name": "Windows 7",
"build": "",
"host": "probackup_nginx_1.probackup_elk",
"tags": [
"_todelete"
],
"minor": "0",
"device": "Other",
"@version": "1",
"verb": "GET"
},
"fields": {
"@timestamp": [
"2018-05-08T10:56:08.000Z"
]
},
"highlight": {
"tags": [
"@kibana-highlighted-field@_todelete@/kibana-highlighted-field@"
]
}
}

My Config looks like that:

filter {

grok {
match => [ "message" => " %{IPV4:clientip}" ]
tag_on_failure => [ "_todelete" ]
}

// if "_todelete" in [tags] {
// drop {}
// }

// grok {
// remove_tag => [ "_todelete" ]
// }

if [type] in [ "apache" , "apache_access" , "apache-access" ] {
grok {
match => [
"message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}",
"message" , "%{COMMONAPACHELOG}+%{GREEDYDATA:extra_fields}"
]
overwrite => [ "message" ]
}
mutate {
convert => ["response", "integer"]
convert => ["bytes", "integer"]
convert => ["responsetime", "float"]
remove_field => "os_name"
}
date {
match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
remove_field => [ "timestamp" ]
}
useragent {
source => "agent"
}
}

I wanted to use one of the two commented solutions as my next step but because every line is tagged it deletes all
(I added // instead of hastag so it doesnt kill the format)

[magnusbaeck (Magnus Bäck)]

Your grok expression is looking for a space followed by an IP address but your IP address comes at the very beginning of the string. So, use ^%{IPV4:clientip} instead.

[nin77]

That worked. Thanks a lot!