Issue with remove_field

[blahnik]

I am having a hard time understanding why my filter to remove some fields is not working.

I grabbed a sample of the JSON from kibana

{
"_index": "logstash-2017.01.27",
"_type": "json",
"_score": 1,
"_source": {
"source": "unknown",
"type": "json",
"duration": 16.13,
"view": 2.68,
"@version": "1",
"host": "ip-10-0-0-111",
"action": "index",
"api": false,
"user_agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36",
"controller": "marketing",
"format": "html",
"message": "{"method":"GET","path":"/","format":"html","controller":"marketing","action":"index","status":200,"duration":16.13,"view":2.68,"db":6.11,"route":"marketing#index","parameters":{},"user_agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36","source":"unknown","tags":["request"],"@timestamp":"2017-01-27T05:20:00Z","@version":"1"}",
"tags": [
"request"
],
"@timestamp": "2017-01-27T05:20:00.000Z",
"route": "marketing#index",
"parameters": {},
"db": 6.11,
"status": 200
},
"fields": {
"@timestamp": [
1485494400000
]
}
}

My logstash.conf file looks like this

filter {
json {
source => "message"
}
mutate {
remove_field => [ "[_source][message][method]" ]
}
}

I have also tried
remove_field => [ "[method]"]
remove_field => [ "[message][method]"]

Also if it matters that data is coming in from redis using the input redis and codec => json.

Thanks in advance.

[magnusbaeck (Magnus Bäck)]

The [_source] part shouldn't be there at all. The reason [message][method] and [method] don't work is because your event has no such fields. It only has a message field that contains a JSON string. You have to parse that JSON payload, e.g. by setting the codec of your redis input to json.

[blahnik]

Maybe I have the input wrong but I do have codex => json as part of the input. Do i need to do something to parse a block of the input?

input {
redis {
host =>...................
data_type => 'list'
codec => "json"
}
}

[magnusbaeck (Magnus Bäck)]

Um, never mind. I read your previous posts too quickly. It does indeed extract stuff from the JSON payload. But there's no method field. Doesn't that in fact indicate that your filter is working?

[blahnik]

I guess to be more clear I am trying to remove some fields from the message part of the event, which is a hash inside the event. For example I want to remove the [message][method]..which is one of the keys inside the message.

ie this part:

"message": "{"method":"GET","path":"/","format":"html","controller":"marketing","action":"index","status":200,"duration":16.13,"view":2.68,"db":6.11,"route":"marketing#index","parameters":{},"user_agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36","source":"unknown","tags":["request"],"@timestamp":"2017-01-27T05:20:00Z","@version":"1"}",

[magnusbaeck (Magnus Bäck)]

I guess to be more clear I am trying to remove some fields from the message part of the event, which is a hash inside the event.

Your message field is a string that happens to be JSON. You can't manipulate that JSON object directly, but you can deserialize it into discrete fields that you can play with. That's what you're doing with the json filter.