Is it possible to use the json codec along with a json filter to further parse things into other fields?

[Tieis]

I've been looking around and experimented a bit, but haven't gotten it to work or so to say.

I have a tcp input with the json codec, along with an elasticsearch output with a template that adds some custom fields. I used to just use a grok-pattern to match the logger message to separate it to different fields, however I'm wondering if I could use a JSON logger message, and use a json filter and set the source of the filter to parse it into fields.

json {
source => "message"
target => "parsedJson"
}
mutate {
add_field => {
"userName" => "%{[parsedJson][userName]}"
"clientName" => "%{[parsedJson][clientName]}"
}
}

This returns a json_message tag/field with value of null.

[magnusbaeck (Magnus Bäck)]

Sure, you can combine a json/json_lines codec with a json filter if you have JSON-encoded data inside JSON-encoded data.

[Tieis]

Ok nice! so if I had this for example in the JSON-encoded data from the codec:

{
"_index": "logstash-2018.02",
"_type": "application",
"_id": "AWG-1QHz2w4sPkeuw0Bb",
"_version": 1,
"_score": null,
"_source": {
"level": "INFO",
"message": "{"userName":"test", "clientName":"test"}",
"type": "application",
"@timestamp": "2018-02-22T18:44:46.150Z",
"HOSTNAME": "EXAMPLE",
"appname": "EXAMPLE",
"json_message": null,
"port": 51080,
"thread_name": "http-nio-8088-exec-10",
"level_value": 20000,
"@version": 1,
"host": "172.16.48.0",
"logger_name": "com.example"
},
"fields": {
"@timestamp": [
1519325086150
]
},
.....
}

If I wanted to extract the JSON from the message property in the _source field, what type of json filter configuration should I be using ?

I've tried setting source => "message", and source => [source][message] and some others, but they didn't seem to work.

[magnusbaeck (Magnus Bäck)]

The message field doesn't contain a string with JSON in it, it contains an object. This is what a string containing JSON data would've looked like:

"message": "{\"userName\":\"test\", \"clientName\":\"test\"}",

[Tieis]

Thanks for the insight, seems to be filtering into an object now after with the json filter. Just need to separate it into fields for my purpose now.

Thanks!

[Tieis]

Hey Magnus, thanks for the pointers before. I am having trouble separating the json filter target into it's own fields now.
I have used this setting:

json {
source => "message"
target => "message"
}

It seems to parse fine when looking at the JSON

"_source": {
"level": "INFO",
"message": "{"userName":"Test", "clientName":"Test"}",
.....
}

Is there an easy way to extract the object keys and values into their own field? I was trying

mutate {
add_field => {
"userName" => "%{[message]["userName"]}"
"clientName" => "%{[message]["clientName"]}"
}
}

but that doesn't seem to work. I've looked around and maybe the Split filter may work if the object is changed into an array? I quickly experimented with it and wasn't able to get it to work.

Sorry for the questions, I'm quite the beginner and am curious if I could do this instead of using our existing grok-pattern filter.

[magnusbaeck (Magnus Bäck)]

It seems to parse fine when looking at the JSON

Your example shows a message field containing an unparsed JSON string so it seems the json filter isn't doing what it's supposed to be doing.

If you want the fields from the JSON string at the top level of the event just omit the target option. Then you won't need the mutate filter.

[Tieis]

Ok, I think you are probably onto something. I tried omitted the "target" option before and got a _jsonparsefailure tag before and didn't think too much about it, but it's possibly my logger message is still not correctly configured. I'll try some other things.

Thanks for the tips.

[Tieis]

holy it worked. I was double json encoding my logger message because i thought there was a json codec and filter, but doesn't seem like i have to.

Thanks for the help!