How to extract json out of an event?

Elastic Stack Logstash
1

Hi,

Here is my sample event,

<6>Jun 5 12:12:45 172.25.29.92 doppler[10]: time="2018-06-05T16:12:46Z" level=info msg="{ "Level": "Info", "Timestamp": "2018-06-05 14:12:45.1195", "Message": "sample message" } " app_id=4456 app_name=dev-app org_id=65739 org_name=Dev timestamp=1528215165150529000

I want to extract msg field(which is a json) and later want to extract key-value pairs out of msg field. Please suggest me a way.

Here is the logstash config I am using,

filter{
      grok {
                                match => {"message" => "%{SYSLOGBASE}\stime\=\"%{NOTSPACE:timest}\"\slevel\=%{NOTSPACE:level}\smsg\=\"\{%{DATA:jsondata}\}%{GREEDYDATA:test}"}
                        }
}

Problem with above grok would be, I will end up not having closing curly bracket }.

Please suggest me a better way to extract json data.

2

Any help here is highly appreciated :slight_smile:

3

If you only problem with the jsondata field is that it is missing a curly brace, then one option is to add one.

mutate { gsub => [ "jsondata", "$", "}' ] }
4

Thanks Badger. Can you please explain what is $ here?

5

It is a regular expression that matches the end of the line. So this says to change the end of the line to be }.

6

Thanks Badger, it works fine.

I have another issue - one of the json fields is named message and also logstash by default gives message field. I want to rename josn field "message" to something like "json_message". If I use mutate filter, I think it would replace both the fields. Is there any way where I can rename fields only for json fields?

7

If you have a field called "message" and you parse the JSON, by default it will overwrite the original "message". If you set target on the JSON filter than you will have the "message" of the JSON inside someField, and then you can rename that using something like

mutate { rename => { "[someField][message]" => "[somefield][someOtherName]" } }
8

Thanks Badger.

My Json object may have several fields depending on the event. It may have a least of 3 fields or it may even have up to 20 fields. If I use a target field named jsondoc, all the fields will be inside jsondoc. But I want all the fields to be independent instead in jsondoc. Is there a way?

9

Yes, do not set a target, and the fields will be at the root level. In that case, if the JSON includes a field called message it will overwrite the message field. If you really want to retain the message field then mutate+rename message to saved_message, parse saved_message with a json filter, mutate+rename message to json_message, mutate+rename saved_message back to message..

1 Like
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.