Parsing log with partial json

[mikemi (Mike)]

2017-03-21T23:53:10-04:00 INFO login {"input":"user123","success":false} {"host":"www.example.com","path":"user","clientip":"127.0.0.1"}

This log file starts with a timestmap, loglevel, and event type.

Next are 2 sections of json.

I'd like to read in the key/values of both json strings, but it's unclear how to approach this if the entire message is not json.

Below is my grok pattern up until the first json string:

%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}? %{WORD:loglevel}

How would I get the json items of both json strings into the record?

[fylie]

This is not a very robust solution, but if you can assume that the JSON will always be in that format (no spaces in the JSON messages, a space between the two JSON messages), you could do something like this:

filter {
   grok  {
       match => {"message" => "%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}? %{WORD:loglevel} %{WORD:event_type} %{NOTSPACE:json1} %{NOTSPACE:json2}" }
       remove_field => [ "message" ]
   }
   json {
       source => "json1" 
       remove_field => [ "json1" ]
   }
   json {
       source => "json2" 
       remove_field => [ "json2" ]
   }
}

[mikemi (Mike)]

The values may have spaces, but this still helps me understand how to approach this.

I'm wondering if there's a way to match on the { }.

For example: {%{DATA:json1}} {%{DATA:json2}} grabs the content inside the braces, but if I can figure out how to either match including the outside braces, or if there's a way I can wrap the content in braces afterwards, it should work for me.

Either way, thanks for getting me back on track.

[fylie]

You could define JSON \{.*\} as a pattern and then do

   grok  {
       patterns_dir => ["./patterns"]
       match => {"message" => "%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}? %{WORD:loglevel} %{WORD:event_type} %{JSON:json1} %{JSON:json2}" }
       remove_field => [ "message" ]
   }