# Parsing log with partial json

**URL:** <https://discuss.elastic.co/t/parsing-log-with-partial-json/79674>\
**Category:** Logstash\
**Created:** [March 23, 2017, 3:00am UTC](https://discuss.elastic.co/t/parsing-log-with-partial-json/79674 "2017-03-23T03:00:32Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![mikemi](https://avatars.discourse-cdn.com/v4/letter/m/ec9cab/32.png) [@mikemi](https://discuss.elastic.co/u/mikemi)\
**Post date:** [March 23, 2017, 3:00am UTC](https://discuss.elastic.co/t/parsing-log-with-partial-json/79674/1 "2017-03-23T03:00:32Z")

</div>

> 2017-03-21T23:53:10-04:00 INFO login {"input":"user123","success":false} {"host":"[www.example.com](http://www.example.com)","path":"user","clientip":"127.0.0.1"}

This log file starts with a timestmap, loglevel, and event type.

Next are 2 sections of json.

I'd like to read in the key/values of both json strings, but it's unclear how to approach this if the entire message is not json.

Below is my grok pattern up until the first json string:

> %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601\_TIMEZONE}? %{WORD:loglevel}

How would I get the json items of both json strings into the record?

---

<div class="post-metadata">

**Author:** ![fylie](https://avatars.discourse-cdn.com/v4/letter/f/e8c25b/32.png) [@fylie](https://discuss.elastic.co/u/fylie)\
**Post date:** [March 23, 2017, 3:30am UTC](https://discuss.elastic.co/t/parsing-log-with-partial-json/79674/2 "2017-03-23T03:30:40Z")

</div>

This is not a very robust solution, but if you can assume that the JSON will always be in that format (no spaces in the JSON messages, a space between the two JSON messages), you could do something like this:

```
filter {
   grok {
       match => {"message" => "%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}? %{WORD:loglevel} %{WORD:event_type} %{NOTSPACE:json1} %{NOTSPACE:json2}" }
       remove_field => ["message"]
   }
   json {
       source => "json1" 
       remove_field => ["json1"]
   }
   json {
       source => "json2" 
       remove_field => ["json2"]
   }
}
```

---

<div class="post-metadata">

**Author:** ![mikemi](https://avatars.discourse-cdn.com/v4/letter/m/ec9cab/32.png) [@mikemi](https://discuss.elastic.co/u/mikemi)\
**Post date:** [March 23, 2017, 3:52am UTC](https://discuss.elastic.co/t/parsing-log-with-partial-json/79674/3 "2017-03-23T03:52:25Z")

</div>

The values may have spaces, but this still helps me understand how to approach this.

I'm wondering if there's a way to match on the `{ }`.

For example: `{%{DATA:json1}} {%{DATA:json2}}` grabs the content inside the braces, but if I can figure out how to either match including the outside braces, or if there's a way I can wrap the content in braces afterwards, it should work for me.

Either way, thanks for getting me back on track.

---

<div class="post-metadata">

**Author:** ![fylie](https://avatars.discourse-cdn.com/v4/letter/f/e8c25b/32.png) [@fylie](https://discuss.elastic.co/u/fylie)\
**Post date:** [March 24, 2017, 1:19am UTC](https://discuss.elastic.co/t/parsing-log-with-partial-json/79674/4 "2017-03-24T01:19:22Z")

</div>

You could define `JSON \{.*\}` as a pattern and then do

```
   grok {
       patterns_dir => ["./patterns"]
       match => {"message" => "%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}? %{WORD:loglevel} %{WORD:event_type} %{JSON:json1} %{JSON:json2}" }
       remove_field => ["message"]
   }
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 21, 2017, 1:19am UTC](https://discuss.elastic.co/t/parsing-log-with-partial-json/79674/5 "2017-04-21T01:19:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.