# Trouble parsing multi-line json logs

**URL:** <https://discuss.elastic.co/t/trouble-parsing-multi-line-json-logs/147628>\
**Category:** Beats\
**Created:** [September 6, 2018, 8:00pm UTC](https://discuss.elastic.co/t/trouble-parsing-multi-line-json-logs/147628 "2018-09-06T20:00:10Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![kreg](https://avatars.discourse-cdn.com/v4/letter/k/ed655f/32.png) [@kreg](https://discuss.elastic.co/u/kreg)\
**Post date:** [September 6, 2018, 8:00pm UTC](https://discuss.elastic.co/t/trouble-parsing-multi-line-json-logs/147628/1 "2018-09-06T20:00:10Z")

</div>

- Beat version
  - 6.4.0

- Operating System
  - Windows

Maybe what I'm trying to do is unsupported (the answer in [Error decoding JSON: json: cannot unmarshal string into Go value of type map[string]interface {}](https://discuss.elastic.co/t/error-decoding-json-json-cannot-unmarshal-string-into-go-value-of-type-map-string-interface/134498) would indicate that's the case), but looking at the FileBeats documentation ([Stdin input | Filebeat Reference [6.4] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/6.4/filebeat-input-stdin.html)), it seems like I should be able to do this based on the sentence

`The decoding happens before line filtering and multiline. You can combine JSON decoding with filtering and multiline if you set the "message_key" option.`

To be fair, the line immediately above the statement I mentioned says `These options make it possible for Filebeat to decode logs structured as JSON messages. Filebeat processes the logs line by line, so the JSON decoding only works if there is one JSON object per line.` - so there's a good chance I'm just misunderstanding the documentation. To me, those two statements seem to conflict with one another.

I'm trying to parse logs that look like the following:

> {  
> "messageKey": "Hello!",  
> "blahKey": "blahValue",  
> "kregsKey": "Hello!",  
> "event\_id": "32cebd5d-1542-4703-a5a3-be5eaa90af81",  
> "level": "error",  
> "logger": "android-exception"  
> }

And I'm getting the following error message:

> 2018-09-06T13:25:55.797-0600 DEBUG [publish] pipeline/processor.go:308 Publish event: {  
> "@timestamp": "2018-09-06T19:25:50.792Z",  
> "@metadata": {  
> "beat": "filebeat",  
> "type": "doc",  
> "version": "6.4.0"  
> },  
> "offset": 3508,  
> "json": {  
> "error": {  
> "type": "json",  
> "message": "Error decoding JSON: json: cannot unmarshal string into Go value of type \>map[string]interface {}"  
> },  
> "messageKey": "{\n\t"messageKey": "Hello!",\n\t"blahKey": "blahValue",\n\t"kregsKey": \>"Hello!",\n\t"event\_id": "32cebd5d-1542-4703-a5a3-be5eaa90af81",\n\t"level": \>"error",\n\t"logger": "android-exception""  
> },  
> "input": {  
> "type": "log"  
> },  
> "prospector": {  
> "type": "log"  
> },  
> "beat": {  
> "hostname": "LYNCHC18",  
> "version": "6.4.0",  
> "name": "LYNCHC18"  
> },  
> "host": {  
> "name": "LYNCHC18"  
> },  
> "source": "G:\kregsTestLog2.log"  
> }

If I condense the json message into one line, the json is parsed and sent correctly. I'm hoping to avoid the requirement to condense all logs into one line for readability purposes, as my logs are ingested by multiple people, not all of which use Kibana.

Am I misunderstanding the documentation, and filebeat doesn't actually support multiline json parsing? Here's my current configuration:

> filebeat.inputs:
> 
> - type: log  
> enabled: true  
> paths:
> - G:\kregsTestLog2.log  
> json.message\_key: messageKey  
> json.keys\_under\_root: false  
> json.add\_error\_key: true  
> multiline.pattern: '^{'  
> multiline.negate: true  
> multiline.match: after
> 
> filebeat.config.modules:  
> path: ${path.config}/modules.d/\*.yml  
> reload.enabled: false
> 
> setup.template.settings:  
> index.number\_of\_shards: 3
> 
> setup.kibana:
> 
> output.elasticsearch:  
> hosts: ["search-craigs-test-elasticsearch-uhvnj6zt5s3px2rldniycjslga.us-east-\> [2.es.amazonaws.com:443](http://2.es.amazonaws.com:443)"]  
> protocol: "https"

Thanks so much in advance for your help!

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [September 6, 2018, 10:28pm UTC](https://discuss.elastic.co/t/trouble-parsing-multi-line-json-logs/147628/2 "2018-09-06T22:28:35Z")

</div>

What do you mean by multiline JSON parsing? Aggregating the lines under `messageKey`?

If yes, you need to set `json.keys_under_root` to true. This way Filebeat puts the message "Hello!" under the root of the event as `messageKey`. Then you can do the multiline aggregation based `messageKey`.

---

<div class="post-metadata">

**Author:** ![kreg](https://avatars.discourse-cdn.com/v4/letter/k/ed655f/32.png) [@kreg](https://discuss.elastic.co/u/kreg)\
**Post date:** [September 11, 2018, 8:53pm UTC](https://discuss.elastic.co/t/trouble-parsing-multi-line-json-logs/147628/3 "2018-09-11T20:53:38Z")

</div>

I'm just trying to get similar functionality whether the event is multi-line or on a single line. The following works without issue:

> { "messageKey": "Hello!", "blahKey": "blahValue", "kregsKey": "Hello!", "event\_id": "32cebd5d-1542-4703-a5a3-be5eaa90af81", "level": "error", "logger": "android-exception" }

In Elasticsearch (Kibana), the record shows up like so:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/c/cce7c586b9a2b75bd1781d39eb02b0b75c2a563c.png)

But as soon as I add additional lines like so:

> {  
> "messageKey": "Hello!",  
> "blahKey": "blahValue",  
> "kregsKey": "Hello!",  
> "event\_id": "32cebd5d-1542-4703-a5a3-be5eaa90af81",  
> "level": "error",  
> "logger": "android-exception"  
> }

I get the error `"Error decoding JSON: json: cannot unmarshal string into Go value of type map[string]interface {}"`

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/d/bdda3dd48923483b1d8a08a436cca2167faf98ba.png)

It might also be worth mentioning that that bottom error `invalid character '}' looking for beginning of value` is what shows up as the Elasticsearch error:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/f/cf7dcab1c7f9ca7ffe59420fa96af2c9f1747ff3.png)

Hopefully this makes things clearer. I don't actually care about the `messageKey` key in the json - I only added that to try to get things working, since if I understand the documentation correctly:  
`You can combine JSON decoding with filtering and multiline if you set the "message_key" option` - Then the message key appears to be necessary if I want to capture multiline json logs.

By the way, I did try setting `json.keys_under_root` to true and am getting the same error ☹

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 9, 2018, 10:53pm UTC](https://discuss.elastic.co/t/trouble-parsing-multi-line-json-logs/147628/4 "2018-10-09T22:53:41Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.