# Filebeat - Decoding Json logs with Linebreaks

**URL:** <https://discuss.elastic.co/t/filebeat-decoding-json-logs-with-linebreaks/136394>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [June 19, 2018, 12:38am UTC](https://discuss.elastic.co/t/filebeat-decoding-json-logs-with-linebreaks/136394 "2018-06-19T00:38:17Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![motamedi791](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@motamedi791](https://discuss.elastic.co/u/motamedi791)\
**Post date:** [June 19, 2018, 12:38am UTC](https://discuss.elastic.co/t/filebeat-decoding-json-logs-with-linebreaks/136394/1 "2018-06-19T00:38:18Z")

</div>

I was able to make FileBeat work with json log files. However, when a log message contains line breaks, the json parser can not decode the json.

For example, the following log line can not be decoded correctly:

`{"timestamp" :"2018-06-18 20:28:22.121", "message": "line1 \nline2 \nline3", "level":"info"}`

Is it there a way to fix this problem in filebeat?

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [June 19, 2018, 3:22pm UTC](https://discuss.elastic.co/t/filebeat-decoding-json-logs-with-linebreaks/136394/2 "2018-06-19T15:22:27Z")

</div>

Does the log message on disk contain `\n` or a newline? What error does Filebeat give while parsing? And what configuration are you using.

Using the exact JSON you posted above works for me.

```auto
filebeat.prospectors:
- paths: [input.json]
  json.keys_under_root: true

output.console.pretty: true

```

```auto
{
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.2.3"
  },
  "@timestamp": "2018-06-19T15:20:31.260Z",
  "beat": {
    "hostname": "macbook",
    "name": "macbook",
    "version": "6.2.3"
  },
  "level": "info",
  "message": "line1 \nline2 \nline3",
  "offset": 93,
  "source": "/Users/akroh/go/src/github.com/elastic/beats/filebeat/.test/json-newline/input.json",
  "timestamp": "2018-06-18 20:28:22.121"
}

```

---

<div class="post-metadata">

**Author:** ![motamedi791](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@motamedi791](https://discuss.elastic.co/u/motamedi791)\
**Post date:** [June 20, 2018, 12:53pm UTC](https://discuss.elastic.co/t/filebeat-decoding-json-logs-with-linebreaks/136394/3 "2018-06-20T12:53:43Z")

</div>

The logfile itself contains line breaks (not just \n) and json decoding fails.

Here is my filebeat config:

```
filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /log/*.log  
  multiline.pattern: '^{'
  multiline.negate: true
  multiline.match: after
  processors:
     - decode_json_fields:
         fields: ["timestamp", "message", "level"]
         process_array: false
         max_depth: 1
         target: ""
         overwrite_keys: true

```

Here is the error message:

```
Invalid format: \"line2 \"
Invalid format: \"line3\", \"level\":\"info\"}\"

```

It look like the the line break cause the json decoder to receive part of of line (as opposed to the whole line ) and fail.

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [June 20, 2018, 1:09pm UTC](https://discuss.elastic.co/t/filebeat-decoding-json-logs-with-linebreaks/136394/4 "2018-06-20T13:09:05Z")

</div>

If the string value contains line feeds then it's not valid JSON because all control characters must be escaped (see [rfc 7151 section 7](https://tools.ietf.org/html/rfc7159#section-7)).

Filebeat (decode\_json\_fields) can handle pretty printed JSON where the object spans multiple lines, but it cannot handle this case where the string values contain control characters.

One possible solution is to do the multiline in Filebeat and the JSON decoding in Logstash. Prior to the JSON filter you could replace the line feeds with `\n` or `\u000a`.

Or you could modify the thing writing the logs to do JSON escaping.

---

<div class="post-metadata">

**Author:** ![motamedi791](https://avatars.discourse-cdn.com/v4/letter/m/5e9695/32.png) [@motamedi791](https://discuss.elastic.co/u/motamedi791)\
**Post date:** [June 20, 2018, 2:50pm UTC](https://discuss.elastic.co/t/filebeat-decoding-json-logs-with-linebreaks/136394/5 "2018-06-20T14:50:15Z")

</div>

Make sense. Thanks. Is there a way to pre-process messages in FileBeat to replace line break withs with '\n' without using LogStash? ( By using pipelines, etc)

Log message -\> Replace line breaks with \n -\> Decode Json -\> Pretty print in Kibana.

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [June 21, 2018, 9:50pm UTC](https://discuss.elastic.co/t/filebeat-decoding-json-logs-with-linebreaks/136394/6 "2018-06-21T21:50:04Z")

</div>

Without LS... you could probably accomplish it with an Ingest Node pipeline that uses [gsub](https://www.elastic.co/guide/en/elasticsearch/reference/master/gsub-processor.html) and then [json](https://www.elastic.co/guide/en/elasticsearch/reference/master/json-processor.html).

Once you create the pipeline you add it to your prospector config. See [`pipeline`](https://www.elastic.co/guide/en/beats/filebeat/6.3/filebeat-input-log.html#_literal_pipeline_literal).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 19, 2018, 11:50pm UTC](https://discuss.elastic.co/t/filebeat-decoding-json-logs-with-linebreaks/136394/7 "2018-07-19T23:50:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.