# Json logs separated by blank lines

**URL:** <https://discuss.elastic.co/t/json-logs-separated-by-blank-lines/143523>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [August 8, 2018, 1:24pm UTC](https://discuss.elastic.co/t/json-logs-separated-by-blank-lines/143523 "2018-08-08T13:24:16Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![yodog](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yodog/32/4822_2.png) [@yodog](https://discuss.elastic.co/u/yodog)\
**Post date:** [August 8, 2018, 1:24pm UTC](https://discuss.elastic.co/t/json-logs-separated-by-blank-lines/143523/1 "2018-08-08T13:24:17Z")

</div>

i have a json log file, where each entry is separated by a blank line

```
{"@timestamp":1456544565,"valid_domain":true,"ip":"8.8.8.8"}

{"@timestamp":2464564543,"valid_domain":false,"ip":"1.2.3.4"}

{"@timestamp":3454678735,"valid_domain":false,"ip":"9.8.7.6"}

```

filebeat refused to parse it, giving the error

```
2018-08-08T10:08:43.626-0300	ERROR	reader/json.go:33	Error decoding JSON: EOF

```

so i tried excluding empty lines with

```
exclude_lines : ['^$']

```

and it worked! until... i restarted the service. then filebeat refused to start, informing this error

```
2018-08-08T10:09:15.257-0300	ERROR	instance/beat.go:691	Exiting: Error in initing input: When using the JSON decoder and line filtering together, you need to specify a message_key value accessing 'filebeat.inputs.2' (source:'/etc/filebeat/filebeat.logstash.yml')

```

i don't have (want?) a `message_key`.

so, i have 3 questions here:

1- why aren't the blank lines ignored by default when parsing json?  
the way i see it, if the curly braces are closed `}` it should end the parsing and ignore all empty lines until the next opening curly brace `{` is found.

2- `exclude_lines : ['^$']` does work. but after the restart filebeat makes sure it doesn't. why? it works! leave it be...

3- what can i do now?

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [August 8, 2018, 2:07pm UTC](https://discuss.elastic.co/t/json-logs-separated-by-blank-lines/143523/2 "2018-08-08T14:07:09Z")

</div>

> 1- why aren't the blank lines ignored by default when parsing json?  
> the way i see it, if the curly braces are closed } it should end the parsing and ignore all empty lines until the next opening curly brace { is found.

The current Filebeat engine works on lines and by default the JSON reader that we use assume that each line is a valid document, in your case an empty line trigger an error. Each line is parsed individually, maybe in a future version we could change it to emit a new json document when a document is valid (complete) but this is not the case today. Also I think even by switching to that, it would still trip on the empty line.

> 2- exclude\_lines : ['^$'] does work. but after the restart filebeat makes sure it doesn't. why? it works! leave it be...

I am not sure, it should have returned the same errors everytime filebeat is started.

> 3- what can i do now?

You have to define a `message_key`, when you use _exclude lines_, in filebeat it will generate an internal event with the json document in the field `message`, so you have to use that value to correctly extract the document.

---

<div class="post-metadata">

**Author:** ![yodog](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yodog/32/4822_2.png) [@yodog](https://discuss.elastic.co/u/yodog)\
**Post date:** [August 9, 2018, 7:41pm UTC](https://discuss.elastic.co/t/json-logs-separated-by-blank-lines/143523/3 "2018-08-09T19:41:09Z")

</div>

but if the log line is perfectly parsed without defining `message_key`, why make it obligatory?  
(as long as i never restart the service, i get exactly the result i want).

or did i just got lucky?

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [August 9, 2018, 7:55pm UTC](https://discuss.elastic.co/t/json-logs-separated-by-blank-lines/143523/4 "2018-08-09T19:55:15Z")

</div>

> but if the log line is perfectly parsed without defining `message_key` , why make it obligatory?

It because when we skip line we have to do 2 steps to parse it.

> or did i just got lucky?

I would expect the same result without restart.

What application produces space in their json log?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 6, 2018, 7:55pm UTC](https://discuss.elastic.co/t/json-logs-separated-by-blank-lines/143523/5 "2018-09-06T19:55:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.