# Filebeat is not able to parse json from files where \\n separated json lines (events) are written

**URL:** <https://discuss.elastic.co/t/filebeat-is-not-able-to-parse-json-from-files-where-n-separated-json-lines-events-are-written/192927>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [July 30, 2019, 2:54pm UTC](https://discuss.elastic.co/t/filebeat-is-not-able-to-parse-json-from-files-where-n-separated-json-lines-events-are-written/192927 "2019-07-30T14:54:22Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![nyet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nyet/32/51254_2.png) [@nyet](https://discuss.elastic.co/u/nyet)\
**Post date:** [July 30, 2019, 2:54pm UTC](https://discuss.elastic.co/t/filebeat-is-not-able-to-parse-json-from-files-where-n-separated-json-lines-events-are-written/192927/1 "2019-07-30T14:54:22Z")

</div>

I have a log source that generates json log files with excess line feeds.

I can get around this (partially) by adding `exclude_lines: ['^$']` as suggested here:

> [@Json logs separated by blank lines](https://discuss.elastic.co/t/json-logs-separated-by-blank-lines/143523):
>
> i have a json log file, where each entry is separated by a blank line {"@timestamp":1456544565,"valid\_domain":true,"ip":"8.8.8.8"} {"@timestamp":2464564543,"valid\_domain":false,"ip":"1.2.3.4"} {"@timestamp":3454678735,"valid\_domain":false,"ip":"9.8.7.6"} filebeat refused to parse it, giving the error 2018-08-08T10:08:43.626-0300 ERROR reader/json.go:33 Error decoding JSON: EOF so i tried excluding empty lines with exclude\_lines : ['^$'] and it worked! until... i restarted the service. th…

This solves the problem on the logstash/ES side, but syslog where file beats is running is completely filled with

`2019-07-30T14:45:41.319Z#011ERROR#011readjson/json.go:52#011Error decoding JSON: EOF`

See also:

> <https://github.com/elastic/beats/issues/5551>
>
> I am using BELK stack for Log Analytics.
> \--------------------------------------…----------------------------------------------------------
> versions:
> ES-5.1.1 is on AWS service
> Logstash-5.3.0, filebeat-5.3.0 and kibana-5.1.1
> \------------------------------------------------------------------------------------------------
> filebeat is running on the server where cdrs are generated.
> Logstash is running on separate server and sends data to date based rolling index in elasticsearch.
> 
> We are using java logback.xml to write single line json events in .log file which rotates basis size and time logic
> 
> \<appender name="SIZE\_AND\_TIME\_BASED\_LOG" class="ch.qos.logback.core.rolling.RollingFileAppender"\>
> \<file\>${CDR\_NEW\_LOG\_PATH}//cdr.log\</file\>
> \<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"\>
> \<fileNamePattern\>${CDR\_NEW\_LOG\_PATH}//cdr-${IP\_ADD}.%d{yyyy-MM-dd-HH-mm}.%i.log\</fileNamePattern\>
> \<timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP"\>
> \<maxFileSize\>${CDR\_NEW\_LOG\_MAX\_FILE\_SIZE}\</maxFileSize\>
> \</timeBasedFileNamingAndTriggeringPolicy\>
> \</rollingPolicy\>
> \------------------------------------------------------------------------------------------------
> In filebeat.yml,
> filebeat.prospectors:
> 
> \- input\_type: log
> 
> paths:
> - /opt/cdrs/\*-\*.log
> 
> 
> json.keys\_under\_root: true
> json.add\_error\_key: true
> 
> ignore\_older: 24h
> close\_inactive: 12h
> scan\_frequency: 30s
> clean\_inactive: 48h
> clean\_removed: true
> 
> close\_removed: true
> 
> close\_eof: true
> output.logstash:
> hosts: \["VALID IP:5043"\]
> fields\_under\_root: false
> \------------------------------------------------------------------------------------------------
> Logstash Config
> 
> input {
> beats {
> port =\> 5043
> client\_inactivity\_timeout =\> 86400
> }
> }
> 
> filter
> {
> grok {
> match =\> { "g2uEvent" =\> "%{TIMESTAMP\_ISO8601:g2uEventTime}"}
> }
> 
> date {
> match =\> \["g2uEventTime", "ISO8601"\]
> target =\> "@timestamp"
> }
> 
> }
> 
> output {
> 
> amazon\_es {
> hosts =\> \["VALID ELASTIC SEARCH END POINT"\]
> index =\> "d2c-%{+YYYY-MM-dd}"
> }
> 
> stdout { codec =\> rubydebug }
> }
> 
> \------------------------------------------------------------------------------------------------
> Issues come randomly when there is traffic. If the same file is sent to filebeat, all data is correctly send to ES.
> 
> Common erros in filebeat:
> ERR Error decoding JSON: invalid character '}' looking for beginning of value
> 
> ERR Error decoding JSON: json: cannot unmarshal string into Go value of type map\[string\]interface {}
> 
> ERR Error decoding JSON: EOF
> 
> ERR Error decoding JSON: invalid character '\\n' in string literal
> 
> ERR Error decoding JSON: invalid character 'm' in literal true (expecting 'r')
> 
> \------------------------------------------------------------------------------------------------
> Sample CDR
> {"g2uEvent":{"g2uEventName":"CAMPAIGN\_USER\_LOGIN\_MOBILE\_DATA\_EVENT","g2uEventTime":"2017-11-10T05:22:32.270Z","g2uEventDate":"2017-11-10"},"g2uCountry":{"g2uCountryIsoname":"IN","g2u\_analytic\_country\_name":"IND"},"g2uOperatorInfo":{"g2uOperatorName":"Idea","g2uOperatorAnalyticsname":"IDEA"},"g2uLanguage":"en","g2uUser":{"g2uUserMsisdn":"91xxxxxxxx","g2uUserChannel":"mobileData","g2uUserStatus":"STATUS\_EXPIRED"},"g2uGames":{"g2uGamesId":"","g2uGamesName":"","g2uGamesCategory":""},"g2uGamesVendor":{"g2uGamesVendorName":""},"g2uGamesUtm":{"g2uUtmSource":"test","g2uUtmMedium":"18333108","g2uUtmCampaign":"ADS"}}
> {"g2uEvent":{"g2uEventName":"CAMPAIGN\_USER\_ADVERTISER\_CALLBACK\_BLOCKED\_EVENT","g2uEventTime":"2017-11-09T14:21:45.918Z","g2uEventDate":"2017-11-09"},"g2uCountry":{"g2uCountryIsoname":"MY","g2uAnalyticCountryName":""},"g2uOperatorInfo":{"g2uOperatorName":"Maxis","g2uOperatorAnalyticsname":""},"g2uUser":{"g2uUserMsisdn":"60xxxxxxx","g2uUserChannel":"MobileData","g2uUserStatus":"STATUS\_SUBSCRIBED"},"g2uGames":{"g2uGamesId":"","g2uGamesName":"","g2uGamesCategory":""},"g2uGamesVendor":{"g2uGamesVendorName":""},"g2uGamesUtm":{"g2uUtmSource":"test1","g2uUtmMedium":"18290630","g2uUtmCampaign":"ADS"}}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 27, 2019, 3:03pm UTC](https://discuss.elastic.co/t/filebeat-is-not-able-to-parse-json-from-files-where-n-separated-json-lines-events-are-written/192927/2 "2019-08-27T15:03:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.