# Filebeat is not able to parse json from files where \\n separated json lines (events) are written. It sometimes misses the records and sometimes gives error when traffic is high

**URL:** <https://discuss.elastic.co/t/filebeat-is-not-able-to-parse-json-from-files-where-n-separated-json-lines-events-are-written-it-sometimes-misses-the-records-and-sometimes-gives-error-when-traffic-is-high/107155>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [November 10, 2017, 8:39am UTC](https://discuss.elastic.co/t/filebeat-is-not-able-to-parse-json-from-files-where-n-separated-json-lines-events-are-written-it-sometimes-misses-the-records-and-sometimes-gives-error-when-traffic-is-high/107155 "2017-11-10T08:39:33Z")\
**Posts on this page:** 1\
**Showing post:** 9

<div class="post-metadata">

**Author:** ![Aman\_Sehgal](https://avatars.discourse-cdn.com/v4/letter/a/b38774/32.png) [@Aman\_Sehgal](https://discuss.elastic.co/u/Aman_Sehgal)\
**Post date:** [November 17, 2017, 7:07am UTC](https://discuss.elastic.co/t/filebeat-is-not-able-to-parse-json-from-files-where-n-separated-json-lines-events-are-written-it-sometimes-misses-the-records-and-sometimes-gives-error-when-traffic-is-high/107155/9 "2017-11-17T07:07:38Z")

</div>

Hi Ruffin,

If I transfer the same file again, it is processed successfully without any error.

Filebeat picks already rotated files.

filebeat config :

In filebeat.yml,

filebeat.prospectors:

- input\_type: log  
paths:  
/root/cdrs/\*-\*.log  
json.keys\_under\_root: true  
json.add\_error\_key: true

ignore\_older: 24h  
close\_inactive: 12h  
scan\_frequency: 30s  
clean\_inactive: 48h  
clean\_removed: true  
close\_removed: true  
close\_eof: true

output.logstash:

hosts: ["VALID IP:5043"]  
fields\_under\_root: false

////////////////////////////////////////////////////////////////////////  
Logstash Config:

input {  
beats {  
port =\> 5043  
client\_inactivity\_timeout =\> 86400  
}  
}

filter  
{  
grok {  
match =\> { "g2uEvent" =\> "%{TIMESTAMP\_ISO8601:g2uEventTime}"}  
}

date {  
match =\> ["g2uEventTime", "ISO8601"]  
target =\> "@timestamp"  
}

}

output {

amazon\_es {  
hosts =\> ["VALID ELASTIC SEARCH END POINT"]  
index =\> "d2c-%{+YYYY-MM-dd}"  
}

stdout { codec =\> rubydebug }  
}

---

_[View the full topic](https://discuss.elastic.co/t/filebeat-is-not-able-to-parse-json-from-files-where-n-separated-json-lines-events-are-written-it-sometimes-misses-the-records-and-sometimes-gives-error-when-traffic-is-high/107155)._