# Filebeat include is not working when logs are in both json and non-json format

**URL:** <https://discuss.elastic.co/t/filebeat-include-is-not-working-when-logs-are-in-both-json-and-non-json-format/126372>\
**Category:** Beats\
**Created:** [April 2, 2018, 5:27am UTC](https://discuss.elastic.co/t/filebeat-include-is-not-working-when-logs-are-in-both-json-and-non-json-format/126372 "2018-04-02T05:27:02Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Abhishek\_Jadhav](https://avatars.discourse-cdn.com/v4/letter/a/bc79bd/32.png) [@Abhishek\_Jadhav](https://discuss.elastic.co/u/Abhishek_Jadhav)\
**Post date:** [April 2, 2018, 5:27am UTC](https://discuss.elastic.co/t/filebeat-include-is-not-working-when-logs-are-in-both-json-and-non-json-format/126372/1 "2018-04-02T05:27:03Z")

</div>

filebeat.yml configuration  
\<  
paths:  
#- /var/log/\*.log  
- /ap/logs/error\_log  
include\_lines: ['^{']  
json.keys\_under\_root: true  
json.add\_error\_key: true  
json.overwrite\_keys: true  
tail\_files: true  
/\>

filebeat logs:  
018-03-30T19:19:13.578+0530 ERROR reader/json.go:32 Error decoding JSON: invalid character 'G' looking for beginning of value  
2018-03-30T19:19:13.578+0530 ERROR reader/json.go:32 Error decoding JSON: invalid character 'A' looking for beginning of value  
2018-03-30T19:19:13.578+0530 ERROR reader/json.go:32 Error decoding JSON: invalid character 'S' looking for beginning of value  
2018-03-30T19:19:13.578+0530 ERROR reader/json.go:32 Error decoding JSON: EOF  
2018-03-30T19:19:13.578+0530 ERROR reader/json.go:32 Error decoding JSON: invalid character 'P' looking for beginning of value  
2018-03-30T19:19:13.579+0530 ERROR reader/json.go:32 Error decoding JSON: EOF  
2018-03-30T19:19:13.579+0530 ERROR reader/json.go:32 Error decoding JSON: invalid character 'O' looking for beginning of value

in error\_log file:

1. json logs are single line.
2. non json log can be single or multiline

Problem:  
if include is applied to filebeat.yml then error logs are also shipped to elasticsearch with json logs. I want to tranfer only json logs from error\_log file.  
I need a quick help..  
Thanks,

---

<div class="post-metadata">

**Author:** ![adrisr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrisr/32/25423_2.png) [@adrisr](https://discuss.elastic.co/u/adrisr)\
**Post date:** [April 2, 2018, 1:05pm UTC](https://discuss.elastic.co/t/filebeat-include-is-not-working-when-logs-are-in-both-json-and-non-json-format/126372/2 "2018-04-02T13:05:42Z")

</div>

Can you post a snippet of the log file including both JSON and non-JSON lines? Thanks

---

<div class="post-metadata">

**Author:** ![Abhishek\_Jadhav](https://avatars.discourse-cdn.com/v4/letter/a/bc79bd/32.png) [@Abhishek\_Jadhav](https://discuss.elastic.co/u/Abhishek_Jadhav)\
**Post date:** [April 2, 2018, 2:41pm UTC](https://discuss.elastic.co/t/filebeat-include-is-not-working-when-logs-are-in-both-json-and-non-json-format/126372/3 "2018-04-02T14:41:43Z")

</div>

log file contain similar type of logs  
\<  
Fail to add element to local cache.

Fail to add element to local cache.

Fail to add element to local cache.

{"timestamp":"2018-04-02T19:56:44:231","level":"INFO","module":"TIME\_PROFILING","filename":"../cache\_time.c","line":2214,"event":"Setting time profiling","user\_point":"filter\_and\_inits","unique\_id":"1234","total\_time":782054.354492,"component\_time":23.689912}

Fail to add element to local cache.  
/\>

json logs are on single line

---

<div class="post-metadata">

**Author:** ![Abhishek\_Jadhav](https://avatars.discourse-cdn.com/v4/letter/a/bc79bd/32.png) [@Abhishek\_Jadhav](https://discuss.elastic.co/u/Abhishek_Jadhav)\
**Post date:** [April 2, 2018, 2:44pm UTC](https://discuss.elastic.co/t/filebeat-include-is-not-working-when-logs-are-in-both-json-and-non-json-format/126372/4 "2018-04-02T14:44:08Z")

</div>

actually, firstly I want to apply a filter to include only json logs and then I want to parse it in json through filebeat.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 30, 2018, 4:44pm UTC](https://discuss.elastic.co/t/filebeat-include-is-not-working-when-logs-are-in-both-json-and-non-json-format/126372/5 "2018-04-30T16:44:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.