Filebeat include is not working when logs are in both json and non-json format

Elastic Stack Beats
1

filebeat.yml configuration
<
paths:
#- /var/log/*.log
- /ap/logs/error_log
include_lines: ['^{']
json.keys_under_root: true
json.add_error_key: true
json.overwrite_keys: true
tail_files: true
/>

filebeat logs:
018-03-30T19:19:13.578+0530 ERROR reader/json.go:32 Error decoding JSON: invalid character 'G' looking for beginning of value
2018-03-30T19:19:13.578+0530 ERROR reader/json.go:32 Error decoding JSON: invalid character 'A' looking for beginning of value
2018-03-30T19:19:13.578+0530 ERROR reader/json.go:32 Error decoding JSON: invalid character 'S' looking for beginning of value
2018-03-30T19:19:13.578+0530 ERROR reader/json.go:32 Error decoding JSON: EOF
2018-03-30T19:19:13.578+0530 ERROR reader/json.go:32 Error decoding JSON: invalid character 'P' looking for beginning of value
2018-03-30T19:19:13.579+0530 ERROR reader/json.go:32 Error decoding JSON: EOF
2018-03-30T19:19:13.579+0530 ERROR reader/json.go:32 Error decoding JSON: invalid character 'O' looking for beginning of value

in error_log file:

  1. json logs are single line.
  2. non json log can be single or multiline

Problem:
if include is applied to filebeat.yml then error logs are also shipped to elasticsearch with json logs. I want to tranfer only json logs from error_log file.
I need a quick help..
Thanks,

1 Like
2

Can you post a snippet of the log file including both JSON and non-JSON lines? Thanks

3

log file contain similar type of logs
<
Fail to add element to local cache.

Fail to add element to local cache.

Fail to add element to local cache.

{"timestamp":"2018-04-02T19:56:44:231","level":"INFO","module":"TIME_PROFILING","filename":"../cache_time.c","line":2214,"event":"Setting time profiling","user_point":"filter_and_inits","unique_id":"1234","total_time":782054.354492,"component_time":23.689912}

Fail to add element to local cache.
/>

json logs are on single line

4

actually, firstly I want to apply a filter to include only json logs and then I want to parse it in json through filebeat.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.