Filebeat Elastic Kibana - Json based Log Processing

[RashidKhan]

Hi Team,

I have a log file that has mixed structure, some are multiple line json, some are single line json and some are plain text.
My goal is to Parse Json and send to elastic, and ignore the plain text.

I read through the documentation and it says that Filebeat can work with multiline json/ single line json and plain text (mixed logs)

now when i try to implement it some things are off

1. Multiline json are not parsed at all
2. Ignore plain text does not seem to work because it throws parsing error and still dumps the content into elastic

can you please assist

[legoguy1000 (Alex)]

Can u post ur config and examples of the different logs?

[RashidKhan]

Sure,

filebeat.yml configuration is as below

filebeat.inputs:
- type: filestream
  paths:
    - C:\Temp\FileBeatsLogs\*
  parsers:
    - ndjson:
        keys_under_root: true
        add_error_key: false
  exclude_lines: ['^#','^=','^$']
  include_lines: ['^{']

and example of log file is as below (the below is a considered a single file)

##comment line##
Request from example
System.Web.Http.Results
Ext Duration: 701
{"param1":null,"param2":null, "param3":"validmessage1","id":1}
{"param1":null,"param2":null, "param3":"validmessage2","id":2}
code : xyz


{
"param1":"value1",
"param2":"value2",
"param3":"validmessage3",
"id":3
}

##comment line##
Request from example
System.Web.Http.Results
Ext Duration: 701
{"param1":null,"param2":null, "param3":"validmessage1","id":1}
{"param1":null,"param2":null, "param3":"validmessage2","id":2}

out of this i am expecting only 5 documents in elastic but i see all even after trying numerous things. Could you please look into this and help?

[legoguy1000 (Alex)]

U shouldn't need the exclude since all u want is the json logs, the ^{ should be all u need. Have u tried it without the ndjson parser?

[RashidKhan]

yes, tried with input type "log", without the ndjson, nothing seems to work