How to configure filebeat to send multiline logs only after the log file is finished updating?

Elastic Stack Beats
1

I am using filebeat-5.6
My config filebeat -> logstash -> elasticsearch -> kibana

I have hundreds of log files with 1KB - 2KB data in the path "c:\programdata\AICdameon\ "
All the files get created when the background jobs start's and the files get updated at regular intervals.
How to make file beat send the whole log file as a single message to elasticsearch only when the log file is finished updating??.
All log file's start with the line "starting jobid 247HQD51" and ends with the line "end job 247HQD51".
My below filebeat config is not working as expected, my filebeat sends logs as soon as they get updated in the log file.
---
filebeat:
prospectors:
- input_type: log
paths:
- c:\programdata\AICdameon*
fields:
type: backupHQD
fields_under_root: true
multiline.pattern: '*'
multiline.negate: true
multiline.match: after

2

You could use the following multiline config:

multiline.pattern: "^end job [0-9A-Z]+$"
multiline.negate: true
multiline.match: "before"

Thus, the whole file will be in one event.
Please note, if your log file it too big, it might lead to too big resource consumption or your Beat might run out of memory.

3

Hi kvch, @kvch I tried your config but i can see each line in the log file getting populated into seperate document's in kibana
Below is my log file with 9 lines
starting jobid 247HQD51
Status 83
LOG 1517035863 4 sgtm 15937 Backup started
LOG 1517035863 4 sgtm 2788 start
LOG 1517035863 4 sgtm 2788 using 262144 data buffer size
LOG 1517035863 4 sgtm 2788 setting receive network buffer to 1049600 bytes
LOG 1517035863 4 sgtm 2788 using 30 data buffers
LOG 1517036526 32 sgtm 2788 sts_get_server_prop_byname failed: error 2060046 plugin error
end job 247HQD51

when the file gets populated with first 5 lines in first 2 minutes then am seeing these five log messages in five different documents in elastisearch(but i am trying to get the whole log file as a single message depending on the last line) and in the next 2 minutes when the log file gets populated with remianing 4 lines i am seeing the next 4 lines getting populated into a single document .

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.