Multiline and flush pattern issue

[vipkabra (Vipin)]

Hi All,

I have a requirement to pick log lines from log based on some start word, lets assume 'ABC' and combine all following lines until 'XYZ' appears in log file. If XYZ found, combine all log lines and ship them to logstash as one request, if XYZ not found then filebeat should wait and keep appending coming log lines to earlier one unless 'XYZ' found or flush timeout happens. To test this scenario i am using some sample file with below input

hydra2.log
ABC
1
2
XYZ
ABC
5
6
XYZ
ABC
9
10

I am expecting two documents to be inserted into elasticsearch
First
ABC
1
2
XYZ

Second
ABC
5
6
XYZ

While third document should not be inserted as end matching pattern 'XYZ' is not yet added in file after 10 value in abv sample.

However, when i run filebeat, I can see three documents inserted with the third document not having XYZ at the end it is still got flushed and pushed to elasticsearch.

Filbeat Config

filebeat.inputs:

• type: filestream
  enabled: true
  paths:
  • /sbclocal/elk/filebeat-8.6.2-linux-x86_64/hydra2.log
    parsers:
  • multiline:
    type: pattern
    pattern: 'ABC'
    negate: true
    match: after
    flush_pattern: 'XYZ'
    fields:
    type: Test
    logging:
    level: debug
    filebeat.config.modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

setup.template.settings:
index.number_of_shards: 1
output.logstash:
hosts: ["localhost:5044"]

I see similar questions asked earlier as well but they are unanswered till date. Below are the links.

Missing events while using Filebeat multiline.flush_pattern

Filebeat "multiline.flush_pattern" is flushing the multiline line event starting with "multiline.pattern" and ending with "multiline.flush_pattern". I am facing 2 issues with multiline.flush_pattern, The lines after "multiline.flush_pattern" are not stored in elasticsearch. I do not see those lines even in the filebeat debug logs. I expect the events that are not matching with multiline.flush_pattern to be stored in the elasticsearch but I do not see any of them in elasticsearch or in the fil…

Multiline pattern & flush pattern in filebeat

Hi All, I am using multiline pattern within filebeat.yml to format the logs as follows, filebeat.inputs: document_type: webapp enabled: true paths: /opt/sample/app.log multiline.pattern: '^., [[^]]+]' multiline.negate: true multiline.match: after tags: ["shop"] exclude_lines: ["^$"] fields: {application: shop} fields_under_root: true tail_files: true My logs are having some unique ID to identify the block of logs to combine multiline logs. So I tried with flush pattern to do th…

I had tried adding multiline.timeout: 50s, which makes it wait before pushing third document with message ABC 9 10 unless timeout happens, but it create another issues, if i add another entry into log before timeout lets say added (11 12 XYZ) then filebeat insert 11 and 12 as seperate document and ABC 9 10 11 12 XYZ message as another document after timeout.

Any suggestion on how to effectively read all log lines which are coming between two specific patterns while there is a possibility of long wait time before file is appended with matching closing pattern ?