# Multiline and flush pattern issue
**URL:** https://discuss.elastic.co/t/multiline-and-flush-pattern-issue/327958
**Category:** Beats
**Tags:** filebeat
**Created:** [March 17, 2023, 5:09pm UTC](https://discuss.elastic.co/t/multiline-and-flush-pattern-issue/327958 "2023-03-17T17:09:48Z")
**Posts on this page:** 2
**Page:** 1
### Author:  [@vipkabra](https://discuss.elastic.co/u/vipkabra)
#### Post date: [March 17, 2023, 5:09pm UTC](https://discuss.elastic.co/t/multiline-and-flush-pattern-issue/327958/1 "2023-03-17T17:09:48Z")
Hi All,
I have a requirement to pick log lines from log based on some start word, lets assume 'ABC' and combine all following lines until 'XYZ' appears in log file. If XYZ found, combine all log lines and ship them to logstash as one request, if XYZ not found then filebeat should wait and keep appending coming log lines to earlier one unless 'XYZ' found or flush timeout happens. To test this scenario i am using some sample file with below input
**hydra2.log**
ABC
1
2
XYZ
ABC
5
6
XYZ
ABC
9
10
I am expecting two documents to be inserted into elasticsearch
**First**
ABC
1
2
XYZ
**Second**
ABC
5
6
XYZ
While third document should not be inserted as end matching pattern 'XYZ' is not yet added in file after 10 value in abv sample.
However, when i run filebeat, I can see three documents inserted with the third document not having XYZ at the end it is still got flushed and pushed to elasticsearch.
**Filbeat Config**
filebeat.inputs:
- type: filestream
enabled: true
paths:
- /sbclocal/elk/filebeat-8.6.2-linux-x86\_64/hydra2.log
parsers:
- multiline:
type: pattern
pattern: 'ABC'
negate: true
match: after
flush\_pattern: 'XYZ'
fields:
type: Test
logging:
level: debug
filebeat.config.modules:
path: ${path.config}/modules.d/\*.yml
reload.enabled: false
setup.template.settings:
index.number\_of\_shards: 1
output.logstash:
hosts: ["localhost:5044"]
I see similar questions asked earlier as well but they are unanswered till date. Below are the links.
> [@Missing events while using Filebeat multiline.flush\_pattern](https://discuss.elastic.co/t/missing-events-while-using-filebeat-multiline-flush-pattern/279175):
>
> Filebeat "multiline.flush\_pattern" is flushing the multiline line event starting with "multiline.pattern" and ending with "multiline.flush\_pattern". I am facing 2 issues with multiline.flush\_pattern, The lines after "multiline.flush\_pattern" are not stored in elasticsearch. I do not see those lines even in the filebeat debug logs. I expect the events that are not matching with multiline.flush\_pattern to be stored in the elasticsearch but I do not see any of them in elasticsearch or in the fil…
> [@Multiline pattern & flush pattern in filebeat](https://discuss.elastic.co/t/multiline-pattern-flush-pattern-in-filebeat/159885):
>
> Hi All, I am using multiline pattern within filebeat.yml to format the logs as follows, filebeat.inputs: document\_type: webapp enabled: true paths: /opt/sample/app.log multiline.pattern: '^., [[^]]+]' multiline.negate: true multiline.match: after tags: ["shop"] exclude\_lines: ["^$"] fields: {application: shop} fields\_under\_root: true tail\_files: true My logs are having some unique ID to identify the block of logs to combine multiline logs. So I tried with flush pattern to do th…
I had tried adding multiline.timeout: 50s, which makes it wait before pushing third document with message ABC 9 10 unless timeout happens, but it create another issues, if i add another entry into log before timeout lets say added (11 12 XYZ) then filebeat insert 11 and 12 as seperate document and ABC 9 10 11 12 XYZ message as another document after timeout.
Any suggestion on how to effectively read all log lines which are coming between two specific patterns while there is a possibility of long wait time before file is appended with matching closing pattern ?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 14, 2023, 7:10pm UTC](https://discuss.elastic.co/t/multiline-and-flush-pattern-issue/327958/2 "2023-04-14T19:10:17Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.