# If conditions when dividing logs?

Hi,
I separate the logs with this config in filebeat

- type: log
enabled: true
paths:
- D:\elastic_stack\LOGS\CMS_LOGS\*
fields:
kafka_topic: "kafka-topic-cms-test1"
tags: ["CMS"]
multiline.type: pattern
multiline.pattern: '^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\s{3}-\srequest\sin\sformatHexDump\sis\s\S{2}'
multiline.negate: true
multiline.match: after
multiline.max_lines: 15000


if I found 'FU' or 'FT' in this log I want to ignore this log and add the hole message of this log in the previous message here an example of my logs:

2022-01-13 17:16:05   - request in formatHexDump is =>
000: 12 15 46 77 77 55 | ak55s2 ....
000: 12 15 46 77 77 55 | ak55s2 ....
..
FO ...
2022-01-13 17:16:10   - request in formatHexDump is =>
000: 44 66 99 11  77 55 | h44a5000 ....
000: 45 66 33 22 00 11  | adandakj ....
..
FT ....
..
2022-01-13 17:17:44   - request in formatHexDump is =>
000: 44 33 22 00 11 55 | jjdll1122 ....
000: 12 15 46 77 77 55 | h555666 ....
..
AO ...
...


so in this example i would like to find only two logs not three the first contains the logs which contain FO and FT together in one message, and the second log contains AO
I don't know if it's possible maybe with filebeat or logstash or Elasticsearch between the _id of documents maybe.
I hope you understand me.

Any help would be sincerely appreciate!
Thanks!

I haven't tested directly in FB, maybe this

^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\s{3}-\srequest\sin\sformatHexDump\sis\s\S{2}.*(?:FO|FT)?

Thank you very much, that's exactly what I want @Rios just if you can tell me what exactly is the role or your meaning of ?: in the regular expression and thank you!

( ?: non-capture groups) - result without group
(FO|FT) ?- multi line which includes FO or FT, optionally
Check here

or

1 Like

Thanks @Rios but I'm sorry it still didn't work, it was my fault, I was testing with a wrong message

What is result? FO and FT as separate lines or FO and FT lines are missing?
Can you try will this extract only FO and FT messages?
^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\s{3}-\srequest\sin\sformatHexDump\sis\s\S{2}.*(FO|FT)

here my example message :

2022-01-13 17:16:05   - request in formatHexDump is =>
000: 12 15 46 77 77 55 | ak55s2 ....
000: 12 15 46 77 77 55 | ak55s2 ....
..
FO ...
2022-01-13 17:16:10   - request in formatHexDump is =>
000: 44 66 99 11  77 55 | h44a5000 ....
000: 45 66 33 22 00 11  | adandakj ....
..
FT ....
..
2022-01-13 17:17:44   - request in formatHexDump is =>
000: 44 33 22 00 11 55 | jjdll1122 ....
000: 12 15 46 77 77 55 | h555666 ....
..
AO ...
...
2022-01-13 17:18:12   - request in formatHexDump is =>
000: 44 44 66 99 00 11 55 55 | h44a5000 ....
000: 45 66 33 22 00 11  | adandakj ....
..
FT ....
...


It always gives me 4 documents with this ^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\s{3 }-\srequest\sin\sformatHexDump\sis\s\S{2}.*(?:FO|FT)? and in this case I only want 2 documents, the first containing FO+FT and the second containing AO+ FT.
and I tried this expression ^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\s{3} -\srequest\sin\sformatHexDump\sis\s\S{2}.*(FO|FT) it gives me a document which contains the hole message

If you want like that, try to use

multiline.type: count
multiline.lines_count: 11