If conditions when dividing logs?

Elastic Stack Beats
1

Hi,
I separate the logs with this config in filebeat

- type: log
  enabled: true
  paths:
    - D:\elastic_stack\LOGS\CMS_LOGS\*
  fields:
        kafka_topic: "kafka-topic-cms-test1"
  tags: ["CMS"]
  multiline.type: pattern
  multiline.pattern: '^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\s{3}-\srequest\sin\sformatHexDump\sis\s\S{2}'
  multiline.negate: true
  multiline.match: after
  multiline.max_lines: 15000

if I found 'FU' or 'FT' in this log I want to ignore this log and add the hole message of this log in the previous message here an example of my logs:

2022-01-13 17:16:05   - request in formatHexDump is => 
000: 12 15 46 77 77 55 | ak55s2 ....
000: 12 15 46 77 77 55 | ak55s2 ....
..
FO ... 
2022-01-13 17:16:10   - request in formatHexDump is => 
000: 44 66 99 11  77 55 | h44a5000 ....
000: 45 66 33 22 00 11  | adandakj ....
..
FT ....
..
2022-01-13 17:17:44   - request in formatHexDump is => 
000: 44 33 22 00 11 55 | jjdll1122 ....
000: 12 15 46 77 77 55 | h555666 ....
..
AO ... 
...

so in this example i would like to find only two logs not three the first contains the logs which contain FO and FT together in one message, and the second log contains AO
I don't know if it's possible maybe with filebeat or logstash or Elasticsearch between the _id of documents maybe.
I hope you understand me.

Any help would be sincerely appreciate!
Thanks!

2

I haven't tested directly in FB, maybe this

^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\s{3}-\srequest\sin\sformatHexDump\sis\s\S{2}.*(?:FO|FT)?

3

Thank you very much, that's exactly what I want @Rios just if you can tell me what exactly is the role or your meaning of ?: in the regular expression and thank you!

4

( ?: non-capture groups) - result without group
(FO|FT) ?- multi line which includes FO or FT, optionally
Check here

or

1 Like
5

Thanks @Rios but I'm sorry it still didn't work, it was my fault, I was testing with a wrong message :confused:

6

What is result? FO and FT as separate lines or FO and FT lines are missing?
Can you try will this extract only FO and FT messages?
^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\s{3}-\srequest\sin\sformatHexDump\sis\s\S{2}.*(FO|FT)

7

here my example message :

2022-01-13 17:16:05   - request in formatHexDump is => 
000: 12 15 46 77 77 55 | ak55s2 ....
000: 12 15 46 77 77 55 | ak55s2 ....
..
FO ... 
2022-01-13 17:16:10   - request in formatHexDump is => 
000: 44 66 99 11  77 55 | h44a5000 ....
000: 45 66 33 22 00 11  | adandakj ....
..
FT ....
..
2022-01-13 17:17:44   - request in formatHexDump is => 
000: 44 33 22 00 11 55 | jjdll1122 ....
000: 12 15 46 77 77 55 | h555666 ....
..
AO ... 
...
2022-01-13 17:18:12   - request in formatHexDump is => 
000: 44 44 66 99 00 11 55 55 | h44a5000 ....
000: 45 66 33 22 00 11  | adandakj ....
..
FT ....
...

It always gives me 4 documents with this ^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\s{3 }-\srequest\sin\sformatHexDump\sis\s\S{2}.*(?:FO|FT)? and in this case I only want 2 documents, the first containing FO+FT and the second containing AO+ FT.
and I tried this expression ^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\s{3} -\srequest\sin\sformatHexDump\sis\s\S{2}.*(FO|FT) it gives me a document which contains the hole message :confused:

8

If you want like that, try to use

multiline.type: count
multiline.lines_count: 11

Link

9

I can't fix the amount of line I need that's why. it's not 11

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.