Grok for this log format

Elastic Stack Elastic Agent
1

Hi, I am trying to parse log file. I am using elastic agent.

below is the log format

[<1 15, 2025 11:12:12 AM>: This is message1]
[<1 15, 2025 11:12:12 AM>:

This is message2

]

This is the grok i am using and it only parses the message1.
it wouldn't parse the message2

grok - [<%{NUMBER:month} %{NUMBER:day}, %{NUMBER:year} %{TIME:time} [AP]M>: %{GREEDYDATA:message}.

can someone help me fix this?

2

Hello,

You need to make grok multiline-aware using (?m):

image
image757×777 21.2 KB

Best regards
Wolfram

3

yes, this works for just one message. but if you put all the messages in the log file. it will consider everything into one document. that is not right.

image
image1531×818 40.4 KB

4

@Wolfram_Haussig

Hi @nkknkk Welcome to the community

You are going to need to use multiline on the input... that is how this works....

Provide a sample log with 5-10 messages in text and someone will show you...

and Share your conf file... in text not screen shots... screen shots of text are discouraged.

and please format you logs and code with 3 backticks before and after ```

5

Hi @Wolfram_Haussig

here is my sample log

[<1 14, 2025 06:06:26 AM>: 

The JDBC database driver version is: 1.2.0.0.0

]
[<1 14, 2025 06:06:26 AM>: Util.getInfo: close statement]
[<1 14, 2025 06:06:26 AM>: 
Change=Y
]
[<1 14, 2025 06:06:26 AM>: Util.getFlag: close statement]
[<1 14, 2025 06:06:26 AM>: 
currentYr=2025
]
[<1 14, 2025 06:06:26 AM>: Util.getYear: close statement]

and here is my config

paths:
          - /opt/logs/*.out
        exclude_files:
          - .gz$
        tags:
          - preserve_original_event
          - forwarded
          - apache_tomcat-catalina
        publisher_pipeline.disable_host: true
        close.on_state_change.inactive: 5m
        fields_under_root: true
        parsers:
          - multiline:
              pattern: ^\[<%{NUMBER:day} %{NUMBER:month}, %{NUMBER:year} %{TIME:time} %{WORD:period}>:
              negate: false
              match: after
6

Wait correction...

you are trying to use GROK for multiline that is not how multiline works...

First, you do multiline on incoming logs in the Agent Input that will put all the lines into a single field message

THEN you parse with GROK in an ingest pipeline

I think your multiline should just be

parsers:
- multiline:
    type: pattern
    pattern: '^\['
    negate: true
    match: after

Then in your Ingest Pipeline you will use GROK to parse the message field

Get the multiline working first.... then work on your ingest pipeline..

What version are you on?

Exactly which integration are you using ... Custom Logs?

Is this a fleet managed agent to stand alone?

The more precise you are the better we can help...

7

Thanks you. This works.
now, my log file has 2 different formats.
one line that starts with [<1 21, 2025
and the other line that starts with the format 21-Jan-2025

is it possible to make the agent look for both the formats?

8

Unclear if you mean the multiline or the Grok... always best to provide a couple actual samples. but you should be able to create a regex with an or | operator... do a little google and you will find it

9

sorry, i meant for the multiline.
trying to get all the data from the logs before going to Grok

10

Something like this strict

^(\[|(0[1-9]|[12][0-9]|3[01])-(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)-\d{4})

Or less strict

^(\[|(0[1-9]|[12][0-9]|3[01])-[a-zA-Z]{3}-\d{4})
11

This is what I have and hopefully it works. I will have to wait for a day and see if this actually is working as expected.

pattern: '^(\[\<|\d{2}-\w{3}-\d{4})'
12

so, looks like it didn't work.
I used the about pattern and i can see only the data that starts with23-Jan-2025. it ignored the other data that starts with [<. what am i missing?

13

You added another \< don't think it is needed

pattern: '^(\[\<|\d{2}-\w{3}-\d{4})'
............. ^^

pattern: '^(\[|\d{2}-\w{3}-\d{4})'

14

couple of days back, I used pattern: '^\[\<' . This got me all the lines that begin with [<
now this new pattern: '^(\[\<|\d{2}-\w{3}-\d{4})' ignores the first pattern [< and only gets the lines starting with 23-Jan-2025.
I don't understand why i should remove the \< from the pattern? can you please explain?

15

No I can't I am not a regex expert perhaps you should try with a regex debugger like

Take a look

Screenshot 2025-01-23 at 7.49.31 AM
Screenshot 2025-01-23 at 7.49.31 AM3576×1330 458 KB

all these work

^(\[|\d{2}-\w{3}-\d{4})
^(\[<|\d{2}-\w{3}-\d{4})
^(\[\<|\d{2}-\w{3}-\d{4})
16

ok, Thank you! I appreciate it! let me try the different scenarios and see if it works

17

There are more examples at

Agent uses same syntax