# Grok for this log format

**URL:** <https://discuss.elastic.co/t/grok-for-this-log-format/373247>\
**Category:** Elastic Agent\
**Created:** [January 15, 2025, 5:56pm UTC](https://discuss.elastic.co/t/grok-for-this-log-format/373247 "2025-01-15T17:56:54Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![nkknkk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nkknkk/32/140707_2.png) [@nkknkk](https://discuss.elastic.co/u/nkknkk)\
**Post date:** [January 15, 2025, 5:56pm UTC](https://discuss.elastic.co/t/grok-for-this-log-format/373247/1 "2025-01-15T17:56:54Z")

</div>

Hi, I am trying to parse log file. I am using elastic agent.

# below is the log format

[\<1 15, 2025 11:12:12 AM\>: This is message1]  
[\<1 15, 2025 11:12:12 AM\>:

This is message2

# ]

This is the grok i am using and it only parses the message1.  
it wouldn't parse the message2

grok - [\<%{NUMBER:month} %{NUMBER:day}, %{NUMBER:year} %{TIME:time} [AP]M\>: %{GREEDYDATA:message}.

can someone help me fix this?

---

<div class="post-metadata">

**Author:** ![Wolfram\_Haussig](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wolfram_haussig/32/70528_2.png) [@Wolfram\_Haussig](https://discuss.elastic.co/u/Wolfram_Haussig)\
**Post date:** [January 16, 2025, 7:59am UTC](https://discuss.elastic.co/t/grok-for-this-log-format/373247/2 "2025-01-16T07:59:11Z")

</div>

Hello,

You need to make grok multiline-aware using `(?m)`:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/2/2238c34dd528a68bd679629138617d69500b4a82.png)

Best regards  
Wolfram

---

<div class="post-metadata">

**Author:** ![nkknkk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nkknkk/32/140707_2.png) [@nkknkk](https://discuss.elastic.co/u/nkknkk)\
**Post date:** [January 16, 2025, 2:55pm UTC](https://discuss.elastic.co/t/grok-for-this-log-format/373247/3 "2025-01-16T14:55:37Z")

</div>

yes, this works for just one message. but if you put all the messages in the log file. it will consider everything into one document. that is not right.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/1/41a7e3ad158eeea68368611850be5745badc0cea.png)

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [January 16, 2025, 5:10pm UTC](https://discuss.elastic.co/t/grok-for-this-log-format/373247/4 "2025-01-16T17:10:07Z")

</div>

@Wolfram_Haussig

Hi @nkknkk Welcome to the community

You are going to need to use multiline on the input... that is how this works....

> **[Multiline codec plugin | Logstash Reference \[8.17\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html)**

Provide a sample log with 5-10 messages in text and someone will show you...

and Share your conf file... in text not screen shots... screen shots of text are discouraged.

and please format you logs and code with 3 backticks before and after `````

---

<div class="post-metadata">

**Author:** ![nkknkk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nkknkk/32/140707_2.png) [@nkknkk](https://discuss.elastic.co/u/nkknkk)\
**Post date:** [January 16, 2025, 9:24pm UTC](https://discuss.elastic.co/t/grok-for-this-log-format/373247/5 "2025-01-16T21:24:07Z")

</div>

Hi @Wolfram_Haussig

here is my sample log

```auto
[<1 14, 2025 06:06:26 AM>: 

The JDBC database driver version is: 1.2.0.0.0

]
[<1 14, 2025 06:06:26 AM>: Util.getInfo: close statement]
[<1 14, 2025 06:06:26 AM>: 
Change=Y
]
[<1 14, 2025 06:06:26 AM>: Util.getFlag: close statement]
[<1 14, 2025 06:06:26 AM>: 
currentYr=2025
]
[<1 14, 2025 06:06:26 AM>: Util.getYear: close statement]

```

and here is my config

```auto
paths:
          - /opt/logs/*.out
        exclude_files:
          - .gz$
        tags:
          - preserve_original_event
          - forwarded
          - apache_tomcat-catalina
        publisher_pipeline.disable_host: true
        close.on_state_change.inactive: 5m
        fields_under_root: true
        parsers:
          - multiline:
              pattern: ^\[<%{NUMBER:day} %{NUMBER:month}, %{NUMBER:year} %{TIME:time} %{WORD:period}>:
              negate: false
              match: after

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [January 16, 2025, 9:47pm UTC](https://discuss.elastic.co/t/grok-for-this-log-format/373247/6 "2025-01-16T21:47:21Z")

</div>

Wait correction...

you are trying to use GROK for multiline that is not how multiline works...

First, you do multiline on incoming logs in the Agent Input that will put all the lines into a single field `message`

THEN you parse with GROK in an ingest pipeline

I think your multiline should just be

```auto
parsers:
- multiline:
    type: pattern
    pattern: '^\['
    negate: true
    match: after

```

Then in your Ingest Pipeline you will use GROK to parse the `message` field

Get the multiline working first.... then work on your ingest pipeline..

> **[Tutorial: Transform data with custom ingest pipelines | Fleet and Elastic...](https://www.elastic.co/guide/en/fleet/current/data-streams-pipeline-tutorial.html)**

What version are you on?

Exactly which integration are you using ... Custom Logs?

Is this a fleet managed agent to stand alone?

The more precise you are the better we can help...

---

<div class="post-metadata">

**Author:** ![nkknkk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nkknkk/32/140707_2.png) [@nkknkk](https://discuss.elastic.co/u/nkknkk)\
**Post date:** [January 21, 2025, 8:18pm UTC](https://discuss.elastic.co/t/grok-for-this-log-format/373247/7 "2025-01-21T20:18:19Z")

</div>

Thanks you. This works.  
now, my log file has 2 different formats.  
one line that starts with `[<1 21, 2025`  
and the other line that starts with the format `21-Jan-2025`

is it possible to make the agent look for both the formats?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [January 21, 2025, 8:19pm UTC](https://discuss.elastic.co/t/grok-for-this-log-format/373247/8 "2025-01-21T20:19:54Z")

</div>

Unclear if you mean the multiline or the Grok... always best to provide a couple actual samples. but you should be able to create a regex with an or `|` operator... do a little google and you will find it

---

<div class="post-metadata">

**Author:** ![nkknkk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nkknkk/32/140707_2.png) [@nkknkk](https://discuss.elastic.co/u/nkknkk)\
**Post date:** [January 21, 2025, 8:21pm UTC](https://discuss.elastic.co/t/grok-for-this-log-format/373247/9 "2025-01-21T20:21:26Z")

</div>

sorry, i meant for the multiline.  
trying to get all the data from the logs before going to Grok

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [January 21, 2025, 8:38pm UTC](https://discuss.elastic.co/t/grok-for-this-log-format/373247/10 "2025-01-21T20:38:47Z")

</div>

Something like this strict

```auto
^(\[|(0[1-9]|[12][0-9]|3[01])-(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)-\d{4})

```

Or less strict

```auto
^(\[|(0[1-9]|[12][0-9]|3[01])-[a-zA-Z]{3}-\d{4})

```

---

<div class="post-metadata">

**Author:** ![nkknkk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nkknkk/32/140707_2.png) [@nkknkk](https://discuss.elastic.co/u/nkknkk)\
**Post date:** [January 21, 2025, 10:42pm UTC](https://discuss.elastic.co/t/grok-for-this-log-format/373247/11 "2025-01-21T22:42:45Z")

</div>

This is what I have and hopefully it works. I will have to wait for a day and see if this actually is working as expected.

```auto
pattern: '^(\[\<|\d{2}-\w{3}-\d{4})'

```

---

<div class="post-metadata">

**Author:** ![nkknkk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nkknkk/32/140707_2.png) [@nkknkk](https://discuss.elastic.co/u/nkknkk)\
**Post date:** [January 23, 2025, 2:58pm UTC](https://discuss.elastic.co/t/grok-for-this-log-format/373247/12 "2025-01-23T14:58:50Z")

</div>

so, looks like it didn't work.  
I used the about pattern and i can see only the data that starts with`23-Jan-2025`. it ignored the other data that starts with `[<`. what am i missing?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [January 23, 2025, 3:01pm UTC](https://discuss.elastic.co/t/grok-for-this-log-format/373247/13 "2025-01-23T15:01:27Z")

</div>

You added another `\<` don't think it is needed

`pattern: '^(\[\<|\d{2}-\w{3}-\d{4})'`  
`............. ^^`

`pattern: '^(\[|\d{2}-\w{3}-\d{4})'`

---

<div class="post-metadata">

**Author:** ![nkknkk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nkknkk/32/140707_2.png) [@nkknkk](https://discuss.elastic.co/u/nkknkk)\
**Post date:** [January 23, 2025, 3:08pm UTC](https://discuss.elastic.co/t/grok-for-this-log-format/373247/14 "2025-01-23T15:08:11Z")

</div>

couple of days back, I used `pattern: '^\[\<' `. This got me all the lines that begin with `[<`  
now this new `pattern: '^(\[\<|\d{2}-\w{3}-\d{4})'` ignores the first pattern `[<` and only gets the lines starting with `23-Jan-2025`.  
I don't understand why i should remove the `\<` from the pattern? can you please explain?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [January 23, 2025, 3:51pm UTC](https://discuss.elastic.co/t/grok-for-this-log-format/373247/15 "2025-01-23T15:51:05Z")

</div>

No I can't I am not a regex expert perhaps you should try with a regex debugger like

> **[regex101: build, test, and debug regex](https://regex101.com/)**
>
> Regular expression tester with syntax highlighting, explanation, cheat sheet for PHP/PCRE, Python, GO, JavaScript, Java, C#/.NET, Rust.

Take a look

 ![Screenshot 2025-01-23 at 7.49.31 AM](https://us1.discourse-cdn.com/elastic/original/3X/d/a/da3976469eb77c669874d60c0904d60737f89325.png)

all these work

```auto
^(\[|\d{2}-\w{3}-\d{4})
^(\[<|\d{2}-\w{3}-\d{4})
^(\[\<|\d{2}-\w{3}-\d{4})

```

---

<div class="post-metadata">

**Author:** ![nkknkk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nkknkk/32/140707_2.png) [@nkknkk](https://discuss.elastic.co/u/nkknkk)\
**Post date:** [January 23, 2025, 3:58pm UTC](https://discuss.elastic.co/t/grok-for-this-log-format/373247/16 "2025-01-23T15:58:04Z")

</div>

ok, Thank you! I appreciate it! let me try the different scenarios and see if it works

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [January 23, 2025, 3:59pm UTC](https://discuss.elastic.co/t/grok-for-this-log-format/373247/17 "2025-01-23T15:59:02Z")

</div>

There are more examples at

> **[Manage multiline messages | Filebeat Reference \[8.17\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html)**

Agent uses same syntax