# Grok pattern not getting full message

**URL:** <https://discuss.elastic.co/t/grok-pattern-not-getting-full-message/205834>\
**Category:** Elasticsearch\
**Created:** [October 30, 2019, 11:26am UTC](https://discuss.elastic.co/t/grok-pattern-not-getting-full-message/205834 "2019-10-30T11:26:35Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![tomsozolins](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomsozolins/32/57007_2.png) [@tomsozolins](https://discuss.elastic.co/u/tomsozolins)\
**Post date:** [October 30, 2019, 11:26am UTC](https://discuss.elastic.co/t/grok-pattern-not-getting-full-message/205834/1 "2019-10-30T11:26:35Z")

</div>

Hello!

I have have a problem with grok getting full message from log file. Can you please help me?

Log message:

> 2019-10-29 19:27:21.779+02:00 [27] INFO - State has changed: Rule: Recording FPS, Counter: \VideoOS Recording Server Device(test 2 [64542b95-e5e9-4800-8f10-3ee8ba09773d])\Media/sec perc, Before: Critical, now: Normal  
> Changes based on data: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2116,678,

Grok pattern:

> %{TIMESTAMP\_ISO8601:system.syslog.timestamp} [ %{SPACE} %  
> {NUMBER:milestone\_lognum}] %{LOGLEVEL:milestone\_loglevel} %{SPACE} - %{GREEDYDATA:milestone\_message}

Structured data result:

> {  
> "system": {  
> "syslog": {  
> "timestamp": "2019-10-29 19:27:21.779+02:00"  
> }  
> },  
> "milestone\_lognum": "27",  
> "milestone\_loglevel": "INFO",  
> "milestone\_message": "State has changed: Rule: Recording FPS, Counter: \VideoOS Recording Server Device(test 2 [64542b95-e5e9-4800-8f10-3ee8ba09773d])\Media/sec perc, Before: Critical, now: Normal "  
> }

---

<div class="post-metadata">

**Author:** ![B.M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/b.m/32/56771_2.png) [@B.M](https://discuss.elastic.co/u/B.M)\
**Post date:** [October 30, 2019, 11:40am UTC](https://discuss.elastic.co/t/grok-pattern-not-getting-full-message/205834/2 "2019-10-30T11:40:12Z")

</div>

Could it be because you return to ligne after Normal ?

Try running your `Grok pattern` with this log

```
2019-10-29 19:27:21.779+02:00 [27] INFO - State has changed: Rule: Recording FPS, Counter: \VideoOS Recording Server Device(test 2 [64542b95-e5e9-4800-8f10-3ee8ba09773d])\Media/sec perc, Before: Critical, now: Normal Changes based on data: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2116,678,
```

---

<div class="post-metadata">

**Author:** ![tomsozolins](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomsozolins/32/57007_2.png) [@tomsozolins](https://discuss.elastic.co/u/tomsozolins)\
**Post date:** [October 30, 2019, 11:59am UTC](https://discuss.elastic.co/t/grok-pattern-not-getting-full-message/205834/3 "2019-10-30T11:59:06Z")

</div>

Exactly the message after Normal is in new line in the source log file, but i need to parse this as one whole message field.

---

<div class="post-metadata">

**Author:** ![B.M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/b.m/32/56771_2.png) [@B.M](https://discuss.elastic.co/u/B.M)\
**Post date:** [October 30, 2019, 12:19pm UTC](https://discuss.elastic.co/t/grok-pattern-not-getting-full-message/205834/4 "2019-10-30T12:19:52Z")

</div>

Have you tried setting `multiline.pattern`? see more:  
[https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html](https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html)

---

<div class="post-metadata">

**Author:** ![tomsozolins](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomsozolins/32/57007_2.png) [@tomsozolins](https://discuss.elastic.co/u/tomsozolins)\
**Post date:** [November 1, 2019, 7:41am UTC](https://discuss.elastic.co/t/grok-pattern-not-getting-full-message/205834/6 "2019-11-01T07:41:35Z")

</div>

I did grok debugging on this log:

> 2019-10-31 09:13:10.178+02:00 [20] INFO - State has changed: Rule: Recording FPS, \> Counter: \VideoOS Recording Server Device(c7 [3793243f-0055-4250-b98f-b15792b055e0])\Media/sec perc, Before: Warning, now: Normal  
> Changes based on data: 208,0519, 0, 0, 0, 208,4882, 0, 0, 415,8712, 0, 0, 0, 0, 208,2306, 0, 0, 0, 208,4588, 0, 0, 0, 208,1291, 0, 0, 0, 208,338, 0, 0, 0, 207,3737,

With this grok pattern:

> %{TIMESTAMP\_ISO8601:system.syslog.timestamp} \[%{SPACE} %{NUMBER:milestone\_lognum}\] %{LOGLEVEL:milestone\_loglevel} %{SPACE} \- (?\<milestone\_message\>(.|\r|\n)\*)

Structured data:

> {  
> "system": {  
> "syslog": {  
> "timestamp": "2019-10-31 09:13:10.178+02:00"  
> }  
> },  
> "milestone\_lognum": "20",  
> "milestone\_loglevel": "INFO",  
> "milestone\_message": "State has changed: Rule: Recording FPS, Counter: \VideoOS Recording Server Device(c7 [3793243f-0055-4250-b98f-b15792b055e0])\Media/sec perc, Before: Warning, now: Normal \n Changes based on data: 208,0519, 0, 0, 0, 208,4882, 0, 0, 415,8712, 0, 0, 0, 0, 208,2306, 0, 0, 0, 208,4588, 0, 0, 0, 208,1291, 0, 0, 0, 208,338, 0, 0, 0, 207,3737,"  
> }

But in elasticsearch i still get grok parse failure events for new line ☹

---

<div class="post-metadata">

**Author:** ![MiTschMR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mitschmr/32/48254_2.png) [@MiTschMR](https://discuss.elastic.co/u/MiTschMR)\
**Post date:** [November 1, 2019, 9:45am UTC](https://discuss.elastic.co/t/grok-pattern-not-getting-full-message/205834/7 "2019-11-01T09:45:22Z")

</div>

Hi @tomsozolins

I used the [Heroku Grok Debugger](https://grokdebug.herokuapp.com/) to analyze your pattern. The following pattern returned matches:

> %{TIMESTAMP\_ISO8601:system.syslog.timestamp} [%{NUMBER:milestone\_lognum}] %{LOGLEVEL:milestone\_loglevel} - (?\<milestone\_message\>(.|\r|\n)\*)

I think the mistake in your pattern were those `%{SPACE}`.

Hope this helps.

---

<div class="post-metadata">

**Author:** ![tomsozolins](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomsozolins/32/57007_2.png) [@tomsozolins](https://discuss.elastic.co/u/tomsozolins)\
**Post date:** [November 1, 2019, 5:29pm UTC](https://discuss.elastic.co/t/grok-pattern-not-getting-full-message/205834/8 "2019-11-01T17:29:13Z")

</div>

Tried without spaces. It matches in heroku website, but does not match in elasticsearch grok debugger. There is a space at the end of the first line and also at the start of the second line.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 29, 2019, 5:29pm UTC](https://discuss.elastic.co/t/grok-pattern-not-getting-full-message/205834/9 "2019-11-29T17:29:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.