# Grok Pattern not matching in Logstash, but matches in Grok Debuggger

**URL:** <https://discuss.elastic.co/t/grok-pattern-not-matching-in-logstash-but-matches-in-grok-debuggger/194985>\
**Category:** Logstash\
**Created:** [August 13, 2019, 8:47am UTC](https://discuss.elastic.co/t/grok-pattern-not-matching-in-logstash-but-matches-in-grok-debuggger/194985 "2019-08-13T08:47:01Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![man0l](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/man0l/32/52176_2.png) [@man0l](https://discuss.elastic.co/u/man0l)\
**Post date:** [August 13, 2019, 8:47am UTC](https://discuss.elastic.co/t/grok-pattern-not-matching-in-logstash-but-matches-in-grok-debuggger/194985/1 "2019-08-13T08:47:01Z")

</div>

Hi,  
I have this grok pattern:

> ```
> (?<timestamp>%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME})%{SPACE}%{LOGLEVEL:level}%{SPACE}%{NUMBER:pid} --- \[%{SPACE}(?<thread>[A-Za-z0-9-]+)\] [A-Za-z0-9.]*\.(?<class>[A-Za-z0-9#_]+)\s*:%{SPACE}(\{%{GREEDYDATA:fields.CorrelationId}\})?:(\{%{GREEDYDATA:fields.proTokenBalance}\}%{SPACE})?\s%{GREEDYDATA:logmessage}
> 
> ```

I am matching this log line:

> 2019-08-13 11:25:39,454 DEBUG 9772 --- [scheduling-1] c.p.r.s.TokenService : :{10248780001247} The PRO Token balance is 102487.80001247

When I start the ELK stack, it gives me \_grokparsefailure, but it it being parsed in the grok debugger tool.

Would someone help me with this issue?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 13, 2019, 1:17pm UTC](https://discuss.elastic.co/t/grok-pattern-not-matching-in-logstash-but-matches-in-grok-debuggger/194985/2 "2019-08-13T13:17:06Z")

</div>

With that pattern and that message I get

```
"fields.proTokenBalance" => "10248780001247",
            "@timestamp" => 2019-08-13T13:14:52.224Z,
                 "class" => "TokenService",
            "logmessage" => "The PRO Token balance is 102487.80001247",
                 "level" => "DEBUG",
                   "pid" => "9772",
                "thread" => "scheduling-1"

```

Are you sure there are single spaces around the ---?

---

<div class="post-metadata">

**Author:** ![man0l](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/man0l/32/52176_2.png) [@man0l](https://discuss.elastic.co/u/man0l)\
**Post date:** [August 14, 2019, 9:14am UTC](https://discuss.elastic.co/t/grok-pattern-not-matching-in-logstash-but-matches-in-grok-debuggger/194985/3 "2019-08-14T09:14:15Z")

</div>

Hi, thank you for your reply.

Yes I am sure about the single spaces around the ---

The logging was working fine until I added the extra field proTokenBalance.

Here's my output in the grok debugger console:

> {  
> "level": "INFO",  
> "logmessage": "The PRO Token balance is 271.99996582",  
> "pid": "5684",  
> "thread": "scheduling-1",  
> "fields": {  
> "proTokenBalance": "271.99996582"  
> },  
> "class": "AccountBalanceCheckService",  
> "timestamp": "2019-08-14 11:25:11,853"  
> }

I have a test config with this logstash config:

input {  
stdin {}  
}  
output { stdout { codec =\> rubydebug } }  
filter {  
grok {  
match =\> {  
"message" =\> "(?%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME})%{SPACE}%{LOGLEVEL:level}%{SPACE}%{NUMBER:pid} --- [%{SPACE}(?[A-Za-z0-9-]+)] [A-Za-z0-9.]_.(?[A-Za-z0-9#\_]+)\s_:%{SPACE}({%{GREEDYDATA:fields.CorrelationId}})?:({%{GREEDYDATA:fields.proTokenBalance}}%{SPACE})?\s%{GREEDYDATA:logmessage}"  
}  
}  
}

This is how I execute the command:

> echo '2019-08-13 11:25:39,454 DEBUG 9772 --- [scheduling-1] c.p.r.s.TokenService : :{10248780001247} The PRO Token balance is 102487.80001247' | C:\logstash\bin\logstash.bat -f .\test.config

it outputs me the correct result as well:

> {  
> "logmessage" =\> "The PRO Token balance is 102487.80001247\r",  
> "message" =\> "2019-08-13 11:25:39,454 DEBUG 9772 --- [scheduling-1] c.p.r.s.TokenService : :{10248780001247} The PRO Token balance is 102487.80001247\r",  
> "thread" =\> "scheduling-1",  
> "pid" =\> "9772",  
> "timestamp" =\> "2019-08-13 11:25:39,454",  
> "class" =\> "TokenService",  
> "@timestamp" =\> 2019-08-14T08:57:06.094Z,  
> "@version" =\> "1",  
> "host" =\> "Manol",  
> "level" =\> "DEBUG",  
> "fields.proTokenBalance" =\> "10248780001247"  
> }

I forwarded the request to the elasticsearch also as well and in Kibana I saw the field.

But when I start reading the log through the file, I use this configuration:

> input {
> 
> ```
> file {
> path => 'C:\projects\Propy.Ethereum.Rest\logs\test\log.log'
> start_position => "beginning"
> }
> 
> ```
> 
> }  
> output {  
> stdout { codec =\> rubydebug }  
> elasticsearch {  
> hosts =\> "localhost:9200"  
> user =\> "elastic"  
> password =\> "changeme"
> 
> ```
> }
> 
> ```
> 
> }  
> filter {  
> grok {  
> match =\> {  
> "message" =\> "(?%{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME})%{SPACE}%{LOGLEVEL:level}%{SPACE}%{NUMBER:pid} --- [%{SPACE}(?[A-Za-z0-9-]+)] [A-Za-z0-9.]_.(?[A-Za-z0-9#\_]+)\s_:%{SPACE}({%{GREEDYDATA:fields.CorrelationId}})?:({%{GREEDYDATA:fields.proTokenBalance}}%{SPACE})?\s%{GREEDYDATA:logmessage}"  
> }  
> }  
> }

And I get this result in Kibana:

> {  
> "\_index": "logstash-2019.08.13-000001",  
> "\_type": "\_doc",  
> "\_id": "j5Fij2wBxXzNBooJFSk2",  
> "\_version": 1,  
> "\_score": null,  
> "\_source": {  
> "message": "2019-08-14 12:07:44,055 INFO 14732 --- [scheduling-1] c.p.r.s.AccountBalanceCheckService : :{271.99996582} The PRO Token balance is 271.99996582\r",  
> "@timestamp": "2019-08-14T09:07:45.471Z",  
> "path": "/tmp/log.log",  
> "host": "3330d93c90be",  
> "@version": "1",  
> "tags": [  
> "\_grokparsefailure"  
> ]  
> },  
> "fields": {  
> "@timestamp": [  
> "2019-08-14T09:07:45.471Z"  
> ]  
> },  
> "sort": [  
> 1565773665471  
> ]  
> }

As you can see, it gives me \_grokparsefailure on this line.

The only difference between the file and the stdin is that the log file has multiline log and the logstash configuration doesn't log multiline. I am not sure is this somehow related.

What is your opinion?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 14, 2019, 12:57pm UTC](https://discuss.elastic.co/t/grok-pattern-not-matching-in-logstash-but-matches-in-grok-debuggger/194985/4 "2019-08-14T12:57:37Z")

</div>

> [@man0l](#):
>
> As you can see, it gives me \_grokparsefailure on this line.

I get an unmatched parenthesis error with that grok filter. Please edit your post, select the configuration and click on \</\>. Then verify that the configuration displayed in the preview panel actually matches your configuration.

---

<div class="post-metadata">

**Author:** ![man0l](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/man0l/32/52176_2.png) [@man0l](https://discuss.elastic.co/u/man0l)\
**Post date:** [August 14, 2019, 1:51pm UTC](https://discuss.elastic.co/t/grok-pattern-not-matching-in-logstash-but-matches-in-grok-debuggger/194985/5 "2019-08-14T13:51:51Z")

</div>

This is a pastebin of the configuration:

[https://pastebin.com/KVwKgMDQ](https://pastebin.com/KVwKgMDQ)

This is a past bin of a part of my log file:  
[https://pastebin.com/nuE23JFJ](https://pastebin.com/nuE23JFJ)

Thank you for your help,  
Regards,  
Manol.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 14, 2019, 2:00pm UTC](https://discuss.elastic.co/t/grok-pattern-not-matching-in-logstash-but-matches-in-grok-debuggger/194985/6 "2019-08-14T14:00:40Z")

</div>

In 7.3.0 that gets me

```
{
             "timestamp" => "2019-08-14 16:28:23,667",
                 "class" => "AccountBalanceCheckService",
                 "level" => "INFO",
"fields.proTokenBalance" => "271.99996582",
            "logmessage" => "The PRO Token balance is 271.99996582",
               "message" => "2019-08-14 16:28:23,667 INFO 1848 --- [scheduling-1] c.p.r.s.AccountBalanceCheckService : :{271.99996582} The PRO Token balance is 271.99996582",
                   "pid" => "1848",
                "thread" => "scheduling-1"
}

```

I am running on UNIX, so I do not have that \r at the end of line, but I would expect GREEDYDATA to consume that.

---

<div class="post-metadata">

**Author:** ![rorixrebel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rorixrebel/32/41778_2.png) [@rorixrebel](https://discuss.elastic.co/u/rorixrebel)\
**Post date:** [August 14, 2019, 2:16pm UTC](https://discuss.elastic.co/t/grok-pattern-not-matching-in-logstash-but-matches-in-grok-debuggger/194985/7 "2019-08-14T14:16:30Z")

</div>

Not sure if it matters but in your pipeline why do you have the order as:

1. Input
2. Output
3. Filter

As for your patterns, they seem to match properly on the debugger.  
[https://grokdebug.herokuapp.com/](https://grokdebug.herokuapp.com/)

---

<div class="post-metadata">

**Author:** ![man0l](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/man0l/32/52176_2.png) [@man0l](https://discuss.elastic.co/u/man0l)\
**Post date:** [August 14, 2019, 2:51pm UTC](https://discuss.elastic.co/t/grok-pattern-not-matching-in-logstash-but-matches-in-grok-debuggger/194985/8 "2019-08-14T14:51:31Z")

</div>

Well, this is confusing for me 🙂

I have tried everything which I've remembered. It seems like the configuration is okay, but somehow it doesn't work.

We have java stack traces in the log, so a part of the log becomes multi line, this could be the issue (and the logstash configuration doesn't handle multi line log at this moment)

I will play a bit more with my local ELK installation.  
Regards,  
Manol.

---

<div class="post-metadata">

**Author:** ![man0l](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/man0l/32/52176_2.png) [@man0l](https://discuss.elastic.co/u/man0l)\
**Post date:** [August 14, 2019, 2:52pm UTC](https://discuss.elastic.co/t/grok-pattern-not-matching-in-logstash-but-matches-in-grok-debuggger/194985/9 "2019-08-14T14:52:17Z")

</div>

Actually it doesn't, when I start logstash it founds properly the file path, it initialize the connection to Elasticsearch also as well.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 11, 2019, 2:52pm UTC](https://discuss.elastic.co/t/grok-pattern-not-matching-in-logstash-but-matches-in-grok-debuggger/194985/10 "2019-09-11T14:52:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.