# \_grokparsefailure inspite of working well with grok debugger

**URL:** <https://discuss.elastic.co/t/grokparsefailure-inspite-of-working-well-with-grok-debugger/175669>\
**Category:** Logstash\
**Created:** [April 6, 2019, 8:59am UTC](https://discuss.elastic.co/t/grokparsefailure-inspite-of-working-well-with-grok-debugger/175669 "2019-04-06T08:59:31Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![yogeshwar\_Trehan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yogeshwar_trehan/32/43590_2.png) [@yogeshwar\_Trehan](https://discuss.elastic.co/u/yogeshwar_Trehan)\
**Post date:** [April 6, 2019, 8:59am UTC](https://discuss.elastic.co/t/grokparsefailure-inspite-of-working-well-with-grok-debugger/175669/1 "2019-04-06T08:59:31Z")

</div>

**I am trying to send logs to elasticsearch using logstash. Following is the logstash config file -**

```
input {
        beats {
            port => "5044"
        }
    }
filter {
  grok {
    patterns_dir => ["/etc/logstash/conf.d/patterns"]
    match => {
      "message" => "%{DATA:remoteaddr},%{DATA:remoteuser},%{DATA:datetimeoflog},<<<<%{DATA:httpmethod} %{DATA:requestcontents} %{DATA:httpversion}>>>>,%{DATA:requeststatus},%{DATA:bodysizeinbytes},<<<<%{DATA:httprefferer}>>>>,<<<<%{DATA:useragent}>>>>,%{DATA:requesttime},%{DATA:upstreamconnecttime},%{DATA:upstreamheadertime},%{DATA:upstreamresponsetime}"
      }
   }
   date {
     match => ["datetimeoflog", "dd/MMM/yyyy:HH:mm:ss Z"]
     target => "@timestamp"
     timezone => "Asia/Kolkata"
    }
}
output {
     elasticsearch {
     hosts => ["localhost:9200"]
     index => "mammoth-%{[nameofindex]}"
     }
  }

```

And the pattern I am using are :-

```
DATA (.*?)

```

Corresponding to the log statement of the input file:-

**`172.25.2.1,-,06/Apr/2019:14:24:41 +0530,<<<<POST /api/v1/weburls HTTP/1.1>>>>,200,95,<<<<http://localhost:3000/>>>>,<<<<Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36>>>>,1.223,0.000,1.216,1.216`**

The message from logstash is as follows :-

`DEBUG] 2019-04-06 08:54:43.964 [[main]>worker4] pipeline - output received {"event"=>{"@version"=>"1", "typeoflog"=>"nginx_access", "source"=>"/home/vagrant/logs/nginx/mycode_app_access.log", "host"=>{"name"=>"ubuntu-bionic", "containerized"=>false, "id"=>"aa4c8e0e252d4e4687a49b82a0798def", "architecture"=>"x86_64", "os"=>{"family"=>"debian", "name"=>"Ubuntu", "codename"=>"bionic", "platform"=>"ubuntu", "version"=>"18.04.1 LTS (Bionic Beaver)"}}, "nameofindex"=>"nginx_wip", "log"=>{"file"=>{"path"=>"/home/vagrant/logs/nginx/mycode_app_access.log"}}, "message"=>"172.25.2.1,-,06/Apr/2019:14:24:41 +0530,<<<<POST /api/v1/weburls HTTP/1.1>>>>,200,95,<<<<http://localhost:3000/>>>>,<<<<Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36>>>>,1.223,0.000,1.216,1.216", "input"=>{"type"=>"log"}, "tags"=>["vagrant_nginx_access", "beats_input_codec_plain_applied", "_grokparsefailure"], "offset"=>28999, "@timestamp"=>2019-04-06T08:54:42.748Z, "beat"=>{"name"=>"ubuntu-bionic", "version"=>"6.6.2", "hostname"=>"ubuntu-bionic"}, "prospector"=>{"type"=>"log"}}}`

**Working fine in debugger**

 ![screen](https://us1.discourse-cdn.com/elastic/original/3X/e/2/e275540b2593694c5e5b3ccee52b680f4204c6e7.png)

---

<div class="post-metadata">

**Author:** ![BKG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bkg/32/43591_2.png) [@BKG](https://discuss.elastic.co/u/BKG)\
**Post date:** [April 6, 2019, 9:17am UTC](https://discuss.elastic.co/t/grokparsefailure-inspite-of-working-well-with-grok-debugger/175669/2 "2019-04-06T09:17:03Z")

</div>

Doesn't look like you're using any custom grok patterns. Not sure if that's conflicting somehow, comment it out and see if it works.

---

<div class="post-metadata">

**Author:** ![yogeshwar\_Trehan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yogeshwar_trehan/32/43590_2.png) [@yogeshwar\_Trehan](https://discuss.elastic.co/u/yogeshwar_Trehan)\
**Post date:** [April 6, 2019, 9:49am UTC](https://discuss.elastic.co/t/grokparsefailure-inspite-of-working-well-with-grok-debugger/175669/3 "2019-04-06T09:49:52Z")

</div>

I tried that also but effect. Can you suggest anything else

---

<div class="post-metadata">

**Author:** ![BKG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bkg/32/43591_2.png) [@BKG](https://discuss.elastic.co/u/BKG)\
**Post date:** [April 6, 2019, 11:48am UTC](https://discuss.elastic.co/t/grokparsefailure-inspite-of-working-well-with-grok-debugger/175669/4 "2019-04-06T11:48:55Z")

</div>

I have never seen this log format with `<<<<` and `>>>>`. Even though the parse works in the test, maybe there's something internal messing it up. For instance, I have some patterns like this in my config that contain angle brackets:

`(?<function>\S*)`

Perhaps try simplifying your regex with `<+` and `>+` instead of `<<<<`. And try escaping them too `\<+` and `\>+`.

---

<div class="post-metadata">

**Author:** ![BKG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bkg/32/43591_2.png) [@BKG](https://discuss.elastic.co/u/BKG)\
**Post date:** [April 6, 2019, 11:54am UTC](https://discuss.elastic.co/t/grokparsefailure-inspite-of-working-well-with-grok-debugger/175669/5 "2019-04-06T11:54:51Z")

</div>

> [@yogeshwar\_Trehan](#):
>
> Can you suggest anything else

Ah - try this pattern with a $ on the end:

`%{DATA:remoteaddr},%{DATA:remoteuser},%{DATA:datetimeoflog},<<<<%{DATA:httpmethod} %{DATA:requestcontents} %{DATA:httpversion}>>>>,%{DATA:requeststatus},%{DATA:bodysizeinbytes},<<<<%{DATA:httprefferer}>>>>,<<<<%{DATA:useragent}>>>>,%{DATA:requesttime},%{DATA:upstreamconnecttime},%{DATA:upstreamheadertime},%{DATA:upstreamresponsetime}$`

---

<div class="post-metadata">

**Author:** ![yogeshwar\_Trehan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yogeshwar_trehan/32/43590_2.png) [@yogeshwar\_Trehan](https://discuss.elastic.co/u/yogeshwar_Trehan)\
**Post date:** [April 6, 2019, 12:53pm UTC](https://discuss.elastic.co/t/grokparsefailure-inspite-of-working-well-with-grok-debugger/175669/6 "2019-04-06T12:53:15Z")

</div>

None of the above helped

---

<div class="post-metadata">

**Author:** ![BKG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bkg/32/43591_2.png) [@BKG](https://discuss.elastic.co/u/BKG)\
**Post date:** [April 6, 2019, 12:58pm UTC](https://discuss.elastic.co/t/grokparsefailure-inspite-of-working-well-with-grok-debugger/175669/7 "2019-04-06T12:58:52Z")

</div>

> [@yogeshwar\_Trehan](#):
>
> 172.25.2.1,-,06/Apr/2019:14:24:41 +0530,\<\<\<\<POST /api/v1/weburls HTTP/1.1\>\>\>\>,200,95,\<\<\<\<[http://localhost:3000/&gt;&gt;&gt;&gt;,&lt;&lt;&lt;&lt;Mozilla/5.0](http://localhost:3000/&gt;&gt;&gt;&gt;,&lt;&lt;&lt;&lt;Mozilla/5.0) (X11; Linux x86\_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36\>\>\>\>,1.223,0.000,1.216,1.216

Well, the problem is, the last field, **upstreamresponsetime** isn't getting matched. You need to change your grok to ensure it gets matched.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 6, 2019, 1:06pm UTC](https://discuss.elastic.co/t/grokparsefailure-inspite-of-working-well-with-grok-debugger/175669/8 "2019-04-06T13:06:27Z")

</div>

```
input { generator { count => 1 message => '172.25.2.1,-,06/Apr/2019:14:24:41 +0530,<<<<POST /api/v1/weburls HTTP/1.1>>>>,200,95,<<<<http://localhost:3000/>>>>,<<<<Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36>>>>,1.223,0.000,1.216,1.216' } }

filter {
    grok { match => { "message" => "%{DATA:remoteaddr},%{DATA:remoteuser},%{DATA:datetimeoflog},<<<<%{DATA:httpmethod} %{DATA:requestcontents} %{DATA:httpversion}>>>>,%{DATA:requeststatus},%{DATA:bodysizeinbytes},<<<<%{DATA:httprefferer}>>>>,<<<<%{DATA:useragent}>>>>,%{DATA:requesttime},%{DATA:upstreamconnecttime},%{DATA:upstreamheadertime},%{DATA:upstreamresponsetime}" } pattern_definitions => { "DATA" => "(.*?)" } }
}

```

works just fine for me...

```
         "remoteaddr" => "172.25.2.1",
      "datetimeoflog" => "06/Apr/2019:14:24:41 +0530",
       "httprefferer" => "http://localhost:3000/",
         "httpmethod" => "POST",

```

etc.

---

<div class="post-metadata">

**Author:** ![BKG](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bkg/32/43591_2.png) [@BKG](https://discuss.elastic.co/u/BKG)\
**Post date:** [April 6, 2019, 1:09pm UTC](https://discuss.elastic.co/t/grokparsefailure-inspite-of-working-well-with-grok-debugger/175669/9 "2019-04-06T13:09:28Z")

</div>

> [@Badger](#):
>
> works just fine for me...

Did it match the last field ok?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 6, 2019, 1:18pm UTC](https://discuss.elastic.co/t/grokparsefailure-inspite-of-working-well-with-grok-debugger/175669/10 "2019-04-06T13:18:38Z")

</div>

> [@BKG](#):
>
> Did it match the last field ok?

No. I do not get upstreamresponsetime unless I add the trailing $ (which I think is really strange), but I get all the other fields and I do not get a \_grokparsefailure.

Personally I would be using custom patterns instead of DATA: (?\<remoteaddr\>[^,]\*) etc. And anchoring the pattern with ^ at the start would make it more efficient.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 4, 2019, 1:22pm UTC](https://discuss.elastic.co/t/grokparsefailure-inspite-of-working-well-with-grok-debugger/175669/11 "2019-05-04T13:22:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.