# Grok error while loading

**URL:** <https://discuss.elastic.co/t/grok-error-while-loading/181515>\
**Category:** Logstash\
**Tags:** elastic-stack-security\
**Created:** [May 17, 2019, 6:08am UTC](https://discuss.elastic.co/t/grok-error-while-loading/181515 "2019-05-17T06:08:14Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![pathri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pathri/32/54484_2.png) [@pathri](https://discuss.elastic.co/u/pathri)\
**Post date:** [May 17, 2019, 6:08am UTC](https://discuss.elastic.co/t/grok-error-while-loading/181515/1 "2019-05-17T06:08:14Z")

</div>

i am running a logstash for the log file with below a sample entry

Sample entry :  
72.14.199.105 - - [30/Apr/2019:06:21:26 +0000] "GET /catalog/view/javascript/font-awesome/css/font-awesome.min.css HTTP/1.1" 200 4748 "[https://www.orderhealth.in/drmorepen/drmorepenbg03](https://www.orderhealth.in/drmorepen/drmorepenbg03)" "Mozilla/5.0 (iPhone; CPU iPhone OS 9\_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)"

using grok as

grok {  
match =\> ["%{IP:Clientip} %{USER:user} %{USER:auth} [%{HTTPDATE:apache\_timestamp}] "%{WORD:method} /%{NOTSPACE:request\_page} HTTP/%{NUMBER:http\_version}" %{WORD:response} %{NUMBER:bytes} "%{URI:page}" "(?[\w/\d.\s(;)(,-]+) +(?[+\w:/.)]+)" ]  
}

on running logstash  
below error is coming.

[ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, input, filter, output at line 10, column 1 (byte 230) after ", :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in`compile\_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in`map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in`initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:23:in`initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline\_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in`block in converge\_state'"]}

"grok" is at line 10 in my config file,  
please help me in understanding that where i am going wrong here

Thanks in advance

---

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [May 17, 2019, 7:32am UTC](https://discuss.elastic.co/t/grok-error-while-loading/181515/2 "2019-05-17T07:32:52Z")

</div>

Hi,

one issue I see is that you do not tell the `grok` filter which _field_ to match the pattern to. From the [documentation](https://www.elastic.co/guide/en/logstash/7.0/plugins-filters-grok.html#plugins-filters-grok-match)

```
   filter {
     grok {
       match => {
         "message" => "Duration: %{NUMBER:duration}"
       }
     }
   }

```

The issue you are having is something else though

> [@pathri](#):
>
> :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, input, filter, output at line 10, column 1

Could you post your Logstash config. Sounds like the whole syntax that Logstash is expecting is missing. Do you use more than one config file? Line 10 is line 10 of the concatenated config file (if you use several files.

---

<div class="post-metadata">

**Author:** ![pathri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pathri/32/54484_2.png) [@pathri](https://discuss.elastic.co/u/pathri)\
**Post date:** [May 17, 2019, 7:37am UTC](https://discuss.elastic.co/t/grok-error-while-loading/181515/3 "2019-05-17T07:37:41Z")

</div>

here is the conf file

# Sample Logstash configuration for creating a simple

# Beats -\> Logstash -\> Elasticsearch pipeline.

input {  
file {  
path =\> "/usr/share/logstash/logs-data/orderhealth.in-ssl\_log-Apr-2019"  
start\_position =\> "beginning"  
}  
}

grok {  
match =\> ["%{IP:Clientip} %{USER:user} %{USER:auth} [%{HTTPDATE:apache\_timestamp}] "%{WORD:method} /%{NOTSPACE:request\_page} HTTP/%{NUMBER:http\_version}" %{WORD:response} %{NUMBER:bytes} "%{URI:page}" "(?[\w/\d.\s(;)(,-]+) +(?[+\w:/.)]+)" ]  
}

output {  
elasticsearch {  
hosts =\> ["localhost"]  
index =\> "logs-ssl"  
}  
}

and yes i am using 4 config files but all are for different indexes.

---

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [May 17, 2019, 7:45am UTC](https://discuss.elastic.co/t/grok-error-while-loading/181515/4 "2019-05-17T07:45:27Z")

</div>

As far as I know, all those files will be combined into one on Logstash startup.

Except for the issue with how the `grok`pattern is defined, that particular config file looks ok.

I would test each config file individually with `./logstash -f first_config_file.conf` to see if they are all ok as far as syntx goes. Once they are ok individually you can try to start Logstash with all of them.

---

<div class="post-metadata">

**Author:** ![pathri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pathri/32/54484_2.png) [@pathri](https://discuss.elastic.co/u/pathri)\
**Post date:** [May 17, 2019, 7:48am UTC](https://discuss.elastic.co/t/grok-error-while-loading/181515/5 "2019-05-17T07:48:36Z")

</div>

in that case it should show error when i am running logstash with other config files but it is working fine with them.

> [@A\_B](#):
>
> one issue I see is that you do not tell the `grok` filter which _field_ to matc

let me check this point although i tried it earlier with this but no luck

---

<div class="post-metadata">

**Author:** ![pathri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pathri/32/54484_2.png) [@pathri](https://discuss.elastic.co/u/pathri)\
**Post date:** [May 17, 2019, 7:54am UTC](https://discuss.elastic.co/t/grok-error-while-loading/181515/6 "2019-05-17T07:54:31Z")

</div>

so now my config file is as below

input {  
file {  
path =\> "/usr/share/logstash/logs-data/orderhealth.in-ssl\_log-Apr-2019"  
start\_position =\> "beginning"  
}  
}

filter {  
grok {  
match =\> { "message" =\> "%{IP:Clientip} %{USER:user} %{USER:auth} [%{HTTPDATE:apache\_timestamp}] "%{WORD:method} /%{NOTSPACE:request\_page} HTTP/%{NUMBER:http\_version}" %{WORD:response} %{NUMBER:bytes} "%{URI:page}" "(?[\w/\d.\s(;)(,-]+) +(?[+\w:/.)]+)"  
}  
}  
}

output {  
elasticsearch {  
hosts =\> ["localhost"]  
index =\> "logs-ssl"  
}  
}

and now the error is

Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, =\> at line 11, column 4 (byte 429) after filter {\n\tgrok {\n \t\tmatch =\> { "message" =\> "%{IP:Clientip} %{USER:user} %{USER:auth} \[%{HTTPDATE:apache\_timestamp}\] \"%{WORD:method} /%{NOTSPACE:request\_page} HTTP/%{NUMBER:http\_version}\" %{WORD:response} %{NUMBER:bytes} \"%{URI:page}" "(?[\w/\d.\s(;)(,-]+) +(?[+\w:/.)]+)" \n\t\t\t", :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in`compile\_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in`map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in`initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:23:in`initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline\_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in`block in converge\_state'"]}

---

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [May 17, 2019, 8:59am UTC](https://discuss.elastic.co/t/grok-error-while-loading/181515/7 "2019-05-17T08:59:11Z")

</div>

Did a quick test with your grok pattern and looks like that is the root of the problem...

The whole pattern is quoted with double quotes, so any double quotes inside the grok pattern have to be escaped somehow.

Did you test your grok pattern in any way?

This seems to work for me

```
%{IP:Clientip} %{USER:user} %{USER:auth} \[%{HTTPDATE:apache_timestamp}\] \"%{WORD:method} %{UNIXPATH:request_page} HTTP/%{NUMBER:http_version}\" %{WORD:response} %{NUMBER:bytes} \"%{URI:page}\" \"%{GREEDYDATA:field_name}\"$
```

---

<div class="post-metadata">

**Author:** ![pathri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pathri/32/54484_2.png) [@pathri](https://discuss.elastic.co/u/pathri)\
**Post date:** [May 17, 2019, 9:01am UTC](https://discuss.elastic.co/t/grok-error-while-loading/181515/8 "2019-05-17T09:01:55Z")

</div>

> [@A\_B](#):
>
> Did you test your grok pattern in any way?

yes used debugger only  
let me check again 🙂

---

<div class="post-metadata">

**Author:** ![pastechecker](https://avatars.discourse-cdn.com/v4/letter/p/0ea827/32.png) [@pastechecker](https://discuss.elastic.co/u/pastechecker)\
**Post date:** [May 17, 2019, 9:10am UTC](https://discuss.elastic.co/t/grok-error-while-loading/181515/9 "2019-05-17T09:10:40Z")

</div>

Add the ^ and $ to your grok.  
You are making it more expensive.

Use also:  
break\_on\_match =\> true

---

<div class="post-metadata">

**Author:** ![pathri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pathri/32/54484_2.png) [@pathri](https://discuss.elastic.co/u/pathri)\
**Post date:** [May 17, 2019, 9:31am UTC](https://discuss.elastic.co/t/grok-error-while-loading/181515/10 "2019-05-17T09:31:37Z")

</div>

meanwhile .please help me with 1 more thing

i want to break the below part in 2 fields like

"Mozilla/5.0 (iPhone; CPU iPhone OS 9\_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)"

want [http://www.google.com/mobile/adsbot.html](http://www.google.com/mobile/adsbot.html) in 1 and rest in another  
actually i was trying to achieve this only with my grok pattern (which i was already doubtful that if i am doing it correct or not 😃 )

---

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [May 17, 2019, 10:32am UTC](https://discuss.elastic.co/t/grok-error-while-loading/181515/11 "2019-05-17T10:32:35Z")

</div>

Just some test results (not included the additional which above)

I tested with this config

```
# cat ls-test1.conf
input { stdin { codec => "json" } }

filter {

  grok {
    match => {
      "message" => "%{IP:Clientip} %{USER:user} %{USER:auth} \[%{HTTPDATE:apache_timestamp}\] \"%{WORD:method} %{UNIXPATH:request_page} HTTP/%{NUMBER:http_version}\" %{WORD:response} %{NUMBER:bytes} \"%{URI:page}\" \"%{GREEDYDATA:field_name}\"$"
    }
  }
}
output {
  stdout { codec => rubydebug }
}

```

And got this result when putting your sample data through Logstash

```
...
[INFO] 2019-05-17 09:01:22.670 [Api Webserver] agent - Successfully started Logstash API 
endpoint {:port=>9601}
{"field1":"hello","message":"72.14.199.105 - - [30/Apr/2019:06:21:26 +0000] \"GET /catalog/view/javascript/font-awesome/css/font-awesome.min.css HTTP/1.1\" 200 4748 \"https://www.orderhealth.in/drmorepen/drmorepenbg03\" \"Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)\""}
{
              "field1" => "hello",
            "response" => "200",
    "apache_timestamp" => "30/Apr/2019:06:21:26 +0000",
              "method" => "GET",
                "user" => "-",
            "Clientip" => "72.14.199.105",
        "request_page" => "/catalog/view/javascript/font-awesome/css/font-awesome.min.css",
             "message" => "72.14.199.105 - - [30/Apr/2019:06:21:26 +0000] \"GET /catalog/view/javascript/font-awesome/css/font-awesome.min.css HTTP/1.1\" 200 4748 \"https://www.orderhealth.in/drmorepen/drmorepenbg03\" \"Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)\"",
                "host" => "foo.bar.net",
            "@version" => "1",
                "page" => "https://www.orderhealth.in/drmorepen/drmorepenbg03",
          "@timestamp" => 2019-05-17T09:01:35.610Z,
                "auth" => "-",
          "field_name" => "Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)",
               "bytes" => "4748",
        "http_version" => "1.1"
}
```

---

<div class="post-metadata">

**Author:** ![pathri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pathri/32/54484_2.png) [@pathri](https://discuss.elastic.co/u/pathri)\
**Post date:** [May 17, 2019, 6:26pm UTC](https://discuss.elastic.co/t/grok-error-while-loading/181515/12 "2019-05-17T18:26:54Z")

</div>

> [@pastechecker](#):
>
> Add the ^ and $ to your grok.

what are these signs for in grok?  
without "$" what are the disadvantages?

---

<div class="post-metadata">

**Author:** ![pathri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pathri/32/54484_2.png) [@pathri](https://discuss.elastic.co/u/pathri)\
**Post date:** [May 17, 2019, 6:35pm UTC](https://discuss.elastic.co/t/grok-error-while-loading/181515/13 "2019-05-17T18:35:28Z")

</div>

> [@pathri](#):
>
> want [Google के क्रॉलर (उपयोगकर्ता एजेंट) की खास जानकारी | Google Search Central &nbsp;|&nbsp; दस्तावेज़ &nbsp;|&nbsp; Google for Developers](http://www.google.com/mobile/adsbot.html) in 1 and rest in another

@A_B & @pastechecker : i have tried with below reg-ex it is giving required output, is it corerct performance wise?

%{IP:Clientip} %{USER:user} %{USER:auth} [%{HTTPDATE:apache\_timestamp}] "%{WORD:method} %{UNIXPATH:request\_page} HTTP/%{NUMBER:http\_version}" %{WORD:response} %{NUMBER:bytes} "%{URI:page}" "%{GREEDYDATA:browserdetails}+%{URI:referby}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 17, 2019, 6:51pm UTC](https://discuss.elastic.co/t/grok-error-while-loading/181515/14 "2019-05-17T18:51:04Z")

</div>

You should definitely read [Do you grok Grok?](https://www.elastic.co/blog/do-you-grok-grok) on the official elastic blog.

^ anchors your pattern to the beginning of line. Suppose we have a line like

```
2016-09-19T18:19:00 DEBUG this is an example log message

```

and we try to match it with

```
%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log_level} %{GREEDYDATA:message}

```

That is OK, it will match. But suppose we have a different line such as

```
Hello, world!

```

That is not going to match. But to test that it has to see if the string "Hello, world!" matches a TIMESTAMP\_ISO8601, then test if the string "ellow, world!" matches, then test if the string "llow, world!" matches and so on. If we had started with the pattern

```
^%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log_level} %{GREEDYDATA:message}

```

then it would only have to "Hello, world!" and not any of the sub-strings. Lots of folks assume grok patterns are implicitly anchored by they are not. Our original pattern would match both these lines

```
Hello, I found this in a log file: 2016-09-19T18:19:00 DEBUG this is an example log message

2016-09-19T18:19:00 DEBUG this is an example log message

```

$ works the same way for end of line. Rarely has anywhere near as much performance impact as ^.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 14, 2019, 6:51pm UTC](https://discuss.elastic.co/t/grok-error-while-loading/181515/15 "2019-06-14T18:51:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.