# Grok pattern query for access logs **URL:** \ **Category:** Elasticsearch\ **Created:** [November 19, 2019, 2:37am UTC](https://discuss.elastic.co/t/grok-pattern-query-for-access-logs/208413 "2019-11-19T02:37:32Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Mehak\_Bhargava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehak_bhargava/32/54750_2.png) [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\ **Post date:** [November 19, 2019, 2:37am UTC](https://discuss.elastic.co/t/grok-pattern-query-for-access-logs/208413/1 "2019-11-19T02:37:32Z")
I am using the below query to extract IP address, timestamp and such from an access log like this below: ```auto 192.168.10.182 - - - 18/Nov/2019:13:42:14 -0800 "GET /xxxx-xxxx-xxxx/xxxxxxxxxx/xxx/xxxx/xxxxxxxxxx HTTP/1.1" 200 4534 GET xxxx-xxxx-xxxx/xxxxxxxxxx/xxx/xxxx/xxxxxxxxxx HTTP/1.1 ``` ```auto PUT _ingest/pipeline/access_log { "description" : "Ingest pipeline for Combined Log Format", "processors" : [ { "grok": { "field": "message", "patterns": ["%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp}\\] \"%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int})"] } }, { "date": { "field": "timestamp", "formats": ["dd/MMM/YYYY:HH:mm:ss Z"] } } ] } ``` Why is the grok pattern giving error at this? ---
**Author:** ![nishant.saini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nishant.saini/32/41489_2.png) [@nishant.saini](https://discuss.elastic.co/u/nishant.saini)\ **Post date:** [November 19, 2019, 2:58am UTC](https://discuss.elastic.co/t/grok-pattern-query-for-access-logs/208413/2 "2019-11-19T02:58:24Z")
This seems to be fine. What is the error that you are getting? ---
**Author:** ![Mehak\_Bhargava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehak_bhargava/32/54750_2.png) [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\ **Post date:** [November 19, 2019, 2:58am UTC](https://discuss.elastic.co/t/grok-pattern-query-for-access-logs/208413/3 "2019-11-19T02:58:57Z")
This is the error ```auto { "docs" : [ { "error" : { "root_cause" : [ { "type" : "exception", "reason" : """java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Provided Grok expressions do not match field value: [192.168.10.182 - - - 18/Nov/2019:13:42:14 -0800 "GET /jdbc-data-server/jdbcdataserver/data/discover/TransactionsDemo HTTP/1.1" 200 4534]""", "header" : { "processor_type" : "grok" } } ], "type" : "exception", "reason" : """java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: Provided Grok expressions do not match field value: [192.168.10.182 - - - 18/Nov/2019:13:42:14 -0800 "GET /jdbc-data-server/jdbcdataserver/data/discover/TransactionsDemo HTTP/1.1" 200 4534]""", "caused_by" : { "type" : "illegal_argument_exception", "reason" : """java.lang.IllegalArgumentException: Provided Grok expressions do not match field value: [192.168.10.182 - - - 18/Nov/2019:13:42:14 -0800 "GET /jdbc-data-server/jdbcdataserver/data/discover/TransactionsDemo HTTP/1.1" 200 4534]""", "caused_by" : { "type" : "illegal_argument_exception", "reason" : """Provided Grok expressions do not match field value: [192.168.10.182 - - - 18/Nov/2019:13:42:14 -0800 "GET /jdbc-data-server/jdbcdataserver/data/discover/TransactionsDemo HTTP/1.1" 200 4534]""" } }, "header" : { "processor_type" : "grok" } } } ] } ``` ---
**Author:** ![nishant.saini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nishant.saini/32/41489_2.png) [@nishant.saini](https://discuss.elastic.co/u/nishant.saini)\ **Post date:** [November 19, 2019, 3:05am UTC](https://discuss.elastic.co/t/grok-pattern-query-for-access-logs/208413/4 "2019-11-19T03:05:12Z")
You need to re look your grok pattern. It in not matching the fields value as clearly mentioned in the error. The log pattern has an extra `-`. So there should be handling for that. The correct grok pattern would be : ``` %{IPORHOST:clientip} %{USER:ident} %{USER:auth} %{USER:extra} %{HTTPDATE:timestamp} \"%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) ``` ---
**Author:** ![Mehak\_Bhargava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehak_bhargava/32/54750_2.png) [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\ **Post date:** [November 19, 2019, 5:55pm UTC](https://discuss.elastic.co/t/grok-pattern-query-for-access-logs/208413/5 "2019-11-19T17:55:59Z")
Hi @nishant.saini, Thanks for the feedback. It helped! Until the %{HTTPDATE:timestamp} the query runs without exception but below still gives error for the text after GET ```auto \"%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int} ``` > [@Mehak\_Bhargava](#): > > "GET /xxxx-xxxx-xxxx/xxxxxxxxxx/xxx/xxxx/xxxxxxxxxx HTTP/1.1" 200 4534 GET xxxx-xxxx-xxxx/xxxxxxxxxx/xxx/xxxx/xxxxxxxxxx HTTP/1.1 What is the issue here? And is there a place for further reference and doc? ---
**Author:** ![nishant.saini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nishant.saini/32/41489_2.png) [@nishant.saini](https://discuss.elastic.co/u/nishant.saini)\ **Post date:** [November 20, 2019, 4:16am UTC](https://discuss.elastic.co/t/grok-pattern-query-for-access-logs/208413/6 "2019-11-20T04:16:27Z")
Are you still getting the same error? Because I don't find anything wrong in the pattern that you mentioned, other that the missing `)` at the end in below: > ``` > \"%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int} > > ``` Regular expressions for grok patterns can be found [here](https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns). You can use kibana dev tools to check the grok pattern against the string (log line). ---
**Author:** ![Mehak\_Bhargava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehak_bhargava/32/54750_2.png) [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\ **Post date:** [November 20, 2019, 7:30pm UTC](https://discuss.elastic.co/t/grok-pattern-query-for-access-logs/208413/7 "2019-11-20T19:30:23Z")
Using the grok debugger in Kibana dev tools, I parsed this as you provided but it gives an error that " Provided grok pattern grok patterns do not match data in the input" ```auto %{IPORHOST:clientip} %{USER:ident} %{USER:auth} %{USER:extra} %{HTTPDATE:timestamp} \"%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response:int} (?:-|%{NUMBER:bytes:int}) ``` For this data- ```auto 192.168.10.182 - - - 18/Nov/2019:13:42:14 -0800 "GET /xxxxxxxxxxxxxxxx/xxxxxxxxxxx/data/discover/xxxxxxxxxxxxxxxx HTTP/1.1" ``` So there is a grok pattern issue? ---
**Author:** ![nishant.saini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nishant.saini/32/41489_2.png) [@nishant.saini](https://discuss.elastic.co/u/nishant.saini)\ **Post date:** [November 21, 2019, 3:28am UTC](https://discuss.elastic.co/t/grok-pattern-query-for-access-logs/208413/8 "2019-11-21T03:28:35Z")
There is difference between the log pattern and that is why the issue of not matching. Below are the two type of logs you are dealing with: ``` 192.168.10.182 - - - 18/Nov/2019:13:42:14 -0800 "GET /jdbc-data-server/jdbcdataserver/data/discover/TransactionsDemo HTTP/1.1" 200 4534 192.168.10.182 - - - 18/Nov/2019:13:42:14 -0800 "GET /xxxxxxxxxxxxxxxx/xxxxxxxxxxx/data/discover/xxxxxxxxxxxxxxxx HTTP/1.1" ``` Notice that in the second log `response` and `bytes` are missing and that is the reason you are getting the error: > " Provided grok pattern grok patterns do not match data in the input" Looking at the grok pattern, `%{NUMBER:response:int}` implies that `response` part is always expected to be present in the log line where as` (?:-|%{NUMBER:bytes:int})` implies that it expects numeric value or `-` for `bytes` To solve this you have to make sure that each log line has a fixed pattern or you can make the grok expressions optional as well. ---
**Author:** ![Mehak\_Bhargava](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mehak_bhargava/32/54750_2.png) [@Mehak\_Bhargava](https://discuss.elastic.co/u/Mehak_Bhargava)\ **Post date:** [November 21, 2019, 6:21pm UTC](https://discuss.elastic.co/t/grok-pattern-query-for-access-logs/208413/9 "2019-11-21T18:21:30Z")
It worked, thankyou!! but this is in dev tools. What if I want this parsed way of information on kibana discover tab? So where is this PUT query for pattern stored in? Which file in particular? ---
**Author:** ![nishant.saini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nishant.saini/32/41489_2.png) [@nishant.saini](https://discuss.elastic.co/u/nishant.saini)\ **Post date:** [November 22, 2019, 1:58am UTC](https://discuss.elastic.co/t/grok-pattern-query-for-access-logs/208413/10 "2019-11-22T01:58:56Z")
Go through the ingest node documentation [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest.html) and the subsequent topic to understand it's working. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 20, 2019, 1:59am UTC](https://discuss.elastic.co/t/grok-pattern-query-for-access-logs/208413/11 "2019-12-20T01:59:04Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.