# \_grokparsefailure is occurring even after the grok pattern is success in grokdebugger
**URL:** https://discuss.elastic.co/t/grokparsefailure-is-occurring-even-after-the-grok-pattern-is-success-in-grokdebugger/280784
**Category:** Logstash
**Created:** [August 9, 2021, 9:26am UTC](https://discuss.elastic.co/t/grokparsefailure-is-occurring-even-after-the-grok-pattern-is-success-in-grokdebugger/280784 "2021-08-09T09:26:50Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@vaseemQA](https://discuss.elastic.co/u/vaseemQA)
#### Post date: [August 9, 2021, 9:26am UTC](https://discuss.elastic.co/t/grokparsefailure-is-occurring-even-after-the-grok-pattern-is-success-in-grokdebugger/280784/1 "2021-08-09T09:26:50Z")
Hi Team,
I'm using logstash 6.8.3, and I'm trying to parse ES slow logs and my sample field is a ES source\_query which looks like
```auto
"source_query":{"from":0,"size":0,"post_filter":{"bool":{"must":[{"term":{" **someId**":{"value":1234,"boost":1.0}}},{"bool":{"must":[{"term":{"indexedAttributes.some_id.long":{"value":1234,"boost":1.0}}},{"term":{"deleted":{"value":"false","boost":1.0}}},{"term":{"someGroupIds":{"value":121221,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"version":true,"_source":{"includes":["orderId"],"excludes":},"sort":[{"sortedAttributes.lastUpdatedOn.date":{"order":"desc","missing":"_last","unmapped_type":"keyword"}}]}
```
and the grok pattern i've used is like below.
`^{\"from\":%{INT},\"size\":(%{DATA:totalSize}),%{DATA}{\"someId\":{\"value\":%{INT:someId},%{DATA}}$`
and it works perfect in [https://grokdebug.herokuapp.com/](https://grokdebug.herokuapp.com/), see below screenshot but however it returns "tags" =\> [
[0] "\_grokparsefailure"
] and i don't see the extracted fields **totalSize** and **someId**
Thanks,
Vaseem
---
### Author:  [@Cad](https://discuss.elastic.co/u/Cad)
#### Post date: [August 9, 2021, 1:16pm UTC](https://discuss.elastic.co/t/grokparsefailure-is-occurring-even-after-the-grok-pattern-is-success-in-grokdebugger/280784/2 "2021-08-09T13:16:21Z")
> [@vaseemQA](#):
>
> `" **someId**"`
This part don't respect your grok pattern.
Replace
```auto
^{\"from\":%{INT},\"size\":(%{DATA:totalSize}),%{DATA}{\"someId\":{\"value\":%{INT:someId},%{DATA}}$
```
With
```auto
^{%{QUOTEDSTRING}:%{INT},%{QUOTEDSTRING}:%{DATA:totalSize},%{DATA}{%{QUOTEDSTRING}:{%{QUOTEDSTRING}:%{INT:someId},%{DATA}}$
#I use quoted string and remove unecessary parentheses.
```
Your input respect the json format, use the json plugin should be the easiest way to index your values.
---
### Author:  [@vaseemQA](https://discuss.elastic.co/u/vaseemQA)
#### Post date: [August 9, 2021, 1:32pm UTC](https://discuss.elastic.co/t/grokparsefailure-is-occurring-even-after-the-grok-pattern-is-success-in-grokdebugger/280784/3 "2021-08-09T13:32:16Z")
Hey @Cad ,
Awesome, it worked, a big shout to you 🙂
Thank you so much for helping me out!.
---
### Author:  [@vaseemQA](https://discuss.elastic.co/u/vaseemQA)
#### Post date: [August 9, 2021, 2:43pm UTC](https://discuss.elastic.co/t/grokparsefailure-is-occurring-even-after-the-grok-pattern-is-success-in-grokdebugger/280784/4 "2021-08-09T14:43:17Z")
Hey @Cad ,
Yes you are right, I think I should use json plugin, but from the entire json I'm interested in only one key value and that json path is not static in nature, how can I solve this, any thoughts?
Thanks in Advance!
Vaseem.
---
### Author:  [@vaseemQA](https://discuss.elastic.co/u/vaseemQA)
#### Post date: [August 10, 2021, 8:13am UTC](https://discuss.elastic.co/t/grokparsefailure-is-occurring-even-after-the-grok-pattern-is-success-in-grokdebugger/280784/5 "2021-08-10T08:13:04Z")
Hey @Cad ,
I've used Json filter, and I'm trying to extract another josn object field value with in it, its basically like below:
however, I got an exception like below
```auto
[2021-08-10T11:36:03,605][WARN][logstash.filters.json] Error parsing json {:source=>"query", :raw=>{"bool"=>{"must"=>[{"term"=>{"someId"=>{"value"=>12345, "boost"=>0.1e1}}}, {"term"=>{"active"=>{"value"=>true, "boost"=>0.1e1}}}, {"term"=>{"deleted"=>{"value"=>false, "boost"=>0.1e1}}}], "adjust_pure_negative"=>true, "boost"=>0.1e1}}, :exception=>java.lang.ClassCastException: org.jruby.RubyHash cannot be cast to org.jruby.RubyIO}
```
My configuration look like this
```auto
if "_grokparsefailure" in [tags] {
json {
source => "source_query"
target => "parsedJson"
}
json {
source => "query"
#add_field => { "json_someId" => "%{[query][bool][must[0]][term][busId][value]}" }
}
}
```
It will be grateful for me, if you can help me in this
Thanks,
Vaseem
---
### Author:  [@vaseemQA](https://discuss.elastic.co/u/vaseemQA)
#### Post date: [August 10, 2021, 8:50am UTC](https://discuss.elastic.co/t/grokparsefailure-is-occurring-even-after-the-grok-pattern-is-success-in-grokdebugger/280784/6 "2021-08-10T08:50:19Z")
Hey @Cad ,
Finally I got the fix for the exception
```auto
[2021-08-10T11:36:03,605][WARN][logstash.filters.json] Error parsing json {:source=>"query", :raw=>{"bool"=>{"must"=>[{"term"=>{"someId"=>{"value"=>12345, "boost"=>0.1e1}}}, {"term"=>{"active"=>{"value"=>true, "boost"=>0.1e1}}}, {"term"=>{"deleted"=>{"value"=>false, "boost"=>0.1e1}}}], "adjust_pure_negative"=>true, "boost"=>0.1e1}}, :exception=>java.lang.ClassCastException: org.jruby.RubyHash cannot be cast to org.jruby.RubyIO}
```
, I used below code block in input filter and the issue got fixed.
```auto
codec => json{
charset => "UTF-8"
}
```
However I'm unable to extract the key value from the parsed josn, and this is how I'm trying to extract, can you please put me in right path
```auto
json {
source => "query"
add_field => { "json_someId" => "%{[query][bool][must[0]][term][busId][value]}" }
}
```
---
### Author:  [@Cad](https://discuss.elastic.co/u/Cad)
#### Post date: [August 10, 2021, 9:09am UTC](https://discuss.elastic.co/t/grokparsefailure-is-occurring-even-after-the-grok-pattern-is-success-in-grokdebugger/280784/7 "2021-08-10T09:09:52Z")
Hi,
> [@vaseemQA](#):
>
> `add_field => { "json_someId" => "%{[query][bool][must[0]][term][busId][value]}" }`
According to the data you show us, `term` have a nested field named `someId` not `busId`.
And to access to a specific array position, you have to use `[must][0]` instead of `[must[0]]`
Here the final line:
`add_field => { "json_someId" => "%{[query][bool][must][0][term][someId][value]}" }`
---
### Author:  [@vaseemQA](https://discuss.elastic.co/u/vaseemQA)
#### Post date: [August 10, 2021, 9:36am UTC](https://discuss.elastic.co/t/grokparsefailure-is-occurring-even-after-the-grok-pattern-is-success-in-grokdebugger/280784/8 "2021-08-10T09:36:46Z")
Hey @Cad ,
I tried the above code but it doesn't throw any error or doesn't add the requested filed, any thoughts please?
---
### Author:  [@Cad](https://discuss.elastic.co/u/Cad)
#### Post date: [August 10, 2021, 10:13am UTC](https://discuss.elastic.co/t/grokparsefailure-is-occurring-even-after-the-grok-pattern-is-success-in-grokdebugger/280784/9 "2021-08-10T10:13:49Z")
Can you show us the complet conf file ?
---
### Author:  [@vaseemQA](https://discuss.elastic.co/u/vaseemQA)
#### Post date: [August 10, 2021, 10:21am UTC](https://discuss.elastic.co/t/grokparsefailure-is-occurring-even-after-the-grok-pattern-is-success-in-grokdebugger/280784/10 "2021-08-10T10:21:32Z")
Hey @Cad ,
It's fixed, I used this code now and it worked, and thank you so much for your kind and quick response 🙂
```auto
if "_grokparsefailure" in [tags] {
json {
source => "source_query"
target => "parsedJson"
add_field => { "someId" => "%{[parsedJson][query][bool][must][0][term][someId][value]}"
}
}
```
---
### Author:  [@vaseemQA](https://discuss.elastic.co/u/vaseemQA)
#### Post date: [August 12, 2021, 9:46am UTC](https://discuss.elastic.co/t/grokparsefailure-is-occurring-even-after-the-grok-pattern-is-success-in-grokdebugger/280784/11 "2021-08-12T09:46:13Z")
Hey @Cad ,
I'm trying to calculate the time taken to parse by JSON filter but the given field is not being added in the output, can you please see if there is anything wrong with my code
```auto
ruby {
code => "event.set('before_json_processed_at', Time.now().nsec)"
}
json {
source => "source_query"
target => "parsedJson"
add_field => { "someBusid" => "%{[parsedJson][query][bool][must][0][term][someBusid][value]}" }
}
ruby {
code => 'event.set("after_json_processed_at", Time.now().nsec)
duration_in_n=event.get("after_json_processed_at")-event.get("before_json_processed_at")
event.set("duration_in_nanos", duration_in_n )'
}
```
---
### Author:  [@Cad](https://discuss.elastic.co/u/Cad)
#### Post date: [August 12, 2021, 10:52am UTC](https://discuss.elastic.co/t/grokparsefailure-is-occurring-even-after-the-grok-pattern-is-success-in-grokdebugger/280784/12 "2021-08-12T10:52:46Z")
Hi,
> [@vaseemQA](#):
>
> the given field is not being added
Which one is not added ?
Do you see the other two ?
---
### Author:  [@vaseemQA](https://discuss.elastic.co/u/vaseemQA)
#### Post date: [August 12, 2021, 5:24pm UTC](https://discuss.elastic.co/t/grokparsefailure-is-occurring-even-after-the-grok-pattern-is-success-in-grokdebugger/280784/13 "2021-08-12T17:24:16Z")
Hey @Cad ,
No I don't see either of them.
Thanks you,
Vaseem
---
### Author:  [@vaseemQA](https://discuss.elastic.co/u/vaseemQA)
#### Post date: [August 13, 2021, 7:10am UTC](https://discuss.elastic.co/t/grokparsefailure-is-occurring-even-after-the-grok-pattern-is-success-in-grokdebugger/280784/14 "2021-08-13T07:10:39Z")
Hey @Cad ,
I found my self the solution BUT not with the above code, However I'm using the awesome endpoint provided by the Elasticsearch [Node Stats API | Logstash Reference [8.11] | Elastic](https://www.elastic.co/guide/en/logstash/current/node-stats-api.html)
And I've used.
```auto
localhost:9600/_node/stats/pipelines?pretty
```
And I got the output like this,
```auto
{
"id": "a1756021d535d74b0f29412d959bb091cbddf78bce667bba8c3a4bd82b169cc2",
"failures": 2,
"events": {
"out": 927,
"duration_in_millis": 1274,
"in": 927
},
"name": "grok",
"matches": 925,
"patterns_per_field": {
"source_query": 1
}
},
{
"id": "84f54948542022bfb6ea79821ff56800a92d34848b7f69577928773286794507",
"events": {
"out": 2,
"duration_in_millis": 39,
"in": 2
},
"name": "json"
}
```
from the above output, My understanding is, the event json took total 2 records and parsed/transformed/processed it in 39 milliseconds which is basically 39/2 = 19.5 milliseconds for each document , Can you please correct me if my understanding is not correct?
Also, can you please tell me the difference between these
> duration\_in\_millis
Vs
> queue\_push\_duration\_in\_millis
Thanks,
Vaseem
---
### Author:  [@Cad](https://discuss.elastic.co/u/Cad)
#### Post date: [August 13, 2021, 10:16am UTC](https://discuss.elastic.co/t/grokparsefailure-is-occurring-even-after-the-grok-pattern-is-success-in-grokdebugger/280784/15 "2021-08-13T10:16:12Z")
Hi vaseemQA,
> [@vaseemQA](#):
>
> My understanding is, the event json took total 2 records and parsed/transformed/processed it in 39 milliseconds which is basically 39/2 = 19.5 milliseconds for each document
I think so, just to be more precise, it's an average per event
> [@vaseemQA](#):
>
> Also, can you please tell me the difference between these
>
> > duration\_in\_millis
>
> Vs
>
> > queue\_push\_duration\_in\_millis
I never use this API but according to the documentation, `queue_push_duration_in_millis` seems to be related to [Persistent queues](https://www.elastic.co/guide/en/logstash/current/persistent-queues.html#persistent-queues).
[This answer](https://discuss.elastic.co/t/measuring-the-taking-time/113900/4) from a previous topic seems pretty accurate about what they means and about my understanding from the documentation.
Sorry for not being able to be more helpful on this subject
Cad.
---
### Author:  [@vaseemQA](https://discuss.elastic.co/u/vaseemQA)
#### Post date: [August 18, 2021, 4:37pm UTC](https://discuss.elastic.co/t/grokparsefailure-is-occurring-even-after-the-grok-pattern-is-success-in-grokdebugger/280784/16 "2021-08-18T16:37:36Z")
Hey @Cad ,
I need help on json filter plugin, the thing is I've the capture a value from one json path but some times the json pattern changes so, how can we achieve that do we have any fallback for that?
for example see below code please.
```auto
json {
source => "source_query"
target => "parsedJson"
add_field => { "es_someId" => "%{[parsedJson][query][bool][must][0][term][someId][value]}"}
if "es_someId" =~ "parsedJson" {
update => {"es_someId" => "%{[parsedJson][query][bool][filter][0][bool][must][3][bool][should][0][term][someId][value]}"}
}
remove_field => [tags]
}
```
but is giving failed exception as like below
```auto
Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 69, column 16 (byte 2013) after filter
```
and in the line number 69 , there is a if block statement which is ` if "es_someId" =~ "parsedJson" {` as shown in above code example.
can you please help me
Thanks in Advance,
Vaseem
---
### Author:  [@Cad](https://discuss.elastic.co/u/Cad)
#### Post date: [August 19, 2021, 10:20am UTC](https://discuss.elastic.co/t/grokparsefailure-is-occurring-even-after-the-grok-pattern-is-success-in-grokdebugger/280784/17 "2021-08-19T10:20:24Z")
> [@vaseemQA](#):
>
> `if "es_someId" =~ "parsedJson" {`
I think here you want to check the content of es\_someId. In a condition, to access to a field value you have to use this syntax :
`if [es_someId] =~ "parsedJson" {`
---
### Author:  [@vaseemQA](https://discuss.elastic.co/u/vaseemQA)
#### Post date: [August 20, 2021, 10:11am UTC](https://discuss.elastic.co/t/grokparsefailure-is-occurring-even-after-the-grok-pattern-is-success-in-grokdebugger/280784/18 "2021-08-20T10:11:28Z")
Hey @Cad,
Thanks for that, can you look at the below poblem please
Problem: Trying to extract the key value pair from a json BUT the JSON is not static in nature it may have number of patterns like expected key pair value json path will be changed dynamically, so in grok we use
> break\_on\_match=\>true
it helps us to check the match against the number of patterns and if matched one of the pattern then it wont check for another one, similarly do we have anything to match against different jsonPaths for a value?
Thanks in Advance,
Vaseem
---
### Author:  [@Cad](https://discuss.elastic.co/u/Cad)
#### Post date: [August 20, 2021, 1:15pm UTC](https://discuss.elastic.co/t/grokparsefailure-is-occurring-even-after-the-grok-pattern-is-success-in-grokdebugger/280784/19 "2021-08-20T13:15:13Z")
> [@vaseemQA](#):
>
> similarly do we have anything to match against different jsonPaths for a value?
I'm not aware of an option who can do that.
But you can try to use the ruby filter to browse your fields and found the data you want.
---
### Author:  [@vaseemQA](https://discuss.elastic.co/u/vaseemQA)
#### Post date: [August 21, 2021, 6:56am UTC](https://discuss.elastic.co/t/grokparsefailure-is-occurring-even-after-the-grok-pattern-is-success-in-grokdebugger/280784/20 "2021-08-21T06:56:58Z")
Hey @Cad ,
Yeah finally I've moved to ruby instead JSON plugin as the JSON is not static in nature and below is the logic I used , let me know if there is any better way.
```auto
if "_grokparsefailure" in [tags]{
ruby{
code => 'event.get("message")
if event.get("message").include? "someId"
someIdIndexStarts=event.get("message").index("someId")
someIdValueIndexStart=someIdIndexStarts+19
lengthOfTotalString=event.get("message").length
for i in someIdValueIndexStart..lengthOfTotalString
if event.get("message").slice(i) == ","
endOfSomeIdValueIndex=i
event.set("es_someId",event.get("message").slice(someIdValueIndexStart..endOfSomeIdValueIndex-1));
break;
end
end
end'
}
}
```
Thanks for your support all the time @Cad
Vaseem,
[Next page](https://discuss.elastic.co/t/grokparsefailure-is-occurring-even-after-the-grok-pattern-is-success-in-grokdebugger/280784.md?page=2)