# Grok Parsing an Internal JSON Part (IMAP Plugin)

**URL:** <https://discuss.elastic.co/t/grok-parsing-an-internal-json-part-imap-plugin/49471>\
**Category:** Logstash\
**Created:** [May 7, 2016, 9:32pm UTC](https://discuss.elastic.co/t/grok-parsing-an-internal-json-part-imap-plugin/49471 "2016-05-07T21:32:35Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![m0tek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/m0tek/32/9656_2.png) [@m0tek](https://discuss.elastic.co/u/m0tek)\
**Post date:** [May 7, 2016, 9:32pm UTC](https://discuss.elastic.co/t/grok-parsing-an-internal-json-part-imap-plugin/49471/1 "2016-05-07T21:32:35Z")

</div>

I Am familiar with Grok and Regexes , I Have the following Logstash Conf file :

Basically it uses the IMAP Plugin in order to read Emails that are sent into the mailbox , As you can see - it tries to parse out (grok) a specific data from the email in order to parse the JSON Part

The Plugin :

```
input {
imap {
host => "imap.gmail.com"
user => "user@pork.com"
password => "pass"
port => 993
secure => true
fetch_count => 100
check_interval => 10
}
} 

#Grokking the Message #
filter {
grok {
#match => {"message" => "Full Response\:\\n%{GREEDYDATA:json}\}"}
match => {"message" => "Full Response: %{GREEDYDATA:json}\}"}
#match => {"message" => "(?<json>Full Response:\\n(.|\r|\n)*)"}
break_on_match => false

}

json { source => "json"
add_tag => "Parsed"
}

output {
file {
path => "/tmp/emailtmp.log"
}
stdout {
codec => rubydebug }
}

```

For some reason i keep recieving \_\_grokparsefailure and the JSON is , ofcourse , not parsed - I Only need the JSON Part (After the Full Response)

Tried various ways , any idea ?

Thanks!

---

<div class="post-metadata">

**Author:** ![m0tek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/m0tek/32/9656_2.png) [@m0tek](https://discuss.elastic.co/u/m0tek)\
**Post date:** [May 7, 2016, 9:35pm UTC](https://discuss.elastic.co/t/grok-parsing-an-internal-json-part-imap-plugin/49471/2 "2016-05-07T21:35:22Z")

</div>

Email Body available here - \>

[http://pastebin.com/wtmJztB5](http://pastebin.com/wtmJztB5)

---

<div class="post-metadata">

**Author:** ![fbaligand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fbaligand/32/5657_2.png) [@fbaligand](https://discuss.elastic.co/u/fbaligand)\
**Post date:** [May 7, 2016, 10:33pm UTC](https://discuss.elastic.co/t/grok-parsing-an-internal-json-part-imap-plugin/49471/3 "2016-05-07T22:33:09Z")

</div>

grok doesn't like line returns.

So before the grok filter, I advice you to replace all line returns by a space.

---

<div class="post-metadata">

**Author:** ![m0tek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/m0tek/32/9656_2.png) [@m0tek](https://discuss.elastic.co/u/m0tek)\
**Post date:** [May 9, 2016, 8:54am UTC](https://discuss.elastic.co/t/grok-parsing-an-internal-json-part-imap-plugin/49471/4 "2016-05-09T08:54:53Z")

</div>

Hi ,  
Do you mean GSubbing /r/n to /s?

---

<div class="post-metadata">

**Author:** ![m0tek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/m0tek/32/9656_2.png) [@m0tek](https://discuss.elastic.co/u/m0tek)\
**Post date:** [May 9, 2016, 12:35pm UTC](https://discuss.elastic.co/t/grok-parsing-an-internal-json-part-imap-plugin/49471/5 "2016-05-09T12:35:59Z")

</div>

Ok , Managed Via this 🙂

```
filter {
mutate {
    gsub => [
    "message", "[\\?#-]", "",
    "message", "\n", " "
]}

grok { match => {"message" => "(?<json_raw>\{.+?\}})"}}

```

Now , I have the folowing Json in a field called json\_raw

Json looks escaped

IE

{"md5":"809e29c9765542645c882dddad35","internalID":3821800,"submitDate":"20160T02:40:48.0000700","versionID":"20160317\_3820030","batch":"api\_c14ca113752f98ce","url":" [http://www.porkalolza.co.il/עי1עמק\_%D](http://www.porkalolza.co.il/%D7%A2%D7%991%D7%A2%D7%9E%D7%A7_%25D)}

Any idea how to parse this ? using the json filter results in jsonparsefailure

---

<div class="post-metadata">

**Author:** ![fbaligand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fbaligand/32/5657_2.png) [@fbaligand](https://discuss.elastic.co/u/fbaligand)\
**Post date:** [May 9, 2016, 1:56pm UTC](https://discuss.elastic.co/t/grok-parsing-an-internal-json-part-imap-plugin/49471/6 "2016-05-09T13:56:39Z")

</div>

json filter is the right filter to do the job.

Looking to your json, it seems that a `"` miss at the end just before `}`

If it's not this, can you give the full error message that json filter raises.

---

<div class="post-metadata">

**Author:** ![m0tek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/m0tek/32/9656_2.png) [@m0tek](https://discuss.elastic.co/u/m0tek)\
**Post date:** [May 9, 2016, 2:52pm UTC](https://discuss.elastic.co/t/grok-parsing-an-internal-json-part-imap-plugin/49471/7 "2016-05-09T14:52:51Z")

</div>

Its not the full JSON - The Json looks like this :

[http://pastebin.com/53He3BPn](http://pastebin.com/53He3BPn)

---

<div class="post-metadata">

**Author:** ![fbaligand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fbaligand/32/5657_2.png) [@fbaligand](https://discuss.elastic.co/u/fbaligand)\
**Post date:** [May 9, 2016, 9:47pm UTC](https://discuss.elastic.co/t/grok-parsing-an-internal-json-part-imap-plugin/49471/8 "2016-05-09T21:47:07Z")

</div>

The problem is that your json is invalid.  
You have some values that are HTML, and where you have `"` as attribute delimiter, like here :  
`,\"location\":\"<iframe id=\"a61d045b\" name=\"a61d045b\"`

These html attribute delimiters (`"`) are in conflict with json value delimiters (`"`).

So, to work, you must transform each html value into a valid json value. For example, replacing `"` by `'`

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:58am UTC](https://discuss.elastic.co/t/grok-parsing-an-internal-json-part-imap-plugin/49471/9 "2017-07-06T04:58:29Z")

</div>

Closed on July 6, 2017, 4:58am UTC