Grok Parsing an Internal JSON Part (IMAP Plugin)

I Am familiar with Grok and Regexes , I Have the following Logstash Conf file :

Basically it uses the IMAP Plugin in order to read Emails that are sent into the mailbox , As you can see - it tries to parse out (grok) a specific data from the email in order to parse the JSON Part

The Plugin :

input {
imap {
host => ""
user => ""
password => "pass"
port => 993
secure => true
fetch_count => 100
check_interval => 10

#Grokking the Message #
filter {
grok {
#match => {"message" => "Full Response\:\\n%{GREEDYDATA:json}\}"}
match => {"message" => "Full Response: %{GREEDYDATA:json}\}"}
#match => {"message" => "(?<json>Full Response:\\n(.|\r|\n)*)"}
break_on_match => false


json { source => "json"
add_tag => "Parsed"

output {
file {
path => "/tmp/emailtmp.log"
stdout {
codec => rubydebug }

For some reason i keep recieving __grokparsefailure and the JSON is , ofcourse , not parsed - I Only need the JSON Part (After the Full Response)

Tried various ways , any idea ?


Email Body available here - >

grok doesn't like line returns.

So before the grok filter, I advice you to replace all line returns by a space.

Hi ,
Do you mean GSubbing /r/n to /s?

Ok , Managed Via this :slight_smile:

filter {
mutate {
    gsub => [
    "message", "[\\?#-]", "",
    "message", "\n", " "

grok { match => {"message" => "(?<json_raw>\{.+?\}})"}}

Now , I have the folowing Json in a field called json_raw

Json looks escaped



Any idea how to parse this ? using the json filter results in jsonparsefailure

json filter is the right filter to do the job.

Looking to your json, it seems that a " miss at the end just before }

If it's not this, can you give the full error message that json filter raises.

Its not the full JSON - The Json looks like this :

The problem is that your json is invalid.
You have some values that are HTML, and where you have " as attribute delimiter, like here :
,\"location\":\"<iframe id=\"a61d045b\" name=\"a61d045b\"

These html attribute delimiters (") are in conflict with json value delimiters (").

So, to work, you must transform each html value into a valid json value. For example, replacing " by '